plugout.net

by **samrise** » Thu Aug 20, 2015 11:10 pm

Hi guys!

Every now and then i'm having some out of memory error. When that happens kernel kills one of the process to get memory back. If it choses the wrong process i lose access to nas (for example, sshd or mdadm).

This link shows how i can disable out of memory killer. The problem is that using sysctl -w vm.overcommit_memory=2 is lost after reboot and i dont have a /etc/systcl.conf file. I created that file but vm.overcommit_memory is always back to 0 after reboot.

:twak

Edit: I forgot the link: http://serverfault.com/questions/141988 ... n-teardown

by **fvdw** » Fri Aug 21, 2015 8:06 am

Never seen that before, is the swap active? (It should use sda3)
You should be able to see it using the command

Code: Select all: free

About the other issue, the firmware has a function to let you execute script and special commands at each boot. It is in task planner

by **samrise** » Fri Aug 21, 2015 9:44 am

Code: Select all: root@nas:/ # free total used free shared buffers Mem: 125848 123588 2260 0 37168 -/+ buffers: 86420 39428 Swap: 514076 0 514076

Is this regular behavior? Mem almost full and Swap empty?

by **fvdw** » Fri Aug 21, 2015 10:26 am

not it is not normal but load a few applications (mediaserver or so) more and see if it starts to use it

on my nwsp2 I see this

Code: Select all: root@nwsp2-3:/ # free total used free shared buffers Mem: 250404 228404 22000 0 43788 -/+ buffers: 184616 65788 Swap: 514076 148 513928 root@nwsp2-3:/ # root@nwsp2-3:/ # cat /proc/swaps Filename Type Size Used Priority /dev/sda3 partition 514076 148 -1 root@nwsp2-3:/ # swapon -s Filename Type Size Used Priority /dev/sda3 partition 514076 148 -1 root@nwsp2-3:/ #

not spectacular but not zero

also the top command is handy to see whats going on

Code: Select all: top - 12:25:47 up 26 days, 18:55, 1 user, load average: 0.34, 0.35, 0.33 Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie Cpu(s): 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 250404k total, 230288k used, 20116k free, 44008k buffers Swap: 514076k total, 148k used, 513928k free, 145044k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3476 root 20 0 2428 1116 916 R 0.7 0.4 0:00.13 top 1508 root 20 0 24896 12m 2056 S 0.3 5.2 334:53.72 twonkyserver 3430 root 20 0 2288 896 732 S 0.3 0.4 0:00.97 dropbear 1 root 20 0 3032 68 0 S 0.0 0.0 0:01.75 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.59 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 1:00.14 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0.0 0.0 2:14.06 rcu_preempt 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_sched 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 241 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 243 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 245 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 250 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff 261 root 20 0 0 0 0 S 0.0 0.0 0:00.04 khubd 268 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md

by **samrise** » Fri Aug 21, 2015 11:18 am

Code: Select all: Mem: 125848k total, 123708k used, 2140k free, 18316k buffers Swap: 514076k total, 0k used, 514076k free, 68236k cached

I've been checking top and i think this is the lowest i manage to catch. Only 2140k free and still not using swap.

by **fvdw** » Fri Aug 21, 2015 9:56 pm

on my 5big2 which has 512 MB RAM the swap is also 0

but what does this command tells you

Code: Select all: cat /proc/sys/vm/swappiness

swappiness can have a value of between 0 and 100
swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible
swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

Typically a value is 60 is used (should be also the value on your system)
However you should have to adapt it, but of course you could try a higher value.
Higher values will reduce performance as swapping takes cpu and disk time

by **samrise** » Sat Aug 22, 2015 1:17 pm

My swappiness is 60 and after some further tests seems all is correct with swap.

I think i found out what is wrong. Seems to be fan-d.

Dmesg:

Code: Select all: [ 6055.126195] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6055.133442] CPU: 0 PID: 840 Comm: fan-d Not tainted 3.14.2 #1 [ 6055.139211] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6055.146984] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6055.155153] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6055.163898] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6055.172095] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6055.180810] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6055.189603] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6055.197257] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6055.204898] [<c0018b0c>] (mm_init.isra.68) from [<c001970c>] (copy_process.part.73+0x6f8/0x1140) [ 6055.213758] [<c001970c>] (copy_process.part.73) from [<c001a2a0>] (do_fork+0xb4/0x230) [ 6055.221690] [<c001a2a0>] (do_fork) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6055.229060] Mem-info: [ 6055.231351] Normal per-cpu: [ 6055.234145] CPU 0: hi: 42, btch: 7 usd: 0 [ 6055.238942] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6055.238942] active_file:193 inactive_file:242 isolated_file:0 [ 6055.238942] unevictable:0 dirty:0 writeback:178 unstable:0 [ 6055.238942] free:14550 slab_reclaimable:224 slab_unreclaimable:13326 [ 6055.238942] mapped:0 shmem:0 pagetables:132 bounce:0 [ 6055.238942] free_cma:0 [ 6055.269570] Normal free:58200kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:772kB inactive_file:968kB unevictable:0kB isolated(a$ [ 6055.310433] lowmem_reserve[]: 0 0 [ 6055.313766] Normal: 5972*4kB (EM) 4289*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 58200kB [ 6055.325613] 435 total pagecache pages [ 6055.329270] 0 pages in swap cache [ 6055.332603] Swap cache stats: add 2534, delete 2534, find 400/697 [ 6055.338676] Free swap = 507152kB [ 6055.341997] Total swap = 514076kB [ 6055.347844] 32768 pages of RAM [ 6055.350920] 14723 free pages [ 6055.353795] 1306 reserved pages [ 6055.356923] 13452 slab pages [ 6055.359858] 67009 pages shared [ 6055.362902] 0 pages swap cached [ 6055.366031] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6055.373898] [ 840] 0 840 422 0 4 46 0 fan-d [ 6055.381874] [ 861] 0 861 504 0 5 31 0 mdadm [ 6055.389800] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6055.397699] [ 1432] 0 1432 4187 0 11 587 0 httpd [ 6055.405612] [ 1433] 99 1433 4207 0 11 593 0 httpd [ 6055.413525] [ 1434] 99 1434 4207 0 11 593 0 httpd [ 6055.421437] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6055.429613] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6055.437425] [ 1452] 0 1452 2259 0 7 155 0 smbd [ 6055.445250] [ 1453] 0 1453 2259 0 7 155 0 smbd [ 6055.453079] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6055.461338] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6055.469599] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6055.477670] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6055.485841] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6055.494190] [ 1527] 0 1527 649 0 5 141 0 rpc.statd [ 6055.502449] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6055.510363] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6055.518694] [ 1882] 0 1882 813 0 5 20 0 sh [ 6055.526345] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6055.534519] [ 1884] 0 1884 813 0 5 22 0 sh [ 6055.542171] [ 1977] 0 1977 636 0 5 56 0 top [ 6055.549903] Out of memory: Kill process 1433 (httpd) score 3 or sacrifice child [ 6055.557189] Killed process 1433 (httpd) total-vm:16828kB, anon-rss:0kB, file-rss:0kB [ 6058.442480] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6058.449734] CPU: 0 PID: 840 Comm: fan-d Not tainted 3.14.2 #1 [ 6058.455501] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6058.463265] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6058.471373] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6058.480002] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6058.488172] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6058.496883] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6058.505679] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6058.513350] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6058.520935] [<c0018b0c>] (mm_init.isra.68) from [<c001970c>] (copy_process.part.73+0x6f8/0x1140) [ 6058.529727] [<c001970c>] (copy_process.part.73) from [<c001a2a0>] (do_fork+0xb4/0x230) [ 6058.537628] [<c001a2a0>] (do_fork) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6058.545016] Mem-info: [ 6058.547287] Normal per-cpu: [ 6058.550097] CPU 0: hi: 42, btch: 7 usd: 0 [ 6058.554898] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6058.554898] active_file:132 inactive_file:136 isolated_file:0 [ 6058.554898] unevictable:0 dirty:0 writeback:9 unstable:0 [ 6058.554898] free:15026 slab_reclaimable:217 slab_unreclaimable:13323 [ 6058.554898] mapped:0 shmem:0 pagetables:121 bounce:0 [ 6058.554898] free_cma:0 [ 6058.585347] Normal free:60104kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:528kB inactive_file:544kB unevictable:0kB isolated(a$ [ 6058.626081] lowmem_reserve[]: 0 0 [ 6058.629408] Normal: 5998*4kB (UEM) 4472*8kB (UEM) 19*16kB (M) 1*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60104kB [ 6058.642137] 268 total pagecache pages [ 6058.645795] 0 pages in swap cache [ 6058.649104] Swap cache stats: add 3121, delete 3121, find 811/1409 [ 6058.655275] Free swap = 507296kB [ 6058.658577] Total swap = 514076kB [ 6058.664411] 32768 pages of RAM [ 6058.667460] 15198 free pages [ 6058.670357] 1306 reserved pages [ 6058.673489] 13442 slab pages [ 6058.676359] 131536 pages shared [ 6058.679503] 0 pages swap cached [ 6058.682633] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6058.690481] [ 840] 0 840 422 0 4 46 0 fan-d [ 6058.698378] [ 861] 0 861 504 0 5 31 0 mdadm [ 6058.706294] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6058.714205] [ 1432] 0 1432 4187 0 11 587 0 httpd [ 6058.722117] [ 1434] 99 1434 4207 0 11 593 0 httpd [ 6058.730032] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6058.738190] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6058.746016] [ 1452] 0 1452 2259 0 7 155 0 smbd [ 6058.753849] [ 1453] 0 1453 2259 0 7 155 0 smbd [ 6058.761670] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6058.769932] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6058.778177] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6058.786260] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6058.794435] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6058.802781] [ 1527] 0 1527 649 0 5 141 0 rpc.statd [ 6058.811042] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6058.818939] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6058.827285] [ 1882] 0 1882 813 0 5 20 0 sh [ 6058.834946] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6058.843121] [ 1884] 0 1884 813 0 5 22 0 sh [ 6058.850773] [ 1977] 0 1977 636 0 5 56 0 top [ 6058.858499] Out of memory: Kill process 1434 (httpd) score 3 or sacrifice child [ 6058.865806] Killed process 1434 (httpd) total-vm:16828kB, anon-rss:0kB, file-rss:0kB [ 6058.926386] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6058.933729] CPU: 0 PID: 840 Comm: fan-d Not tainted 3.14.2 #1 [ 6058.939532] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6058.947279] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6058.955404] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6058.964040] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6058.972227] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6058.980939] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6058.989733] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6058.997387] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6059.004977] [<c0018b0c>] (mm_init.isra.68) from [<c001970c>] (copy_process.part.73+0x6f8/0x1140) [ 6059.013764] [<c001970c>] (copy_process.part.73) from [<c001a2a0>] (do_fork+0xb4/0x230) [ 6059.021688] [<c001a2a0>] (do_fork) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6059.029053] Mem-info: [ 6059.031344] Normal per-cpu: [ 6059.034137] CPU 0: hi: 42, btch: 7 usd: 5 [ 6059.038936] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6059.038936] active_file:129 inactive_file:131 isolated_file:0 [ 6059.038936] unevictable:0 dirty:0 writeback:0 unstable:0 [ 6059.038936] free:15041 slab_reclaimable:217 slab_unreclaimable:13323 [ 6059.038936] mapped:0 shmem:0 pagetables:110 bounce:0 [ 6059.038936] free_cma:0 [ 6059.069387] Normal free:60164kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:516kB inactive_file:524kB unevictable:0kB isolated(a$ [ 6059.110039] lowmem_reserve[]: 0 0 [ 6059.113369] Normal: 5995*4kB (UEM) 4481*8kB (UEM) 21*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60164kB [ 6059.125821] 261 total pagecache pages [ 6059.129493] 0 pages in swap cache [ 6059.132806] Swap cache stats: add 3122, delete 3122, find 812/1411 [ 6059.138962] Free swap = 507452kB [ 6059.142285] Total swap = 514076kB [ 6059.148097] 32768 pages of RAM [ 6059.151169] 15216 free pages [ 6059.154038] 1306 reserved pages [ 6059.157166] 13442 slab pages [ 6059.160048] 131530 pages shared [ 6059.163179] 0 pages swap cached [ 6059.166309] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6059.174158] [ 840] 0 840 422 0 4 46 0 fan-d [ 6059.182068] [ 861] 0 861 504 0 5 31 0 mdadm [ 6059.189986] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6059.197881] [ 1432] 0 1432 4187 0 11 587 0 httpd [ 6059.205796] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6059.213967] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6059.221792] [ 1452] 0 1452 2259 0 7 155 0 smbd [ 6059.229622] [ 1453] 0 1453 2259 0 7 155 0 smbd [ 6059.237436] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6059.245694] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6059.253952] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6059.262037] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6059.270212] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6059.278544] [ 1527] 0 1527 649 0 5 141 0 rpc.statd [ 6059.286806] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6059.294717] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6059.303062] [ 1882] 0 1882 813 0 5 20 0 sh [ 6059.310715] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6059.318866] [ 1884] 0 1884 813 0 5 22 0 sh [ 6059.326520] [ 1977] 0 1977 636 0 5 56 0 top [ 6059.334259] Out of memory: Kill process 1432 (httpd) score 3 or sacrifice child [ 6059.341565] Killed process 1432 (httpd) total-vm:16748kB, anon-rss:0kB, file-rss:0kB [ 6059.815760] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6059.823111] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6059.828968] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6059.836745] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6059.844860] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6059.853484] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6059.861672] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6059.870384] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6059.879159] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6059.886836] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6059.894421] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6059.902269] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6059.909839] Mem-info: [ 6059.912114] Normal per-cpu: [ 6059.914907] CPU 0: hi: 42, btch: 7 usd: 0 [ 6059.919724] active_anon:0 inactive_anon:5 isolated_anon:0 [ 6059.919724] active_file:126 inactive_file:138 isolated_file:0 [ 6059.919724] unevictable:0 dirty:0 writeback:5 unstable:0 [ 6059.919724] free:14973 slab_reclaimable:211 slab_unreclaimable:13363 [ 6059.919724] mapped:0 shmem:0 pagetables:103 bounce:0 [ 6059.919724] free_cma:0 [ 6059.950174] Normal free:59892kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:20kB active_file:504kB inactive_file:552kB unevictable:0kB isolated($ [ 6059.991040] lowmem_reserve[]: 0 0 [ 6059.994374] Normal: 5939*4kB (UEM) 4503*8kB (EM) 7*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59892kB [ 6060.006687] 269 total pagecache pages [ 6060.010360] 5 pages in swap cache [ 6060.013671] Swap cache stats: add 3197, delete 3192, find 863/1506 [ 6060.019842] Free swap = 509860kB [ 6060.023144] Total swap = 514076kB [ 6060.028966] 32768 pages of RAM [ 6060.032046] 15140 free pages [ 6060.034920] 1306 reserved pages [ 6060.038049] 13476 slab pages [ 6060.040940] 66148 pages shared [ 6060.043985] 5 pages swap cached [ 6060.047123] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6060.054971] [ 840] 0 840 422 0 4 46 0 fan-d [ 6060.062883] [ 861] 0 861 504 0 5 31 0 mdadm [ 6060.070795] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6060.078694] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6060.086866] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6060.094693] [ 1452] 0 1452 2259 0 7 155 0 smbd [ 6060.102518] [ 1453] 0 1453 2259 0 7 155 0 smbd [ 6060.110353] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6060.118595] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6060.126853] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6060.134940] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6060.143112] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6060.151464] [ 1527] 0 1527 649 0 5 141 0 rpc.statd [ 6060.159722] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6060.167618] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6060.175962] [ 1882] 0 1882 813 0 5 20 0 sh [ 6060.183616] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6060.191787] [ 1884] 0 1884 813 0 5 22 0 sh [ 6060.199422] [ 1977] 0 1977 636 0 5 56 0 top [ 6060.207163] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6060.215068] Out of memory: Kill process 1452 (smbd) score 0 or sacrifice child [ 6060.222286] Killed process 1453 (smbd) total-vm:9036kB, anon-rss:0kB, file-rss:0kB [ 6060.263589] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6060.270979] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6060.276840] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6060.284623] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6060.292732] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6060.301354] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6060.309547] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6060.318236] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6060.327033] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6060.334706] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6060.342296] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6060.350058] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6060.357600] Mem-info: [ 6060.359889] Normal per-cpu: [ 6060.362684] CPU 0: hi: 42, btch: 7 usd: 0 [ 6060.367483] active_anon:6 inactive_anon:6 isolated_anon:0 [ 6060.367483] active_file:127 inactive_file:128 isolated_file:0 [ 6060.367483] unevictable:0 dirty:0 writeback:0 unstable:0 [ 6060.367483] free:15034 slab_reclaimable:211 slab_unreclaimable:13363 [ 6060.367483] mapped:0 shmem:0 pagetables:96 bounce:0 [ 6060.367483] free_cma:0 [ 6060.397846] Normal free:60136kB min:1416kB low:1768kB high:2124kB active_anon:24kB inactive_anon:24kB active_file:508kB inactive_file:512kB unevictable:0kB isolated$ [ 6060.438664] lowmem_reserve[]: 0 0 [ 6060.442012] Normal: 5992*4kB (UEM) 4507*8kB (UEM) 7*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60136kB [ 6060.454376] 266 total pagecache pages [ 6060.458034] 11 pages in swap cache [ 6060.461446] Swap cache stats: add 3209, delete 3198, find 869/1518 [ 6060.467605] Free swap = 510024kB [ 6060.470926] Total swap = 514076kB [ 6060.476746] 32768 pages of RAM [ 6060.479823] 15198 free pages [ 6060.482698] 1306 reserved pages [ 6060.485825] 13476 slab pages [ 6060.488694] 66145 pages shared [ 6060.491750] 11 pages swap cached [ 6060.494969] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6060.502817] [ 840] 0 840 422 0 4 46 0 fan-d [ 6060.510727] [ 861] 0 861 504 0 5 31 0 mdadm [ 6060.518629] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6060.526540] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6060.534712] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6060.542537] [ 1452] 0 1452 2259 0 7 155 0 smbd [ 6060.550368] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6060.558613] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6060.566870] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6060.574958] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6060.583131] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6060.591474] [ 1527] 0 1527 649 0 5 141 0 rpc.statd [ 6060.599741] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6060.607636] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6060.615981] [ 1882] 0 1882 813 0 5 20 0 sh [ 6060.623643] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6060.631815] [ 1884] 0 1884 813 0 5 22 0 sh [ 6060.639457] [ 1977] 0 1977 636 6 5 50 0 top [ 6060.647199] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6060.655104] Out of memory: Kill process 1452 (smbd) score 0 or sacrifice child [ 6060.662318] Killed process 1452 (smbd) total-vm:9036kB, anon-rss:0kB, file-rss:0kB [ 6061.316166] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6061.323545] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6061.329543] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6061.337285] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6061.345404] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6061.354027] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6061.362216] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6061.370927] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6061.379720] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6061.387375] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6061.394964] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6061.402720] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6061.410282] Mem-info: [ 6061.412555] Normal per-cpu: [ 6061.415348] CPU 0: hi: 42, btch: 7 usd: 0 [ 6061.420164] active_anon:10 inactive_anon:12 isolated_anon:0 [ 6061.420164] active_file:166 inactive_file:208 isolated_file:0 [ 6061.420164] unevictable:0 dirty:0 writeback:12 unstable:0 [ 6061.420164] free:14921 slab_reclaimable:211 slab_unreclaimable:13363 [ 6061.420164] mapped:0 shmem:0 pagetables:89 bounce:0 [ 6061.420164] free_cma:0 [ 6061.450786] Normal free:59684kB min:1416kB low:1768kB high:2124kB active_anon:40kB inactive_anon:48kB active_file:664kB inactive_file:832kB unevictable:0kB isolated$ [ 6061.491690] lowmem_reserve[]: 0 0 [ 6061.495020] Normal: 5863*4kB (UEM) 4509*8kB (UEM) 10*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59684kB [ 6061.507472] 394 total pagecache pages [ 6061.511144] 20 pages in swap cache [ 6061.514543] Swap cache stats: add 3303, delete 3283, find 950/1658 [ 6061.520713] Free swap = 510648kB [ 6061.524017] Total swap = 514076kB [ 6061.529867] 32768 pages of RAM [ 6061.532917] 15081 free pages [ 6061.535792] 1306 reserved pages [ 6061.538920] 13476 slab pages [ 6061.541808] 66146 pages shared [ 6061.544856] 20 pages swap cached [ 6061.548072] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6061.555921] [ 840] 0 840 422 0 4 46 0 fan-d [ 6061.563832] [ 861] 0 861 504 0 5 31 0 mdadm [ 6061.571745] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6061.579659] [ 1437] 0 1437 434 6 4 14 0 noflushd [ 6061.587817] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6061.595645] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6061.603903] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6061.612160] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6061.620249] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6061.628409] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6061.636752] [ 1527] 0 1527 649 0 5 141 0 rpc.statd [ 6061.645016] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6061.652927] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6061.661270] [ 1882] 0 1882 813 0 5 20 0 sh [ 6061.668904] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6061.677075] [ 1884] 0 1884 813 0 5 22 0 sh [ 6061.684730] [ 1977] 0 1977 636 9 5 47 0 top [ 6061.692473] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6061.700380] Out of memory: Kill process 1527 (rpc.statd) score 0 or sacrifice child [ 6061.708012] Killed process 1527 (rpc.statd) total-vm:2596kB, anon-rss:0kB, file-rss:0kB [ 6062.217799] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6062.225048] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6062.230920] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6062.238662] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6062.246771] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6062.255515] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6062.263715] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6062.272422] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6062.281220] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6062.288874] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6062.296463] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6062.304219] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6062.311776] Mem-info: [ 6062.314042] Normal per-cpu: [ 6062.316836] CPU 0: hi: 42, btch: 7 usd: 0 [ 6062.321651] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6062.321651] active_file:128 inactive_file:128 isolated_file:0 [ 6062.321651] unevictable:0 dirty:0 writeback:0 unstable:0 [ 6062.321651] free:15066 slab_reclaimable:211 slab_unreclaimable:13363 [ 6062.321651] mapped:0 shmem:0 pagetables:84 bounce:0 [ 6062.321651] free_cma:0 [ 6062.352025] Normal free:60264kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:512kB inactive_file:512kB unevictable:0kB isolated(a$ [ 6062.392679] lowmem_reserve[]: 0 0 [ 6062.396008] Normal: 6008*4kB (UEM) 4509*8kB (UEM) 10*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60264kB [ 6062.408469] 256 total pagecache pages [ 6062.412141] 0 pages in swap cache [ 6062.415453] Swap cache stats: add 3500, delete 3500, find 1110/1930 [ 6062.421710] Free swap = 511204kB [ 6062.425012] Total swap = 514076kB [ 6062.430856] 32768 pages of RAM [ 6062.433904] 15223 free pages [ 6062.436771] 1306 reserved pages [ 6062.439925] 13476 slab pages [ 6062.442793] 131680 pages shared [ 6062.445921] 0 pages swap cached [ 6062.449051] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6062.456900] [ 840] 0 840 422 0 4 46 0 fan-d [ 6062.464815] [ 861] 0 861 504 0 5 31 0 mdadm [ 6062.472727] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6062.480636] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6062.488796] [ 1449] 0 1449 1587 0 6 124 0 nmbd [ 6062.496623] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6062.504883] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6062.513142] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6062.521227] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6062.529388] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6062.537737] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6062.545647] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6062.553992] [ 1882] 0 1882 813 0 5 20 0 sh [ 6062.561645] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6062.569821] [ 1884] 0 1884 813 0 5 22 0 sh [ 6062.577460] [ 1977] 0 1977 636 0 5 56 0 top [ 6062.585202] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6062.593108] Out of memory: Kill process 1449 (nmbd) score 0 or sacrifice child [ 6062.600321] Killed process 1449 (nmbd) total-vm:6348kB, anon-rss:0kB, file-rss:0kB [ 6062.649374] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6062.656624] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6062.662496] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6062.670262] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6062.678352] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6062.686976] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6062.695172] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6062.703882] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6062.712680] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6062.720346] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6062.727915] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6062.735675] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6062.743232] Mem-info: [ 6062.745506] Normal per-cpu: [ 6062.748300] CPU 0: hi: 42, btch: 7 usd: 0 [ 6062.753115] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6062.753115] active_file:128 inactive_file:128 isolated_file:0 [ 6062.753115] unevictable:0 dirty:0 writeback:0 unstable:0 [ 6062.753115] free:15086 slab_reclaimable:208 slab_unreclaimable:13356 [ 6062.753115] mapped:0 shmem:0 pagetables:78 bounce:0 [ 6062.753115] free_cma:0 [ 6062.783479] Normal free:60344kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:512kB inactive_file:512kB unevictable:0kB isolated(a$ [ 6062.824124] lowmem_reserve[]: 0 0 [ 6062.827455] Normal: 6010*4kB (UEM) 4512*8kB (UEM) 13*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60344kB [ 6062.842314] 256 total pagecache pages [ 6062.846478] 0 pages in swap cache [ 6062.850184] Swap cache stats: add 3506, delete 3506, find 1122/1947 [ 6062.856435] Free swap = 511700kB [ 6062.859830] Total swap = 514076kB [ 6062.865680] 32768 pages of RAM [ 6062.868730] 15172 free pages [ 6062.871630] 1306 reserved pages [ 6062.874760] 13466 slab pages [ 6062.877628] 66456 pages shared [ 6062.880745] 0 pages swap cached [ 6062.883886] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6062.893019] [ 840] 0 840 422 0 4 46 0 fan-d [ 6062.901676] [ 861] 0 861 504 0 5 31 0 mdadm [ 6062.909602] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6062.917495] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6062.925792] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6062.935056] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6062.944316] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6062.952608] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6062.960790] [ 1525] 0 1525 724 0 5 121 0 rpc.mountd [ 6062.969126] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6062.977086] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6062.986198] [ 1882] 0 1882 813 0 5 20 0 sh [ 6062.993869] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6063.002044] [ 1884] 0 1884 813 0 5 22 0 sh [ 6063.009700] [ 1977] 0 1977 636 0 5 56 0 top [ 6063.017430] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6063.025378] Out of memory: Kill process 1525 (rpc.mountd) score 0 or sacrifice child [ 6063.033448] Killed process 1525 (rpc.mountd) total-vm:2896kB, anon-rss:0kB, file-rss:0kB [ 6071.014982] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6071.022234] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6071.028085] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6071.035855] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6071.044031] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6071.052807] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6071.061005] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6071.069721] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6071.078500] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6071.086176] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6071.093808] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6071.101652] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6071.109199] Mem-info: [ 6071.111496] Normal per-cpu: [ 6071.114291] CPU 0: hi: 42, btch: 7 usd: 0 [ 6071.119091] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6071.119091] active_file:139 inactive_file:170 isolated_file:0 [ 6071.119091] unevictable:0 dirty:0 writeback:54 unstable:0 [ 6071.119091] free:14794 slab_reclaimable:189 slab_unreclaimable:13346 [ 6071.119091] mapped:0 shmem:0 pagetables:73 bounce:0 [ 6071.119091] free_cma:0 [ 6071.149541] Normal free:59176kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:556kB inactive_file:680kB unevictable:0kB isolated(a$ [ 6071.190367] lowmem_reserve[]: 0 0 [ 6071.193697] Normal: 5932*4kB (UEM) 4431*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59176kB [ 6071.205510] 309 total pagecache pages [ 6071.209167] 0 pages in swap cache [ 6071.212587] Swap cache stats: add 4137, delete 4137, find 1577/2712 [ 6071.218832] Free swap = 512184kB [ 6071.222164] Total swap = 514076kB [ 6071.227999] 32768 pages of RAM [ 6071.231074] 14939 free pages [ 6071.233942] 1306 reserved pages [ 6071.237069] 13437 slab pages [ 6071.239955] 66883 pages shared [ 6071.242997] 0 pages swap cached [ 6071.246135] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6071.254034] [ 840] 0 840 422 0 4 46 0 fan-d [ 6071.261961] [ 861] 0 861 504 0 5 31 0 mdadm [ 6071.269884] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6071.277786] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6071.285957] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6071.294217] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6071.302474] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6071.310563] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6071.318726] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6071.326642] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6071.334987] [ 1882] 0 1882 813 0 5 20 0 sh [ 6071.342643] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6071.350815] [ 1884] 0 1884 813 0 5 22 0 sh [ 6071.358456] [ 1977] 0 1977 636 0 5 56 0 top [ 6071.366198] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6071.374105] Out of memory: Kill process 1977 (top) score 0 or sacrifice child [ 6071.381231] Killed process 1977 (top) total-vm:2544kB, anon-rss:0kB, file-rss:0kB [ 6071.854758] fan-d invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6071.862012] CPU: 0 PID: 2033 Comm: fan-d Not tainted 3.14.2 #1 [ 6071.867866] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6071.875629] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6071.883742] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6071.892364] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6071.900558] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6071.909244] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6071.918043] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6071.925715] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6071.933304] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6071.941055] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6071.948591] Mem-info: [ 6071.950874] Normal per-cpu: [ 6071.953668] CPU 0: hi: 42, btch: 7 usd: 0 [ 6071.958467] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6071.958467] active_file:273 inactive_file:330 isolated_file:0 [ 6071.958467] unevictable:0 dirty:0 writeback:322 unstable:0 [ 6071.958467] free:14779 slab_reclaimable:192 slab_unreclaimable:13353 [ 6071.958467] mapped:17 shmem:0 pagetables:68 bounce:0 [ 6071.958467] free_cma:0 [ 6071.989090] Normal free:59116kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:1092kB inactive_file:1320kB unevictable:0kB isolated$ [ 6072.030262] lowmem_reserve[]: 0 0 [ 6072.033591] Normal: 5631*4kB (UEM) 4534*8kB (UEM) 20*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59116kB [ 6072.046051] 603 total pagecache pages [ 6072.049724] 0 pages in swap cache [ 6072.053037] Swap cache stats: add 4187, delete 4187, find 1610/2767 [ 6072.059278] Free swap = 512408kB [ 6072.062595] Total swap = 514076kB [ 6072.068420] 32768 pages of RAM [ 6072.071496] 14923 free pages [ 6072.074372] 1306 reserved pages [ 6072.077499] 13447 slab pages [ 6072.080385] 131883 pages shared [ 6072.083513] 0 pages swap cached [ 6072.086643] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6072.094492] [ 840] 0 840 422 0 4 46 0 fan-d [ 6072.102403] [ 861] 0 861 504 0 5 31 0 mdadm [ 6072.110319] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6072.118215] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6072.126387] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6072.134648] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6072.142906] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6072.150991] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6072.159156] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6072.167073] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6072.175420] [ 1882] 0 1882 813 0 5 20 0 sh [ 6072.183073] [ 1883] 0 1883 572 0 4 50 0 dropbear [ 6072.191244] [ 1884] 0 1884 813 17 5 22 0 sh [ 6072.198881] [ 2033] 0 2033 422 0 4 46 0 fan-d [ 6072.206785] Out of memory: Kill process 1883 (dropbear) score 0 or sacrifice child [ 6072.214348] Killed process 1884 (sh) total-vm:3252kB, anon-rss:0kB, file-rss:68kB [ 6073.401932] sh invoked oom-killer: gfp_mask=0x4d0, order=2, oom_score_adj=0 [ 6073.408976] CPU: 0 PID: 2035 Comm: sh Not tainted 3.14.2 #1 [ 6073.414605] [<c00139fc>] (unwind_backtrace) from [<c001075c>] (show_stack+0x10/0x14) [ 6073.422425] [<c001075c>] (show_stack) from [<c02a5040>] (dump_header.isra.17+0x48/0x128) [ 6073.430560] [<c02a5040>] (dump_header.isra.17) from [<c005ad74>] (oom_kill_process+0x7c/0x350) [ 6073.439164] [<c005ad74>] (oom_kill_process) from [<c005b498>] (out_of_memory+0x2d8/0x308) [ 6073.447360] [<c005b498>] (out_of_memory) from [<c005dfd8>] (__alloc_pages_nodemask+0x62c/0x690) [ 6073.456068] [<c005dfd8>] (__alloc_pages_nodemask) from [<c005e04c>] (__get_free_pages+0x10/0x3c) [ 6073.464864] [<c005e04c>] (__get_free_pages) from [<c0015d58>] (pgd_alloc+0x14/0xe4) [ 6073.472538] [<c0015d58>] (pgd_alloc) from [<c0018b0c>] (mm_init.isra.68+0x84/0xc4) [ 6073.480125] [<c0018b0c>] (mm_init.isra.68) from [<c008c348>] (do_execve+0x170/0x480) [ 6073.487866] [<c008c348>] (do_execve) from [<c000d620>] (ret_fast_syscall+0x0/0x2c) [ 6073.495421] Mem-info: [ 6073.497693] Normal per-cpu: [ 6073.500499] CPU 0: hi: 42, btch: 7 usd: 0 [ 6073.505303] active_anon:0 inactive_anon:0 isolated_anon:0 [ 6073.505303] active_file:129 inactive_file:128 isolated_file:0 [ 6073.505303] unevictable:0 dirty:0 writeback:0 unstable:0 [ 6073.505303] free:15134 slab_reclaimable:182 slab_unreclaimable:13388 [ 6073.505303] mapped:0 shmem:0 pagetables:65 bounce:0 [ 6073.505303] free_cma:0 [ 6073.535666] Normal free:60536kB min:1416kB low:1768kB high:2124kB active_anon:0kB inactive_anon:0kB active_file:516kB inactive_file:512kB unevictable:0kB isolated(a$ [ 6073.576312] lowmem_reserve[]: 0 0 [ 6073.579660] Normal: 5966*4kB (EM) 4540*8kB (UEM) 22*16kB (UEM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60536kB [ 6073.592111] 257 total pagecache pages [ 6073.595769] 0 pages in swap cache [ 6073.599078] Swap cache stats: add 4407, delete 4407, find 1768/3010 [ 6073.605340] Free swap = 512580kB [ 6073.608647] Total swap = 514076kB [ 6073.614487] 32768 pages of RAM [ 6073.617538] 15273 free pages [ 6073.620432] 1306 reserved pages [ 6073.623567] 13472 slab pages [ 6073.626436] 131651 pages shared [ 6073.629580] 0 pages swap cached [ 6073.632710] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 6073.640561] [ 840] 0 840 422 0 4 46 0 fan-d [ 6073.648463] [ 861] 0 861 504 0 5 31 0 mdadm [ 6073.656388] [ 1038] 0 1038 445 0 5 22 -1000 udevd [ 6073.664308] [ 1437] 0 1437 434 0 4 20 0 noflushd [ 6073.672477] [ 1466] 0 1466 421 0 4 13 0 syncawayd [ 6073.680740] [ 1469] 0 1469 422 0 5 13 0 discoverd [ 6073.688985] [ 1481] 0 1481 546 0 5 39 0 rpcbind [ 6073.697069] [ 1503] 0 1503 552 0 4 22 0 dropbear [ 6073.705247] [ 1547] 0 1547 421 0 5 15 0 hostd [ 6073.713159] [ 1575] 0 1575 422 0 4 47 0 temp-guard [ 6073.721502] [ 1882] 0 1882 813 0 5 20 0 sh [ 6073.729147] [ 2033] 0 2033 812 0 5 16 0 sh [ 6073.736795] [ 2035] 0 2035 812 0 5 17 0 sh [ 6073.744446] Out of memory: Kill process 1575 (temp-guard) score 0 or sacrifice child [ 6073.752177] Killed process 1575 (temp-guard) total-vm:1688kB, anon-rss:0kB, file-rss:0kB

Out of memory is being invoked always by fan-d and everytime a new random process is killed trying to get memory for fan-d. I can also add that this happens after big data transfers (more than 20gb) and not if transfer small amounts and wait between them. Also, i never heard fan working (maybe its really silent).

Also, when i was using old kernel sometimes i had an error on dmesg about I2C bus locked. I search for it a bit i can think it was fan related, but then we changed kernel and i didnt search anymore.

by **fvdw** » Sat Aug 22, 2015 7:34 pm

yes fan-d seems to invoke it but I don't think it is the cause. It uses only little amount of the memory.
This is a daemon that control the fan, if temp stays below the tress hold then it will not start the fan.
So as long as that is the case you won't hear the fan.

Anyhow in the attachment a newer version. It will be included in 16-1 and has a better temperature control, it monotors temperatures of all installed disks insead of only sda.

You must replace the existing fan5big binary in /bin_cab

To be able to do that you probably need to kill fan-d first

Code: Select all: killall -9 fan-d

(in fact fan-d is a symlink to fan5big)

Of course you could also try to run the nas with no fan-d daemon to see if oom comes back but be careful because your disk could be overheated.(see web interface disk settings)

I think some other process is eating your memory.
You can use the command

Code: Select all: ps aux

to get a list of all processes running and the amount of memory they use.
This is the output of my 5big2 and you can see fan-d is only using 0.1% of the memory

Code: Select all: root@fvdwsl-5big2:/ # ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.1 3032 620 ? Ss 18:24 0:01 init root 2 0.0 0.0 0 0 ? S 18:24 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 18:24 0:00 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S< 18:24 0:00 [kworker/0:0H] root 7 0.0 0.0 0 0 ? S 18:24 0:01 [rcu_preempt] root 8 0.0 0.0 0 0 ? S 18:24 0:00 [rcu_sched] root 9 0.0 0.0 0 0 ? S 18:24 0:00 [rcu_bh] root 10 0.0 0.0 0 0 ? S< 18:24 0:00 [khelper] root 242 0.0 0.0 0 0 ? S< 18:24 0:00 [writeback] root 244 0.0 0.0 0 0 ? S< 18:24 0:00 [bioset] root 246 0.0 0.0 0 0 ? S< 18:24 0:00 [kblockd] root 251 0.0 0.0 0 0 ? S< 18:24 0:00 [ata_sff] root 262 0.0 0.0 0 0 ? S 18:24 0:00 [khubd] root 269 0.0 0.0 0 0 ? S< 18:24 0:00 [md] root 279 0.0 0.0 0 0 ? S< 18:24 0:00 [cfg80211] root 295 0.0 0.0 0 0 ? S< 18:24 0:00 [rpciod] root 304 0.0 0.0 0 0 ? S 18:24 0:00 [kswapd0] root 305 0.0 0.0 0 0 ? S 18:24 0:00 [fsnotify_mark] root 306 0.0 0.0 0 0 ? S< 18:24 0:00 [nfsiod] root 307 0.0 0.0 0 0 ? S< 18:24 0:00 [cifsiod] root 311 0.0 0.0 0 0 ? S 18:24 0:00 [jfsIO] root 312 0.0 0.0 0 0 ? S 18:24 0:00 [jfsCommit] root 313 0.0 0.0 0 0 ? S 18:24 0:00 [jfsSync] root 314 0.0 0.0 0 0 ? S< 18:24 0:00 [xfsalloc] root 315 0.0 0.0 0 0 ? S< 18:24 0:00 [xfs_mru_cache] root 316 0.0 0.0 0 0 ? S< 18:24 0:00 [xfslogd] root 317 0.0 0.0 0 0 ? S< 18:24 0:00 [crypto] root 441 0.0 0.0 0 0 ? S 18:24 0:00 [scsi_eh_0] root 442 0.0 0.0 0 0 ? S< 18:24 0:00 [scsi_tmf_0] root 445 0.0 0.0 0 0 ? S 18:24 0:00 [scsi_eh_1] root 446 0.0 0.0 0 0 ? S< 18:24 0:00 [scsi_tmf_1] root 456 0.0 0.0 0 0 ? S< 18:24 0:00 [smflush] root 546 0.0 0.0 0 0 ? S< 18:24 0:00 [aoe_tx0] root 548 0.0 0.0 0 0 ? S< 18:24 0:00 [aoe_ktio0] root 566 0.0 0.0 0 0 ? S< 18:24 0:00 [raid5wq] root 568 0.0 0.0 0 0 ? S< 18:24 0:00 [dm_bufio_cache] root 569 0.0 0.0 0 0 ? S< 18:24 0:00 [kmpathd] root 570 0.0 0.0 0 0 ? S< 18:24 0:00 [kmpath_handlerd] root 588 0.0 0.0 0 0 ? S 18:24 0:00 [mv_crypto] root 598 0.0 0.0 0 0 ? S< 18:24 0:00 [ipv6_addrconf] root 604 0.0 0.0 0 0 ? S< 18:24 0:00 [deferwq] root 703 0.0 0.0 0 0 ? S< 18:24 0:00 [kworker/0:1H] root 704 0.0 0.0 0 0 ? S 18:24 0:00 [kjournald] root 716 0.0 0.0 0 0 ? S 18:24 0:00 [kjournald] root 749 0.0 0.0 1552 164 ? Ss 18:24 0:00 /usr/sbin/buttonsd root 761 0.0 0.0 0 0 ? S 18:24 0:00 [kjournald] root 792 0.0 0.0 0 0 ? S< 18:24 0:00 [iscsi_eh] root 910 0.0 0.1 1812 564 ? Ss 18:24 0:00 /usr/sbin/fan-d root 914 0.0 0.0 0 0 ? S 18:24 0:00 [scsi_eh_2] root 915 0.0 0.0 0 0 ? S< 18:24 0:00 [scsi_tmf_2] root 916 0.0 0.0 0 0 ? S 18:24 0:00 [usb-storage] root 935 0.0 0.1 3656 552 ? Ss 18:24 0:00 /sbin/syslogd -b 5 root 1069 0.0 0.0 0 0 ? S 18:25 0:00 [kjournald] root 1074 0.0 0.0 0 0 ? S 18:25 0:00 [kjournald] root 1079 0.0 0.0 0 0 ? S 18:25 0:00 [kjournald] root 1084 0.0 0.0 0 0 ? S 18:25 0:00 [kjournald] root 1089 0.0 0.1 1860 544 ? S<s 18:25 0:00 /sbin/udevd --daemo root 1512 0.0 0.0 3656 292 ? Ss 18:25 0:00 /sbin/udhcpc -b -i root 1547 0.0 2.3 45520 12240 ? Ss 18:25 0:00 /usr/bin/httpd -f h nobody 1550 0.0 0.9 45600 4796 ? S 18:25 0:00 /usr/bin/httpd -f h nobody 1551 0.0 0.8 45600 4228 ? S 18:25 0:00 /usr/bin/httpd -f h nobody 1552 0.0 0.8 45600 4208 ? S 18:25 0:00 /usr/bin/httpd -f h root 1555 0.0 0.1 1740 536 ? Ss 18:25 0:07 /usr/sbin/noflushd root 1736 0.0 0.0 1684 312 ? S<s 18:25 0:08 /usr/sbin/syncawayd root 1740 0.0 0.0 1688 364 ? Ss 18:25 0:00 /usr/bin/discoverd root 1747 0.0 0.1 2180 620 ? Ss 18:25 0:00 /usr/bin/rpcbind root 1763 0.0 0.0 2208 488 ? Ss 18:25 0:00 dropbear -p 22 -r / root 1776 0.0 0.0 1684 436 ? Ss 18:25 0:00 /usr/sbin/hostd 720 root 1804 0.0 0.1 1812 576 ? Ss 18:25 0:00 /usr/sbin/temp-guar root 1849 0.0 0.1 2256 604 ? Ss 18:25 0:00 /usr/sbin/rsync --d root 1856 0.0 0.0 3656 400 ? Ss 18:25 0:00 /bin/ntpd -l nobody 1923 0.0 0.9 10908 5084 ? Ss 18:25 0:00 proftpd: (accepting root 1931 0.0 0.0 3656 508 ? Ss 18:25 0:00 /bin/ntpd -p 193.67 root 1943 0.0 0.1 3656 588 ? Ss 18:25 0:00 /usr/sbin/crond -c root 1965 1.7 2.1 77032 10968 ? Ssl 18:25 3:03 /usr/local/sbin/bts root 1996 0.0 0.2 6372 1388 ? Ss 18:25 0:00 /usr/sbin/nmbd -D - root 1998 0.0 0.5 9324 2564 ? Ss 18:25 0:00 /usr/sbin/smbd -D - root 2006 0.0 0.2 9324 1136 ? S 18:25 0:00 /usr/sbin/smbd -D - root 2378 0.0 0.1 3036 620 ? Ss 18:25 0:00 /bin/sh root 2417 0.0 0.0 0 0 ? S 20:25 0:00 [kworker/u2:1] root 2465 0.0 0.1 2288 892 ? Ss 20:56 0:00 dropbear -p 22 -r / root 2466 0.0 0.1 3660 860 pts/0 Ss 20:56 0:00 -sh root 2483 0.0 0.1 2288 852 ? Ss 20:57 0:00 dropbear -p 22 -r / root 2484 0.0 0.1 3660 800 ? S 20:57 0:00 -sh root 2638 0.0 0.0 0 0 ? S 21:06 0:00 [kworker/0:2] root 2644 0.0 0.0 0 0 ? S 21:15 0:00 [kworker/u2:0] root 2645 0.0 0.0 0 0 ? S 21:16 0:00 [kworker/0:0] root 2654 0.0 0.5 9648 2724 ? S 21:21 0:00 /usr/sbin/smbd -D - root 2655 0.0 0.0 0 0 ? S 21:21 0:00 [kworker/0:1] root 2660 0.0 0.2 4596 1148 pts/0 R+ 21:24 0:00 ps aux root@fvdwsl-5big2:/ # ps aux | grep fan-d root 910 0.0 0.1 1812 564 ? Ss 18:24 0:00 /usr/sbin/fan-d

by **samrise** » Sat Aug 22, 2015 8:10 pm

I cant find the attachment, maybe you forgot?

Regarding fan-d, i dont think its using a lot of memory, i think maybe its not working correctly and the result is some kind of memory leak, since it seems to only happens when disks become hotter. Is there a way to check if fans are working correctly or a way to make them work manual?

by **samrise** » Sat Aug 22, 2015 8:34 pm

Some more tips regarding temperature.

In this post you say 56º recorded is a lot. Mine recorded 64º as maximum before i activate temp guard on 55º.

In this pic temp is 30º and you say its ok. Mine is always around 43º-45º even if idle.

Code: Select all: root@nas:/ # lsmod | grep g762 root@nas:/ #

This module should also be loaded?

I didnt read the entire thread so maybe you have another module now?

plugout.net

Sysctl and out of memory killer

Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Re: Sysctl and out of memory killer

Who is online