OOM invoked with plenty of free swap

Evengard asked:

I am running an ARM router with i2p and tor on it – a Netgear R7000.
Of course I’ve added a full 512 MB of SWAP to it to prevent OOMs, understanding that it could slow down the system… But then I still get OOM killer starting with plenty of SWAP free, and killing tor! More interesting, after killing tor it seems that the system is OK for an unlimited amount of time… But still seems like tor couldn’t be swapped.
I even tried to turn off the overcommit, didn’t helped at all.

See the dmesg log below

resetbutton invoked oom-killer: gfp_mask=0x2000d0, order=0, oom_score_adj=0
CPU: 0 PID: 1500 Comm: resetbutton Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:c7a4cdc0 r5:00000000 r4:c6818000 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007c5cc>] (dump_header.isra.13+0x84/0x194)
[<c007c548>] (dump_header.isra.13+0x0/0x194) from [<c007c958>] (oom_kill_process+0x90/0x3e8)
[<c007c8c8>] (oom_kill_process+0x0/0x3e8) from [<c007d17c>] (out_of_memory+0x2c0/0x304)
[<c007cebc>] (out_of_memory+0x0/0x304) from [<c007fcbc>] (__alloc_pages_nodemask+0x5b0/0x694)
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c00a2464>] (cache_alloc_refill+0x2c8/0x600)
[<c00a219c>] (cache_alloc_refill+0x0/0x600) from [<c00a214c>] (kmem_cache_alloc+0x88/0xd8)
[<c00a20c4>] (kmem_cache_alloc+0x0/0xd8) from [<c00afdd4>] (getname_flags+0x2c/0x104)
 r7:b6f5509c r6:00000000 r5:00000000 r4:00020000
[<c00afda8>] (getname_flags+0x0/0x104) from [<c00afec4>] (getname+0x18/0x1c)
[<c00afeac>] (getname+0x0/0x1c) from [<c00a5270>] (do_sys_open+0xbc/0x178)
[<c00a51b4>] (do_sys_open+0x0/0x178) from [<c00a5350>] (SyS_open+0x24/0x28)
 r9:c6818000 r8:c0008d24 r7:00000005 r6:00000000 r5:00000000
r4:00000000
[<c00a532c>] (SyS_open+0x0/0x28) from [<c0008ba0>] (ret_fast_syscall+0x0/0x30)
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:10263 inactive_anon:12797 isolated_anon:0
 active_file:957 inactive_file:1335 isolated_file:0
 unevictable:151 dirty:0 writeback:0 unstable:0
 free:13787 slab_reclaimable:912 slab_unreclaimable:5734
 mapped:1171 shmem:0 pagetables:283 bounce:0
 free_cma:0
Normal free:26184kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:96kB inactive_file:64kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3648kB slab_unreclaimable:22936kB kernel_stack:1192kB pagetables:1132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4440 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:28656kB min:128kB low:8772kB high:17416kB active_anon:41052kB inactive_anon:51512kB active_file:3764kB inactive_file:5244kB unevictable:540kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:0kB writeback:0kB mapped:4684kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 102*4kB (UEM) 118*8kB (UEM) 360*16kB (UM) 158*32kB (UM) 28*64kB (UEM) 8*128kB (UM) 4*256kB (UM) 4*512kB (UM) 0*1024kB 0*2048kB 2*4096kB (R) = 26248kB
HighMem: 1177*4kB (UMR) 1620*8kB (UMR) 225*16kB (UMR) 30*32kB (UMR) 29*64kB (UMR) 8*128kB (UMR) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 28436kB
9347 total pagecache pages
6914 pages in swap cache
Swap cache stats: add 484419, delete 477505, find 187469/228822
Free swap  = 408788kB
Total swap = 524284kB
65536 pages of RAM
13904 free pages
1661 reserved pages
6646 slab pages
549590 pages shared
6923 pages swap cached
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  622]     0   622      189        7       3       13             0 hotplug2
[  626]     0   626      199       30       3        6             0 mstpd
[  971]     0   971      217        9       3        8             0 dropbear
[  987]     0   987      297      105       3       30             0 dnsmasq
[ 1112]     0  1112      315       29       4       30             0 ttraff
[ 1195]     0  1195      209       25       3        6             0 dhcp6c
[ 1288]     0  1288      755        3       4       83             0 httpd
[ 1330]     0  1330      315       26       3       18             0 process_monitor
[ 1334]     0  1334      361       44       3       38             0 upnp
[ 1337]     0  1337      334       50       4        9             0 wland
[ 1338]     0  1338      190       42       3        3             0 igmprt
[ 1339]     0  1339      291       10       3        6             0 udhcpc
[ 1449]     0  1449      182       27       3        3             0 cron
[ 1457]     0  1457      370       38       3       32             0 nas
[ 1460]     0  1460      370       36       4       32             0 nas
[ 1495]     0  1495      291       11       3        6             0 syslogd
[ 1497]     0  1497      291       15       3        6             0 klogd
[ 1500]     0  1500      334       32       3       12             0 resetbutton
[ 1684] 65534  1684     1008       18       5      499             0 tinyproxy
[ 1690] 65534  1690     1021        9       5      508             0 tinyproxy
[ 1691] 65534  1691     1021        9       5      508             0 tinyproxy
[ 1692] 65534  1692     1021        9       5      508             0 tinyproxy
[ 1693] 65534  1693     1008        6       5      502             0 tinyproxy
[ 1694] 65534  1694     1008        6       5      502             0 tinyproxy
[ 1695] 65534  1695     1008        6       5      502             0 tinyproxy
[ 1696] 65534  1696     1008        6       5      502             0 tinyproxy
[ 1697] 65534  1697     1008        6       5      502             0 tinyproxy
[ 1698] 65534  1698     1008        6       5      502             0 tinyproxy
[ 1699] 65534  1699     1008        6       5      502             0 tinyproxy
[ 1711]   106  1711    36556    12756      73    10236             0 tor
[ 1730]    13  1730      580        9       4       38             0 polipo
[ 1806]  1000  1806     2828       41       5      124             0 wrapper
[ 1808]  1000  1808    59927     7979      69    13797             0 java
[ 1921]     0  1921      241       18       3       31             0 dropbear
[ 1923]     0  1923      293        9       3        9             0 sh
[ 3705]     0  3705      804        5       4       86             0 bash
[ 5859]     0  5859      320       61       3       12             0 radvd
[ 5860]     0  5860      320       18       3       18             0 radvd
Out of memory: Kill process 1711 (tor) score 118 or sacrifice child
Killed process 1711 (tor) total-vm:146224kB, anon-rss:49036kB, file-rss:1988kB
tor: page allocation failure: order:0, mode:0x20058
CPU: 1 PID: 1711 Comm: tor Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:00000000 r5:00000000 r4:00020058 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007def4>] (warn_alloc_failed+0x100/0x110)
[<c007ddf4>] (warn_alloc_failed+0x0/0x110) from [<c007faf0>] (__alloc_pages_nodemask+0x3e4/0x694)
 r3:00040001 r2:00000000
 r7:c0405180 r6:00000001 r5:00000000 r4:00020058
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c007a274>] (find_or_create_page+0x48/0x94)
[<c007a22c>] (find_or_create_page+0x0/0x94) from [<c00cfc08>] (__getblk+0x16c/0x308)
 r9:00000000 r8:0223764c r7:c7418110 r6:00000001 r5:00001000
r4:00000000
[<c00cfa9c>] (__getblk+0x0/0x308) from [<c00d0668>] (__bread+0x1c/0xd8)
[<c00d064c>] (__bread+0x0/0xd8) from [<bf5e96c8>] (_8+0x208/0x4a0 [ext2])
 r4:c6bb5b58
[<bf5e963c>] (_8+0x17c/0x4a0 [ext2]) from [<bf5ea2e0>] (_35+0x78/0x884 [ext2])
[<bf5ea268>] (_35+0x0/0x884 [ext2]) from [<c00d6da8>] (do_mpage_readpage+0x268/0x6d4)
[<c00d6b40>] (do_mpage_readpage+0x0/0x6d4) from [<c00d72d4>] (mpage_readpages+0xc0/0x104)
[<c00d7214>] (mpage_readpages+0x0/0x104) from [<bf5e99ac>] (_61+0x24/0x28 [ext2])
[<bf5e9988>] (_61+0x0/0x28 [ext2]) from [<c0082f08>] (__do_page_cache_readahead+0x170/0x1f8)
[<c0082d98>] (__do_page_cache_readahead+0x0/0x1f8) from [<c0083428>] (ra_submit+0x30/0x38)
[<c00833f8>] (ra_submit+0x0/0x38) from [<c007bb4c>] (filemap_fault+0x1bc/0x434)
[<c007b990>] (filemap_fault+0x0/0x434) from [<c008e5a4>] (__do_fault+0xac/0x490)
[<c008e4f8>] (__do_fault+0x0/0x490) from [<c0091284>] (handle_pte_fault+0x78/0x6d4)
[<c009120c>] (handle_pte_fault+0x0/0x6d4) from [<c0091968>] (__handle_mm_fault+0x88/0x90)
[<c00918e0>] (__handle_mm_fault+0x0/0x90) from [<c00920e0>] (handle_mm_fault+0x28/0x2c)
[<c00920b8>] (handle_mm_fault+0x0/0x2c) from [<c001948c>] (do_page_fault+0x118/0x288)
[<c0019374>] (do_page_fault+0x0/0x288) from [<c0008580>] (do_DataAbort+0x40/0x9c)
[<c0008540>] (do_DataAbort+0x0/0x9c) from [<c0008998>] (__dabt_usr+0x38/0x40)
Exception stack(0xc6bb5fb0 to 0xc6bb5ff8)
5fa0:                                     b6fe43d8 b6cfd864 00000005 00000053
5fc0: 00000001 b6fe43dd b6d0a358 b6cfd864 b866141c 00000000 b6fe43d8 00000000
5fe0: b6d0b658 be8798f0 b6cf2330 b6a4c388 20000010 ffffffff
 r8:b866141c r7:b6cfd864 r6:ffffffff r5:20000010 r4:b6a4c388
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:10263 inactive_anon:13171 isolated_anon:0
 active_file:971 inactive_file:1322 isolated_file:0
 unevictable:151 dirty:1 writeback:0 unstable:0
 free:13621 slab_reclaimable:912 slab_unreclaimable:5739
 mapped:1172 shmem:0 pagetables:283 bounce:0
 free_cma:0
Normal free:27136kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:40kB inactive_file:120kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3648kB slab_unreclaimable:22956kB kernel_stack:1192kB pagetables:1132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:27380kB min:128kB low:8772kB high:17416kB active_anon:41052kB inactive_anon:52684kB active_file:3764kB inactive_file:5248kB unevictable:540kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:4kB writeback:0kB mapped:4688kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 96*4kB (UEM) 120*8kB (UEM) 360*16kB (UM) 182*32kB (UM) 32*64kB (UEM) 8*128kB (UM) 4*256kB (UM) 4*512kB (UM) 0*1024kB 0*2048kB 2*4096kB (R) = 27264kB
HighMem: 918*4kB (UMR) 1620*8kB (UMR) 225*16kB (UMR) 30*32kB (UMR) 29*64kB (UMR) 8*128kB (UMR) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 27400kB
9533 total pagecache pages
7099 pages in swap cache
Swap cache stats: add 484625, delete 477526, find 187576/228974
Free swap  = 408868kB
Total swap = 524284kB
65536 pages of RAM
13939 free pages
1661 reserved pages
6652 slab pages
549523 pages shared
7099 pages swap cached
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets

The sysctl vm related settings:

vm.admin_reserve_kbytes = 7914
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.highmem_is_dirtyable = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32    32
vm.max_map_count = 65530
vm.min_free_kbytes = 32768
vm.mmap_min_addr = 4096
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 2
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 100
vm.user_reserve_kbytes = 7914
vm.vfs_cache_pressure = 100

What am I doing wrong (except for using a SOHO router for tor, lol)?

PS: Using DD-WRT firmware

EDIT:
Another OOM killer log – this time nothing about resetbutton:

kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0
CPU: 0 PID: 2 Comm: kthreadd Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:c782fb80 r5:00000000 r4:c7834000 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007c5cc>] (dump_header.isra.13+0x84/0x194)
[<c007c548>] (dump_header.isra.13+0x0/0x194) from [<c007c958>] (oom_kill_process+0x90/0x3e8)
[<c007c8c8>] (oom_kill_process+0x0/0x3e8) from [<c007d17c>] (out_of_memory+0x2c0/0x304)
[<c007cebc>] (out_of_memory+0x0/0x304) from [<c007fcbc>] (__alloc_pages_nodemask+0x5b0/0x694)
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c002bf70>] (copy_process.part.68+0xdc/0xe30)
[<c002be94>] (copy_process.part.68+0x0/0xe30) from [<c002cdd8>] (do_fork+0xe0/0x2bc)
[<c002ccf8>] (do_fork+0x0/0x2bc) from [<c002cfe4>] (kernel_thread+0x30/0x38)
 r9:00000001 r8:00000000 r7:c7834000 r6:c0454014 r5:c04096a8
r4:c1109e88
[<c002cfb4>] (kernel_thread+0x0/0x38) from [<c004be50>] (kthreadd+0xe4/0x12c)
[<c004bd6c>] (kthreadd+0x0/0x12c) from [<c0008c38>] (ret_from_fork+0x14/0x3c)
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:7196 inactive_anon:7223 isolated_anon:0
 active_file:1054 inactive_file:1422 isolated_file:0
 unevictable:199 dirty:3 writeback:0 unstable:0
 free:22911 slab_reclaimable:969 slab_unreclaimable:6549
 mapped:685 shmem:0 pagetables:210 bounce:0
 free_cma:0
Normal free:29372kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:128kB unevictable:44kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3876kB slab_unreclaimable:26196kB kernel_stack:576kB pagetables:840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1064 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:62272kB min:128kB low:8772kB high:17416kB active_anon:28784kB inactive_anon:28892kB active_file:4068kB inactive_file:5676kB unevictable:752kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:12kB writeback:0kB mapped:2740kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 465*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 92*32kB (UEM) 51*64kB (UEM) 25*128kB (UEM) 12*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29340kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
5489 total pagecache pages
2805 pages in swap cache
Swap cache stats: add 1237151, delete 1234346, find 539424/688368
Free swap  = 484712kB
Total swap = 524284kB
65536 pages of RAM
23095 free pages
1661 reserved pages
7517 slab pages
557492 pages shared
2805 pages swap cached
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  622]     0   622      189        1       3       12             0 hotplug2
[  626]     0   626      199       31       3        6             0 mstpd
[  971]     0   971      217        1       3        7             0 dropbear
[  987]     0   987      297      105       3       30             0 dnsmasq
[ 1112]     0  1112      315       61       4       10             0 ttraff
[ 1195]     0  1195      209       40       3        2             0 dhcp6c
[ 1288]     0  1288      755        1       4       82             0 httpd
[ 1330]     0  1330      315       38       3       17             0 process_monitor
[ 1334]     0  1334      366       89       3       11             0 upnp
[ 1337]     0  1337      334       50       4        9             0 wland
[ 1338]     0  1338      190       45       3        4             0 igmprt
[ 1339]     0  1339      291       30       3        5             0 udhcpc
[ 1449]     0  1449      182       32       3        2             0 cron
[ 1457]     0  1457      370       36       3       32             0 nas
[ 1460]     0  1460      370       34       4       32             0 nas
[ 1495]     0  1495      291       20       3        5             0 syslogd
[ 1497]     0  1497      291       37       3        2             0 klogd
[ 1500]     0  1500      334       32       3       12             0 resetbutton
[ 1684] 65534  1684     1008       15       5      499             0 tinyproxy
[ 1690] 65534  1690     1021        5       5      507             0 tinyproxy
[ 1691] 65534  1691     1021        5       5      507             0 tinyproxy
[ 1692] 65534  1692     1021        5       5      507             0 tinyproxy
[ 1693] 65534  1693     1008        4       5      501             0 tinyproxy
[ 1694] 65534  1694     1008        4       5      501             0 tinyproxy
[ 1695] 65534  1695     1008        4       5      501             0 tinyproxy
[ 1696] 65534  1696     1008        4       5      501             0 tinyproxy
[ 1697] 65534  1697     1008        4       5      501             0 tinyproxy
[ 1698] 65534  1698     1008        4       5      501             0 tinyproxy
[ 1699] 65534  1699     1008        4       5      501             0 tinyproxy
[ 1730]    13  1730      580       10       4       37             0 polipo
[ 1806]  1000  1806     2828       78       5      116             0 wrapper
[ 1921]     0  1921      241       51       3       19             0 dropbear
[ 1923]     0  1923      293        1       3        8             0 sh
[16658]   106 16643    33786    12722      65     7647             0 tor
[16661]     0 16661      804        4       4       85             0 bash
[17143]     0 17143      790      167       4       26             0 top
[20851]     0 20851      320       64       3       12             0 radvd
[20853]     0 20853      320       33       3       17             0 radvd
Out of memory: Kill process 16658 (tor) score 104 or sacrifice child
Killed process 16658 (tor) total-vm:135144kB, anon-rss:49088kB, file-rss:1800kB
tor: page allocation failure: order:0, mode:0x20058
CPU: 1 PID: 16658 Comm: tor Tainted: P             3.10.79 #381
Backtrace:
[<c0015cb8>] (dump_backtrace+0x0/0x118) from [<c0015ec0>] (show_stack+0x18/0x1c)
 r6:00000000 r5:00000000 r4:00020058 r3:00000000
[<c0015ea8>] (show_stack+0x0/0x1c) from [<c012e5c0>] (dump_stack+0x24/0x28)
[<c012e59c>] (dump_stack+0x0/0x28) from [<c007def4>] (warn_alloc_failed+0x100/0x110)
[<c007ddf4>] (warn_alloc_failed+0x0/0x110) from [<c007faf0>] (__alloc_pages_nodemask+0x3e4/0x694)
 r3:00040001 r2:00000000
 r7:c0405180 r6:c6488000 r5:00000000 r4:00020058
[<c007f70c>] (__alloc_pages_nodemask+0x0/0x694) from [<c007a274>] (find_or_create_page+0x48/0x94)
[<c007a22c>] (find_or_create_page+0x0/0x94) from [<c00cfc08>] (__getblk+0x16c/0x308)
 r9:00000000 r8:0220f844 r7:c7418110 r6:00000001 r5:00001000
r4:00000000
[<c00cfa9c>] (__getblk+0x0/0x308) from [<c00d0668>] (__bread+0x1c/0xd8)
[<c00d064c>] (__bread+0x0/0xd8) from [<bf5e96c8>] (_8+0x208/0x4a0 [ext2])
 r4:c6489b58
[<bf5e963c>] (_8+0x17c/0x4a0 [ext2]) from [<bf5ea2e0>] (_35+0x78/0x884 [ext2])
[<bf5ea268>] (_35+0x0/0x884 [ext2]) from [<c00d6da8>] (do_mpage_readpage+0x268/0x6d4)
[<c00d6b40>] (do_mpage_readpage+0x0/0x6d4) from [<c00d72d4>] (mpage_readpages+0xc0/0x104)
[<c00d7214>] (mpage_readpages+0x0/0x104) from [<bf5e99ac>] (_61+0x24/0x28 [ext2])
[<bf5e9988>] (_61+0x0/0x28 [ext2]) from [<c0082f08>] (__do_page_cache_readahead+0x170/0x1f8)
[<c0082d98>] (__do_page_cache_readahead+0x0/0x1f8) from [<c0083428>] (ra_submit+0x30/0x38)
[<c00833f8>] (ra_submit+0x0/0x38) from [<c007bb4c>] (filemap_fault+0x1bc/0x434)
[<c007b990>] (filemap_fault+0x0/0x434) from [<c008e5a4>] (__do_fault+0xac/0x490)
[<c008e4f8>] (__do_fault+0x0/0x490) from [<c0091284>] (handle_pte_fault+0x78/0x6d4)
[<c009120c>] (handle_pte_fault+0x0/0x6d4) from [<c0091968>] (__handle_mm_fault+0x88/0x90)
[<c00918e0>] (__handle_mm_fault+0x0/0x90) from [<c00920e0>] (handle_mm_fault+0x28/0x2c)
[<c00920b8>] (handle_mm_fault+0x0/0x2c) from [<c001948c>] (do_page_fault+0x118/0x288)
[<c0019374>] (do_page_fault+0x0/0x288) from [<c000861c>] (do_PrefetchAbort+0x40/0x9c)
[<c00085dc>] (do_PrefetchAbort+0x0/0x9c) from [<c0008b58>] (ret_from_exception+0x0/0x10)
Exception stack(0xc6489fb0 to 0xc6489ff8)
9fa0:                                     0000021c 00000000 6df1d800 6df1d800
9fc0: b470b880 0000000f da4afeed b6fe04bc 00000040 5eedf00d b6fdb8f4 b6fba134
9fe0: 00000000 b470b858 b6f0ead4 b6f0ead4 60000010 ffffffff
 r8:00000040 r7:b6fe04bc r6:ffffffff r5:60000010 r4:b6f0ead4
Mem-info:
Normal per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
HighMem per-cpu:
CPU    0: hi:   42, btch:   7 usd:   0
CPU    1: hi:   42, btch:   7 usd:   0
active_anon:7196 inactive_anon:7223 isolated_anon:0
 active_file:1028 inactive_file:1457 isolated_file:0
 unevictable:199 dirty:3 writeback:0 unstable:0
 free:22844 slab_reclaimable:969 slab_unreclaimable:6547
 mapped:685 shmem:0 pagetables:210 bounce:0
 free_cma:0
Normal free:29092kB min:32768kB low:40960kB high:49152kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:128kB unevictable:44kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:124196kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:3876kB slab_unreclaimable:26188kB kernel_stack:576kB pagetables:840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1024 1024
HighMem free:62252kB min:128kB low:8772kB high:17416kB active_anon:28784kB inactive_anon:28892kB active_file:4068kB inactive_file:5700kB unevictable:752kB isolated(anon):0kB isolated(file):0kB present:131072kB managed:131072kB mlocked:0kB dirty:12kB writeback:0kB mapped:2740kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 467*4kB (UEM) 285*8kB (UEM) 219*16kB (UEM) 93*32kB (UEM) 50*64kB (EM) 25*128kB (UEM) 11*256kB (UEM) 2*512kB (U) 0*1024kB 0*2048kB 2*4096kB (R) = 29060kB
HighMem: 5997*4kB (UM) 2743*8kB (UM) 478*16kB (UM) 27*32kB (UM) 2*64kB (M) 2*128kB (U) 1*256kB (U) 2*512kB (U) 0*1024kB 1*2048kB (U) 1*4096kB (R) = 62252kB
5489 total pagecache pages
2805 pages in swap cache
Swap cache stats: add 1237151, delete 1234346, find 539424/688368
Free swap  = 484712kB
Total swap = 524284kB
65536 pages of RAM
23025 free pages
1661 reserved pages
7515 slab pages
819780 pages shared
2805 pages swap cached
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets
TCP: too many orphaned sockets

And again, Free swap = 484712kB – huge amount not used. Why?

My answer:


You have a couple of obvious problems.

First:

vm.swappiness = 100

You’ve told the kernel to swap as aggressively as possible, even when it’s not strictly necessary to do so.

On an embedded device (which usually won’t have swap at all) this is completely insane. This should be set to 0 (probably best) or 1.

Second:

vm.overcommit_memory = 2

Aside from preventing any overcommit at all, and preventing the system from using all free memory and swap, this also causes additional memory to be reserved for admin and user processes as specified in admin_reserve_kbytes and user_reserve_kbytes.

I can’t find a single system I manage, embedded, desktop or server, where this setting is anything other than 0.

Finally, about DD-WRT… I’m not happy with them. They went overly commercial some time back and stopped making regular releases and updates. These days I recommend OpenWrt as a platform for building embedded systems.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.