Server crashed, suspect a hardware failure

rabotalius asked:

Today my server crashed, apparently 10-15 minutes after I edited some php files. Maybe it’s not related but it’s the second time I change configs in php and after 15 minutes or so server becomes not responsive.

Can it be my hardware?

server info:

centos6
Linux 1 3.0.0+ #1 SMP Fri Oct 26 07:55:47 EEST 2012 x86_64 x86_64 x86_64 GNU/Linux
no raid

/var/log/messages:

Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:39:22 stephan kernel: TCP: Possible SYN flooding on port 8080. Dropping request.
Aug 11 21:40:04 stephan kernel: net_ratelimit: 60 callbacks suppressed
Aug 11 22:06:49 stephan kernel: find: page allocation failure: order:4, mode:0x50
Aug 11 22:06:49 stephan kernel: Pid: 25644, comm: find Not tainted 3.0.0+ #1
Aug 11 22:06:49 stephan kernel: Call Trace:
Aug 11 22:06:49 stephan kernel: [<ffffffff811089c3>] warn_alloc_failed+0xf3/0x160
Aug 11 22:06:49 stephan kernel: [<ffffffff8110acc9>] ? __alloc_pages_direct_compact+0xc9/0x190
Aug 11 22:06:49 stephan kernel: [<ffffffff8110b199>] __alloc_pages_slowpath+0x409/0x6b0
Aug 11 22:06:49 stephan kernel: [<ffffffff8110b5e4>] __alloc_pages_nodemask+0x1a4/0x1f0
Aug 11 22:06:49 stephan kernel: [<ffffffff81149c92>] kmem_getpages+0x62/0x170
Aug 11 22:06:49 stephan kernel: [<ffffffff8114a7aa>] fallback_alloc+0x1ba/0x270
Aug 11 22:06:49 stephan kernel: [<ffffffff8114a223>] ? cache_grow+0x2c3/0x2f0
Aug 11 22:06:49 stephan kernel: [<ffffffff8114a535>] ____cache_alloc_node+0x95/0x150
Aug 11 22:06:49 stephan kernel: [<ffffffff8114b0db>] __kmalloc_track_caller+0x16b/0x1f0
Aug 11 22:06:49 stephan kernel: [<ffffffff8111d8fb>] ? krealloc+0x1b/0x60

Aug 11 22:34:04 stephan kernel: active_anon:17963140 inactive_anon:2065253 isolated_anon:0
Aug 11 22:34:04 stephan kernel: active_file:1256602 inactive_file:1269966 isolated_file:0
Aug 11 22:34:04 stephan kernel: unevictable:0 dirty:3241 writeback:0 unstable:0
Aug 11 22:34:04 stephan kernel: free:101331 slab_reclaimable:276886 slab_unreclaimable:1169794
Aug 11 22:34:04 stephan kernel: mapped:382601 shmem:9232188 pagetables:37365 bounce:0
Aug 11 22:34:04 stephan kernel: Node 0 DMA free:15864kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_$
Aug 11 22:34:04 stephan kernel: lowmem_reserve[]: 0 2170 48378 48378
Aug 11 22:34:04 stephan kernel: Node 0 DMA32 free:186904kB min:2016kB low:2520kB high:3024kB active_anon:1580540kB inactive_anon:430692kB activ$
Aug 11 22:34:04 stephan kernel: lowmem_reserve[]: 0 0 46207 46207
Aug 11 22:34:04 stephan kernel: Node 0 Normal free:82568kB min:42980kB low:53724kB high:64468kB active_anon:40600164kB inactive_anon:2053444kB $
Aug 11 22:34:04 stephan kernel: lowmem_reserve[]: 0 0 0 0
Aug 11 22:34:04 stephan kernel: Node 1 Normal free:119988kB min:45096kB low:56368kB high:67644kB active_anon:29671856kB inactive_anon:5776876kB$
Aug 11 22:34:04 stephan kernel: lowmem_reserve[]: 0 0 0 0
Aug 11 22:34:04 stephan kernel: Node 0 DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15864kB
Aug 11 22:34:04 stephan kernel: Node 0 DMA32: 1200*4kB 899*8kB 614*16kB 341*32kB 207*64kB 151*128kB 117*256kB 89*512kB 41*1024kB 2*2048kB 0*409$
Aug 11 22:34:04 stephan kernel: Node 0 Normal: 20707*4kB 23*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 83012$
Aug 11 22:34:04 stephan kernel: Node 1 Normal: 18553*4kB 5316*8kB 155*16kB 19*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = $
Aug 11 22:34:04 stephan kernel: 11759340 total pagecache pages
Aug 11 22:34:04 stephan kernel: 263 pages in swap cache
Aug 11 22:34:04 stephan kernel: Swap cache stats: add 143022, delete 142759, find 2120395833/2120420681
Aug 11 22:34:04 stephan kernel: Free swap  = 0kB
Aug 11 22:34:04 stephan kernel: Total swap = 569340kB
Aug 11 22:34:04 stephan kernel: 25165808 pages RAM
Aug 11 22:34:04 stephan kernel: 380608 pages reserved
Aug 11 22:34:04 stephan kernel: 984379 pages shared
Aug 11 22:34:04 stephan kernel: 24085536 pages non-shared
Aug 11 22:34:08 stephan kernel: find: page allocation failure: order:4, mode:0x50
Aug 11 22:34:08 stephan kernel: Pid: 28687, comm: find Not tainted 3.0.0+ #1
Aug 11 22:34:08 stephan kernel: Call Trace:
Aug 11 22:34:08 stephan kernel: [<ffffffff811089c3>] warn_alloc_failed+0xf3/0x160

Aug 11 22:34:08 stephan kernel: [<ffffffff8110abf6>] ? drain_local_pages+0x16/0x20
Aug 11 22:34:08 stephan kernel: [<ffffffff8110b199>] __alloc_pages_slowpath+0x409/0x6b0
Aug 11 22:34:08 stephan kernel: [<ffffffff8110b5e4>] __alloc_pages_nodemask+0x1a4/0x1f0
Aug 11 22:34:08 stephan kernel: [<ffffffff81149c92>] kmem_getpages+0x62/0x170
Aug 11 22:34:08 stephan kernel: [<ffffffff8114a7aa>] fallback_alloc+0x1ba/0x270
Aug 11 22:34:08 stephan kernel: [<ffffffff8114a223>] ? cache_grow+0x2c3/0x2f0
Aug 11 22:34:08 stephan kernel: [<ffffffff8114a535>] ____cache_alloc_node+0x95/0x150
Aug 11 22:34:08 stephan kernel: [<ffffffff8114b0db>] __kmalloc_track_caller+0x16b/0x1f0
Aug 11 22:34:08 stephan kernel: [<ffffffff8111d8fb>] ? krealloc+0x1b/0x60
Aug 11 22:34:08 stephan kernel: [<ffffffff8111d8af>] __krealloc+0x6f/0xa0
Aug 11 22:34:08 stephan kernel: [<ffffffff8111d8fb>] krealloc+0x1b/0x60
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef14c>] append_deblk+0x2c/0xa0
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef318>] append_de+0x158/0x190
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef6cb>] fillvdir+0x1cb/0x1e0
Aug 11 22:34:08 stephan kernel: [<ffffffffa03b9c27>] xfs_dir2_leaf_getdents+0x547/0x780 [xfs]
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef500>] ? au_nhash_append_wh+0xc0/0xc0
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef500>] ? au_nhash_append_wh+0xc0/0xc0
Aug 11 22:34:08 stephan kernel: [<ffffffffa03b696c>] xfs_readdir+0xec/0x130 [xfs]
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef500>] ? au_nhash_append_wh+0xc0/0xc0
Aug 11 22:34:08 stephan kernel: [<ffffffffa03eb1d9>] xfs_file_readdir+0x39/0x50 [xfs]
Aug 11 22:34:08 stephan kernel: [<ffffffff81171608>] vfs_readdir+0xb8/0xe0
Aug 11 22:34:08 stephan kernel: [<ffffffff814f8afe>] ? mutex_lock+0x1e/0x50
Aug 11 22:34:08 stephan kernel: [<ffffffff811e2d2e>] vfsub_readdir+0x1e/0x40
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef88e>] au_do_read_vdir+0x14e/0x1e0
Aug 11 22:34:08 stephan kernel: [<ffffffff811ef9e0>] read_vdir+0xc0/0x140
Aug 11 22:34:08 stephan kernel: [<ffffffff811efbe0>] au_vdir_init+0x30/0x110
Aug 11 22:34:08 stephan kernel: [<ffffffff811edff3>] aufs_readdir+0xa3/0x1b0
Aug 11 22:34:08 stephan kernel: [<ffffffff81171470>] ? sys_ioctl+0xb0/0xb0
Aug 11 22:34:08 stephan kernel: [<ffffffff81171470>] ? sys_ioctl+0xb0/0xb0
Aug 11 22:34:08 stephan kernel: [<ffffffff81171608>] vfs_readdir+0xb8/0xe0
Aug 11 22:34:08 stephan kernel: [<ffffffff81171799>] sys_getdents+0x89/0xf0
Aug 11 22:34:08 stephan kernel: [<ffffffff81501f42>] system_call_fastpath+0x16/0x1b

Aug 11 22:34:08 stephan kernel: Mem-Info:
Aug 11 22:34:08 stephan kernel: Node 0 DMA per-cpu:
Aug 11 22:34:08 stephan kernel: CPU    0: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    1: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    2: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    3: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    4: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    5: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    6: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    7: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    8: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    9: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU   10: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: CPU   11: hi:    0, btch:   1 usd:   0
Aug 11 22:34:08 stephan kernel: Node 0 DMA32 per-cpu:
Aug 11 22:34:08 stephan kernel: CPU    0: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    1: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    2: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    3: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    4: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    5: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    6: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    7: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    8: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    9: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU   10: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU   11: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: Node 0 Normal per-cpu:
Aug 11 22:34:08 stephan kernel: CPU    0: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    1: hi:  186, btch:  31 usd:   2
Aug 11 22:34:08 stephan kernel: CPU    2: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    3: hi:  186, btch:  31 usd:   1
Aug 11 22:34:08 stephan kernel: CPU    4: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    5: hi:  186, btch:  31 usd:   3
Aug 11 22:34:08 stephan kernel: CPU    6: hi:  186, btch:  31 usd:  50
Aug 11 22:34:08 stephan kernel: CPU    7: hi:  186, btch:  31 usd:   1
Aug 11 22:34:08 stephan kernel: CPU    8: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    9: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU   10: hi:  186, btch:  31 usd:  19
Aug 11 22:34:08 stephan kernel: CPU   11: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: Node 1 Normal per-cpu:
Aug 11 22:34:08 stephan kernel: CPU    0: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    1: hi:  186, btch:  31 usd:   8
Aug 11 22:34:08 stephan kernel: CPU    2: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    3: hi:  186, btch:  31 usd: 175
Aug 11 22:34:08 stephan kernel: CPU    4: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    5: hi:  186, btch:  31 usd:   4
Aug 11 22:34:08 stephan kernel: CPU    6: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    7: hi:  186, btch:  31 usd:   4
Aug 11 22:34:08 stephan kernel: CPU    8: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU    9: hi:  186, btch:  31 usd:  31
Aug 11 22:34:08 stephan kernel: CPU   10: hi:  186, btch:  31 usd:   0
Aug 11 22:34:08 stephan kernel: CPU   11: hi:  186, btch:  31 usd:   0

I answered:

It appears you ran out of memory — and then ran out of swap space!

Figure out what is using up all the memory on your system.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.