My server crashed today. It switched the /sda8 device (/home) to read only and the log said “ext3_xattr_block_get: inode 590080: bad block 6”
From rescue mode (something my hosting provider offers where the OS is loaded from the network) I ran e2fsck -c on the offending (unmounted) device.
This was the result:
/home: Updating bad block inode. Pass 1: Checking inodes, blocks, and sizes Inode 590080 has a bad extended attribute block 6. Clear<y>? yes Inode 590080 has illegal block(s). Clear<y>? yes Illegal block #6 (3225896002) in inode 590080. CLEARED. Illegal indirect block (161766104) in inode 590080. CLEARED. Illegal triple indirect block (161776560) in inode 590080. CLEARED. Inode 590080, i_size is 693150323631456341, should be 4096. Fix<y>? yes Deleted inode 8669125 has zero dtime. Fix<y>? yes Pass 2: Checking directory structure Entry '1362510759.H198758P4400' in /mail/tmp (8666577) has deleted/unused inode 8669125. Clear<y>? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -8685568 -8687060 Fix<y>? yes Free blocks count wrong for group #265 (5593, counted=5595). Fix<y>? yes Free blocks count wrong (7313501, counted=7313503). Fix<y>? yes Inode bitmap differences: -8669125 Fix<y>? yes Free inodes count wrong for group #265 (29385, counted=29386). Fix<y>? yes Free inodes count wrong (11764285, counted=11764286). Fix<y>? yes /home: ***** FILE SYSTEM WAS MODIFIED ***** /home: 630530/12394816 files (4.3% non-contiguous), 5076620/12390123 blocks
I don’t really understand that, but it sounds like it found something wrong.
But when I ran “dumpe2fs -b /dev/sda8” it doesn’t list any bad blocks.
So are there bad blocks? If yes, where are they? If no, what was the server complaining about?
This is just plain old filesystem corruption.
You didn’t give enough information about your environment to make an educated guess as to what happened, but the most common causes I’ve seen are:
- Not installing updates. Old kernels, especially on EL5 and older versions, have many filesystem related bugs which have been fixed and updates pushed. If you are not on EL5.9 / EL6.x then you’re doing it wrong.
- Loss of power while the system is active and thus the filesystem is in an indeterminate state. You would hopefully know if an event like this had happened.
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.