Tag Archive for mdadm

RAID rebuilding seems to have stopped

Thomas Wang asked:

My server was running a RAID 1 array with two disks. One of those disk failed today and was replaced.

I’ve copied the GPT partition to the new hard disk (sda) with:

sgdisk -R /dev/sda /dev/sdb

and changed the UDID with

sgdisk -G /dev/sda

I’ve then added both partitions to the RAID array:

mdadm /dev/md4 -a /dev/sda4

and

mdadm /dev/md5 -a /dev/sda5

/dev/md4 was rebuilt correctly, but not /dev/md5.

When I run cat /proc/mdstat shortly after running those commands, it showed this:

Personalities : [raid1]
md5 : active raid1 sda5[2] sdb5[1]
      2820667711 blocks super 1.2 [2/1] [_U]
      [>....................]  recovery =  0.0% (2109952/2820667711) finish=423.0min speed=111050K/sec

md4 : active raid1 sda4[2] sdb4[0]
      15727544 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Which was correct; it was trying to rebuild md5, but a few minutes later, it stopped and now cat /proc/mdstat returns:

Personalities : [raid1]
md5 : active raid1 sda5[2](S) sdb5[1]
      2820667711 blocks super 1.2 [2/1] [_U]

md4 : active raid1 sda4[2] sdb4[0]
      15727544 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Why did it stop rebuilding on that new disk? Here’s what I get when running mdadm --detail /dev/md5

    /dev/md5:
        Version : 1.2
  Creation Time : Sun Sep 16 15:26:58 2012
     Raid Level : raid1
     Array Size : 2820667711 (2690.00 GiB 2888.36 GB)
  Used Dev Size : 2820667711 (2690.00 GiB 2888.36 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sat Dec 27 04:01:26 2014
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           Name : rescue:5  (local to host rescue)
           UUID : 29868a4d:f63c6b43:ee926581:fd775604
         Events : 5237753

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       21        1      active sync   /dev/sdb5

       2       8        5        -      spare   /dev/sda5

Thanks @Michael Hampton for your answer. I’m back at it after a night of sleep :-) So I checked dmesg and I get this:

[Sat Dec 27 04:01:04 2014] md: recovery of RAID array md5
[Sat Dec 27 04:01:04 2014] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[Sat Dec 27 04:01:04 2014] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[Sat Dec 27 04:01:04 2014] md: using 128k window, over a total of 2820667711k.
[Sat Dec 27 04:01:04 2014] RAID1 conf printout:
[Sat Dec 27 04:01:04 2014]  --- wd:2 rd:2
[Sat Dec 27 04:01:04 2014]  disk 0, wo:0, o:1, dev:sdb4
[Sat Dec 27 04:01:04 2014]  disk 1, wo:0, o:1, dev:sda4
[Sat Dec 27 04:01:21 2014] ata2.00: exception Emask 0x0 SAct 0x1e000 SErr 0x0 action 0x0
[Sat Dec 27 04:01:21 2014] ata2.00: irq_stat 0x40000008
[Sat Dec 27 04:01:21 2014] ata2.00: cmd 60/80:68:00:12:51/03:00:0d:00:00/40 tag 13 ncq 458752 in
[Sat Dec 27 04:01:21 2014]          res 41/40:80:68:14:51/00:03:0d:00:00/00 Emask 0x409 (media error) <F>
[Sat Dec 27 04:01:21 2014] ata2.00: configured for UDMA/133
[Sat Dec 27 04:01:21 2014] sd 1:0:0:0: [sdb] Unhandled sense code
[Sat Dec 27 04:01:21 2014] sd 1:0:0:0: [sdb]  
[Sat Dec 27 04:01:21 2014] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Sat Dec 27 04:01:21 2014] sd 1:0:0:0: [sdb]  
[Sat Dec 27 04:01:21 2014] Sense Key : Medium Error [current] [descriptor]
[Sat Dec 27 04:01:21 2014] Descriptor sense data with sense descriptors (in hex):
[Sat Dec 27 04:01:21 2014]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[Sat Dec 27 04:01:21 2014]         0d 51 14 68 
[Sat Dec 27 04:01:21 2014] sd 1:0:0:0: [sdb]  
[Sat Dec 27 04:01:21 2014] Add. Sense: Unrecovered read error - auto reallocate failed
[Sat Dec 27 04:01:21 2014] sd 1:0:0:0: [sdb] CDB: 
[Sat Dec 27 04:01:21 2014] Read(16): 88 00 00 00 00 00 0d 51 12 00 00 00 03 80 00 00
[Sat Dec 27 04:01:21 2014] end_request: I/O error, dev sdb, sector 223417448
[Sat Dec 27 04:01:21 2014] ata2: EH complete
[Sat Dec 27 04:01:24 2014] ata2.00: exception Emask 0x0 SAct 0x8 SErr 0x0 action 0x0
[Sat Dec 27 04:01:24 2014] ata2.00: irq_stat 0x40000008
[Sat Dec 27 04:01:24 2014] ata2.00: cmd 60/08:18:68:14:51/00:00:0d:00:00/40 tag 3 ncq 4096 in
[Sat Dec 27 04:01:24 2014]          res 41/40:08:68:14:51/00:00:0d:00:00/00 Emask 0x409 (media error) <F>
[Sat Dec 27 04:01:24 2014] ata2.00: configured for UDMA/133
[Sat Dec 27 04:01:24 2014] sd 1:0:0:0: [sdb] Unhandled sense code
[Sat Dec 27 04:01:24 2014] sd 1:0:0:0: [sdb]  
[Sat Dec 27 04:01:24 2014] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Sat Dec 27 04:01:24 2014] sd 1:0:0:0: [sdb]  
[Sat Dec 27 04:01:24 2014] Sense Key : Medium Error [current] [descriptor]
[Sat Dec 27 04:01:24 2014] Descriptor sense data with sense descriptors (in hex):
[Sat Dec 27 04:01:24 2014]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[Sat Dec 27 04:01:24 2014]         0d 51 14 68 
[Sat Dec 27 04:01:24 2014] sd 1:0:0:0: [sdb]  
[Sat Dec 27 04:01:24 2014] Add. Sense: Unrecovered read error - auto reallocate failed
[Sat Dec 27 04:01:24 2014] sd 1:0:0:0: [sdb] CDB: 
[Sat Dec 27 04:01:24 2014] Read(16): 88 00 00 00 00 00 0d 51 14 68 00 00 00 08 00 00
[Sat Dec 27 04:01:24 2014] end_request: I/O error, dev sdb, sector 223417448
[Sat Dec 27 04:01:24 2014] ata2: EH complete
[Sat Dec 27 04:01:24 2014] md/raid1:md5: sdb: unrecoverable I/O read error for block 4219904
[Sat Dec 27 04:01:24 2014] md: md5: recovery interrupted.
[Sat Dec 27 04:01:24 2014] RAID1 conf printout:
[Sat Dec 27 04:01:24 2014]  --- wd:1 rd:2
[Sat Dec 27 04:01:24 2014]  disk 0, wo:1, o:1, dev:sda5
[Sat Dec 27 04:01:24 2014]  disk 1, wo:0, o:1, dev:sdb5
[Sat Dec 27 04:01:24 2014] RAID1 conf printout:
[Sat Dec 27 04:01:24 2014]  --- wd:1 rd:2
[Sat Dec 27 04:01:24 2014]  disk 1, wo:0, o:1, dev:sdb5

So it does seem to be a read error. But SMART doesn’t seem to be too bad (if I understand it correctly):

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   088   087   006    Pre-fail  Always       -       154455820
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       5
  5 Reallocated_Sector_Ct   0x0033   084   084   036    Pre-fail  Always       -       21664
  7 Seek_Error_Rate         0x000f   072   060   030    Pre-fail  Always       -       38808769144
  9 Power_On_Hours          0x0032   071   071   000    Old_age   Always       -       26073
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       5
183 Runtime_Bad_Block       0x0032   099   099   000    Old_age   Always       -       1
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       721
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       4295032833
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   063   061   045    Old_age   Always       -       37 (Min/Max 33/37)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       3
193 Load_Cycle_Count        0x0032   095   095   000    Old_age   Always       -       10183
194 Temperature_Celsius     0x0022   037   040   000    Old_age   Always       -       37 (0 21 0 0)
197 Current_Pending_Sector  0x0012   088   088   000    Old_age   Always       -       2072
198 Offline_Uncorrectable   0x0010   088   088   000    Old_age   Offline      -       2072
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       157045479198210
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       4435703883570
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       5487937263078

SMART Error Log Version: 1
ATA Error Count: 6 (device log contains only the most recent five errors)

Anyway thanks for your answer. And yes, if I was setting up the server again I definitely would not use more than one partition for my RAID array (in this case actually md5 is even using LVM.

Thanks,


I answered:

It looks like you physically removed the faulty disk without Linux being fully aware of it, thus when you added the new disk it was marked as a spare (and the system is still waiting for you to put the old disk back in). It is likely that /dev/md4 failed and Linux detected the failure, but since /dev/md5 is a separate array (that didn’t itself fail) Linux still believed it was good.

To recover from this situation, you need to tell the system to start using the spare, and to forget about the removed disk.

First, grow the RAID array to three devices, so that it can make use of the spare.

mdadm --grow /dev/md5 --raid-devices=3

At this point it should begin syncing to the spare, which will be listed as spare rebuilding in mdadm --detail, and you should see the sync operation in /proc/mdstat.

When the sync completes, you’ll tell mdadm to forget about the device that is no longer present.

mdadm --remove /dev/md5 detached

Finally, you set the number of devices back to 2.

mdadm --grow /dev/md5 --raid-devices=2

How your system got into this state, I can’t be sure. But it could be that your other disk had a read error, causing the resync to stop and this failed state to occur. If this is the case, you’ll see log entries to that effect in dmesg when the sync operation dies. If this turns out to be case you’ll need some deeper magic (update your question if this happens) and possibly to have your backups handy.


You may also want to read this virtually identical question on Super User as it contains some other possible solutions.


Finally, it is a best practice to use whole disks as RAID array members, or at most a single partition of the disk, you can then divide up the RAID block device with LVM if necessary. This configuration would have prevented this problem.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Is mdadm RAID Toast?

Andrew Wei asked:

I took all my drives out and put them onto a new cpu/mobo. (upgrading)

I boot up and mdadm claims it can’t boot my degraded RAID.

/dev/sdb

    sudo mdadm --examine /dev/sdb
    /dev/sdb:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 91a6c44d:21226975:8d2dc41a:7fcff414
               Name : desktop:0  (local to host desktop)
      Creation Time : Tue Jun 25 19:03:31 2013
         Raid Level : raid5
       Raid Devices : 3

     Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
         Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
      Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 367cb248:993e2658:ecd4b56d:2aaa0a6a

        Update Time : Tue Mar  4 17:48:54 2014
           Checksum : d4572f50 - correct
             Events : 12635

             Layout : left-symmetric
         Chunk Size : 512K

       Device Role : Active device 1
       Array State : AAA ('A' == active, '.' == missing)

/dev/sdc

    sudo mdadm --examine /dev/sdc
    /dev/sdc:
       MBR Magic : aa55
    Partition[0] :   4294967295 sectors at            1 (type ee)

/dev/sdd

    sudo mdadm --examine /dev/sdd
    /dev/sdd:
       MBR Magic : aa55
    Partition[0] :   4294967295 sectors at            1 (type ee)

What happens when I tried to “recreate” it

    sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 --spare-devices=0 /dev/sd[bcd]
    mdadm: /dev/sdb appears to be part of a raid array:
        level=raid5 devices=3 ctime=Tue Jun 25 19:03:31 2013
    mdadm: /dev/sdc appears to be part of a raid array:
        level=raid0 devices=0 ctime=Wed Dec 31 18:00:00 1969
    mdadm: partition table exists on /dev/sdc but will be lost or
           meaningless after creating array
    mdadm: /dev/sdd appears to be part of a raid array:
        level=raid0 devices=0 ctime=Wed Dec 31 18:00:00 1969
    mdadm: partition table exists on /dev/sdd but will be lost or
           meaningless after creating array

I’m hoping there is a slight chance I can get my stuff back, considering mdadm doesn’t see that sdc/sdd are part of a raid, but just not the same one.

Is my raid toast?

EDIT: Trying to assembling by specifying

    sudo mdadm --assemble /dev/md0 /dev/sd[bcd]
    mdadm: no RAID superblock on /dev/sdc
    mdadm: /dev/sdc has no superblock - assembly aborted

Try Using –scan

    sudo mdadm --assemble --scan
    mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

EDIT #2

    cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
    md0 : inactive sdb[1](S)
          2930135512 blocks super 1.2

    unused devices: <none>

EDIT #3

/dev/sdb

    sudo gdisk -l /dev/sdb
    GPT fdisk (gdisk) version 0.8.7

    Partition table scan:
      MBR: not present
      BSD: not present
      APM: not present
      GPT: not present

    Creating new GPT entries.
    Disk /dev/sdb: 5860533168 sectors, 2.7 TiB
    Logical sector size: 512 bytes
    Disk identifier (GUID): 76360B85-31EF-4155-8F9E-767C0C14454E
    Partition table holds up to 128 entries
    First usable sector is 34, last usable sector is 5860533134
    Partitions will be aligned on 2048-sector boundaries
    Total free space is 5860533101 sectors (2.7 TiB)

    Number  Start (sector)    End (sector)  Size       Code  Name

/dev/sdc

    sudo gdisk -l /dev/sdc
    GPT fdisk (gdisk) version 0.8.7

    Partition table scan:
      MBR: protective
      BSD: not present
      APM: not present
      GPT: present

    Found valid GPT with protective MBR; using GPT.
    Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
    Logical sector size: 512 bytes
    Disk identifier (GUID): DBD9F056-E1AE-4C22-826F-2D359EF6680E
    Partition table holds up to 128 entries
    First usable sector is 34, last usable sector is 5860533134
    Partitions will be aligned on 2048-sector boundaries
    Total free space is 2925 sectors (1.4 MiB)

    Number  Start (sector)    End (sector)  Size       Code  Name
       1            2048      5860532223   2.7 TiB     0700

/dev/sdd

    sudo gdisk -l /dev/sdd
    GPT fdisk (gdisk) version 0.8.7

    Partition table scan:
      MBR: protective
      BSD: not present
      APM: not present
      GPT: present

    Found valid GPT with protective MBR; using GPT.
    Disk /dev/sdd: 5860533168 sectors, 2.7 TiB
    Logical sector size: 512 bytes
    Disk identifier (GUID): BE9B843B-62CB-4D12-A661-5FA9AF871493
    Partition table holds up to 128 entries
    First usable sector is 34, last usable sector is 5860533134
    Partitions will be aligned on 2048-sector boundaries
    Total free space is 2925 sectors (1.4 MiB)

    Number  Start (sector)    End (sector)  Size       Code  Name
       1            2048      5860532223   2.7 TiB     0700  

I answered:

For some reason, you seem to have used the whole disk for one of the drives, and a single partition on each of the other two. Try assembling using the partitions.

mdadm --assemble /dev/sdb /dev/sd[cd]1

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

DegradedArray event on /dev/md1

user1821484 asked:

This morning I got this message:

This is an automatically generated mail message from mdadm
running on 

A DegradedArray event had been detected on md device /dev/md1.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1]
md1 : active raid1 sdb3[2](F) sda3[1]
      1860516800 blocks [2/1] [_U]

md0 : active raid1 sdb1[0] sda1[1]
      499904 blocks [2/2] [UU]

unused devices: <none>

Does it mean that 1 of the hard drives is not working anymore? How can I fix this problem? Should I ask the data center to replace the hard drive? Can I try to re-add the missing device? If yes, what command should I run and is it safe to re-add? I just don’t want my server to go offline.

serv397:/var/log# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2](F) sda3[1]
      1860516800 blocks [2/1] [_U]

md0 : active raid1 sdb1[0] sda1[1]
      499904 blocks [2/2] [UU]

unused devices: <none>

serv397:/var/log# mdadm -D /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Sun Apr 29 22:51:51 2012
     Raid Level : raid1
     Array Size : 1860516800 (1774.33 GiB 1905.17 GB)
  Used Dev Size : 1860516800 (1774.33 GiB 1905.17 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sat Feb 23 09:26:39 2013
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

           UUID : ec02d5ce:8554d4ad:7792c71e:7dc17aa4
         Events : 0.11225668

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8        3        1      active sync   /dev/sda3

       2       8       19        -      faulty spare   /dev/sdb3


kern.log

Feb 23 09:00:58 triton1017 kernel: [24015352.812156] __ratelimit: 134 callbacks suppressed
Feb 23 09:00:58 triton1017 kernel: [24015352.812165] mdadm: sending ioctl 1261 to a partition!
Feb 23 09:00:58 triton1017 kernel: [24015352.812172] mdadm: sending ioctl 1261 to a partition!


mdam:

[    1.929981] mdadm: sending ioctl 1261 to a partition!
[    1.930211] mdadm: sending ioctl 800c0910 to a partition!
[    1.930241] mdadm: sending ioctl 800c0910 to a partition!
[    1.944515] md: md0 stopped.
[    1.945700] md: bind<sda1>
[    1.945944] md: bind<sdb1>
[    1.947709] raid1: raid set md0 active with 2 out of 2 mirrors
[    1.947784] md0: detected capacity change from 0 to 511901696
[    1.948516]  md0: unknown partition table
[    1.984932] md: md1 stopped.
[    1.986131] md: bind<sda3>
[    1.986332] md: bind<sdb3>
[    1.987377] raid1: raid set md1 active with 2 out of 2 mirrors
[    1.987421] md1: detected capacity change from 0 to 1905169203200
[    1.988287]  md1: unknown partition table
[    2.164118] kjournald starting.  Commit interval 5 seconds
[    2.164130] EXT3-fs: mounted filesystem with ordered data mode.
[    3.181350] udev[346]: starting version 164
[    3.644863] input: PC Speaker as /devices/platform/pcspkr/input/input3
[    3.654062] Error: Driver 'pcspkr' is already registered, aborting...
[    3.663045] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
[    3.810284] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    3.812865] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    3.860102] [drm] Initialized drm 1.1.0 20060810
[    3.884550] hda-intel: no codecs found!
[    3.884672] HDA Intel 0000:01:05.1: setting latency timer to 64
[    3.925197] [drm] radeon defaulting to userspace modesetting.
[    3.925973] pci 0000:01:05.0: setting latency timer to 64
[    3.926082] [drm] Initialized radeon 1.32.0 20080528 for 0000:01:05.0 on minor 0
[    4.123784] Adding 1998840k swap on /dev/sda2.  Priority:-1 extents:1 across:1998840k
[    4.126482] Adding 1998840k swap on /dev/sdb2.  Priority:-2 extents:1 across:1998840k
[    4.332550] EXT3 FS on md1, internal journal
[    5.247285]   alloc irq_desc for 25 on node -1
[    5.247287]   alloc kstat_irqs on node -1
[    5.247299] tg3 0000:02:00.0: irq 25 for MSI/MSI-X
[    5.275326] ADDRCONF(NETDEV_UP): eth0: link is not ready

Tried to readd:

sudo mdadm --re-add /dev/md1 /dev/sdb3
mdadm: Cannot open /dev/sdb3: Device or resource busy
sudo mdadm --remove /dev/md1 /dev/sdb3
mdadm: hot removed /dev/sdb3 from /dev/md1
sudo mdadm --add /dev/md1 /dev/sdb3
mdadm: re-added /dev/sdb3

/var/log# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
      1860516800 blocks [2/1] [_U]
      [>....................]  recovery =  0.1% (2849024/1860516800) finish=455.9min speed=67898K/sec

md0 : active raid1 sdb1[0] sda1[1]
      499904 blocks [2/2] [UU]

unused devices: <none>

Re-syncing didn’t solve the problem:

triton1017:/var/log# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2](S) sda3[1]
      1860516800 blocks [2/1] [_U]

md0 : active raid1 sdb1[0] sda1[1]
      499904 blocks [2/2] [UU]

unused devices: <none>

triton1017:/var/log# mdadm -D /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Sun Apr 29 22:51:51 2012
     Raid Level : raid1
     Array Size : 1860516800 (1774.33 GiB 1905.17 GB)
  Used Dev Size : 1860516800 (1774.33 GiB 1905.17 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sat Feb 23 18:14:08 2013
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           UUID : ec02d5ce:8554d4ad:7792c71e:7dc17aa4
         Events : 0.11245156

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8        3        1      active sync   /dev/sda3

       2       8       19        -      spare   /dev/sdb3


kern.log files shows the following:

Feb 23 14:55:19 triton1017 kernel: [24036613.378608] ata1.00: error: { UNC }
Feb 23 14:55:19 triton1017 kernel: [24036613.398590] ata1.00: configured for UDMA/133
Feb 23 14:55:19 triton1017 kernel: [24036613.398627] ata1: EH complete
Feb 23 14:55:21 triton1017 kernel: [24036616.262518] ata1.00: exception Emask 0x0 SAct 0x1dfbe SErr 0x0 action 0x0
Feb 23 14:55:21 triton1017 kernel: [24036616.262525] ata1.00: irq_stat 0x40000008
Feb 23 14:55:21 triton1017 kernel: [24036616.262531] ata1.00: failed command: READ FPDMA QUEUED
Feb 23 14:55:21 triton1017 kernel: [24036616.262539] ata1.00: cmd 60/80:28:00:5a:b4/00:00:75:00:00/40 tag 5 ncq 65536 in
Feb 23 14:55:21 triton1017 kernel: [24036616.262540]          res 41/40:80:38:5a:b4/00:00:75:00:00/00 Emask 0x409 (media error) <F>
Feb 23 14:57:16 triton1017 kernel: [24036730.503323] ata1.00: status: { DRDY ERR }
Feb 23 14:57:16 triton1017 kernel: [24036730.503328] ata1.00: error: { UNC }
Feb 23 14:57:16 triton1017 kernel: [24036730.523346] ata1.00: configured for UDMA/133
Feb 23 14:57:16 triton1017 kernel: [24036730.523356] ata1: EH complete
Feb 23 14:57:17 triton1017 kernel: [24036732.116026] INFO: task mysqld:6067 blocked for more than 120 seconds.
Feb 23 14:57:17 triton1017 kernel: [24036732.116032] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 14:57:17 triton1017 kernel: [24036732.116040] mysqld        D 0000000000000002     0  6067    938 0x00000000
Feb 23 14:57:17 triton1017 kernel: [24036732.116049]  ffffffff814891f0 0000000000000086 0000000000000000 00000000ffffffff
Feb 23 14:57:17 triton1017 kernel: [24036732.117353]  ffff880016dcfc00 0000000000015780 000000000000f9e0 ffff8805c4c65fd8
Feb 23 14:57:17 triton1017 kernel: [24036732.117367]  0000000000015780 0000000000015780 ffff880618825bd0 ffff880618825ec8
Feb 23 14:57:17 triton1017 kernel: [24036732.117380] Call Trace:
Feb 23 14:57:17 triton1017 kernel: [24036732.117391]  [<ffffffff810168f3>] ? read_tsc+0xa/0x20
Feb 23 14:57:17 triton1017 kernel: [24036732.117400]  [<ffffffff8110e656>] ? sync_buffer+0x0/0x40
Feb 23 14:57:17 triton1017 kernel: [24036732.117408]  [<ffffffff812fbb4a>] ? io_schedule+0x73/0xb7
Feb 23 14:57:17 triton1017 kernel: [24036732.117419]  [<ffffffff8110e691>] ? sync_buffer+0x3b/0x40
Feb 23 14:57:17 triton1017 kernel: [24036732.117426]  [<ffffffff812fbf5a>] ? __wait_on_bit_lock+0x3f/0x84
Feb 23 14:57:17 triton1017 kernel: [24036732.117433]  [<ffffffff8110e656>] ? sync_buffer+0x0/0x40
Feb 23 14:57:17 triton1017 kernel: [24036732.117441]  [<ffffffff812fc00a>] ? out_of_line_wait_on_bit_lock+0x6b/0x77
Feb 23 14:57:17 triton1017 kernel: [24036732.117451]  [<ffffffff81065070>] ? wake_bit_function+0x0/0x23
Feb 23 14:57:17 triton1017 kernel: [24036732.117459]  [<ffffffff8110ea83>] ? sync_dirty_buffer+0x29/0x93
Feb 23 14:57:17 triton1017 kernel: [24036732.117474]  [<ffffffffa018ce04>] ? journal_dirty_data+0xd1/0x1b0 [jbd]
Feb 23 14:57:17 triton1017 kernel: [24036732.117486]  [<ffffffffa01a3f1f>] ? ext3_journal_dirty_data+0xf/0x34 [ext3]
Feb 23 14:57:17 triton1017 kernel: [24036732.117499]  [<ffffffffa01a23f9>] ? walk_page_buffers+0x65/0x8b [ext3]
Feb 23 14:57:17 triton1017 kernel: [24036732.117510]  [<ffffffffa01a3f44>] ? journal_dirty_data_fn+0x0/0x13 [ext3]
Feb 23 14:57:17 triton1017 kernel: [24036732.117521]  [<ffffffffa01a5a66>] ? ext3_ordered_write_end+0x73/0x10f [ext3]
Feb 23 14:57:17 triton1017 kernel: [24036732.117532]  [<ffffffffa01b0bbb>] ? ext3_xattr_get+0x1ef/0x271 [ext3]
Feb 23 14:57:17 triton1017 kernel: [24036732.117542]  [<ffffffff810b517e>] ? generic_file_buffered_write+0x18d/0x278
Feb 23 14:57:17 triton1017 kernel: [24036732.117552]  [<ffffffff810b561a>] ? __generic_file_aio_write+0x25f/0x293
Feb 23 14:57:17 triton1017 kernel: [24036732.117560]  [<ffffffff810b56a7>] ? generic_file_aio_write+0x59/0x9f
Feb 23 14:57:17 triton1017 kernel: [24036732.117569]  [<ffffffff810eef1a>] ? do_sync_write+0xce/0x113
Feb 23 14:57:17 triton1017 kernel: [24036732.117577]  [<ffffffff81103a85>] ? mntput_no_expire+0x23/0xee
Feb 23 14:57:17 triton1017 kernel: [24036732.117584]  [<ffffffff81065042>] ? autoremove_wake_function+0x0/0x2e
Feb 23 14:57:17 triton1017 kernel: [24036732.117593]  [<ffffffff812fce69>] ? _spin_lock_bh+0x9/0x25
Feb 23 14:57:17 triton1017 kernel: [24036732.117600]  [<ffffffff810ef86c>] ? vfs_write+0xa9/0x102
Feb 23 14:57:17 triton1017 kernel: [24036732.117607]  [<ffffffff810ef91c>] ? sys_pwrite64+0x57/0x77
Feb 23 14:57:17 triton1017 kernel: [24036732.117615]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Feb 23 14:57:17 triton1017 kernel: [24036732.117622] INFO: task flush-9:1:1456 blocked for more than 120 seconds.
Feb 23 14:57:17 triton1017 kernel: [24036732.117628] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 14:57:17 triton1017 kernel: [24036732.117636] flush-9:1     D 0000000000000000     0  1456      2 0x00000000
Feb 23 14:57:17 triton1017 kernel: [24036732.117645]  ffffffff814891f0 0000000000000046 0000000000000000 0000000000000001
Feb 23 14:57:17 triton1017 kernel: [24036732.117659]  0000000000000086 ffffffff8104a45a 000000000000f9e0 ffff88061905dfd8
Feb 23 14:57:17 triton1017 kernel: [24036732.117672]  0000000000015780 0000000000015780 ffff8806190646a0 ffff880619064998
Feb 23 14:57:17 triton1017 kernel: [24036732.117683] Call Trace:
Feb 23 14:57:17 triton1017 kernel: [24036732.117691]  [<ffffffff8104a45a>] ? try_to_wake_up+0x289/0x29b
Feb 23 14:57:17 triton1017 kernel: [24036732.117701]  [<ffffffff8119255f>] ? radix_tree_tag_clear+0x93/0xf1
Feb 23 14:57:17 triton1017 kernel: [24036732.117709]  [<ffffffff8110e656>] ? sync_buffer+0x0/0x40
Feb 23 14:57:17 triton1017 kernel: [24036732.117716]  [<ffffffff812fbb4a>] ? io_schedule+0x73/0xb7
Feb 23 14:57:17 triton1017 kernel: [24036732.117724]  [<ffffffff8110e691>] ? sync_buffer+0x3b/0x40
Feb 23 14:57:17 triton1017 kernel: [24036732.117731]  [<ffffffff812fbf5a>] ? __wait_on_bit_lock+0x3f/0x84
Feb 23 14:57:17 triton1017 kernel: [24036732.117738]  [<ffffffff8110e656>] ? sync_buffer+0x0/0x40
Feb 23 14:57:17 triton1017 kernel: [24036732.117745]  [<ffffffff812fc00a>] ? out_of_line_wait_on_bit_lock+0x6b/0x77
Feb 23 14:57:17 triton1017 kernel: [24036732.117753]  [<ffffffff81065070>] ? wake_bit_function+0x0/0x23
Feb 23 14:57:17 triton1017 kernel: [24036732.117762]  [<ffffffff8110fa23>] ? __block_write_full_page+0x159/0x2ac

I answered:

That disk is actually defective. Have it replaced. Re-sync after replacing the disk with a good one.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

How to detect hard disk failure?

Devator asked:

So, one of my servers has a hard disk failure. It’s running software RAID, the system locked up and according to /proc/mdstat (and /var/log/messages), it’s really down:

Personalities : [raid1]
md2 : active raid1 sdb2[1]
      104320 blocks [2/1] [_U]

md5 : active raid1 sdb5[1]
      2104448 blocks [2/1] [_U]

md6 : active raid1 sdb6[1]
      830134656 blocks [2/1] [_U]

md1 : active raid1 sdb1[1]
      143363968 blocks [2/1] [_U]

and

Nov  5 22:04:37 m38501 smartd[4467]: Device: /dev/sda, not capable of SMART self-check

However

when I do smartctl -H /dev/sda, it passes the test. It also passes the test with smartctl --test=short /dev/sda.

So, is smartctl a broken testing tool, or am I doing something completely off?


I answered:

Maybe an intermittent error with the drive electronics? That’s the first thing that comes to mind. Be safe and replace the drive.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Shell Script To Monitor Completion of RAID Assembly

VinnyD asked:

How can I pause execution of my shell script after calling the following command until the raid array has been assembled? From what I understand, this is an asynchronous process and status of the raid array needs to be polled.

mdadm –create -l10 -n4 /dev/md0 /dev/xvdh*


I answered:

Since you’re creating a new RAID 10, you can begin using the array immediately. The initial resync will continue in the background. You only need to wait for it to complete if building a RAID 5 array (and it’s a good idea for a RAID 6, too).

See Initial Array Creation in the Linux RAID Wiki for further details.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

How many really distinct permutations of volumes are in mdadm raid5 degraded array?

Adam Ryczkowski asked:

I’m attempting to recover from a failure of my raid volume after upgrade from ubuntu 10.04 to 12.04.

I’ve tried re-creating the array in any combination of 5 elementary volumes with one replaced with “missing” to ensure that the array wakes as degraded.

Next, with help of dd if=/dev/md1, I made a backup of the first 256kB of each version of my reassembled raid for inspection.

To my astonishment I see only 5 distinct version of the first 256kB chunk out of possible 120 permutations on a 5 disk set or even 24 on a 4 disk set. I assume the 4 disk set number should be correct, because 1 volume must be set as missing and henceforth shouldn’t be accounted for.

How can this occur?


I answered:

The Linux RAID Wiki has a script permute_array.pl designed to go through all the possible permutations and find the “right” one. You should be able to use this to begin recovering your array.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

how to know what is disk uuid for md raid1 and hdparm?

MealstroM asked:

ive got md0 (raid 1) array and want to make write cache off on them during system boot (ubuntu 12.04 server).

md0: /dev/sda /dev/sdc

blkid:

/dev/sda: UUID="3e502de5-696d-f4b4-470e-XXX" TYPE="linux_raid_member" 
/dev/sdb1: UUID="4ba40aae-65e2-416b-8f17-XXX" TYPE="ext2" 
/dev/sdb5: UUID="LNt5uO-ZFik-eQ0g-BEhP-FDLi-XXX" TYPE="LVM2_member" 
/dev/md0: UUID="a7eb2443-c3be-45e6-a3eb-XXX" TYPE="ext4" 
/dev/mapper/mydev-root: UUID="b560f808-db97-4a56-bbf1-XXX" TYPE="ext4" 
/dev/sdc: UUID="3e502de5-696d-f4b4-470e-XXX" TYPE="linux_raid_member" 
/dev/mapper/mydev-swap_1: UUID="49b806fe-95a6-4ddf-9c47-XXX" TYPE="swap" 

hdparm -W 0 /dev/sda (or /dev/sdc) works ok, but this letters could be changed during boot. and i want to use this via disk-uuid.

*stat /dev/disk/by-uuid/

 File: `/dev/disk/by-uuid/4ba40aae-65e2-416b-8f17-XXX' -> `../../sdb1'
 File: `/dev/disk/by-uuid/a7eb2443-c3be-45e6-a3eb-XXX' -> `../../md0'
 File: `/dev/disk/by-uuid/49b806fe-95a6-4ddf-9c47-XXX' -> `../../dm-1'
 File: `/dev/disk/by-uuid/b560f808-db97-4a56-bbf1-XXX' -> `../../dm-0'

if i use hdparm -W 0 /dev/disk/by-uuid/a7eb2443-c3be-45e6-a3eb-XXX — this fails.

/sdb1 -- system hdd
/dm-0 -- /boot on sdb1
/dm-1 -- /root on sdb1

I’m trying to use native /etc/hdparm.conf to disable write_cache on disk-by-uuid.

i dont want to write some script to check what /dev/sdX i should use with hdparm, so im asking what to do. Please help.


I answered:

You’ve tried to use hdparm on the by-uuid device file corresponding to your RAID array (md0). Instead, try running it on the ones corresponding to the physical disks.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.