While the RAID array is rebuilding from the previous section, this is what happens when a second disk is pulled.
root@debian:/data# ls
lost+found testfile1 testfile2 testfile3
root@debian:/data# cat testfile3 >> testfile4
[first disk is pulled here]
root@debian:/data# tail -f /var/log/messages
sd 2:0:3:0: [sdd] Unhandled error code
sd 2:0:3:0: [sdd] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
sd 2:0:3:0: [sdd] CDB: Write(10): 2a 00 00 14 20 00 00 30 00 00
RAID5 conf printout:
- - - rd:4 wd:3
disk 0, o:1, dev:sdb
disk 1, o:1, dev:sdc
disk 2, o:0, dev:sdd
disk 3, o:1, dev:sde
RAID5 conf printout:
- - - rd:4 wd:3
disk 0, o:1, dev:sdb
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
RAID5 conf printout:
- - - rd:4 wd:3
disk 0, o:1, dev:sdb
disk 1, o:1, dev:sdc
disk 2, o:1, dev:sdf
disk 3, o:1, dev:sde
md: recovery of RAID array md0
[second disk is pulled here]
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 00 13 28 00 00 04 00 00
lost page write due to I/O error on md0p1
lost page write due to I/O error on md0p1
lost page write due to I/O error on md0p1
lost page write due to I/O error on md0p1
lost page write due to I/O error on md0p1
JBD2: Detected IO errors while flushing file data on md0p1-8
JBD2: Detected IO errors while flushing file data on md0p1-8
md: md0: recovery done.
RAID5 conf printout:
- - - rd:4 wd:2
disk 0, o:0, dev:sdb
disk 1, o:1, dev:sdc
disk 2, o:1, dev:sdf
disk 3, o:1, dev:sde
RAID5 conf printout:
- - - rd:4 wd:2
disk 1, o:1, dev:sdc
disk 2, o:1, dev:sdf
disk 3, o:1, dev:sde
RAID5 conf printout:
- - - rd:4 wd:2
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
cat: write error: Read-only file system
This box is an unhappy camper. As usual, when the first drive is pulled it is discovered and md removes the bad drive from the array and places the spare in. About 20 seconds later I pulled another drive from the VM and it is detected. Since reads and writes were being performed on the volume the system complains about I/O errors. Interestingly, md removes the second failed disk (/dev/sdb) from the array and then it removes the hot spare that was rebuilding (/dev/sdf). The filesystem is then placed in read-only mode.
root@debian:/data# cat /proc/mdstat
md0 : active raid5 sde[5] sdf[4](S) sdd[2](F) sdc[1] sdb[0](F)
15724032 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [_U_U]
mdstat shows that md has failed /dev/sdd and /dev/sdb. /dev/sdf is changed back to a spare device. /data is no longer usable but the system does keep it mounted.