Around 9 years ago I built a machine with the sole purpose of serving as my personal file server. It took me several months to find the parts, and well, I was in a tight budget, but still, I managed to create a little machine with six 3TB hard drives, from which I created a RAID5 array (by software, RAID controllers are pretty expensive). I assembled the array with no spare drive, as I wanted to make the most out of all that space… for, you know, backups (wink wink).
Around 5 years later, one of the drives died, so I shucked an external drive that my then girlfriend (and present wife) lended me, which ended up being almost the same model as the original drives I bought.
So, then there I was, trying to rebuild my software array. I don’t
know by heart how to use mdadm, but I had no issues then, I just
found the failing drive (it was clicking), remove it, connect the new
one, configure the drive for the array and then add it so it starts
rebuilding.
To the date I don’t know which is the best approach, but at the time, I created the partitions first and then assembled the array later over the partitions. I didn’t used LVM, well… just because I didn’t knew enough about it.
Well… the thing is that, a couple of months ago, around four years
after the first dead drive, another one died, but luckily, my Linux
computer had two: one 3TB and one 4TB. The operating system was
installed in the 3TB drive, and the 4TB drive was full
of… ejem… backups, so I had to find another drive to backup the
backups, in order to move the operating system to the 4TB drive (I
hate reinstallations). I use a partition scheme with a partition for
each important directory: one for /, one for /home, and so on, so
I copied the same scheme, just adding the extra space to the /home
partition, and then it was time to do the array recovery again.
Now, one thing about the file server is that I installed the operating system in a 16GB USB flash drive. I had the intention of getting a compact flash card for that, but they were (and still are) somewhat expensive. So another surprise is that said flash drive also died, but that’s why I keep a backup image in case that happens.
So, I found another temporary flash drive (let’s hope it also lasts
9 years, hehe), I wrote the image, rebooted the file server, and
started the process of recovering the array, and to my surprise,
another drive is marked faulty!!! So the array had only 4 working
drives out of 6, one needed to be rebuilt and one faulty… I checked
the logs and there they are, a lot of low level read errors from the
drive. Using smartctl I also found that the drive has logged errors
internally, so this one is about to die…
But it still works… so if I can reassemble the array, forcing the faulty drive so the new one can be rebuilt… then I can try to rescue the most valuable … ejem… backups from the server. At this point, any other drive could fail: four of them have more than 60,000 hours (this is actually not that much for 9 years… it should be over 100k).
Anyway… let’s try to save the data. I turned off the computer, made sure all the cables were ok, restarted the computer, and then I recreated the array, forcing the faulty one to be added as being clean, but keeping out the one that needs rebuilding:
$ mdadm --create /dev/md0 --assume-clean -l5 -n6 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 missing \
--readonly
I found that the --readonly option will help me verify that it would
work, and it did, the data seemed to be ok. So I stopped and
reassembled the array as normal (still with the missing drive):
$ mdadm --stop /dev/md0
$ mdadm --create /dev/md0 -l5 -n6 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 missing
And then I added the last one so it starts rebuilding:
$ mdadm --manage /dev/md0 -a /dev/sdf1
And voilà! An almost working array!
$ mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Mar 11 21:13:45 2021
Raid Level : raid5
Array Size : 14650670080 (13971.97 GiB 15002.29 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Mar 11 23:08:28 2021
State : clean, degraded, recovering
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Rebuild Status : 19% complete
Name : aleph:0 (local to host aleph)
UUID : 0efe8b93:fe4d0b42:48858b52:f6031e84
Events : 1352
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
6 8 81 5 spare rebuilding /dev/sdf1
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdf1[6] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
14650670080 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/5] [UUUUU_]
[===>.................] recovery = 19.7% (579513304/2930134016) finish=468.0min speed=83705K/sec
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
Now let’s hope it lives long enough to get all those precious backups out…