The backup of my damaged RAID array finished yesterday afternoon (it only took about three days, rather than the week I’d originally thought).  I spent some time verifying that all the files had been copied and opened a random selection of them to verify their contents to satisfy myself that the backup would be useful* if the RAID array died completely.

After confirming the backup, I took down the system and pulled the bad drive and replaced it, rebooted, ran the RAID BIOS setup tool to add the new drive, and booted into Linux.  Once the system was up, the driver initiated the rebuild and it was a matter of waiting.  Fortunately, it rebuilt without any errors or problems.  So now I have at least double-redundancy for most of my data, and in some cases triple-redundancy. 

I keep the majority of my work files on the Linux RAID-5 system and access them from my desktop via Samba shares.  On the desktop I have the directory set to “Make available offline” so that if the Linux system goes down I can continue working.  The Linux system also serves as a secondary backup of this website (Dreamhost has backups, but it’s prudent to assume they won’t work and keep a backup of your own).  Every night at 1:00am the Linux system uses ssh to invoke a couple of database backup commands that leave tar-gzipped files in a temp directory on my web hosting account.  Then it uses rsync via ssh to backup everything in my account’s home directory.  Now, I also have a cron job on my second Linux system (the one with the scanner) that runs at 4:00am and backs up everything in my home directory on the RAID system (I picked 4:00am to give the web backup plenty of time to complete, although it’s usually done in 5 to 10 minutes).  Both backup jobs email me their output so I can confirm that they ran to completion.  This means that my work files are triple-redundant (on the RAID-5 system, on the desktop, and on the scanner backup system), my regular files (photos, emails, etc) are double-redundant (on RAID-5 and on the scanner backup), and my web files are quadruple-redundant (web server, Dreamhost backup, RAID-5 backup, scanner backup). 

That first rsync of my home directory (which took over two days) transferred 120GB and it got me to wondering just what I was using so much space for.  On inspection I found that I was using a lot of space for system backups from old systems that I no longer had (one backup was from 2003).  I also had a bunch of space taken up with software images that I no longer needed (like old SuSE DVD ISO’s, J2EE server install packages, application development toolkits, etc).  Deleting all that junk reclaimed 90GB of space. 

* (Click “Read More…” for the rest of the story)

Verifying a backup if just as important has having the backup in the first place.  If you rely on a third party to backup your data, it’s a good idea to ask them to retrieve a file for you from time to time just to make sure the backups work.  If they do incremental backups, you may even wish to ask for the file from a specific date. 

I do this now because of an incident that happened a number of years ago when I was still a programmer.  Our development was done on a Unix system.  The source code was in a source code control system, so even if the system died, we’d still have our code.  However, many of us had example code, documentation, and notes in our home directories.  The system was backed up to tape every night, and the operators were diligent about changing the tape and seeing that the backups completed each night. 

I don’t remember why it had to be done (it was either an OS upgrade or replacing a hard drive), but over a weekend an operator was going to have to wipe the system, reinstall, and then restore our data.  Unfortunately, he did not verify the latest backup before wiping the disks.  When he went to restore our files he found that the tape was bad.  The backup software didn’t catch it, so even though it had appeared to complete successfully, the tape was useless.  Luckily he found an earlier backup, but we still lost a month or so of data in our home directories.