The conference I’m attending is for technical people from my company.  A variety of technical sessions are being offered through Wednesday.  I attended one today concerning creating highly-available systems.  The presenter was a former chief engineer on a nuclear sub who now works for the company.  His point throughout the presentation was that your system’s reliability depends not only on technology, but also on your people and the processes that they follow. 

The most interesting point he brought out was something that was learned from 9/11.  We in the industry were quick to set up redundant systems in geographically separated datacenters and to put procedures in place for offsite backups, but we forgot about distributing people.  The companies that were located in the World Trade Center had offsite backups and redundant systems in place, but they didn’t have people who were ready to step in and fill the positions of those who were killed.  All that physical preparation wasn’t very useful because there were no people available who knew the systems.

I have to admit that I hadn’t given much thought to this topic before, even though I’ve seen what happens when people become too specialized and aren’t cross-trained on each other’s jobs.  It’s one more thing to factor into the next system that I’m involved with.