After the previous post, we had one subcomponent that needed to be run, but the run failed, due to a file not being found (again).  The servers that our app is run on use a read-only networked filesystem to store code.  The people who run them have to follow a process of extracting code and releasing the files to the production filesystem.  Something in this obviously failed, causing some files to be missing.  This isn’t something we’ve ever had happen before, and I’m tempted to blame operator error.  But I did learn that the guy doing the install is also their on-call guy this week.  He’d been working 16 hours yesterday, so it’s likely he missed something.  I feel bad for him, but at the same time, it’s delaying the deployment of our application, and we have to answer back to our customer.  Anyway, by the time we got to that point, it was 2:30am (3:30am for one guy who’s in Atlanta), and we gave up for the night.  We were supposed to start again at 9:00am, but the deployment guy got called out on a severity 1 issue, so we’re on hold again.

Meanwhile, I feel like warmed-over death after only getting about 4.5 hours of sleep.  This is pretty frustrating, since our code has been through three test cycles and has been installed a number of times along the way.  We’re confident that the code is OK and that our installation instructions are good.  It’s just been a series of unforseen obstacles.

Sometimes, it seems like no matter how well planned and run a project is, it just has a dark cloud hanging over it.  We’ve had screw-ups from inexperienced developers, hardware failures, and various other setbacks, yet we managed to pull the schedule out of the fire through extra hours (and good planning on the part of the project manager who allowed some “contingency” time in the test phase).  However, it’s out of our hands now and we’re dependent upon the group hosting the application to get it installed.  All we can do is watch and offer advice and observations.  Unfortunately, I think we’re going to miss our deployment date, given the current situation.  It’s just a feeling right now, because it’s still possible to hit it, if everything comes together just right.  But I’m not very confident, and when it comes to project estimation, my gut feel is usually pretty close.