It seems that the FBI is going to throw out a $170 million software system and start over.

A new FBI (news – web sites) computer program designed to help agents share information to ward off terrorist attacks may have to be scrapped, the agency has concluded, forcing a further delay in a four-year, half-billion-dollar overhaul of its antiquated computer system.

The bureau is so convinced that the software, known as Virtual Case File, will not work as planned that it has taken steps to begin soliciting proposals from outside contractors for new software, officials said.

…

A prototype of the Virtual Case File was delivered to the FBI last month by Science Applications International Corp. of San Diego. But bureau officials consider it inadequate and already outdated, and are using it mainly on a trial basis to glean information from users that will be incorporated in a new design.

Science Applications has received about $170 million from the FBI for its work on the project. Sources said about $100 million of that would be essentially lost if the FBI were to scrap the software.

As someone who works in the field of building software systems, I can speculate a little about the possible reasons for this problem.  One common failing we see in the computer industry is that the software as delivered is not accepted by the customer.  Usually, this comes down to a fundamental misunderstanding of the requirements, either by the designers or sometimes by the customer (that’s not as weird as it sounds, more later on this) and a failure to properly involve stakeholders in design and development.  Further, the fact that they had to spend $170 million to get the system and only then find it isn’t suitable suggests they’re using a waterfall software development model, which while a fairly standard practice (especially in government), is especially prone to this type of failure, since the customer doesn’t really get to see anything until very late in the process (e.x. at system test or even at the user acceptance test stage).

I mentioned above that sometimes systems fail because the customer didn’t understand the requirements.  Some people are probably wondering how this is possible.  What often happens is that a customer sees a “pain point” in their current way of doing things and wants to “fix” it.  Often, you’ll also get conflicting viewpoints on this from different elements within the customer’s organization.  The job of a good designer is to ask questions and ferret out the real core requirements from the customer and get buy-in from all of the stakeholders.  The other side of this coin is the gung-ho, get-it-done-now type of customer who sets requirements that are really too low (i.e. what hardware or software to use, even if it doesn’t really fit the solution) and the designers don’t go back up the chain to find the real system requirements.  It can be difficult to get this kind of customer to reveal their true requirements, since asking questions is seen as either a challenge or a waste of time (e.x. Why aren’t you doing something productive?).  There is also a third problem in some organizations, where requirements are funneled through another group, so the primary stakeholders are not directly involved in requirements reviews between the designers and the customer.  Frequently, these third parties “filter” the requirements based on their own organizational biases so that the real requirements may or may not make it to the designer.

This is the area where I work—I have to understand the customer’s requirements and turn them into a system design.  This is a huge area of work in academia and is of great concern to all IT organizations.  I’ve had some training in the Systems Engineering and Architecture methodology from the SDOE Program offered through the Stevens Institute of Technology (specifically, the 625 and 650 courses).  My employer paid for this training because they’re trying to bring some order to the process of software development and I’ve found that while a bit tedious at first it offers some advantages.  Primarily, the proper management of requirements, while it takes longer up front, is shown to reduce the overall cost of the project as well as reducing defects and their associated costs.  A defect in requirements that is caught during a requirements review may only cost a few hours to fix, whereas a defect that makes it to the system test stage could cost hundreds or thousands of hours to fix (and programming time isn’t cheap), with corresponding delays in schedule as well as reduced customer satisfaction.  The goal of SE&A is to design the system the customer wants and be able to prove it when done (i.e. be able to show objectively that the requirements were met, rather than arguing over what the system is supposed to have done at the end).

While I don’t know what methodology the FBI and SAIC used, I would expect that for this level of failure to occur, the requirements were either not properly documented or understood or they were wrong to begin with and no one questioned them.  If the requirements changed during development (9/11?), then if they were using this type of process they could have evaluated the change in requirements and determined fairly quickly how they would ripple through the system as well as the impact in cost (SE&A promotes traceability of requirements from business/stakeholder needs to System Requirements down to components, so a change to the business requirements immediately tells you which components are likely to be affected). 

The waterfall method I mentioned above is simply one where you do requirements analysis, design, coding, testing, and delivery in that order and it’s all preplanned and laid out at the beginning.  Each step has to finish before the next one starts and the customer doesn’t see anything until testing or delivery.  If there is any doubt about requirements, this type of method is likely doomed to failure or will require heroic efforts on the part of the development team to save, since any misunderstandings can’t be fixed until afterwards and only with great effort and cost.

Since we saw such a spectacular failure with this project, perhaps the FBI should consider an iterative approach.  In this approach, once the requirements are analyzed and agreed to by the stakeholders, design and coding begins on some core part of the system and it is presented to the customer as a prototype.  The customer evaluates the prototype and any feedback is incorporated into the next iteration, which adds some new features as well.  This process repeats as needed until the system is complete. 

In addition to the iterative approach, they should also consider involving the real stakeholders in the requirements gathering and review process.  This includes real, live field agents in addition to their chain of command and the IT staff.  Everyone who touches the system in their jobs must be considered and there is no better way to do that than to actually talk to them.  This may be difficult in the government procurement arena, where these things are typically sent out as requests for bids with the requirements predetermined by some government functionary.  But if the FBI could break through this type of mentality, they may have a chance of delivering a system that actually works.

Link via Slashdot.

If you think spyware is simply the product of some maladjusted geeks in a basement somewhere, you’d be wrong.  It turns out that there is quite a bit of venture capital involved.  The four biggest spyware/malware offenders have a total of $139 million in venture capital investments.

Link via Slashdot.

I got a weird comment to the “Fundamentally Unserious” post from last week.  It consisted of simply the phrase, “I’m shocked” and claimed to be from someone named “Diarmuid Moroney” of the “Dublin International Foundation College.”  The website given seemed kind of weird and I found out that it was a ripoff of the real website with some interesting and naughty bits added to the HTML.  The page didn’t render right in Firefox, and I think I’m glad it didn’t, given this bit that was hidden there:


<a href=“cartoon/01.html”><span class=“red”>disney porn</span></a><span class=“red”>   
            <a href=“cartoon/05.html”>simpsons sex</a>   
            <a href=“cartoon/10.html”>kim possible nude</a>   
            <a href=“cartoon/15.html”>futurama xxx</a></span><br>


Disney Porn!  WTF?!  I had to Google to find out who Kim Possible was.  If you’re the kind of squirrel who gets off on a cartoon teenager, I don’t want to know you.  But I suppose there’s no accounting for taste or the weird variety of things that get people’s motors started.

Anyhow, I’ve deleted the comment, but this is an insidious sort of thing that is hard to catch since you have to investigate the linked website (and in this case look at the source).  The only red flag is that the comment itself doesn’t quite make sense.  For example, I could understand being shocked by the bishop’s statement, but people who would be shocked would say more than just “I’m shocked.” 

As usual, I ignored the repeated reminder emails that my passwords on a number of systems were going to expire soon.  I get so much other email that I tend to see them and immediately forget about them.  So this morning I go to log on and get the message that I have to change my password now.  I always hate picking new passwords, because they’re such a pain.  Not only do I have to adhere to my company’s password rules, I have to be able to remember the damn thing.  I tried several and was growing increasingly frustrated at finding something memorable and “secure” (at least according to their rules).  Anyhow, I finally found something that would work and went into one of our support websites to change the password on six Unix systems all at once.  Except it did something I didn’t expect this time (something it’s never done in the past three years I’ve been using this system).  Whoever runs the system had changed it to send me an email with my new password in cleartext.  Normally, this is just annoying, but it’s not that big of a deal.

Unfortunately, the system also CC’ed my manager on the emails.  One email for each of the six systems.  Did I mention that my password used one of the seven words that are forbidden on TV and radio?  That made for an interesting phone call from my manager about ‘appropriate’ words for use in a business environment.  Of course, the fact that she was never supposed to see the password didn’t make much difference (managers are supposed to be copied on passwords for new accounts and for password resets, but not with user-selected password changes).

If I’d have known that anyone else would have seen the password that I might have chosen a different password selection algorithm.  Anyhow, I sent a nastygram to the support address for the password website.  Perhaps they can be pursuaded to adhere to the password rules before anyone else gets their wrists slapped for an ‘inappropriate’ password that is supposed to be kept private.

Within a day of upgrading to Expression Engine I started getting spam on my contact email address.  The reason for this is that unlike in the old weblog I’d put a live “mailto:” link in the “Contact” part of the page header.  I could have used some Javascript to obfuscate the address to keep it from being harvested, but I’ve always resisted using Javascript when it could interfere with the site’s function if it is disabled.  I know that the majority of web designers hate it when they can’t use Javascript, but I’ve been strongly affected by my company’s guidelines concerning Section 508 compliance for web sites (i.e. disabled access).  One of the core rules is that you can’t use Javascript for “essential tasks” unless there is an alternate method for performing the task.  If Javascript is disabled, the site should degrade gracefully.  Accessibility is one reason why I’ve never enabled ‘CAPTCHAS’ for my comments section.  It’s also why I despise websites that use Flash for their core pages.  It’s OK for special content or presentations, but for basic site navigation and presentation it blows chunks (not only does it interfere with assistive technology, you can’t do freakin’ bookmarks).  While I haven’t bothered to run the site through an accessibility scanner, I try to use valid XHTML in all of my posts as well as in the templates.  But I digress . . .

While Expression Engine has a contact module, it seemed like a nice little diversionary programming task to write my own contact form.  My requirements for the contact module:

1. Must not reveal the email address on any part of the HTML that is generated for the form (to prevent spammers from harvesting the email).
2. Must not accept an email address as a parameter (to prevent spammers from relaying through it).
3. Must support invocation from multiple parts of my site with different destination addresses for each one (i.e. I have different contact addresses for my gun show listings than I do for my guns or my weblog).

Along the way, I also decided I wanted to be able to use it as a pop-up window and use CSS for formatting.  Invoking it as a pop-up window involved a bit of Javascript, but it degrades well when Javascript is disabled and displays the form in the original browser window.  As I was coding, I realized that it might be useful for other people to use on their sites, especially if they didn’t have Expression Engine.  So I made sure that all configuration and customization fields were in a single file, which is separated from the code.  In addition, all of the strings used to build the form are in arrays at the top of the code module, to make future enhancements for translation easier.  The script is in use right now and you’ll see it if you click the “Contact” link in the header of any of my Expression Engine pages.

If anyone wants to use these scripts, they can be downloaded here:

• contact.tar.tz (Tarred, Gzipped)
• contact.zip(Zipped)

These scripts require PHP, but shouldn’t need any additional modules. 

Installation
Note: If you only have FTP access to your server (my condolences), you’ll need to extract the archive on your system and then upload the files after making the changes I give below.

1. Upload the files to your server and extract the archive/zip file to the directory where you’ll be running the script.
2. Modify continfo.php as follows:
      
   • Modify “$contacts” to specify the public target name along with the email address associated with it.  You can have as many targets as you like, although only one is accepted per invocation.

      

   • Modify “$base_url” to be the web accessible URL of the directory where the script will be invoked.  It must end with a “/”.  (Example: If the script will be at “http://www.yourdomain/your/directory/contact.php”, then you’d put “http://www.yourdomain/your/directory/” here).

      

   • (Optional) Modify “$page_title” to be the title you want to use for your contact page.

    

Invocation/Usage
When invoking the script, you must specify one of the targets you used in the “$contacts” array in continfo.php.  This is done using CGI GET parameters.  Using the example URL from the installation section, and assuming you created a target called “mycontact”, you’d invoke the script by calling “http://www.yourdomain/your/directory/contact.php?target=mycontact”. 

The CSS and the HTML assume that they’ll be used in a 600×600 pop-up window.  To invoke them in a pop-up, you’ll need to use some Javascript on the invocation (if Javascript is disabled in the browser, the pop-up code is ignored and the script opens in the current window).  Here’s a sample:

<a href=“http://www.yourdomain/your/directory/contact.php?target=mycontact” onclick=“OpenContactForm(this.href); return false”>Contact</a>


You’ll also need the Javascript function “OpenContactForm”:

function OpenContactForm(c)
{
window.open(c,
'contact',
'width=600,height=600,scrollbars=yes,status=yes');
}


Further customization
You can customize the colors and page formatting by editing ‘contact.css’.  The text that is displayed on the form is all contained in arrays at the beginning of ‘contact.php’.  You can rename ‘contact.php’ to anything you like.  You can also rename ‘contact’.css’, but you’ll need to edit ‘continfo.php’ to tell the script the new name of the CSS file.

Warning
This code is released ‘as-is’, so you use it at your own risk.  This script doesn’t use any kind of user token, so it is vulnerable to DOS attack if someone took the time to understand the parameters in the form and write another script to post the form over and over.  The attacker could not cause the script to send email to arbitrary addresses (without cracking the server, but you might have bigger problems if that happened).  Instead, the attacker would end up flooding the inbox for the selected target address.  The Expression Engine contact module uses a form of user token (in fact, all of their modules that generate forms have this built-in), so you’re better off using the EE module if you have that.  This script isn’t likely to be modified to fix this problem, since it would require a database to fix it right, and I didn’t want to make this script any more complicated than it is. 

Now that I’m getting the hang of Expression Engine, it didn’t take me very long to convert my gun show listings to an EE weblog. It took a little hacking of the EE code itself, because it doesn’t display future entries by default. However, the support forums and knowledge weblog explained what to do.

Now, each show is a weblog entry whose entry date is the date of the show. The table of shows on the main page is generated from the weblog entries, displayed in ascending order. In addition to being automatically generated (in the past I had to hand edit the HTML every time the page changed), I now allow comments on each show (although they’re moderated to prevent any spam nonsense).

To prevent anyone who linked to my old page from being broken, I installed a custom 404 handler that uses PHP to look at the requested URL and determines where to send the user. Here’s what that code looks like, in case anyone is interested in doing something similar.

<?php

?>

The Federal Trade Commission finally made its report to Congress today on a proposed do-not-spam registry.

The Federal Trade Commission today told Congress that, at the present time, a National Do Not Email Registry would fail to reduce the amount of spam consumers receive, might increase it, and could not be enforced effectively. In a report filed in response to a statutory mandate, the FTC also said that anti-spam efforts should focus on creating a robust e-mail authentication system that would prevent spammers from hiding their tracks and thereby evading Internet service providers’ anti-spam filters and law enforcement. To help focus these efforts, the FTC today announced that it will be sponsoring a Fall 2004 Authentication Summit to encourage a thorough analysis of possible authentication systems and their swift deployment.

…

The FTC’s report analyzed three types of possible registries: a registry containing individual e-mail addresses; a registry containing the names of domains that did not wish to receive spam; and a registry of individual names that required all unsolicited commercial e-mail to be sent via an independent third party that would deliver messages only to those email addresses not on the registry.

The FTC studied these three possible registry models by reviewing registry proposals from some of the nation’s largest Internet, computer, and database management firms; consulted with more than 80 individuals representing more than 50 organizations including consumer groups, e-mail marketers, anti-spam advocates, and others; demanded information from the seven ISPs that control over 50 percent of the market for consumer e-mail accounts; and retained the services of three of the nation�s preeminent computer scientists.

The Report concludes that all three possible registry models could not be enforced effectively. A registry of individual email addresses also suffers from severe security/privacy risks that would likely result in registered addresses receiving more spam because spammers would use such a registry as a directory of valid email addresses. It ultimately would become the National Do Spam List. Furthermore, a registry of domains would have no impact on spam and a third-party forwarding service model could have a devastating impact on the e-mail system.

It seems kind of silly that they had to waste all this time figuring this out.  Anyone with any knowledge of the unscruplous practices of spammers could have told them that any kind of registry would just turn into a list of validated email addresses that the spammers would hit harder than ever.

I know that Microsoft and a number of other big companies are behind the idea of an authenticated sender system.  The only problem I see is that given time the spammers will likely find some way to corrupt the system or work around it.  Perhaps it’s time for the real spam solution: find some of the spammers and kill them slowly and painfully as examples to the others.

I’ve been playing with the 14-day free trial version of Expression Engine.  I wasn’t able to install it on my hosting account because the trial version uses Zend Optimizer to encrypt/obfuscate their PHP source.  Unfortunately, Zend Optimizer required root access for installation.  The fully licensed version will not have this problem, though.

In the meantime I installed the trial on my Linux system at home and created two weblogs.  I then imported all the entries from this site as well as The Bitch Girls into those weblogs and set about trying to figure out their template system.  While you can create multiple weblogs in Expression Engine, you have to do a lot of template customization to make those weblogs truly separate.  I think I’ve finally gotten it all figured out, at least in terms of how to code the templates.

There appears to be some kind of problem with the archive page, though.  It just displays the year and month and then nothing.  I will check the support forums to see if this is a known problem or if there is a fix.

I’m not sure if I’ll be implementing Expression Engine anytime soon, but I wanted to give it a try to see if it would be useful.  So far, though, it’s far superior in terms of posting performance in that no rebuilds have to be done for new posts or when changing templates/layout.  Also, comments seem to work faster than with MT.  However, this all comes at the price of being a little harder on the server since all pages are dynamic and require DB access to display them.  In a typical installation, using PHP as an Apache module, this would be mitigated somewhat by caching.  However, my webhost’s default configuration is to run PHP as a CGI process.  This provides better security in that they use a SUEXEC wrapper to allow the CGI process to run under my user ID rather than the ID of the server.  This allows for better file access control (i.e. you don’t have to set files and directories with 666 or 777 permissions) at the expense of the loss of caching.  I may try setting up my home system to run PHP as a CGI process to see how that affects performance.  I’d hate to finish the evaluation and buy the product only to find that it performs poorly in the actual production environment.

I’ve been running four weblogs with a total of nine authors on my current copy of MT (I interpreted this as being legal, since I don’t charge for it and I don’t offer a service; all the weblogs are run from a single instance of MT on a single server).  Movable Type just announced a new license model for MT 3.0.  From what I can see of their new license model, my current usage would require the purchase of “Movable Type 3.0 Personal Edition Volume License II” at a cost of $149.95.  They also offer a free version, but it is limited to no more than four weblogs with one author, so that isn’t an option.

Before I put down $150 for weblogging software, I’m going to investigate the alternatives.  Of course, the hassle of migrating to new software might make it worth the cost, but that remains to be seen.  MT 3.0 had better have improved a lot of things significantly if they want to charge those kinds of prices.

More at Slashdot. 

Update:  I really like some of the features I see in pMachine’s Expression Engine.  For the same price as the “personal” MT 3.0 version I could get Expression Engine for noncommercial use and have unlimited authors and weblogs.  I may give it a try one of these days.  But for now, as long as MT 2.661 does the job I’m going to leave well enough alone.

OptIn Real Big has obtained an injunction against IronPort, the company that owns SpamCop.  For those who don’t know, SpamCop is a service that automatically decodes the headers on spam emails and sends notifications to the spammer’s ISP about the spam.  Since sending spam is a violation of the TOS or AUP for most ISPs, this usually gets the spammer kicked off the service.

Scott Richter, the self-professed “Spam King” and president of e-mail marketing company OptIn, sued IronPort and SpamCop on April 29 for allegedly interfering with his business and causing his Internet service providers to block his company’s e-mail. He also charged SpamCop with not disclosing the identity of people who complain about its e-mail, thereby aiding potential violations of the Can-Spam Act, which requires the removal of people from future mailings if they so choose.

“This whole system is done in the dark—we don’t know who’s complaining, what the substance of the complaint is, and there’s no opportunity to correct the complaint” to comply with provisions in the Can-Spam Act that require a company to remove people from a mailing list, said Steven Richter, an attorney for OptIn.

“We’re asking for the right to handle complaints.”

First, there should be enough information in a SpamCop ISP report to identify the offender and the recipient.  In fact, there is more information than that there, because I have one spammer who added my SpamCop address to their “opt-in” list (as I have noted before, I use a unique address for all of my online dealings, one which indicates the company that I’m doing business with, and since I own several domains I know these addresses have not been used before).

Frankly, this points out one of the weaknesses (of which there are many) of the CAN SPAM act.  It requires that the receiver of spam contact the sender to be removed.  Since spammers have proven themselves to be unscrupulous bastards, no one with any sense will contact the spammer to ask to be removed, since this usually just confirms that the address is active and results in more spam. 

I know that in my own dealings with Opt In Real Big that their claims of having obtained permission to send me spam are lies.  They were using an address that I had given to a company that I knew I had told not to send me any promotional emails.  Further, they were advertising things from another company entirely. 

If they think this is going to make life any easier for them, they’re sadly mistaken, though.  If I get any spam from these bastards while the injunction is in progress I’ll just report them to their ISP myself.  I have the knowledge to hunt them down on my own, it’s just that SpamCop provided a handy automated interface to do what would otherwise would take me several steps to do myself.

More discussion of this topic is available on Slashdot.