% fortune -ae paul murphy

Data Processing scores big success! Linux loses, again!

On September 29, 2008, just about all major newscasts across Canada carried something about our new national copy of the American do not call line - and virtually every TV station and newspaper web site had a prominent link to www.dnlc.gc.ca.

The supporting legislation passed late in 2005 and the contract to IBM loyalist Bell ICT out-sourcing was formalized only two years later: in December of 2007.

Here's a bit from a a CBC piece announcing that contract:

The CRTC has picked Bell Canada to operate the National Do Not Call List, giving the company a five-year mandate to block calls from telemarketers for customers who request the service.

Bell will be responsible for registering numbers, providing telemarketers with up-to-date versions of the list and receiving customer complaints about telemarketing calls. The CRTC said it has not yet been decided how those complaints will be resolved, but that should be determined early in the new year.

Phone customers can register their numbers with Bell at no extra charge, and telemarketers are required to subscribe to the list, which the CRTC said must be operational by Sept. 30. Telemarketers must pay Bell to operate the list.

And here's a bit from the Financial Post's write-up for the site's operational success nine months later - on September 30th:

By Sarah Schmidt, Canwest News Service Published: Tuesday, September 30, 2008

OTTAWA -- Consumers eager to add their phone numbers to the newly launched national Do Not Call list to block telemarketers ran into trouble on Tuesday when the Web site crashed on its first day and the telephone sign-up number was not accessible.

The online service went down Tuesday morning within nine hours of the launch at midnight. Canadian Radio-television and Telecommunications Commission spokesman Denis Carmel chalked up early glitches to a "very higher volume that's causing a few initial problems."

About 157,000 people were able to register online (www.LNNTE-DNCL.gc.ca) before the site crashed Tuesday morning. It remained unstable for most of the working day, during which time nearly another 100,000 were able to register. More than 1.5 million people tried to call the toll-free number, but only 74,445 people got through to register.

Really? and only two days after the Chinese communists released audio tape of their astronauts conversing in space - several hours before their launch?

At 8:30 AM in our national capital (Ottawa) on September 30th, the only response available from the https://www.lnnte-dncl.gc.ca site was that: "The site is down for planned maintenance."

(By the way, the site has no obvious user identification controls: when the site was finally working on October 5th it cheerfully let me register one of the the CRTC's own complaint line numbers - but note the politically correct use of https.)

According to netcraft on September 28:
Netblock Owner Bell ICT Outsourcing Solutions 671 La Gauchetiere Ouest Montreal QC CA
IP address 207.35.39.20
OS Linux
Web Server unknown
Last reboot 28-Sep-2008

At 9:30 AM the front page worked and you could enter your phone number, but when you did and hit return you got the informative "The service is not available. Please try again later."

By 10:45AM that helpful note had been replaced; by this:

The page cannot be found

The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

Please try the following:

HTTP Error 404 - File or directory not found.
Internet Information Services (IIS)

Technical Information (for support personnel)

An hour later, the number entry page had been replaced too - by the Microsoft error message.

By 11:45AM the front page looked like this, in Firefox:

but if you clicked on "Register My Number" you got the "The service is not available. Please try again later" message.

By 12:45, you could again see the number registration page (complete with a "Last modified on 07/07 2008" claim) - but filling it in and clicking their "Continue" button still produced the Microsoft server error message.

At 2:15PM it took over a minute to load the front page message: "The service is not available. Please try again later".

At 3:15 the second page loaded - in 82 seconds - but there was a new error message:

Please correct the following errors:

* We are experiencing a temporary technical problem. Please call one of the numbers below for registration, or try again later.

This seems to refer to a CAPTCHA code because it asks you to input one, but doesn't show one - and when you click the line headed "Can't read? Try another" you get "The service is not available. Please try again later."

And, finally, at 9:00 PM on the 30th.. "The connection was refused when attempting to contact https://www.dnlc.gc.ca/" - and that was still happening at 9:00 AM on October 1st.

According to the press release, however "About 157,000 people were able to register online before the site crashed Tuesday morning. It remained unstable for most of the working day, during which time nearly another 100,000 were able to register."

So my first question is simple: if 257,000 successful registrations makes the site a contract conforming success, just when and how did these lucky few get through?

And my second question? Well, at 2:30 PM on Friday, October 2nd it still refused connection attempts but the record at Netcraft had changed:

According to Netcraft on October 2nd:
Netblock Owner unknown
IP address unknown
OS Unknown
Web Server Unknown
Last reboot Unknown

So here's what I think happened: sometime in June or July they assigned someone to develop this site, loaded it on mainframe Linux, and declared it ready. Then when the PR blitz hit on September 29th a lot of journalists and web people testing their links hit it - and problems appeared. In response they opted to rehost on Windows, declared the contract a success - and hung (at least) most of the people who responded to the PR blitz by trying to register out to dry.

And here's a bet: this project will eventually appear on dozens of resumes as an enormous personal success pulled off against overwhelming odds, under unbelievably tight deadlines, and with nothing, absolutely nothing, but the user's welfare and the taxpayer's dollars in mind throughout.

Paul Murphy wrote and published The Unix Guide to Defenestration. Murphy is a 25-year veteran of the I.T. consulting industry, specializing in Unix and Unix-related management issues.