% fortune -ae paul murphy

In praise of IT standardization

Here's the "Synopsis" from a TIGA ([U.S] Treasury Inspector General for Tax Administration) report about the use of web servers in the IRS:

The IRS requires that business units register all internal web sites and web servers with the Web Services Division in the Modernization and Information Technology Services organization.

We obtained a September 2007 network scan from the IRS Computer Security Incident Response Center that identified 2,093 potential web servers connected to the IRS network. We compared the scan results to the web registration database and identified 1,811 web servers that were not in the web registration database. These 1,811 web servers were not authorized to connect to the IRS network. We recognize that some of these unauthorized web servers could be legitimate web servers supporting IRS operations. For example, the Enterprise Operations organization was able to show that 661 (36 percent) of the 1,811 web servers had a legitimate business purpose.

The risk exists that the remaining 1,150 unauthorized web servers are being used for non-business purposes. Due to resource constraints, we conducted only limited tests to identify non-business web servers and found none. We did identify situations in which some unauthorized web servers were inadvertently running web services.

We attribute the existence of unauthorized web servers to 1) web server owners not registering their servers with the web registration program, and 2) responsibility for the web registration program remaining unassigned since September 2006. Lack of ownership over the web registration program adversely affected the maintenance and inventory of the web registration database. According to IRS procedures, if a web server is not registered, it might be blocked from delivering information to the network. Because no office had responsibility for the web registration program, this requirement was not enforced, and web servers were allowed to be connected without proper authorization and accountability.

And here are the report's recommendations:

We recommended that the Chief Information Officer establish official ownership and assign responsibilities for the web registration program, enforce IRS procedures to block unauthorized web servers from providing data over the IRS network, and require an annual scan of web servers and comparison to the web registration database to identify unauthorized web servers. Unauthorized web servers should be immediately disconnected from the IRS network, and inappropriate web sites should be referred to the Treasury Inspector General for Tax Administration Office of Investigations. In addition, web server owners should be required to revalidate the need for the servers annually and immediately notify the Chief Information Officer upon decommission of any web server. The Chief Information Officer should also require quarterly network scans of web servers to measure compliance with security requirements and limit the number of approved web software packages used in the non-modernized environment.

("Modernized, incidently, means "Windows").

You get a better picture of what's really behind this if you read the full report. Basically, one of the re-orgs gutted the office responsible for standardizing, counting, and vetting internal web services in September of 2006 but by August of 2007 this listing still existed officially and still showed 2,878 active web servers desite an on going consolidation effort. Thus a network scan done in September of 2007 showed a total of only 2,093 active servers including 282 survivors from the previous year and 1,811 undocumented internal sites.

Here's a part of what the report has to say about that:

Of greater concern are the 1,811 web servers identified by the CSIRC scan that were not included in the web registration database, shown as the blue portion in Figure 1. These 1,811 web servers represent those that have not been authorized, yet are connected to the IRS network. However, the unauthorized web servers could be legitimate servers supporting IRS operations. For example, during our review, the Enterprise Operations organization within the MITS was able to demonstrate that 661 (36 percent) of the 1,811 web servers had legitimate business purposes.

Or, to paraphrase, 64% (1200) of the servers found to be active, didn't have an obvious official business purpose.

And here's their security situation summary on all 2,093 servers found in the security scan:

To evaluate compliance with security guidance, we obtained a CSIRC vulnerability scan of web servers conducted in September 2007. This scan identified 2,093 web servers with at least 1 security vulnerability.

The scan report contained:

- 540 web servers with at least 1 of 160 high-risk vulnerabilities,

- 1,101 web servers with at least 1 of 117 moderate-risk vulnerabilities, and

- 2,092 web servers with at least 1 of 135 low-risk vulnerabilities.

The number of web servers did not equal 2,093 because most web servers contained at least 1 high-, 1 medium-, and 1 low-risk vulnerability.

To complicate matters the software server numbers shown (827 Apache, 1393 IIS) are from a separate June 2007 scan showing a total of 2,568 servers - and "the Office of Enterprise Architecture has approved [only] three web software packages for use: Microsoft® Internet Information Server, IBM WebSphere® Application Server, and Oracle® web software."

There are a couple of things that bother me about this whole report - including:

1. the picture presented is one of internal disarray with obvious evidence of massive internal IT battles and organizational service disruption going on over long periods of time.

2. far too many of the numbers cited in this report end in the digits 0,1, or 3 to be coincidence. As a result I'm guessing these guys lost control of what came from which scan or which report and "balanced" the numbers a bit.

3. Some obvious questions are left unraised: there are 2,093 servers exactly one of which shows no current security vulnerabilities. Wow! what was different about it? IBM websphere is listed as an allowed server technology - but Apache isn't and yet 827 servers in the June scan self identified as Apache. How many of these were not websphere implementations and how did their users get away with that?

4. Who's minding the shop? Consider this bit:

   During our review, we were able to identify whether web servers were laptop and desktop computers based on the computer naming convention. In the population of laptop and desktop computers, we identified the location of 54 unauthorized web servers. We judgmentally selected 19 of these 54 computers at 3 IRS offices and confirmed that they were valid computers on the network but were unintentionally running web services.

   Oh.

So my bottom line is judgemental too: the report is worth reading closely because it exemplifies what can happen in large organizations when the clueless audit the clueless, when leadership is replaced by standardization, and when senior management abdicates all responsibility to set and maintain IT direction.