My January 13th blog To beat Linux: scam the customer drew quite a lot of talkback comments. One of the themes there was fired off by a comment from No-Ax-To-Grind to the effect that the Linux community is somehow afraid of a direct performance comparison against Microsoft.
I'm pretty sure that Microsoft's challenge was little more than a publicity stunt carefully structured to ensure no one could take them up on it, but the fundamental question is interesting: for some range of tasks, which is really faster given identical hardware and skills?
Running a publishable benchmark isn't cheap and, of course, there are both sales and ego reasons for companies to ensure that the results they publish support their sales. In most cases that means they try to get the best possible results -i.e. you're not going to see IBM hire just any idiot off the street to set up and run their TPC/C Linux benchmark: they're going to put some real expertise into getting exactly the results they want, and so will Microsoft and HP and everybody else who gets into this game.
As a result the public benchmarks offered by TPC and SPEC may be good places to look for an answer. Unfortunately, there are few pairs of comparable results and most of the ones that do exist reflect HP's attempts to convince people that the Itanium is worth looking at.
The TPC/H series, for example, shows an that HP Itanium2 server running Windows with SQL-Server on 32 CPUs reached a score of 30,956 on the 3000GB test while essentially the same machine, but with 64 of the same CPUs and running Oracle under HP-UX, claimed a score of 71,847 - 31% more than the doubling you'd expect if scaling were linear and the performance per CPU for the Windows server combination equaled that of Oracle under HP-UX.
You get somewhat better comparability under the TPC/C V5 benchmark. Here, for example, an HP 64 CPU Itanium2 running at 1.5Ghz was used under both software scenarios. With Microsoft SQL Server 2000 Enterprise Ed. 64-bit, running under Microsoft Windows Server 2003 Datacenter Edition 64-bit and the Microsoft COM+ Transaction Monitor, this machine obtained a throughput score of 786,646. With Oracle Database 10g Enterprise Edition running under HP UX 11.iv2 64-Bit Base OS and the BEA Tuxedo 8.0 Transaction Monitor, it reached 1,008,144 to suggest about a 30% performance advantage to Unix.
A pair of 32CPU Itanium2 based NEC machines show the same pattern. With Microsoft SQL Server 2000 Enterprise Ed. 64-bit, Microsoft Windows Server 2003 Datacenter Edition 64-bit, and Microsoft COM+ the test system achieved a score of 577,531. With Oracle Database 10g Enterprise Edition, and BEA Tuxedo 8.1 on SUSE LINUX Enterprise Server 9 it reached 683,575 - an 18% advantage for Unix.
Beyond that, however, there are a pair of two way comparisons in which both sides use the same database and the same Xeon hardware to produce a pure OS to OS comparison that Linux wins - but there's a strange gotcha hiding in the weeds.
The first of these pairings offers a direct comparison between IBM DB2 UDB 8.1 running under Microsoft Windows Server 2003 Enterprise Edition and the same database product running under SUSE LINUX Enterprise Server 9. The Unix result, 5090 on TPC/H at 300GB, is trivially better (about 1.7%) than the Windows result (5,003).
Similarly, a TPC/C V5 comparison between an HP Proliant ML350-T03-X2.8/533 running IBM DB2 UDB Express Edition v8.1 with Microsoft COM+ on Microsoft Windows Server 2003 Standard Edition and the same hardware, database, and monitor but using SUSE LINUX Enterprise Server 9, produced a 1.8% victory for Linux: 18,661 to 18,318.
However close inspection of the detailed reports suggests something very odd: the machine's controller and RAM specifications were the same in both cases - and that's not what you would expect.
Because of the waterfall object control heirarchy built into Windows 5.X (and, indirectly, the Intel Itanium), you maximize performance by using one controller (and one NIC) per CPU. Linux, in contrast, is a true SMP system so you maximize performance by reducing device interupts and loading up on memory for your database cache instead. That's why if you look at Linux benchmark configurations you'll typically see 0.5 controllers or fewer per CPU while Windows benchmarks almost always have 1:1 CPU/Controller ratios.
Since RAM is actually cheaper and faster than controllers and disks, keeping machine configurations essentially identical artificially slows Linux and raises its cost relative to Windows. Cost per transaction is, of course, one of the TPC's critical metrics, and Linux did "win" both comparisons; but it won by a much smaller margin than it should have.
Overall, however, there seem to be two trends visible in both the TPC and SPEC benchmarks:
So there's an interesting twist to the answer for the original question: Although the evidence we do have heavily favors Unix, there isn't enough information to draw firm conclusions. On the other hand the near total absence of truly comparable information can't be a coincidence. Combine that with Microsoft's general refusal to play in the more complex benchmarks and we can guess who's afraid of who - and it isn't Solaris, Linux, or any other Unix.