I got asked, the other day, why big systems projects usually fail - and asked a question in response: "why does it take so long to walk to Calgary?" (About 200 KM).
The key reason IT is often a source of cost and frustration is that senior managers - CEOs, CFOs, and board members - usually have neither the expertise to direct IT, nor the interest and resources needed to get that expertise externally. In effect most of the senior people I know cause IT to fail, and thereby defraud shareholders by failing to discharge their own responsibilities, essentially by assuming away their responsibility for IT -and then blaming "computers" when that behaviour proves to have unhappy practical consequences.
What's going on is that they have the expertise and social skills needed to see that their IT Joe is a decent, hard working, expert in his field - but they don't have either the knowledge or the courage needed to see beyond that: to understand what he actually does or how that compares to what he should be doing.
Senior management can, of course, hire consultants and auditors to review Joe's operation, but the people doing that job are usually either frauds or Joes: they may wear more expensive suits and they usually have better social skills, but their credentials for doing the review are usually either non existent or about the same as Joe's credentials for doing the job. Thus they can, at best, evaluate Joe's work in the same terms that he uses: verifying that industry standard processes and tools are used in industry standard ways -Why? because if they weren't Joe, they wouldn't be experts, and if they're experts they share Joe's world view, Joe's career, and Joe's commitment to walking.
Peel that problem away and ask why Joe doesn't get better at his job - why nearly forty five years after the 360 revolutionised data processing the industry still can't reliably deliver a working system on time and budget - and you'll find that Joe has gotten better at his job, but no matter how fast he walks, he can't get to Calgary in time to meet expectations set by cars and airplanes.
So why does he insist on walking? Because that's what Joe's do: they walk. You'd think external pressure would force change - that normal evolutionary processes would drive adaptation, but what we see is that this only happens within the context set by walking. Why? largely, I think, because the CEOs and other senior people Joe reports to are unable to break out of his context: so instead of asking why he insists on walking when he could drive, they try to get him walking faster.
And, from Joe's perspective, we have the quintessential IT bottom line: in IT failure succeeds and success fails.
Joe's career wouldn't be helped by becoming the guy who fixes the phones: that guy's systems work, and nobody knows who he is. What Joe's career needs is bigger budgets, more staff, face time with the bosses - things he gets by being the photocopier guy: that guy's systems fail a lot, but he's known and liked by everyone because he's always there to shoulder the load when crisises hit. In an industry defined by backpackers, the guy with the truck can't be seen - and doesn't get promoted.
Bottom line, why do big organizations like Ontario's department of Social Services lurch from one hundred million dollar IT failure to another? because the pressure driving industry evolution is the pressure to build careers and make money, while the top managers whose job it is to prune off evolutionary dead ends, lack both the knowledge and the courage needed to do it.