In the realm of technology, the promise of software has often collided with reality, revealing a troubling trend: despite trillions of dollars spent, software projects continue to fail at alarming rates. Major contributors to these shortcomings are boundaries to human creativity and too ambitious project objectives. On top of that, most are not good at handling complexity and risks. Such non-delivery ripples across countries, sending dangerous shockwaves. They affect the credibility of our governments but more importantly the everyday lives of millions of people who rely on these systems.
Despite budgets having ballooned, success rates in software and IT projects have not meaningfully improved over the past 20 years. Examples abound, such as Michigan’s MiDAS unemployment system and Australia’s Centrelink “Robodebt” welfare programs, both of which represent significant missteps in automated decision-making. These failures expose a much bigger, pervasive issue throughout the software development industry. Firms often grossly misjudge or entirely ignore the expenses associated with IT project failures.
The Human Element in Software Development
At the end of the day human imagination is more important than software code. Projects without a strong articulation of purpose and need, or ones that struggle with the tension of complexity, always risk walking a tightrope.
Failures occur not simply because of technical deficiencies but due to deep-rooted misconceptions regarding what algorithms can and cannot do. Government officials in Michigan and Australia put too much faith in automated systems, believing they could do a better job than human judgement.
This blind faith in deregulation has been quite literally disastrous. In Michigan, state officials used MiDAS to wrongly accuse tens of thousands of residents of unemployment fraud. Likewise, Centrelink’s Robodebt system incorrectly identified hundreds of thousands of Australians as welfare cheats. Beyond just the loss of dollars gained or lost from a net perspective, these decisions have created incalculable financial pressure and worry for millions of families.
“Anyone can make a mistake, but only an idiot persists in his error.” – Cicero
The assumption that technology would make everything more efficient without taking into account some of the pitfalls has turned out to be the wrong one. The impact of these failures civil rights groups are calling attention to the necessity of human oversight and intervention when necessary in automated systems.
The Cost of IT Failures
The cost of IT failures can be monumental. The Canadian government’s CA $310 million Phoenix payroll system should be a warning. Initially launched in April 2016, it soon became clear that the project was a disaster in terms of mismanagement and improper conception. Executives were sure they were the ones sophisticated enough to build their own payroll. They were going to adapt a publicly available package for 80,000 different pay rules in 105 collective agreements.
In practice, the project’s execution was an utter disaster. The developer/technical team worked hard to plan to achieve this at less than 60 percent of the vendor’s initial estimated cost. They soon found that this goal was totally infeasible. Close to 70 percent of the 430,000 active and retired federal workers paid through the Phoenix system have experienced paycheck problems. More than eight-and-a-half years, in fact.
The fiscal impact is devastating. IT project failures can create generational scars that last for years, or in some cases, decades. The federal government has not fully credited these costs. This oversight generates a dangerous and unrealistic picture of what is actually required to create successful software systems of the future.
The Future of Software Development
Against this backdrop of these past failures, the majority of the thousands of existing, outdated Canadian government IT systems need to be replaced or modernized. There’s a big concern that new developments might end up recreating disasters like Phoenix or other past debacles.
We need to resolve not to forget the lessons learned from earlier failures as we plan for a new expedition. Project managers and stakeholders need to be careful about the systems they’re introducing. Most importantly, they need a realistic picture of what it all entails.
Henry Petroski’s claim that “to engineer is human” perfectly conveys the heart of this difficulty. Value failure as an important aspect of the design process. It aids teams in understanding what is realistic to expect and leads to better success.