You have all had them -- IT projects that you've wholeheartedly sponsored, that you sincerely believed would fly. And then you've seen them stall and fall awkwardly back to earth, grounded by their inherent flaws.
You're not alone.
The Standish Group International Inc., a research firm in Dennis, Mass., has been tracking project failure since 1993. Its latest survey indicates that 46 per cent of IT projects were over budget and overdue, and 28 per cent failed altogether. Another of its studies cites even grimmer success rates--only 24 per cent of IT projects undertaken by Fortune 500 companies in 1998 will be completed successfully. The problem has been going on as long as companies have been using information technology.
You may not be able to avoid them, but at least you can recognise them while they're still half--baked. To that end, a group of brave CIOs and IT staffers have granted us insight into the failures with which they've regrettably been involved. Our purpose in recounting these setbacks is not to focus on the failures, but to draw lessons learned and perhaps help other CIOs avoid the same traps.
Some of the CIOs with whom Senior Writers Tom Field, David Pearson and Polly Schneider spoke did so completely on the record; others requested anonymity either for themselves or their employers, and we've respected their wishes.
Reading these case studies does not ensure that you'll never hatch another turkey, but the expert advice may help you avoid eating crow.
Lights Out On LAMP
THE PROJECT:
In 1990 the Washington State Department of Licensing launched its License Application Mitigation Project (LAMP), a five--year, $USUS41.8 million mission to automate the state's vehicle registration and license renewal processes. But when George Lindamood stepped in as director of the state's IS department in 1993, LAMP was hardly glowing. The budget had swelled to a projected $USUS51 million, the legislature changed its schedule in the course of the project, and even if the project were completed, it promised to be a colossal money--waster.
"It was like the Lilliputians dragging down Gulliver," Lindamood says. "The project was stopped dead in its tracks." And even had the project not stalled, he says, "it would have been much too big and obsolete by the time it was finished." Lindamood tried to save the project, but to no avail. In 1997 the plug was pulled on LAMP, but not before seven years and roughly $USUS40 million had been wasted.
THE PROBLEM:
The project's failure stems in part from bad project management, Lindamood says--the undertaking was too big with too few solid deliverables along the way. But the chief downfall was administrative meddling. From authorisation to purchasing to quality assurance, LAMP was overseen by a mix of elected officials and political appointees whose agendas were more personal than project--driven, Lindamood says. He adds, "In government, it's almost irresistible for new people to get in [office] and want to stick their fingers into these things." By design, the project was inexplicably split between in--house developers and a private industry contractor, which led to poor coordination and delays. After Lindamood left, legislators diverted money from LAMP to fund construction of an offramp at a racetrack and passed new licensing and registration laws that altered the project scope and caused further delays.
"The real shame of the failure of a project like LAMP," Lindamood laments today, is that "the people who are ultimately responsible are the least likely to be blamed for it." THE RECOVERY: There was none. LAMP was turned off in 1997, after legislators calculated that the project ultimately would cost $USUS4.2 million more annually to run than the state's $USUS800,000 per year incumbent system. Before the plug was pulled, Lindamood washed his hands of LAMP and of his IS position, opting instead to be a consultant with GartnerGroup Inc., where he now is charged with (his description) "the care and feeding of CIOs." Asked what lessons he learned from the LAMP experience, Lindamood cites several: "LAMP should have been bid entirely to a private industry contractor.
"Government doesn't build [systems] well in--house," he says.
" The project should have been broken down into smaller, measurable chunks, rather than spread out over several years.
" Experienced quality--control personnel should have been hired from private industry and empowered upfront, as opposed to putting political appointees in charge during the project.
" Project managers should have expressed from the start exactly what conditions would necessitate shutting down the project rather than letting it limp along.
Looking back, Lindamood regrets that although he could see from the start that LAMP was headed for disaster, he couldn't save it. "The system doesn't want to hear [that a project is failing]," he says. "The system is the problem; it's the system that leads to the failure." --Tom FieldSure Cure for The BluesTHE PROJECT: Back in the 80s, independent Blue Cross licensees around the country found themselves in a major consolidation mode. Big blues were swallowing up little ones, and the newly massive companies that resulted were having a difficult time merging disparate claims--payment systems. Inside one of the biggest blue sharks in the water, not to be named here, the time was ripe for a major new system--one that could grow along with the company, which was looking to expand beyond health care.
The company brought in a major consultancy, conducted a study, made recommendations and acquired a new claims system. And the $USUS8 billion company got comfortable with the idea of spending $USUS5 million to $USUS15 million over two to three years to complete a project it saw as critical to its continued steady growth.
Chuck Southworth, an applications manager at the project's inception and CIO by the time of its demise, says that, looking back, the directive from senior management to "go get a new system, put it in place and make it work" should have tipped off the company's IT leadership to the problems that were soon to follow. "Things handed down from on high by fiat tend not to ever work," he says.
THE PROBLEM:
Southworth, now CIO of The Martin Agency, an advertising firm in Richmond, Va., says the fatal flaw in the doomed claims system lay in executives' blindness to the human factors involved. Of an IT staff of around 400, 75 of the best and brightest were selected--ahem, make that encouraged--to "volunteer" to implement the new system. They faithfully stepped up and faltered.
"The big issue was backfilling," says Southworth. "You can't have people on two important endeavours at once, so who's going to do these folks' other work while they're concentrating on this big project?" The answer from all those good and talented individuals was that they'd work in two worlds at once. "It's a basic law of physics that you can't be in two places at one time," says Southworth.
Nor did it help matters that the staffers had been motivated to accept the new project largely out of job insecurity. They knew their employer was weighing outsourcing against in--sourcing on the project--and that, for them, the former choice might spell joblessness down the road. After all, the new claims system represented the future of the company. Their willingness to take on "extra" work convinced the company to keep most of the work in--house and provided management with a chance to cast a vote of confidence in its people. Then, too, good and loyal "team players" will often take on more work than they can handle. The consequence, revealed over the ensuing months, was constant conflict over priorities. "It scarred people really badly," says Southworth.
THE RECOVERY:
Southworth was promoted to CIO around nine months into the project. He laboured to bring it under control until the end of the budget year, then approached the vice chairman of the company with the bad news. "I think you'd better sit down," Southworth told his boss. "We need to kill this thing now, before it gets any further out of hand." Southworth was surprised and relieved at the reaction his tidings elicited. After a brief denial--What do you mean kill it? That's out of the question!--he stepped back and let Southworth make the call. "The whole thing actually helped build my personal credibility in the organisation," says Southworth. "They respected that I had the [guts] to come forward with something so painful, and to do it so forthrightly." The company abandoned the project, at a cost of millions. "We had moved people out, rented the space, bought the system, gone through all the planning, and it all wound up on the scrap heap." Eventually, the company took up revamping its claims system once again, notes Southworth, and succeeded. What was different this time? Better dedication of resources.
Here's what Southworth took away from the experience:" No matter how far your staff goes to convince you they're master jugglers, don't assign anyone to more than one major undertaking at a time.
" Downward buy--in means convincing staff that a new priority is more important than whatever they've been working on. To secure acceptance, stress ownership.
"When the system was subsequently deployed--successfully--it was because people believed and understood that it was really theirs" to bring to life, says Southworth.
" Don't let a doomed project run on--or die a quiet death. Admit it's failed and announce the failure.
Southworth says that, over the years, his few failures have taught him more than his many successes. "The key," he adds, "is never having to learn the same lesson twice." Customer DisserviceTHE PROJECT: Four years ago, a state chapter of a well--known national consumer group embarked on what was to be an 18--month, $USUS1 million project to replace its customer database. The new system, to be designed, built and installed by the company's IS staff, would distinguish among different "preferred customer" levels, affording the appropriate products and levels of service on demand. The good news was that the company's IS team did deliver a new system on time. The bad news was that the system didn't work as promised, handling routine transactions smoothly but tripping over more complex ones. Within three weeks of throwing the switch, managers found that the new system couldn't distinguish among customers and that some transactions were being cancelled while others were kept on hold. Immediately, the database was shut down, transactions were processed by hand and new IS leadership was brought in to rebuild both the system and the strained relationship between business and IS.
THE PROBLEM:
IS couldn't--or wouldn't--say no. So eager to please their business partners, IS executives said yes, they could build this system, yes, it would be scaleable and yes, it would be Y2K compliant, on time and within budget. But in reality, IS could deliver on none of the promises. The other big mistake by IS was making this a date--driven project. There was no flexibility for mistakes or unforeseen challenges; developers simply kept their eyes on the calendar and failed to speak up about any glitches they saw along the way. "It was kind of like one of those horror stories you hear about," says the anonymous CIO who inherited the mess a couple of years ago, after more than a dozen people (including the former CIO) lost their jobs because of their roles in this disaster.
THE RECOVERY:
The new CIO immediately empowered "SWAT" teams to diagnose the problems and analyse whether the best course was to repair or replace the new database.
"There wasn't enough good code there to repair," the CIO says, so plan B went into effect. IS partnered with two vendors--one to design and build, the other to oversee the first vendor's work--and a new multimillion--dollar database is projected to be installed sometime in 1999. But replacing the database is only half the battle. IS also must rebuild its fractious relationship with the business side, and that project has its own unique requirements. "I'm trying to dig out of a hole," the new CIO says. "Anything negative that happens now just reinforces the perception [among business people] that IS doesn't know what it's doing." But IS is making headway. The new CIO is making sure new projects are on time and within budget, and he is convinced that his organisation's recovery from the database debacle will secure a new reputation for the group.
Key lessons learned so far:
" IS must break its "code of silence" when working on challenging projects. "If [IS managers] had spoken up sooner, when things started to go wrong, 14 jobs might have been saved," the new CIO says.
" The best cure for failure is a good recovery plan. The new CIO gained his bosses' confidence quickly by stepping in, assessing the problem and developing a plan of action that stated clearly when the recovery would be completed and at what cost.
" You can't do too much too soon. Like the project itself, recovering IS's reputation is a slow process. "I'm trying to do things incrementally to restore customers' confidence," the new CIO says. "You can't do it all in a big bang--although you sure can tear it all down real fast." How long will the recovery take? Can the new CIO really pull it off? "Those are loaded questions," he says, "but I remind people that IS didn't get this way overnight; it won't get turned around overnight."--T. FieldToo Many Cooks In the KitchenTHE PROJECT: In 1996 a major San Francisco bank was poised to roll out an application for tracking customer calls routed to its "elite" group of customer service reps who handled problem cases. Reports provided by the new system would be going directly to the president of the bank and board of directors. An initial product demo seemed sluggish, yet the consultants assured both IS and the telephone banking division managers that all was well. They were wrong. "The source code was so bad it took 20 minutes to load the program on the PC," recalls Jim Daviner, the systems analyst on the project, now a business systems manager for marketing at The Gymboree Corp. in Burlingame, Calif., a children's retail chain.
THE PROBLEM:
The system crashed constantly, could not support multiple users at once and did not meet the bank's security requirements. IS hired a new consultant to help rewrite the application, but after three months the project was killed--resulting in a loss of approximately $USUS200,000 in staff time and consulting fees, and a bad rap for IS.
According to Daviner, the first mistake was the bank's failure to check references and work samples of the consulting firm. Daviner caught a major flaw in the database design in time, but as the project progressed, the programming team became increasingly isolated and hard to work with, refusing to release the source code to the project managers. Daviner says the programmers didn't want the bank to find out that they were, in his words, "pretty inept."But the root of the project's failure was a complicated reporting structure that left no clear line of command. Between Daviner, the lead analyst from the business side, and the four consulting programmers located in Arizona, there were two other layers of management from IS. Another layer, Diviner's boss, was the lead project manager and sponsor--yet she had no direct contact with or control over the programmers and left the company in the middle of the project.
Worse, the project correspondence was lost in the process of cleaning out her PC, Daviner says. At the same time, the bank's IS department was being reorganised. One of the two IS managers on the project resigned to go to another company, and the other was restructured out of a job after the project's problems came to light. In late 1996, with no leadership or business sponsor to rescue a coding disaster, Daviner had to mop up the mess.
THE RECOVERY:
The project was never revived, and Daviner says the biggest loss for the bank is not having access to valuable customer information the system was to deliver. On a positive note, the business units are getting rid of unnecessary layers in projects. Today project managers have direct oversight of the programming consultants and approve all hiring of IS personnel. "Things happen more smoothly now," he says. Daviner (who, as an aside, says his reasons for leaving the bank were not related to this project) shares the following tips: Institute a formal review process for hiring consultants.
" Require change control documentation so that managers can see what changes were made during development, when and why.
" Ownership is essential. When sponsors or top players leave the company or the project, new owners should be identified immediately and supported with documentation.
" Assign a central manager for the project team who is the conduit for communication and decisions. Result: Everyone is on the same page rather than working in parallel and reporting to different managers.
In the end, a disorganised project team with unstable leadership wrought the ruin of an important customer application. Unfortunately, due to the ongoing IS reorganisation at the bank, Daviner says that IS has not improved its project management practices--a clear example of chaos breeding chaos.
--Polly Schneider
Payroll On the Loose
THE PROJECT:
The night before the launch of a new payroll system in a major Boston health--care organisation, project managers were sweating. During a sample run, the off--the--shelf package began cutting checks for negative amounts, for sums larger than the top executive's annual take--home pay and checks that incorrectly matched employee numbers with names, recalls a former director of systems analysis who asked not to be identified. Called in to review the project before the launch, he wound up sitting in the payroll department for six months to help fix the problem.
Payroll was still delivered on time, and out of 7,000 employees, the fiasco affected the checks of only roughly 50 to 100 people. Still, the payroll office had to pull the bad checks and rerun them manually to create new ones every pay--day until the application was debugged. The incident damaged the relationship between IS and the payroll and finance departments, and the programming manager resigned in disgrace.
THE PROBLEM:
"It became apparent the entire system was never tested or run in parallel" with the old system, says the director of systems analysis. IS did not check to ensure that an existing human resources database was compatible with the new payroll system software; the problem was that limit controls on hourly rates were not in place. An entry of 8.0 dollars per hour could be read as 800 dollars per hour if the user did not enter the decimal point, which revealed another problem: No one had been adequately made aware of the differences between the old system and the new one. "There was no turnover between development and production," the director says. Business managers were signing off on the system without really understanding how the people in their department would be using it, he recalls.
"A lack of clear leadership was a problem from the beginning," says the director of systems analysis. The main sponsor, the payroll director, suffered a heart attack three weeks before the launch date and was placed on disability; he was gone for months. Throughout the course of the project, the organisation's top IT director was "uninvolved," the director of systems analysis reveals.
THE RECOVERY:
The organisation hired a consultant to help and provide support, and eight months after the implementation date hired a new director of IS. An administrator who oversaw all IS--related projects was transferred to another area of the company altogether. With the new management, IS established a more formal reporting methodology and some ground rules. Outside consultants would be used on every major project, and business units had to identify a project manager for every project and provide adequate training for the staff. Overall, project managers began to forge better partnerships with users. "The effect was really a mandate to change," says the director of systems analysis. "It created the demand for a more professional way of managing projects." The director of systems analysis (who subsequently went on to become CIO of this organisation and others) has a few lessons to share from the experience:" Test, test and test again: "It's much harder to fix things afterward."" Hire an outside contractor to complete an independent project review.
" Communicate with the business: Send a monthly letter to senior management with updates as the project progresses.
" When buying off--the--shelf software, hire a consultant who understands how the application will run in your environment. The former director says that this is the puzzle piece software vendors often can't deliver.
Looking back, the director--turned--CIO says that the bungled project was a blessing because it helped raise awareness of the company's growing dependence on IT. "Without a major catastrophe, there was a perception that IS didn't affect operations." --P. Schneider The Plan Man LearnethTHE PROJECT: You do the math: Anticipating growth, a $USUS100 million division of a $USUS740 million manufacturing business earmarks $USUS5 million for a new distribution and customer service system to replace its old, sputtering set--up. The project is to take a year and a half to complete, involve 20 business and tech staffers, make up for 20 per cent of the missing source code and set the division in good stead for Y2K.
Two years later, the CIO is fired. A knight in armour, an IT executive with years of experience fixing troubled projects, is brought in to save the day.
He's informed that the challenge ahead is serious, but he's kept in the dark as to how goofed up things have gotten.
THE PROBLEM:
"This project had the two deadly sins built in," reflects the old pro, who's since changed jobs and agreed to share memories of his nightmare under cover of anonymity. "They'd chosen the wrong software solution, thanks to a terribly naive RFP process, and they had no project plan." Worse still, no one owned the project. "IS thought the business users owned it, the business users thought IS owned it and the CEO thought the vendor owned it." There was an unwritten project plan with five or six major milestones, but not a single one of those had been met and no one on the project team could say when they might be met.
The new IT boss pulled together all hands and, in just two days, orchestrated a 2,000 line--item plan. Three months later, with the company's best IT talent and the software vendor's consultants busting their chops, the system broke down altogether. "It was supposed to be the show--stopper for us," says the source, "and we couldn't get customer orders through the system." Internal users began panicking; customers began complaining about incompetence. "We were sending people the wrong stuff and sending everything late." THE RECOVERY: Finally, three years and $US4 million into the project, the CIO polled the 20 players anonymously. Eighteen said they thought the project was beyond saving.
The two who held out were the project manager (a business--side person) and his boss, the VP of manufacturing--apart from the CIO, the two whose necks were most at risk and, in the end, the only two to lose their jobs.
The CIO approached his boss, the CEO. "It was kind of like telling him a relative had died," he recalls. "First he denied it, then he went through a grieving process, then he accepted it. It was just so much money for a division that size to wave in the wind." A settlement was negotiated with the software vendor, and corporate IT stepped in and assumed direct control of all IT operations in the division.
For the CIO, it was time to leave--and to reflect on some hard lessons:" If you don't have extensive experience with the RFP process, get help from someone who does.
" Decide upfront how decisions will be made through the duration of the project. "An essential aspect of good project management is a consistent approach to the inevitable forks in the road. You can't decide by show of hands at one turn, edict on another and a secret ballot on the third."" Look before you leap into a new career "opportunity," and don't be afraid to conduct due diligence if you're taking on a project that's already underway.
"Before I accepted that job, I should have talked to the project manager, asked to see the project documentation, studied the RFP, spoken with the software vendor and talked to some internal users," says the CIO. "Had I known how amiss this project was, I would not have joined that company."--D. Pearson
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.