It Ain't Over . . . Until You Do the Post-Implementation Audit

Scrutinising every aspect of a finished project and acting on the lessons learned can be controversial and difficult. But post-implementation audits are an important way to avoid repeating costly mistakes.

Reader ROI

• Why post-implementation audits are valuable
• Learn from companies doing it right
• Overcome resistance to PIAs in your organisation

As soon as Michael Baker Corporation's IT department installing a Web-based procurement system from ePlus, Bruce Higgins, CIO of the $US405 million engineering and construction company, was bombarded with enquiries from colleagues about the system's effectiveness. He had ample anecdotal evidence suggesting that orders were getting turned around more quickly, but he didn't have tangible proof that the system was improving efficiency.

So he decided to conduct his first ever post-implementation audit (PIA) of the new system. A PIA is a top-to-bottom evaluation of the hard and soft benefits derived from a strategic information system, the security of that system and the project management process for deploying it. From the PIA, Higgins learned that because IT miscalculated the number of people needing to use the new system, the ROI was driven down by the cost of ordering additional licences. The PIA also showed that the system was saving the company more than $US150,000 yearly, however, so Higgins was able to prove the system's worth to his colleagues. And he learned some valuable lessons on what not to do on subsequent projects.

CIOs would do well to follow Higgins's lead in realising that a PIA is a worthwhile way to prove the value of IT. But many don't. In fact, Barbara Gomolski, a research director with Gartner, estimates that a mere 20 per cent of companies take the time to conduct PIAs.

Companies avoid post-implementation audits for many reasons: They take too much time and drain away valuable personnel resources - two things currently in short supply. They require reams of documentation so that processes and results can be validated. Finally, project sponsors and implementers fear that the results of an audit, if unfavourable, will be used against them.

The CIOs at companies successfully performing audits have identified critical success factors, including getting the right people involved, timing the audit properly, and collecting enough documentation to facilitate the smooth execution of a PIA. They pick the right projects to audit (for more information on this, see "How to Select the Right Project for Your First Post-Implementation Audit", page 108). And these CIOs share several key traits on how they ensure that PIAs become a sustained practice in their organisations: They are all committed to continuous improvement, they've made PIAs a part of their project management methodologies, and they have their CEOs' support.

But companies not performing PIAs are missing out on the important benefits such data provides. PIAs provide a thorough approach for proving the value of high-cost, mission-critical IT investments and for gleaning project management best practices, which CIOs can then apply to keep subsequent projects on track. At a time when the value of CIOs, IT departments and IT investments are under increased scrutiny, PIAs are now more critical to CIOs' success and survival than ever before. "With the pressure on business today and the responsibility of IT to help the business units understand where their dollars are being spent, I really have trouble with anyone saying you shouldn't be doing [PIAs] in this [hard economic] time," says Jim Smith, CIO of Sun Life Financial and a strong advocate of PIAs.

Page Break

How to Overcome Resistance to Audits

Many CIOs and their IT organisations shy away from audits because they think they'll be scapegoated if the audit reveals problems or lack of ROI. That's why Michael Barilla, vice president for business information services of Greif, a $US1.6 billion manufacturer of industrial and paper packaging, frequently reminds himself and his IT staff that systems implementations and software deployments are not IT projects, but business projects. So if a project fails, he says, "it's going to fail as a team", and everyone - not just IT - will be accountable.

Gartner's Gomolski concurs. If an audit exposes a shortcoming, she says, that problem shouldn't only reflect on IT, especially if the project was jointly sponsored by the business, IT and finance. "Exposing problems can be a double-edged sword," says Gomolski. "But it's better to be proactive. If you haven't achieved the value you thought you would, figure out what you can do to change that."

Another common concern about PIAs shared by CIOs, IT employees and even businesspeople is that they suck up too much time and require too much toil. But companies that perform PIAs say they shouldn't - and don't - take a lot of time to execute. Honeywell Aerospace aims to spend no more than seven to 10 days conducting an audit, and Sun Life Financial tries to complete PIAs within two weeks.

Furthermore, CIOs who conduct PIAs say the time and resources audits use get recouped on subsequent project implementations. Resistance to audits also often stems from a desire to be rid of a project once the deployment is complete and to move on to the next challenge. The trick is to make it easy for workers to conduct and provide feedback for PIAs. For example, when Sun Life Financial's IT project office needs to solicit feedback on a system from business users, it asks them to fill out an electronic survey, on their own time, that takes no more than 20 minutes to complete. The response rate for those surveys is usually better than 50 per cent and is sometimes close to 100 per cent, says Ed Esposito, director of the project office.

Because Honeywell Aerospace's IT organisation for its Aviation Aftermarket Services unit also has trouble re-engaging business users in a project during an audit, the IT organisation leverages its staffers who have completed training in Six Sigma, a process improvement methodology. Those with Six Sigma training, who are experts in business processes, provide the IT department with the feedback it needs to determine whether a system has streamlined workflows.

Engage the Right People

Who should perform PIAs is a matter of great debate. The most common groups of workers include one or more of the following:

• Members of the project implementation team from IT
• Members of the project implementation team from both IT and the business
• Representatives from a company's internal audit department

At Sun Life Financial, IT's project office leads the PIA process on its own IT and non-IT projects. But it does so in conjunction with the finance department and the company's internal audit department. IT projects are audited from the beginning and on an ongoing basis, rather than at the end of an implementation, which ensures that IT follows sound project methodologies, meets user requirements, stays on budget and implements proper security controls.

Sun Life Financial's approach comes closest to being the best, according to PIA experts. Don Christian, a partner with PricewaterhouseCoopers (PwC), says the PIA team should consist of a businessperson and an IT person who were involved with the implementation, and that it should be led by someone independent, such as an internal auditor who was not part of the project team. Christian says it's better to have a group of people from different functions participate, rather than just an IT team or just internal audit, because they all provide valuable input. The advantage of having members of the IT project team involved is that they're intimately familiar with the benefits, deliverables and requirements of the project. And because they know the project so well, it is easier for them to fully evaluate a project. Having a businessperson on the audit team is important because she can more easily determine if an external factor rather than a systems failure is causing a system to not generate expected value. And an independent auditor is important because he's not afraid to ask tough questions and will prevent the members of the project team who are involved in the audit from softening any findings.

Page Break

Time It Right

When to conduct an audit, just as who should lead it, is a matter of debate. When to start depends on the type of system or application deployed, the amount of time it will take before the application begins generating some results or data, and the amount of time it takes staffers to get acclimated to the new system and new processes. Generally the audit should take place well within a year of implementation.

"When we started the audit a month after the implementation, we didn't have enough data to see if the system was successful or not," says Michael Baker's Higgins. "It's really important to make sure the system is up and running long enough to have enough data in the system that you can analyse."

Ken Cunneen, IT leader for technology and integrated supply chain systems in Honeywell Aerospace's Aviation Aftermarket Services division, says if a company implements a financial system that only gives results once a quarter, the audit team should wait at least three to six months until the system has generated enough data before performing a PIA.

On the other hand, the sooner you start the audit, the fresher the data and the easier it is to cull lessons learned. Which is why PIAs should not be a onetime event, says PwC's Christian.

Sun Life Financial's project office begins PIAs within a month of project completion to do a post-mortem on the implementation process and to get feedback from users to make enhancements to the system. "When we have benefits that are realised over a period of years after the project is implemented, we do follow-up reviews," says Sun Life Financial's Esposito.

Higgins plans to take that iterative approach with Michael Baker's ERP implementation, which is scheduled to be completed by the end of this year. "We will begin to audit the effectiveness of the new system within a couple of months after we finish implementing it," says Higgins. "Then we'll probably start to look at the ROI six to eight months after we go live so that we have enough data to perform a good analysis."

Maintain Meticulous Documentation

Another key to successful PIAs is good documentation, which takes many forms. It includes the business case that outlines the system's expected cost, benefits and ROI; the project's time line, including key metrics and milestones; a breakdown of the system's technical requirements; a record of the security and financial controls that have been put inside the system; and a compendium of all the changes that have been made to the system or project plan. But getting good documentation is one of the biggest challenges CIOs and IT departments face in conducting PIAs, particularly if they're doing one for the first time.

Sharon Thompson, director of IT audit for the AARP (American Association of Retired People), says that if an auditor discovers that the implementation team does not have a thorough business case, a detailed project time line or meticulous records of changes and security controls, the auditor should note that discovery in the audit report and include a recommendation for the implementation team to keep better records.

To make life easier for himself and his post-implementation auditors, Higgins uses a Web-based collaboration tool called MPOWR - developed by members of his IT department to do version control and to archive changes made to the project's scope, templates, time line and budget. When it comes time to do a PIA, Higgins sends the auditors a link to the MPOWR site with a user name and password.

Sun Life Financial's IT project office also developed a project portfolio management system in-house that the project manager uses to report on a monthly basis changes to a project's cost, milestones and schedule.

Page Break

Act on Lessons Learned

If you don't use PIAs to identify ways to improve project management, you're not realising their full value. Sun Life Financial's Esposito says project team members who served on the audit and implementation teams hold a meeting during the PIA process to discuss what went well and what went wrong both during the implementation and the audit, and to identify areas for improvement.

To ensure that people take these lessons to heart, Sun Life Financial posts them on the corporate intranet and submits to senior management a summary of findings from PIAs of all projects done during the course of the year. Esposito also makes changes to project management and implementation processes if a project's PIA indicates that changes are warranted.

Sun Life Financial annually reviews the audits of different projects to identify common areas that need improvement. Gartner's Gomolski says it's important to do this type of project-to-project comparison. "The real value," she says, "is in seeing some of the patterns. For example, 'We screw up every time we do an integration project. What does that mean?' Or, 'Every time we do a project with this business unit, we suffer from amazing scope creep because they don't really know what they want, so maybe we have to work with them differently.'"

SIDEBAR: Sustain the Momentum

Now that you're armed with convincing arguments on the importance of PIAs and success factors for pulling them off, making them a sustained practice in your organisation should look less daunting.

"If you have the project's objectives, approach, how it's going to be organised, how the business areas are involved in the project, the expected costs, and the anticipated hard and soft benefits well-defined up front, and you manage change along the way, then when you come in to do the post-project audit, there are no surprises," says Esposito.

Not only do PIAs go smoother when you've done all the hard work on the front end, but you increase your odds of actualising the system's value. "By getting a better understanding of project costs up front," says Higgins, "you'll be more likely to hit the ROI you're trying to meet."

Eight Steps for a Successful PIA

• Long before the post-implementation audit (PIA) begins, write up a clear business case that delineates and breaks down the cost of the project, the soft and hard benefits, the expected ROI, and when the ROI will be fully achieved.
• Keep scrupulous records of all the changes that are made to the system and to the project plan during the implementation. A Web-based version control or collaboration tool can make this easier.
• Identify the members of the audit team.
• Schedule a kick-off PIA meeting to identify all the logistics that need to take place to perform the audit.
• Review the business case, the initial scope of work and all the change orders. Around the same time, schedule interviews with users or send out satisfaction surveys. Also survey implementation team members to find out how well they think the various project objectives were met. Test the system and its controls. Examine the quality of the data. Take a sample of the transactions to make sure they're being processed the way they should. Engage the finance department to conduct its review of the system's pay-off.
• Compare audit findings with the business case to see if you hit your target.
• Gather lessons learned. Identify what went well, what didn't and why. Determine how you can address what didn't go well in future implementations.
• Perform follow-up reviews, if necessary.

Page Break

Doing It Right Questions to ask during an audit

About the functionality of the system:

• How does the system work? How are transactions processed? Are transactions processed in a timely, secure and accurate manner?
• How well is the system working?
• Are people using the system? Is it easy to use?
• Has the system automated all previously manual processes?
• Does the system need any enhancements or additional modules?

About the security of the system:

• How timely and accurate is the data in the system?
• Is the system secure? How is access to the system defined? Who has access to it and what level of access do they possess?
• Does the system have audit trails that will raise a red flag if someone goes into the system who shouldn't and starts manipulating data?

About the implementation process:

• What went well? What didn't go well and why? Was it a technical problem? An organisational problem? A change in the marketplace or business climate? A snafu with a vendor?
• How can these things be prevented in future implementations?
• Was there a project management methodology that was followed? What was it? Was it adequate?
• Was the project structured and scheduled to facilitate completing it on time and on budget?
• Did the implementation team possess the skills it needed to get the project done on time and on budget?
• Was the project on time and on budget? Were there components of the project that went over budget or were delayed

About the post-implementation audit:

• What training is being offered to users?
• How is the help desk structured to deal with a potential influx of calls about the new system and new processes?
• Is there a process for logging problems with the system if people do have problems? What is it?
• What enhancements are or have been made to the system? Is there a log listing all the changes and enhancements, who made them and whether they had approval?

About the results:

• Did we achieve all the soft benefits that were laid out in the business case?
• Did we achieve all the hard benefits?
• Did we achieve the cost savings and ROI?

Page Break

SIDEBAR: How to Select the Right Project for Your First Post-Implementation Audit

Small is better, especially in the beginning

If your first post-implementation audit (PIA) goes smoothly, you'll be more inclined to make them a standard step in your project life cycle. The trick to ensuring that your first audit goes off without a hitch is to pick the right project, says Barbara Gomolski, a research director for Gartner (US).

Auditing projects that are "a political minefield where results are inconclusive", won't inspire anyone to perform PIAs regularly, she says. Instead, she tells her clients to "go for the low-hanging fruit". By low-hanging fruit, Gomolski means projects that have a fairly specific expected financial return and that are sponsored by a cooperative business unit. If the business unit members that cosponsored the project are easy to work with, they'll be more likely to give the audit team the information and feedback it needs during the audit phase of the project.

She also suggests that CIOs audit projects that involve software or technologies that are going to be around for a while. If the audit reveals the system did not yield the expected benefits, the CIO has time to figure out why that is and what to do about it.

The first project Bruce Higgins, CIO of engineering and construction company Michael Baker Corporation, ever audited in his career followed similar guidelines. In January 2002, Higgins installed an automated, Web-based system for procuring IT hardware and software from ASP ePlus. Before, procurement had been a costly, time-consuming, paper-based, people-intensive process. Higgins hoped the new, Web-based system would be a clear improvement over the old process and would yield a system that was easy to use and would lead to cost-savings, improved quality of service and a faster procurement cycle. The project team listed those expectations in a business case for the project. They also scoped out all the work involved with getting the new system up and running. As the project's scope changed, the implementation team wrote change orders with the executive sponsor's authorisation, and brought the changes to the users' attention so they wouldn't be surprised by the system or its actual cost once the implementation was complete.

While the audit had a few hiccups, it was an overall success. It went well because Higgins chose to audit a project that was small in scope, had clear benefits that were defined up front and recorded in the business case, and because the implementation team documented all changes in scope. As a result, it was relatively easy for the audit team to compare the new systems' expected benefits and cost savings with the actual benefits and ROI. Because the process went so smoothly and because his company learned from the audit how to better manage projects, Higgins plans to conduct PIAs on all strategic IT projects, including an ERP system in January 2004.

"You really get an understanding of how to run your IT projects better," says Higgins in regard to PIAs. "You can throw stones at the one you've just implemented, praise yourself for the things you did right, criticise yourself for things you did wrong, understand why they went wrong. and how you can make them better on other projects throughout your enterprise."

Higgins says PIAs are also useful in determining if you've actualised your financial return from the system: "We figured out a real way to establish whether or not there was an ROI on the project by collecting and analysing data. Now we have a process that we can utilise on other IT projects throughout our enterprise that will allow us to evaluate the savings and efficiencies we created in streamlining processes by implementing new pieces of software."