Playing with Fire

IT is late to embrace risk analysis, but without it, project portfolio management is nothing more than a fad.

Reader ROI

• Realise the importance of quantifying risk
• Understand how to use statistical simulations to map risks and probabilities
• Deliver a risk analysis process

Montserrat is one of the many islands sprinkled among the Caribbean West Indies. In just 10,000 hectares, it boasts mountains, rain forests, beaches and groves of bananas, mangoes and coconuts. The air temperature rarely dips below 26 degrees and neither does the water.

In short, Montserrat is paradise. Or it would be but for the Soufriere Hills volcano, which erupted for the first time in July 1995 and hasn't stopped since.

Soufriere Hills has rendered nearly two-thirds of the island - an area now called the Exclusion Zone - uninhabitable. Since 1995, the island's population has fallen from 11,000 to 4000. The volcano has buried Plymouth, the former capital. It killed 20 people in one violent belch in 1997. It has suffocated the economy, once driven by tourism and rock stars like Sting, the Stones and Paul McCartney, who partied and recorded music there at Air Studios, the recording facility once owned by the Beatles' former producer George Martin but now buried by the volcano.

This dichotomy - Eden on one side of the island, the fires of Hell on the other - makes Montserrat a perfect laboratory for risk analysis. Just as much of Montserrat is buried in ash, it's also buried in probabilities. Scientists know, for example, that there's only a 3 per cent chance that Soufriere Hills will stop erupting in the next six months. They also know there's a 10 per cent chance of injury from the volcano at the border of the Exclusion Zone, and they can draw an imaginary line across the island where the threat from the volcano equals the threat from hurricanes and earthquakes.

"Thirty years ago, you needed the biggest computer in the world to do the statistical risk analysis," says Willy Aspinall, who helped develop these figures in the shadow of Soufriere Hills. "Now all you need is a laptop and a spreadsheet." He says the risk calculations get better and more textured all the time. He uses Monte Carlo risk analysis simulation software and spreadsheets to quantify the risk levels that help decision-makers minimise the volcano's threat to people's lives.

If this type of risk analysis is good enough for Aspinall, it ought to be good enough for CIOs, especially now that they're working in an economic environment looming as ominously over their businesses as Soufriere Hills looms over Montserrat. For the most part, though, CIOs have not adopted statistical analysis tools to analyse and mitigate risk for software project management.

This is why they should.

Page Break

Risky Business

Experts will tell you that statistical risk analysis is as essential to real portfolio management as a processor is to a computer. Without it, portfolio management is simply a way to organise the view of projects that will almost certainly fail. CIOs who are serious about portfolio management need to be serious about statistical risk management. (For more on portfolio management, see "Portfolio Management: How to Do It Right", CIO June.)

"If you don't succeed with risk management, you won't succeed with project portfolio management," says Raytheon CIO Rebecca Rhoads, who credits risk management with lowering her project failure rate and helping Raytheon IT achieve its cost-performance targets. Rhoads is ahead of the curve, but despite her engineering background, she has yet to apply the kind of sophisticated statistical analysis that Aspinall uses for his volcano.

Robert Sanchez, senior vice president and CIO of Ryder, credits risk analysis with bringing order to his company's decision-making process for projects. He would welcome statistical analysis, but he's not there yet. "Have we really embraced it completely and understood it in all of its detail?" Sanchez asks rhetorically. "No, we haven't. But we will."

CIOs should become familiar with two statistical tools. They are the colourfully named workhorses of risk analysis: Monte Carlo simulation and decision tree analysis. Probabilities figure heavily into both, which means that risk has to be quantified. CIOs must draw their own line between the Exclusion Zone, where it's too risky to venture, and the beaches, rain forests and coconut groves, where the living is easy and the threats are manageable.

The Trap of Common Sense

Even a simple task like choosing to drive to work requires a risk assessment, although not a computational one; you can do shorthand probability in your head. Though the cost of being wrong is high, the risk is relatively low (a 5 per cent probability of being seriously hurt in a car accident) and easily mitigated by wearing a seat belt.

This sort of informal risk analysis can sometimes be useful. Steve Snodgrass, CIO of construction materials supplier Granite Rock, has the misfortune of managing IT for a company that literally straddles the San Andreas Fault. Snodgrass doesn't need statistics to tell him that it would be a bad idea to do nothing to mitigate the possibility that a quake will take out his critical applications. So he outsources his applications' backup far from the fault line.

However, CIOs often use this kind of common sense reasoning as a way to avoid doing real risk analysis, say Tom DeMarco and Timothy Lister, authors of Waltzing with Bears: Managing Risk on Software Projects, a primer on statistical risk analysis for IT. "It's been very frustrating to see a best practice like statistical analysis shunned in IT," says Lister. "It seems there's this enormously strong cultural pull in IT to avoid looking at the downside."

In lieu of choosing projects based on acceptable risk, Ryder's Sanchez says, IT often uses what he calls the moral argument, in which the greatest risk lies in not doing the project. Therefore, the risk is mitigated by doing the project. This reasoning was particularly valid during the boom years when there was a palpable fear of getting left behind technologically. But it was never called risk analysis. "I came into IT and was never really comfortable with the moral argument," says Sanchez, whose background is in engineering and finance. "I was looking at it thinking: We analyse the risk of building a new office, but we don't on an ERP system that costs the same amount."

Page Break

How to Create a Risk Analysis Process

As the director of foreign exchange at Merck, Art Misyan uses statistical risk analysis for evaluating the impact of foreign currency volatility. Like Sanchez, he's puzzled by IT's laissez-faire attitude towards risk analysis. "Risk gives you the ability to look at a whole range of outcomes, but IT looks at only two possible outcomes," he says. "Either you hit deadlines or budgets, or you don't."

IT needs to think in probabilities, Misyan says, not ones and zeros. The best way to start is for the CIO to formalise the risk process. "First you have to set up a process to determine and track risks," he says. The good news is that much of the risk process is built into project management methodologies CIOs have been adopting anyway, so it should be familiar. Here are the basics for developing a risk analysis process.

Gather experts to determine project risks. These brainstorming sessions should be free and creative. "You want the pessimist in the group, the dark cloud," says Anne Rogers, director of information safeguards at Waste Management, who teaches risk analysis. "You want the person that will ask, What if a truck ran into the building?"

When you don't ask the off-the-wall question, you run the risk of smacking into it. "Motorola gambled on developing Iridium satellite phones and charging $US7 a minute," recalls DeMarco. "No one seemed to wonder what would happen if [mobile] phones came along offering similar service for 10 cents a minute and free nights and weekends."

Assign researchers to uncover known risks. "We came up with 20 or 30 risks we knew we'd face by research," says Sandy Lazar, director of key systems for the District of Columbia, who is overseeing a five-year, $US71.5 million administrative systems modernisation program (see "Get a Grip on Risk", page 62). "If you read up, you realise ERP has failed over and over for the same reasons for 15 years now." In fact, there are five typical risks to software projects that every CIO should include in a risk analysis .

Divide risks into two categories - local and global. The risk of staff turnover during a project is a local risk. War is a global risk. Often, those new to risk analysis focus only on the local risks, but they need to consider the global risks and their impact.

Create a template for each risk. The template should include a unique risk number, a risk owner, potential costs (in dollars and other terms), a probability of occurrence (a low-medium-high scale will do at this point), any potential red flags or signs that the risk is materialising, mitigation strategies and a post mortem for noting if the risk factor actually happened. (A good example of such a template can be found in Waltzing with Bears. Go to http://www.cio.com/archive/070103/risk_form.html for a sample Risk Control Form.)

One important footnote for developing this process: Value consistency over accuracy. If you do things in a consistent manner and the numbers are off, at least they'll be off in a consistent - and therefore fixable - way. "The process," says Raytheon's Rhoads, "is so much more important than the maths rigour. Mature, consistent processes - you need that first."

Page Break

How to Use Monte Carlo Simulations

Once you have a repository of project risks, you can get statistical. The most commonly used tool for this is the Monte Carlo simulation. This technique was developed in the 1940s for the Manhattan Project. It's used today for everything from deciding where to dig for oil to optimising the process of compacting trash at a waste treatment facility. It's a deceptively simple but powerful tool for risk analysis. All Monte Carlo really does is roll the dice (hence the name).

Here's the theory: Roll a die 100 times, and record the results. Each face will come up approximately one-sixth of the time - but not exactly. That's because of randomness. Roll the die 1000 times, and the distribution becomes closer to one-sixth. Roll it a million times, and it gets much closer still.

The die represents risks - albeit evenly distributed, predictable risks - where each side has about a one-sixth probability of occurrence or a five-sixths probability of not occurring. What if each die were a project risk and each side represented a possible outcome of that risk? Say one die was for the risk of project delays due to staff turnover. One side would represent the possibility that the project is six months late because of 20 per cent turnover. Another side could represent a two-year delay due to 80 per cent turnover. The die could also be unevenly weighted so that certain outcomes are more or less likely. There would, of course, be dice for other risks - sloppy development, budget cuts or any other factor unearthed during preliminary research.

Monte Carlo simulators "roll" all those risks together and record the combined outcomes. The more you roll the dice, the more exact they make the distribution of possible outcomes. What you end up with resembles an anthill (see "The Shape of Risk", page 59), where the highest point on the curve is the most likely outcome and the lowest ends are possible but less likely.

Once you determine a project's risk profile, you can build in extra resources (like money and time) to mitigate the risks on the highest points of the curve. If the distribution says there's a 50 per cent probability the project will run six months late, you might decide to build three extra months into the schedule to mitigate that risk.

Monte Carlo simulators also let you run "sensitivity analyses" - rolling only one die while keeping the others fixed on a particular outcome to see what happens when just one risk changes. A health-care company (that requested anonymity) using a Monte Carlo simulator from Glomark ran a sensitivity analysis for a pending software project. Each die was rolled, one at a time, 500 times while the other dice were kept fixed on their most likely outcomes. The exercise showed that three of the nine risks represented 87 per cent of the potential impact on the project - allowing the company to focus its energy there.

You can (and should) repeat Monte Carlo simulations for all the projects in your portfolio, ranking them from riskiest to safest. This will help you generate an "efficient frontier" - a line that shows the combination of projects that provide the highest benefit at a predetermined level of risk - something like the line across Montserrat. An efficient frontier helps you avoid unnecessary risk. It will help stop you from choosing one project portfolio that has the same risk but lower benefits than another.

Admittedly, this description glosses over some of Monte Carlo's dirty work. Someone has to determine which dots to put on the dice and how to weight the individual dots. That's your job. Canvass your experts, mine historical data, and do whatever else you can to come up with possible outcomes from each risk, and then estimate the probability of that result occurring. In other words, the risks themselves are a range of outcomes contributing to a further range of possible outcomes for any given project, or even combinations of projects.

The best way to create the risk dice is with a triangle distribution. Determine three data points: the best case outcome, the worst case and the most likely case. Assume the best and worst cases have low probabilities and the most likely case is somewhere in between.

In the staff turnover example, the worst case might be a two-year delay due to 80 per cent turnover. The best case may be no delay due to no turnover. The most likely - based on experience and research - might be the previously stated six-month delay from 20 per cent turnover. Chart this on a probability distribution grid, and you get a triangle.

Take that triangle and others you create for all project risks, run Monte Carlo simulations, and you'll come up with the smooth anthill curve that shows overall risks to your project.

Vitro, a $US2.6 billion glass company in Monterrey, Mexico, has done this on many IT projects (it's now required for projects valued at more than $US20,000). "No one wanted to measure at first," says Gustavo Benitez, manager of Vitro's IT supply chain. "Because measuring makes you accountable. We're not that deep into it; we only use best case, worst case and most likely, and already it helps. It helps you see different scenarios."

Certain risk metrics are predetermined. DeMarco and Lister's five core risks to software projects have been given probability distributions based on historical data. If you're still worried about assigning a meaningful number to risks, Lister says relax and just guess. "Guess a number just to get going," he says. "Even that will be better than how IT approaches risk today."

Page Break

How to Do Decision Tree Analysis

The other major risk analysis methodology applicable to IT project portfolio management is decision tree analysis. Where Monte Carlo excels at shaping what happens when many risks are in play at once (such as launching an ERP project), a decision tree proves most useful at mapping either-or situations and the sequential risks that follow each decision (for example, either I build a new factory or retrofit an old one). Each choice is a branch with an attached probability, derived the same way all risks are derived - through brainstorming and research. Each branch leads to other branches, which are the risks that result from choosing the original branch.

The key to analysing decision trees is knowing that the probabilities compound. This is why Waste Management's Rogers likes them. Decision trees often show that good risk decisions are counterintuitive, if you follow the branches. "I'll give some outlandish choice, like rewiring your office or building a new one," Rogers says. "Everyone thinks they know which is less risky, but you watch the risks compound over time and guess what, it's not nearly as risky as you think to build a new office in certain situations."

Decision trees are also powerful presentation tools for executives, as long as you don't drop an entire tree on the CEO. They can be unwieldy. They branch out quickly with thousands of potential paths. A CIO must keep them under control. To demonstrate the path of certain decisions, limit the branches to only the most important risks, and simply leave out remote risks.

Final Analysis

Risk analysis takes some getting used to. Anyone not steeped in this world may misinterpret the results of any given analysis. For example, if you say there's a 90 per cent chance it will rain tomorrow, and the day ends up being sunny, your analysis still may have been perfect. There was, after all, a one in 10 chance the sun would shine, and indeed it did.

The hardest part of risk, especially for CIOs, is that it doesn't provide concrete answers. Risk analysis will not tell you which project to do. It will tell you which ones fall into a certain level of risk and payoff. Provide the same IT portfolio and the same risk analysis to Rogers of Waste Management and Lynn Caddell, CIO of Yellow Freight, and they will most likely choose a different set of projects to pursue.

"We're not real risk-averse right now," Rogers notes. "If they're handled right, we're ready to take some risks."

On the other hand, Caddell's current crusade is to ensure timely delivery of projects, and she's not eager to take on anything that might compromise that mission. "Last year we were 96 per cent on time with projects, and I attribute that to risk management," she says.

Ultimately, the CIO must understand his enterprise's culture and know which risks are worth taking. No amount of risk analysis will lift the decision-making burden from the CIO's shoulders or ensure the success of any project. Risk analysis is a tool, not a substitute for leadership. After all, it takes a leader to understand that although there's no such thing as a sure bet, decisions must be made.

"In these matters, the only certainty is that nothing is certain," famously noted Pliny the Elder, who, incidentally, died from ash inhalation in 79 AD, when the volcano Mount Vesuvius erupted.

Page Break

SIDEBAR: INTERACTIVE MONTE CARLO

Monte Carlo simulation is a deceptively simple but powerful tool for risk analysis.To try the tool: go to

www.cio.com/montecarlo

SIDEBAR: The Shape of Risk

Even without any numbers, the basic probability curve can convey plenty of information about the risk it describes. Here's a cheat sheet for deciphering probability curves

The Basic Curve The basic probability curve looks like an anthill. Here, the X axis represents potential outcomes from worst to best, going left to right. The Y axis represents the probability of those outcomes, from lowest to highest, going bottom to top. The highest point on the curve indicates the most likely outcome of the risk. The best case falls at the far right and worst case far left, both with the lowest probabilities of occurrence.

A steeper, narrower curve (the red line) represents more certainty about the outcome, since more potential outcomes fall in a smaller range. A low, broad curve (the blue line) represents less certainty about a risk's potential impact on a project. With this understanding, you can determine the likelihood of potential risk outcomes with a quick look at a distribution chart.

The Optimistic Curve While steepness of the curve indicates certainty, its tilt describes relative outlook. A risk distribution that tilts to the right represents a more optimistic outlook, since the higher probability results are closer to the best possible outcome.

The Pessimistic Curve On the other hand, a curve that leans to the left shows a more pessimistic view of the risk, since there's more probability that the outcome will fall on the worst-case side of the spectrum.

SIDEBAR: The Five Universal Risks to Software Projects

1 Schedule flaws Either an error in the original schedule or an error in the way the project is run can affect its timing.

2 Requirements inflation This happens when what is needed from the project changes during development.

3 Staff turnover When key people leave during a project, it can have a serious impact on continuity and schedule.

4 Specification breakdown Anything less than complete agreement on project specifications can be fatal.

5 Underperformance Substandard work by anyone on the development team will affect project quality.

SOURCE: WALTZING WITH BEARS

SIDEBAR: When to Use Which Tool

Both Monte Carlo and decision tree analyses are powerful tools, but each has its particular strengths. Monte Carlo simulations are good for accounting for multiple risks occurring simultaneously. Decision trees excel at analysing sequential risks compounding over time. Given those frameworks, here's a look at several scenarios and whether you would be better off rolling the dice or climbing the tree.

Decision Tree

• Decision based on monetary value

• Sequential decisions required

• Few variables or low probability variables that are easily calculated

• Analysing two possible decisions against each other

Monte Carlo

• Decision based on criteria other than value, such as a schedule

• Decisions involve one variable

• More than five variables in complex environment

• Analysing an entire portfolio strategy

SOURCE: RISK AND DECISION ANALYSIS IN PROJECTS BY JOHN R SCHUYLER

Page Break

SIDEBAR: Get a Grip on Risk

How one IT department tracks 43 risks

Sandy Lazar, director of key systems for the District of Columbia, hasn't yet applied statistical analysis to his major software project, but he could. He's done all the prep work. His project management-based risk analysis is good, and it even includes some rough quantifications. Lazar has gone far with his risk management, and you'd do well to imitate his process.

Lazar and his team did all the basics. They researched risks from published literature, brainstormed for other risks, catalogued them in a standard format, planned mitigation and so on. They placed risks into three categories: risks to budget, risks to benefit realisation and risks to schedule.

The severity assigned to these risks is low, medium and high. The severity is further defined by a money or time scale. For example, a medium budget risk might equal 21 per cent to 60 per cent overrun on annual project costs. A high schedule risk might equal nine months or more delay.

Each risk is also given a probability of actually happening. This is derived in the research and brainstorming phase. Probabilities are high (51 per cent to 100 per cent likely), medium (16 per cent to 50 per cent) and low (0 per cent to 15 per cent).

Right now, Lazar's project tracks 43 risks. One local risk is "scope and complexity". One global risk is "political climate". Scope and complexity carries the potential to delay the entire project. Understandably, Lazar assigns this a high severity, although the probability of this risk materialising is medium, based on conversations with his team and research. For political climate - the risk of a change in DC's leadership - he assigned a medium severity and a low probability based on his team's knowledge of the current government's popularity and the chances a new administration could pull back on the project.

Each risk has a mitigation target and proposed mitigation, and there is a risk meeting each week to revisit and track risks. "The eye opener has not been the intellectual realisation of what risk is or how you use it, but the actual practice of it makes you realise how it has to be ingrained," says Lazar. "Really doing risk is work. Most IT departments don't want to do more work. They want to hit deadlines."

There are few IT departments that get beyond this, but with the data Lazar has, he easily could start quantifying his risks and running probability simulations. When you get to this point, we suggest you keep going.

SIDEBAR: Bookshelf Essentials

A short reading list for your crash-course in risk analysis

Title: Waltzing with Bears: Managing Risk on Software Projects

Authors: Tom DeMarco and Timothy Lister

The Skinny: A good introduction to statistical risk analysis written accessibly and humorously.

Excerpt: "First we need to take on a bit of organisational folklore that will otherwise get in the way - the notion that initiating a project without slack is the sign of really gutsy management. On the contrary, it's a sign of cowardice."

Title: Decision Making with Insight

Author: Sam Savage

The Skinny: An authoritative textbook with accompanying software on using Excel add-ons for statistical risk analysis. It's practical and helps visualise risk, but be prepared for some statistical rigour.

Excerpt: "When you have estimates of a low, most likely and high value for an uncertainty, it is often reasonable to use a random number generator with what is known as a Triangular Distribution."

Title: Risk and Decision Analysis in Projects

Author: John R Schuyler

The Skinny: Comprehensive overview of several types of methodologies including Monte Carlo simulation and decision tree analysis. Technical but accessible, it's more focused on concept, less focused on practice than Savage's book.

Excerpt: "The triangle distribution is very popular, though I've never seen a system in business or in nature generate values with a triangular shape. Simplicity is the appeal."