How to Use Monte Carlo Simulations
Once you have a repository of project risks, you can get statistical. The most commonly used tool for this is the Monte Carlo simulation. This technique was developed in the 1940s for the Manhattan Project. It's used today for everything from deciding where to dig for oil to optimising the process of compacting trash at a waste treatment facility. It's a deceptively simple but powerful tool for risk analysis. All Monte Carlo really does is roll the dice (hence the name).
Here's the theory: Roll a die 100 times, and record the results. Each face will come up approximately one-sixth of the time - but not exactly. That's because of randomness. Roll the die 1000 times, and the distribution becomes closer to one-sixth. Roll it a million times, and it gets much closer still.
The die represents risks - albeit evenly distributed, predictable risks - where each side has about a one-sixth probability of occurrence or a five-sixths probability of not occurring. What if each die were a project risk and each side represented a possible outcome of that risk? Say one die was for the risk of project delays due to staff turnover. One side would represent the possibility that the project is six months late because of 20 per cent turnover. Another side could represent a two-year delay due to 80 per cent turnover. The die could also be unevenly weighted so that certain outcomes are more or less likely. There would, of course, be dice for other risks - sloppy development, budget cuts or any other factor unearthed during preliminary research.
Monte Carlo simulators "roll" all those risks together and record the combined outcomes. The more you roll the dice, the more exact they make the distribution of possible outcomes. What you end up with resembles an anthill (see "The Shape of Risk", page 59), where the highest point on the curve is the most likely outcome and the lowest ends are possible but less likely.
Once you determine a project's risk profile, you can build in extra resources (like money and time) to mitigate the risks on the highest points of the curve. If the distribution says there's a 50 per cent probability the project will run six months late, you might decide to build three extra months into the schedule to mitigate that risk.
Monte Carlo simulators also let you run "sensitivity analyses" - rolling only one die while keeping the others fixed on a particular outcome to see what happens when just one risk changes. A health-care company (that requested anonymity) using a Monte Carlo simulator from Glomark ran a sensitivity analysis for a pending software project. Each die was rolled, one at a time, 500 times while the other dice were kept fixed on their most likely outcomes. The exercise showed that three of the nine risks represented 87 per cent of the potential impact on the project - allowing the company to focus its energy there.
You can (and should) repeat Monte Carlo simulations for all the projects in your portfolio, ranking them from riskiest to safest. This will help you generate an "efficient frontier" - a line that shows the combination of projects that provide the highest benefit at a predetermined level of risk - something like the line across Montserrat. An efficient frontier helps you avoid unnecessary risk. It will help stop you from choosing one project portfolio that has the same risk but lower benefits than another.
Admittedly, this description glosses over some of Monte Carlo's dirty work. Someone has to determine which dots to put on the dice and how to weight the individual dots. That's your job. Canvass your experts, mine historical data, and do whatever else you can to come up with possible outcomes from each risk, and then estimate the probability of that result occurring. In other words, the risks themselves are a range of outcomes contributing to a further range of possible outcomes for any given project, or even combinations of projects.
The best way to create the risk dice is with a triangle distribution. Determine three data points: the best case outcome, the worst case and the most likely case. Assume the best and worst cases have low probabilities and the most likely case is somewhere in between.
In the staff turnover example, the worst case might be a two-year delay due to 80 per cent turnover. The best case may be no delay due to no turnover. The most likely - based on experience and research - might be the previously stated six-month delay from 20 per cent turnover. Chart this on a probability distribution grid, and you get a triangle.
Take that triangle and others you create for all project risks, run Monte Carlo simulations, and you'll come up with the smooth anthill curve that shows overall risks to your project.
Vitro, a $US2.6 billion glass company in Monterrey, Mexico, has done this on many IT projects (it's now required for projects valued at more than $US20,000). "No one wanted to measure at first," says Gustavo Benitez, manager of Vitro's IT supply chain. "Because measuring makes you accountable. We're not that deep into it; we only use best case, worst case and most likely, and already it helps. It helps you see different scenarios."
Certain risk metrics are predetermined. DeMarco and Lister's five core risks to software projects have been given probability distributions based on historical data. If you're still worried about assigning a meaningful number to risks, Lister says relax and just guess. "Guess a number just to get going," he says. "Even that will be better than how IT approaches risk today."
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.