Reader ROI
- Read about the real-time test of an incident response team.
- Learn why incident response planning is worth the pain.
- Find out how planning ahead can save time, money and your reputation.
See "The Truth About Cyberterrorism"to learn how to respond to cyberterrorism.
Think you can't afford to create an incident response plan? Think again. Here's a budget-conscious guide to getting started.
For Barry L Woolsey, CIO of Fleet Credit Card Services, the first public test of his plan for dealing with security incidents began not with a hacker, a cyberterrorist or a disgruntled employee. It started when a curious customer logged on to Mycard.Fleet.com to make a credit card payment - and inadvertently discovered a security hole that would earn headlines on MSNBC.com and force Fleet to shut down its site for five hours.
Logging in on a Friday afternoon, Jonathan Bryce, a 20-year-old online operations manager, noticed that the Web page where he accessed his credit card account had a long identification number in the URL. Wondering how Fleet kept track of transaction history, he entered a random number. To his shock, he pulled up someone else's transaction.
"The hole allowed you to see people's personal information,"says Bryce, who works for Rackspace Managed Hosting in San Antonio. "Mainly it was nonsensitive information, but there were several cases where it was personal information like Social Security numbers, account numbers and addresses."
Alarmed that his account information could be compromised, Bryce called the customer service line at Fleet. After being transferred several times, he was told that someone would call him back on Monday.
That wasn't good enough. Bryce called the media, and hours later both MSNBC.com and The Boston Globe online had picked up the story.
Behind the scenes at Fleet Credit Card Services, the FleetBoston Financial subsidiary north of Philadelphia, Bryce's phone calls and the ensuing media coverage set in motion an incident response plan long in the making. It took more than five hours from the time Bryce called Fleet to the time Woolsey found out. So while Woolsey is generally happy with the way the plan unfolded, he acknowledges that it could have worked better - a whole lot better.
At least Fleet had a plan. Throughout most of the business world, incident response planning is one of those best practices that rarely gets done. That's because of its vague underlying assumption: something could go wrong.
Obviously, prevention is key. But in a world where security is nothing if not fallible, knowing how to respond to a security incident - be it a computer worm, mistake, hacker or the mere suspicion of a problem - can save a company time, money and even its reputation. Indeed, a better response from the customer service representatives at Fleet could have kept the problem from making headlines in the first place.
Nevertheless, the sad fact is that Fleet handled the situation better than many companies could have. Outside of the heavily regulated health-care and financial services industries, experts say, few companies are prepared to deal with isolated security incidents, let alone calculated attacks from organised crime or even, perhaps, cyberterrorists (for more on this, read "The Truth About Cyberterrorism", page 68).
"There's only now the beginning of awareness that this thing is needed,"says Jay Ehrenreich, senior manager in the cybercrime prevention and response group at PricewaterhouseCoopers in New York City. "Companies do come to us to help them develop incident response plans, but lots of times it's only after they've been burned."
And getting burned is expensive. Companies have spent billions of dollars recovering from malicious worms such as Nimda and Code Red. And in a 2001 study by the Computer Security Institute and the FBI, respondents who could put a dollar amount on the cost of a security breach averaged more than $US2 million in financial losses.
Not everyone has to learn the hard way, though. Here are the steps to take to jump-start the process with less time and effort than you might expect.
1 PULL TOGETHER A TEAM.
As long as upper management is serious about security, the first step of incident response planning - pulling together an incident response team - costs next to nothing. "Many companies envision this fully dedicated, highly paid SWAT team doing nothing, waiting for an emergency,"Ehrenreich says. "It's really not that way. It's almost like a volunteer fire department, where people have other duties, but when there's an emergency people respond to it."
The night Fleet learned of its security breach, Woolsey, COO Susan Gleason (who is no longer with the company), CEO Patrick J Coll, and representatives from the legal team, communications department and FleetBoston gathered on a pre-established bridge telephone line to discuss how to minimise the damage to Fleet's customers and its reputation. Meanwhile, the IS team worked to analyse the security breach, find out what information had been accessed and fix the security hole. Based on the IS team's findings, the fraud department then called customers whose personal information had been compromised.
That's pretty typical of the groups that need to come together after a security breach, although some organisations would have added human resources to the mix. The list of team members can get long, but not everyone needs to be involved with every situation. Also, some people are responsible for fixing the problem; others just need to know what's happening. The incident response group will be closely aligned with or perhaps even the same as the business continuity or disaster recovery teams.
Once team members know their role, a project manager should create a list with multiple ways to contact team members 24/7. The list should also include contact information for security vendors, Web hosting companies and other relevant technology providers, and it should be available in hard copy at every business location and at people's homes. The need for that last recommendation was painfully illustrated on September 11.
2 COORDINATE YOUR EFFORTS.
The plan can't stop with that group. As the Fleet example illustrates, everyone across the organisation - right down to the newest person at the call centre - needs to know how to react to a potential security breach. That's why there needs to be a centralised way to report, respond to and track incidents. Customer service representatives need to know who to call about a problem; security employees need to know when to call the CIO; the CIO needs to know when to call the CEO; and someone needs to track what's going on organisationwide so that different business units can prepare for attacks. "A good bit of incident response is coordination,"says David Nelson, deputy CIO at NASA, who's in charge of the agency's IT security. "You want to be sure that the right people have the right information at the right time."
A security incident at any of NASA's centres is first reported to an onsite IT security manager. The incident is put into one of seven categories ranging from a legitimate user misusing the system to an access problem in which a hacker has obtained the password of a systems administrator. Anything deemed serious is reported immediately to the centralised NASA Incident Response Centre (NASIRC), which shares this information with Nelson. If the incident could be criminal in nature, NASA's inspector general's office also gets involved.
During large-scale attacks such as the Code Red worm, NASIRC sets up a conference call with IT security managers from all the centres. "We would in real time go over what's happening, what's the damage, what's being done, and do we have to take any extraordinary measures to protect ourselves,"Nelson says. "If we didn't have [this system] in place, we'd be like sitting ducks without any means of finding out that the hunters are shooting at us. Our experience is that any substantial attack hits a lot of computer systems or IP addresses or places almost simultaneously, and so early awareness can help us take appropriate measures before we're all dead. That first gun goes off, and all the ducks fly out - maybe one got killed, and we're sorry about that, but the rest of them are all alive."
3 GRANT AUTHORITY.
One of the most political parts of incident response planning - but one that can save precious time if an attack is successful - is deciding ahead of time who's in charge of incident response and which people could pull the plug on the Web site or network if need be. But the fact that some people high on the corporate food chain have to relinquish control causes friction, says Rebecca Bace, a National Security Agency alum who wrote Intrusion Detection. Unfortunately, there simply might not be time to contact everyone.
"The velocity of [an intrusion] proceeds in seconds or minutes, rarely hours. If management wants people to be able to respond effectively to quick-moving attacks, they've got to empower them to shut certain portions of the system down,"says Bace, who is now a faculty member for The Institute for Applied Network Security in Massachusetts.
The Georgia Student Finance Commission learned that the hard way - but in slow motion. Bill Spernow, the commission's new chief information security officer (CISO), was hired after a public security snafu in which personal information about HOPE scholarship recipients was disclosed on the Internet. The organisation was painfully slow in responding to the breach, caused by a technical error, because there was no one in-house who could take control of the situation and shut down the Web site. Now Spernow is in control.
"You can do this [incident response planning] on the cheap as long as senior management has come to the decision that they're going to delegate this to one person who can make a decision,"says Spernow, a former Gartner analyst who says he took the job because he wanted to apply his advice in the real world. "That will always be the biggest bottleneck whether you're a Mom and Pop or a Fortune 100. It ends up being a very intense political discussion that brings in lots of fears and control issues. And once you've done that, you've handled 60 per cent of the problem."
4 READY YOUR IT RESOURCES AHEAD OF TIME.
No doubt about it, the technical aspect of intrusion response is the most complicated part. The IS team or an outsourced monitoring service has to be able to identify problems when they do occur - by examining logs for unusual behaviour, looking for vulnerabilities, watching for faulty configurations and monitoring intrusion detection systems, which can generate a tremendous number of false positives. "You have to know when someone's attacking you because if you don't know, you can't do anything. And that's the hardest part,"says Michael Young, CISO and vice principal of State Street Global Advisors in Boston.
Once a problem is identified, IT staffers need to be able to look at system logs and analyse exactly what happened. Finding and preserving evidence is the key to fixing a problem and keeping it from happening again. Also, knowing exactly which files were compromised will help a company ensure data integrity and figure out which customers, employees or business partners may be affected (to learn more about computer forensics, read "IT Autopsy", CIO April, 2001).
Whether this can be done in-house or needs to be outsourced depends on the size of the company, how attractive a target it is to hackers and how sensitive its information is. As a general rule, the more crucial and extensive a company's information assets, the more sense it makes for executives to keep security operations in-house.
Fortunately, though, the more a company works to prevent security problems in the first place, the cheaper it will be to deal with them once they do arise. "An ounce of prevention is worth a pound of cure, and that absolutely applies in the cyberintrusion space,"says Bruce Moulton, former CISO at Fidelity Investments and a cofounder of the Financial Services Information Sharing and Analysis Centre.
5 DECIDE WHEN TO INVOLVE LAW ENFORCEMENT.
Reluctant to call the cops and risk having customers and stockholders find out about a problem? That's normal. But if a situation gets bad enough - a key competitor steals intellectual property, a hacker publishes customer records or a former employee takes down the Web site - you might change your mind. And in some cases, publicity will ensue anyway. Executives can save time and heartache by discussing ahead of time what situations might cause them to call in law enforcement. For most companies, these situations involve serious financial loss that law enforcement may help recover in damages, or a high-profile case that's already publicly known.
At the University of Washington Academic Medical Centre in Seattle, which suffered a security breach that was reported by The Washington Post and other news outlets, CIO Tom Martin initially decided not to call the FBI because he didn't think the hacker had accessed any patient data. But as soon as he discovered otherwise, he says, he changed his mind.
"There is a stigma to being a victim because it implies some level of incompetence, but really it's an inevitability. As we become more Internet-based, it's only a matter of time before you have problems,"Martin says. "To underreport is a disservice to the industry. It's building in an infrastructure of illegal activity, and we just decided we won't tolerate illegal activity or accept it as a given."
The threat of organised cyberattacks or even cyberterrorism has made this even more of a concern, many experts say, because law enforcement needs to be able to watch for trends or attacks that spread across companies and industries. For research purposes, Carnegie Mellon's CERT Coordination Centre tracks security incidents and asks that companies report even unsuccessful attacks (for more information, visit www.cert.org or www.auscert.org.au).
Increasingly, companies are also being encouraged to contact law enforcement outside of a particular security event. By talking with law enforcement ahead of time, a CIO or CISO is better equipped to evaluate the impact of an investigation on business operations, determine whether law enforcement might be able to help and find out how sensitive information is handled by the authorities.
"When we get the call [after a security incident] has already happened, it's difficult for us to help,"says Weaver, head of the Secret Service's New York City Electronic Crimes Task Force, which was made a national model for information sharing between law enforcement and business by the USA Patriot Act, passed by Congress shortly after September 11. "Call who best serves you. It may not always be federal - you may be best served by a local police department, a state police department, an attorney general. If you don't have [that relationship] and an incident does arrive, it's like driving your car down the street and trying to change the tyre while you're doing it. You don't want to dial [000]. You want to know who you're bringing in."Once in place, incident response plans are morphing, living documents. As Fleet Credit Card Services learned, there are things to be tested and tweaked and learned from - processes that become better and easier to justify to management after they've been used.
So the next time that organisation has a security incident - be it a malicious worm or something more ominous - the company will be that much better prepared. CIO Woolsey is working on making sure customer representatives know how to handle security calls from alarmed customers, and since the security hole was in a vendor's application, he's also instituting tighter controls around third-party software. He's heartened by the fact that to the best of his knowledge, no customers have closed their account because of the incident.
"I guess the one lesson I would walk away with is never assume that anything is perfect,"Woolsey says. "What you need to do is understand that you can't plan for everything, so you need to have a contingency that's broad enough and open enough that it can in essence deal with everything."
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.