When Bad Things Happen to Good Companies

Were there a computer incident Hall of Fame, you could probably imagine strolling the halls and browsing through exhibits of history's most dynamic electronic viruses and worms - the villains whose names have sent shivers down the spine of any security expert equipped with a decent memory: The Morris Worm, Melissa, Nimda, Code Red, LoveLetter, Klez and, of course, the most recent inductee, SQL Slammer. You might also see some of the more notorious service outages, hacker penetrations, denials of service, malicious e-mail and Internet attacks on display. All have caused varying degrees of chaos, and some have even stopped businesses in their tracks, crippling productivity and costing millions of dollars in lost commerce.

And yet all could have been tamed. Had someone the foresight to put an incident response plan in place, those viruses and worms and outages and attacks might not be so infamous today.

Of course, such a place doesn't really exist, but the threat of cyber-attacks does. And it's growing every day, due in part to the widespread use of e-mail and the Internet. According to statistics from Carnegie Mellon's CERT Coordination Centre (CERT/CC), the number of reported cyberincidents has surged from only six in 1988 to a whopping 82,000 in 2002. Despite the rising threat, however, CERT/CC finds that most CIOs and CSOs don't even think about their response to an incident until after they've experienced an intrusion of some sort, says Chad Dougherty, an Internet security analyst at CERT/CC. "That's because most companies feel relatively safe. They believe that the hackers won't target them, specifically," he says.

But they'd be wrong, says Dougherty. The majority of computer incidents are no longer focused on a particular company. "Most attacks now are automated," he says. "They spread with the intent to damage everyone and everything they can."

Clearly, it's time for CIOs and security execs to come to terms with the need for response planning. "For a long time, incident response meant having a loose team of people on call if something went wrong," says Gene Fredriksen, vice president of information security at Raymond James Financial. "Then companies started getting hit regularly, and I think [people] are finally beginning to realise that incident response is not optional."

Not optional, but also not easy. Even a well-prepared CIO knows that an incident response plan can't keep his or her company completely safe from attack - even with the latest tools for intrusion detection. "There's just no such thing as zero risk," says Leslie Macartney, CISO for Reuters. "And you can't always predict the number, nature or severity of the attacks. But incident response plans are necessary because, in short, no matter how much you try, things will occasionally go wrong. Your company is at its greatest exposure in the time between when an incident occurs and when the containment actions are completed - that's when most of the damage occurs."

And it's not just an internal matter, says Macartney. "Customer confidence can be damaged if it appears the company has been remiss in its handling of security events. The company's reputation could be at stake."

But you can't protect everything completely, so you must prioritise, Macartney adds. By creating a specific strategy that states what to prioritise and how to react if an incident does happen - and by making your security organisation capable of detecting, analysing, and responding quickly and knowledgeably to an event - you can limit the damage done and lower the costs of recovery. And then, by knowing who to call and what to do next, you can decrease the amount of time it takes to recover and possibly save you and your staff from additional disasters along the way.

"The organisations that don't know how to respond to incidents are the ones that will really get hurt," says Kevin Connell, director of information security for the shared data centre of the Securities Industry Automation Corporation, which runs the computer systems and communications networks of the New York and American stock exchanges. "And while it's hard to protect against something you can't predict, it's not so hard to react decisively in crisis situations once you have a plan in place and a procedure to follow."

Page Break

Getting Started

When thinking about incident response planning, remember that the best defence is a good offence. But before you do anything, says Ariel Silverstone, CISO at Temple University, it's important to define the nature of a cyberattack. That way, you can decide what constitutes an incident for your company (for guidelines on reporting security incidents see www.csoonline.com/response/index.html). Generally speaking, a computer incident is anything that potentially compromises the confidentiality, integrity or availability of a computer system. Sometimes such incidents can be real - like a service outage. Other times, the incident is merely a perceived attack - like when a file disappears because an employee simply moved it from one server to another without telling anyone.

Drafting the response plan includes four main activities, according to Kenneth van Wyk, coauthor of Incident Response and director of technology for Tekmark Global Service's technology risk management practice. First, pull together a response team that broadly represents the entire organisation - HR, legal, media relations - and build a phone list to make alerting the necessary people more efficient. Then, create an incident reporting form - a checklist of sorts - to help document the incident and track costs along the way. Next, build a flow chart detailing the process that the team should follow during an incident (see chart, page 69). And finally, map out a post-incident review process to ensure continuous improvement with your overall plan. Each part will play an important role in helping you deal with incidents before, during and after they occur.

Go Team

Incident response teams go by different names in different companies: Some call it the IRT; others use the acronym CIRT or CSIRT, for computer security incident response team. Whatever you call it, the group is pretty much your version of a SWAT team, called into action when a computer incident occurs.

Because every incident (and its potential effect on your systems) has its own particular traits and required responses, it's important to first get a grasp of the kind of incident-handling expertise your network staff and others on the team already have, says Walt Foultz, director of IT security for Farmers Insurance Group. "Incident response is not only a security activity," he says. "All sources of qualified and competent assistance must be assessed so you can be sure, collectively, that you have the skills to handle the job."

During the early stages of creating an incident response program, Foultz suggests surveying your potential team members to scope out the depth of their incident response skills and technical knowledge. Find out if anyone has a specialty, such as dealing with network probes or e-mail viruses. Foultz gives his own staff verbal pop quizzes to make sure they know their stuff. "One technique I use is to set up hypothetical situations, and they have to tell me what they'd do," he says. He also makes sure every staff member allocates a percentage of her regular work time to learn about the latest cyberincident trends and security technologies. "We do that with individual training and by disseminating internal research to the team through management and scheduled awareness sessions," he says.

How your team is structured depends on the skills and available resources within your company. Large companies often have response teams staffed with people dedicated solely to handling incidents, while smaller companies often create a team consisting of a core group of people from several IT and business departments who get tapped if something happens.

George Wade, Lucent Technologies' regional security manager for North America, recommends casting a wide net when choosing your incident response team. The ideal team should include members of your IT security team who know the company's networks, applications and systems inside and out. Don't forget to include representatives from other departments in the company. Many CIOs and security execs do not think to include people from media relations on their response teams, Wade says. "But if someone defaces your corporate Web site and reporters suddenly start calling, you'll understand very quickly how important it is to have a company spokesperson informed and involved," he explains.

Page Break

Some companies decide to involve their disaster recovery or business continuity departments in their response teams - the reason is that other voices often prove helpful when things really go wrong and systems need to be shut down completely.

The team also needs a certain degree of flexibility. "Response teams shouldn't be static," Wade says. "They can be added to or subtracted from at any time if you decide that something needs to change."

Once the team is in place, you'll need to create a contact list - a staple of any response plan, says van Wyk. "If you overlook creating one, you do so at your peril," he says. It's essentially a phone tree, including emergency phone, pager and e-mail information for members of your incident response team. The list should also include contact information for outside authorities, such as state and federal police, AusCERT and any third-party provider that your company may rely on for backup assistance. Contacting the authorities won't be necessary for every incident, but it's good to have the information at your fingertips.

For continuity purposes, list contacts according to job function, authority and skill set rather than by name. That way, if someone leaves the company, you won't have to rework the entire list. It also means that there's a clear reporting structure in place: When an incident occurs at 3am, for instance, and the system administrator sleeps through his pager alarm, the team member who discovers the incident can quickly alert the next person in the chain of command.

Go with the Flow

Once your team is in place, you should create a diagram that spells out, step by step, what each part of the security organisation needs to do when a breach occurs. And while the incident process needs to be flexible in order to handle various kinds of attacks, Silverstone says, you won't want to leave any of the steps in the diagram to interpretation. "Be precise. Everyone should know who to call and what to do in every type of situation," Silverstone says. "If you leave it open-ended and someone makes the wrong decision, you'll leave your organisation open to liability."

Once you determine that you have a genuine incident on your hands, you and your designated team members can formulate a response strategy. Is the incident major or minor? Does it threaten vital business functions? Do you want to contain the incident and maintain business continuity, or do you want to allow the incident to unfold so that you can gather forensic evidence for an investigation further down the road?

Should you contact outside agencies yet? Is it necessary to communicate with the general employee population? The answers to such questions will help the process move along more quickly and predictably, saving precious time and money, minimising damage and maintaining business continuity for your company.

Consider making a team member the designated note-taker so that when a crisis hits, there's no confusion about who's capturing all the important information.

Page Break

It Ain't Over Till It's Over

Finally, after every incident, CIOs or their security execs need to lead their incident response teams in a post-mortem review process that examines how well the incident team dealt with the attack. Did team members follow the response diagram? Did staff members handle the incident calmly? Did everyone on the contact list respond promptly? Should the contact information be updated or changed in any way? And, finally, do you need to add anyone to the team or adjust the procedures?

"If you don't learn from what you've just experienced, you open yourself up to more attacks," says Raymond James Financial's Fredriksen. The review is your chance to improve the plan and the team so that you can work out any kinks before the next incident strikes. Fredriksen recommends doing a risk analysis after every incident to make sure as many vulnerabilities as possible are secured.

After the review, you will find it useful to complete an incident report for your records. Among other details, the report should include all the information you've gathered about the incident, both during the response process and in the post-mortem. That way, if you decide to pursue an investigation, you'll have all the evidence on hand.

Remember that the steps to a clear, planned response are not complicated. Once you are sure that an incident has actually happened, determine whether it's a major or minor event.

Decide whether your priority is to pursue an investigation and allow the incident to play out, or to shut down the problem as quickly as possible.

And finally, work to defend against further attacks. Take a look at the way in which the attack happened and determine if an application needs to be patched or a port reconfigured. Take whatever action is necessary to prevent the attack from happening again. And be sure to let everyone on the response team know that the problem is fixed.

IT threats may be coming faster and faster. But by having a clearly defined response process, you can prevent attacks from devastating your systems. "Plans are not a panacea," Reuters' Macartney says. "But if you use them strategically, you can limit your exposure to risk."

What Does a Security Incident Cost Your Company?

Determining the cost of a breach can be difficult - it often depends on the type of event that occurred and what damage, if any, the prolonged exposure added. We've found that few hardy souls are willing to buck up and disclose how much a security breach cost them. Part of the reason for their reticence is that they can't tell you how much it cost - their incident response plans are either nonexistent or they lack the means to track how much the incident cost in terms of lost productivity, systems downtime, staff overtime and estimated damage to the company's reputation (see "It's Not Easy Being Breached", February CIO). Then, when incidents do occur, no one has the wherewithal to sit down and take notes to determine how many hours it took to stave off a hack, how many pizzas were ordered for people working until 4am, and how many hours of downtime the systems suffered.

Creating a method of tracking the events and effects of a cyberattack at the time it occurs is both simple and smart. By making incident documentation a part of your response plan, you avoid trying to recount the incident and estimate its cost after the fact.

Here are the basic questions to ask when evaluating an incident:

• What happened?
• How did it happen?
• When were you aware of the incident?
• What is the damage?
• What systems have been affected?
• Are they working normally now?
• Which employees are affected?
• Who else is aware of the problem?
• What is the suspected attack method?
• What information is compromised?
• Is it sensitive?

Those questions and their answers should be recorded as part of an incident reporting form, a scorecard of sorts to keep track of events and help you document any damage or costs to your company.