Testing in-house and vendor-built software for security holes should be an enterprise priority, said a group of vulnerability research experts speaking on a panel at the Gartner IT Security Summit held in the US.
But Rich Mogull, the Gartner analyst who hosted the panel, questioned how practical it would be for companies to dedicate the dollars and resources required for this testing.
Thomas Ptacek, founder of application security consulting firm Matasano Security, defined vulnerability research as analyzing software for holes that attackers could take advantage of before the product is deployed, using techniques such as reverse engineering and source-code auditing.
Software vendors and many enterprises have teams of engineers in-house to perform this testing, or rely on third parties such as the panellists' companies that specialize in finding vulnerabilities.
Every piece of software is not a Boeing flight-control system where everyone will die if there's a bug
The benefit of this testing is being able to avoid the damage an attacker could cause by fixing software problems before implementation.
"If you don't find the problems, someone [else] will find the problems," said Chris Wysopal, co-founder of VeriCode. "If you leave crumbs on the floor the ants are going to show up. That's a huge liability ... for your company."
For software built in-house, vulnerability testing should be part of the software development lifecycle, not an afterthought, Wysopal said.
"Threat modelling to find out what are the weakest parts and easiest attack vectors [of an application] is what people should do when designing software; you find the weak points through threat modelling then start reverse engineering," he said.
Simply using tools that scan for vulnerabilities is not enough, the panellists agreed.
"Scanning tools can reduce the amount of time you spend [analyzing the code] manually, but if you care about the security of the application ... you need to go deep and augment the scanner," says Ptacek. "The place of the scanner is to accelerate testing, but you can't rely on them."
Gartner's Mogull did an electronic poll of the roughly 1000 conference attendees, asking what level of vulnerability testing they performed at their organizations. The majority said their testing was limited to using commercial scanning tools.
One reason enterprises may not be doing more intense vulnerability testing is because the necessary skills are rare, Mogull suggested.
"It's a huge skills issue," conceded Wysopal. "It would be best to have an expert researcher looking at every piece of code out there, but you just can't find them."
Ptacek disagreed, saying services such as Web application penetration testing are readily available.
Another panellist, Errata Security co-founder David Maynor, added that any steps an organization can take to find vulnerabilities in software are worth it.
"You're not wasting your money just because you don't find bugs," Maynor said.
Yet the process is still an expensive one, Mogull said, and enterprises can't be expected to dedicate such time and money to extensively testing every application.
"You have to have the appropriate level of testing [correlate to] the risk of the application," said Wysopal. "Not every application requires hired experts coming in, or training up a large team. Some applications are not as risky as others."
Ptacek again disagreed, saying that every application offers entry into an organization.
"Even innocuous applications deployed inside an organization can be lethal" if exploited, since just about every application contains or connects to sensitive data. The one example the panel came up with of a program that wouldn't pose a threat if compromised was an employee lunch-ordering application.
"Every piece of software is not a Boeing flight-control system where everyone will die if there's a bug," countered Wysopal.
Mogull asked what percentage of the application development lifecycle budget should be devoted to vulnerability research. Wysopal answered at least 5 percent, Ptacek said between 5 percent and 10 percent - adding that the cost for such testing should actually come out of the quality and assurance budget - but Maynor said closer to 25 percent.
"It's easier to allocate dollars before deployment than fix [a vulnerability] after it's been deployed," he said.
Turning to commercial software, Mogull asked the audience for a show of hands if they reverse engineer products they buy from vendors to look for vulnerabilities. Outside of the panellists, few hands went up.
"It would be ridiculous to test [to that level] every single product you bring into your organization. What level of testing is OK?" Mogull asked the panel.
"Put one to two person weeks on any new product and you'll find stuff," said Ptacek. "Use a sniffer and look at packets on the wire. You'll find vulnerabilities, and you'll gain control over how they're going to be fixed."
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.