Open Source Code Contains Fewer Defects, But There's a Catch

Open source code is lower quality than proprietary code. At least, that's how many people now perceive it.

Until this year, you could make a persuasive argument that defects in freely available source code are more likely to be spotted and fixed promptly than defects in proprietary software. Then along came Goto Fail, Heartbleed, Shellshock and Poodle. These four high-profile bugs in open source software weren't detected and fixed for years, in some cases, despite the code having been freely available for anyone to inspect.

That's been enough to put a question mark back in many people's minds about the way that open source software is developed -- and whether it's enough to count on someone, somewhere, analyzing the code and spotting defects. There's a risk that everyone assumes someone else analyzes the code when, in fact, no one with the necessary skills is actually doing so. This calls into question the wisdom of adopting open source software in the enterprise at all.

[ Survey: Security, Quality Top Companies' Reasons for Using Open Source ]

But proprietary software frequently contains defects, including security vulnerabilities. Is there any real evidence to suggest that open source code is better or worse than its closed source counterpart?

The annual Coverity scan report provides one source of objective information about the amount of code defects in open source and proprietary software. The report analyzes the levels of defects found in software developed using the two different models, which it runs though its static analysis system.

It's important to bear in mind that the scan report only includes software that's submitted for scanning; in a sense, this is a self-selected sample. That said, it turns out that the defect density -- the number of bugs per 1,000 lines of code -- of open source and proprietary software are broadly similar.

In fact, the most recent report (2013) found open source software written in C and C++ to have a lower defect density than proprietary code. The average defect density across projects of all sizes was 0.59 for open source, and 0.72 for proprietary software.

Applications with few lines of code had, in general, lower defect densities than larger ones, although large apps with more than 1 million lines of code actually had a lower density than some medium-sized apps.

Open Source Code May Be Secure, But Fewer Teams Protect It

That seems to give open source software an endorsement as far as code quality is concerned. When it comes to security bugs specifically, though, many open source projects take inadequate steps to prevent them, according to Zack Samocha, Coverity's senior director of products.

"A typical proprietary software company will have a security team that ensures that best practices are carried out. They have the budgets to maintain those teams," he says. "I don't see those entities ... in the open source community."

[ Feature: Why Open Source Software Isn't as Secure as You Think ]

That may be true of smaller projects, but commercial open source companies such as Red Hat, along with many large open source projects -- especially those with software intended for enterprise deployment -- almost always have a concerted security effort.

Samocha says the Linux community is taking steps to increase security -- although more needs to be done, in his opinion -- but adds that all open source projects need to add security to their day to day thinking. "In commercial software companies, developers and CIOs think about security all the time. In open source projects, I'm not sure that that happens."

Samocha also believes that open source projects could better use the ecosystem around security, and the commercial tools that exist. "The majority of tools are aimed at security teams rather than developers," he says. "Security teams should be filtering the defects and giving relevant ones to the developer community to fix."

Static Analysis Tools Help, But Only to a Point

Of course, projects that don't have a security team can't work in this way. Individual developers who run analysis tools risk being swamped by too many irrelevant bugs or false positives. This may lead them to ignore these tools altogether.

"If developers have to waste precious cycles investigating too many 'don't care' findings, or if they spend too much time waiting for the analysis to finish, they will inevitably stop using those tools," says Jon Jarboe, a senior technical manager at Coverity.

Security teams also prove useful because, while running code through static analysis tools can spot many code defects, it certainly won't catch them all.

The Heartbleed bug is a good case in point: At the time, Coverity's static analysis wouldn't have detected it before it was discovered. (The company has added a new analysis heuristic that does now detect it, but that's beside the point.) The Shellshock bug is another: Some static analysis tools would have spotted it, but only along with a large number of issues that would have turned out to be false positives.

"Static analysis can find known vulnerabilities. What about the unknown vulnerabilities?" asks Mike Gualtieri, a principal analyst at Forrester Research. "That's where threat modeling comes in. This technique allows you to find the threats to your application and to identify mitigation strategies for each of these threats."

When in Doubt, Audit Your Code

Such a holistic approach to security may be difficult to achieve in a project with many contributors and no security team to coordinate it. How, then, can the level of confidence in open source software be improved?

[ Related: 5 Ways to Get Open Source Software Support ]

For businesses that decide to use open source code from projects that may not have the resources to devote to security, there's always the option to have the code audited by a suitably qualified person or team.

Such audits can be expensive and time-consuming, though, so it could be that large companies or even industry groups end up funding these projects for the benefit of the wider community of users. Alternatively, crowd-funding audits -- in the way that the TrueCrypt audit has been funded since it shut down in May -- may become increasingly common.

When it comes to the core open source projects, such as OpenSSL, which make up much of the software infrastructure on which the Internet operates, the Linux Foundation's Core Infrastructure Initiative may make a significant difference. Funded by the likes of Microsoft, IBM, Google and Dell, it aims to provide resources to assist these projects in improving their security and paying for outside code reviews.

Confidence in open source software has certainly taken a hit over the last several months -- but, in the end, it's worth remembering that all software can be found to contain code defects. What's more, there's certainly no evidence that software developed using the open source model is more likely to contain serious defects that any other type of software.