On December 31, all the 30GB Zune models turned into bricks because of a Leap Year firmware coding error. This quality assurance and testing debacle demonstrates three lessons every software developer should take to heart.
On the last day of 2008, every one of an older model of the Microsoft Zune MP3 player (the ones with 30GB of storage) locked up. The devices were back in operation again a day later, and Microsoft explained the cause of the trouble:
"A bug in the internal clock driver related to the way the device handles a leap year. The issue should be resolved over the next 24 hours as the time change moves to January 1, 2009. We expect the internal clock on the Zune 30GB devices will automatically reset tomorrow (noon, GMT)."
With that data, Zune and technical users have some idea of what happened in the "Z2K9" incident. Microsoft's Scott Hanselman wrote a very good technical analysis for programmers about the dangers of such edge cases, and apparently he's not the only one to cover the bug. (My thanks to Indrajit Chakrabarty for the pointer.)
However, aside from "how not to write code like that," there are three important things for developers and software QA professionals—and their managers—to take away from the experience.
This Was a Failure of the Software Development Process and QA Testing
It's great that the technical problem was so easily addressed ("wait a day"), but it's one heck of an embarrassment for Microsoft. I'm not talking about their PR issues per se, though Microsoft is still trying to live down the Red Ring of Death debacle with its XBox. However, Microsoft has a long history of, shall we say, a less-than-stellar reputation for quality, and they did not do themselves any favors with this incident. I feel especially sorry for the authors of the new book, How We Test Software at Microsoft (cue: pointing and giggling) and the many smart people I have met from the company. (They have great people. Really. Some of the smartest techies I've met. But somehow Microsoft doesn't seem to create a culture that demands quality.)
But the bottom line is that this problem was entirely preventable. As a London-based web developer pointed out to me, "Edge conditions such as year transitions on leap years really ought to be tested as a matter of course, and shouldn't be that difficult to do on devices where you can adjust the clock." The date problem really should have been spotted before it was checked in, he says; any sort of code review probably would have spotted the infinite loop possibility. So why wasn't it done? Why wasn't it caught?
I do understand the notion of "ship on time," and that some things get lost in the eternal desire to make a production date. Quality assurance testing is not the only victim. But this is a well-defined problem set with pretty darned obvious unit tests. (I won't be surprised if I get e-mail messages from QA Tools companies telling me that their products include such tests as a matter of course. Just post a response to this post, folks. In this context, it's fine.)
We all make mistakes. But the purpose of software engineering is to catch and fix errors before the product is released.
For further contemplation: Would your company's software development process have caught an error like this?
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.