Charles 'Charlie' Pellerin.
There's nothing unusual about having a bad day at the office. But some people have worse days than others, and in his time Charles (Charlie) Pellerin has had a few notable ones. Not many people find themselves having to explain why an organisation has invested a decade and half and in the vicinity of $3 billion on a project that has failed.
That's the position Pellerin found himself in as NASA's director of astrophysics in the wake of the 1990 launch of the Hubble Space Telescope, which had what appeared to be an unfixable flaw in its optical system.
It's difficult to overstate what a disaster this was and the humiliation faced by NASA; not just as an organisation but also the individuals who worked for the agency. A good friend of Pellerin who worked on the telescope fell ill in the wake of the launch and died. Two of Pellerin's senior staffers had to be removed from their offices by guards and taken to alcohol rehab facilities. "These are PhDs sitting at their desk getting drunk; this is how bad the stress was," says Pellerin.
Most people faced with a disaster on the scale of Hubble might want to either bury themselves under a blanket in bed for a decade or two, or try (no doubt unsuccessfully) to forget it ever happened. Instead Pellerin set out to try to fix Hubble — and succeeded, in the process winning NASA's Distinguished Service Medal, the highest honour conferred by the agency. And with a stubbornness that some people may consider verges on the perverse, set out to discover exactly what went wrong. The problem with Hubble, Pellerin concluded, wasn't merely a technical failure. It was a leadership failure and a product of the culture surrounding the project.
Techworld Australia features:
Hadoop: How open source can whittle Big Data down to size
The road to a successful open source project: Learning lessons from Drupal
Open Source Ecology: Can open source save the planet?
Python vs. PHP: Choosing your next project's language
His study of why projects fail also led him to draw links between Hubble and an earlier NASA disaster: The disintegration of Space Shuttle Challenger on January 28, 1986, which killed seven astronauts.
"In 1986 I drove into my parking place at NASA headquarters and my staff members are waiting for me there and it's very unusual," Pellerin says. "They ran up to me and said, 'Charlie, Charlie, Challenger exploded." Pellerin's division had the biggest payload on Challenger, and in the lead-up to the launch there had been a lot of cross-training between his division and the astronauts. "I had this sinking feeling that maybe my payload had come loose, because it was a very low cost effort and it was big and heavy and maybe it had broken through the cargo bay doors and caused this accident."
Pellerin watched the Challenger failure review closely. "I had good friends who I thought were good solid engineering managers who worked on the boosters so I'm trying to figure out what happened," he says.
"I saw this guy, Richard Feynman, who was a review board member, take a piece of rubber O-ring and put it in his icy water on television, and showed that it stiffened up. So immediately I said, 'Oh, that's the technical problem, they didn't do the O-ring well.'"
"That was nuts," Pellerin says. "These guys understood the O-ring, but I put that story in my head because technical people look for technical answers. I never read the conclusion of [the review board] report that said it was a social shortfall."
Four years after Challenger Pellerin was getting ready to launch Hubble and grappling with the difficulties of readying a telescope that wasn't intended for in-atmosphere operation. The advantage of a telescope in space is that light from stars won't be moved around by atmospheric incoherencies. No-one had attempted to design a telescope that would offer the accuracy promised by Hubble.
"So the question is, what are you going to tell people if someone asks you if it's going to work? What would you say? 'Of course.' It's the only answer right?" Pellerin says. "You spend 15 years and $3 billion or whatever, so of course it's going to work — there's no other answer.
"The head of NASA congressional appropriations asked me, 'Is it going to work?' And I said, 'Of course!' So she went and identified herself very closely with the project. So then we find it doesn't…"
The circumstances under which Pellerin discovered Hubble's flawed mirror were "awkward". There was a 'first light' event for the opening of Hubble's aperture door — "we call it the toilet seat," Pellerin says. The aperture door was opened, and a little spot of light appears — the first light from Hubble. "Everybody whoops and cheers," Pellerin says. But he noticed the spot of light was fuzzy. He was reassured by a colleague that it was nothing serious; Hubble's secondary mirror was attached to a stepping motor that would allow minor alterations to cope with dimensional changes brought about by the outgassing of water vapour in space.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.