Thursday, April 21, is a day that Michael Downing, the CEO and CFO of social media start-up Tout, won't soon forget. In the wee hours of the morning, Downing learned a harsh lesson: cloud computing is not bulletproof.
Tout, which had launched its real-time video status update service a week and a half earlier, was among the numerous customers taken down by Amazon's EC2 outage. Not only was the main database, which houses critical account information, impacted, but Downing also quickly learned that the company's application server partner, Heroku, also was an Amazon customer -- and offline. "The first 90 days is the critical time when you're trying to establish your brand and you build momentum. That wasn't possible when our systems were at a complete standstill," Downing says.
Before this incident, Downing was proud that more than 90% of his applications were being hosted in the cloud so the company could get off the ground without the shackles of high infrastructure costs. "I've trusted and used cloud services for years and this technology is transformational for the start-up world," he says.
Broken Trust
That trust is now irrevocably broken, he says. While Heroku came back online relatively quickly, his database remained down for almost 48 hours. At some point, after little communication from Amazon about a fix, Downing and his team uploaded a three-day-old snapshot of the database to a server at another Amazon location -- far from the ailing Virginia data center. "Although we permanently lost some data, we were at least able to get back online," he says.
As much as a week after the incident began, Downing says that Amazon still hadn't been in touch with him to explain the outage that we now know stemmed from a configuration error, other than generic, mass messages. "Part of the whole value proposition when you sign on for these services is there will be no one single point of failure and even if a whole node goes down, your systems won't be tanked. This was a huge eye opener that proved that is definitely not the case," he says.
As this story was being published, Amazon hadn't responded to a request for comment.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.