It happened again: Amazon Web Service suffered its latest outage in late June. Now, as the dust has settled, customers are reassessing what lessons can be learned and how to prepare for the inevitable next one.
Compared to AWS's major outage last summer, which was caused by human error and resulted in an overloaded network, the most recent incident resulted from an electrical storm causing a power outage in an AWS Virginia data center. While the actual outage only lasted about 20 minutes, the domino effect of a backup generator not kicking in, combined with software bugs AWS had not seen before, caused about 7% of customers in the impacted area to be down, some for as much as three hours on the evening of Friday, June 29.
READ: Amazon takes blame for Amazon takes blame for outages, bugs and bottlenecks
COMPETITION HEATING UP: Amazon in the crosshairs of Google and Microsoft
As storms ripped through the mid-Atlantic coast that Friday night and into Saturday morning, parts of sites such as Netflix, Pinterest and Instagram were down, sometimes for as much as three hours. But it didn't have to be that way. Software startup Newvem tracks AWS customer usage, and officials say misconfigurations by customers exacerbated the problem on that Friday night. Newvem and Netflix have four suggestions of how the latest outage could have been mitigated and how to prepare for future incidents.
1: Use snapshots
Backing up data is critically important to ensure high availability and AWS gives customers the option of backing up their Elastic Block Store (EBS), which is a file storage service impacted during the latest outage, with a "snapshot." EBS Snapshots make a copy of the EBS volume and back it up in Amazon's accompanying Simple Storage Service (S3) offering. User to have to initially back up their entire EBS volume to S3, but then whenever there is a change to the content of the EBS volume, only the new data has to be captured in another snapshot for the volume to be recreated. Of Newvem's more than 500 customers, 45% of users who have large AWS clouds, meaning those with more than 101 instances, did not have effective EBS snapshots.
2. Ensure correct ELB configurations
One of the advantages of using Elastic Load Balancers (ELBs) is they can automatically reroute traffic based on availability and need. But Newvem found that up to 20% of heavy users aren't properly configuring their ELBs either. One of the most common misconfigurations is to reroute ELB traffic within the same availability zone (AZ). AWS has multiple availability zones within its regions, which are meant to be isolated from one another. By not configuring the ELB to route traffic to a separate AZ, users aren't protected if their AZ is impacted, Newvem says.
3. Test, test, test
One of the bigger names that went down during the latest AWS outage was Netflix, which during the past few years has migrated much of the company's video streaming services to the AWS cloud. During the latest outage, the site had selective service disruptions between 8 and 11 p.m. PDT on the Friday night of the outage.
In response, Netflix wrote a blog post outlining changes they will make to prepare for an AWS disruption and area they hope to ramp up is testing. Netflix already has "Chaos Monkey," which simulates an outage of random instances within the Netflix AWS cloud. But that's apparently not good enough. The company is now developing a "Chaos Gorilla," which will simulate an entire availability zone going down to ensure the system can automatically handle the situation.
4. Not just multi-AZ, but multi-region
After last year's outage, AWS officials reminded users that using multiple availability zones is the best way to ensure AWS cloud resiliency. Now, Netflix and Newvem officials agree that instead of a multiple AZ architecture, spreading workloads across multiple regions, or even across multiple cloud providers is the best way to ensure high availability. "Using multiple regions is really the new best practices for customers that really require high availability," Newvem CEO Zev Laderman says. Netflix says as it is expanding its global footprint to allow streaming of its video content around the world, it will be moving to a multi-region support system as well.
Network World staff writer Brandon Butler covers cloud computing and social collaboration. He can be reached at BButler@nww.com and found on Twitter at @BButlerNWW.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.