'Power event' at AWS knocks out websites and services

A ‘power event’ at Amazon Web Services’ data centre in Sydney on Sunday night was to blame for hundreds of disgruntled end users being unable to access websites and apps.

Foxtel Play, Channel Nine, Presto, Stan, Dominos, Fairfax Media and Go Get services were among those unavailable, as the city was hit by severe storms.

An investigation into connectivity issues was first reported on Amazon’s service dashboard at 3.47pm on Sunday and an hour later the company confirmed that its instances “have experienced a power event”. By 5.31pm AWS told users: “We have restored power to the affected Availability Zone and are working to restore connectivity to the affected instances.”

The power event affected AWS’ Elastic Compute Cloud (EC2), which in turn caused connectivity issues for other Sydney services including AWS ElastiCache, Redshift, Relational Database Service, Route 53 Private DNS, CloudFormation, CloudHSM, Database Migration Service, Elastic Beanstalk and Storage Gateway.

In an update this morning AWS said that 80 per cent of affected instances and volumes had been recovered by automated systems by Sunday evening but the remainder were not due to “a couple of unexpected issues”.

Those issues had been fixed by 1am this morning and “all but a small number of instances and volumes were recovered.”

“Most of these instances are hosted on hardware which was adversely affected by the loss of power. While we will continue to work to recover any affected instances or volumes, we recommend replacing any remaining affected instances or volumes if possible,” the AWS status update, marked ‘Resolved’, said.

Service users turned to social media to mark a victory for “real clouds”, while @AWSCloudANZ, the account for Amazon Web Services Australia & New Zealand remained notably silent. The most recent tweet on the account plugs an upcoming podcast that helps listeners “keep informed”.

CIO Australia has contacted AWS for further comment.