IBM ignites Census war as blame game begins

IBM has pointed to government agencies and at least two internet service providers over the failure of the Australian Bureau of Statistics’ (ABS) Census website to withstand a series of distributed denial of service (DDoS) attacks in early August.

The technology giant, which was contracted by the ABS to develop, implement, and host the eCensus platform for the 2016 Census, said in its submission to the government committee investigating the 2016 Census project that it had anticipated and planned for the risk DDoS attacks to the site.

“The main defence mechanism utilised was a form of protection known as geo-blocking (known internally at IBM as ‘Island Australia’),” the company said in its submission.

“In short, the geo-blocking arrangement involves blocking or diverting international traffic intended for the eCensus site before it reaches the site, while leaving the system free to continue to process domestic traffic.

“This method was chosen because the primary risk of DDoS attacks of sufficient size to disrupt site availability was considered to be from foreign sources,” it said.

Additionally, IBM claims that the ABS and the Australian Signals Directorate (ASD) were aware that the technology company intended to use geo-blocking.

At the time, according to IBM, the ABS’ IT security personnel considered geo-blocking to be an “extremely effective control”.

“IBM understands that the ASD was asked by the ABS to review the security arrangements for the 2016 eCensus site, but the ASD declined to undertake a detailed review,” it said.

Big Blue said that it met with the ABS and the ASD on 21 July 2016 to seek the ASD’s input on security threats for the project.

During the course of the discussion, IBM asked the ASD if it was aware of any intelligence relating to planned denial of service attack risk - the ASD said it was not.

The vendor also claims that the geo-blocking arrangements were implemented by the internet service providers engaged to provide public access to the eCensus site – in this case, Nextgen Networks and Telstra.

However, Nextgen said its offer of DDoS protection was flatly rejected by the vendor.

“In accordance with IBM’s order, Nextgen supplied IBM with a standard internet service, and met all of its service levels on that product,” according to Nextgen’s submission to the committee.

“Although Nextgen strongly recommended to IBM to take up an internet DDoS protection option for the purposes of the 2016 census, it was declined by IBM,” the company said.

Nextgen said its commercial proposal to IBM dated 12 January 2015, with the company claiming email confirmation of rejection of its DDoS protection plan came through on 24 May 2016 from IBM.

“This additional feature offered by Nextgen is designed to effectively detect and defend against DDoS attacks,” Nextgen added.

According to IBM, under its arrangement with the ISPs, if a DDoS attack was attempted, and was severe enough to warrant the implementation of the geo-blocking arrangement, IBM would direct Nextgen and Telstra to put ‘Island Australia’ into place.

While Nextgen provided IBM with its commercial proposal on 12 January 2015, however, the ISP said it was “not privy” to the IBM 'Island Australia' strategy until 20 July 2016, just six days before the eCensus site went live.

“Nextgen provided all possible assistance to IBM (which is well beyond what is provided for a standard internet service) to put in place 'Island Australia',” Nextgen’s submission stated.

“Nextgen complied with the IBM 'Island Australia' framework requirements provided by IBM, which was activated for testing on 5 August 2016 by IBM.

Page Break

“IBM advised Nextgen that the test results were successful and positive. IBM’s intention was to activate IBM’s 'Island Australia' only when there was a DDoS attack and would accordingly instruct Nextgen to do so when needed.”

After becoming aware of 'Island Australia', Nextgen claims that it advised the vendor that the IP address range requested was part of a larger aggregate network, and therefore it was not possible to provide specific international routing restrictions for this range.

“Nextgen recommended using an alternative IP address range, which would give IBM better control, but this was rejected by IBM,” Nextgen’s submission stated.

Furthermore, IBM requested the application of IP address blocking filters by Nextgen’s upstream suppliers, and international remote black holes for 20 specific individual host routes, which Nextgen complied with.

Nextgen said the individual host routes picked by IBM “may not be exhaustive” however, and DDoS attacks could come from other routes in the IP address range - which they did in the third DDoS attack on Census Day.

Prior to Census Day, IBM instructed Nextgen to activate 'Island Australia' for testing on August 5 at 6am, and reported a successful operation, yet just over 24 hours later, the first DDoS attack hit.

Coming through at approximately 10:10am, Nextgen claimed IBM was alerted to the breach, with the attack subsiding by 10:20am.

At approximately 11.45am however, the eCensus system experienced its second DDoS attack, with IBM instructing Nextgen to activate 'Island Australia', which was allegedly put in place within two minutes, using the same pre-configuration that had undergone testing the day previous.

As a result, the eCensus site returned to normal by 11.49am.

“At all times Nextgen was in contact with IBM, and IBM’s 'Island Australia' remained in place after the second DDoS attack,” Nextgen’s submission stated.

“IBM’s router facing the Nextgen link was rebooted soon after the [third] attack and IBM kept the Nextgen IBM link down until it was comfortable there was no data breach,” Nextgen’s submission stated.

“After the fourth DDoS attack, Nextgen offered to implement the DDoS protection option. This was provided at Nextgen’s cost and continued to provide full support to IBM on the service,” it said.

For its part, IBM said that the site underwent performance and security testing by the ABS before it went live. The company said it also performed hundreds of tests itself in the course of developing the site and the eCensus application.

“The geo-blocking arrangement was tested prior to Census Day and worked. A geo-blocking arrangement had also been implemented as a DDoS defence for the 2011 Census,” said IBM.

However, when a DDoS attack – the fourth of the day – was detected by IBM on the eCensus site at 7.27 pm on 9 August 2016, the attack was “of significant size” and had the effect of causing the site to become unresponsive and unavailable to the public, the company said.

“Regrettably, the 7.27 pm DDoS attack also caused one of the mechanisms used by IBM to monitor the performance of the eCensus site to miscarry,” said IBM.

“As a result, some IBM employees who were observing the monitor mistakenly formed the view that there was a risk that data was being exfiltrated from the website and that the risk needed to be further investigated.

“Out of an abundance of caution, IBM shut down access to the site and assessed the situation. The cause of the problem was identified. No data exfiltration occurred,” it said.

Following the fourth DDoS attack, the firewall to the eCensus site, through which IBM’s control link to the routers on both the NextGen link and the Telstra link operated, became overloaded with data.

The overload of the firewall required manual rebooting of an IBM router on the open Telstra link which, due to a configuration error, took more than an hour to resolve, the company said.

Page Break

The attack, according to IBM, was foreign-sourced and hit the eCensus site via the NextGen link at a time when IBM had already told NextGen and Telstra that ‘Island Australia’ was to be in place. NextGen had provided “repeated assurances” to IBM prior to the attack that it had done this, the company said.

“In fact, the assurances were incorrect. IBM was informed – later that day after the attack had passed – that a Singapore link operated by one of NextGen’s upstream suppliers (Vocus Communications) had not been closed off, and this was the route through which the attack traffic had entered the NextGen link to the eCensus site.

“Vocus admitted the error in a teleconference with IBM, NextGen and Telstra around 11.00 pm on 9 August 2016,” said IBM.

IBM claimed that if NextGen and Vocus properly implemented ‘Island Australia,’ it would have been effective to prevent the final DDoS attack, and the effects that it had on the eCensus site which, ultimately, led to its shutdown.

“As a result, the eCensus site would not have become unavailable to the public during the peak period on 9 August 2016,” said IBM.

For its part, Vocus – the upstream supplier of Nextgen and provided IP Transit Services and DDoS protection services to Nextgen which were resupplied to IBM for the eCensus project – has questioned IBM’s version of events.

“Vocus does not agree that the fourth DDoS attack was the cause of the site becoming unresponsive,” Vocus’ submission to the committee stated.

“The fourth attack comprised of attack traffic which peaked at 563Mbps which is not considered significant in the industry, and lasted 14 minutes.

“Such attacks would not usually bring down the census website which should have had relevant preparations in place to enable it to cater for the expected traffic from users as well as high likelihood of DDoS attacks,” it said.

This mirrors earlier comments by the ABS, which said in its submission that the DDoS attack should not have been able to disrupt the system and that, despite extensive planning and preparation by the ABS for the 2016 Census, the risk was “not adequately addressed” by IBM.

According to Vocus, the cause of the census website being unreachable was “IBM employees falsely identifying normal traffic patterns” as data exfiltration, and “manually turning off their Internet gateway routers” – a claim that IBM’s submission supports.

In addition, Vocus said IBM “took approximately three hours to configure and bring the website back up again”.

“The traffic coming through the Singapore link amounted to a total of 563Mbps, and not of a size to cause the census website to become unresponsive, had appropriate network security measures been implemented by IBM,” Vocus’ submission stated.

“In addition, it is incorrect for IBM to represent that DDoS attack traffic travels through a single link, in this case, the Vocus Singapore peering link,” it said.

Referring to IBM’s technical description of the DDoS attacks, Vocus added that devices (‘botnets’) can be located anywhere in the world, including inside Australia.

Furthermore, the telco claims that the ‘Island Australia approach’ “does not consider the reality of overseas network operators” connecting to Australian service providers inside Australian borders.

“In fact, during the fourth DDoS attack, Vocus had blocked the vast majority of DDoS traffic, only passing on a small percentage of the total traffic from botnet hosts in Asia and Australia,” Vocus’ submission added.

“Once Vocus was made aware of the fourth DDoS Attack, it implemented a static null route to block additional DDoS traffic at its international border routers within 15 minutes,” the company said.

Although Big Blue said it accepts its responsibility as the head contractor for the eCensus project, the use of ISPs, such as NextGen, to provide links to the eCensus site was required for the projects and could not be avoided.

The company also revealed that, since the 11 August shutdown of the site, there have been further DDoS attacks on the eCensus site, all of which have been successfully defended.

“The DDoS attacks on 9 August 2016 highlight the importance of the risk that cybersecurity threats present to both government and industry, now and into the future,” the company said.

Additional reporting by James Henderson.