IBM questions internal processes after Census outage

IBM managing director A/NZ, Kerry Purcell

IBM has questioned its own dealings with an internet service provider and its upstream partner in its handling of the 2016 Census project, after the government portal was taken offline for 40 hours in early August after being hit by a series of DDoS attacks.

According to IBM engineer, Michael Shallcross, the company’s efforts to instruct Australian telecommunications provider, Nextgen, and its upstream partner, Vocus, in the implementation of a geo-blocking distributed denial-of-service (DDoS) prevention plan (named Island Australia) in the lead up to Census day, had failed.

“It’s apparent from the submissions brought by Nextgen and Vocus that perhaps the internal communications had not conveyed adequately the intent and instructions of and surrounding the implementation of Island Australia,” Shallcross told the senate committee investigating the 2016 Census project in a public hearing on October 25.

“We, as the prime contractor, dealt with both Telstra and Nextgen as our ISPs, and expected them, as large internet service providers, to be able to implement those instructions correctly.

"Telstra did [that] for us in 2011, and again in 2016. We had difficulty with the implementation by Vocus of those particular instructions.

“If there was one thing [I would do differently]…it would be trying to gain a greater certainty that the ISPs upstream partners correctly understood our instructions,” Shallcross said.

Nextgen has previously said it had "provided all possible assistance" to IBM to put in place the Island Australia plan, and that it complied with the technology company's framework to hold DDoS attacks at bay.

The IBM engineer also outlined a failure in the configuration in one of two routers it was using to channel data traffic to the Census site from its two ISP partners which, ultimately, led to a failed reboot after it was shut down following a data surge caused by a fourth DDoS attack - this left the router inoperable for more than an hour.

“We did, during the lead up to the Census, test the impact of a failure of that router and test that the failover mechanisms on the rest of the site worked effectively,” Shallcross said.

“But we tested that router failure by simulating it, which is relatively easy to do in a repeatable fashion. If we had our time again, we would probably test a hard ‘power-off, power-on’ that router. That would have discovered earlier that we had that reboot and configuration and loading problem,” he said.

The DDoS attack that ultimately overwhelmed the infrastructure put in place by IBM, which had been contracted by the Australian Bureau of Statistics in a $9.7 million deal to lead the 2016 Census project, and its ISP partners, was routed through Singapore.

According to IBM, the first of the four attacks notched up data rates of 3 Gbps. By the time the fourth attack was underway, the company said its staff noticed a “qualitative” difference in the traffic. It was this shift, and IBM's subsequent misinterpretation of the data, that prompted the company and the ABS to shut down the site in the interest of protecting it.

Since the inquiry began, IBM has claimed that the geo-blocking DDoS protection strategy was discussed in length with the ABS and signed-off by the agency.

However, it had not been formally approved by the Australian Signals Directorate (ASD), the national agency responsible for the collection and analysis of foreign signals intelligence, and the provision of cyber security for the government.

“It was discussed with ASD, but I’m not aware that the ASD ever passed a comment saying one way or the other that they endorsed it or not,” Shallcross told the committee.

The geo-blocking strategy was an appropriate approach to protecting the Census site from DDoS attacks, according to Shallcross, given that it was only meant to be used by people who were within Australia at the time.

“We stand by very firmly of the view that geo-blocking is an effective DDoS attack prevention mechanism,” IBM A/NZ managing director, Kerry Purcell, told the senate committee.

Purcell's stance on geo-blocking stands in stark contrast to comments by the special adviser to the Prime Minister on cyber security, Alastair MacGibbon, who told the committee that IBM's approach was flawed.

“There certainly were better alternatives, yes," MacGibbon said.

"The concept of the ‘Island Australia’ geo-blocking was to prevent internet traffic coming in from overseas to an Australian website. And on face value that might seem reasonably logical," he said.

But it is not uncommon for internet traffic sourced from Australia to look as though it has come from overseas, MacGibbon explained. In fact, IBM's own password reset system for the Census website relied on data coming in from overseas. As such, the geo-blocking approach was problematic, he suggested.

“There was a fundamental failure in the logic of an ‘Island Australia’. I could see it as part of a series of protections, adding some value. But to rely solely on it, clearly, was a failure," he said.

Page Break

Meanwhile, Purcell also confirmed that IBM has neither sacked nor disciplined any employees following the failure of the Census site and IBM-supplied infrastructure to withstand a DDoS attack.

This is despite previous reports that two senior executives had resigned soon after the debacle, and calls by Prime Minister, Malcolm Turnbull, that “heads will roll” over the outage.

At the same time, Purcell revealed that he is in talks with the federal treasury to pay compensation costs to the government, following the Census debacle.

“I’m engaged directly with the secretary of the treasury, John Fraser, and we are looking to constructively resolve the matter as soon as possible. I’m confident we’re able to achieve some kind of outcome in the very near future,” Purcell said.

Purcell’s comments followed claims by former ABS chief, Bill McLennan, that the impact of job cuts at the agency may have led to a “gross lack of experience” among senior staff.

This is a charge that IBM representatives have challenged, with Purcell and Shallcross both suggesting they had not seen or experienced any technical weaknesses within the Bureau’s IT ranks.

For his part, current ABS boss, David Kalisch, apologised for any inconvenience caused by the outage, but has continued to hold IBM accountable, telling the committee that IBM’s systems should have been more “robust”.

“The ABS made a number of poor judgments in our preparation for the Census that led to poor service experienced by many households, and I apologise to the community on behalf of the ABS,” Kalisch told the committee.

“We made a difficult decision to taker the system offline on August 9 to ensure the security of Census data, but we should not have got to that point, and the IBM systems should have been more robust to DDoS events,” he said.

Kalisch said that he was surprised to learn that the series of DDoS attacks overwhelmed the Census website, given the assurances that systems had been put into place to secure it against such attacks.

Additionally, Kalisch announced the establishment an independent panel to assess the quality of the 2016 Census data, and promised that the next Census, due in 2021, will adopt a more “rigorous” approach, following the events of this year’s Census.

It is not clear, however, how this move may affect IBM’s long-time relationship with the Bureau.

The inquiry continues.