Inside the mop-up of a huge data “cock-up”

In late 2016, the Australian Red Cross Blood Service made headlines for all the wrong reasons after it emerged that the personal information of more than half a million blood donors had somehow found its way onto a publicly-facing website.

It was discovered that a file containing donor information, including names, gender, addresses, blood type, and phone numbers, had been placed in an non-secure environment by a third party technology partner that had developed and maintained the Blood Service’s website.

Rob Van Selm, Asia Pacific delivery director for one of the Blood Service's technology partners, Precedent, subsequently confirmed the company was working with the organisation in relation to the breach.

The breach came to light after an anonymous source sent security researcher, Troy Hunt, one of 647 different donor tables that were publicly discoverable online. The 1.74GB MySQL database back-up that had been discovered online contained more than 1.28 million records.

“There's no escaping the fact that this was a major cock-up on many levels and that's the simple, honest truth,” Hunt said in a blog post at the time.

The incident was remarkable due to the sheer volume of information that had been compromised – with some pundits at the time referring to it as Australia’s largest data breach to date.

However, it was also remarkable to some degree due to the speed and efficiency with which the Red Cross Blood Service acted to remedy the situation.

Not only did the organisation immediately launch an investigation into the breach, it swiftly engaged Australia’s Computer Emergency Response Team (AusCERT).

As a paying member of the non-profit organisation, the Australian Red Cross Blood Service was able to get AusCERT and its general manager at the time, Thomas King, on board early to help it stem the fallout of the incident.

From the perspective of King, who has since moved on to become general manager of managed security services at Telstra, the move by the Red Cross Blood Service’s CEO, Shelley Park, to be open and up-front with the public about the breach was an important first step to dealing with the problem.

The organisation even launched a dedicated hotline and organised access to a national identity and cyber support service for concerned donors to find out more information about the incident.

“The Blood Service is an exemplar…of owning the problem, and not trying to deflect it, even though it wasn’t actually fully their fault. A third party let them down; it was human error. It wasn’t a hack,” King told ARN. “Even though that was the case, they owned it, as though it was a hack and it was 100 per cent their responsibility.”

Page Break

Once engaged, King and his team worked with the Australian Red Cross Blood Service and, presumably, the organisation’s technology partner, Precedent, to resolve the breach and mop-up the potential fallout from the incident.

King and his team habitually take a six-phase incident response approach to such breaches. The first step is preparation, followed by identification, containment, eradication, recovery, and lessons learned

“You’re trying to lessen the chance that you will have a data breach or a cyber incident. But where companies generally fall down, is that they don’t plan for what happens afterwards; they don’t develop, as part of their preparation, an incident response plan,” King said.

“I’d say that for the vast majority of cyber incidents that we help manage, the companies don’t have an incident response plan. And even most of the ones that do [have one] don’t use it in an incident because it’s not usable. Situations in that case are often so fluid that the plans they’ve written really start coming apart.

“With mandatory breach notification coming in, that will be an issue,” he said.

In the case of the Red Cross Blood Service, it was the willingness of the organisation’s leadership to comprehend the situation, act quickly, and ask for help that got the ball rolling.

“She [Park] got it, and she owned it, and she understood how important this was,” King said. “They [the Red Cross Blood Service] stood up a business stream and a technical stream in terms of the crisis management, and they worked tirelessly.

“They let us in every meeting. They took the advice. At the end of the day they had to make the decision about what was right for them, but they took all our advice, and I’d say that is another very important thing,” he said.

After preparation, the next phase of the response was identification and analysis, according to King. For the Red Cross Blood Service, this step involved others, with Hunt and his anonymous source informing the organisation of the breach. This is not unusual, with a large proportion of breaches going unknown until a third party picks them up and reports them – or not.

“Troy [Hunt] disclosed to us very ethically and very early,” King said. “That was good start to a very bad situation. And we were able to get the offending website taken offline the same day, later that night we were able to get it down.”

“At the end of the day, that mattered. For Australia’s biggest incident so far, that made a material difference. That we could talk to someone in Australia who had the authority to make a decision, and took that website down for us,” he said.

Page Break

Once the website was taken offline King and his team were able to move from the identification phase to the containment phase of the incident response process.

Containment activities often start with the forensics, according to King, and involve response teams, war rooms, crisis centres and, importantly, media and communications activities.

From containment, King and his team moved into the eradication phase. This step was made more difficult than usual in some ways due to the severity of the breach, but easier in others as there were no adversaries to contend with – as is the case with many breaches resulting from malicious activities.

“Generally you’ll need to have an eradication strategy that you’ll need to come up with,” King said. “So you need to reimage and reinstall affected systems. But, depending on your supply chain, logistics, third parties, that is not as simple as it used to be; where can that data end up? Is it communicating via APIs? You’ve got to work all that out.”

This is where forensics come in handy, according to King, with the eradication phase often made more difficult if organisations don’t have a clear understanding of their digital assets. Fortunately, the Australian Red Cross Blood Service did.

Next came the recovery and remediation phase, which King suggests should leave an organisation in a stronger position than it was previously, in terms of its digital assets and data security regime.

“You need to recover to better than you were before, so you’re no longer at the same risk level,” said King. “It’s not an exact science, it’s mostly science, but it’s partly art, because these situations a very fluid.”

The final step of the process, lessons learned, is perhaps one of the most important phases, according to King. It is this step that helps lessen the chances of a similar event occurring again in the future.

“You need to do a post-incident review, and be honest in your appraisal of what went well, what went wrong, what can you put in place to reduce the risk of that happening again,” King said.

King also recommends that organisations create a culture where employees don’t have fears around failure, as it will encourage open communications about potential issues, meaning they are likely to be addressed early, before they turn into problems.

“If you have a culture of fail early, fail fast, or don’t fail alone. If you’re having issues, don’t hide them, let someone know,” he said. “That’s how you have to manage the incident. It’s about being open, honest, and transparent.”

It is, perhaps, this culture of openness that helped the Australian Red Cross Blood Service deal with the breach as effectively as it did.

Ultimately, the Australian Red Cross Blood Service said that, following its forensic investigation, it could confirm the relevant data was accessed by only one person, the anonymous source who subsequently shared the information with Hunt.

“Our investigation indicates that the copy of the data which had been accessed and all known copies of the data have been deleted,” Park said in a statement on 14 November last year.

“We are continuing to strengthen our protection of your personal information and over the coming weeks we will be announcing newer, stronger steps to enhance this,” she said.