Firefighters at CFA Victoria deployed content delivery management (CDM) software during the recent Victorian bushfires as an influx of residents using a mobile app that provides fire warnings caused its website to fail.
CFA is a first responder and emergency management organisation funded by the state government in Victoria. It has 61,000 people, including 59,000 volunteers, operating out of 1253 fire stations around Victoria. Bushfires are just one area in CFA’s jurisdiction; it is also a first responder for road rescue, hazardous materials and other emergencies.
IT’s role at CFA is to communicate information to both internal operations people and the general public who the organisation protects, CFA executive manager, ICT services, Michael Foreshew told Computerworld Australia.
This includes informing fire fighters how best to extinguish a bushfire and sending warnings to the public about potential danger areas.
CFA used to rely on mainstream media to communicate warnings to the general public, but about three years ago began using Internet technologies to directly warn citizens abut emergencies, Foreshew said.
CFA has a system called “One Source, One Message” that allows CFA officials in the field to produce alerts that are automatically and simultaneously sent to the CFA website, Twitter, Facebook, mainstream media and the federal SMS alert system.
CFA also produces RSS and XML data feeds that can be read by mobile apps and other readers. Apps have been developed for Android, Apple, Windows Phone and BlackBerry mobile operating systems.
On 4 January with bushfires raging across Victoria, one mobile app called FireReady “bombarded our website ... with so much traffic that the website would time out waiting for the data to come back,” he said.
“We initially thought it was a denial of service attack,” Foreshew said. “However, none of the perimeter security went off.”
Eventually, CFA discovered that a poorly designed FireReady app was the source of the problems.
The app drew “a lot of data every 60 seconds,” he said. “It doesn’t just take the changes; it reloads everything.”
In addition, “rather than query our proxy servers, which had been set up to handle this ... it forced our proxies to go back and request the data” from the database.”
The inefficiencies combined with 417,000 users using that app at once crippled the CFA website.
“The website was still producing the pages,” but anybody who tried to access the website received server timeout errors, he said. “What was then happening is people grew frustrated with the website, so they downloaded the app. It was a compounding problem.”
After identifying the app was the problem, CFA engaged Internet content delivery management software vendor Akamai, a company it had already been talking to about another initiative, Foreshew said. On 9 January, Akamai presented a possible solution on and deployed it in about four hours, “free of charge,” he said.
The software quickly resolved the network congestion, and CFA this week signed a one-year contract with Akamai after navigating a government review, said Foreshew.
At the peak of the emergency, CFA served 1.9TB of data in one day, Foreshew said. CFA’s 1Gbps link was running at about 650Mbps to 670Mbps trying to serve the large volume of data, he said.
However, after about 15 minutes with the Akamai service, “we were down below 13Mbps and we haven’t gone above that ever since,” he said.
CFA had to find money and satisfy procurement guidelines before signing the contract, Foreshew said. “CFA and a number of government organisations are facing reduced budgets, so we needed to reprioritise.”
The Akamai CDM was attractively priced and “provided a large number of servers across a variety of locations,” Foreshew said. Alternative CDM providers presented a smaller number of larger capacity servers in a centralised location, he said. CFA decided that the Akamai CDM provided better redundancy.
Akamai “properly” addressed cloud provisioning, he added. “What we saw in some of the other providers were outsourcers trying to buy their way through an inflection point of technology.”
Meanwhile, CFA is redesigning the mobile app that caused the traffic congestion in the first place. It is also exploring alternative ways to deliver information to mobile devices, Foreshew said.
“We’ve seen advancements in the use of HTML5, for example,” which could allow CFA to consolidate its mobile delivery rather than have a separate app for each mobile OS, he said.
“There are no sacred cows,” he said. “We [will] have a look at what is the most appropriate, most efficient, most effective way of getting the information out to people.”
Follow Adam Bender on Twitter: @WatchAdam
Follow Computerworld Australia on Twitter: @ComputerworldAU, or take part in the Computerworld conversation on LinkedIn: Computerworld Australia
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.