Credit: Dreamstime
The Australian Securities Exchange (ASX) has produced a post-mortem report on a hardware failure which temporarily halted trading on Monday night and early Tuesday morning.
The ASX described the failure – in the main database used by its equities trading system, ASX Trade – as ‘complex and unprecedented’ which took time to investigate and involved input from third party vendors.
In its report, the ASX said problems with hardware do happen from time to time but don’t typically disrupt the market because the organisation has full system replication in a backup data centre (its DRS site). It also has automated failure processes that enable faulty hardware within its primary data centre to be replaced.
“ASX regularly tests this failover, including testing hardware components every time there is an upgrade to ASX Trade,” the report said.
“ASX conducts business continuity tests with industry participants and includes testing failover processes. This was last performed and successfully completed on 27 July 2016.”
On Monday, an ‘unprecedented’ hardware function triggered a complete database failover to DRS and not all parts of ASX Trade successfully connected to the DRS database, the report said.
As a result, ASX could not open the market until 11:30am and took action to close the market at 2:05pm.”
The broken database hardware failed over to the backup data centre correctly and most part of ASX Trade failed over and connected to the new database, the ASX explained.
“However, some parts remained connected to the original database, thinking it was operating normally. ASX operators were originally unaware of this anomaly. This issue was identified before the normal market open and most of the consequences were manually rectified. The market was open again at 11:30am,” the ASX said in its report.
At 1:42pm, a reference data inconsistency emerged for a group of securities and as a result, the system stopped functioning properly for those securities, the ASX said.
“After investigation it became apparent that the data inconsistency had been caused by the original hardware malfunction and partial failover. ASX halted the market in a controlled manner,” the report said.
The failed hardware was replaced overnight, ASX Trade was reset to its standard operating configuration, and the system was validated as being ready for market open.
The ASX and its technology suppliers have commenced a detailed investigation to identify why the hardware malfunction caused the failover issues.
“At this stage, neither ASX nor its vendors can confirm having ever seen this malfunction before. ASX has updated its procedures to address the unprecedented scenario were it to recur,” the report said.
ASX CEO, Dominic Stevens said in a statement that the outage to ASX trade last Monday was unacceptable to the market and unacceptable to the ASX.
"I offer my apologies for the disruption caused,” he said.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.