"Performance testing is a best guess at what you think is going to happen and the real world always manages to blindside you": Yarra Valley Water's CIO, Leigh Berrell.
In early 2014, Yarra Valley Water was a victim of its own success.
Melbourne’s largest water and sanitation organisation had just rolled out an information exchange application – integrated with several core systems and databases – that enables builders, plumbers, conveyancers and suppliers to complete online transactions.
Its EasyACCESS Portal uses a rules-based engine to process around 35 different types of transactions from a water service request to providing an information statement used by a solicitor for conveyancing purposes. The system reduces the time it takes to complete transactions from days and weeks to minutes and hours by eliminating manual paperwork.
But from the outset, Yarra Valley Water, which services 1.7 million people and 50,000 businesses, had a problem. The high number of users accessing the system and various mix of transactions – which pulled information from databases and processed them through middleware – caused a number of service outages.
Leigh Berrell, CIO and GM of business technology services at Yarra Valley Water tells CIO that system issues were mainly buried in the SOA [service-oriented architecture] and the middleware layer, which made it difficult for technical staff to identify the root cause.
“We got a lot more users than we expected and as the database started to fill, we hit performance issues,” says Berrell.
“The system fell over a number of times once the load started to build. We ended up with a transaction backlog that wasn't making anyone happy. We had to clear a lot of those transactions manually and the customers were being impacted by that.
“Things were looking a bit gloomy. After a very successful launch, we were struggling after the first month,” Berrell says.
The applications also had some unexpected performance issues, despite substantial testing prior to its installation.
“Performance testing is a best guess at what you think is going to happen and the real world always manages to blindside you,” he says. “It wasn't a matter of just throttling the application back, cutting half the customers off and it would go away,” he said.
In March 2014, the company approached several vendors in the application performance management market before eventually selecting software developed by US firm, Dynatrace.
“Within a couple of hours, the Dynatrace system had identified the biggest bottleneck – the way an SQL query was built with some of the middleware layers. It was pretty innocuous when you looked at it,” said Berrell.
Software agents at the network and database layers gather transaction load and timing information, which Berrell and his IT team can view on a dashboard.
Read more: Yarra Valley Water announces email upgrade to improve its disaster recovery
“To try and pull that [information] together manually, even with the best practitioners, is a really difficult thing to do.
“You can filter by transaction type, for instance, so it can go through and say ‘show me what happened with this particular transaction?’ Straight away you can see that right there in that piece of middleware and down to that SQL call is where everything fell in a heap and the extra 15 seconds came into this transaction,” he said.
By April last year, the software has helped the organisation solve its major issues and smaller problems that could have caused issues with transaction load increases. Over the past year, ‘hundreds of thousands’ of transactions have run through the EasyACCESS Portal.
“We’ve used it to proactively go through and solve integration issues, restructure some indexes … and identify the things we need to focus on,” he said.
Read more: Victorian tax agency taps Oracle in refresh
Other projects
Last September, Yarra Valley Water launched a new asset management system to track items such as pipes and pumps. It rolled out the IBM Maximo Application Management software which was heavily integration with the company’s GIS, SCADA, billing and property systems, and the EasyACCESS Portal.
“We couldn't afford to have a performance issue with that – there’s no going back when you transition from one asset system to another. It was a ‘big bang’ transition as well.
“We used Dynatrace extensively before it went live to look at all the transaction types. We used it when we were running all our performance tests. It did actually find – despite the best efforts of our programmers, developers and designers – things that may have been a problem and resolved them,” says Berrell.
On April 21, the company also launched a service portal that enables its water and sewage customers to log in, pay bills, ask for concessions and payment extensions, and see if there are outages in their local areas.
Again, the company used the software to search transactions and make sure there weren't any ‘time bombs' present, says Berrell.
The customer service portal was created to provide an alternative platform for customers to interact with the company.
“There’s a millennial generation in the population that is taking out water accounts, they are tenants in houses … and they've got an expectation that they can deal with us online.
“There are also certain transactions that it just makes no sense for a customer to talk to a human being to achieve. For example, if they want a payment extension – it’s inconvenient to pick up the phone and talk to someone when they can have it sorted in 30 seconds with a click,” he said.
This also frees up contact centre staff to deal with customer enquiries that need a 'personal touch', Berrell says.
Follow CIO Australia on Twitter and Like us on Facebook… Twitter: @CIO_Australia, Facebook: CIO Australia, or take part in the CIO conversation on LinkedIn: CIO Australia
Follow Byron Connolly on Twitter:@ByronConnolly
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.