Why big data is big business

The big data hype cycle is well and truly over, and CIOs need to lead big data projects or risk getting left behind by competitors that are gaining business advantage from analysing the goldmine of data that lives in databases and on the corporate network.

IT leaders recently gathered for a roundtable luncheon in Melbourne to discuss what big data means to their organisations, and the strategies they are using to analyse information to improve decision making. The luncheon was sponsored by Tableau Software.

One CIO says his organisation had mastered traditional transaction-style reporting, becoming highly proficient using lag indicators to forecast future performance, direct investment, and drive efficiency improvements.

“In the last 12 months, there has been a strong shift towards ‘scenario and what if-based’ analysis of data with a heavy leaning toward visualisation as opposed to traditional transactional and summary-style reporting,” he says.

“The degree of sophistication available today in the reporting and visualisation suites and the ease with which our business users can aggregate disparate data sources, is fuelling this evolving trend.”

Jeremy Bree, CIO, at Henley Properties, said the organisation is using tools to create a more graphical and visual view of sales data that is now providing sales managers with a better understanding of leads, profitability of each sale, and sale types per region.

“This allows a better focus of time and effort into customer engagements that are more likely to be successful, and allows us to analyse future products that we put into different markets,” he says.

Anthony Russo, senior manager, technology and security, at Open Universities Australia, says exploring data to gain insights is nothing new and there are copious tools available to analyse data. Each type of data set can give an insight into a particular aspect of the customer and hence, potentially open up competitive advantage, he says.

However, he says the difficulty is in finding the right tool and attracting the expertise to process the huge volumes of data sets to form a holistic view of the customer, which is useful to the business.

“So data is not the issue, it is how to view the data and the expertise to know how to interpret the data,” he says.

Overcoming big data challenges

Australian CIOs have reported that big data projects are being held up by the high cost of setting up infrastructure to support the capturing of potentially hundreds of millions of data points each day. Lack of a business case and the need to integrate many data sources were also holding back adoption.

Jan Spinder, CIO at the Royal Australasian College of Surgeons, agrees that the high cost of setting up infrastructure to support big data projects is slowing adoption as return on investment is not necessarily understood up front.

“Through the data insight, great understanding is available on core business figures, and as a result, ROI can be measured. But building the business case with up front ROI is harder and is slowing down acceptance.

“Complexity of the number of data sources and their interconnection is slowing down adoption as well,” he says.

Open Universities Australia’s Russo adds that cost would have been a huge blocker in the past but infrastructure is now cloud-based so it easier to configure and run when doing big data projects.

“Collecting data and storing it is not an issue anymore,” he says.

Alan Eldridge, sales consulting manager, at Tableau Software, says rather than create a ‘big-bang’ project to overhaul the entire data architecture at once, CIOs should spin off ‘proof of concepts’ with each business case.

By taking an agile, incremental approach, driven by specific use of cases, CIOs can get value as they inform later projects.

He says that investment in new infrastructure creates the problem of integrating data sources but there are several approaches to overcome this issue. Many companies integrate at the analytical layer, he says, which means they look for an analytics approach that lets them use data from disparate sources.

“So instead of trying how to figure out how to get their Hadoop data into their enterprise data warehouse, they leave the data where it is and simply analyse one or both data sets as needed,” he says. “This has the advantage of letting different types of infrastructure handle the workloads they are best suited to.”

Another approach is to use workflows to selectively bring data together into a data warehouse,” he says.

“An iterative approach here is helpful as well,” he says. “Use analysis to identify key data that should be combined, and bring it into the same environment selectively,” he advises.

“There may not ultimately be a reason to combine all the data in one place, as long as you can answer the questions that you need to answer,” he said.

One CIO says his business does not rely on collecting huge volumes of data points at this time. Rather than the main challenge is blending captured data to gain insights into behaviour and to challenge existing theories.

“There’s plenty of data to support ‘what’ happened, it is when the question becomes ‘how’, that visualisation and the ability to interrogate our data in a very interactive manner, becomes most relevant,” he says.

He notes that without a solid business case, organisations run the risk of creating more ‘noise’ with data rather than gaining insight from it.

“The density of the business case is a result of the expertise within the business,” he says. “Without existing theories and direct questions coming for experts within the business, it is easy to be lead astray and answer questions nobody is asking or that don’t help the business grow and learn.”

Bridging the gap

Roundtable attendees agreed that bridging the gap between their department’s focus on systems and applications and the need for executives to gain almost instant access to critical information.

This has been an issue for some time, says Open Universities Australia’s Russo, as instant access means huge overheads on systems.

“We are moving all services to an Amazon cloud to [deal with] scalability issues,” says Russo. “We have already been able to combine multiple data sets to form dashboards, which holistically track business health and performance,” he says. “Providing access to easy to understand dashboards improves business transparency, which aids in better decision making.”

Alex Watson, general manager, IT applications at Lonsec Fiscal, says his organisation has made good progress over the past 12 months streamlining and automating many manual processes. This has made more accurate and timely data available to users.

“This is a critical dependency that we still need to complete before we can build an electronic view of the state of our business,” he says.

“The second stage of this plan it to ensure that business users can self-serve the data and reports that they need without IT needing to be involved. The final state would be to create executive dashboards and reports, but executives themselves have typically de-prioritised this to focus resources on other business initiatives,” he says.

Bryce Turner, enterprise account manager, at Tableau Software, believes the key to bridging this gap is to be able to unlock the value from each system independently rather than requiring all data to be centralised and conformed before it can be used.

“Drawing data from each system through existing interfaces – either by directly querying the data in-place or consuming file extracts – and blending these data feeds together in an agile fashion helps executives address the need for immediate information.

“Over time, these information flows can be formalised and made more robust as the IT team becomes involved. This approach also helps the IT team prioritise as they can direct their efforts to those reports and dashboards that have proven themselves to be most useful to the business,” he says.