Big data analytics today lets businesses play Moneyball

Hype continues to surround big data analytics. But hype or not, data-driven decision-making is becoming central to management decisions at most enterprises. They are taking inspiration from Billy Beane, the general manager of the Oakland Athletics baseball team that turned to data analytics (known as Sabermetrics in baseball and made popular in the movie, "Moneyball") to assemble a competitive baseball team despite the organization's limited financial resources.

Today, enterprises are no longer merely trying to get their heads around what the term big data means; the overwhelming number of enterprises are now seeking to leverage data insights and adopt data-driven processes.

A recent study by the Economist Intelligence Unit, commissioned by Capgemini, found that two-thirds of respondents--607 executives (43 percent of them C-level and board executives) hailing from 20 different industries across the globe--say that the collection and analysis of data underpins their firm's business strategy and day-to-day decision making. In fact, just more than half say that management decisions based purely on intuition or experience are regarded as suspect.

Another recent study by IT consulting firm Avanade, a subsidiary of Accenture, backs up those findings. Avanade's study of 569 C-level executives, business unit leaders and IT decision-makers in 18 countries found that 91 percent of companies are already using tools to manage and analyze data, beginning with data storage, reporting, data integration and enterprise search. A majority of companies (73 percent) have leveraged data to increase revenue. Fifty-seven percent of that subset used data to increase an existing revenue stream, while the remaining 43 percent used data to create entirely new sources of revenue.

Move to a Data-Driven Culture Is Underway

Avanade's study also shows a cultural shift occurring. Data management and analysis were once considered IT jobs. But that's changing quickly. Today, 58 percent of respondents say that data management is now embedded throughout business operations. And 95 percent of businesses with dedicated business analysts do not consider data analysts a part of their IT staff. Instead, enterprises are distributing that expertise to line-of-business groups throughout the company. And 59 percent of global companies also say that more employees than ever before are involved in making decisions as a result of more widely available company data.

Companies that focus on data analysis skills are having the most success transforming their data into revenue: 88 percent of enterprises with dedicated data analysts have turned data into revenue, compared with 49 percent of companies without dedicated data analysts.

Do Big Data Analytics Mean a Competitive Edge?

In Moneyball: The Art of Winning an Unfair Game, author Michael Lewis lays out how Billy Beane used statistics and analytics to turn the Oakland Athletics around. But that's just one data point. The elephant in the room when it comes to big data analytics has always been: Do companies that base their decisions on data and analytics actually perform better than their competitors?

The answer appears to be 'Yes.'

In 2011, Prof. Erik Brynjolfsson, an economist at the Sloan School of Management at the Massachusetts Institute of Technology (MIT), set out to find the answer to this question.

"Companies that use this type of 'data-driven decision making' actually show higher performance," Brynjolfsson wrote. "Working with Lorin Hitt and Heekyung Kim, I analyzed 179 large publicly traded firms and found that the ones that adopted this method are about 5 percent more productive and profitable than their competitors.

Furthermore, the study found a relationship between this method and other performance measures such as asset utilization, return on equity and market value. There is a lot of low-hanging fruit for companies that are able to use big data to their advantage."

Enterprises are taking those findings to heart. Avanade's study found that 57 percent of respondents say their firms added technology options for analyzing and managing data in the past 12 months. A further 75 percent say their company will make additional investments to improve their capability to analyze data within the next 12 months.

"Big data is reaching a tipping point where it is becoming much more mainstream," says Steve Palmer, Business Intelligence Leader for North America at Avanade. "We are beginning to see companies on a much broader basis derive both financial and competitive advantage from data."

Obstacles to Managing and Analyzing Data

That said, there's a long way to go. While the majority of firms say they are leveraging their data, 85 percent of Avanade's respondents say they face obstacles in managing and analyzing data. These obstacles range from the sheer volume of data to data security to a lack of dedicated staff.

The Economist Business Unit and Capgemini found that organizations are struggling with the enormous volumes of data coming in. But perhaps more important, they are struggling with poor quality data and siloed data.

"It is the complexity of the data they have, not the sheer volume, that is really slowing down their ability to make decisions," says Scott Schlesinger, vice president and head of Business Information Management for Capgemini U.S. "You've got to get your arms around data quality. Garbage in is garbage out."

While the majority of respondents in the study say their firms are data-driven, they also say their organizations do not have enough of a 'big data culture': 55 percent say that big data management is not viewed strategically at senior levels of the organization.

When it comes to these obstacles, there is a split between C-level executives and IT decision-makers about which constitute the biggest hurdles. The majority of C-level executives (58 percent) believe finding the right technology is a bigger challenge than finding the right staff. However, 56 percent of IT decision-makers disagree: They believe finding the right staff is a bigger challenge than finding the right technology.

"Big data is not new," Schlesinger says. "It's a new term. It's a problem that's becoming more and more prevalent, not due to the volume of data but to the new types of data that we're facing. Unless we face it from the people, process and technology perspectives, we won't be able to get our arms around it."

"Throwing technology at it is just not the answer," he adds. "Technology is a component of it."

How to Approach Big Data

To better make the transition to a data-driven culture, Avanade recommends taking what it calls the MORE approach:

• Merge. To squeeze the value out of your data, you need to merge data from multiple sources-for instance bringing together structured data like customer information from your CRM with unstructured data like customer sentiment data from social news feeds to gain a more holistic understanding of how customers are responding to your product. The challenge here is in understanding which data to bring together to provide the actionable intelligence you need.

• Optimize. Not all data is good data. Making decisions faster using data-driven processes is an excellent goal, but if you start with bad data, you'll just be making bad decisions faster. You need to identify, select and capture the optimal data set to make the decisions you need to make. This involves framing the right questions and utilizing the right tools and processes.

• Respond. Having data and acting on it in a meaningful way are two different things. You need to have the proper reporting tools in place to surface the right information to the people who need it, and those people then need the processes and tools to take action on their insights.

• Empower. Data can't be locked in silos and it can be the province of a select few. You need to train your staff to recognize and act on big data insights.

Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at tolavsrud@cio.com