Big data - Part 2
- 20 September, 2011 10:12
binary data
Data memories
A second technology making a significant impact on solving Big Data problems is in-memory computing, which takes workloads that were traditionally resident on disk-based storage and moves them into main memory. This delivers a performance improvement many times above that which has been possible previously.
The beauty of these technologies, however, is that they enable organisations to analyse events in their transactional systems in real time, rather than having to extract and translate data into a relational data warehouse.
While Big Data is creating new opportunities for some businesses, it is also creating entirely new business models
Read Part 1 of Big Data.
The global general manager for business analytics at SAP, Steve Lucas, says the company has 25 active beta users on HANA today and has strong interest from close to another thousand organisations.
“We’re seeing some large consumer packaged goods companies that are looking to do massive analysis on very Big Data, like on-shelf availability and trade promotion effectiveness,” Lucas says. “And we are seeing things like financial services companies that want to do fraud analytics.
“There are companies looking at using HANA to do real time analysis on what TV shows consumers are watching. We’re seeing companies that are looking to do some unexpected types of activities with HANA.”
The advantage these companies are seeking is speed. In-memory technology and the database optimisation it enables allows organisations to perform analytics functions at speeds which are in some cases thousands of times greater than has been possibly previously. They can handle much larger and more complex data sets without losing the granularity of individual records.
While Big Data is creating new opportunities for some businesses, it is also creating entirely new business models. New companies such as US-based Infochimps and Factual have emerged to gather and provide access to numerous data sets (on both a paid and unpaid basis), in a model that is becoming known as data-as-a-service (DaaS).
Both are increasingly being used by developers to create new applications, particularly through calling in geolocation data held within their repositories.
Palantir Technologies is another US company specifically created to help organisations to integrate, visualise and analyse multiple data types, including structured, unstructured, relational, temporal, and geospatial data.
Some of the service providers are not so new, but Big Data tools are enabling them to become more sophisticated in what they offer. The Melbourne-based doubleIQ has been solving customers’ data problems for a decade by conforming and matching up data types for analysis, such as providing an integrated view of customer behaviour.
Cofounder, Dennis Claridge, says new possibilities are opening up from the volumes of data now available for processing. He says the channels for data have increased tenfold, but the volumes of data have increased a hundredfold, to the point where it might now be analysing billions of transactions.
Read more about Big Data.
“As a result of things being digitised there is more data that can be generated — a lot of Web data, such as Web logs and buying behaviour,” Claridge says. “There is a lot more transaction data and a lot more fragmentation in terms of channels that are being used.
“That’s great for us because it give us more and more opportunity to remix those, repackage those, and re-aggregate them back together to make sense of it.”
Also working in doubleIQ’s favour has been the plummeting cost of the hardware required for analytics, with Greenplum being run on commodity hardware for much of the data crunching. This is enabling doubleIQ to offer a wider range of service to a broader range of companies.
“Ten years ago the cost would have been prohibitive,” Claridge says. “It’s possible for us to develop quite a reasonable capability at a low cost now. Because there is more data available there are more possibilities across different industries, and more breadth of how it could be applied and used within a particular industry.”
Claridge says the long term vision is for doubleIQ to set itself up as a service, where it would host customer data in its own cloud and perform analysis, then publish access back to the client’s users. Should it be able to do so, Claridge says the company will one step further along in its goal of helping customers get maximum value from their data assets — something that many have failed to do.
“They don’t make as much use of their information as they could, and it gets lost over time,” Claridge says. “Data’s never valued correctly in an organisation.”
In the era of Big Data, that lost value is only likely to increase.
Read Part 1 of Big Data.
Recommended reading:
