IIIS: Big Data driving new trends

Steve Duplessie

Reality mining, tagging data at its source and Cloud reputation agencies are three of the emerging trends likely to come out of the explosion of ‘Big Data’ delegates at the Implementing Information Infrastructure Symposium (IIIS) have heard.

See our preview of Ross Dawson’s IIIS presentation

Speaking at the event, co-hosted by Storage Networking Industry Association A/NZ and Computerworld Australia, strategy advisor, author and futurist, Ross Dawson, said “reality mining” — the gathering of data based on the activities of people in a given environment — was a major trend to emerge out of, and contributor to, Big Data.

“If you look at an office environment there is an extraordinary amount of data to look at. For example, what gestures people are making, where are they looking, what conversations are they having, how much are they smiling when they speak to each other?” he said.

“You can literally get terabytes of data out of just a few hours of this. That data is being collected to drive productivity; to design new ways to enhance collaboration and create value inside organisations.”

Dawson said an added emphasis for this type of data collection was research suggesting that organisations which used data-based decision making had been able to achieve 5 to 6 per cent greater productivity.

“That requires this kind of reality mining,” he said. “This requires all that data within the organisation to be captured and used effectively.”

Read more on the IIIS event

To enable effective data capture organisations would also have to transition away from the mass collection of unstructured data to a process where all data was tagged at the point of capture.

“The vast majority of the information we have is unstructured… so what is critical is tagging data at its source,” he said. “In this explosion of data… the key thing is to add structure to it by tagging at the source as the cost of doing it after vastly more.”

Dawson said video was a good example of this, where not only the speakers in the video, but the themes and even emotions of the speakers could be tagged and used later in reality mining exercises.

Such tagging work would likely be carried out by machines but also by services such as Amazon’s Mechanical Turk which offers an on-demand workforce to “build human intelligence” directly into applications.

With the security of the public Cloud being a major inhibitor to adoption, Dawson said Cloud ratings agencies would likely emerge that would rate the security of Cloud providers in a similar to financial ratings agencies Standard & Poor’s and Moody’s.

“This goes alongside the ‘reputation economy’. This decade is the one where reputation is driving almost everything we do — reputation of individuals, people’s influence on social media and the like,” he said.

Read Big Data - Part 1.

“So for companies, being in the Cloud doesn’t automatically mean you are reputable. We need the creation of organisations like S&P and Moody’s rate institutions in the Cloud."

As well as Cloud ratings agencies, companies which offered services such as Cloud-based distributed data bases, and Cloud brokers — organisations trading storage and processing power on spot markets — will emerge.

Flash storage to make major inroads in the next year

Also speaking at the event, Steve Duplessie, founder and senior analyst, Enterprise Strategy Group, said flash-based storage would make major inroads within the next 12 months — largely due to the increased input/output loads caused by virtualisation.

“For the last 15 years just about everything in storage has been just about fast enough and that hasn’t been a problem till now, where most people are at least 40 per cent virtualised in their server rooms,” he said. “Server virtualisation all of a sudden exposes I/O problems we didn’t know we had.”

“We are now not that far away from you not buying disk drives anymore. You certainly won’t be buying 1500rpm fibre-channel disk drives as they are pointless. Everything in your data centre is solid-state — except disk drives. That is a fascinating.”

The “only” reason for organisations not to buy solid state was an economic one, Duplessie said.

“If I tell you that you can buy $1000 a terabyte of mechanical or $1000 a terabyte of solid state you will probably buy that solid state,” he said. “Sooner or later that will be the case.”

Data backup was arguably one of the most problematic areas of storage, largely due to organisations placing the same value of every piece of data over time.

“We put a man on the moon with the effective computer power of less than your wristwatch and got them back — we figured that out — but we can’t figure out backup,” Duplessie said.

Follow Tim Lohman on Twitter: @Tlohman

Follow Computerworld Australia on Twitter: @ComputerworldAU