Skills required
The skill sets required to manage big data are less typical of our preconceived image of what an IT professional should be. While there is still an element of technical acumen needed, the actual tasks involved in measuring and interpreting unstructured data are more suited to people skilled in mathematics and physics. It is more scientific than technical, giving rise to such titles as data scientist, data modeller or analytician.
In an attempt to accelerate the availability of these skill sets in the market, many vendors are incubating universities with programs and courses to attract students to big data as a lucrative career option, but it is yet to be seen how quickly supply will meet demand in a landscape that is becoming more competitive.
Furthermore, people who have these skill sets or come out of universities where these skills have been honed to perfection will be resources in high demand. We are more likely to see these people make a name for themselves through new start-ups in the style of industry luminaries such as Gates, Jobs, Ellison and Page.
This raises a very interesting scenario for end-user organisations. If skilled graduates are absorbed by the vendor community or go into business for themselves, this leaves a significant skills gap for end-user organisations to fill; and until they are able to do so, they face the very real possibility of costly consulting engagements in their big data projects. End-user organisations will be challenged to circumvent this scenario with strategies to attract skilled professionals.
Big data has been correctly labelled a disruptive force in IT as it caught the market by surprised and left organisations flummoxed about how to deal with it. No one could have anticipated that the third platform technologies they had adopted as a way of doing business would result in the Big Bang that created the Digital Universe.
A sense of ownership
Discussion about big data continues as technology vendors vie with one another to “own” the concept of big data by putting certain exclusivity to it through some cutting edge technology in attempts to harness its commercial potential. This is not necessarily a bad thing.
The phenomenon of big data, along with other disruptive forces such as cloud computing, mobility and social media, has fuelled a new renaissance in innovation and technological advancement that has implications far beyond the corporate sphere.
The discussion around big data has led to a certain air of expectation. The Australian market was initially seen as slow in adopting big data technologies, but this was fuelled by a misconception that big data was only about volume.
As yet, most Australian organisations do not have anywhere near the hundreds of terabytes of data typical of organisations in Europe and the US. The focus for Australian organisations revolves more around the variety of data they collect and what use they put it to.
There are many in the market and in the industry that dismiss big data as mere hype. However, this does not make the reality of the situation go away. We are still faced with a growing volume and variety of data at a pace never experienced before.
Where technology has traditionally been able to evolve to meet the demands of the collection and analysis of information, the data deluge we are seeing currently is like a tsunami we were unprepared for.
It is not difficult to see how hype and misconception of big data have led some to become disillusioned with the whole concept. I would suggest however, that the disillusionment is only a result of not being able to see its full potential.
There are already examples of it being used in many instances outside the corporate sphere such as crime prevention, where law enforcement agencies are able to identify crime hotspots through crowd sourcing and take pre-emptive action that in the end contributes towards a better quality of life in the neighbourhood.
While this is just one example, there are many other use cases for big data that would have a tremendous impact on the quality and indeed the value of human life. The digital universe is constantly expanding and if not already, will soon hold the formula to eradicate terminal disease; and to predict weather pattern that lead to drought and famine; and to identify and trace the root of terrorism.
While there still may be a way to go for big data to solve many of the world’s problems, it is the individual, through the use of third platform technologies, that gave rise to big data. And it will be the individual in the form of the data scientist, the data modeller and analytician that drives innovation in the footsteps of Herman Hollerith.
Big data is a relative term. It has come full circle and is likely to do so many more times yet.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.