The Big Data Boom
Big Data is nothing new; organisations have been gathering large amounts of information for decades. But it's only recently, thanks to advances in internet technology, that they have been able to really analyse it, and extract meaningful value from it.
Indeed, Big Data as we know it today is transforming the IT landscape. Where once only large multinational companies were able to take advantage of it, now even the smallest of businesses can use big data analysis to drive their decision making.
As a result, information has become one of the most valuable assets that an organisation has, which means managing information has become a major priority. That's why the Big Data trend has given rise to demand for a new generation of IT professionals with a very specific set of skills.
Roles such as data scientists, big data developers and data architects simply would not have existed ten or even five years ago, but now organisations are scrambling for the best talent. And because Big Data Analysis is such a recent concept, people with experience in this area can be difficult to find.
What is Big Data?
The term Big Data is used to describe massive amounts of unstructured data that are simply too large and too complex to be managed, manipulated or analysed using standard methods or tools. Back in 2001, IT industry analyst Doug Laney, who is now a vice president at Gartner Research, came up with a mainstream definition of big data as 'the three Vs'. These are volume, velocity and variety.
The first one is pretty self explanatory. Every minute of every hour of every day, more and more data is being generated, and so the volume of new information that is available for analysis is constantly growing. Of course, the old data also remains relevant as well and will for many years to come.
Previously, the challenge was how to store all of this data. Businesses needed huge servers and data centres on site, which were very costly. Today, new cloud-based methods of storage have brought costs down significantly and eliminated the need to house physical servers, so attention has turned to another challenge; how to use analytics to create value from this data.
Velocity refers to the speed at which data is being generated in today's super fast online world and the need to react quickly in order to harness it. Indeed, day-old data, while still entirely relevant, is not as valuable as seconds-old information, which makes data management a real-time activity.
As far as variety is concerned, in the past data tended to be very structured and came in the form of sales figures, financial details and inventories, for example. Today, there is a vast amount of unstructured data coming from a huge array of sources.
There are emails, videos, photographs, social media posts, mobile text messages, instant messages, word processing documents, audio files, web pages and presentations, to name but a few. It is estimated that between 80 and 90 per cent of the data in any organisation is unstructured, and it is thought that as social media and online sharing becomes even more prolific, the growth of this type of data could far outweigh that of structured data.
Why do businesses want to analyse it?
Information in itself is pretty worthless. To extract value from it, you need to analyse it. What good is a long list of social media posts or product reviews unless you take the time to examine them and identify trends or themes within them? By doing so, companies can make more informed decisions about how they run their business.
For example, a retailer may be able to use data on customer buying patterns to better target their marketing material or their promotions, while a bank might be able to offer targeted financial advice or make product recommendations to a customer based on data from his or her profile.
But it's not just about marketing. Big data analysis can also be useful in the development of new products and services or the redevelopment of existing ones, in identifying and managing risk and in boosting productivity by making better management decisions.
Effectively, Big Data analysis is about turning data into revenue by finding actionable insights, and gaining a competitive advantage over rivals in the process. There will always be room for instinct and intuition within business, but Big Data has removed some of the risks associated with following one's gut by allowing companies to make smarter, data-driven decisions that have a sound basis in reality.
The role of cloud computing
Big Data analysis was, in the past, an activity reserved for large organisations with the money and the physical space to house huge clusters of in-house servers. However, the growth of cloud computing has changed all that, and now businesses of all sizes can get involved.
It may be true to say that cloud computing has given rise to the growth of Big Data. It is effectively an enabler of Big Data storage and processing on a commercial scale, because it would be almost impossible for the majority of companies to collect, analyse and process big data sets without it.
Companies can not only store vast amounts of data in the cloud, but a new breed of online applications has allowed them to analyse their data in the cloud as well, without having to invest in expensive hardware.
Cloud analytics solutions work on a pay-per-use model, providing unlimited resources wherever and whenever they are required and allowing businesses to run query after query in real time without worrying about whether their servers can keep up with the demand. This has made big data analysis significantly cheaper and has provided much greater flexibility.
These applications use non-relational database technology that allows them to capture the kind of unstructured data that would not be possible with traditional relational databases. Prior to the growth of Big Data Analytics and cloud computing, businesses were only able to analyse a fraction of the data they owned. Now, they are able to utilize it all.
Who is benefiting from Big Data Analysis?
Big data analysis is having huge benefits for businesses in a vast range of sectors, and retail is most definitely one of them. Big supermarket chains, for example, are collating data using loyalty cards, which allows them to monitor buying habits and offer personalised promotions to their shoppers based on the products they purchase the most.
They are also making merchandising decisions based on information about buying habits throughout the year and in specific locations, and making product recommendations based on data about what customers with similar profiles have purchased.
Financial institutions are using big data to their advantage as well, monitoring customer spending and saving habits to develop new products and personalise their services, creating customer profiles that allow call centre staff to respond quickly to individual problems or requests, and managing risk by identifying customers who are likely to default and pinpointing instances of fraud.
They are even using big data analysis to drive efficiencies in activities such as cash machine reloading; making sure that ATMs do not run out of money in those crucial peak periods, that reloading is carried out when it is likely to be least disruptive to customers and that unnecessary reloading is eliminated.
But it is not just customer-facing institutions that are benefiting from Big Data. Pharmaceutical firms are also tapping into it to understand prescribing behaviour and drive sales, to link patient genotypes to clinical trial results in order to personalise drug development, and to improve safety by comparing clinical data with trials of similar agents to identify possible negative effects on patient populations, for example.
There are many other uses for Big Data analysis within these sectors, and of course in others. Government agencies are using it, healthcare companies are using it, manufacturers are using it, telecommunications firms are using it, utilities companies are using it and it is even finding uses in sectors such as agriculture.
Demand for analytics experts
Cloud computing may have brought Big Data Analytics to the masses, but harnessing the power of Big Data is not quite as simple as downloading a cloud-based application and getting started. It takes people with a specific set of skills to manipulate the data and extract meaningful insights from it.
Of course, Big Data Analysis is a relatively new phenomenon, so people with these skills as well as the experience to back them up can be difficult to find. It's why there is such fierce demand around the world for talented data scientists, analysts and architects.
And it is not just technical skills and experience that are required to be a successful big data analyst. As Richard Rodts, manager of global analytics academic programmes at IBM, points out, there are some important soft skills that are needed, too.
"There are the very human attributes, such as a knack for both strategic and creative thinking, the ability to collaborate with colleagues across the business, and strong communication skills that enable you to convey data-driven findings to senior decision-makers in a compelling way," he told IEEE-USA Today's Engineer Online.
In 2012, technology research firm Gartner predicted that by 2015 some 4.4 million IT jobs would have been created around the world to support the growth of big data. However, it warned that only around a third of these vacancies would be filled, as supply simply cannot keep up with demand.
"Data experts will be a scare, valuable commodity," said Peter Sondergaard, senior vice president and global head of research at the group. "IT leaders will need immediate focus on how their organisation develops and attracts the skills required. These jobs will be needed to grow your business. These jobs are the future of the new information economy."