Managing Information in the Age of Big Data

The Big Data revolution taking place across many industry sectors is presenting challenges to business managers in traditional line functions like production and marketing, and to corporate IT professionals alike. Big Data goes to the heart of the IT professional’s domain – the way that the enterprise warehouses and manages information – and potentially upends long-established systems and practices. At the same time it presents business manages with a new level of urgency to incorporating deep data analytics into their daily array of tactical business decisions. As important as it has always been for business and technology professionals to be on the same page, it is now more important than ever. Both sides also need to understand the specific concerns and challenges that are keeping the other up at night, and work together to solve them and identify critical gaps.

Not Just Rows and Columns Anymore

The world of the corporate IT professional has long been an orderly environment of rows and columns – the architectural foundation of relational databases containing daily transaction activity. This is not to say that the job was ever easy – there are many things that can compromise the integrity and accuracy of the information residing in those rows and columns. But at least the mission was clear – figure out the best way to warehouse the data, scale the size of the warehouse to accommodate the growth in data volume, and make clean, usable data available to the business user community every day. The key to this was structure – the data in this closed, managed environment were structured and supported by the technology systems in place. That structure is no longer applicable to all of the data coming into the company’s technology environment every day – in fact, a large part of the growth in data volume is coming from other sources that are decidedly unstructured.

Rise of the Machines

What is coming into corporate environments with a higher frequency now are open source data in many new formats that evolve rapidly and scale even more rapidly. In 2011 the amount of data created and replicated around the world exceeded 1.8 zettabytes – that’s 1.8 trillion gigabytes, representing a growth factor of nine times over the previous five years. The really daunting challenge facing IT professionals as they contemplate this staggering growth rate of information is that most of the growth is coming from various types of machine generated data (MGD), i.e. data generated automatically by an independent computational agent that is not caused by human action. Clickstream logs, social media, stock exchange trading data are all examples of MGD.

Chances are that sources of MGD data exist in just about every corner of major US industry sector. Where do all these types of data come from? According to a 2011 white paper by telecommunications company Ericsson, there are expected to be over 50 billion connected devices in the world by 2021, each of which will be continually emitting data-rich signals. Buried within these seemingly unintelligible signals are data about consumer behavior, location, transportation methods and other insights that can provide business managers with a fuller, deeper picture of their demand environments. The challenge of managing this information goes well beyond traditional challenge of uploading and warehousing the data in the most optimal way for the business users to extract and analyze them.

It’s Not the Data, It’s What You Do With the Data

Making sense of this tidal wave of multi-source and multi-format data will be challenging; yet there are some basic rules business managers can follow to keep themselves from becoming overwhelmed. Any data – whether from a traditional proprietary transaction database or from an RFID scanner or from social network weblog files – are useful only to the extent that human intelligence enters the picture and asks the right questions from which to derive actionable insights from them. For example, your raw data may tell you that one of your customers – a restaurant that buys food products from you – tends to buy a preponderance of raw ingredients while another customer – a restaurant of a similar type – buys mostly prepared foods. That basic information leads to a derived data point that you can call something like “chef skill level”, the idea being that the restaurant offering menu items prepared from scratch places a comparatively high premium on the quality of their dishes and invests in chefs of commensurate experience. That data point in turn can lead to other predictive insights, such as which of these two customers would be more receptive to a particular product promotion or a loyalty incentive discount.

Source of Competitive Advantage?

Can information management practices in the age of Big Data offer a path to competitive advantage? There’s plenty of reason to be skeptical – there is no shortage of historical cases where emerging technologies have been trumpeted as the Holy Grail, and the vast majority of such claims have failed to ring true. Big Data can sound like little more than hype – but there are stronger reasons to argue that it is real, and here to stay. Achieving sustainable advantage for an enterprise requires an intelligent approach to managing and using information based on the following:

• Coordination between suppliers, distributors and retail operators to reduce waste along their demand chain and realize opportunities for mutual profit;
• Connection to all the new sources of data emerging into the ecosystem, and the ability to take action on insights from the data in real time;
• Customization of offers that provide the right product (or bundle of products) to the right customer at the right time, at a price that maximizes the seller’s profit while retaining the buyer’s loyalty.

Whether one believes that Big Data represents an evolution or a revolution, it is still new enough to be a large question mark in the minds of many business decision makers. The opportunity is present for visionary enterprises to establish a market driving position; but that opportunity won’t last forever.