It stands to change the outcome - for video gamers, heart attack victims and many others

Over the past three years, many businesses have shifted from thinking of big data as a challenge to perceiving it as the opportunity it is.

In order to better understand enterprise big data trends, Microsoft conducted a study among 280 top decision-makers last year. The findings were interesting in the sense that more than 38 percent of respondents' current data stores contained unstructured data (valuable, but unorganized information).

Nearly 53 percent of the respondents rated increased amounts of unstructured data to analyze as extremely important. This trend is only increasing as more businesses realize that unstructured data holds the key to new value not accessible in their existing structures.

There are two key measures to tapping into ultimate value in big data. The first is time to gain insight, and the second is return on accessible data. These measures are, in turn, enabled through a process of producing information, which means converting data or information from one domain into another.

Consider the following example. A fleet of ambulances is equipped with GPS units that collect telemetry. Information production techniques allow conversion of the raw GPS telemetry, a sequence of records containing time and location, into an incident response time.

The magic of information production is that it takes data difficult to deal with in traditional information systems, such as raw GPS telemetry, and transforms it into information, that is both more structured and more business relevant. Once the incident response time is produced, patient outcome can be predicted.

Great information production tools can also allow a reduction in the time to gain insight. They allow one to get from a hunch to validation quickly. In fact, there is an emerging class of these tools that can make great predictions by finding correlations in diverse data sets, which may hold the new untapped value.

Valuable answers require joining different data sets in logical ways. In traditional databases, what you would call "accessible" data is constrained to data contained in the database. This data has been normalized, cleaned and indexed so it can be used to efficiently answer a fixed set of questions over that data domain.

Big data and information production, on the other hand, enable a much larger definition of accessible data. Going back to the ambulance example, it could determine how many lives could be saved through shorter response time. By using accessible demographic and population data, one could determine how many heart attack victims could be saved by moving or adding ambulances.

We all know big data has become mainstream when it makes everyday experiences better. For example, by doing an analysis on unstructured data with big data tools, the Microsoft Halo 4 (video game) team gained insights on the use patterns and thereby made changes to further improve the overall gaming experience.

There is no doubt that we are all starting to benefit from big data. Companies like Microsoft are working to make sure that the future promise of big data is fulfilled in ways that we can now only scarcely imagine. It is focused on developing big data offerings that allow faster insight time and greater return on accessible data.

All this means we are in the midst of an incredibly exciting time, and the next decade offers even more promise.

The author is chief technology officer of the Cloud and Enterprise Engineering Group, Microsoft.

(China Daily European Weekly 04/11/2014 page9)