Can Dark Data Become a Part of Companies’ BI Efforts? by Dan Radak

Written by Dan Radak.

A few years ago, people from Gartner coined the term dark data to denote something all business intelligence experts have had experience with – vast amounts of data that are saved and stored without anyone even considering utilizing them. IBM has since come forward with a mind-boggling piece of statistic, which is that “about 90 percent of data generated by most sensors and A-to-D conversions on the market never gets utilised”.

We can all agree that this is a staggering percentage and that this kind of waste would probably not be tolerated in any other field. So, is dark data destined for an eternity in the shadows or is there a chance it becomes an integral part of companies’ BI efforts?


Dark Data

What Constitutes Dark Data?

Modern companies and organizations receive insane amounts of data every second, especially since mobile technologies have become so advanced and widely used. For example, this can include email communications, web traffic data, network routing information, content streaming data, as well as certain mobile-based momentary insights such as someone’s location or a decision to make an action on their device.

Some of this data goes dark because of its nature which makes it unusable unless acted upon within seconds or minutes. Some of the data goes dark because there is so much of it that an organization could never hope to analyze it in any sensible way.

Why Keep It?

Before we move any further, we need to understand why companies and other entities even store dark data, these copious amounts of information that is completely unstructured and all but unusable. Understanding the price and the complexities of storing ever-growing sets of data, why do not organizations simply get rid of dark data?

The main reasons, it seems, is that companies and organizations are legally required to keep such data and in order to comply with the law, they do it. This is much like the infamous case of Hillary Clinton’s emails which she was required to keep and which her team deleted, thus breaking the law.

Some people argue that it is the simple human nature to hoard data in the hopes that it will one day be used.

This is also paradoxical, as storing this basically useless data becomes a big expense (for the organization.

Is There Hope?

Like they say, there is always hope. The fact is that much of dark data is still quite alluring and interesting to business intelligence professionals and this has caused numerous leading entities in the BI industry to focus their attention on dark data, as well as to come up with ways to shed some light on it.

IBM has been one of the most active players in the whole dark data field, even though they might not call it that. Their cognitive computing efforts and Watson are one of the possible solutions for the problem of dark data which is just too huge to be dealt with in traditional BI terms. For some time, they have been organizing events aimed at raising awareness and addressing many of the issues that have a lot to do with dark data.

People whose OLAP software technology was purchased by Microsoft in the 1990s,BI software company Panorama, have also recently introduced the latest iteration of their software called Necto which is, according to them, able to tap into dark data as well thanks to their own patented algorithms.

Closing Word

It is safe to say that dark data is going to get less and less dark as time goes by. Big players know very well that dark data can be a fantastic source of insights and they are working hard on trying to make sense of it all. We can imagine a future in which dark data is used for business insights.

Future is definitely bright for dark data.

To read more from Dan Radak click here.

For the link to the original article click here. 

Subscribe to our blog