The Problem with Business Data Overload & How to Fix It

The Promise of Big Data

Does more data lead to better decisions? Or better yet, does collecting every bit of data equate to a value-added strategy that moves the needle forward? 

Ever since the tools for managing and understanding big data have improved to a certain level of maturity, all of the emphasis in corporate IT departments and strategy sessions seems to have been to “collect all the data” - and to put it somewhere, anywhere… but preferably into a “data lake”.

If a data lake sounds drab and unstructured, you’re not wrong. Benedict Evans touches on this topic quite eloquently:

Technology is full of narratives, but one of the loudest is around something called ‘data’. AI is the future, and it’s all about data, and data is the future, and we should own it and maybe be paid for it, and countries need data strategies and data sovereignty. Data is the new oil!

This is mostly nonsense. There is no such thing as "data", it isn’t worth anything and it doesn’t really belong to you anyway.

According to Evans, 

"Data" is not one thing, but innumerable different collections of information, each of them specific to a particular application, that aren’t interchangeable. Siemens has wind turbine telemetry and Transport for London has ticket swipes, and you can’t use the turbine telemetry to plan a new bus route. If you gave both sets of data to Google or Tencent, that wouldn’t help them build a better image recognition system.

Fast forward to now.

Data Science and machine learning are two heavily touted elements of the current IT orthodoxy. But there is a dirty little secret that is not often talked about: the real work is in data engineering to get “good data” - the relevant data, cleaned up and in a consistent format - into a place the data scientists can do the most with it; and to get “good data” into the machine learning algorithms so that they learn the right things, and not the wrong things.

It turns out that high-quality data is much more valuable than high-quality data buried in an ocean of low-quality data. The more you just “collect all the data” regardless of quality and context, the harder the work is to engineer out the good, relevant, cleansed data. In other words, the signal-to-noise ratio is terrible. And fixing it afterward is really expensive.

Knowing which data you need has always been important. Knowing which data you don’t need is just as important. However, the “data is oil” argument treats it as all potentially valuable - when in fact, it’s not.

How to Achieve a High Signal-to-Noise Ratio

Believe it or not, building your organization around business processes can drastically improve the quality of the data you collect by ensuring that you get a higher signal-to-noise ratio:

• The valuable business data itself is readily available to your business process by reference or by copy.
  
  
• The business process definition itself allows you to choose more accurately where, when, and what data you want to capture, in order to get the most accurate picture.
  
  
• All of the data you collect from a business process has process context available to improve the data, so that you have not just the snapshot of the data, but also “how you got there” and “where you are” in the process.
  
  
• If there’s data that needs to be referenced from other systems, it can be by reference or by copy (snapshot in time).

The potential value of understanding your end-to-end processes is immense. The process model - and the data the engine facilitates collecting - allows you to have really information-rich and context-rich data to analyze your business’s performance. This also allows for better machine learning, better data science, and less work on data engineering (and yes, this will actually save you time and money)

The noise gridlock is an enigma. There’s a natural tendency to think that the more data we consume, the better. But it’s quite the opposite. By placing too much emphasis on collecting data (and continuing to collect), we lose sight of what’s really important.

For more on how BP3 can help you make the most out of your data, get in touch with one of our experts.