Mind + Machine

Big Data: Hype, Fallacies and Reality

Big data and related fields like analytics suffer from hype. Overblown claims about big data often overwhelm useful, focused, rational discussions of big-data reality. The first fallacy is that “big data solves everything.” It doesn’t. Many people embrace big data without necessary tools and contexts. They need better structures to govern data usage, as well as a definition of how they’ll use analytics, goals, metrics and accountable, expert staff.

Some people falsely believe you need a “data lake and tools.” A data lake is a huge collection of data. The fallacious promise is that you can process data more cheaply if you collect mass quantities of it. Unfortunately, you’re more likely to duplicate data, let data die in the lake and risk handling “intellectual property” badly. People also believe that more data “means more insight.” That can be true – if you process the right data the right way. More often, it isn’t true.

A Data Pyramid

Consider a pyramid of data use with four levels. Level 1 is “raw data,” like surveillance camera photos. Level 2 is “information,” data you’ve partially...