The big data fallacy: Why big data is not objective and why it shouldn’t be

Big Data is not as objective as we think, it is vulnerable to subjectivities and confirmation bias. It is also unable to capture the emotions and beliefs that motivate customer experience. Complementing Big Data with Thick Data provides rich ground for quality consumer insights.

Big Data is big business. Globally, the Big Data market is expected to be worth $103 billion by 2027. The business world loves the cool, objective truths of Big Data that are absent of any misinterpretations or messy interference by a human analyst. Business is adopting Big Data at record speed, confident that it will provide clear, stable predictions on future consumer behaviour. Consultancy firm Ernst & Young are not alone in espousing Big Data as enabling business “to make better, smarter, real-time, fact-based decisions”. You would be in good company, then, if you assumed that Big Data was the key to answering that ever-elusive question of how humans behave.

You would also be wrong.Big Data Market Size 2011 to 2027




The realms of advertising and marketing abound with hyperbole around Big Data’s potential to revolutionise the way we understand and interact with customers. With data on virtually every digital action of the modern consumer, marketers are confident that they are able to produce a genuine, data-rich picture of their customers. They extoll the ability to interpret their customers’ personality and preferences based on the vast data points around their patterns of behaviour. It’s not uncommon to see business analytics reports littered with pictograms like this one:

Big Data Process: drivers - risks- success


And, who can argue with a pictogram, right? If a pictogram tells me that these simple steps will produce “Big data success” then who am I to disagree?

Why, then, do 85% of Big Data projects supposedly fail?




open bible sitting on a tableBecause of its apparent objectivity and absence of human error, Big Data has achieved a some-what godly status; seen by many in business as an omniscient tool that will provide the absolute truth about consumer behaviour. And there are good grounds for this.

In the idiosyncratic new book “Raw Data is an Oxymoron”, Daniel Rosenberg traces the history of the word ‘data’ back to 17thCentury theologians. It was first used in reference to the undeniable truth of scriptural text. Since then its meaning may have changed, but its God-given status doesn’t seem to have been challenged much at all.




In reality, Big Data is far from God-given. It is created by ordinary people. And, unlike the God of the 17thCentury scriptures, people aren’t infallible – they have opinions, biases and subjectivities. Confirmation Bias is one such fallibility. In interpreting data, people have a “tendency to look for things that support a viewpoint while ignoring conflicting information”. Access to huge amounts of data and sophisticated analysis programmes allows us to pick and choose data as we like. Whilst this technology allows us to understand the world in ways unimagined before, it also allows us to manipulate data to confirm our pre-existing views.


“People show confirmation bias, which is the tendency to look for things that support a viewpoint while ignoring conflicting information”





The supposed power of Big Data lies in its approach. It looks at what people actually do – their behaviours – rather than what they believe or say they do. By focusing exclusively on measurable behaviours, it claims to remove the risk of subjective bias that comes with a research participant telling you how they think they behave. Now, don’t get me wrong, I strongly believe there is merit in that. As the famous Anthropologist, Margaret Mead, succinctly put it:


“What people do, what people say, and what people say they do are entirely different things”.

However, looking only at behaviour patterns is not enough. And this is where Big Data falls short: it doesn’t provide an explanation of whypeople do what they do, it doesn’t expose their motivations. Two people could follow the exact same customer journey to buy your product, but their motivations may be entirely different. So, the way you market to them should, ideally, be different as well.





Disappointed man opening gift of socksOur purchasing decisions are heavily-laden with emotion – we often buy with our hearts, more than our heads. Think about Christmas gifts. Your mother-in-law doesn’t buy you socks every year because you need new socks. No, she buys them to show that she cares. It’s a gesture of affection, an emotional purchase.

Big Data does not capture this emotional side. If you want to truly understand your customer, you need to understand why they buy your product or why they shop in a certain way. Without that knowledge, you cannot understand how to improve your product to fit into their world – which is, ultimately, what you want to be doing as a business, right?



Now, I’m aware there has been a lot of Big Data bashing in this article, which is not strictly the message I want to convey (okay, maybe a little). Big Data is great: I strongly believe it can provide incredible insights into human behaviour. I just want it to be taken down from its pedestal. For two reasons:

  • Big Data is not entirely objective. It is subject to interpretation and bias just like any other research method. And like all methods, it excels when it is utilised by a skilled analyst who really understands how to navigate its pitfalls and hazards.
  • Big Data only shows behaviour. It misses out on the irrational, messy and unexpected nature of human decisions. It works best when complimented with qualitative data that explains this “human” side of consumer decisions.

Humans are so much more than the behaviour patterns Big Data would have us believe we are. It is our complexities, emotions and irrationalities that give the world colour and make us human. We do not stop being human just because we consume or engage with a product. And god forbid we ever do.


Billy Jones

Leave a Reply

Your email address will not be published. Required fields are marked *