Striking the right balance between data and intuition
Blog Posts:Mu Sigma
Published On: 04 February 2015
During the Vietnam War, U.S. defense secretary Robert McNamara – obsessed with quantification – infamously relied on body count as a key component of his strategy. Ron Johnson, a CEO who dismissed data in favor of instinct, became well known for his disastrous tenure as the head of JC Penney.
Both of these leaders failed due to overreliance on either data or intuition. Striking the right balance between the two is a challenge that plagues every business decision maker.
In the era of Big Data, it has become critical for businesses to start institutionalizing analytics and decision sciences in order to make impactful decisions. With modern technology allowing businesses to gather all sorts of information, data-driven decisions have led to impactful insights and thrown light on many gray areas of our businesses which would have not been possible by relying on just intuitions, ideologies and business experiences. However, overreliance on data has its pitfalls. The risk is that in managing what we have measured, we might miss what really matters.
With that in mind, here are some proven tactics that will help you adopt a scientific approach to addressing business problems and driving the right decisions.
Observation and inquiry: Everyone saw apples falling from trees but it wasn’t until Isaac Newton explored why that we learned of gravity. Curiosity, powers of observation and the ability to ask the right questions are all essential to defining problems correctly and guiding us towards the right solution. Businesses often fall into the trap of rushing to solve the wrong problem or just the immediate problem without looking at the bigger picture, or even overlooking seemingly trivial issues that are actually significant indicators.
Hypothesis or theory: Framing a hypothesis means explaining observations and giving a plausible explanation. This is the aspect we call “theory” that we frame based on our observations, business knowledge and intuitions. A hypothesis could be a prediction or an opinion but it should be specific, testable and falsifiable.
Experimentation: Science teaches us not to believe anything without evidence. Even if it seems like an obvious hypothesis, it needs to be proved or disproved with data. The null hypotheses that we test are always to falsify the hypotheses we framed, to doubt our intuitions and closely observe what our data is telling us. If a hypothesis is rejected, we need to find out why and check if our data is pointing out the opposite with enough confidence. And if the data validates our hypothesis, we can accept our theory but only with the disclaimer that it may be true now but not always. Newton did not know his universal laws of gravitation could be later falsified with Einstein’s theory of relativity.
Tools and techniques: Our ability to boast about experimentation and evidence comes from technological advancements. Without statistical techniques and modern analytical tools, our data would be a dump of overwhelming information. From basic correlation which derives its roots from pen-paper calculations, to machine learning and neural networks, which enable highly complex computations, we have an intense amount of statistical techniques to choose from. However, many analysts often limit themselves to relatively familiar techniques like regression. They need to learn to adapt themselves to richer computational techniques or soon we will start falsely blaming our data for misguiding us.
A key concept of science is to fail fast and learn faster. In this rapidly changing business environment, companies no longer have the luxury of procrastinating decision making for too long due to fear of failing. Failure is something that can propel a company to future greatness if they can learn fast from their mistakes.
Nevertheless, there are limits to how scientific decision making can be used in a business environment. Unlike scientists, who have the luxury of withholding judgment until sufficient evidence has accumulated, businesses often have to act in a state of partial ignorance or else lose to competitors. While making decisions for devising a new strategy or policy or starting a new line of products, we might not have enough evidence to predict success or failure with complete confidence. Even here, though, the scientific method is instructive, not for eliciting answers but rather for highlighting the limits of what can be known. This is where co-existence of information and intuition comes into play.
Science teaches us how to make decisions based on evidence, but also not to hold these evidences as eternal truth. Is it time to employ just a few more scientists in the C-suite?