The Flaw of Averages
Applied math usually comes easy to decision scientists. This is not to say all math is easy – but most decisions we make involve relatively intuitive mathematical principles and algorithms. Sometimes, however, simple math doesn’t make common sense.
Consider this scenario: Brand managers at a large CPG company are compensated based on market share growth across different retailers. We made an interesting observation while reviewing the company’s annual performance. The market share across every single retailer grew – yet the overall market share of its products went down. How is this possible? It presented a paradox for group managers, since brand managers argued that they deserved their bonuses.
The key to understanding what happened lies in looking at the Product Volume Sold columns. First, there is a massive difference in volume between the first four retailers and the fifth. Though the brand’s market share increased in each of the first four, the overall volume at each store decreased precipitously. The stores appear to be doing far less volume in the brand’s category.
The fifth retailer does much higher volume, and, combined with a market share increase, helped the CPG company make up for the volume shortages at the other four – although its overall industry market share declined.
This phenomenon is called Simpson’s Paradox – when data from unequal-sized groups are combined into a single data set, the trends are reversed. The shift in market volume from the small retailers to the large retailer causes the market share change to reverse at an aggregate level.
Be mindful of this paradox when analyzing data – looking for trends, causal relations and propensities. While looking for overall trends, we often poke holes around the aggregate data, where details and variation within categories is missed. One resolution is to separate segmentation, group and profile data into tranches and study the behavior separately. The key is not being satisfied with ‘analyses of averages,’ and to always keep in mind that even when an analysis violates conventional wisdom, one should always critique through tough questions.