Institutionalized a test and learn philosophy for a specialty retailer

  • November 10th, 2018

What We Did: Enabled FP&A managers to design, measure tests and devise go-to-market strategies

The Impact We Made: Provided decision makers with “one version of the truth” and increased profitability by 12% 

Summary – In depth understanding of customer behavior

To increase profi­tability during the housing downturn, a large retailer wanted to experiment various strategies from pricing and marketing to store remodeling. Mu Sigma developed an enterprise level system to design tests and provided business groups with a unifi­ed framework to measure the impact. This collaborative platform helped finance teams tweak go-to-market strategies and make better informed decisions leading to a better understanding of customer behavior and a 12% rise in profi­tability.

About The Client – A specialty retailer

The client is one of the world’s largest retailers and specializes in home improvement and construction products and services. Mu Sigma worked with the finance team to develop a test and learn system that provided feedback on changes in go-to-market strategy implemented by Merchandising, Marketing and Store Operations.

The Challenge – Inability to measure efficiency of store strategies

During the economic crisis and housing downturn of 2010, store profi­tability declined dramatically. Changes to assortment, pricing and operational strategy were being considered as part of eff-orts to improve profi­tability and revenue per square foot at existing stores. To determine which of these strategies would prove most effective, many different tests needed to be designed and evaluated before chain wide roll out. It was critical for the retailer to have a unifi­ed framework in place that could measure the impact and provide decision makers with “one version of the truth”.

The Approach – A test and learn system

Tests were designed in collaboration with the client’s merchandising, pricing, store operations and marketing teams. Opportunity assessment, duration of a test and changes to current strategy were discussed in collaborative forums with stakeholders to align with organizational goals.

  • Test and control groups were identifi­ed by evaluating similarity in sales patterns, location, competition and demographics
  • Statistical tests were performed to ensure that test groups were representative of the population
  • An ensemble of predictive models was used to isolate lift due to events while accounting for seasonality, economic trends, weather, promotions and holidays
  • Impact of events on sales, profi­tability and customer satisfaction was continuously measured at various levels- customer, item/SKU, store and market level

Deep dive analysis on test results were performed and the root causes of exceptions were uncovered. Discussions with ­finance and relevant business teams were conducted to fi­nalize go-to-market strategy.

The entire algorithm was developed in a modular fashion and was automated to design and measure over 200 tests a day. An R-Hadoop stack was leveraged along with a HTML 5 user interface to provide stakeholders with a system to design tests, measure and institutionalize a culture of low-cost experimentation. 

The Outcome – Increased store profitability

  • The test and learn framework was highly scalable; End-to-end process automation successfully tested and measured impact for over 2000 events
  • A web-based application was built to centrally manage event design and deliver easy-to-understand reporting for executives
  • Identifi­ed opportunities worth $150M from positive testing and highlighted risks worth $70M from negative testing
  • Enabled the client to test various strategies across merchandising, pricing, store operations and pricing, understand their impact prior to an organization roll out, thereby increasing profi­tability by 12%
  • Institutionalized a test and learn philosophy and provided a unifi­ed approach to measuring the impact of events across business functions