Imagine a company that operates 10,000 convenient stores of which 8,000 are named QwikMart and 2,000 are named FastMart. It is observed that the average revenue per store is $1 million for QwikMart and $1.1 million for FastMart. Will the company benefit by renaming all QwikMarts to FastMarts? One way to answer such a strategic question is to build an analytical model of a typical convenient store with several variables including the name of the store as one. And then perform the analysis under different scenarios. An alternate method is to actually rename a few dozen randomly chosen QwikMart stores and test the revenue impact against another randomly chosen QwikMart stores whose name is not changed. Author Jim Manzi argues in “Uncontrolled: The surprising payoff of trial-and-error for business, politics and society” that our current methods carry a huge bias for the former (analysis) and can benefit from doing more of the latter (experimentation).
Causal density – key challenge: Experimentation hasn’t been a popular method in social sciences especially if you contrast it with correlation analysis. A classic example is presented in the bestseller Freakonomics where the authors argue that a significant fraction of US crime reduction can be linked to legalization of abortions in 1970s. The research involved rigorous correlation analysis. However, in subsequent analysis two Federal Reserve economists found a bug in the software model and with a small change in assumptions the result shows no correlation between abortion legalization and crime reduction. Back-and-forth has continued without any conclusive result.
Building analytical models in social setting where behaviours are involved has a significant challenge. The causal density of a social setting is very high compared to physics laws applied to large objects. That means the number of variables that can impact the observed outcome can be very large, very difficult to find out and hence building a reliable analytical model is difficult. If finding all the causes is too cumbersome, why not experiment and find out? That is the view Jim Manzi presents at least for those situations where experimentation is practical.
Experimentation and business strategy: In my earlier articles, I have observed that Strategy gurus like Michael Porter and Richard Rumelt don’t emphasize experimentation. They present analytical models through which you decipher the internal and external context and create a game-plan as an outcome. And then you implement it. It has been over a quarter of a century since Rumelt-Henderson-Porter started publishing frameworks. It hasn’t worked predictably. It is time managers consider experimentation as a complementary method to successful strategy building. Note that Manzi is not denying the role of Porter-style analytical models in creating hypothesis. He is suggesting that it is worth checking if we can test the crucial assumptions behind the strategy at low-cost quickly. And do those experiments whenever possible.
A throwaway prototype of what later became AdSense was built overnight at Google. Within a week hundred Googlers experienced and assessed the usefulness of content targeted ads. This eventually led to creating a successful monetization model for Google. Google today performs tens of thousands of experiments on search algorithm alone every year.
Experimental revolution in business: Manzi cites companies like Capital One which has turned business into a scientific laboratory. Every decision about product design, marketing, channels of communication, credit lines, customer selection, collection policies and cross-selling could be subjected to systematic testing using thousands of experiments. In fact, Manzi’s own company Applied Predictive Technologies is helping 30 to 40 percent of largest retailers, hotel chains, restaurant chains and retail banks in America perform repeated standardized tests on its platform.
Whether experimentation really becomes a revolution worldwide is to be seen. However, if you want to understand how experimentation is pushing the boundaries in social sciences, Uncontrolled is an excellent place to start.
Note: I am thankful to Prof. Stefan Thomke of Harvard Business School for suggesting this book to me. Thomke himself is an authority on experimentation and has written an excellent book – “Experimentation matters: Unlocking the potential of new technologies for innovation”.