My 3 takeaways from “Experimentation works” by Stefan Thomke

I have been a fan of both systematic experimentation and Prof. Stefan Thomke for over a decade. Hence, it is not surprising that I enjoyed his recent book “Experimentation works: The surprising power of business experiments”. In this book, Thomke makes some of the core concepts from his earlier book “Experimentation matters” more accessible and brings out the increased scope and scale of disciplined business experimentation in the digital era. I have several takeaways from this book. However, for the sake of creating interest, let me highlight three of them.

Bad vs good experiments:  The book brings out characteristics of what makes a good business experiment. When a CEO of a retail chain J.C. Penney implements a bold plan of revamping the retail stores based on what worked in his earlier stint at Apple, the company is demonstrating HiPPO, a bias for Highest Paid Person’s Opinion. That is not even a bad experiment. When you tinker with rewards to see if it increases productivity, it is an example of trial-and-error or an uncontrolled or a bad experiment. Why? This is because you don’t know the counterfactual. i.e. How do you know that productivity would have increased even without the changed reward? Thomke dedicates a chapter “What makes a good business experiment?” to explain this which I found useful. The attributes of a good business experiment include falsifiability of the hypothesis, feasibility, and repeatability of the experiment among others.  

Scale of online experimentation: With the advent of online digital platforms, designing and running randomized control trials became cheap, fast, and scalable. Thomke dedicates three chapters to present various aspects of online, large scale experimentation. One chapter takes a peek into his favorite experimentation organization - Booking.com which runs more than one thousand concurrent tests on its website, servers and apps every single day. When it is designing a “Book” button, it creates two versions one with say, a yellow button, and the other with a blue button, and then it gets tested live with millions of customers. The color that attracts the most bookings gets used. David Vismans, chief product officer, says, “Our customers decide where to take the website, not our managers.” With millions of page hits every day, even a small one percent improvement in conversion can have a big impact on the business. Booking.com is not alone, LinkedIn runs between five hundred and one thousand experiments concurrently through the year. Goole, Amazon, IBM, and even start-ups have been using this approach to experimentation.

Ethical issues in business experimentation: What if you are testing a differential pricing rather than different colors of the button? Could it be unfair to the customers who pay more? While designing experiments, has care been taken to see safety and emotional impact on customers? In other words, experimenters carry ethical responsibility to test new ideas for integrity before running randomized experiments. Hence, some of the leading experimentation organizations are adding ethical guidelines and case studies as part of their employee training. In one chapter, Thomke looks at seven attributes of experimentation culture such as integrity.

One area which I wish the book covered more is – replication crisis. As of today (June 2020), it is a decade long ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate. It mainly affects social sciences and medicine. Since business experimentation is akin to social science experimentation, I feel it is relevant here.

In the epilogue, Thomke imagines future directions of business experimentation. With the advent of Artificial Intelligence (AI), could design, execution and analysis of experiments be automated – outsourced to experiment-bots like chat-bots? What if the business decisions themselves are taken automatically without human intervention? Thomke feels, based on the current research, that some of the required ingredients for this to work exist already today.

The book gives a number of pointers for further study which I find very helpful. I strongly recommend this book to managers who care about innovation and experimentation.

image source: amazon.com