Sunday, January 18, 2009

Overcoming prediction disability: Applying the black swan wisdom to idea selection

Apollo 13 explosion, a black swan: Let’s re-visit the Apollo 13 story we discussed last year. Ken Cox, Technical Manager of the control systems program proposed that they should build a contingency mode of coming back in case there is an explosion in the main command module. Apollo Program Office (like any other project review committee) asked the question, “What is the probability of this event occurring?” Ken didn’t have the foggiest idea of what the probability was. So the Program Office turned him down saying, “Well, but you haven’t proved yet that this is really needed.” With his deep conviction and rapport with software control board chair, Ken went ahead and implemented his idea anyway. The explosion did occur and it brought the astronauts back safely to earth. Apollo 13 explosion is a classic black swan and the question program committee asked about predicting the probability of such an event is the classic mistake we make in evaluating ideas.

Prediction disability: Nassim quotes a famous baseball coach Yogi Berra in The Black Swan, “It is tough to make predictions, especially about the future”. Black swan wisdom says that we can’t predict a black swan especially in Extremistan where a single event in future can dwarf all other events cumulatively occurred so far. However, prediction is deeply institutionalized in our world and we routinely fill probability of events in the risk plan templates. If we apply the traditional risk plan template of a maintenance project to innovation management, it becomes easier to predict the future of the innovation. If Marissa Mayer had applied risk plan template to Paul Buchheit’s idea of AdSense, perhaps Google wouldn’t be where it is today (see the AdSense story). So if prediction of success is not the way to go in evaluating ideas, what do we do?

How to evaluate ideas? Here is an approach which does not require either the innovator or review committee to predict the future. Every idea comes with a set of assumptions why it makes sense (about needs, technology, production capability, profitability). While one can’t predict probability of each assumption turning real, one can estimate (a) cost of an experiment that can validate or clarify each assumption and (b) implication of an assumption turning true or false. For example, in case of Apollo 13, Ken was in a position to tell the cost of implementing his idea and the impact of success was saving lives. Similarly, cost of prototyping the AdSense technology was very low (Paul did it overnight) and implication of the acceptance of this idea was a huge upside. We can plot these ideas with costs on the x-axis and impact on the y-axis. Then we can decide to fund all the ideas in quadrant D and selected ones in quadrant A & C.


  1. As mentioned and translate in the matrix the idea to verify and assumption as quick as possible that is for the cheapest cost. The matrix is valid for one identified item and as mentioned the highest disaster in project are due to systemic effect. Based on this the planning method should be adapted to the level of assumptions. An oprating plan will be disastrous for an innivation but an discovery learning plan will help to monitor the unknown.

  2. Some observations:

    1. Apollo 13 explosion is technically not a Black Swan. Because a Black Swan event should be "unexpected". Ken Cox and the people he convinced visualized such a possibility and prepared for it. Nassim Taleb calls such an event Grey Swan - rare, high impact but not unexpected.

    2. On the cost-impact matrix: Is it always possible to visualize the potential impact of success for an idea? In my experience "No". Neither James Watt nor his first two financiers, Prof. Black and Dr. Roebuck, could visualize the potential of Watt's modified engine. It took ten years and a Boulton to say that the engine can be useful to "anything that moves". What Venture Capitalists do is to identify broad trends which they believe are creating high impact and then fund lots of experiments. Boulton was doing something similar: the sector he was betting on was "energy". For more on Watt-Boulton story see Two experimentation loops in an innovation: story of Watt-Boulton steam engine

  3. In this video, Scott Cook from Intuit presents how they try to move idea selection away from "Politics and PowerPoint" to "Leadership by experimentation" or more precisely "Decisions by hypothesis validation". Cook says that an important role of a leader is to create an environment where experiments can happen at low-cost and at high speed. And then, as much as possible, run experiments to validate assumptions embedded in ideas. He says he is surprised as to on how many ideas you can run experiments.

    In case you want to jump directly to his "Leadership by experimentation" piece in the video go to 6:40.