Ron Kohavi was the Technical Fellow and VP of the Analysis & Experimentation team at Microsoft. He had a team of brilliant engineers, data scientists and program managers working under him. Kohavi’s team built Microsoft’s “Experimentation Platform.” Before the Experimentation Platform was in use, Microsoft teams were delivering well-thought out features, some small, others larger multi-month projects. Output was great – and Microsoft products like Bing, Edge, Exchange, Office, Skype, Windows, and Xbox were benefitting from the exceptional work produced.
Or were they?
There was seemingly no need to check the outcomes. After all, these were well-thought out features designed by the finest minds and delivered by the best technical skills in the software engineering industry. It was only logical that the expected user and business benefits would follow. The focus on outputs was highlighted by Microsoft offering “Ship It” incentives on the basis that, “Every time a product ships, it takes us one step closer to the vision.”
Kohavi took a different approach. He encouraged the teams to evaluate ideas with the scientific rigor of controlled experiments, the same process used by the Federal Drug Administration in the US to approve drugs. To everyone’s horror, they found that only one change in three resulted in the expected benefit, one change in three had zero impact and the remaining third actually had a negative result*.
* I used the slide below in a presentation I did at Agile Africa 2018 on ‘The Power of Feature Hypotheses‘. Negative results are surprisingly common. I cite the example of a traffic circle that was constructed near my home to “alleviate traffic” but made traffic congestion much worse.
In fact, a 33% success rate is pretty decent. Of the few organisations that run experiments and assess the actual outcomes, better results are seldom experienced (for example on optimised systems like Bing and Google Search, success rates are much lower, typically ranging between 10% and 20%).
Therefore, I would propose Kohavi’s Law: Only one change in three will yield a positive result (with the obvious corollary that two out of three changes will have no detectable impact or a negative outcome).
We tend to treat the stated benefits of new products, features and functional changes as fact rather than as experiments. On top of this, it is incredibly rare that we measure the actual results of a change.
The reality is that every change – large or small; no matter how well-thought out; how much market research was conducted beforehand; or how intelligent and important the person behind the idea – is actually an opinion.
Every new feature we build is actually a bet. Every business case and benefits statement is actually an estimate*. We think that the odds are in our favour but the data suggests otherwise. Standish Group studies on feature usage indicate that the bigger the bet, the worse the odds – with upwards of 50% of features within a system never or hardly ever used (and only 20% of features are regularly used).
* And this is normally a grossly overly optimistic estimate to ensure that a feature gets approved and prioritised.
Aside: The good news is that there is an easy way to avoid this problem by using minimum viable products (MVPs) and feature hypotheses (a topic which I will explore in more detail in a future article).
READ: The Power of Feature Hypotheses
So we’ve given Ron his share of the limelight, but where does Harry Potter come into this story?
In the mid-1990s, single mother Joanne Rowling was desperately trying to get her masterpiece published. She described her situation being, “as poor as it is possible to be in modern Britain without being homeless.” The manuscript for Harry Potter and the Philosopher’s Stone had been rejected by every major publisher – and most of the minor publishers.
The first three chapters of Harry Potter and the Philosopher’s Stone arrived on the desk Nigel Newton, the chairman and editor of a small publishing house called Bloomsbury. Bloomsbury had only recently started publishing children’s books, however Newton had still not taken a look at Rowling’s manuscript. He arrived home one evening and gave the copy to his eight-year-old daughter Alice, asking her to “Take a read and tell me what you think.”
Alice devoured those first three chapters in one sitting. According to a rare interview with Nigel Newton in The Independent, “She came down from her room an hour later, glowing, saying ‘Dad, this is so much better than anything else.’ She nagged and nagged me in the following months, wanting to see what came next.”
It seems incredible that the best-selling book series of all time was only published because of the persistent “pester power” of an eight-year-old girl, especially in an industry like publishing where revered editors rely on their experience to make decisions. The same editors will brag about their intuition – every now and then they’ll take a flyer on a new author or back a hunch on something a little different, using the rationale that, “I’ve got a good feeling about this one.”
Children’s books about wizards don’t sell (especially those written by female authors*) and no one had a good feeling about Harry Potter. After suffering so many rejections, it seemed that Harry Potter would be confined to a broom cupboard under the stairs in Little Whinging, until little Alice Newton got her hands on those first few chapters.
* Joanne Rowling has no middle name. Just before the first book went to print, the publishers requested that she use a gender-neutral pen name as “boys don’t read books written by female authors”. She chose the K in J.K. Rowling after her grandmother Kathleen.
The magic of Harry Potter dispels the illusion that we should place our trust in experience, intuition and expert opinion. Despite Alice Newton’s endorsement, even Bloomsbury only went with a paltry first print run of 500 copies for Harry Potter and the Philosopher’s Stone (not such a giant leap of faith after all). However, those 500 first edition copies spawned a dynasty whose estimated worth is over $25 billion and made Joanne Rowling the world’s first and only billion-dollar author.
What Harry Potter highlights and Kohavi’s Law proves, is that we grossly overexaggerate the importance of our experience and intuition. The best way to check the viability and value of an idea is to perform an experiment with the target audience and observe the results. How we do this is a topic that I will be exploring in future articles in this series on “Outcomes over Output”.
This is the first in a series of articles on “Outcomes over Outputs”.
Read Part 2: Outcomes: The Forgotten Test Case
Acknowledgements: If you’re looking for a good lockdown read, check out Kohavi’s latest book (co-authored with Diane Tang, and Ya Xu), “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing“.
Follow Running Mann:
I think that one of the reasons for this seeming anomaly is that there too many people with not enough to do. In order not to be made redundant from their very well paid jobs, they dream up lots of completely useless ideas which make our lives more complicated without adding any benefit.
That was a great article Stuart. I really hope it gets everyone talking world wide especially the leaders of big business, and also our political leaders.
Great read Stuart, thank you