In this article, I’ll use a simple, easily accessible example feature to show how a feature hypothesis should evolve and how monitoring results can impact our plan and backlog priorities.
I am going to stick with the “Password Reset” example from the previous article in this “Outcomes versus Outputs” series since resetting one’s own password can be applied to almost any website and application.
This is the third in a series of articles on “Outcomes over Outputs”.
Part 1: Kohavi’s Law & Harry Potter explain why your experience and intuition suck
Part 2: Outcomes: The Forgotten Test Case
Password Reset Scenario
When users forget their password, they must phone the call centre to get their password changed. Analysis of the call centre logs reveal that there is currently an average of 3,500 “password reset” requests to the call centre per month. We predict that building self-service password reset functionality will reduce call centre volumes by 90%.
Hypothesis example
Typically, this would hit the feature team as something like “Provide self-service password reset functionality by email and SMS.”
But what are we really trying to do? I would propose something like, “Provide the ability for a user to reset their own password securely without having to phone the call centre.”
This is a subtle but important difference since we are moving from what we want to do (‘build a feature’ which is an output) to what we want to achieve (‘obtain a valuable result’ which is an outcome). A correlating feature hypothesis might be, “If we provide the ability for users to reset their own password securely, we will reduce the call centre volumes by 90% (3,150 incidents) per month.”
The temptation is to try to monetise every feature. In the “Password Reset” example, the actual feature I worked with was given a cost savings “value” of R800,000 per month. The problem is that even if the feature achieved the 90% reduction it would be very difficult to prove or quantify the actual rand value savings. However, it is very easy to measure call centre volumes and use that as our proxy for value.
From a team motivation perspective, it is also far more inspiring to “reduce call centre volumes by 90%” than “save the bank R800,000” or “build functionality that allows customers to reset their password by email.”
Evolving the Hypothesis
Our “Password Reset” feature is prioritised and deemed important enough to take into the next iteration. During backlog refinement, business and IT discuss the true business need, user requirements, technical possibilities and system constraints.
In this example let’s assume that, based on analysis of call centre logs, customer feedback and expert opinion we believe we’ll get a:
- 50% reduction with email self-service password reset option
- 30% reduction with a SMS self-service password reset option
- 10% reduction with a USSD self-service password reset option
Therefore, we only expect to achieve our stated objective of a 90% reduction if we offer all three channels.
It’s important to note that these are estimates (read “best guesses”) based on the current knowledge available to the team.
Lean 101 says that the smaller the feature, the faster it will flow through the system and be delivered. Therefore, we split the original “Password Reset” feature into three features (one per channel). From a sizing point of view, let’s assume that each of the options has the following relative size in story points based on anticipated volume, complexity and associated risks:
- Email: 20 story points – the team is comfortable as they understand and have full control all system components to produce the feature; they are confident that most email addresses in the system are correct as monthly statements are sent to the email addresses and failed delivery is closely monitored (and incorrect email addresses corrected).
- SMS: 40 story points – the team is relatively comfortable but have only rudimentary experience working with SMSs; there is a risk that some mobile numbers may be incorrect in the system.
- USSD: 100 story points – the team has never worked with USSD functionality before and the size reflects the risk and learning curve of understanding and implementing a new technology; there is a risk that some mobile numbers may be incorrect in the system.
In this scenario, the decision is easy. The first feature we focus on is “Password Reset by email” as this is the most important (50% reduction) and easiest (20 story points) option.
Note: If the scenario changed so that SMS was the most important option but the above sizes/complexity remains unchanged (or for that matter any alternate scenarios changing priority and size), more discussion and the use of a prioritisation technique like Weighted Shortest Job First would be required. The rule of thumb is to go with the quickest delivery of value – as this is the fastest way to assess value achieved and learn.
Evaluating the Hypothesis
The feature team builds the “Password Reset by email” feature and it is deployed into production. After go-live, the team monitors the actual results. Let’s play out a few possible scenarios:
- Our predictions were 100% correct and call centre volumes drop by 50%:
In this case we continue as per our plan and “Password Reset via SMS” will be delivered next followed by the “Password Reset via USSD” feature.
- We get a 90% reduction in call centre volumes:
Suddenly the “Password Reset via SMS” and the “Password Reset via USSD” features become a lot less important because we have got all the value with the first feature. The remaining password reset features will go to the bottom of the backlog (or may even be removed completely) as delivering the additional password reset options will result in vastly diminished value.
- We only get a 10% reduction in call centre volumes:
This does not automatically mean that the other password reset options become much more important. This is an unexpected result and we first want to understand why the achieved value was so low. Perhaps users just don’t know about the new feature and some ‘marketing’ is required, perhaps we’ve built it so badly that it’s easier to phone the call centre than use the self-service option or perhaps we were wrong in our predictions and our users would much rather use an SMS or USSD option.
- We get an 85% reduction in call centre volumes but notice that one market, in this example Kenya, is not using the new feature:
Further analysis determines that the Kenyan market requires a USSD option as this is what they are familiar with (e.g. as customers there use USSD applications like M-Pesa as their primary means for financial transactions). In this scenario, priority is adjusted and USSD jumps above the SMS option on the feature backlog.
Learn from the Outcomes
It’s important to realise that hypotheses are estimates of the expected value we’ll achieve. They only become ‘fact’ when they go live in production. Hypotheses enable us to run experiments and chase outcomes rather than outputs. The features we build and the hypotheses they contain should be evolving over time.
My definition of agility is, “the ability to change direction as fast and efficiently as possible”. Using feature hypotheses to evaluate the outcomes of each feature enables us to learn faster and become more agile by changing direction based on actual results.
This is the third in a series of articles on “Outcomes over Outputs”.
Part 1: Kohavi’s Law & Harry Potter explain why your experience and intuition suck
Part 2: Outcomes: The Forgotten Test Case
Follow Running Mann:
