When it comes to behavioural science, I am an enthusiastic amateur. I guess the same could be said for my marathon running infatuation which, although prolific with over 250 completed, is still firmly footed in the ‘recreational amateur’ category. My payslip reads ‘Agile Coach’ so I’d like to think that I’ve managed to breach the amateur category in my chosen career (but you might need to check with my colleagues to confirm whether I do indeed qualify as a professional). It’s not often that running, behavioural science and agile coaching intersect but recently they did, so I decided to use the opportunity to run an experiment and this is what happened.
The Running Backstory
I was fortunate enough to run the Timbavati Traverse in July. This is a 45km single day event inside the Timbavati Private Game Reserve, whose primary objective is to raise awareness and funding for rhino anti-poaching efforts. We were privileged to see two crashes of rhino during the run. However, the sad predicament in the fight against poaching was unfortunately highlighted a couple of weeks later when Anton Mzimba, Timbavati’s Head Ranger, was murdered in cold blood in an apparent hit by the rhino poaching syndicates .
The Agile Assessment Challenge
At Vitality Group, a company that uses behavioural science to improve people’s health and fitness, we are just under a year into an agile transformation using the Scaled Agile Framework (SAFe). We are at the stage where we are looking to improve our value streams and DevOps practices. As such, we decided to conduct SAFe’s DevOps Health Radar.
This is a self-assessment consisting of 16 questions that covers the full ‘concept to cash’ life cycle. The scope is much wider than what most of us narrowly think of as ‘DevOps’ and requires that leaders in various roles participate so that we can get an accurate and useful result.
Although it would take less than half-an-hour per leader to complete, it’s always a challenge to get busy people to complete questionnaires and this is especially true when you are dealing with people at the senior leadership level. Therefore, this seemed like a perfect opportunity to run a quick behavioural science experiment: Would promising to donate to a rhino anti-poaching charity result in a better assessment completion rate?
The Method
I sorted a group of 24 leaders by first name and randomly allocated them into 3 groups of 8.
- Group 1: “Polite Message” Control Group (no nudge)
- Group 2: If 6 out of 8 completed the assessment I’d donate R1,000 to the rhinos
- Group 3: If everyone (8 out of 8) completed the assessment I’d donate R1,000 to the rhinos
Each group received the email below together with an attached spreadsheet. The spreadsheet contained a tab (worksheet) with each group member’s name. Once someone completed the assessment, I changed that person’s tab to green.
The emails were identical except:
- The header for Groups 2 and 3 was suffixed with “(& help save the rhinos)”
- The following sentence was added to the body of Groups 2 and 3 respectively
- “If 6 out of 8 of you in this group complete the assessment, I’ll donate R1,000 to a rhino anti-poaching charity.”
- “If all 8 of you in this group complete the assessment, I’ll donate R1,000 to a rhino anti-poaching charity.”
The Hypotheses
- Group 1 will have the lowest completion rate
- Group 2 will have the highest completion rate.
Hypothesis #1 is self-explanatory. My logic on hypothesis #2 is that a 100% completion rate was unlikely, so fewer people in group #3 would try. However, other people I spoke to expected group 3 would have the best completion rate because the could be no freeloaders when everyone in the group had to complete the assessment.
What Happened
I sent out the emails at about 6pm on Tuesday and asked everyone to complete the assessment by the end of Friday. By lunchtime on Wednesday, the completion scores were:
G1 = 0; G2 = 2; G3 = 5.
A good start. Less than a day in and things were looking good for the rhinos.
Only one more assessment (in G2) was completed by the end of Thursday bringing the scores to:
G1 = 0; G2 = 3; G3 = 5.
I had made a decision not to send any follow-up “chasing” emails but I had set a reminder flag for noon on Friday (this means that each person would get a pop-up notification in Outlook and, assuming the original message had not been deleted, the message would turn red in their inbox). Unfortunately, I managed to mess this up and only the control group got the Friday ‘flagged’ reminder.
The noon pop-up did result in Group 1 getting onto the scoresheet with two Friday assessments completed. In fact this was the only movement in all three groups leaving the score at:
G1 = 2; G2 = 3; G3 = 5.
The results would be presented at our LACE (Lean Agile Centre of Excellence) meeting on Tuesday so I let the assessment run until Monday close of business and during this time two more G1 members reacted to the polite email and completed their assessment. This left the final score at:
G1 = 4; G2 = 3; G3 = 5.
A Failed Experiment?
Both my hypotheses proved false (G2 had the least and G3 the most responses). However, a failed experiment is one where not enough data is obtained to determine whether the hypothesis is true or false and / or when the experiment itself is flawed.
The experiment was definitely flawed. Apart from omitting the pop-up reminder for groups 2 and 3, the sample size of 8 people per group would not have been statistically significant. However, in this case my objectives were:
- inject a bit of fun to a relatively dreary activity,
- illustrate that experimentation is easy (in the hope that that his will inspire more experimentation in the work we do) and,
- most importantly, ensure that a critical mass of our leaders complete the DevOps Health Radar assessment.
On this basis, I was happy to take a 50% (12 out of 24) completion rate as a success.
From a personal perspective, I was hoping to donate some money to rhino anti-poaching efforts so unfortunately that was a hard fail. However, I will be looking for future opportunities to run some more experiments to ensure that the anti-poaching donation goes to its intended source.
Learnings and Observations
I was hoping the people within the group would encourage other members of their group to complete the survey but this did not happen. I expect that the result would have been different had I appointed a “team captain” to chase the non-conformers. This would be easier in a team environment rather than with a leadership group as there is normally someone in the team (e.g. a Scrum Master) who would naturally be on point to chase, track and remind.
I made a decision not to send follow-up chasing emails but it would have been interesting to see the impact of sending emails to the non-conformers (and copying the rest of the team) reminding them that they were standing in the way of a donation.
The pop-up reminder seemed to have a big impact on compliance – all 4 of group 1’s assessments were completed after the reminder. It would have been interesting to see what happened in groups 2 and 3 if the Friday pop-up has worked as intended.
The “100% compliance group” had much better initial results than all the other methods (all 5 of the group who completed the assessment did so within half a working day of the original email being sent). Email is an easy method to communicate but might not be an effective method for all participants. All of the leadership group get high email volumes but I suspect that some are better than others at controlling their inboxes. Having said that, the two most senior leaders within the groups both completed the assessment.
Future Applications
From the (admittedly cursory) Google search I did, I could not find any data on ‘donations as an incentive’ social science experiments (if anyone is aware of one, please let me know).
I am keen to run more experiments where the reward for a group of people ‘doing something’ is a donation to a worthy charity. Personally, I would be much more inclined to complete an assessment or compliance activity with this nudge. The chance to win a lucky draw prize is a much-used carrot within corporates, but a donation to charity is likely to be a much more effective incentive.
I also think that organisations running consumer surveys would get higher response rates if a donation nudge was used. I typically delete emails from businesses asking for “5 minutes of my time” even if there is the chance of a prize. However, I would happily donate 5 minutes of my time if the message was that “For every x people who complete this survey, we’ll donate y to [this charity].”
One Last Nudge
If you enjoyed this article and would like to donate to a worthy rhino anti-poaching charity, consider making a donation to Rhino Revolution via Grant Murphy’s givengain page: https://www.givengain.com/ap/grant-murphy-raising-funds-for-rhino-revolution/
Grant is the founder of the Timbavati Traverse and the Head Guide at King’s Camp in the Timbavati Private Game Reserve.
Oh, and if 10 readers donate R100 – then I’ll donate R1,000 myself…
