Evidence Shows That When Making Decisions, People Hate Experiments

We like to think that this blog, while born out of a trivial New Year’s resolution, has managed to become a reasonably serious outlet. That said, we fully appreciate that today’s post might read like satire, so unbelievable is the topic. It concerns a study published in the Proceedings of the National Academy of Sciences that identified an interesting response to experiment-based decision making.

Taking a step back, it is worth reminding ourselves that choosing between two options is not always easy. Should the government introduce electric buses or a bike sharing scheme? Is it better to invest in more classes, or should we prioritise after school programs? Do I want my cake, or do I want to eat it instead? These are difficult choices where we are not easily able to assess the impact of each decision beforehand.

To help with these matters we have various tools at our cognitive disposal. Arguably the most powerful of these is the randomised controlled trial (RCT). Without rehashing high school science classes, suffice to say that they enable us to compare two (or more) scenarios in a way that minimises bias. Commonly known as A/B tests, they have found use in medical research, policy making, and product development.

However, despite this pedigree, the authors of the study noted the following:

“We find robust evidence—across 16 studies of 5,873 participants from three populations spanning nine domains—that people often approve of untested policies or treatments (A or B) being universally implemented but disapprove of randomized experiments (A/B tests) to determine which of those policies or treatments is superior. This effect persists even when there is no reason to prefer A to B and even when recipients are treated unequally and randomly in all conditions (A, B, and A/B).”

Logically, we know that to make better long term decisions we need to collect data, and to collect data, we often need to run experiments. Testing two scenarios that are neither individually objectionable nor obviously worse than one another seems like the ideal example; both groups will not lose out to any meaningful extent and we will gain valuable data to improve outcomes for all. Yet, this is the exact situation that the study found such an aversion to.

The authors draw no universal conclusion regarding the driver of this irrational behaviour, although they do mention a few contributing factors. What is clear, however, is that this behaviour is more pervasive than you might expect. They note:

“This ‘A/B effect’ is as strong among those with higher educational attainment and science literacy and among relevant professionals.”

Given the seeming ubiquitousness of this behaviour, it is likely we will encounter it in our organisations too. How many times have you observed comfort with an arbitrary decision disappear when more rigour is put into the decision making process?

Is it the fundamental human dislike of feeling like we are not in control, even if we will also ultimately benefit from the exercise? Is it an aversion to looking too closely at things; shining a spotlight and being afraid of what we might find? Regardless of the driving forces, awareness of these irrational biases can help us make better decisions. After all, if we do not question our assumptions, how will we ever find a better way?

– Colm & Ryan