Chapter Five

Move the needle

Anticipate regression to the mean

BENEFITS OF THIS MENTAL TACTIC

Understanding regression to the mean will remind you that most outcomes stem from skill and luck.⁴² It will encourage you to try and figure out how much of each played a role in a particular outcome. This is important for a number of reasons.

1. It helps assess your outcomes When skill is involved, a relatively small sample size is sufficient to judge the person or process that lead to an outcome (take chess, for example, which is hard to win by being just lucky). But if more luck is involved, then you will have to look at a much larger group.
2. It allows you to forecast results Once you know the relative contributions of skill versus luck in a given outcome, it is easier to forecast subsequent outcomes.
3. It helps you calibrate your feedback Most people agree that praising (or blaming) a person whose success stems only from luck doesn’t make sense. With people who are struggling with failure, you can be more helpful by identifying skills they’re lacking which, if strengthened, could help them become more successful in the future.

Use this mental tactic whenever you suspect randomness is at play to influence the decision situation at hand.

ANTICIPATE REGRESSION TO THE MEAN

In baseball, the Sophomore Jinx is well known: a player who has a great rookie season is unlikely to do as well or better his second year. That is regression to the mean at work.

In baseball, the best rookie player in each league receives the Rookie of the Year (ROY) Award. But the performance of the winners in their next seasons tends to be worse than in their first year. The rookie fails to live up to the standards of their first effort. Professional athletes aren’t the only ones affected by this seeming curse. In the USA, high school and college students who earn the highest grades in their freshman year tend to prove less successful in their second year. And singers and bands who score a big hit with their first album tend to see lower sales of their second album.

So, what’s going on? The culprit is regression to the mean, a statistical maxim stating that exceptional outputs tend to decline (regress) to an average over time.

When it comes to human performance, regression to the mean occurs because success is always a function of both skill and luck. ROYs, star first-year students and new entertainers who hit it big with their first effort, are always identified as successes on the basis of their exceptional performance.

Since most people assume that performance is based on skill, few of us factor in the role of luck in these early successes – and in later experiences. When performance inevitably regresses to the mean over time, people are surprised and even disillusioned. They don’t understand what’s required to sustain success or to become successful. In short, people give too much weight to extremes.

AN INTRIGUING DISCOVERY

On 16 February 1822, one of the world’s most famous statisticians, Francis Galton, was born. A cousin to Charles Darwin, he came from a very talented family, although he never quite achieved the same level of fame that Darwin did. Still, Galton’s achievements were first-rate and his influence on statistics is still felt to this day (he coined the terms correlation, quartile and percentile).

Galton was particularly fascinated by the study of human populations. He was especially devoted to the study of the concept of heredity, how traits were passed from parents to their offspring. Galton first introduced the idea of regression, which meant something completely different from what it stands for today. Instead of typical regression downwards, he meant ‘regression to the mean.’

As part of his research, Galton explored the link between heights of parents and their children. He indicated the heights of 928 adult children on one axis and the mid-parent height (average height of father and mother) on the other axis.⁴³ The result is the chart below.

When Galton examined this plot, he expected to see offspring resembling their parents in size. In other words, he expected most points to lie on or near the 45-degree line, which indicate similar heights for parents and their children. But instead, he found that, with parents who were unusually tall, the offspring tended to be shorter than their mother and father, and, with parents who were shorter than usual, many of the offspring were taller.

The data that he collected didn’t fit the 45-degree line (solid black in the chart), but rather the slightly less steep dashed line. This suggests that children from exceptionally tall parents are typically smaller than them, and children from exceptionally small partners are typically taller than them. Galton observed the following:

“It appeared from these experiments that the offspring did not tend to resemble their parents in size, but always to be more mediocre than they – to be smaller than the parents, if the parents were large; to be larger than the parents, if the parents were small.”⁴⁴

Francis Galton

This seems counterintuitive, but it is a classic case of regression to the mean. It is not about genetics, it’s about statistics. Galton had expected to see offspring resembling their parents in size. Instead, he found that, with parents who were unusually tall, the offspring tended to be shorter than their mother and father. And with parents who were shorter than usual, many of the offspring were taller. A tall mother’s height is partly caused by genetic factors, and partly due to random factors (and environmental influences) which made her grow taller than average during childhood. The genetic part of her height will pass to her child, but the random or environmental factors will not, making it more likely that her child will be shorter.

RANDOMNESS IN EVERYTHING

Pick up any business book in your favourite book store. Many of them (certainly not all) will try their best to convince you about the merits of some new management instrument. Typically, the author selects real-life examples of companies that have seemingly outperformed the competition over time as proof for the efficacy of their brainchildren.

While there is certainly merit to some science on success factors, research often falls prey to regression to the mean. Take, for example, management thinkers who pick companies based on past performance and claim to have identified success factors for their past achievements. Jim Collins’ bestselling book Good to Great⁴⁵ features eleven companies that have outperformed the rest over a period of time. Collins singles out five differentiators or success factors that he considers responsible for the difference of performance. What can we learn from analysing winners? Our common sense says: “A lot.” But common sense can sometimes lead us astray.

Michael Cusumano, a professor at MIT’s Sloan School of Management, uses a simple exercise to help students see the importance of accounting for both skill and luck as drivers for successful outcomes. On day one of his Advanced Strategic Management course, he typically asks his students to stand up. He then tells them he’s going to toss a coin, and asks them to choose heads or tails.

He tosses the coin, reveals the side that came up, and asks the losers to sit down. After a number of rounds, there are usually only one or two students left standing. He then asks these winners to come to the front of the class and give an elaborate explanation on their success factors. By then, everyone clearly understands that the last students standing were merely lucky.

A team of three management thinkers, Raynor, Ahmed and Henderson took a closer look at the companies prominently highlighted as ‘best-in-class’ in popular management books such as Good to Great, In Search of Excellence or What Really Works. By looking at the total shareholder return (a measure for company performance), they confirmed that luck was an important, and mostly ignored, reason for company success.⁴⁶ The smaller the sample size (numbers of years with above-average success) and the bigger the role of luck relative to skill, the more difficult it is to determine best practices.

It is unsurprising, then, that many of Jim Collins’ favoured companies from 2001 have since fallen from grace. One of his top picks was Fannie Mae, the giant mortgage lender that needed to be placed into conservatorship by the Federal Housing Finance Agency in the wake of the financial crisis. Another was Circuit City, which is now bankrupt.

CHECKLIST

FURTHER EXAMPLES

Scold and praise

Let’s say you are coaching your talented daughter in gymnastics. For the next European Championships, she will need to practise performing her vault. After performing a beautiful handspring double front somersault tucked, you praise her on the assumption that praise will motivate her to do even better next time. But surprisingly, after a very good jump, she will usually do worse.

And the opposite is true, too. If you scold her after a particularly bad performance, she will most likely do better in the next round. It is tempting to conclude that your praise and scolding have resulted in a certain type of observable performance. But the difference in the gymnastic performance of your daughter can simply be attributed to regression to the mean.

Your daughter’s performance moves somewhere around her individual average performance. That’s where most of the mass of the bell curve above lies. A really good performance is likely to be a positive outlier, just like a very bad performance. As such, after a very bad performance, it is likely that she will do better than before – and vice versa. This is generally not because of you scolding or praising, but simply due to a statistical fact.

Traffic safety

Imagine you are the president of a large city’s transit authority. Accidents at one particular junction in the city have caused a particularly high number of traffic accidents over the last two years – some of them even fatal.

You call in your advisors to discuss options to bring down the number of accidents and make the junction safer. One of your advisors is able to convince you to put up speed cameras on two roads leading to the junction.

Sure enough, the accident rate starts to fall over the following months. So was it a good idea to install the speed cameras? It is tempting to assume so. After all, the fall in accidents occurred after the cameras had been put up.

But think about it again. Speed cameras are often installed as a response to an exceptionally high number of accidents in the periods before. But if there haven’t been any other fundamental changes to the traffic flow, the increase in accidents is likely to be simply a statistical anomaly. In other words, we would expect the number of accidents to return back to base levels anyway. We would see the numbers of crashes regress to the mean. The upshot is that the speed cameras might have had an impact, but likely a much smaller one than we think.

The chart below makes it clear. By focusing on a certain junction with a high-accident rate, you are more likely to select an outlier. Say the long-term average number of crashes is around five to six, but it had a record number of nine accidents in 2013.

But because of its outlier nature, the number of accidents will naturally go down over the next year. It is important to note, though, that it is not certain that they will go down. It is just vastly more likely that they will go down to four, five, six or seven accidents in the next year.

As the transit authority’s president, how do you find out the true effect of installing speed cameras on traffic accidents? There are two ways. If you have sufficient data from the same site going back in time, you can establish a baseline and verify if the nine accidents per year are actually the norm or the exception. Or, you could compare the intersection with a control situation, i.e. another high-accident site where no additional speed camera has been set up. The key here is picking the right pairs for comparison to establish a true counterfactual.

THE BOTTOM LINE

We are programmed to automatically look for patterns in data. But we often impose patterns on what is, in fact, random. Establishing rules that work reliably is difficult, particularly if you only have a few observations to build on, and if luck plays its part as well. To overcome regression to the mean, think about how much the success you are observing could be due to chance, hone counterfactual reasoning (what could have happened but didn’t), and try to find more historical data points.