Chapter 10

Peapods and Perils

Regression to the mean provides many decision-making systems with their philosophical underpinnings. And for good reason. There are few occasions in life when the large are likely to become infinitely large or when the small are likely to become infinitely small. Trees never reach the sky. When we are tempted—as we so often are—to extrapolate past trends into the future, we should remember Galton’s peapods.

Yet if regression to the mean follows such a constant pattern, why is forecasting such a frustrating activity? Why can’t we all be as prescient as Joseph in his dealings with Pharaoh? The simplest answer is that the forces at work in nature are not the same as the forces at work in the human psyche. The accuracy of most forecasts depends on decisions being made by people rather than by Mother Nature. Mother Nature, with all her vagaries, is a lot more dependable than a group of human beings trying to make up their minds about something.

There are three reasons why regression to the mean can be such a frustrating guide to decision-making. First, it sometimes proceeds at so slow a pace that a shock will disrupt the process. Second, the regression may be so strong that matters do not come to rest once they reach the mean. Rather, they fluctuate around the mean, with repeated, irregular deviations on either side. Finally, the mean itself may be unstable, so that yesterday’s normality may be supplanted today by a new normality that we know nothing about. It is perilous in the extreme to assume that prosperity is just around the corner simply because it always has been just around the corner.

Regression to the mean is most slavishly followed on the stock market. Wall Street folklore is full of such catch phrases as “Buy low and sell high,” “You never get poor taking a profit,” and “The bulls get something and the bears get something but the hogs get nothing.” All are variations on a simple theme: if you bet that today’s normality will extend indefinitely into the future, you will get rich sooner and face a smaller risk of going broke than if you run with the crowd. Yet many investors violate this advice every day because they are emotionally incapable of buying low or selling high. Impelled by greed and fear, they run with the crowd instead of thinking for themselves.

It is not all that easy to keep the peapods in mind. Since we never know exactly what is going to happen tomorrow, it is easier to assume that the future will resemble the present than to admit that it may bring some unknown change. A stock that has been going up for a while somehow seems a better buy than a stock that has been heading for the cellar. We assume that a rising price signifies that the company is flourishing and that a falling price signifies that the company is in trouble. Why stick your neck out?

Professionals are just as likely as amateurs to try to play it safe. For example, in December 1994, analysts at the brokerage firm of Sanford C. Bernstein & Co. found that professionals who tend to forecast a higher-than-average growth rate for a company consistently overestimate the actual results, while pessimists consistently underestimate them.a “[O]n average,” the analysts reported, “expectations are not met.”1

The consequences are clear: stocks with rosy forecasts climb to unreal heights while stocks with dismal forecasts drop to unreal lows. Then regression to the mean takes over. The more realistic and stouthearted investors buy as others rush to sell, and sell as others rush to buy. The payoff comes when the actual earnings surprise those who followed the trend.

History tells us of many legendary investors who made fortunes by betting on regression to the mean, by buying low and selling high. Among them are Bernard Baruch, Benjamin Graham, and Warren Buffett. That contrarian position is confirmed by a wealth of academic research.

But the few who made it big by copping the bets of the crowd receive all the attention. We hear little about those investors who tried the same thing and failed, either because they acted too soon or not at all, or because the mean to which they expected stock prices to regress was not the mean to which they actually did regress.

Consider those investors who had the temerity to buy stocks in early 1930, right after the Great Crash, when prices had fallen about 50% from their previous highs. Prices proceeded to fall another 80% before they finally hit bottom in the fall of 1932. Or consider the cautious investors who sold out in early 1955, when the Dow Jones Industrials had finally regained their old 1929 highs and had tripled over the preceding six years. Just nine years later, prices were double both their 1929 and their 1955 highs. In both cases, the anticipated return to “normal” failed to take place: normal had shifted to a new location.

In discussing the issue of whether regression to the mean governs the behavior of the stock market, we are really asking whether stock prices are predictable, and if so under what conditions. No investor can decide what risks to take before answering that question.

There is some evidence that the prices of certain stocks rise “too high” and fall “too low.” In 1985 at the annual meeting of the American Finance Association, economists Richard Thaler and Werner DeBondt presented a paper titled, “Does the Stock Market Overreact?”2 To test whether extreme movements of stock prices in one direction provoke regression to the mean and are subsequently followed by extreme movements in the other direction, they studied the three-year returns of over a thousand stocks from January 1926 to December 1982. They classified the stocks that had gone up by more or had fallen by less than the market average in each three-year period as “winners,” and the stocks that had gone up by less or had fallen by more than the market average as “losers.” They then calculated the average performance of each group over the subsequent three years.

Their findings were unequivocal: “Over the last half-century, loser portfolios . . . outperform the market by, on average, 19.6% thirty-six months after portfolio formation. Winner portfolios, on the other hand, earn [produce returns] about 5.0% less than the market.”3

Although DeBondt and Thaler’s test methods have been subjected to some criticism, their findings have been confirmed by other analysts using different methods. When investors overreact to new information and ignore long-term trends, regression to the mean turns the average winner into a loser and the average loser into a winner. This reversal tends to develop with some delay, which is what creates the profitable opportunity: we could really say that first the market overreacts to short-term news and then underreacts while awaiting new short-term news of a different character.4

The reason is simple enough. Stock prices in general follow changes in company fortunes. Investors who focus excessively on the short run are ignoring a mountain of evidence demonstrating that most surges in earnings are unsustainable. On the other hand, companies that encounter problems do not let matters slide indefinitely. Managers will set to work making the hard decisions to put their company back on track—or will find themselves out of a job, replaced by others more zealous.

Regression to the mean decrees that it could not be otherwise. If the winners kept on winning and the losers kept on losing, our economy would consist of a shrinking handful of giant monopolies and virtually no small companies at all. The once-admired monopolies in Japan and Korea are now going through the opposite process, as regression to the mean in the form of irresistible waves of imports is gradually weakening their economic power.

The track records of professional investment managers are also subject to regression to the mean. There is a strong probability that the hot manager of today will be the cold manager of tomorrow, or at least the day after tomorrow, and vice versa. This does not mean that successful managers will inevitably lose their touch or that managers with poor records will ultimately see the light—though that does tend to happen. Often investment managers lose ground simply because no one style of management stays in fashion forever.

Earlier, in the discussion of the Petersburg Paradox, we noted the difficulty investors had in valuing stocks that seemed to have infinite payoffs (page 107). It was inevitable that the investors’ unlimited optimism would ultimately lift the price of those growth stocks to unrealistic levels. When regression to the mean sent the stocks crashing, even the best manager of growth-stock portfolios could not help but look foolish. A similar fad took over small stock investing in the late 1970s, when academic research demonstrated that small stocks had been the most successful long-run investment despite their greater risk. By 1983, regression to the mean had once more set in, and small stocks under-performed for years afterward. This time, even the best manager of small-stock investment could not help but look foolish.

In 1994, Morningstar, the leading publication on the performance of mutual funds, published the accompanying table, which shows how various types of funds had fared over the five years ending March 1989 and the five years ending March 1994:5

Objective	5 Years to March 1989	5 Years to March 1994
International stocks	20.6%	9.4%
Income	14.3%	11.2%
Growth and income	14.2%	11.9%
Growth	13.3%	13.9%
Small company	10.3%	15.9%
Aggressive growth	8.9%	16.1%
Average	13.6%	13.1%

This is a spectacular demonstration of regression to the mean at work. The average performance in both periods was almost identical, but the swings in results from the first period to the second were enormous. The three groups that did better than average in the first period did worse than average in the second; the three groups that did worse than average in the first period did better than average in the second.

This impressive evidence of regression to the mean might provide some valuable advice to investors who are constantly switching managers. It suggests that the wisest strategy is to dismiss the manager with the best recent track record and to transfer one’s assets to the manager who has been doing the worst; this strategy is no different from selling stocks that have risen furthest and buying stocks that have fallen furthest. If that contrarian strategy is hard to follow, there is another way to accomplish the same thing. Go ahead and follow your natural instincts. Fire the lagging manager and add to the holdings of the winning manager, but wait two years before doing it.

What about the stock market as a whole? Are the popular averages, like the Dow Jones Industrials and the Standard & Poor’s Composite of 500 stocks, predictable?

The charts in Chapter 8 (page 147) show that market performance over periods of a year or more does not look much like a normal distribution, but that performance by the month and by the quarter does, though not precisely. Quetelet would interpret that evidence as proof that stock-price movements in the short run are independent—that today’s changes tell us nothing about what tomorrow’s prices will be. The stock market is unpredictable. The notion of the random walk was evoked to explain why this should be so.

But what about the longer view? After all, most investors, even impatient ones, stay in the market for more than a month, a quarter, or a year. Even though the contents of their portfolios change over time, serious investors tend to keep their money in the stock market for many years, even decades. Does the long run in the stock market really differ from the short run?

If the random-walk view is correct, today’s stock prices embody all relevant information. The only thing that would make them change is the availability of new information. Since we have no way of knowing what that new information might be, there is no mean for stock prices to regress to. In other words, there is no such thing as a temporary stock price—that is, a price that sits in limbo before moving to some other point. That is also why changes are unpredictable.

But there are two other possibilities. If the DeBondt-Thaler hypothesis of overreaction to recent news applies to the market as a whole and not just to individual stocks, regression to the mean in the performance of the major market averages should become visible as longer-term realities make themselves felt. If, on the other hand, investors are more fearful in some economic environments than in others—say, 1932 or 1974 in contrast to 1968 or 1986—stock prices would fall so long as investors are afraid and would rise again as circumstances change and justify a more hopeful view of the future.

Both possibilities argue for ignoring short-term volatility and holding on for the long pull. No matter how the market moves along the way, returns to investors should average out around some kind of long-term normal. If that is the case, the stock market may be a risky place for a matter of months or even for a couple of years, but the risk of losing anything substantial over a period of five years or longer should be small.

Impressive support for this viewpoint appeared in a monograph published in 1995 by the Association for Investment Management & Research—the organization to which most investment professionals belong—and written by two Baylor University professors, William Reichenstein and Dovalee Dorsett.6 On the basis of extensive research, they conclude that bad periods in the market are predictably followed by good periods, and vice versa. This finding is a direct contradiction of the random-walk view, which denies that changes in stock prices are predictable. Stock prices, like the peapods, have shown no tendency to head off indefinitely in one direction or the other.

Mathematics tells us that the variance—a measure of how observations tend to distribute themselves around their average level—of a series of random numbers should increase precisely as the length of the series grows. Observations over three-year periods should show triple the variance of observations over one year, and observations over a decade should show ten times the variance of annual observations. If, on the other hand, the numbers are not random, because regression to the mean is at work; the mathematics works out so that the ratio of the change variance to the time period will be less than one.b

Reichenstein and Dorsett studied the S&P 500 from 1926 to 1993 and found that the variance of three-year returns was only 2.7 times the variance of annual returns; the variance of eight-year returns was only 5.6 times the variance of annual returns. When they assembled realistic portfolios containing a mixture of stocks and bonds, the ratios of variance to time period were even smaller than for portfolios consisting only of stocks.

Clearly, long-run volatility in the stock market is less than it would be if the extremes had any chance of taking over. In the end, and after their flings, investors listen to Galton rather than dancing along behind the Pied Piper.

This finding has profound implications for long-term investors, because it means that uncertainty about rates of return over the long run is much smaller than in the short run. Reichenstein and Dorsett provide a wealth of historical data and projections of future possibilities, but the following passage suggests their principal findings (based on results after adjustment for inflation):7

For a one-year holding period, there is a five percent chance that investors in the stock market will lose at least 25% of their money, and a five percent chance that they will make more than 40%. Over thirty years, on the other hand, there is only a five percent chance that a 100% stock portfolio will grow by less than 20% and a five percent chance that owners of this portfolio could end up over fifty times richer than where they started.

Over time, the difference between the returns on risky securities and conservative investments widens dramatically. Over twenty years, there is only a five percent chance that a portfolio consisting only of long-term corporate bonds would much more than quadruple while there is a fifty percent chance that a 100% equity portfolio would grow at least eightfold.

Yet, this painstaking research gives us no easy prescription for getting rich. We all find it difficult to hang in through thin as well as thick. And Reichenstein and Dorsett tell us only what happened between 1926 and 1993. Tempting as long-term investing appears in light of their calculations, their analysis is 100% hindsight. Worse, even small differences in annual returns over many years produce big differences in the investor’s wealth at the end of the long-run.

The overreaction to new information that DeBondt and Thaler reported in the behavior of stock prices was the result of the human tendency to overweight recent evidence and to lose sight of the long run. After all, we know a lot more about what is happening right now than we can ever know about what will happen at some uncertain date in the future.

Nevertheless, overemphasizing the present can distort reality and lead to unwise decisions and faulty assessments. For example, some observers have deplored what they allege to be a slowdown in productivity growth in the United States over the past quarter-century. Actually, the record over that period is far better than they would lead us to believe. Awareness of regression to the mean would correct the faulty view of the pessimists.

In 1986, Princeton economist William Baumol published an enlightening study of long-run trends in productivity. His data came from 72 countries and reached back to 1870.8 The study focused on what Baumol calls the process of convergence. According to this process, the countries with the lowest levels of productivity in 1870 have had the highest rates of improvement over the years, while the most productive countries in 1870 have exhibited the slowest rates of improvement—the peapods at work again, in other words. The differences in growth rates have slowly but surely narrowed the gap in productivity between the most backward and the most advanced nations as each group has regressed toward the mean.

Over the 110 years covered by Baumol’s analysis, the difference between the most productive nation and the least productive nation converged from a ratio of 8:1 to a ratio of only 2:1. Baumol points out, “. . . what is striking is the apparent implication that only one variable, a country’s 1870 GDP per work-hour, . . . matters to any substantial degree.”9 The factors that economists usually identify as contributing to growth in productivity—free markets, a high propensity to save and invest, and “sound” economic policies—seem to have been largely irrelevant. “Whatever its behavior,” Baumol concludes, each nation was “fated to land close to its predestined position.”10 Here is a worldwide phenomenon that exactly replicates Galton’s small-scale experiments.

Assessments of the performance of the United States change radically when appraised from this perspective. As the nation with the highest GDP per work-hour among industrial countries since the turn of the century, the relatively slow rate of growth in productivity in the United States in recent years should come as no surprise. Each successive technological miracle counts for less as the base from which we measure gets bigger. In fact, Baumol’s data show that the U.S. growth rate in productivity has been “just middling” for the better part of a century, not merely for the past couple of decades. Between 1899 and 1913 it was already slower than the growth rates of Sweden, France, Germany, Italy, and Japan.

Although Japan has had the highest long-run growth rate of all the developed economies, except during the Second World War, Baumol points out that it had the lowest level of output per worker in 1870 and still ranks behind the United States. But the process of convergence proceeds inexorably, as technology advances, as education spreads, and as increasing size facilitates economies of scale.

Baumol suggests that dissatisfaction with the U.S. record since the late 1960s is the result of myopia on the part of commentators who overemphasize recent performance and ignore long-term trends. He points out that the huge jump in U.S. levels of productivity from about 1950 to 1970 was not our preordained destiny, even for a nation as technologically oriented as the United States. Seen in a longer perspective, that leap was only an aberration that roughly offset the sharp decline from historical growth rates suffered during the 1930s and the Second World War.

Even though the subject matter is entirely different, Baumol’s main conclusions echo DeBondt and Thaler:

We cannot understand current phenomena . . . without systematic examination of earlier events which affect the present and will continue to exercise profound effects tomorrow. . . . [T]he long run is important because it is not sensible for economists and policymakers to attempt to discern long-run trends and their outcomes from the flow of short-run developments, which may be dominated by transient conditions.11

Sometimes the long run sets in too late to bail us out, even when regression to the mean is at work. In a famous passage, the great English economist John Maynard Keynes once remarked:

In the long run, we are all dead. Economists set themselves too easy, too useless a task if in the tempestuous seasons they can only tell us that when the storm is long past the ocean will be flat.12

But we are obliged to live in the short run. The business at hand is to stay afloat and we dare not wait for the day the ocean will be flat. Even then its flatness may be only an interlude of unknown duration between tempests.

Dependence on reversion to the mean for forecasting the future tends to be perilous when the mean itself is in flux. The Reichenstein-Dorsettt projections assume that the future will look like the past, but there is no natural law that says it always will. If global warming indeed lies ahead, a long string of hot years will not necessarily be followed by a long string of cold years. If a person becomes psychotic instead of just neurotic, depression may be permanent rather than intermittent. If humans succeed in destroying the environment, floods may fail to follow droughts.

If nature sometimes fails to regress to the mean, human activities, unlike sweet peas, will surely experience discontinuities, and no risk-management system will work very well. Galton recognized that possibility and warned, “An Average is but a solitary fact, whereas if a single other fact be added to it, an entire Normal Scheme, which nearly corresponds to the observed one, starts potentially into existence.”13

Early in this book we commented on the stability of the daily lives of most people century after century. Since the onset of the industrial revolution about two hundred years ago, so many “single other facts” have been added to the “Average” that defining the “Normal Scheme” has become increasingly difficult. When discontinuities threaten, it is perilous to base decisions on established trends that have always seemed to make perfect sense but suddenly do not.

Here are two examples of how people can be duped by overreliance on regression to the mean.

In 1930, when President Hoover declared that “Prosperity is just around the corner,” he was not trying to fool the public with a soundbite or a spin. He meant what he said. After all, history had always supported that view. Depressions had come, but they had always gone.c Except for the period of the First World War, business activity had fallen in only seven years from 1869 until 1929. The single two-year setback during those years was 1907–1908, from a very high point; the average annual decline in real GDP was a modest 1.6%, and that included one decline of 5.5%.

But production fell in 1930 by 9.3% and in 1931 by 8.6%. At the very bottom, in June 1932, GDP was 55% below its 1929 peak, even lower than it had been at the low point of the short-lived depression of 1920. Sixty years of history had suddenly become irrelevant. The trouble stemmed in part from the loss of youthful dynamism over the long period of industrial development; even during the boom of the 1920s, economic growth was below the long-term trend defined by the years from 1870 to 1918. The weakening of forward momentum, combined with a sequence of policy errors here and abroad and the shock of the stock market crash in October 1929, drove prosperity away from the corner it was presumably around.

The second example: In 1959, exactly thirty years after the Great Crash, an event took place that made absolutely no sense in the light of history. Up to the late 1950s, investors had received a higher income from owning stocks than from owning bonds. Every time the yields got close, the dividend yield on common stocks moved back up over the bond yield. Stock prices fell, so that a dollar invested in stocks brought more income than it had brought previously.

That seemed as it should be. After all, stocks are riskier than bonds. Bonds are contracts that specify precisely when the borrower must repay the principal of the debt and provide the schedule of interest payments. If borrowers default on a bond contract, they end up in bankruptcy, their credit ruined, and their assets under the control of creditors.

With stocks, however, the shareholders’ claim on the company’s assets has no substance until after the company’s creditors have been satisfied. Stocks are perpetuities: they have no terminal date on which the assets of the company must be distributed to the owners. Moreover, stock dividends are paid at the pleasure of the board of directors; the company has no obligation ever to pay dividends to the stockholders. Total dividends paid by publicly held companies were cut on nineteen occasions between 1871 and 1929; they were slashed by more than 50% from 1929 to 1933 and by about 40% in 1938.

So it is no wonder that investors bought stocks only when they yielded a higher income than bonds. And no wonder that stock prices fell every time the income from stocks came close to the income from bonds.

Until 1959, that is. At that point, stock prices were soaring and bond prices were falling. This meant that the ratio of bond interest to bond prices was shooting up and the ratio of stock dividends to stock prices was declining. The old relationship between bonds and stocks vanished, opening up a gap so huge that ultimately bonds were yielding more than stocks by an even greater margin than when stocks had yielded more than bonds.

The cause of this reversal could not have been trivial. Inflation was the main factor that distinguished the present from the past. From 1800 to 1940, the cost of living had risen an average of only 0.2% a year and had actually declined on 69 occasions. In 1940 the cost-of-living index was only 28% higher than it had been 140 years earlier. Under such conditions, owning assets valued at a fixed number of dollars was a delight; owning assets with no fixed dollar value was highly risky.

The Second World War and its aftermath changed all that. From 1941 to 1959, inflation averaged 4.0% a year, with the cost-of-living index rising every year but one. The relentlessly rising price level transformed bonds from a financial instrument that had appeared inviolate into an extremely risky investment. By 1959, the price of the 2 1/2% bonds the Treasury had issued in 1945 had fallen from $1,000 to $820—and that $820 bought only half as much as in 1949!

Meanwhile, stock dividends took off on a rapid climb, tripling between 1945 and 1959, with only one year of decline—and even that a mere 2%. No longer did investors perceive stocks as a risky asset whose price and income moved unpredictably. The price paid for today’s dividend appeared increasingly irrelevant. What mattered was the rising stream of dividends that the future would bring. Over time, those dividends could be expected to exceed the interest payments from bonds, with a commensurate rise in the capital value of the stocks. The smart move was to buy stocks at a premium because of the opportunities for growth and inflation hedging they provided, and to pass up bonds with their fixed-dollar yield.

Although the contours of this new world were visible well before 1959, the old relationships in the capital markets tended to persist as long as people with memories of the old days continued to be the main investors. For example, my partners, veterans of the Great Crash, kept assuring me that the seeming trend was nothing but an aberration. They promised me that matters would revert to normal in just a few months, that stock prices would fall and bond prices would rally.

I am still waiting. The fact that something so unthinkable could occur has had a lasting impact on my view of life and on investing in particular. It continues to color my attitude toward the future and has left me skeptical about the wisdom of extrapolating from the past.

How much reliance, then, can we place on regression to the mean in judging what the future will bring? What are we to make of a concept that has great power under some conditions but leads to disaster under others?

Keynes admitted that “as living and moving beings, we are forced to act. . . [even when]our existing knowledge does not provide a sufficient basis for a calculated mathematical expectation.”14 With rules of thumb, experience, instinct, and conventions—in other words, gut—we manage to stumble from the present into the future. The expression “conventional wisdom,” first used by John Kenneth Galbraith, often carries a pejorative sense, as though what most of us believe is inevitably wrong. But without conventional wisdom, we could make no long-run decisions and would have trouble finding our way from day to day.

The trick is to be flexible enough to recognize that regression to the mean is only a tool; it is not a religion with immutable dogma and ceremonies. Used to make mechanical extrapolations of the past, as President Hoover or my older associates used it, regression to the mean is little more than mumbo-jumbo. Never depend upon it to come into play without constantly questioning the relevance of the assumptions that support the procedure. Francis Galton spoke wisely when he urged us to “revel in more comprehensive views” than the average.

a I am not related to Sanford Bernstein, by the way.

b Opposite tendencies are apparent in the historical record of interest rates, which reflect “aversion” to the mean. A trend once in place has a higher probability of continuing than reversing. Over two-year periods, the variance of the yield on 90-day Treasury bills is 2.2 times annual data; over eight-year periods, the variance is nearly 32 times the annual data; for longer-term interest rates, the pattern is similar but muted.

c They called depressions “panics” in those days; “depression” was a euphemism coined for the occasion. Later, “recession” became the accepted euphemism. One can only speculate on how deep a recession would have to be before the experts would decide to call it a “depression.”

Notes

1. Sanford C. Bernstein & Co., 1995.

2. DeBondt and Thaler, 1986.

3. Ibid.

4. See Dreman and Berry, 1995.

5. Morningstar Mutual Funds, April 1, 1994.

6. Reichenstein and Dorsett, 1995, pp. 46–47.

7. These conclusions are distilled from Reichenstein and Dorsett, 1995, Table 11, p. 32.

8. Baumol, 1986. For an extended analysis of this process, see Baumol, Nelson, and Wolff, 1994.

9. Baumol, 1986, p. 1077.

10. Ibid., p. 1077.

11. Ibid., p. 1084.

12. Keynes, 1924, p. 88.

13. Forrest, 1974, pp. 201–202.

14. Despite help from others, I have been unable to locate the source of this quotation, which has long been in my files. A similar comment appears in Keynes, 1936, pp. 152–153.