by Nate Silver
Or say that you are considering buying another type of asset: a mortgage-backed security. This type of commodity may be even harder to value. But the more investors buy them—and the more the ratings agencies vouch for them—the more confidence you might have that they are safe and worthwhile investments. Hence, you have a positive feedback—and the potential for a bubble.
A negative feedback did eventually rein in the housing market: there weren’t any Americans left who could afford homes at their current prices. For that matter, many Americans who had bought homes couldn’t really afford them in the first place, and soon their mortgages were underwater. But this was not until trillions of dollars in bets, highly leveraged and impossible to unwind without substantial damage to the economy, had been made on the premise that all the people buying these assets couldn’t possibly be wrong.
“We had too much greed and too little fear,” Summers told me in 2009. “Now we have too much fear and too little greed.”
Act III: This Time Wasn’t Different
Once the housing bubble had burst, greedy investors became fearful ones who found uncertainty lurking around every corner. The process of disentangling a financial crisis—everyone trying to figure out who owes what to whom—can produce hangovers that persist for a very long time. The economists Carmen Reinhart and Kenneth Rogoff, studying volumes of financial history for their book This Time Is Different: Eight Centuries of Financial Folly, found that financial crises typically produce rises in unemployment that persist for four to six years.86 Another study by Reinhart, which focused on more recent financial crises, found that ten of the last fifteen countries to endure one had never seen their unemployment rates recover to their precrisis levels.87 This stands in contrast to normal recessions, in which there is typically above-average growth in the year or so following the recession88 as the economy reverts to the mean, allowing employment to catch up quickly. Yet despite its importance, many economic models made no distinction between the financial system and other parts of the economy.
Reinhart and Rogoff’s history lesson was one that the White House might have done more to heed. Soon, they would be responsible for their own notoriously bad prediction.
In January 2009, as Barack Obama was about to take the oath of office, the White House’s incoming economic team—led by Summers and Christina Romer, the chair of the Council of Economic Advisers—were charged with preparing the blueprint for a massive stimulus package that was supposed to make up for the lack of demand among consumers and businesses. Romer thought that $1.2 trillion in stimulus was called for.89 Eventually, the figure was revised downward to about $800 billion after objections from the White House’s political team that a trillion-dollar price would be difficult to sell to Congress.
To help pitch the Congress and the country on the stimulus, Romer and her colleagues prepared a memo90 outlining the depth of the crisis and what the stimulus might do to ameliorate it. The memo prominently featured a graphic predicting how the unemployment rate would track with and without the stimulus. Without the stimulus, the memo said, the unemployment rate, which had been 7.3 percent when last reported in December 2008, would peak at about 9 percent in early 2010. But with the stimulus, employment would never rise above 8 percent and would begin to turn downward as early as July 2009.
Congress passed the stimulus on a party-line vote in February 2009. But unemployment continued to rise—to 9.5 percent in July and then to a peak of 10.1 percent in October 2009. This was much worse than the White House had projected even under the “no stimulus” scenario. Conservative bloggers cheekily updated Romer’s graphic every month—but with the actual unemployment rate superimposed on the too-cheery projections (figure 1-6).
FIGURE 1-6: WHITE HOUSE ECONOMIC PROJECTIONS, JANUARY 2009
People see this graphic now and come to different—and indeed entirely opposite—conclusions about it. Paul Krugman, who had argued from the start that the stimulus was too small,91 sees it as proof that the White House had dramatically underestimated the shortfall in demand. “The fact that unemployment didn’t come down much in the wake of this particular stimulus means that we knew we were facing one hell of a shock from the financial crisis,” he told me. Other economists, of course, take the graph as evidence that the stimulus had completely failed.92
The White House can offer its version of S&P’s “everyone else made the same mistake” defense. Its forecasts were largely in line with those issued by independent economists at the time.93 Meanwhile, the initial economic statistics had significantly underestimated the magnitude of the crisis.94 The government’s first estimate—the one available to Romer and Summers at the time the stimulus was being sold—was that GDP had declined at a rate of 3.8 percent in the fall of 2008.95 In fact, the financial crisis had taken more than twice as large a bite out of the economy. The actual rate of GDP decline had been closer to 9 percent,96 meaning that the country was about $200 billion poorer than the government first estimated.
Perhaps the White House’s more inexcusable error was in making such a precise-seeming forecast—and in failing to prepare the public for the eventuality that it might be wrong. No economist, whether in the White House or elsewhere, has been able to predict the progress of major economic indicators like the unemployment rate with much success. (I take a more detailed look at macroeconomic forecasting in chapter 6.) The uncertainty in an unemployment rate forecast97 made during a recession had historically been about plus or minus 2 percent.98 So even if the White House thought 8 percent unemployment was the most likely outcome, it might easily enough have wound up in the double digits instead (or it might have declined to as low as 6 percent).
There is also considerable uncertainty about how effective stimulus spending really is. Estimates of the multiplier effect—how much each dollar in stimulus spending contributes to growth—vary radically from study to study,99 with some claiming that $1 in stimulus spending returns as much as $4 in GDP growth and others saying the return is just 60 cents on the dollar. When you layer the large uncertainty intrinsic to measuring the effects of stimulus atop the large uncertainty intrinsic to making macroeconomic forecasts of any kind, you have the potential for a prediction that goes very badly.
What the Forecasting Failures Had in Common
There were at least four major failures of prediction that accompanied the financial crisis.
The housing bubble can be thought of as a poor prediction. Homeowners and investors thought that rising prices implied that home values would continue to rise, when in fact history suggested this made them prone to decline.
There was a failure on the part of the ratings agencies, as well as by banks like Lehman Brothers, to understand how risky mortgage-backed securities were. Contrary to the assertions they made before Congress, the problem was not that the ratings agencies failed to see the housing bubble. Instead, their forecasting models were full of faulty assumptions and false confidence about the risk that a collapse in housing prices might present.
There was a widespread failure to anticipate how a housing crisis could trigger a global financial crisis. It had resulted from the high degree of leverage in the market, with $50 in side bets staked on every $1 that an American was willing to invest in a new home.
Finally, in the immediate aftermath of the financial crisis, there was a failure to predict the scope of the economic problems that it might create. Economists and policy makers did not heed Reinhart and Rogoff’s finding that financial crises typically produce very deep and long-lasting recessions.
There is a common thread among these failures of prediction. In each case, as people evaluated the data, they ignored a key piece of context:
The confidence that homeowners had about housing prices may have stemmed from the fact that there had not been a substantial decline in U.S. housing prices in the recent past. However, there had never before been such a widespread increase in U.S. housing prices like the one that preceded the collapse.
The confide
nce that the banks had in Moody’s and S&P’s ability to rate mortgage-backed securities may have been based on the fact that the agencies had generally performed competently in rating other types of financial assets. However, the ratings agencies had never before rated securities as novel and complex as credit default options.
The confidence that economists had in the ability of the financial system to withstand a housing crisis may have arisen because housing price fluctuations had generally not had large effects on the financial system in the past. However, the financial system had probably never been so highly leveraged, and it had certainly never made so many side bets on housing before.
The confidence that policy makers had in the ability of the economy to recuperate quickly from the financial crisis may have come from their experience of recent recessions, most of which had been associated with rapid, “V-shaped” recoveries. However, those recessions had not been associated with financial crises, and financial crises are different.
There is a technical term for this type of problem: the events these forecasters were considering were out of sample. When there is a major failure of prediction, this problem usually has its fingerprints all over the crime scene.
What does the term mean? A simple example should help to explain it.
Out of Sample, Out of Mind: A Formula for a Failed Prediction
Suppose that you’re a very good driver. Almost everyone thinks they’re a good driver,100 but you really have the track record to prove it: just two minor fender benders in thirty years behind the wheel, during which time you have made 20,000 car trips.
You’re also not much of a drinker, and one of the things you’ve absolutely never done is driven drunk. But one year you get a little carried away at your office Christmas party. A good friend of yours is leaving the company, and you’ve been under a lot of stress: one vodka tonic turns into about twelve. You’re blitzed, three sheets to the wind. Should you drive home or call a cab?
That sure seems like an easy question to answer: take the taxi. And cancel your morning meeting.
But you could construct a facetious argument for driving yourself home that went like this: out of a sample of 20,000 car trips, you’d gotten into just two minor accidents, and gotten to your destination safely the other 19,998 times. Those seem like pretty favorable odds. Why go through the inconvenience of calling a cab in the face of such overwhelming evidence?
The problem, of course, is that of those 20,000 car trips, none occurred when you were anywhere near this drunk. Your sample size for drunk driving is not 20,000 trips but zero, and you have no way to use your past experience to forecast your accident risk. This is an example of an out-of-sample problem.
As easy as it might seem to avoid this sort of problem, the ratings agencies made just this mistake. Moody’s estimated the extent to which mortgage defaults were correlated with one another by building a model from past data—specifically, they looked at American housing data going back to about the 1980s.101 The problem is that from the 1980s through the mid-2000s, home prices were always steady or increasing in the United States. Under these circumstances, the assumption that one homeowner’s mortgage has little relationship to another’s was probably good enough. But nothing in that past data would have described what happened when home prices began to decline in tandem. The housing collapse was an out-of-sample event, and their models were worthless for evaluating default risk under those conditions.
The Mistakes That Were Made—and What We Can Learn from Them
Moody’s was not completely helpless, however. They could have come to some more plausible estimates by expanding their horizons. The United States had never experienced such a housing crash before—but other countries had, and the results had been ugly. Perhaps if Moody’s had looked at default rates after the Japanese real estate bubble, they could have had some more realistic idea about the precariousness of mortgage-backed securities—and they would not have stamped their AAA rating on them.
But forecasters often resist considering these out-of-sample problems. When we expand our sample to include events further apart from us in time and space, it often means that we will encounter cases in which the relationships we are studying did not hold up as well as we are accustomed to. The model will seem to be less powerful. It will look less impressive in a PowerPoint presentation (or a journal article or a blog post). We will be forced to acknowledge that we know less about the world than we thought we did. Our personal and professional incentives almost always discourage us from doing this.
We forget—or we willfully ignore—that our models are simplifications of the world. We figure that if we make a mistake, it will be at the margin.
In complex systems, however, mistakes are not measured in degrees but in whole orders of magnitude. S&P and Moody’s underestimated the default risk associated with CDOs by a factor of two hundred. Economists thought there was just a 1 in 500 chance of a recession as severe as what actually occurred.
One of the pervasive risks that we face in the information age, as I wrote in the introduction, is that even if the amount of knowledge in the world is increasing, the gap between what we know and what we think we know may be widening. This syndrome is often associated with very precise-seeming predictions that are not at all accurate. Moody’s carried out their calculations to the second decimal place—but they were utterly divorced from reality. This is like claiming you are a good shot because your bullets always end up in about the same place—even though they are nowhere near the target (figure 1-7).
FIGURE 1-7: ACCURACY VERSUS PRECISION
Financial crises—and most other failures of prediction—stem from this false sense of confidence. Precise forecasts masquerade as accurate ones, and some of us get fooled and double-down our bets. It’s exactly when we think we have overcome the flaws in our judgment that something as powerful as the American economy can be brought to a screeching halt.
2
ARE YOU SMARTER THAN A TELEVISION PUNDIT?
For many people, political prediction is synonymous with the television program The McLaughlin Group, a political roundtable that has been broadcast continually each Sunday since 1982 and parodied by Saturday Night Live for nearly as long. The show, hosted by John McLaughlin, a cantankerous octogenarian who ran a failed bid for the United States Senate in 1970, treats political punditry as sport, cycling through four or five subjects in the half hour, with McLaughlin barking at his panelists for answers on subjects from Australian politics to the prospects for extraterrestrial intelligence.
At the end of each edition of The McLaughlin Group, the program has a final segment called “Predictions,” in which the panelists are given a few seconds to weigh in on some matter of the day. Sometimes, the panelists are permitted to pick a topic and make a prediction about anything even vaguely related to politics. At other times, McLaughlin calls for a “forced prediction,” a sort of pop quiz that asks them their take on a specific issue.
Some of McLaughlin’s questions—say, to name the next Supreme Court nominee from among several plausible candidates—are difficult to answer. But others are softballs. On the weekend before the 2008 presidential election, for instance, McLaughlin asked his panelists whether John McCain or Barack Obama was going to win.1
That one ought not to have required very much thought. Barack Obama had led John McCain in almost every national poll since September 15, 2008, when the collapse of Lehman Brothers had ushered in the worst economic slump since the Great Depression. Obama also led in almost every poll of almost every swing state: in Ohio and Florida and Pennsylvania and New Hampshire—and even in a few states that Democrats don’t normally win, like Colorado and Virginia. Statistical models like the one I developed for FiveThirtyEight suggested that Obama had in excess of a 95 percent chance of winning the election. Betting markets were slightly more equivocal, but still had him as a 7 to 1 favorite.2
But McLaughlin’s first panelist, Pat Buchanan, dodged the question. “The undecided
s will decide this weekend,” he remarked, drawing guffaws from the rest of the panel. Another guest, the Chicago Tribune’s Clarence Page, said the election was “too close to call.” Fox News’ Monica Crowley was bolder, predicting a McCain win by “half a point.” Only Newsweek’s Eleanor Clift stated the obvious, predicting a win for the Obama-Biden ticket.
The following Tuesday, Obama became the president-elect with 365 electoral votes to John McCain’s 173—almost exactly as polls and statistical models had anticipated. While not a landslide of historic proportions, it certainly hadn’t been “too close to call”: Obama had beaten John McCain by nearly ten million votes. Anyone who had rendered a prediction to the contrary had some explaining to do.
There would be none of that on The McLaughlin Group when the same four panelists gathered again the following week.3 The panel discussed the statistical minutiae of Obama’s win, his selection of Rahm Emanuel as his chief of staff, and his relations with Russian president Dmitry Medvedev. There was no mention of the failed prediction—made on national television in contradiction to essentially all available evidence. In fact, the panelists made it sound as though the outcome had been inevitable all along; Crowley explained that it had been a “change election year” and that McCain had run a terrible campaign—neglecting to mention that she had been willing to bet on that campaign just a week earlier.
