In sum, the belief system defense hypothesis ties together many strands of evidence. Defensive cognitions are activated when forecasters most need them. And endorsement of defensive cognitions—in aggregate—distinguishes better from worse Bayesian belief updaters. But it is worth stressing that our focus has been psychological, not epistemological. We postpone a thorough discussion of the defensibility of defenses until chapter 6.15
HINDSIGHT EFFECTS: ARTIFACT AND FACT
When we recontacted experts to gauge their reactions to the confirmation or disconfirmation of their predictions, we frequently ran into an awkward problem. Our records of the probability judgments made at the beginning of forecast periods often disagreed with experts’ recollections of what they predicted. There was, moreover, a systematic bias in these recollections. Experts claimed that they assigned higher probabilities to outcomes that materialized than they did. From a narrowly Bayesian perspective, this 20/20 hindsight effect was a methodological nuisance: it is hard to ask someone why they got it wrong when they think they got it right. But from a psychological perspective, the hindsight effect is intriguing in its own right. What counts as fact or artifact depends on the goals of inquiry.16
For purposes of assessing Bayesian belief updating, it was necessary to remind experts, as delicately as possible, of their original predictions. Only then could we pose the question: given your earlier position and given subsequent events, do you want to change your mind? But the opportunity to build on psychological work on hindsight bias was irresistible,17 so we decided, in six cases, to ask experts to recollect their positions prior to receiving the reminder from our records. Those cases were the Soviet Union/Russia (1988–1993), South Africa (1988–1989, 1993–1994), Canada (1993–1998), China (1992–1997), European Monetary Union (1992–1997, 1998–2001), and the Korean peninsula (1992–1997).
Figure 4.2 shows that we replicated two well-established laboratory effects: (a) widespread susceptibility to the hindsight bias; (b) more pronounced hindsight bias among hedgehogs.18 When we asked experts to recall their original likelihood judgments, experts, especially hedgehogs, often claimed that they attached higher probabilities to what subsequently happened than they did. Figure 4.2 also adds a new wrinkle. Experts shortchanged the competition. When experts recalled the probabilities they once thought their most influential rivals would assign to the future that materialized, they imputed lower probabilities after the fact than before the fact. In effect, experts displayed both the classic hindsight effect (claiming more credit for predicting the future than they deserved) and the mirror-image hindsight effect (giving less credit to their opponents for anticipating the future than they deserved).
Figure 4.2. The relative magnitude of the hindsight bias when experts try to recall: (a) the probabilities that they themselves once assigned to possible futures (own perspective); and (b) the probabilities that they once said intellectual rivals would assign the same possible futures. Positive scores on the y-axis mean a “knew it all along” or positive hindsight bias; negative scores mean a “never would have known it” or negative hindsight bias. Hedgehogs show stronger “I know it all along” bias as well as the complementary “They never would have known it all along” bias.
Hindsight effects are undoubtedly partly rooted in the simple human desire to portray oneself as smarter, and to portray rivals as dumber, than is the case. In the most cynical variant of this view, people knew what their prior positions were and dissembled. This explanation cannot, however, explain all the evidence. Experimenters find the memory-distortion effect even when, as was the case here, people know the researcher has access to the correct answers and can detect false self-promotion.
A fuller explanation must trace hindsight bias to a deeper cause capable of producing genuine self-deception: the largely unconscious cognitive processing that is automatically activated whenever we learn what has happened and that allows us to rapidly assimilate the observed outcome into our network of beliefs about what makes things happen. People manage to convince themselves, sometimes within milliseconds, that they knew it all along. This explanation dovetails nicely with the greater propensity of hedgehogs to exhibit the effect. Hedgehogs should place a higher value on cognitive continuity, on minimizing gaps between their current and past opinions. Hedgehogs should thus be more predisposed—by dint of their cognitive and emotional makeup—to assimilate outcomes, as soon as they become known, into their favorite explanatory categories.
This explanation also helps to account for why the hindsight bias was so selective, inflating the powers of foresight only of like-minded, right-thinking observers and deflating those of one’s rivals. The world did not become retrospectively foreseeable for everyone. The clarity of ex post determinism was reserved for those with the correct worldviews.19
Discerning readers might, however, sense a contradiction between two results: the greater susceptibility of hedgehogs to hindsight effects and the greater interest of hedgehogs in invoking close-call counterfactuals that rescue forecasts from disconfirmation. Hindsight bias portrays what happened as, in retrospect, inevitable: hence, something one should have foreseen. By contrast, close-call counterfactuals portray what happened as highly contingent: hence, unforeseeable. How could the same people invoke such contradictory defenses?
The short answer is that the same people did not usually invoke these two defenses. Although the correlation between being a hedgehog and endorsing close-call counterfactuals is statistically significant (.36), as is the correlation between being a hedgehog and hindsight bias (.29), the correlation between endorsing close-call counterfactuals and susceptibility to hindsight bias is a meager .11.20
The pieces of the puzzle now fit together. The hindsight bias and belief system defenses are complementary strategies of reinforcing our self-images as rational beings: hindsight bias pumps up the likelihood we recall attaching to futures that materialized, whereas the belief system defenses stress the reasonableness of the opinions that once led us to think other things would happen. Why change one’s mind in response to the unexpected when one can convince oneself that one saw it coming all along, and to the degree one must concede an element of surprise, one can still argue that one’s earlier expectations were at least “almost right”?
LINKING PROCESS AND CORRESPONDENCE CONCEPTIONS OF GOOD JUDGMENT
We can close the circumstantial circle of correlational evidence. Chapter 3 used a variety of indicators to show that foxes attached more realistic probability estimates to possible futures than did hedgehogs. The best-fitting explanation traced the performance differential to the different reasoning styles of foxes and hedgehogs. The fox advantages in forecasting accuracy disappeared when we controlled for the influence of styles of reasoning—the tendency of foxes to report thoughts that were both self-critical in content and dialectical in structure, alternating between advancing reasons for expecting an outcome, then shifting into critiques of that reasoning and generating arguments for expecting opposing outcomes, and finally shifting into self-reflective efforts to forge viable syntheses of the clashing considerations. Foxes were, in psychological jargon, more “integratively complex” than hedgehogs.21
Chapter 4 has shown that, although foxes were far from perfect belief updaters by Bayesian standards, they were more likely to change their minds in the right direction and to the right degree when the unexpected happened, as often it did. The best-fitting explanation traced the performance differential to two factors: (a) the greater reliance of hedgehogs on belief system defenses such as close-call counterfactuals and off-on-timing that gave them intellectual cover for arguing that no serious mistake had been made and for refusing to abandon prior positions; (b) the greater susceptibility of hedgehogs to hindsight bias that allowed them to maintain with conviction the fiction that their original predictions were not all that far off the mark.
In both chapters, the root cause of hedgehog underperformance has been a reluctance to entertain the possibility that they
might be wrong. This interpretation ties together additional loose ends. The integrative-complexity index, the measure of self-critical reasoning that predicted correspondence indicators of good judgment in chapter 3, also predicted the tendencies to resist changing probability estimates in response to the unexpected, to mobilize belief system defenses that justify those prior estimates, and to exhibit the hindsight bias in recall of past positions (r’s = .26, .35, and .27). The same, self-justifying, hedgehog style of reasoning that translated into poorer forecasting performance translated into poorer belief-updating performance. Inspection of correlation matrices also reveals that better belief updaters had better forecasting records, especially calibration scores. These correlations are also not stunningly large—ranging between .25 and .36—but they are consistently significant. When we formally factor the fallibility of our measures into the equation by correcting for attenuation of correlations due to unreliability, the convergence of evidence is all the more impressive.
Figure 4.3. A conceptual framework that builds on figure 3.5. It inserts what we learned in chapter 4 about the tendency of integratively complex foxes to be better Bayesian belief updaters. It posits that integratively complex thinkers enjoy advantages in forecasting skill, in part, because they are more willing to change their minds in response to the unexpected, in part, because they are more likely to remember past mistakes (reduced hindsight bias), and in part, because they are more likely to see the case for expecting opposing outcomes and thus make cautious probability estimates. Those quicker to acknowledge past mistakes are less prone to make future mistakes.
Figure 4.3 integrates these new findings into the conceptual framework for good judgment laid out in Figure 3.6. The effects of a foxlike, integratively complex style of reasoning on forecasting skill are now mediated by the tendencies both to hedge probability bets and to be better belief updaters.
But have we learned anything surprising about good judgment in the late twentieth century? Here we run into the defining dilemma of the social scientist: the choice between being judged either obvious or obviously wrong. Intellectual foxes will see the current results as a rather unsurprising, although still welcome, vindication of what they have been saying all along. These scholars have repeatedly traced the psychological roots of intelligence failures to an unwillingness to be sufficiently self-critical, to reexamine underlying assumptions, to question dogma, and to muster the imagination to think daringly about options that others might ridicule.22 Political observers are well advised to heed Oliver Cromwell’s (unintentionally ironic but intentionally ominous) advice to his foes in 1650: “I beseech you, in the bowels of Christ, think it possible you may be mistaken.”23
Hedgehog commentators will not be so welcoming. They will see the current results as deeply misleading, for reasons laid out in chapter 6: policy and intelligence failures can more often be traced to paralysis and self-doubt induced by excessive self-criticism. To paraphrase Dean Acheson’s admonition to Richard Neustadt during the Cuban missile crisis, “I know your theory, professor. You think the president should be warned. But you are wrong. He needs confidence.” So-called biases such as over-confidence and belief perseverance put sorely needed backbone in policy.
Arguments over the right process prescriptions are rooted as much in temperament as in evidence. Still, evidence is not irrelevant. The data tell us something that one camp suspected was true, another camp suspected was false, but neither camp had investigated systematically because both camps were convinced of the blindingly obvious truth of their positions. It must be left to posterity to judge whether the results to this point are obvious or obviously wrong.
1 From the standpoint of work on conversational norms, the order of questioning used in this study—posing the questions about alternative perspectives right before the bottom-line probability assessment—constitutes a particularly tough test of the notion that forecasters are oblivious to alternative perspectives. Recency is often a cue in conversations that the speaker considers the just-discussed information to be relevant (H. P. Grice, “Logic and Conversation,” in Syntax and Semantics, vol. 3, Speech Acts, ed. P. Cole and J. L. Morgan [New York: Academic Press, 1975], 41–58).
2 This result converges with several lines of experimental research, including (a) work on pseudo-diagnosticity that shows that people give too little weight to the denominator of likelihood ratios; (b) work on egocentricity biases. See H. R. Arkes and M. Rothbart, “Memory, Retrieval, and Contingency Judgments,” Journal of Personality and Social Psychology 49 (1985): 598–606; B. Fischhoff and R. Beyth-Marom, “Hypothesis Evaluation from a Bayesian Perspective,” Psychological Review 90 (1983): 239–60; L. Ross and D. Griffin, “Subjective Construal, Social Inference, and Human Misunderstanding,” in Advances in Experimental Social Psychology, vol. 24, ed. M. Zanna (New York: Academic Press, 1991), 319–59; R. E. Nisbett and L. Ross, Human Inference: Strategies and Shortcomings of Social Judgment (Englewood Cliffs, NJ: Prentice-Hall, 1980).
3 Fischhoff and Beyth-Marom, “Hypothesis Evaluation from a Bayesian Perspective.”
4 This alternative format “depersonalized” hypothesis testing by no longer pitting experts against their “rivals.” For example, in addition to asking experts on Western Europe in 1998 to judge the likelihood of various countries adopting the euro in the next three to five years, we asked them to judge the truth or falsity or the hypothesis “there is a long-term process of economic and political integration at work in Europe,” and then make two sets of conditional-likelihood judgments: (a) assume for sake of argument that the hypothesis is definitely (100 percent) true and judge the conditional likelihood of various countries adopting the euro by January 2001 or 2008; (b) assume the opposite and then make the same conditional likelihood judgments. The results from the two different formats were sufficiently similar to justify pooling the data.
5 Some researchers have concluded that people are such bad Bayesians that they are not Bayesians at all (Fischhoff and Beyth-Marom, “Hypothesis Evaluation from a Bayesian Perspective”). Early work on cognitive conservatism showed that people clung too long to the only information they initially had—information that typically took the form of base rates of variously colored poker chips that defined judges’ “priors.” See L. D. Phillips and D. Edwards, “Conservatism in a Simple Probability Inference task,” Journal of Experimental Psychology 54 (1966): 346–54. But later work showed that people often ignored base rates when they could construct plausible causal stories from case-specific information (Nisbett and Ross, Human Inference). This confusion is readily resolved here. Our forecasters did cling too long to their prior hypotheses (replicating “cognitive conservatism”), but those hypotheses were grounded in strong beliefs about political causality at work in specific cases, not in statistical summaries of how often outcomes occur in specified regions or periods. Indeed, these ideological schemata often cause experts to overpredict low-frequency outcomes, as noted in chapter 3, and this can be viewed as a form of base-rate neglect. My experience is that base rates shape political judgments mostly when people have no other information or have imbued the base rate with causal potency (e.g., sweeping stereotypes about Russians or Islam).
6 A. W. Kruglanski and D. M. Webster, “Motivated Closing of the Mind: ‘Seizing’ and ‘Freezing,’ ” Psychological Review 103 (1996): 263–68.
7 The average likelihood ratio for hedgehogs was 3.2:1, whereas the ratio for foxes was 2.3:1. Hedgehogs were also twice as likely as foxes to assign a zero likelihood to competing prior hypotheses (approximately 9 percent versus 4 percent of the time). This created a technical problem. Belief-updating equations become undefined whenever forecasters assign a hypothesis a value of zero (hence the need for recoding zero as .01 and 1.0 as .99). Once someone commits to the view something is impossible, no amount of evidence—within a Bayesian framework—can move them. One can view this outcome as a failure of the framework or as a failure of respondents to appreciate how closed-minded they are when they us
e zero on the subjective probability scale.
8 For an experimental demonstration of an analogous effect, see C. Lord, M. Lepper, and E. Preston, “Considering the Opposite: A Corrective Strategy for Social Judgement,” Journal of Personality and Social Psychology 46 (1984): 1254–66; C. Lord, L. Ross, and M. Lepper, “Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence,” Journal of Personality and Social Psychology 37 (1979): 2098–2109.
9 “Conservatism” carries no ideological connotation here. It refers to conserving existing mental structures, regardless of content. Liberals can be, and often are, as “guilty” of cognitive conservatism as conservatives. Some researchers find cognitive conservatism effects to be ideologically symmetrical: Nisbett and Ross, Human Inference; Z. Kunda, Social Cognition: Making Sense of People (Cambridge: MIT Press, 1999); P. E. Tetlock and A. Levi, “Attribution Bias: On the Inconclusiveness of the Cognition-Motivation Debate,” Journal of Experimental Psychology 18 (1982): 68–88. Other researchers, however, find that ideological conservatives tend to be more cognitively conservative. See P. E. Tetlock, “Cognitive Structural Analysis of Political Rhetoric,” in Political Psychology: A Reader, ed. S. Iyengar and W. J. McGuire (Durham, NC: Duke University Press, 1992), 380–407.
Expert Political Judgment Page 25
