Expert Political Judgment

by Philip E. Tetlock

  These asymmetries do suggest, though, there may be some, albeit limited, potential for hedgehogs to “catch up” via value adjustments that invoke the “I made the right mistake” defense (a point we revisit in chapter 6). For now, it must suffice to note that hedgehogs’ tendency to assign too high probabilities to lower-frequency outcomes fits snugly within the emerging cognitive-process account of the data. We discover just how snugly when we explore the linkages between forecasting accuracy (either in aggregate or broken down into types of under- and overprediction) and the thought processes that forecasters reported when we called on them to explain their predictions.

  We asked all participants twice (one inside and once outside their area of expertise): Why are you, on balance, optimistic, pessimistic, or mixed in your assessment of the future of x?” Our analyses of the resulting thought protocols targeted two properties of thinking styles that, if the cognitive-process account is correct, should distinguish foxes from hedgehogs and “explain” their differential forecasting performance. The key targets were as follows:

  a. the evaluative differentiation index that taps into how often people use qualifying conjunctions such as “however,” “but,”—and so on, that imply thoughts are in tension with one another.

  b. the conceptual integration index that taps into how often people struggle to specify guidelines for resolving tensions among differentiated cognitions (e.g., grappling with trade-offs or acknowledging that sensible people, not just fools and scoundrels, could wind up viewing the same problem in clashing ways).

  The Methodological Appendix provides coding details, as well as the rationale for combining the two measures into an integrative complexity index.15

  Our analyses of forecasters’ arguments reinforced the cognitive-process account of hedgehog-fox effects in several ways:

  a. As one would expect if foxes and hedgehogs were equally knowledgeable but differed in their tolerance of dissonance and motivation to generate integrative cognitions, we find that (i) hedgehogs and foxes do not differ in the total number of thoughts they generate; (ii) they do differ on both evaluative differentiation and cognitive integration, each of which rises as we move from the “pure” hedgehog to the “pure” fox quartile of respondents. The composite-process measure, integrative complexity, correlated .38 with the hedgehog-fox scale.

  b. As one would expect if these differences in styles of thinking were linked not only to the hedgehog-fox measure but also to forecasting skill, integrative complexity correlates with aggregate accuracy indicators, such as calibration (.32) and discrimination (.24), as well as with the directional indicator of tendency to overpredict change (.33).

  c. As one would expect if these differences in styles of thinking partly mediate the connection between more foxlike cognitive styles and forecasting skill, the correlations between the hedgehog-fox scale and both calibration and overprediction of change take a significant tumble after we control for the overlap between these measures and integrative complexity (although the partial correlations remain significant).

  d. As one would expect if there were an affinity between hedgehog styles of reasoning and ideological extremism, hedgehogs were more likely to be extremists (average r = .31 across the three content of belief system scales). Consistent with the notion that these affinities are rooted in hedgehogs’ aversion to, and foxes’ tolerance of, dissonant combinations of ideas, extremists were also less integratively complex (r = .32).

  e. As one would expect if hedgehog performance had been dragged down by forecasters with extreme convictions making extreme predictions that stray far from base rates, hedgehogs were more prone to use the high-confidence ends of the subjective probability scales. Relative to foxes, hedgehogs call significantly more things impossible or highly improbable (31.9 percent of judgments versus 24.3 percent) and more things certain or highly probable (7.4 percent of judgments versus 4.0 percent). And hedgehog extremists are the most prone of all subgroups to use these end point values: calling 34 percent of things impossible or nearly so and 9.4 percent of things certain or nearly so. To appreciate the magnitude of the performance drag, hedgehogs not only used the extreme end points more frequently, when they did, they also had higher miss rates (the “impossible” or “nearly impossible” happened almost 19.8 percent of the time compared to foxes’ rate of 9.9 percent) and higher false alarm rates (sure things or nearly sure things failed to happen 31.5 percent of the time compared to foxes’ rate of 20.8 percent).

  f. Finally, some might try here to resurrect the “foxes are just chickens” hypothesis (which took a hammering when foxes beat hedgehogs on discrimination). Hedgehogs clearly do make braver forecasts—forecasts that will prove more embarrassing if the unexpected occurs. But the evidence is again consistent with the notion that the greater caution among foxes is rooted in balanced cognitive appraisals of situations, not a mindless clinging to the midpoints of the scales. The hedgehog-fox differential on extremity of predictions, once highly significant (r = .35), shrinks significantly when we control for the fact that foxes engage in more integratively complex thinking about problems than do hedgehogs (partial r = .14). It is hard to build up a lot of momentum for extreme predictions if one is slowed down by lots of buts and howevers.

  Overall, these quantitative analyses yield a strikingly consistent portrait of good forecasting judgment. Figure 3.5 lays out a conceptual model that captures the pattern of correlations among key constructs. Good judges tend to be moderate foxes: eclectic thinkers who are tolerant of counterarguments, and prone to hedge their probabilistic bets and not stray too far from just-guessing and base-rate probabilities of events. However, the quantitative analyses give us only a vague picture of how foxes managed to outperform hedgehogs with such regularity in real-world settings. To get a richer sense for what transpired, we need to get behind the numbers, compare the reasoning strategies of hedgehogs and foxes in particular domains, and trace the linkages between those strategies and forecasting triumphs and fiascoes.16

  Figure 3.5. The foxes’ advantage in forecasting skill can be traced to two proximal mediators, greater integrative complexity of free-flowing thoughts and a cautious approach to assigning subjective probabilities. These two mediators are, in turn, traced to broader individual differences in cognitive style (the hedgehog-fox scale) and in ideological extremism (scores on the three content-of-belief scales). Cognitive style and ideological extremity reciprocally influence each other: a fox style of reasoning encourages ideological moderation, and ideological extremism encourages a hedgehog style of reasoning.

  THE QUALITATIVE SEARCH FOR GOOD JUDGMENT

  Isaiah Berlin argued that the fox-hedgehog distinction captured

  one of the deepest differences which divide writers and thinkers, and it may be, human beings in general. For there exists a great chasm between those, on one side, who relate everything to a single central vision, one system, more or less coherent or articulate, in terms of which they understand, think and feel … and, on the other side, those who pursue many ends, often unrelated and even contradictory, connected, if at all, only in some de facto way, … related to no moral or esthetic principle. These last lead lives, perform acts and entertain ideas that are centrifugal rather than centripetal; their thought is scattered or diffused, moving on many levels, seizing upon the essence of a vast variety of experiences and objects for what they are in themselves, without, consciously or unconsciously, seeking to fit them into, or exclude them from, any one unchanging, all-embracing … inner vision. The first kind of intellectual belongs to the hedgehogs, the second to the foxes; and without insisting on a rigid classification, … Dante belongs to the first category and Shakespeare to the second; Plato, Lucretius, Pascal, Hegel, Dostoyevsky, Nietzsche, Ibsen, and Proust are, in varying degrees, hedgehogs; Herodotus, Aristotle, Montaigne, Erasmus, Molière, Goethe, Pushkin, Balzac, and Joyce are foxes.17

  Berlin recognized that few fit the ideal-type template of fox or hedgehog. Most of us are hybrids, a
wkward hedge-fox and fox-hog amalgams. Indeed, Berlin suspected that his beloved Tolstoy was a fox who aspired to be a hedgehog. In the same vein, we should recognize that “hedgehogs” and “foxes” are defined here by arbitrary quartile cutoffs along a fuzzy measurement continuum. I met some participants whom formal measures classified as “foxes” but who admired the crisp, deductive style of reasoning of hedgehogs, and—imitation being the sincerest form of flattery—even sometimes snuck a bit of syllogistic certainty into their own cognitive repertoire. One “fox,” by psychometric criteria, wistfully told me that it would be “nice to close an argument with QED.” I also met participants whom the formal measures pigeonholed as “hedgehogs” but who grudgingly conceded that it may sometimes be futile to try to reduce the booming, buzzing confusion of this world into a single vision. One “hedgehog” feared that “God did give the damn physicists all the solvable problems.” Another observed that “grown-ups understand the tragedy of knowledge: people do not fit into neat logical categories but neat logical categories are indispensable to the advance of knowledge.”

  Yet another respondent, who deserved his position near the midpoint of the continuum, offered the old chestnut: “There are two types of people—those who classify people into two types and those who don’t.” The remark was a pointed reminder that my participants were fully capable of thinking about thinking (“metacognition”) and of transcending my procrustean categories by making midstream-of-consciousness adjustments when they suspect that they have gone too far in any one direction. I should not fall into the essentialist trap of viewing “hedgehogs” and “foxes” as distinct cognitive species. Reification leads only to quibbling: Are hedgehogs still hedgehogs when they engage in fox-style self-mockery? Are foxes still foxes when they pause to admire the elegance of a hedgehog framework?

  Qualifications noted, it is still useful to inventory the distinctive attributes of the reasoning styles of hedgehogs and foxes that emerged from their free-flowing commentaries on how they went about forming and revising expectations. But we should approach the list in a foxlike spirit, as cognitive maneuvers that, when we look backward in time, worked well for some forecasters and not so well for others, but might well have worked out in the opposite manner save for quirky twists of fate. Acknowledging the tentativeness of our knowledge will protect us from disappointment when, looking forward in time, we discover how frequently extrapolations of past regularities into the future are upended. In this spirit, then, are six basic ways in which foxes and hedgehogs differed from each other. Foxes were more

  a. skeptical of deductive approaches to explanation and prediction

  b. disposed to qualify tempting analogies by noting disconfirming evidence

  c. reluctant to make extreme predictions of the sort that start to flow when positive feedback loops go unchecked by dampening mechanisms

  d. worried about hindsight bias causing us to judge those in the past too harshly

  e. prone to a detached, ironic view of life

  f. motivated to weave together conflicting arguments on foundational issues in the study of politics, such as the role of human agency or the rationality of decision making

  Foxes Are More Skeptical of the Usefulness of Covering Laws for Explaining the Past or Predicting the Future

  When senior hedgehogs dispense advice to junior colleagues, they stress the virtue of parsimony. Good judgment requires tuning out the ephemera that dominate the headlines and distract us from the real, surprisingly simple, drivers of long-term trends. They counsel that deep laws constrain history, and that these laws are knowable and lead to correct conclusions when correctly applied to the real world. They also endorse cognitive ideals that fit Berlin’s characterization of the ideal-type hedgehog (hardly astonishing—a nonnegligible number had read Berlin’s essay). They admire deductive reasoning that uses powerful abstractions to organize messy facts and to distinguish the possible from the impossible, the desirable from the undesirable.

  But, agree though hedgehogs do on “how to think,” they disagree, often fiercely, over what to think—over the correct content to insert into the logical machinery. To invoke the zoological metaphor, there are many ideological subspecies of hedgehogs, each with its own distinctive view of the fundamental drivers of events.

  The propensity of hedgehogs to push their favorite first principles as far as possible, and sometimes beyond, arose on numerous occasions. For example, hedgehogs who stressed the primacy of ethnicity were among the first to suspect that the Soviet Union might not survive Gorbachev’s policies that allowed “captive peoples” to express how miserable they were. Neorealist hedgehogs joined these “primordialists” in 1991–1992 in arguing that the demise of the USSR had now made Eastern Europe safe for conventional warfare among groups that had been compelled in the bipolar NATO–Warsaw Pact world to suppress their enmity. As a result, this combined camp scored impressive “hits.” Even here, though, these hedgehogs did not reap much benefit in aggregate forecasting skill. They overpredicted conflict: war has yet to break out between Hungary and Romania, the divorce between the Czechs and Slovaks was as civilized as these things get, and Russia has not yet invaded the Baltics, the Ukraine, or Kazakhstan. Yugoslavia—which had already begun to unravel in 1991—was their “big hit.”

  These hedgehogs also went out on predictive limbs in 1992 with respect to the European Monetary Union (EMU) and NATO. They felt that the original driving force behind these organizations was the threat of Soviet aggression, and they now suspected that (a) Europeans would become more reluctant to sacrifice sovereign control over monetary and fiscal policy to transnational authorities; (b) Europeans would feel less grateful for the American nuclear umbrella and more irritated by the overbearing habits of the American hegemons. With respect to the European Monetary Union, some of these observers went all the way back to 1776 for the right historical analogy, noting that, notwithstanding their shared language and traditions, the thirteen colonies were wary after the Declaration of Independence to go beyond the weak linkages in the Articles of Confederation. The adoption of stronger central government was propelled by desire for common defense. These experts added that the emotional impetus toward the European Monetary Union was fear of repeating World War II. But, as time passes, the old folks die and the younger generation does not share their obsessions about German expansionism. Another camp of premature obituary writers for the EMU suspected that public support for the monetary union would evaporate when people appreciated the sacrifices needed to satisfy the stringent Maastricht convergence requirements for inflation, interest rates, and budget deficits.

  Of course, every time hedgehogs of one persuasion suffered a setback, hedgehogs of another persuasion were well positioned to claim credit for their farsightedness. Hedgehog institutionalists were not surprised that the euro project survived currency crises, budgetary squeezes, close-call elections, and political scandals. Whether they grounded their cases in transaction cost economics or the evolving political self-images of Europeans, this camp bet on consolidation and gradual expansion of the transnational regime.

  By contrast, foxes doubted that real-world problems could be squeezed, without serious distortion, into syllogistic templates.18 The grounds for doubt included the following:

  a. There is typically ambiguity about which laws apply. This is true, as one fox insisted, even when theory is “as good as it gets” in politics: modeling the vote-winning strategies of candidates. For instance, should we always expect the two major parties’ platforms to converge on the preference profile of the median voter? Or should we back off: Have the conditions for applying the theorem been satisfied? Is the issue space one-dimensional? Do small third or fourth parties nullify all predictions? Is the system truly “winner take all?”

  b. There is typically ambiguity about how to bridge the gap between ethereal abstractions and grubby facts on the ground. This is most true when we most need guidance. One fox brought up the problem of coping with adversa
ries. How much weight should policy makers give to deterrence theory (which stresses the dangers of pusillanimity) versus conflict spiral theory (which stresses the dangers of bellicosity)? When does conciliation become appeasement? When does deterrence grade into provocation? Another fox, who knew a lot about Northeast Asia, had a telling response to long lists of “bridging the gap” questions: “If you know the answers, you can read Kim Jong-il’s and Jiang Zemin’s minds a lot better than I can.”

  This uneasiness toward the “Hempelian” agenda to reduce history to social science mostly served foxes well. For example, although foxes did not assign as high a probability as hedgehogs to the Yugoslav conflagration, foxes did not overpredict wars: Czechs versus Slovaks, Hungarians versus Romanians, and Russians versus Ukrainians, Lithuanians, Latvians, Estonians, or Kazakhs, or—on a more global scale—civil wars in Nigeria, Pakistan, and South Africa.