I suspect the incentive problem is insuperable without concerted action by the big buyers of political prognostication in media, business, and government. The best hope of breaking the status quo is by publicizing that demand for public intellectuals has become partly conditional on their proven track records in drawing correct inferences from relevant real-world events unfolding in real time, and that the buyers are no longer automatically relying on the “Rolodex” or prestigious affiliation or ideological compatibility heuristics. Smart newcomers would then see the new regime as a means of more rapid ascent in the media star or consulting hierarchy than might otherwise be available. And insofar as upstarts began to claim serious market shares, even comfortably ensconced incumbents might feel compelled to rethink how they think.
Resistance would naturally be fierce, especially from influential hedgehogs who would have the most to lose, and we had a foretaste of the forms it is likely to take in the strong relativist objections to this entire project. Few of us look pretty under the cognitive microscope. And one does not need to buy fully into George Bernard Shaw’s definition of professions (conspiracies to defraud the public) to recognize that no profession welcomes in-depth scrutiny of its collective claim to expertise. Sociologists have long known that professions maintain their autonomy and prestige by convincing the world of two things: (1) we professionals possess valuable skills that the uninitiated do not; and (2) these valuable skills cannot be easily “canned” or reduced to reproducible components easily taught to outsiders.18 Resistance, moreover, would not just come from the supply side of the economic equation. Chapter 2 showed that, on the demand side, there is a strong desire among mass-public consumers to believe that they live in a predictable world and an equally strong desire among more elite consumers in the media, business, and government to appear to be doing the right thing by ritualistically consulting the usual suspects from widely recognized interest groups. The fainthearted should be forgiven for concluding that we are fated to fail to break this tight symbiotic embrace between self-confident suppliers of dubious products and their cling-on customers.
But even fierce resistance can be overcome. Low-transaction-cost index funds have benefited—very substantially—from slowly spreading knowledge of how hard-pressed stock-pickers are to best dart-throwing chimps and other mindless algorithms. And prediction markets—in which people put their money where their mouths are—have boomed in popularity.19 We should not take it for granted that incumbents will forever be successful in creating a Wizard-of-Oz-like mystique around their inner-sanctum knowledge. We live in a world of rapidly advancing information technology and more gradually advancing artificial intelligence. It is arguably only a matter of time before these new technologies encroach on well-established professions, including medicine, law, science, and even that last redoubt of obfuscation, politics.
And if the resistance were overcome, where would we be then? Age of Reason optimists would announce that the long-heralded end-of-ideology thesis has been vindicated—a prediction that, incidentally, has used up its allotment of off-on-timing defenses.20 Lord Rutherford’s era of “gentlemen and women, let us calculate rather than debate” would finally be upon us. My own guess is that the announcement would again be premature. Human nature being what it is, and the political system creating the perversely self-justifying incentives that it does, I would expect, in short order, faux rating systems to arise that shill for representatives of points of view who feel shortchanged by even the most transparent evaluation systems that bend over backward to be fair.21 The signal-to-noise ratio will never be great in a cacophonously pluralistic society such as ours.
But that does not mean we must reconcile ourselves to the noisy status quo. A coordinated initiative—from those in academia, foundations, and the media who view public intellectuals as purveyors of credence goods, not just solidarity and entertainment goods—could sharpen the signals and dampen the noise. Imperfect though they are, the research tools in this book should be of use to professionals in applied fields such as intelligence analysis, risk assessment, and journalism. Deployed thoughtfully, these tools can help professionals build self-correcting epistemic communities devoted to monitoring complex events as they unfold in real time, reflecting on the implications of these events for their evolving worldviews, and specifying benchmarks for defining and checking biases.22 Progress will not be as easy as techno-enthusiasts hope. There are ineradicable pockets of subjectivity in political judgment. But progress is not as hopeless as opponents of social science never tire of insisting.
1 For a historical account of the ups and downs of the concept of objectivity in twentieth-century scholarship, see P. Novick, That Noble Dream: The Objectivity Question and the American Historical Profession (New York: Cambridge University Press, 1988).
2 This policy recommendation is grounded in the now large research literature on the effects of accountability on judgment and choice. Accountability by itself is no panacea. But the right types of accountability can help. In engineering accountability systems to promote more self-critical and reflective styles of thinking, one is well advised to ensure that forecasters believe they must answer for judgments they have yet to make to audiences whose own views forecasters cannot readily infer (closing off the option of telling people what they want to hear) and whose respect forecasters value (encouraging sober second thought). For a review, see P. E. Tetlock and J. Lerner, “The Social Contingency Model: Identifying Empirical and Normative Boundary Conditions on the Error-and-Bias Portrait of Human Nature,” in Dual Process Models in Social Psychology, ed. S. Chaiken and Y. Trope (New York: Guilford Press, 1999). The psychologic underlying this proposal bears some similarities to explanations that have been offered for the well-documented power of prediction markets to produce aggregate forecasts that are more accurate than the vast majority of individual forecasters. These explanations often stress the power of competitive interaction among forecasters, driven by a mix of financial and social-image motives, to encourage more flexible, rapid-fire belief updating in response to new evidence (see J. Wolfers and E. Zitzewitz, “Prediction Markets,” Journal of Economic Perspectives 18 [2004]: 107–26).
3 Relativists and their constructivist allies in political science stress the multiplicity of meanings people attach to each other’s actions, and the contestability of those meanings in rough-and-tumble encounters in which people jostle to claim desirable identities for themselves and to “cast” rivals into less desirable ones. Moderate forms of relativism and constructivism are easy for students of human cognition to accept: they warn us of the power of mind-sets to shape how we pose questions and go about answering them. See F. Suppe, The Structure of Scientific Theories (Chicago: University of Chicago Press, 1974). The hard-core forms are another matter: they cast doubt not only on the possibility of understanding the world but also on the wisdom of even trying.
4 Value priorities are also not immutable. Positions can shift quickly in response to shocking events. An example is the metamorphosis in American attitudes toward false-alarm detentions of suspected terrorists in the wake of the 9/11 attacks.
5 J. Bruner, Actual Minds, Possible worlds (Cambridge: Harvard University Press, 1995).
6 For a thoughtful analysis of the debate over covering laws in history, see C. Roberts, The Logic of Historical Explanation (University Park: Pennsylvania State University Press, 1995).
7 L. Cederman, Emergent Actors in World Politics: How States and Nations Develop and Dissolve (Princeton, NJ: Princeton University Press, 1997).
8 G. Gigerenzer, “Dread Risk, September 11, and Fatal Traffic Accidents,” Psychological Science 15 (2004): 286–87.
9 For examples of such work: T. Amer, K. Hackenbrack, and M. Nelson, “Between Auditor Differences in the Interpretation of Probability Phrases,” Auditing: A Journal of Practice and Theory 13 (1994): 126–36; K. H. Teigen, “When Equal Chance-Good Chances: Verbal Probabilities and Equiprobability Effect,” Organizational Behavior and Human D
ecision Processes 85 (2000): 77–108; K. H. Teigen and W. Brun, “Ambiguous Probabilities: When Does p = 0.3 Reflect a Possibility, and When Does It Reflect a Doubt?” Journal of Behavioral Decision Making 13 (2000): 345–62; K. H. Teigen and W. Brun, “Verbal Probabilities: A Question of Frame?” Journal of Behavioral Decision Making 16 (2003): 53–72.
10 Or, mirror imaging Clifford Geertz’s ploy of proclaiming himself an anti-anti-relativist, I might proclaim myself an anti-anti-positivist.
11 A casual glance over twentieth-century history—Hitler’s Germany, Stalin’s Russia, Mao’s China, Pol Pot’s Cambodia, the Taliban’s Afghanistan, and the institutionalized insanity that is still North Korea—should remind us of how oppressively surreal things can get when tiny cliques enforce monopoly claims on truth. The superiority of democratic regimes over totalitarian ones should not count, however, as evidence that there is no room for improvement. Even in relatively open societies, it is far more difficult than it need be for attentive consumers of information to sort out which points of view have proven more prescient on which points of contention.
12 Richard A. Posner, Public Intellectuals: A Study of Decline: A Critical Analysis (Cambridge: Harvard University Press, 2001); for a pioneering study of the criteria that ordinary people use in judging whether an idea is worth keeping alive in the marketplace of ideas, see C. Heath, C. Bell, and E. Sternberg, “Emotional Selection in Memes: The Case of Urban Legends,” Journal of Personality and Social Psychology 81 (2002): 1028–41.
13 For a review of work that suggests integratively simple rhetoric often has a political-psychological advantage over more complex forms of rhetoric, see P. E. Tetlock, “Cognitive Structural Analysis of Political Rhetoric: Methodological and Theoretical Issues,” in Political Psychology: A Reader, ed. S. Iyengar and W. J. McGuire (Durham, NC: Duke University Press, 1992), 380–407.
14 For a review, Tetlock, “Cognitive Structural Analysis.”
15 C. F. Camerer and R. M. Hogarth, “The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework,” Journal of Risk and Uncertainty 19 (1999): 1–3, 7–42.
16 T. R. Tyler and H. J. Smith, “Social Justice and Social Movements,” in The Handbook of Social Psychology, ed. D. T. Gilbert, S. Fiske, and G. Lindzey (New York: McGraw-Hill, 1998).
17 One could argue that much the same end could be achieved by mandating that those offering advice on futures—be they pundits in the public eye or intelligence analysts behind the scenes—regularly participate in prediction markets that require testing their wits against a wide range of fellow experts as well as dilettantes. There is indeed much to recommend this idea. I see the growing interest in prediction markets as a hopeful sign that the era of close-to-zero-accountability-for-predictive-track-records may slowly be coming to a close (see S. Weber and P. E. Tetlock, “New Economy: The Pentagon’s Plan for Futures Trading,” New York Times, August 11, 2003, C3). A task for future work will be to sort the commonalities and differences underlying four approaches that have shown promise in improving political and business forecasting: prediction markets, the Delphi method of integrating divergent perspectives, the power of simple consensus forecasts, and the superior forecasting skill of more self-critical and flexible forecasters. One common theme is that useful information for predicting complex outcomes is typically spread across diverse sources and there is a price to be paid for narrow-mindedness. A second theme is that there is often a price to be paid for extremism, for betting heavily on long shots (R. Thaler and W. Ziemba, “Anomalies: Pari-mutuel Betting Markets: Racetracks and Lotteries,” Journal of Economic Perspectives 2 [1988]: 161–74). A third theme running through three of the four approaches—prediction markets, the Delphi method, and cognitive styles—is that information processing systems generate more accurate forecasts to the degree that competing ideas—all of them—are subject to critical assessment.
18 H. Wilensky, “The Professionalization of Everyone?” American Journal of Sociology 70 (1964): 137–58.
19 Of course, there has also been resistance to prediction markets, especially those in which participants might profit from the grief of others, raising the moral specter of taboo trade-offs (see Weber and Tetlock, “Pentagon Futures Trading.”)
20 Daniel Bell, The Coming of Post-industrial Society: A Venture in Social Forecasting (New York: Basic Books, 1976).
21 “True believers” are quick to claim they are victims of bias: L. Ross and D. Griffin, “Subjective Construal, Social Inference, and Human Misunderstanding,” in Advances in Experimental Social Psychology, Vol. 24, ed. M. Zanna (New York: Academic Press, 1991), 319–59.
22 The recommendations of this book are much in the spirit of Sherman Kent, after whom the CIA named its training school for intelligence analysts. We can draw cumulative lessons from experience only if we are aware of gaps between what we expected and what happened, acknowledge the possibility that those gaps signal shortcomings in our understanding, and test alternative interpretations of those gaps in an evenhanded fashion. This means doing what we did here: obtaining explicit subjective probability estimates (not just vague verbiage), eliciting reputational bets that pit rival worldviews against each other, and assessing the consistency of the standards of evidence experts apply to evidence. (S. Kent, Collected Essays, U.S. Government: Center for the Study of Intelligence, 1970, http://www.cia.gov/csi/books/shermankent/toc.html).
Methodological Appendix
THIS APPENDIX is divided into six sections, each dedicated to describing the research participants, procedures, and stimulus materials used in one of the six major types of studies presented in the book: (1) the regional forecasting exercises in chapters 2 and 3 (from which we derived the correspondence indicators of good judgment such as probability scores and their components); (2) the reputational bet exercises in chapter 4 (from which we derived the Bayesian belief-updating indicators and measures of belief system defenses); (3) the reactions to historical discoveries exercises in chapter 5 (from which we derived turnabout tests of double standards); (4) the reactions to close-call counterfactual exercises in chapter 5 (from which we derived measures of perceptions of the “mutability” of historical outcomes); (5) the unpacking scenarios of possible futures exercises in chapter 7 (from which we derived coherence measures of good judgment, such as violations of formal principles of probability theory); (6) the unpacking scenarios of possible pasts exercise in chapter 7 (from which we derived inevitability and impossibility curves and measured contradictions between the two sets of perceptions).
I. REGIONAL FORECASTING EXERCISES (CHAPTERS 2 AND 3)
Participants and Individual Difference Measures
Our operational definition of an expert was “a professional who makes his or her livelihood by commenting or offering advice on political and economic trends of significance to the well-being of particular states, regional clusters of states, or the international system as a whole.” Expertise could thus take diverse region-specific forms (from southern Africa to the Middle East, etc.) and functional forms (knowledge of local political scenes, of macroeconomic policies and their effects, of interstate relations, of military balances of power and proliferation risks, etc.). We classified the 284 experts (who satisfied the overarching definition and who answered at least half of our forecasting questions) into the following demographic, educational, disciplinary background, current employment, and political affiliation categories (measured in the Professional Background Questionnaire). Participants were mostly male (76 percent), with an average age of 43 (standard deviation of 7.3 years) and an average of 12.2 years of relevant work experience (standard deviation of 4.7). Most had doctoral degrees (52 percent) and almost all had postgraduate training (96 percent). Our participants came from a potpourri of disciplines, including most branches of area studies (41 percent), international relations (24 percent), economics (12 percent), national security and arms control (11 percent), journalism (9 percent), diplomacy (2 percent), and international la
w (1 percent). They worked in a range of settings, including academia (41 percent), government (26 percent), think tanks and foundations (17 percent), international institutions (8 percent), and the private sector (including the media) (8 percent). Approximately 61 percent of participants had been interviewed at least once by a major media outlet and 21 percent had been interviewed more than ten times. Approximately 80 percent of participants had served at least once as formal or informal consultants on international political or economic issues to government, the private sector, international agencies or think tanks. The vast majority of the sample (82 percent) participated in forecasting exercises initiated between 1988 and 1995.
We also tried—as often as possible—to measure individual differences in ideological orientation (the thirteen-item Worldview Questionnaire that was factor analyzed in chapter 3) and in cognitive style (the thirteen-item Styles-of-Reasoning Questionnaire that was also factor analyzed in chapter 3). In each case, experts responded to items on nine-point scales ranging from “completely disagree” (1) to “completely agree” (9), with 5 defined as a point of maximum uncertainty.
The items in the Worldview Questionnaire included the following: “I see an irreversible trend toward global economic interdependence”; “Free markets are the best path to prosperity”; “Our society underestimates the adverse environmental side effects of free markets”; “Our society underestimates the adverse effects of markets on social equality”; “I believe balance-of-power politics remains the reigning principle in world politics”; “It is a mistake to dismiss international institutions as subordinate to whims of great powers”; “I am optimistic about long-term growth trajectory of the world economy”; “I am concerned about pushing the limits of sustainable development”; “In dealing with potential adversaries, reassurance is generally a more useful diplomatic tool than deterrence”; “We should weight financial contagion as a greater threat than moral hazard in deciding whether to aid insolvent governments”; “I expect powerful ethnic and religious identifications to transform the boundaries of dozens of existing states in the near future”; “There is a widespread tendency to underestimate fragility of ecosystems”; and “On balance I see myself as more liberal/conservative.” Maximum likelihood factor analysis with quartimin rotation yielded the three-factor solution in table 1 in chapter 3. The high-loading items on each factor (above 0.25) defined the belief system scale used in later analyses (average Cronbach’s alpha = 0.79).
Expert Political Judgment Page 38
