Rationality- From AI to Zombies

by Eliezer Yudkowsky

  I hope that the part of the Art I have set down, as incomplete as it may be, can surpass that preliminary barrier—give people a base to build on; give them an idea that an Art exists, and somewhat of how it ought to be developed; and give them at least a saving throw before they instantaneously go astray.

  That’s my dream—that this highly-specialized-seeming art of answering confused questions may be some of what is needed, in the very beginning, to go and complete the rest.

  A task which I am leaving to you. Probably, anyway. I make no promises as to where my attention may turn in the future. But y’know, there are certain other things I need to do. Even if I develop yet more Art by accident, it may be that I will not have the time to write any of it up.

  Beyond all that I have said of fake answers and traps, there are two things I would like you to keep in mind.

  The first—that I drew on multiple sources to create my Art. I read many different authors, many different experiments, used analogies from many different fields. You will need to draw on multiple sources to create your portion of the Art. You should not be getting all your rationality from one author—though there might be, perhaps, a certain centralized website, where you went to post the links and papers that struck you as really important. And a maturing Art will need to draw from multiple sources. To the best of my knowledge there is no true science that draws its strength from only one person. To the best of my knowledge that is strictly an idiom of cults. A true science may have its heroes, it may even have its lonely defiant heroes, but it will have more than one.

  The second—that I created my Art in the course of trying to do some particular thing that animated all my efforts. Maybe I’m being too idealistic—maybe thinking too much of the way the world should work—but even so, I somewhat suspect that you couldn’t develop the Art just by sitting around thinking to yourself, “Now how can I fight that akrasia thingy?” You’d develop the rest of the Art in the course of trying to do something. Maybe even—if I’m not overgeneralizing from my own history—some task difficult enough to strain and break your old understanding and force you to reinvent a few things. But maybe I’m wrong, and the next leg of the work will be done by direct, specific investigation of “rationality,” without any need of a specific application considered more important.

  A past attempt of mine to describe this principle, in terms of maintaining a secret identity or day job in which one doesn’t teach rationality, was roundly rejected by my audience. Maybe “leave the house” would be more appropriate? It sounds to me like a really good, healthy idea. Still—perhaps I am deceived. We shall see where the next pieces of the Art do, in fact, come from.

  I have striven for a long time now to convey, pass on, share a piece of the strange thing I touched, which seems to me so precious. And I’m not sure that I ever said the central rhythm into words. Maybe you can find it by listening to the notes. I can say these words but not the rule that generates them, or the rule behind the rule; one can only hope that by using the ideas, perhaps, similar machinery might be born inside you. Remember that all human efforts at learning arcana slide by default into passwords, hymns, and floating assertions.

  I have striven for a long time now to convey my Art. Mostly without success, before this present effort. Earlier I made efforts only in passing, and got, perhaps, as much success as I deserved. Like throwing pebbles in a pond, that generate a few ripples, and then fade away . . . This time I put some back into it, and heaved a large rock. Time will tell if it was large enough—if I really disturbed anyone deeply enough that the waves of the impact will continue under their own motion. Time will tell if I have created anything that moves under its own power.

  I want people to go forth, but also to return. Or maybe even to go forth and stay simultaneously, because this is the Internet and we can get away with that sort of thing; I’ve learned some interesting things on Less Wrong, lately, and if continuing motivation over years is any sort of problem, talking to others (or even seeing that others are also trying) does often help.

  But at any rate, if I have affected you at all, then I hope you will go forth and confront challenges, and achieve somewhere beyond your armchair, and create new Art; and then, remembering whence you came, radio back to tell others what you learned.

  *

  1. David Charles Stove, The Plato Cult and Other Philosophical Follies (Cambridge University Press, 1991).

  Bibliography

  Albert, David Z. Quantum Mechanics and Experience. Harvard University Press, 1994.

  Alexander, Scott. “Why I Am Not Rene Descartes.” Slate Star Codex (blog) (2014). http://slatestarcodex.com/2014/11/27/why-i-am-not-rene-descartes/.

  Allais, Maurice. “Le Comportement de l’Homme Rationnel devant le Risque: Critique des Postulats et Axiomes de l’Ecole Americaine.” Econometrica 21, no. 4 (1953): 503–546. doi:10.2307/1907921.

  Ariely, Dan. Predictably Irrational: The Hidden Forces That Shape Our Decisions. HarperCollins, 2008.

  Asch, Solomon E. “Studies of Independence and Conformity: A Minority of One Against a Unanimous Majority.” Psychological Monographs 70 (1956).

  Ashmore, Richard D., Vasantha Ramchandra, and Russell A. Jones. “Censorship as an Attitude Change Induction.” Paper presented at Eastern Psychological Association meeting (1971).

  Asimov, Isaac. The Relativity of Wrong. Oxford University Press, 1989.

  Baez, John. “The Crackpot Index.” 1998. http://math.ucr.edu/home/baez/crackpot.html.

  Banks, Iain. The Player of Games. Orbit, 1989.

  Baratz, Daphna. How Justified Is the “Obvious” Reaction? Stanford University, 1983.

  Barbour, Julian. The End of Time: The Next Revolution in Physics. 1st ed. New York: Oxford University Press, 1999.

  Baron, Jonathan. Thinking and Deciding. Cambridge University Press, 2007.

  Baron, Jonathan, and Joshua D. Greene. “Determinants of Insensitivity to Quantity in Valuation of Public Goods: Contribution, Warm Glow, Budget Constraints, Availability, and Prominence.” Journal of Experimental Psychology: Applied 2, no. 2 (1996): 107–125. doi:10.1037/1076-898X.2.2.107.

  Barrett, Jeffrey. Everett’s Relative-State Formulation of Quantum Mechanics. Edited by Edward N. Zalta. http://plato.stanford.edu/archives/fall2008/entries/qm-everett/.

  Benson, Peter L., Stuart A. Karabenick, and Richard M. Lerner. “Pretty Pleases: The Effects of Physical Attractiveness, Race, and Sex on Receiving Help.” Journal of Experimental Social Psychology 12 (5 1976): 409–415. doi:10.1016/0022-1031(76)90073-1.

  Bond, Rod, and Peter B. Smith. “Culture and Conformity: A Meta-Analysis of Studies Using Asch’s (1952b, 1956) Line Judgment Task.” Psychological Bulletin 119 (1996): 111–137.

  Bostrom, Nick. “A History of Transhumanist Thought.” Journal of Evolution and Technology 14, no. 1 (2005): 1–25. http://www.nickbostrom.com/papers/history.pdf.

  ———. “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards.” Journal of Evolution and Technology 9 (2002). http://www.jetpress.org/volume9/risks.html.

  ———. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014.

  Bostrom, Nick, and Milan M. Ćirković, eds. Global Catastrophic Risks. New York: Oxford University Press, 2008.

  Bostrom, Nick, and Julian Savulescu. “Human Enhancement Ethics: The State of the Debate.” In Human Enhancement, edited by Nick Bostrom and Julian Savulescu. 2009.

  Bostrom, Nick, and Eliezer Yudkowsky. “The Ethics of Artificial Intelligence.” In The Cambridge Handbook of Artificial Intelligence, edited by Keith Frankish and William Ramsey. New York: Cambridge University Press, 2014.

  Bourget, David, and David J. Chalmers. “What Do Philosophers Believe?” Philosophical Studies (2013): 1–36.

  Boynton, Robert S. “The Birth of an Idea: A Profile of Frank Sulloway.” The New Yorker (October 1999).

  Brehm, Sharon S., and Marsha Weintraub. “Physical Barriers and Psychological Reactance: Two-year-o
lds’ Responses to Threats to Freedom.” Journal of Personality and Social Psychology 35 (1977): 830–836.

  Broeder, Dale. “The University of Chicago Jury Project.” Nebraska Law Review 38 (1959): 760–774.

  Brown, Kevin. Reflections On Relativity. Raleigh, NC: printed by author, 2011. http://www.mathpages.com/rr/rrtoc.htm.

  Brust, Steven. The Paths of the Dead. Vol. 1 of The Viscount of Adrilankha. Tor Books, 2002.

  Budesheim, Thomas Lee, and Stephen DePaola. “Beauty or the Beast?: The Effects of Appearance, Personality, and Issue Information on Evaluations of Political Candidates.” Personality and Social Psychology Bulletin 20 (4 1994): 339–348. doi:10.1177/0146167294204001.

  Buehler, Roger, Dale Griffin, and Michael Ross. “Exploring the ‘Planning Fallacy’: Why People Underestimate Their Task Completion Times.” Journal of Personality and Social Psychology 67, no. 3 (1994): 366–381. doi:10.1037/0022-3514.67.3.366.

  ———. “Inside the Planning Fallacy: The Causes and Consequences of Optimistic Time Predictions.” In Gilovich, Griffin, and Kahneman, Heuristics and Biases, 250–270.

  ———. “It’s About Time: Optimistic Predictions in Work and Love.” European Review of Social Psychology 6, no. 1 (1995): 1–32. doi:10.1080/14792779343000112.

  Bujold, Lois McMaster. Komarr. Miles Vorkosigan Adventures. Baen, 1999.

  Burton, Ian, Robert W. Kates, and Gilbert F. White. The Environment as Hazard. 1st ed. New York: Oxford University Press, 1978.

  Campbell, Richmond, and Lanning Snowden, eds. Paradoxes of Rationality and Cooperation: Prisoner’s Dilemma and Newcomb’s Problem. Vancouver: University of British Columbia Press, 1985.

  Carson, Richard T., and Robert Cameron Mitchell. “Sequencing and Nesting in Contingent Valuation Surveys.” Journal of Environmental Economics and Management 28, no. 2 (1995): 155–173. doi:10.1006/jeem.1995.1011.

  Casscells, Ward, Arno Schoenberger, and Thomas Graboys. “Interpretation by Physicians of Clinical Laboratory Results.” New England Journal of Medicine 299 (1978): 999–1001.

  Castellow, Wilbur A., Karl L. Wuensch, and Charles H. Moore. “Effects of Physical Attractiveness of the Plaintiff and Defendant in Sexual Harassment Judgments.” Journal of Social Behavior and Personality 5 (6 1990): 547–562.

  Chaiken, Shelly. “Communicator Physical Attractiveness and Persuasion.” Journal of Personality and Social Psychology 37 (8 1979): 1387–1397. doi:10.1037/0022-3514.37.8.1387.

  Chalmers, David J. The Conscious Mind: In Search of a Fundamental Theory. New York: Oxford University Press, 1996.

  Chapman, Graham, et al. Monty Python’s The Life of Brian (of Nazareth). Eyre Methuen, 1979.

  Chapman, Gretchen B., and Eric J. Johnson. “Incorporating the Irrelevant: Anchors in Judgments of Belief and Value.” In Gilovich, Griffin, and Kahneman, Heuristics and Biases, 120–138.

  Cherryh, Carolyn J. The Paladin. Baen, 2002.

  Cialdini, Robert B. Influence: Science and Practice. Boston: Allyn & Bacon, 2001.

  ———. Influence: The Psychology of Persuasion: Revised Edition. New York: Quill, 1993.

  Cleaver, Jerry. Immediate Fiction: A Complete Writing Course. Macmillan, 2004.

  Combs, Barbara, and Paul Slovic. “Newspaper Coverage of Causes of Death.” Journalism & Mass Communication Quarterly 56, no. 4 (1979): 837–849. doi:10.1177/107769907905600420.

  Crawford, Charles B., Brenda E. Salter, and Kerry L. Jang. “Human Grief: Is Its Intensity Related to the Reproductive Value of the Deceased?” Ethology and Sociobiology 10, no. 4 (1989): 297–307.

  Darwin, Charles. On the Origin of Species by Means of Natural Selection; or, The Preservation of Favoured Races in the Struggle for Life. 1st ed. London: John Murray, 1859. http://darwin-online.org.uk/content/frameset?viewtype=text&itemID=F373&pageseq=1.

  ———. The Descent of Man, and Selection in Relation to Sex. 2nd ed. London: John Murray, 1874. http://darwin-online.org.uk/content/frameset?itemID=F944&viewtype=text&pageseq=1.

  Darwin, Francis, ed. The Life and Letters of Charles Darwin. John Murray, 1887.

  Dawes, Robyn M. House of Cards: Psychology and Psychotherapy Built on Myth. Free Press, 1996.

  ———. Rational Choice in An Uncertain World. 1st ed. Edited by Jerome Kagan. San Diego, CA: Harcourt Brace Jovanovich, 1988.

  De Camp, Lyon Sprague, and Fletcher Pratt. The Incomplete Enchanter. New York: Henry Holt & Company, 1941.

  Denes-Raj, Veronika, and Seymour Epstein. “Conflict between Intuitive and Rational Processing: When People Behave against Their Better Judgment.” Journal of Personality and Social Psychology 66 (5 1994): 819–829. doi:10.1037/0022-3514.66.5.819.

  Dennett, Daniel C. Breaking the Spell: Religion as a Natural Phenomenon. Penguin, 2006.

  ———. Darwin’s Dangerous Idea: Evolution and the Meanings of Life. Simon & Schuster, 1995.

  ———. Freedom Evolves. Viking Books, 2003.

  ———. “The Unimagined Preposterousness of Zombies.” Journal of Consciousness Studies 2 (4 1995): 322–26.

  Descartes, René. Discours de la Méthode. Librairie des Bibliophiles, 1887.

  Desvousges, William H., F. Reed Johnson, Richard W. Dunford, Kevin J. Boyle, Sara P. Hudson, and K. Nicole Wilson. Measuring Nonuse Damages Using Contingent Valuation: An Experimental Evaluation of Accuracy. Technical report. Research Triangle Park, NC: RTI International, 2010. doi:10.3768/rtipress.2009.bk.0001.1009.

  Downs, A. Chris, and Phillip M. Lyons. “Natural Observations of the Links Between Attractiveness and Initial Legal Judgments.” Personality and Social Psychology Bulletin 17 (5 1991): 541–547. doi:10.1177/0146167291175009.

  Drescher, Gary L. Good and Real: Demystifying Paradoxes from Physics to Ethics. Cambridge, MA: MIT Press, 2006.

  Eagly, Alice H., Richard D. Ashmore, Mona G. Makhijani, and Laura C. Longo. “What Is Beautiful Is Good, But . . . A Meta-analytic Review of Research on the Physical Attractiveness Stereotype.” Psychological Bulletin 110 (1 1991): 109–128. doi:10.1037/0033-2909.110.1.109.

  Eddy, David M. “Probabilistic Reasoning in Clinical Medicine: Problems and Opportunities.” In Judgement Under Uncertainty: Heuristics and Biases, edited by Daniel Kahneman, Paul Slovic, and Amos Tversky. Cambridge University Press, 1982.

  Efran, M. G., and E. W. J. Patterson. “The Politics of Appearance.” Unpublished PhD thesis, 1976.

  Egan, Greg. Quarantine. London: Legend Press, 1992.

  Ehrlinger, Joyce, Thomas Gilovich, and Lee Ross. “Peering Into the Bias Blind Spot: People’s Assessments of Bias in Themselves and Others.” Personality and Social Psychology Bulletin 31, no. 5 (2005): 680–692.

  Eidelman, Scott, and Christian S. Crandall. “Bias in Favor of the Status Quo.” Social and Personality Psychology Compass 6, no. 3 (2012): 270–281.

  Epley, Nicholas, and Thomas Gilovich. “Putting Adjustment Back in the Anchoring and Adjustment Heuristic: Differential Processing of Self-Generated and Experimentor-Provided Anchors.” Psychological Science 12 (5 2001): 391–396. doi:10.1111/1467-9280.00372.

  Epstein, Lewis Carroll. Thinking Physics: Understandable Practical Reality, 3rd Edition. Insight Press, 2009.

  Feldman, Richard. “Naturalized Epistemology.” In The Stanford Encyclopedia of Philosophy, Summer 2012, edited by Edward N. Zalta.

  Festinger, Leon, Henry W. Riecken, and Stanley Schachter. When Prophecy Fails: A Social and Psychological Study of a Modern Group That Predicted the Destruction of the World. Harper-Torchbooks, 1956.

  Fetherstonhaugh, David, Paul Slovic, Stephen M. Johnson, and James Friedrich. “Insensitivity to the Value of Human Life: A Study of Psychophysical Numbing.” Journal of Risk and Uncertainty 14, no. 3 (1997): 283–300. doi:10.1023/A:1007744326393.

  Feynman, Richard P. “Judging Books by Their Covers.” In Surely You’re Joking, Mr. Feynman! New York: W. W. Norton & Company, 1985.