Rationality- From AI to Zombies

by Eliezer Yudkowsky

  Dictionary editors read what other people write, and record what the words seem to mean; they are historians. The Oxford English Dictionary may be comprehensive, but never authoritative.

  But surely there is a social imperative to use words in a commonly understood way? Does not our human telepathy, our valuable power of language, rely on mutual coordination to work? Perhaps we should voluntarily treat dictionary editors as supreme arbiters—even if they prefer to think of themselves as historians—in order to maintain the quiet cooperation on which all speech depends.

  The phrase “authoritative dictionary” is almost never used correctly, an example of proper usage being The Authoritative Dictionary of IEEE Standards Terms. The IEEE is a body of voting members who have a professional need for exact agreement on terms and definitions, and so The Authoritative Dictionary of IEEE Standards Terms is actual, negotiated legislation, which exerts whatever authority one regards as residing in the IEEE.

  In everyday life, shared language usually does not arise from a deliberate agreement, as of the IEEE. It’s more a matter of infection, as words are invented and diffuse through the culture. (A “meme,” one might say, following Richard Dawkins forty years ago—but you already know what I mean, and if not, you can look it up on Google, and then you too will have been infected.)

  Yet as the example of the IEEE shows, agreement on language can also be a cooperatively established public good. If you and I wish to undergo an exchange of thoughts via language, the human telepathy, then it is in our mutual interest that we use the same word for similar concepts—preferably, concepts similar to the limit of resolution in our brain’s representation thereof—even though we have no obvious mutual interest in using any particular word for a concept.

  We have no obvious mutual interest in using the word “oto” to mean sound, or “sound” to mean oto; but we have a mutual interest in using the same word, whichever word it happens to be. (Preferably, words we use frequently should be short, but let’s not get into information theory just yet.)

  But, while we have a mutual interest, it is not strictly necessary that you and I use the similar labels internally; it is only convenient. If I know that, to you, “oto” means sound—that is, you associate “oto” to a concept very similar to the one I associate to “sound”—then I can say “Paper crumpling makes a crackling oto.” It requires extra thought, but I can do it if I want.

  Similarly, if you say “What is the walking-stick of a bowling ball dropping on the floor?” and I know which concept you associate with the syllables “walking-stick,” then I can figure out what you mean. It may require some thought, and give me pause, because I ordinarily associate “walking-stick” with a different concept. But I can do it just fine.

  When humans really want to communicate with each other, we’re hard to stop! If we’re stuck on a deserted island with no common language, we’ll take up sticks and draw pictures in sand.

  Albert’s appeal to the Argument from Common Usage assumes that agreement on language is a cooperatively established public good. Yet Albert assumes this for the sole purpose of rhetorically accusing Barry of breaking the agreement, and endangering the public good. Now the falling-tree argument has gone all the way from botany to semantics to politics; and so Barry responds by challenging Albert for the authority to define the word.

  A rationalist, with the discipline of hugging the query active, would notice that the conversation had gone rather far astray.

  Oh, dear reader, is it all really necessary? Albert knows what Barry means by “sound.” Barry knows what Albert means by “sound.” Both Albert and Barry have access to words, such as “acoustic vibrations” or “auditory experience,” which they already associate to the same concepts, and which can describe events in the forest without ambiguity. If they were stuck on a deserted island, trying to communicate with each other, their work would be done.

  When both sides know what the other side wants to say, and both sides accuse the other side of defecting from “common usage,” then whatever it is they are about, it is clearly not working out a way to communicate with each other. But this is the whole benefit that common usage provides in the first place.

  Why would you argue about the meaning of a word, two sides trying to wrest it back and forth? If it’s just a namespace conflict that has gotten blown out of proportion, and nothing more is at stake, then the two sides need merely generate two new words and use them consistently.

  Yet often categorizations function as hidden inferences and disguised queries. Is atheism a “religion”? If someone is arguing that the reasoning methods used in atheism are on a par with the reasoning methods used in Judaism, or that atheism is on a par with Islam in terms of causally engendering violence, then they have a clear argumentative stake in lumping it all together into an indistinct gray blur of “faith.”

  Or consider the fight to blend together blacks and whites as “people.” This would not be a time to generate two words—what’s at stake is exactly the idea that you shouldn’t draw a moral distinction.

  But once any empirical proposition is at stake, or any moral proposition, you can no longer appeal to common usage.

  If the question is how to cluster together similar things for purposes of inference, empirical predictions will depend on the answer; which means that definitions can be wrong. A conflict of predictions cannot be settled by an opinion poll.

  If you want to know whether atheism should be clustered with supernaturalist religions for purposes of some particular empirical inference, the dictionary can’t answer you.

  If you want to know whether blacks are people, the dictionary can’t answer you.

  If everyone believes that the red light in the sky is Mars the God of War, the dictionary will define “Mars” as the God of War. If everyone believes that fire is the release of phlogiston, the dictionary will define “fire” as the release of phlogiston.

  There is an art to using words; even when definitions are not literally true or false, they are often wiser or more foolish. Dictionaries are mere histories of past usage; if you treat them as supreme arbiters of meaning, it binds you to the wisdom of the past, forbidding you to do better.

  Though do take care to ensure (if you must depart from the wisdom of the past) that people can figure out what you’re trying to swim.

  *

  166

  Empty Labels

  Consider (yet again) the Aristotelian idea of categories. Let’s say that there’s some object with properties A, B, C, D, and E, or at least it looks E-ish.

  FRED: “You mean that thing over there is blue, round, fuzzy, and—”

  ME: “In Aristotelian logic, it’s not supposed to make a difference what the properties are, or what I call them. That’s why I’m just using the letters.”

  Next, I invent the Aristotelian category “zawa,” which describes those objects, all those objects, and only those objects, that have properties A, C, and D.

  ME: “Object 1 is zawa, B, and E.”

  FRED: “And it’s blue—I mean, A—too, right?”

  ME: “That’s implied when I say it’s zawa.”

  FRED: “Still, I’d like you to say it explicitly.”

  ME: “Okay. Object 1 is A, B, zawa, and E.”

  Then I add another word, “yokie,” which describes all and only objects that are B and E; and the word “xippo,” which describes all and only objects which are E but not D.

  ME: “Object 1 is zawa and yokie, but not xippo.”

  FRED: “Wait, is it luminescent? I mean, is it E?”

  ME: “Yes. That is the only possibility on the information given.”

  FRED: “I’d rather you spelled it out.”

  ME: “Fine: Object 1 is A, zawa, B, yokie, C, D, E, and not xippo.”

  FRED: “Amazing! You can tell all that just by looking?”

  Impressive, isn’t it? Let’s invent even more new words: “Bolo” is A, C, and yokie; “mun” is A, C, and xippo; and “merlacdoni
an” is bolo and mun.

  Pointlessly confusing? I think so too. Let’s replace the labels with the definitions:

  “Zawa, B, and E” becomes [A, C, D], B, E

  “Bolo and A” becomes [A, C, [B, E]], A

  “Merlacdonian” becomes [A, C, [B, E]], [A, C, [E, ¬D]].

  And the thing to remember about the Aristotelian idea of categories is that [A, C, D] is the entire information of “zawa.” It’s not just that I can vary the label, but that I can get along just fine without any label at all—the rules for Aristotelian classes work purely on structures like [A, C, D]. To call one of these structures “zawa,” or attach any other label to it, is a human convenience (or inconvenience) which makes not the slightest difference to the Aristotelian rules.

  Let’s say that “human” is to be defined as a mortal featherless biped. Then the classic syllogism would have the form:

  All [mortal, ¬feathers, bipedal] are mortal.

  Socrates is a [mortal, ¬feathers, bipedal].

  Therefore, Socrates is mortal.

  The feat of reasoning looks a lot less impressive now, doesn’t it?

  Here the illusion of inference comes from the labels, which conceal the premises, and pretend to novelty in the conclusion. Replacing labels with definitions reveals the illusion, making visible the tautology’s empirical unhelpfulness. You can never say that Socrates is a [mortal, ¬feathers, biped] until you have observed him to be mortal.

  There’s an idea, which you may have noticed I hate, that “you can define a word any way you like.” This idea came from the Aristotelian notion of categories; since, if you follow the Aristotelian rules exactly and without flaw—which humans never do; Aristotle knew perfectly well that Socrates was human, even though that wasn’t justified under his rules—but, if some imaginary nonhuman entity were to follow the rules exactly, they would never arrive at a contradiction. They wouldn’t arrive at much of anything: they couldn’t say that Socrates is a [mortal, ¬feathers, biped] until they observed him to be mortal.

  But it’s not so much that labels are arbitrary in the Aristotelian system, as that the Aristotelian system works fine without any labels at all—it cranks out exactly the same stream of tautologies, they just look a lot less impressive. The labels are only there to create the illusion of inference.

  So if you’re going to have an Aristotelian proverb at all, the proverb should be, not “I can define a word any way I like,” nor even, “Defining a word never has any consequences,” but rather, “Definitions don’t need words.”

  *

  167

  Taboo Your Words

  In the game Taboo (by Hasbro), the objective is for a player to have their partner guess a word written on a card, without using that word or five additional words listed on the card. For example, you might have to get your partner to say “baseball” without using the words “sport,” “bat,” “hit,” “pitch,” “base” or of course “baseball.”

  As soon as I see a problem like that, I at once think, “An artificial group conflict in which you use a long wooden cylinder to whack a thrown spheroid, and then run between four safe positions.” It might not be the most efficient strategy to convey the word “baseball” under the stated rules—that might be, “It’s what the Yankees play”—but the general skill of blanking a word out of my mind was one I’d practiced for years, albeit with a different purpose.

  In the previous essay we saw how replacing terms with definitions could reveal the empirical unproductivity of the classical Aristotelian syllogism. All humans are mortal (and also, apparently, featherless bipeds); Socrates is human; therefore Socrates is mortal. When we replace the word “human” by its apparent definition, the following underlying reasoning is revealed:

  All [mortal, ¬feathers, biped] are mortal;

  Socrates is a [mortal, ¬feathers, biped];

  Therefore Socrates is mortal.

  But the principle of replacing words by definitions applies much more broadly:

  ALBERT: “A tree falling in a deserted forest makes a sound.”

  BARRY: “A tree falling in a deserted forest does not make a sound.”

  Clearly, since one says “sound” and one says “not sound,” we must have a contradiction, right? But suppose that they both dereference their pointers before speaking:

  ALBERT: “A tree falling in a deserted forest matches [membership test: this event generates acoustic vibrations].”

  BARRY: “A tree falling in a deserted forest does not match [membership test: this event generates auditory experiences].”

  Now there is no longer an apparent collision—all they had to do was prohibit themselves from using the word sound. If “acoustic vibrations” came into dispute, we would just play Taboo again and say “pressure waves in a material medium”; if necessary we would play Taboo again on the word “wave” and replace it with the wave equation. (Play Taboo on “auditory experience” and you get “That form of sensory processing, within the human brain, that takes as input a linear time series of frequency mixes . . .”)

  But suppose, on the other hand, that Albert and Barry were to have the argument:

  ALBERT: “Socrates matches the concept [membership test: this person will die after drinking hemlock].”

  BARRY: “Socrates matches the concept [membership test: this person will not die after drinking hemlock].”

  Now Albert and Barry have a substantive clash of expectations; a difference in what they anticipate seeing after Socrates drinks hemlock. But they might not notice this, if they happened to use the same word “human” for their different concepts.

  You get a very different picture of what people agree or disagree about, depending on whether you take a label’s-eye-view (Albert says “sound” and Barry says “not sound,” so they must disagree) or taking the test’s-eye-view (Albert’s membership test is acoustic vibrations, Barry’s is auditory experience).

  Get together a pack of soi-disant futurists and ask them if they believe we’ll have Artificial Intelligence in thirty years, and I would guess that at least half of them will say yes. If you leave it at that, they’ll shake hands and congratulate themselves on their consensus. But make the term “Artificial Intelligence” taboo, and ask them to describe what they expect to see, without ever using words like “computers” or “think,” and you might find quite a conflict of expectations hiding under that featureless standard word. See also Shane Legg’s compilation of 71 definitions of “intelligence.”

  The illusion of unity across religions can be dispelled by making the term “God” taboo, and asking them to say what it is they believe in; or making the word “faith” taboo, and asking them why they believe it. Though mostly they won’t be able to answer at all, because it is mostly profession in the first place, and you cannot cognitively zoom in on an audio recording.

  When you find yourself in philosophical difficulties, the first line of defense is not to define your problematic terms, but to see whether you can think without using those terms at all. Or any of their short synonyms. And be careful not to let yourself invent a new word to use instead. Describe outward observables and interior mechanisms; don’t use a single handle, whatever that handle may be.

  Albert says that people have “free will.” Barry says that people don’t have “free will.” Well, that will certainly generate an apparent conflict. Most philosophers would advise Albert and Barry to try to define exactly what they mean by “free will,” on which topic they will certainly be able to discourse at great length. I would advise Albert and Barry to describe what it is that they think people do, or do not have, without using the phrase “free will” at all. (If you want to try this at home, you should also avoid the words “choose,” “act,” “decide,” “determined,” “responsible,” or any of their synonyms.)

  This is one of the nonstandard tools in my toolbox, and in my humble opinion, it works way way better than the standard one. It also requires more effort to use; you get what you pay for.
/>
  *

  168

  Replace the Symbol with the Substance

  What does it take to—as in the previous essay’s example—see a “baseball game” as “An artificial group conflict in which you use a long wooden cylinder to whack a thrown spheroid, and then run between four safe positions”? What does it take to play the rationalist version of Taboo, in which the goal is not to find a synonym that isn’t on the card, but to find a way of describing without the standard concept-handle?

  You have to visualize. You have to make your mind’s eye see the details, as though looking for the first time. You have to perform an Original Seeing.

  Is that a “bat”? No, it’s a long, round, tapering, wooden rod, narrowing at one end so that a human can grasp and swing it.

  Is that a “ball”? No, it’s a leather-covered spheroid with a symmetrical stitching pattern, hard but not metal-hard, which someone can grasp and throw, or strike with the wooden rod, or catch.

  Are those “bases”? No, they’re fixed positions on a game field, that players try to run to as quickly as possible because of their safety within the game’s artificial rules.

  The chief obstacle to performing an original seeing is that your mind already has a nice neat summary, a nice little easy-to-use concept handle. Like the word “baseball,” or “bat,” or “base.” It takes an effort to stop your mind from sliding down the familiar path, the easy path, the path of least resistance, where the small featureless word rushes in and obliterates the details you’re trying to see. A word itself can have the destructive force of cliché; a word itself can carry the poison of a cached thought.

  Playing the game of Taboo—being able to describe without using the standard pointer/label/handle—is one of the fundamental rationalist capacities. It occupies the same primordial level as the habit of constantly asking “Why?” or “What does this belief make me anticipate?”