Rationality- From AI to Zombies

by Eliezer Yudkowsky

  And in the same way, when people argue over whether the falling tree makes a sound, or whether Pluto is a planet, they don’t see themselves as arguing over whether a categorization should be active in their neural networks. It seems like either the tree makes a sound, or not.

  We know where Pluto is, and where it’s going; we know Pluto’s shape, and Pluto’s mass—but is it a planet? And yes, there were people who said this was a fight over definitions—but even that is a Network 2 sort of perspective, because you’re arguing about how the central unit ought to be wired up. If you were a mind constructed along the lines of Network 1, you wouldn’t say “It depends on how you define ‘planet,’” you would just say, “Given that we know Pluto’s orbit and shape and mass, there is no question left to ask.” Or, rather, that’s how it would feel—it would feel like there was no question left—if you were a mind constructed along the lines of Network 1.

  Before you can question your intuitions, you have to realize that what your mind’s eye is looking at is an intuition—some cognitive algorithm, as seen from the inside—rather than a direct perception of the Way Things Really Are.

  People cling to their intuitions, I think, not so much because they believe their cognitive algorithms are perfectly reliable, but because they can’t see their intuitions as the way their cognitive algorithms happen to look from the inside.

  And so everything you try to say about how the native cognitive algorithm goes astray, ends up being contrasted to their direct perception of the Way Things Really Are—and discarded as obviously wrong.

  *

  163

  Disputing Definitions

  I have watched more than one conversation—even conversations supposedly about cognitive science—go the route of disputing over definitions. Taking the classic example to be “If a tree falls in a forest, and no one hears it, does it make a sound?,” the dispute often follows a course like this:

  If a tree falls in the forest, and no one hears it, does it make a sound?

  ALBERT: “Of course it does. What kind of silly question is that? Every time I’ve listened to a tree fall, it made a sound, so I’ll guess that other trees falling also make sounds. I don’t believe the world changes around when I’m not looking.”

  BARRY: “Wait a minute. If no one hears it, how can it be a sound?”

  In this example, Barry is arguing with Albert because of a genuinely different intuition about what constitutes a sound. But there’s more than one way the Standard Dispute can start. Barry could have a motive for rejecting Albert’s conclusion. Or Barry could be a skeptic who, upon hearing Albert’s argument, reflexively scrutinized it for possible logical flaws; and then, on finding a counterargument, automatically accepted it without applying a second layer of search for a counter-counterargument; thereby arguing himself into the opposite position. This doesn’t require that Barry’s prior intuition—the intuition Barry would have had, if we’d asked him before Albert spoke—differs from Albert’s.

  Well, if Barry didn’t have a differing intuition before, he sure has one now.

  ALBERT: “What do you mean, there’s no sound? The tree’s roots snap, the trunk comes crashing down and hits the ground. This generates vibrations that travel through the ground and the air. That’s where the energy of the fall goes, into heat and sound. Are you saying that if people leave the forest, the tree violates Conservation of Energy?”

  BARRY: “But no one hears anything. If there are no humans in the forest, or, for the sake of argument, anything else with a complex nervous system capable of ‘hearing,’ then no one hears a sound.”

  Albert and Barry recruit arguments that feel like support for their respective positions, describing in more detail the thoughts that caused their “sound”-detectors to fire or stay silent. But so far the conversation has still focused on the forest, rather than definitions. And note that they don’t actually disagree on anything that happens in the forest.

  ALBERT: “This is the dumbest argument I’ve ever been in. You’re a niddlewicking fallumphing pickleplumber.”

  BARRY: “Yeah? Well, you look like your face caught on fire and someone put it out with a shovel.”

  Insult has been proffered and accepted; now neither party can back down without losing face. Technically, this isn’t part of the argument, as rationalists account such things; but it’s such an important part of the Standard Dispute that I’m including it anyway.

  ALBERT: “The tree produces acoustic vibrations. By definition, that is a sound.”

  BARRY: “No one hears anything. By definition, that is not a sound.”

  The argument starts shifting to focus on definitions. Whenever you feel tempted to say the words “by definition” in an argument that is not literally about pure mathematics, remember that anything which is true “by definition” is true in all possible worlds, and so observing its truth can never constrain which world you live in.

  ALBERT: “My computer’s microphone can record a sound without anyone being around to hear it, store it as a file, and it’s called a ‘sound file.’ And what’s stored in the file is the pattern of vibrations in air, not the pattern of neural firings in anyone’s brain. ‘Sound’ means a pattern of vibrations.”

  Albert deploys an argument that feels like support for the word “sound” having a particular meaning. This is a different kind of question from whether acoustic vibrations take place in a forest—but the shift usually passes unnoticed.

  BARRY: “Oh, yeah? Let’s just see if the dictionary agrees with you.”

  There’s a lot of things I could be curious about in the falling-tree scenario. I could go into the forest and look at trees, or learn how to derive the wave equation for changes of air pressure, or examine the anatomy of an ear, or study the neuroanatomy of the auditory cortex. Instead of doing any of these things, I am to consult a dictionary, apparently. Why? Are the editors of the dictionary expert botanists, expert physicists, expert neuroscientists? Looking in an encyclopedia might make sense, but why a dictionary?

  ALBERT: “Hah! Definition 2c in Merriam-Webster: ‘Sound: Mechanical radiant energy that is transmitted by longitudinal pressure waves in a material medium (as air).’”

  BARRY: “Hah! Definition 2b in Merriam-Webster: ‘Sound: The sensation perceived by the sense of hearing.’”

  ALBERT AND BARRY, CHORUS: “Consarned dictionary! This doesn’t help at all!”

  Dictionary editors are historians of usage, not legislators of language. Dictionary editors find words in current usage, then write down the words next to (a small part of) what people seem to mean by them. If there’s more than one usage, the editors write down more than one definition.

  ALBERT: “Look, suppose that I left a microphone in the forest and recorded the pattern of the acoustic vibrations of the tree falling. If I played that back to someone, they’d call it a ‘sound’! That’s the common usage! Don’t go around making up your own wacky definitions!”

  BARRY: “One, I can define a word any way I like so long as I use it consistently. Two, the meaning I gave was in the dictionary. Three, who gave you the right to decide what is or isn’t common usage?”

  There’s quite a lot of rationality errors in the Standard Dispute. Some of them I’ve already covered, and some of them I’ve yet to cover; likewise the remedies.

  But for now, I would just like to point out—in a mournful sort of way—that Albert and Barry seem to agree on virtually every question of what is actually going on inside the forest, and yet it doesn’t seem to generate any feeling of agreement.

  Arguing about definitions is a garden path; people wouldn’t go down the path if they saw at the outset where it led. If you asked Albert (Barry) why he’s still arguing, he’d probably say something like: “Barry (Albert) is trying to sneak in his own definition of ‘sound,’ the scurvey scoundrel, to support his ridiculous point; and I’m here to defend the standard definition.”

  But suppose I went back in time to before the start of the arg
ument:

  (Eliezer appears from nowhere in a peculiar conveyance that looks just like the time machine from the original The Time Machine movie.)

  BARRY: “Gosh! A time traveler!”

  ELIEZER: “I am a traveler from the future! Hear my words! I have traveled far into the past—around fifteen minutes—”

  ALBERT: “Fifteen minutes?”

  ELIEZER: “—to bring you this message!”

  (There is a pause of mixed confusion and expectancy.)

  ELIEZER: “Do you think that ‘sound’ should be defined to require both acoustic vibrations (pressure waves in air) and also auditory experiences (someone to listen to the sound), or should ‘sound’ be defined as meaning only acoustic vibrations, or only auditory experience?”

  BARRY: “You went back in time to ask us that?”

  ELIEZER: “My purposes are my own! Answer!”

  ALBERT: “Well . . . I don’t see why it would matter. You can pick any definition so long as you use it consistently.”

  BARRY: “Flip a coin. Er, flip a coin twice.”

  ELIEZER: “Personally I’d say that if the issue arises, both sides should switch to describing the event in unambiguous lower-level constituents, like acoustic vibrations or auditory experiences. Or each side could designate a new word, like ‘alberzle’ and ‘bargulum,’ to use for what they respectively used to call ‘sound’; and then both sides could use the new words consistently. That way neither side has to back down or lose face, but they can still communicate. And of course you should try to keep track, at all times, of some testable proposition that the argument is actually about. Does that sound right to you?”

  ALBERT: “I guess . . .”

  BARRY: “Why are we talking about this?”

  ELIEZER: “To preserve your friendship against a contingency you will, now, never know. For the future has already changed!”

  (Eliezer and the machine vanish in a puff of smoke.)

  BARRY: “Where were we again?”

  ALBERT: “Oh, yeah: If a tree falls in the forest, and no one hears it, does it make a sound?”

  BARRY: “It makes an alberzle but not a bargulum. What’s the next question?”

  This remedy doesn’t destroy every dispute over categorizations. But it destroys a substantial fraction.

  *

  164

  Feel the Meaning

  When I hear someone say, “Oh, look, a butterfly,” the spoken phonemes “butterfly” enter my ear and vibrate on my ear drum, being transmitted to the cochlea, tickling auditory nerves that transmit activation spikes to the auditory cortex, where phoneme processing begins, along with recognition of words, and reconstruction of syntax (a by no means serial process), and all manner of other complications.

  But at the end of the day, or rather, at the end of the second, I am primed to look where my friend is pointing and see a visual pattern that I will recognize as a butterfly; and I would be quite surprised to see a wolf instead.

  My friend looks at a butterfly, his throat vibrates and lips move, the pressure waves travel invisibly through the air, my ear hears and my nerves transduce and my brain reconstructs, and lo and behold, I know what my friend is looking at. Isn’t that marvelous? If we didn’t know about the pressure waves in the air, it would be a tremendous discovery in all the newspapers: Humans are telepathic! Human brains can transfer thoughts to each other!

  Well, we are telepathic, in fact; but magic isn’t exciting when it’s merely real, and all your friends can do it too.

  Think telepathy is simple? Try building a computer that will be telepathic with you. Telepathy, or “language,” or whatever you want to call our partial thought transfer ability, is more complicated than it looks.

  But it would be quite inconvenient to go around thinking, “Now I shall partially transduce some features of my thoughts into a linear sequence of phonemes which will invoke similar thoughts in my conversational partner . . .”

  So the brain hides the complexity—or rather, never represents it in the first place—which leads people to think some peculiar thoughts about words.

  As I remarked earlier, when a large yellow striped object leaps at me, I think “Yikes! A tiger!” not “Hm . . . objects with the properties of largeness, yellowness, and stripedness have previously often possessed the properties ‘hungry’ and ‘dangerous,’ and therefore, although it is not logically necessary, auughhhh CRUNCH CRUNCH GULP.”

  Similarly, when someone shouts “Yikes! A tiger!,” natural selection would not favor an organism that thought, “Hm . . . I have just heard the syllables ‘Tie’ and ‘Grr’ which my fellow tribe members associate with their internal analogues of my own tiger concept, and which they are more likely to utter if they see an object they categorize as aiiieeee CRUNCH CRUNCH help it’s got my arm CRUNCH GULP.”

  Figure 164.1: Network 3

  Considering this as a design constraint on the human cognitive architecture, you wouldn’t want any extra steps between when your auditory cortex recognizes the syllables “tiger,” and when the tiger concept gets activated.

  Going back to the parable of bleggs and rubes, and the centralized network that categorizes quickly and cheaply, you might visualize a direct connection running from the unit that recognizes the syllable “blegg” to the unit at the center of the blegg network. The central unit, the blegg concept, gets activated almost as soon as you hear Susan the Senior Sorter say, “Blegg!”

  Or, for purposes of talking—which also shouldn’t take eons—as soon as you see a blue egg-shaped thing and the central blegg unit fires, you holler “Blegg!” to Susan.

  And what that algorithm feels like from inside is that the label, and the concept, are very nearly identified; the meaning feels like an intrinsic property of the word itself.

  The cognoscenti will recognize this as a case of E. T. Jaynes’s “Mind Projection Fallacy.” It feels like a word has a meaning, as a property of the word itself; just like how redness is a property of a red apple, or mysteriousness is a property of a mysterious phenomenon.

  Indeed, on most occasions, the brain will not distinguish at all between the word and the meaning—only bothering to separate the two while learning a new language, perhaps. And even then, you’ll see Susan pointing to a blue egg-shaped thing and saying “Blegg!,” and you’ll think, I wonder what “blegg” means, and not, I wonder what mental category Susan associates to the auditory label “blegg.”

  Consider, in this light, the part of the Standard Dispute of Definitions where the two parties argue about what the word “sound” really means—the same way they might argue whether a particular apple is really red or green:

  ALBERT: “My computer’s microphone can record a sound without anyone being around to hear it, store it as a file, and it’s called a ‘sound file.’ And what’s stored in the file is the pattern of vibrations in air, not the pattern of neural firings in anyone’s brain. ‘Sound’ means a pattern of vibrations.”

  BARRY: “Oh, yeah? Let’s just see if the dictionary agrees with you.”

  Albert feels intuitively that the word “sound” has a meaning and that the meaning is acoustic vibrations. Just as Albert feels that a tree falling in the forest makes a sound (rather than causing an event that matches the sound category).

  Barry likewise feels that:

  sound.meaning == auditory experiences

  forest.sound == false.

  Rather than:

  myBrain.FindConcept("sound") == concept_AuditoryExperience

  concept_AuditoryExperience.match(forest) == false.

  Which is closer to what’s really going on; but humans have not evolved to know this, anymore than humans instinctively know the brain is made of neurons.

  Albert and Barry’s conflicting intuitions provide the fuel for continuing the argument in the phase of arguing over what the word “sound” means—which feels like arguing over a fact like any other fact, like arguing over whether the sky is blue or green.

  You may not e
ven notice that anything has gone astray, until you try to perform the rationalist ritual of stating a testable experiment whose result depends on the facts you’re so heatedly disputing . . .

  *

  165

  The Argument from Common Usage

  Part of the Standard Definitional Dispute runs as follows:

  ALBERT: “Look, suppose that I left a microphone in the forest and recorded the pattern of the acoustic vibrations of the tree falling. If I played that back to someone, they’d call it a ‘sound’! That’s the common usage! Don’t go around making up your own wacky definitions!”

  BARRY: “One, I can define a word any way I like so long as I use it consistently. Two, the meaning I gave was in the dictionary. Three, who gave you the right to decide what is or isn’t common usage?”

  Not all definitional disputes progress as far as recognizing the notion of common usage. More often, I think, someone picks up a dictionary because they believe that words have meanings, and the dictionary faithfully records what this meaning is. Some people even seem to believe that the dictionary determines the meaning—that the dictionary editors are the Legislators of Language. Maybe because back in elementary school, their authority-teacher said that they had to obey the dictionary, that it was a mandatory rule rather than an optional one?