Rationality- From AI to Zombies

by Eliezer Yudkowsky

  You try to treat category membership as all-or-nothing, ignoring the existence of more and less typical subclusters. Ducks and penguins are less typical birds than robins and pigeons. Interestingly, a between-groups experiment showed that subjects thought a disease was more likely to spread from robins to ducks on an island, than from ducks to robins. (Typicality and Asymmetrical Similarity)

  A verbal definition works well enough in practice to point out the intended cluster of similar things, but you nitpick exceptions. Not every human has ten fingers, or wears clothes, or uses language; but if you look for an empirical cluster of things which share these characteristics, you’ll get enough information that the occasional nine-fingered human won’t fool you. (The Cluster Structure of Thingspace)

  You ask whether something “is” or “is not” a category member but can’t name the question you really want answered. What is a “man”? Is Barney the Baby Boy a “man”? The “correct” answer may depend considerably on whether the query you really want answered is “Would hemlock be a good thing to feed Barney?” or “Will Barney make a good husband?” (Disguised Queries)

  You treat intuitively perceived hierarchical categories like the only correct way to parse the world, without realizing that other forms of statistical inference are possible even though your brain doesn’t use them. It’s much easier for a human to notice whether an object is a “blegg” or “rube”; than for a human to notice that red objects never glow in the dark, but red furred objects have all the other characteristics of bleggs. Other statistical algorithms work differently. (Neural Categories)

  You talk about categories as if they are manna fallen from the Platonic Realm, rather than inferences implemented in a real brain. The ancient philosophers said “Socrates is a man,” not, “My brain perceptually classifies Socrates as a match against the ‘human’ concept.” (How An Algorithm Feels From Inside)

  You argue about a category membership even after screening off all questions that could possibly depend on a category-based inference. After you observe that an object is blue, egg-shaped, furred, flexible, opaque, luminescent, and palladium-containing, what’s left to ask by arguing, “Is it a blegg?” But if your brain’s categorizing neural network contains a (metaphorical) central unit corresponding to the inference of blegg-ness, it may still feel like there’s a leftover question. (How An Algorithm Feels From Inside)

  You allow an argument to slide into being about definitions, even though it isn’t what you originally wanted to argue about. If, before a dispute started about whether a tree falling in a deserted forest makes a “sound,” you asked the two soon-to-be arguers whether they thought a “sound” should be defined as “acoustic vibrations” or “auditory experiences,” they’d probably tell you to flip a coin. Only after the argument starts does the definition of a word become politically charged. (Disputing Definitions)

  You think a word has a meaning, as a property of the word itself; rather than there being a label that your brain associates to a particular concept. When someone shouts “Yikes! A tiger!,” evolution would not favor an organism that thinks, “Hm . . . I have just heard the syllables ‘Tie’ and ‘Grr’ which my fellow tribemembers associate with their internal analogues of my own tiger concept and which aiiieeee CRUNCH CRUNCH GULP.” So the brain takes a shortcut, and it seems that the meaning of tigerness is a property of the label itself. People argue about the correct meaning of a label like “sound.” (Feel the Meaning)

  You argue over the meanings of a word, even after all sides understand perfectly well what the other sides are trying to say. The human ability to associate labels to concepts is a tool for communication. When people want to communicate, we’re hard to stop; if we have no common language, we’ll draw pictures in sand. When you each understand what is in the other’s mind, you are done. (The Argument From Common Usage)

  You pull out a dictionary in the middle of an empirical or moral argument. Dictionary editors are historians of usage, not legislators of language. If the common definition contains a problem—if “Mars” is defined as the God of War, or a “dolphin” is defined as a kind of fish, or “Negroes” are defined as a separate category from humans, the dictionary will reflect the standard mistake. (The Argument From Common Usage)

  You pull out a dictionary in the middle of any argument ever. Seriously, what the heck makes you think that dictionary editors are an authority on whether “atheism” is a “religion” or whatever? If you have any substantive issue whatsoever at stake, do you really think dictionary editors have access to ultimate wisdom that settles the argument? (The Argument From Common Usage)

  You defy common usage without a reason, making it gratuitously hard for others to understand you. Fast stand up plutonium, with bagels without handle. (The Argument From Common Usage)

  You use complex renamings to create the illusion of inference. Is a “human” defined as a “mortal featherless biped”? Then write: “All [mortal featherless bipeds] are mortal; Socrates is a [mortal featherless biped]; therefore, Socrates is mortal.” Looks less impressive that way, doesn’t it? (Empty Labels)

  You get into arguments that you could avoid if you just didn’t use the word. If Albert and Barry aren’t allowed to use the word “sound,” then Albert will have to say “A tree falling in a deserted forest generates acoustic vibrations,” and Barry will say “A tree falling in a deserted forest generates no auditory experiences.” When a word poses a problem, the simplest solution is to eliminate the word and its synonyms. (Taboo Your Words)

  The existence of a neat little word prevents you from seeing the details of the thing you’re trying to think about. What actually goes on in schools once you stop calling it “education”? What’s a degree, once you stop calling it a “degree”? If a coin lands “heads,” what’s its radial orientation? What is “truth,” if you can’t say “accurate” or “correct” or “represent” or “reflect” or “semantic” or “believe” or “knowledge” or “map” or “real” or any other simple term? (Replace the Symbol with the Substance)

  You have only one word, but there are two or more different things-in-reality, so that all the facts about them get dumped into a single undifferentiated mental bucket. It’s part of a detective’s ordinary work to observe that Carol wore red last night, or that she has black hair; and it’s part of a detective’s ordinary work to wonder if maybe Carol dyes her hair. But it takes a subtler detective to wonder if there are two Carols, so that the Carol who wore red is not the same as the Carol who had black hair. (Fallacies of Compression)

  You see patterns where none exist, harvesting other characteristics from your definitions even when there is no similarity along that dimension. In Japan, it is thought that people of blood type A are earnest and creative, blood type Bs are wild and cheerful, blood type Os are agreeable and sociable, and blood type ABs are cool and controlled. (Categorizing Has Consequences)

  You try to sneak in the connotations of a word, by arguing from a definition that doesn’t include the connotations. A “wiggin” is defined in the dictionary as a person with green eyes and black hair. The word “wiggin” also carries the connotation of someone who commits crimes and launches cute baby squirrels, but that part isn’t in the dictionary. So you point to someone and say: “Green eyes? Black hair? See, told you he’s a wiggin! Watch, next he’s going to steal the silverware.” (Sneaking in Connotations)

  You claim “X, by definition, is a Y!” On such occasions you’re almost certainly trying to sneak in a connotation of Y that wasn’t in your given definition. You define “human” as a “featherless biped,” and point to Socrates and say, “No feathers—two legs—he must be human!” But what you really care about is something else, like mortality. If what was in dispute was Socrates’s number of legs, the other fellow would just reply, “Whaddaya mean, Socrates’s got two legs? That’s what we’re arguing about in the first place!” (Arguing “By Definition”)

  You claim “Ps, by definition, are Qs!” If y
ou see Socrates out in the field with some biologists, gathering herbs that might confer resistance to hemlock, there’s no point in arguing “Men, by definition, are mortal!” The main time you feel the need to tighten the vise by insisting that something is true “by definition” is when there’s other information that calls the default inference into doubt. (Arguing “By Definition”)

  You try to establish membership in an empirical cluster “by definition.” You wouldn’t feel the need to say, “Hinduism, by definition, is a religion!” because, well, of course Hinduism is a religion. It’s not just a religion “by definition,” it’s, like, an actual religion. Atheism does not resemble the central members of the “religion” cluster, so if it wasn’t for the fact that atheism is a religion by definition, you might go around thinking that atheism wasn’t a religion. That’s why you’ve got to crush all opposition by pointing out that “Atheism is a religion” is true by definition, because it isn’t true any other way. (Arguing “By Definition”)

  Your definition draws a boundary around things that don’t really belong together. You can claim, if you like, that you are defining the word “fish” to refer to salmon, guppies, sharks, dolphins, and trout, but not jellyfish or algae. You can claim, if you like, that this is merely a list, and there is no way a list can be “wrong.” Or you can stop playing games and admit that you made a mistake and that dolphins don’t belong on the fish list. (Where to Draw the Boundary?)

  You use a short word for something that you won’t need to describe often, or a long word for something you’ll need to describe often. This can result in inefficient thinking, or even misapplications of Occam’s Razor, if your mind thinks that short sentences sound “simpler.” Which sounds more plausible, “God did a miracle” or “A supernatural universe-creating entity temporarily suspended the laws of physics”? (Entropy, and Short Codes)

  You draw your boundary around a volume of space where there is no greater-than-usual density, meaning that the associated word does not correspond to any performable Bayesian inferences. Since green-eyed people are not more likely to have black hair, or vice versa, and they don’t share any other characteristics in common, why have a word for “wiggin”? (Mutual Information, and Density in Thingspace)

  You draw an unsimple boundary without any reason to do so. The act of defining a word to refer to all humans, except black people, seems kind of suspicious. If you don’t present reasons to draw that particular boundary, trying to create an “arbitrary” word in that location is like a detective saying: “Well, I haven’t the slightest shred of support one way or the other for who could’ve murdered those orphans . . . but have we considered John Q. Wiffleheim as a suspect?” (Superexponential Conceptspace, and Simple Words)

  You use categorization to make inferences about properties that don’t have the appropriate empirical structure, namely, conditional independence given knowledge of the class, to be well-approximated by Naive Bayes. No way am I trying to summarize this one. Just read the essay. (Conditional Independence, and Naive Bayes)

  You think that words are like tiny little LISP symbols in your mind, rather than words being labels that act as handles to direct complex mental paintbrushes that can paint detailed pictures in your sensory workspace. Visualize a “triangular lightbulb.” What did you see? (Words as Mental Paintbrush Handles)

  You use a word that has different meanings in different places as though it meant the same thing on each occasion, possibly creating the illusion of something protean and shifting. “Martin told Bob the building was on his left.” But “left” is a function-word that evaluates with a speaker-dependent variable grabbed from the surrounding context. Whose “left” is meant, Bob’s or Martin’s? (Variable Question Fallacies)

  You think that definitions can’t be “wrong,” or that “I can define a word any way I like!” This kind of attitude teaches you to indignantly defend your past actions, instead of paying attention to their consequences, or fessing up to your mistakes. (37 Ways That Suboptimal Use Of Categories Can Have Negative Side Effects On Your Cognition)

  Everything you do in the mind has an effect, and your brain races ahead unconsciously without your supervision.

  Saying “Words are arbitrary; I can define a word any way I like” makes around as much sense as driving a car over thin ice with the accelerator floored and saying, “Looking at this steering wheel, I can’t see why one radial angle is special—so I can turn the steering wheel any way I like.”

  If you’re trying to go anywhere, or even just trying to survive, you had better start paying attention to the three or six dozen optimality criteria that control how you use words, definitions, categories, classes, boundaries, labels, and concepts.

  *

  Interlude

  An Intuitive Explanation of Bayes’s Theorem

  Editor’s Note: This is an abridgement of the original version of this essay, which contained many interactive elements.

  Your friends and colleagues are talking about something called “Bayes’s Theorem” or “Bayes’s Rule,” or something called Bayesian reasoning. They sound really enthusiastic about it, too, so you google and find a web page about Bayes’s Theorem and . . .

  It’s this equation. That’s all. Just one equation. The page you found gives a definition of it, but it doesn’t say what it is, or why it’s useful, or why your friends would be interested in it. It looks like this random statistics thing.

  Why does a mathematical concept generate this strange enthusiasm in its students? What is the so-called Bayesian Revolution now sweeping through the sciences, which claims to subsume even the experimental method itself as a special case? What is the secret that the adherents of Bayes know? What is the light that they have seen?

  Soon you will know. Soon you will be one of us.

  While there are a few existing online explanations of Bayes’s Theorem, my experience with trying to introduce people to Bayesian reasoning is that the existing online explanations are too abstract. Bayesian reasoning is very counterintuitive. People do not employ Bayesian reasoning intuitively, find it very difficult to learn Bayesian reasoning when tutored, and rapidly forget Bayesian methods once the tutoring is over. This holds equally true for novice students and highly trained professionals in a field. Bayesian reasoning is apparently one of those things which, like quantum mechanics or the Wason Selection Test, is inherently difficult for humans to grasp with our built-in mental faculties.

  Or so they claim. Here you will find an attempt to offer an intuitive explanation of Bayesian reasoning—an excruciatingly gentle introduction that invokes all the human ways of grasping numbers, from natural frequencies to spatial visualization. The intent is to convey, not abstract rules for manipulating numbers, but what the numbers mean, and why the rules are what they are (and cannot possibly be anything else). When you are finished reading this, you will see Bayesian problems in your dreams.

  And let’s begin.

  * * *

  Here’s a story problem about a situation that doctors often encounter:

  1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

  What do you think the answer is? If you haven’t encountered this kind of problem before, please take a moment to come up with your own answer before continuing.

  * * *

  Next, suppose I told you that most doctors get the same wrong answer on this problem—usually, only around 15% of doctors get it right. (“Really? 15%? Is that a real number, or an urban legend based on an Internet poll?” It’s a real number. See Casscells, Schoenberger, and Graboys 1978;1 Eddy 1982;2 Gigerenzer and Hoffrage 1995;3 and many other studies. It’s a surprising result which is easy to replicate, so it’s been extensively replicated.)

>   On the story problem above, most doctors estimate the probability to be between 70% and 80%, which is wildly incorrect.

  Here’s an alternate version of the problem on which doctors fare somewhat better:

  10 out of 1,000 women at age forty who participate in routine screening have breast cancer. 800 out of 1,000 women with breast cancer will get positive mammographies. 96 out of 1,000 women without breast cancer will also get positive mammographies. If 1,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?

  And finally, here’s the problem on which doctors fare best of all, with 46%—nearly half—arriving at the correct answer:

  100 out of 10,000 women at age forty who participate in routine screening have breast cancer. 80 of every 100 women with breast cancer will get a positive mammography. 950 out of 9,900 women without breast cancer will also get a positive mammography. If 10,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?

  * * *

  The correct answer is 7.8%, obtained as follows: Out of 10,000 women, 100 have breast cancer; 80 of those 100 have positive mammographies. From the same 10,000 women, 9,900 will not have breast cancer and of those 9,900 women, 950 will also get positive mammographies. This makes the total number of women with positive mammographies 950 + 80 or 1,030. Of those 1,030 women with positive mammographies, 80 will have cancer. Expressed as a proportion, this is 80/1,030 or 0.07767 or 7.8%.