Rationality- From AI to Zombies

by Eliezer Yudkowsky

  But that’s only one vision of the future. In another vision, the knowledge we now call “science” is taken out of the public domain—the books and journals hidden away, guarded by mystic cults of gurus wearing robes, requiring fearsome initiation rituals for access—so that more people will actually study it.

  I mean, right now, people can study science but they don’t.

  “Scarcity,” it’s called in social psychology. What appears to be in limited supply is more highly valued. And this effect is especially strong with information—we’re much more likely to try to obtain information that we believe is secret, and to value it more when we do obtain it.

  With science, I think, people assume that if the information is freely available, it must not be important. So instead people join cults that have the sense to keep their Great Truths secret. The Great Truth may actually be gibberish, but it’s more satisfying than coherent science, because it’s secret.

  Science is the great Purloined Letter of our times, left out in the open and ignored.

  Sure, scientific openness helps the scientific elite. They’ve already been through the initiation rituals. But for the rest of the planet, science is kept secret a hundred times more effectively by making it freely available, than if its books were guarded in vaults and you had to walk over hot coals to get access. (This being a fearsome trial indeed, since the great secrets of insulation are only available to Physicist-Initiates of the Third Level.)

  If scientific knowledge were hidden in ancient vaults (rather than hidden in inconvenient pay-for-access journals), at least then people would try to get into the vaults. They’d be desperate to learn science. Especially when they saw the power that Eighth Level Physicists could wield, and were told that they weren’t allowed to know the explanation.

  And if you tried to start a cult around oh, say, Scientology, you’d get some degree of public interest, at first. But people would very quickly start asking uncomfortable questions like “Why haven’t you given a public demonstration of your Eighth Level powers, like the Physicists?” and “How come none of the Master Mathematicians seem to want to join your cult?” and “Why should I follow your Founder when they aren’t an Eighth Level anything outside their own cult?” and “Why should I study your cult first, when the Dentists of Doom can do things that are so much more impressive?”

  When you look at it from that perspective, the escape of math from the Pythagorean cult starts to look like a major strategic blunder for humanity.

  Now, I know what you’re going to say: “But science is surrounded by fearsome initiation rituals! Plus it’s inherently difficult to learn! Why doesn’t that count?” Because the public thinks that science is freely available, that’s why. If you’re allowed to learn, it must not be important enough to learn.

  It’s an image problem, people taking their cues from others’ attitudes. Just anyone can walk into the supermarket and buy a light bulb, and nobody looks at it with awe and reverence. The physics supposedly isn’t secret (even though you don’t know), and there’s a one-paragraph explanation in the newspaper that sounds vaguely authoritative and convincing—essentially, no one treats the lightbulb as a sacred mystery, so neither do you.

  Even the simplest little things, completely inert objects like crucifixes, can become magical if everyone looks at them like they’re magic. But since you’re theoretically allowed to know why the light bulb works without climbing the mountain to find the remote Monastery of Electricians, there’s no need to actually bother to learn.

  Now, because science does in fact have initiation rituals both social and cognitive, scientists are not wholly dissatisfied with their science. The problem is that, in the present world, very few people bother to study science in the first place. Science cannot be the true Secret Knowledge, because just anyone is allowed to know it—even though, in fact, they don’t.

  If the Great Secret of Natural Selection, passed down from Darwin Who Is Not Forgotten, was only ever imparted to you after you paid $2,000 and went through a ceremony involving torches and robes and masks and sacrificing an ox, then when you were shown the fossils, and shown the optic cable going through the retina under a microscope, and finally told the Truth, you would say “That’s the most brilliant thing ever!” and be satisfied. After that, if some other cult tried to tell you it was actually a bearded man in the sky 6000 years ago, you’d laugh like hell.

  And you know, it might actually be more fun to do things that way. Especially if the initiation required you to put together some of the evidence for yourself—together, or with classmates—before you could tell your Science Sensei you were ready to advance to the next level. It wouldn’t be efficient, sure, but it would be fun.

  If humanity had never made the mistake—never gone down the religious path, and never learned to fear anything that smacks of religion—then maybe the PhD granting ceremony would involve litanies and chanting, because, hey, that’s what people like. Why take the fun out of everything?

  Maybe we’re just doing it wrong.

  And no, I’m not seriously proposing that we try to reverse the last five hundred years of openness and classify all the science secret. At least, not at the moment. Efficiency is important for now, especially in things like medical research. I’m just explaining why it is that I won’t tell anyone the Secret of how the ineffable difference between blueness and redness arises from mere atoms for less than $100,000—

  Ahem! I meant to say, I’m telling you about this vision of an alternate Earth, so that you give science equal treatment with cults. So that you don’t undervalue scientific truth when you learn it, just because it doesn’t seem to be protected appropriately to its value. Imagine the robes and masks. Visualize yourself creeping into the vaults and stealing the Lost Knowledge of Newton. And don’t be fooled by any organization that does use robes and masks, unless they also show you the data.

  People seem to have holes in their minds for Esoteric Knowledge, Deep Secrets, the Hidden Truth. And I’m not even criticizing this psychology! There are deep secret esoteric hidden truths, like quantum mechanics or Bayes-structure. We’ve just gotten into the habit of presenting the Hidden Truth in a very unsatisfying way, wrapped up in false mundanity.

  But if the holes for secret knowledge are not filled by true beliefs, they will be filled by false beliefs. There is nothing but science to learn—the emotional energy must either be invested in reality, or wasted in total nonsense, or destroyed. For myself, I think it is better to invest the emotional energy; fun should not be needlessly cast away.

  Right now, we’ve got the worst of both worlds. Science isn’t really free, because the courses are expensive and the textbooks are expensive. But the public thinks that anyone is allowed to know, so it must not be important.

  Ideally, you would want to arrange things the other way around.

  *

  213

  Initiation Ceremony

  The torches that lit the narrow stairwell burned intensely and in the wrong color, flame like melting gold or shattered suns.

  192 . . . 193 . . .

  Brennan’s sandals clicked softly on the stone steps, snicking in sequence, like dominos very slowly falling.

  227 . . . 228 . . .

  Half a circle ahead of him, a trailing fringe of dark cloth whispered down the stairs, the robed figure itself staying just out of sight.

  239 . . . 240 . . .

  Not much longer, Brennan predicted to himself, and his guess was accurate:

  Sixteen times sixteen steps was the number, and they stood before the portal of glass.

  The great curved gate had been wrought with cunning, humor, and close attention to indices of refraction: it warped light, bent it, folded it, and generally abused it, so that there were hints of what was on the other side (stronger light sources, dark walls) but no possible way of seeing through—unless, of course, you had the key: the counter-door, thick for thin and thin for thick, in which case the two would cancel out.

/>   From the robed figure beside Brennan, two hands emerged, gloved in reflective cloth to conceal skin’s color. Fingers like slim mirrors grasped the handles of the warped gate—handles that Brennan had not guessed; in all that distortion, shapes could only be anticipated, not seen.

  “Do you want to know?” whispered the guide; a whisper nearly as loud as an ordinary voice, but not revealing the slightest hint of gender.

  Brennan paused. The answer to the question seemed suspiciously, indeed extraordinarily obvious, even for ritual.

  “Yes,” Brennan said finally.

  The guide only regarded him silently.

  “Yes, I want to know,” said Brennan.

  “Know what, exactly?” whispered the figure.

  Brennan’s face scrunched up in concentration, trying to visualize the game to its end, and hoping he hadn’t blown it already; until finally he fell back on the first and last resort, which is the truth:

  “It doesn’t matter,” said Brennan, “the answer is still yes.”

  The glass gate parted down the middle, and slid, with only the tiniest scraping sound, into the surrounding stone.

  The revealed room was lined, wall-to-wall, with figures robed and hooded in light-absorbing cloth. The straight walls were not themselves black stone, but mirrored, tiling a square grid of dark robes out to infinity in all directions; so that it seemed as if the people of some much vaster city, or perhaps the whole human kind, watched in assembly. There was a hint of moist warmth in the air of the room, the breath of the gathered: a scent of crowds.

  Brennan’s guide moved to the center of the square, where burned four torches of that relentless yellow flame. Brennan followed, and when he stopped, he realized with a slight shock that all the cowled hoods were now looking directly at him. Brennan had never before in his life been the focus of such absolute attention; it was frightening, but not entirely unpleasant.

  “He is here,” said the guide in that strange loud whisper.

  The endless grid of robed figures replied in one voice: perfectly blended, exactly synchronized, so that not a single individual could be singled out from the rest, and betrayed:

  “Who is absent?”

  “Jakob Bernoulli,” intoned the guide, and the walls replied:

  “Is dead but not forgotten.”

  “Abraham de Moivre,”

  “Is dead but not forgotten.”

  “Pierre-Simon Laplace,”

  “Is dead but not forgotten.”

  “Edwin Thompson Jaynes,”

  “Is dead but not forgotten.”

  “They died,” said the guide, “and they are lost to us; but we still have each other, and the project continues.”

  In the silence, the guide turned to Brennan, and stretched forth a hand, on which rested a small ring of nearly transparent material.

  Brennan stepped forward to take the ring—

  But the hand clenched tightly shut.

  “If three-fourths of the humans in this room are women,” said the guide, “and three-fourths of the women and half of the men belong to the Heresy of Virtue, and I am a Virtuist, what is the probability that I am a man?”

  “Two-elevenths,” Brennan said confidently.

  There was a moment of absolute silence.

  Then a titter of shocked laughter.

  The guide’s whisper came again, truly quiet this time, almost nonexistent: “It’s one-sixth, actually.”

  Brennan’s cheeks were flaming so hard that he thought his face might melt off. The instinct was very strong to run out of the room and up the stairs and flee the city and change his name and start his life over again and get it right this time.

  “An honest mistake is at least honest,” said the guide, louder now, “and we may know the honesty by its relinquishment. If I am a Virtuist, what is the probability that I am a man?”

  “One—” brennan started to say.

  Then he stopped. Again, the horrible silence.

  “Just say ‘one-sixth’ already,” stage-whispered the figure, this time loud enough for the walls to hear; then there was more laughter, not all of it kind.

  Brennan was breathing rapidly and there was sweat on his forehead. If he was wrong about this, he really was going to flee the city. “Three fourths women times three fourths Virtuists is nine sixteenths female Virtuists in this room. One fourth men times one half Virtuists is two sixteenths male Virtuists. If I have only that information and the fact that you are a Virtuist, I would then estimate odds of two to nine, or a probability of two-elevenths, that you are male. Though I do not, in fact, believe the information given is correct. For one thing, it seems too neat. For another, there are an odd number of people in this room.”

  The hand stretched out again, and opened.

  Brennan took the ring. It looked almost invisible, in the torchlight; not glass, but some material with a refractive index very close to air. The ring was warm from the guide’s hand, and felt like a tiny living thing as it embraced his finger.

  The relief was so great that he nearly didn’t hear the cowled figures applauding.

  From the robed guide came one last whisper:

  “You are now a novice of the Bayesian Conspiracy.”

  *

  Part R

  Physicalism 201

  214

  Hand vs. Fingers

  Back to our original topic: Reductionism and the Mind Projection Fallacy. There can be emotional problems in accepting reductionism, if you think that things have to be fundamental to be fun. But this position commits us to never taking joy in anything more complicated than a quark, and so I prefer to reject it.

  To review, the reductionist thesis is that we use multi-level models for computational reasons, but physical reality has only a single level.

  Here I’d like to pose the following conundrum: When you pick up a cup of water, is it your hand that picks it up?

  Most people, of course, go with the naive popular answer: “Yes.”

  Recently, however, scientists have made a stunning discovery: It’s not your hand that holds the cup, it’s actually your fingers, thumb, and palm.

  Yes, I know! I was shocked too. But it seems that after scientists measured the forces exerted on the cup by each of your fingers, your thumb, and your palm, they found there was no force left over—so the force exerted by your hand must be zero.

  The theme here is that, if you can see how (not just know that) a higher level reduces to a lower one, they will not seem like separate things within your map; you will be able to see how silly it is to think that your fingers could be in one place, and your hand somewhere else; you will be able to see how silly it is to argue about whether it is your hand that picks up the cup, or your fingers.

  The operative word is “see,” as in concrete visualization. Imagining your hand causes you to imagine the fingers and thumb and palm; conversely, imagining fingers and thumb and palm causes you to identify a hand in the mental picture. Thus the high level of your map and the low level of your map will be tightly bound together in your mind.

  In reality, of course, the levels are bound together even tighter than that—bound together by the tightest possible binding: physical identity. You can see this: You can see that saying (1) “hand” or (2) “fingers and thumb and palm,” does not refer to different things, but different points of view.

  But suppose you lack the knowledge to so tightly bind together the levels of your map. For example, you could have a “hand scanner” that showed a “hand” as a dot on a map (like an old-fashioned radar display), and similar scanners for fingers/thumbs/palms; then you would see a cluster of dots around the hand, but you would be able to imagine the hand-dot moving off from the others. So, even though the physical reality of the hand (that is, the thing the dot corresponds to) was identical with / strictly composed of the physical realities of the fingers and thumb and palm, you would not be able to see this fact; even if someone told you, or you guessed from the correspondence of the dots, you woul
d only know the fact of reduction, not see it. You would still be able to imagine the hand dot moving around independently, even though, if the physical makeup of the sensors were held constant, it would be physically impossible for this to actually happen.

  Or, at a still lower level of binding, people might just tell you “There’s a hand over there, and some fingers over there”—in which case you would know little more than a Good-Old-Fashioned AI representing the situation using suggestively named LISP tokens. There wouldn’t be anything obviously contradictory about asserting:

  ⊢Inside(Room,Hand)

  ⊢¬Inside(Room,Fingers),

  because you would not possess the knowledge

  ⊢Inside(x,Hand)⇒Inside(x,Fingers).

  None of this says that a hand can actually detach its existence from your fingers and crawl, ghostlike, across the room; it just says that a Good-Old-Fashioned AI with a propositional representation may not know any better. The map is not the territory.

  In particular, you shouldn’t draw too many conclusions from how it seems conceptually possible, in the mind of some specific conceiver, to separate the hand from its constituent elements of fingers, thumb, and palm. Conceptual possibility is not the same as logical possibility or physical possibility.

  It is conceptually possible to you that 235,757 is prime, because you don’t know any better. But it isn’t logically possible that 235,757 is prime; if you were logically omniscient, 235,757 would be obviously composite (and you would know the factors). That that’s why we have the notion of impossible possible worlds, so that we can put probability distributions on propositions that may or may not be in fact logically impossible.

  And you can imagine philosophers who criticize “eliminative fingerists” who contradict the direct facts of experience—we can feel our hand holding the cup, after all—by suggesting that “hands” don’t really exist, in which case, obviously, the cup would fall down. And philosophers who suggest “appendigital bridging laws” to explain how a particular configuration of fingers evokes a hand into existence—with the note, of course, that while our world contains those particular appendigital bridging laws, the laws could have been conceivably different, and so are not in any sense necessary facts, etc.