Rationality- From AI to Zombies

by Eliezer Yudkowsky

  I’m not phrasing this well, but then, I’m trying to dissect a confused thought: Einstein belongs to a separate magisterium, the sacred magisterium. The sacred magisterium is distinct from the mundane magisterium; you can’t set out to be Einstein in the way you can set out to be a full professor or a CEO. Only beings with divine potential can enter the sacred magisterium—and then it is only fulfilling a destiny they already have. So if you say you want to outdo Einstein, you’re claiming to already be part of the sacred magisterium—you claim to have the same aura of destiny that Einstein was born with, like a royal birthright . . .

  “But Eliezer,” you say, “surely not everyone can become Einstein.”

  You mean to say, not everyone can do better than Einstein.

  “Um . . . yeah, that’s what I meant.”

  Well . . . in the modern world, you may be correct. You probably should remember that I am a transhumanist, going around looking at people thinking, “You know, it just sucks that not everyone has the potential to do better than Einstein, and this seems like a fixable problem.” It colors one’s attitude.

  But in the modern world, yes, not everyone has the potential to be Einstein.

  Still . . . how can I put this . . .

  There’s a phrase I once heard, can’t remember where: “Just another Jewish genius.” Some poet or author or philosopher or other, brilliant at a young age, doing something not tremendously important in the grand scheme of things, not all that influential, who ended up being dismissed as “Just another Jewish genius.”

  If Einstein had chosen the wrong angle of attack on his problem—if he hadn’t chosen a sufficiently important problem to work on—if he hadn’t persisted for years—if he’d taken any number of wrong turns—or if someone else had solved the problem first—then dear Albert would have ended up as just another Jewish genius.

  Geniuses are rare, but not all that rare. It is not all that implausible to lay claim to the kind of intellect that can get you dismissed as “just another Jewish genius” or “just another brilliant mind who never did anything interesting with their life.” The associated social status here is not high enough to be sacred, so it should seem like an ordinarily evaluable claim.

  But what separates people like this from becoming Einstein, I suspect, is no innate defect of brilliance. It’s things like “lack of an interesting problem”—or, to put the blame where it belongs, “failing to choose an important problem.” It is very easy to fail at this because of the cached thought problem: Tell people to choose an important problem and they will choose the first cache hit for “important problem” that pops into their heads, like “global warming” or “string theory.”

  The truly important problems are often the ones you’re not even considering, because they appear to be impossible, or, um, actually difficult, or worst of all, not clear how to solve. If you worked on them for years, they might not seem so impossible . . . but this is an extra and unusual insight; naive realism will tell you that solvable problems look solvable, and impossible-looking problems are impossible.

  Then you have to come up with a new and worthwhile angle of attack. Most people who are not allergic to novelty will go too far in the other direction, and fall into an affective death spiral.

  And then you’ve got to bang your head on the problem for years, without being distracted by the temptations of easier living. “Life is what happens while we are making other plans,” as the saying goes, and if you want to fulfill your other plans, you’ve often got to be ready to turn down life.

  Society is not set up to support you while you work, either.

  The point being, the problem is not that you need an aura of destiny and the aura of destiny is missing. If you’d met Albert before he published his papers, you would have perceived no aura of destiny about him to match his future high status. He would seem like just another Jewish genius.

  This is not because the royal birthright is concealed, but because it simply is not there. It is not necessary. There is no separate magisterium for people who do important things.

  I say this, because I want to do important things with my life, and I have a genuinely important problem, and an angle of attack, and I’ve been banging my head on it for years, and I’ve managed to set up a support structure for it; and I very frequently meet people who, in one way or another, say: “Yeah? Let’s see your aura of destiny, buddy.”

  What impressed me about Julian Barbour was a quality that I don’t think anyone would have known how to fake without actually having it: Barbour seemed to have seen through Einstein—he talked about Einstein as if everything Einstein had done was perfectly understandable and mundane.

  Though even having realized this, to me it still came as a shock, when Barbour said something along the lines of, “Now here’s where Einstein failed to apply his own methods, and missed the key insight—” But the shock was fleeting, I knew the Law: No gods, no magic, and ancient heroes are milestones to tick off in your rearview mirror.

  This seeing through is something one has to achieve, an insight one has to discover. You cannot see through Einstein just by saying, “Einstein is mundane!” if his work still seems like magic unto you. That would be like declaring “Consciousness must reduce to neurons!” without having any idea of how to do it. It’s true, but it doesn’t solve the problem.

  I’m not going to tell you that Einstein was an ordinary bloke oversold by the media, or that deep down he was a regular schmuck just like everyone else. That would be going much too far. To walk this path, one must acquire abilities some consider to be . . . unnatural. I take a special joy in doing things that people call “humanly impossible,” because it shows that I’m growing up.

  Yet the way that you acquire magical powers is not by being born with them, but by seeing, with a sudden shock, that they really are perfectly normal.

  This is a general principle in life.

  *

  1. Julian Barbour, The End of Time: The Next Revolution in Physics, 1st ed. (New York: Oxford University Press, 1999).

  256

  Class Project

  “Do as well as Einstein?” Jeffreyssai said, incredulously. “Just as well as Einstein? Albert Einstein was a great scientist of his era, but that was his era, not this one! Einstein did not comprehend the Bayesian methods; he lived before the cognitive biases were discovered; he had no scientific grasp of his own thought processes. He was too caught up in the drama of rejecting his era’s quantum mechanics to actually fix it. And while I grant that Einstein reasoned cleanly in the matter of General Relativity—barring that matter of the cosmological constant—he took ten years to do it. Too slow!”

  “Too slow?” repeated Taji incredulously.

  “Too slow! If Einstein were in this classroom now, rather than Earth of the negative first century, I would rap his knuckles! You will not try to do as well as Einstein! You will aspire to do BETTER than Einstein or you may as well not bother!”

  Jeffreyssai shook his head. “Well, I’ve given you enough hints. It is time to test your skills. Now, I know that the other beisutsukai don’t think much of my class projects . . .” Jeffreyssai paused significantly.

  Brennan inwardly sighed. He’d heard this line many times before, in the Bardic Conspiracy, the Competitive Conspiracy: The other teachers think my assignments are too easy, you should be grateful, followed by some ridiculously difficult task—

  “They say,” Jeffreyssai said, “that my projects are too hard; insanely hard; that they pass from the realm of madness into the realm of Sparta; that Laplace himself would catch on fire; they accuse me of trying to tear apart my students’ souls—”

  Oh, crap.

  “But there is a reason,” Jeffreyssai said, “why many of my students have achieved great things; and by that I do not mean high rank in the Bayesian Conspiracy. I expected much of them, and they came to expect much of themselves. So . . .”

  Jeffreyssai took a moment to look over his increasingly disturbed students.
“Here is your assignment. Of quantum mechanics, and General Relativity, you have been told. This is the limit of Eld science, and hence, the limit of public knowledge. The five of you, working on your own, are to produce the correct theory of quantum gravity. Your time limit is one month.”

  “What?” said Brennan, Taji, Styrlyn, and Yin. Hiriwa gave them a puzzled look.

  “Should you succeed,” Jeffreyssai continued, “you will be promoted to beisutsukai of the second dan and sixth level. We will see if you have learned speed. Your clock starts—now.”

  And Jeffreyssai strode out of the room, slamming the door behind him.

  “This is crazy!” Taji cried.

  Hiriwa looked at Taji, bemused. “The solution is not known to us. How can you know it is so difficult?”

  “Because we knew about this problem back in the Eld days! Eld scientists worked on this problem for a lot longer than one month.”

  Hiriwa shrugged. “They were still arguing about many-worlds too, weren’t they?”

  “Enough! There’s no time!”

  The other four students looked to Styrlyn, remembering that he was said to rank high in the Cooperative Conspiracy. There was a brief moment of weighing, of assessing, and then Styrlyn was their leader.

  Styrlyn took a great breath. “We need a list of approaches. Write down all the angles you can think of. Independently—we need your individual components before we start combining. In five minutes, I’ll ask each of you for your best idea first. No wasted thoughts! Go!”

  Brennan grabbed a sheet and his tracer, set the tip to the surface, and then paused. He couldn’t think of anything clever to say about unifying General Relativity and quantum mechanics . . .

  The other students were already writing.

  Brennan tapped the tip, once, twice, thrice. General Relativity and quantum mechanics . . .

  Taji put his first sheet aside, grabbed another.

  Finally, Brennan, for lack of anything clever to say, wrote down the obvious.

  Minutes later, when Styrlyn called time, it was still all he had written.

  “All right,” Styrlyn said, “your best idea. Or the idea you most want the rest of us to take into account in our second components. Taji, go!”

  Taji looked over his sheets. “Okay, I think we’ve got to assume that every avenue that Eld science was trying is a blind alley, or they would have found it. And if this is possible to do in one month, the answer must be, in some sense, elegant. So no multiple dimensions. If we start doing anything that looks like we should call it ‘string theory,’ we’d better stop. Maybe begin by considering how failure to understand decoherence could have led Eld science astray in quantizing gravity.”

  “The opposite of folly is folly,” Hiriwa said. “Let us pretend that Eld science never existed.”

  “No criticisms yet!” said Styrlyn. “Hiriwa, your suggestion?”

  “Get rid of the infinities,” said Hiriwa, “extirpate that which permits them. It should not be a matter of cleverness with integrals. A representation that allows infinity must be false-to-fact.”

  “Yin.”

  “We know from common sense,” Yin said, “that if we stepped outside the universe, we would see time laid out all at once, reality like a crystal. But I once encountered a hint that physics is timeless in a deeper sense than that.” Yin’s eyes were distant, remembering. “Years ago, I found an abandoned city; it had been uninhabited for eras, I think. And behind a door whose locks were broken, carved into one wall: quote .ua sai .ei mi vimcu ty bu le mekso unquote.”

  Brennan translated: Eureka! Eliminate t from the equations. And written in Lojban, the sacred language of science, which meant the unknown writer had thought it to be true.

  “The ‘timeless physics’ of which we’ve all heard rumors,” Yin said, “may be timeless in a very literal sense.”

  “My own contribution,” Styrlyn said. “The quantum physics we’ve learned is over joint positional configurations. It seems like we should be able to take that apart into a spatially local representation, in terms of invariant distant entanglements. Finding that representation might help us integrate with General Relativity, whose curvature is local.”

  “A strangely individualist perspective,” Taji murmured, “for one of the Cooperative Conspiracy.”

  Styrlyn shook his head. “You misunderstand us, then. The first lesson we learn is that groups are made of people . . . no, there is no time for politics. Brennan!”

  Brennan shrugged. “Not much, I’m afraid, only the obvious. Inertial mass-energy was always observed to equal gravitational mass-energy, and Einstein showed that they were necessarily the same. So why is the ‘energy’ that is an eigenvalue of the quantum Hamiltonian necessarily the same as the ‘energy’ quantity that appears in the equations of General Relativity? Why should spacetime curve at the same rate that the little arrows rotate?”

  There was a brief pause.

  Yin frowned. “That seems too obvious. Wouldn’t Eld science have figured it out already?”

  “Forget Eld science existed,” Hiriwa said. “The question stands: we need the answer, whether it was known in ancient times or not. It cannot possibly be coincidence.”

  Taji’s eyes were abstracted. “Perhaps it would be possible to show that an exception to the equality would violate some conservation law . . .”

  “That is not where Brennan pointed,” Hiriwa interrupted. “He did not ask for a proof that they must be set equal, given some appealing principle; he asked for a view in which the two are one and cannot be divided even conceptually, as was accomplished for inertial mass-energy and gravitational mass-energy. For we must assume that the beauty of the whole arises from the fundamental laws, and not the other way around. Fair-rephrasing?”

  “Fair-rephrasing,” Brennan replied.

  Silence reigned for thirty-seven seconds, as the five pondered the five suggestions.

  “I have an idea . . .”

  *

  Interlude

  A Technical Explanation of Technical Explanation

  As Jaynes emphasizes, the theorems of Bayesian probability theory are just that—mathematical theorems that follow inevitably from Bayesian axioms.1 One might naively think that there would be no controversy about mathematical theorems. But when do the theorems apply? How do we use the theorems in real-world problems? The Intuitive Explanation tries to avoid controversy, but the Technical Explanation willfully walks into the whirling helicopter blades. Bluntly, the reasoning in the Technical Explanation does not represent the unanimous consensus of Earth’s entire planetary community of Bayesian researchers. At least, not yet.

  Where the Intuitive Explanation focused on providing a firm grasp of Bayesian basics, A Technical Explanation of Technical Explanation builds, on a Bayesian foundation, theses about human rationality and philosophy of science. The Technical Explanation of Technical Explanation is so named because it begins with this question:

  “What is the difference between a technical understanding and a verbal understanding?”

  * * *

  As a child I read books of popular physics, and fancied myself knowledgeable; I thought I knew that sound was waves of air, that light was waves of electromagnetism, that matter was waves of complex probability amplitudes. When I grew up, I read the Feynman Lectures on Physics and took the time to understand “the wave equation.”2 And then I realized that up to that point, I had not understood or believed “sound is waves” in anything like the way a physicist means and believes that sentence.

  So that is the difference between a technical understanding and a verbal understanding.

  Do you believe that? If so, you should have applied the knowledge, and said: “But why didn’t you give a technical explanation instead of a verbal explanation?”

  * * *

  Visualize probability density or probability mass—probability as a lump of clay that you must distribute over possible outcomes.

  Let’s say there’s a little light t
hat can flash red, blue, or green each time you press a button. The light flashes one and only one color on each press of the button; the possibilities are mutually exclusive. You’re trying to predict the color of the next flash. On each try, you have a weight of clay, the probability mass, that you have to distribute over the possibilities red, green, and blue. You might put a fourth of your clay on the green possibility, a fourth of your clay on the blue possibility, and half your clay on the red possibility—like assigning probabilities of 25% to green, 25% to blue, and 50% to red. The metaphor is that probability is a conserved resource, to dole out sparingly. If you think that blue is more likely to flash on the next experiment, you can assign a higher probability to blue, but you have to take the probability mass from the other hypotheses—maybe steal some clay from red and add it to blue. You can never get any more clay. Your probabilities can’t sum to more than 1.0 (100%). You can’t predict a 75% chance of seeing red and an 80% chance of seeing blue.

  Why would you want to be careful with your probability mass, or dole it out sparingly? Why not slop probability all over the place? Let’s shift the metaphor from clay to money. You can bet up to a dollar of play money on each press of the button. An experimenter stands nearby, and pays you an amount of real money that depends on how much play money you bet on the winning light. We don’t care how you distributed your remaining play money over the losing lights. The only thing that matters is how much you bet on the light that actually won.

  But we must carefully construct the scoring rule used to pay off the winners, if we want the players to be careful with their bets. Suppose the experimenter pays each player real money equal to the play money bet on the winning color. Under this scoring rule, if you observe that red comes up six times out of ten, your best strategy is to bet, not 60 cents on red, but the entire dollar on red, and you don’t care about the frequencies of blue and green. Why? Let’s say that blue and green each come up around two times out of ten. And suppose you bet 60 cents on red, 20 cents on blue, and 20 cents on green. In this case, six times out of ten you would win 60 cents, and four times out of ten you would win 20 cents, for an average payoff of 44 cents. Under that scoring rule, it makes more sense to allocate the entire dollar to red, and win an entire dollar six times out of ten. Four times out of ten you would win nothing. Your average payoff would be 60 cents.