Rationality- From AI to Zombies

by Eliezer Yudkowsky

  “Forming accurate beliefs requires a corresponding amount of evidence” is a very cogent truth both in human relations and in thermodynamics: if blind faith actually worked as a method of investigation, you could turn warm water into electricity and ice cubes. Just build a Maxwell’s Demon that has blind faith in molecule velocities.

  Engines of cognition are not so different from heat engines, though they manipulate entropy in a more subtle form than burning gasoline. For example, to the extent that an engine of cognition is not perfectly efficient, it must radiate waste heat, just like a car engine or refrigerator.

  “Cold rationality” is true in a sense that Hollywood scriptwriters never dreamed (and false in the sense that they did dream).

  So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don’t expect me to believe that a big, elaborate clever argument can do it either.

  *

  187

  Perpetual Motion Beliefs

  The last essay concluded:

  To form accurate beliefs about something, you really do have to observe it. It’s a very physical, very real process: any rational mind does “work” in the thermodynamic sense, not just the sense of mental effort . . . So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don’t expect me to believe that a big, elaborate clever argument can do it either.

  One of the chief morals of the mathematical analogy between thermodynamics and cognition is that the constraints of probability are inescapable; probability may be a “subjective state of belief,” but the laws of probability are harder than steel.

  People learn under the traditional school regimen that the teacher tells you certain things, and you must believe them and recite them back; but if a mere student suggests a belief, you do not have to obey it. They map the domain of belief onto the domain of authority, and think that a certain belief is like an order that must be obeyed, but a probabilistic belief is like a mere suggestion.

  They look at a lottery ticket, and say, “But you can’t prove I won’t win, right?” Meaning: “You may have calculated a low probability of winning, but since it is a probability, it’s just a suggestion, and I am allowed to believe what I want.”

  Here’s a little experiment: Smash an egg on the floor. The rule that says that the egg won’t spontaneously reform and leap back into your hand is merely probabilistic. A suggestion, if you will. The laws of thermodynamics are probabilistic, so they can’t really be laws, the way that “Thou shalt not murder” is a law . . . right?

  So why not just ignore the suggestion? Then the egg will unscramble itself . . . right?

  It may help to think of it this way—if you still have some lingering intuition that uncertain beliefs are not authoritative:

  In reality, there may be a very small chance that the egg spontaneously reforms. But you cannot expect it to reform. You must expect it to smash. Your mandatory belief is that the egg’s probability of spontaneous reformation is ~0. Probabilities are not certainties, but the laws of probability are theorems.

  If you doubt this, try dropping an egg on the floor a few decillion times, ignoring the thermodynamic suggestion and expecting it to spontaneously reassemble, and see what happens. Probabilities may be subjective states of belief, but the laws governing them are stronger by far than steel. I once knew a fellow who was convinced that his system of wheels and gears would produce reactionless thrust, and he had an Excel spreadsheet that would prove this—which of course he couldn’t show us because he was still developing the system. In classical mechanics, violating Conservation of Momentum is provably impossible. So any Excel spreadsheet calculated according to the rules of classical mechanics must necessarily show that no reactionless thrust exists—unless your machine is complicated enough that you have made a mistake in the calculations.

  And similarly, when half-trained or tenth-trained rationalists abandon their art and try to believe without evidence just this once, they often build vast edifices of justification, confusing themselves just enough to conceal the magical steps.

  It can be quite a pain to nail down where the magic occurs—their structure of argument tends to morph and squirm away as you interrogate them. But there’s always some step where a tiny probability turns into a large one—where they try to believe without evidence—where they step into the unknown, thinking, “No one can prove me wrong.”

  Their foot naturally lands on thin air, for there is far more thin air than ground in the realms of Possibility. Ah, but there is an (exponentially tiny) amount of ground in Possibility, and you do have an (exponentially tiny) probability of hitting it by luck, so maybe this time, your foot will land in the right place! It is merely a probability, so it must be merely a suggestion.

  The exact state of a glass of boiling-hot water may be unknown to you—indeed, your ignorance of its exact state is what makes the molecules’ kinetic energy “heat,” rather than work waiting to be extracted like the momentum of a spinning flywheel. So the water might cool down your hand instead of heating it up, with probability ~0.

  Decide to ignore the laws of thermodynamics and stick your hand in anyway, and you’ll get burned.

  “But you don’t know that!”

  I don’t know it with certainty, but it is mandatory that I expect it to happen. Probabilities are not logical truths, but the laws of probability are.

  “But what if I guess the state of the boiling water, and I happen to guess correctly?”

  Your chance of guessing correctly by luck, is even less than the chance of the boiling water cooling your hand by luck.

  “But you can’t prove I won’t guess correctly.”

  I can (indeed, must) assign extremely low probability to it.

  “That’s not the same as certainty, though.”

  Hey, maybe if you add enough wheels and gears to your argument, it’ll turn warm water into electricity and ice cubes! Or, rather, you will no longer see why this couldn’t be the case.

  “Right! I can’t see why couldn’t be the case! So maybe it is!”

  Another gear? That just makes your machine even less efficient. It wasn’t a perpetual motion machine before, and each extra gear you add makes it even less efficient than that.

  Each extra detail in your argument necessarily decreases the joint probability. The probability that you’ve violated the Second Law of Thermodynamics without knowing exactly how, by guessing the exact state of boiling water without evidence, so that you can stick your finger in without getting burned, is, necessarily, even less than the probability of sticking in your finger into boiling water without getting burned.

  I say all this, because people really do construct these huge edifices of argument in the course of believing without evidence. One must learn to see this as analogous to all the wheels and gears that fellow added onto his reactionless drive, until he finally collected enough complications to make a mistake in his Excel spreadsheet.

  *

  188

  Searching for Bayes-Structure

  Gnomish helms should not function. Their very construction seems to defy the nature of thaumaturgical law. In fact, they are impossible. Like most products of gnomish minds, they include a large number of bells and whistles, and very little substance. Those that work usually have a minor helm contained within, always hidden away, disguised to appear innocuous and inessential.

  —Spelljammer campaign set

  We have seen that knowledge implies mutual information between a mind and its environment, and we have seen that this mutual information is negentropy in a very physical sense: If you know where molecules are and how fast they’re moving, you can turn heat into work via a Maxwell’s Demon / Szilárd engine.

  We have seen that forming true beliefs without evidence is the same sort of improbability as a hot glass of water spontaneously reorganizing into ice cubes and electricity. Rationality
takes “work” in a thermodynamic sense, not just the sense of mental effort; minds have to radiate heat if they are not perfectly efficient. This cognitive work is governed by probability theory, of which thermodynamics is a special case. (Statistical mechanics is a special case of statistics.)

  If you saw a machine continually spinning a wheel, apparently without being plugged into a wall outlet or any other source of power, then you would look for a hidden battery, or a nearby broadcast power source—something to explain the work being done, without violating the laws of physics.

  So if a mind is arriving at true beliefs, and we assume that the Second Law of Thermodynamics has not been violated, that mind must be doing something at least vaguely Bayesian—at least one process with a sort-of Bayesian structure somewhere—or it couldn’t possibly work.

  In the beginning, at time T = 0, a mind has no mutual information with a subsystem S in its environment. At time T = 1, the mind has 10 bits of mutual information with S. Somewhere in between, the mind must have encountered evidence—under the Bayesian definition of evidence, because all Bayesian evidence is mutual information and all mutual information is Bayesian evidence, they are just different ways of looking at it—and processed at least some of that evidence, however inefficiently, in the right direction according to Bayes on at least some occasions. The mind must have moved in harmony with the Bayes at least a little, somewhere along the line—either that or violated the Second Law of Thermodynamics by creating mutual information from nothingness.

  In fact, any part of a cognitive process that contributes usefully to truth-finding must have at least a little Bayesian structure—must harmonize with Bayes, at some point or another—must partially conform with the Bayesian flow, however noisily—despite however many disguising bells and whistles—even if this Bayesian structure is only apparent in the context of surrounding processes. Or it couldn’t even help.

  How philosophers pondered the nature of words! All the ink spent on the true definitions of words, and the true meaning of definitions, and the true meaning of meaning! What collections of gears and wheels they built, in their explanations! And all along, it was a disguised form of Bayesian inference!

  I was actually a bit disappointed that no one in the audience jumped up and said: “Yes! Yes, that’s it! Of course! It was really Bayes all along!”

  But perhaps it is not quite as exciting to see something that doesn’t look Bayesian on the surface, revealed as Bayes wearing a clever disguise, if: (a) you don’t unravel the mystery yourself, but read about someone else doing it (Newton had more fun than most students taking calculus), and (b) you don’t realize that searching for the hidden Bayes-structure is this huge, difficult, omnipresent quest, like searching for the Holy Grail.

  It’s a different quest for each facet of cognition, but the Grail always turns out to be the same. It has to be the right Grail, though—and the entire Grail, without any parts missing—and so each time you have to go on the quest looking for a full answer whatever form it may take, rather than trying to artificially construct vaguely hand-waving Grailish arguments. Then you always find the same Holy Grail at the end.

  It was previously pointed out to me that I might be losing some of my readers with the long essays, because I hadn’t “made it clear where I was going” . . .

  . . . but it’s not so easy to just tell people where you’re going, when you’re going somewhere like that.

  It’s not very helpful to merely know that a form of cognition is Bayesian, if you don’t know how it is Bayesian. If you can’t see the detailed flow of probability, you have nothing but a password—or, a bit more charitably, a hint at the form an answer would take; but certainly not an answer. That’s why there’s a Grand Quest for the Hidden Bayes-Structure, rather than being done when you say “Bayes!” Bayes-structure can be buried under all kinds of disguises, hidden behind thickets of wheels and gears, obscured by bells and whistles.

  The way you begin to grasp the Quest for the Holy Bayes is that you learn about cognitive phenomenon XYZ, which seems really useful—and there’s this bunch of philosophers who’ve been arguing about its true nature for centuries, and they are still arguing—and there’s a bunch of AI scientists trying to make a computer do it, but they can’t agree on the philosophy either—

  And—Huh, that’s odd!—this cognitive phenomenon didn’t look anything like Bayesian on the surface, but there’s this non-obvious underlying structure that has a Bayesian interpretation—but wait, there’s still some useful work getting done that can’t be explained in Bayesian terms—no wait, that’s Bayesian too—OH MY GOD this completely different cognitive process, that also didn’t look Bayesian on the surface, ALSO HAS BAYESIAN STRUCTURE—hold on, are these non-Bayesian parts even doing anything?

  Yes: Wow, those are Bayesian too!

  No: Dear heavens, what a stupid design. I could eat a bucket of amino acids and puke a better brain architecture than that.

  Once this happens to you a few times, you kinda pick up the rhythm. That’s what I’m talking about here, the rhythm.

  Trying to talk about the rhythm is like trying to dance about architecture.

  This left me in a bit of a pickle when it came to trying to explain in advance where I was going. I know from experience that if I say, “Bayes is the secret of the universe,” some people may say “Yes! Bayes is the secret of the universe!”; and others will snort and say, “How narrow-minded you are; look at all these other ad-hoc but amazingly useful methods, like regularized linear regression, that I have in my toolbox.”

  I hoped that with a specific example in hand of “something that doesn’t look all that Bayesian on the surface, but turns out to be Bayesian after all”—and an explanation of the difference between passwords and knowledge—and an explanation of the difference between tools and laws—maybe then I could convey such of the rhythm as can be understood without personally going on the quest.

  Of course this is not the full Secret of the Bayesian Conspiracy, but it’s all that I can convey at this point. Besides, the complete secret is known only to the Bayes Council, and if I told you, I’d have to hire you.

  To see through the surface adhockery of a cognitive process, to the Bayesian structure underneath—to perceive the probability flows, and know how, not just know that, this cognition too is Bayesian—as it always is—as it always must be—to be able to sense the Force underlying all cognition—this, is the Bayes-Sight.

  “. . . And the Queen of Kashfa sees with the Eye of the Serpent.”

  “I don’t know that she sees with it,” I said. “She’s still recovering from the operation. But that’s an interesting thought. If she could see with it, what might she behold?”

  “The clear, cold lines of eternity, I daresay. Beneath all Shadow.”

  —Roger Zelazny, Prince of Chaos1

  *

  1. Roger Zelazny, Prince of Chaos (Thorndike Press, 2001).

  Part P

  Reductionism 101

  189

  Dissolving the Question

  “If a tree falls in the forest, but no one hears it, does it make a sound?”

  I didn’t answer that question. I didn’t pick a position “Yes!” or “No!” and defend it. Instead I went off and deconstructed the human algorithm for processing words, even going so far as to sketch an illustration of a neural network. At the end, I hope, there was no question left—not even the feeling of a question.

  Many philosophers—particularly amateur philosophers, and ancient philosophers—share a dangerous instinct: If you give them a question, they try to answer it.

  Like, say, “Do we have free will?”

  The dangerous instinct of philosophy is to marshal the arguments in favor, and marshal the arguments against, and weigh them up, and publish them in a prestigious journal of philosophy, and so finally conclude: “Yes, we must have free will,” or “No, we cannot possibly have free will.”

  Some philosophers are wise enough to recall the
warning that most philosophical disputes are really disputes over the meaning of a word, or confusions generated by using different meanings for the same word in different places. So they try to define very precisely what they mean by “free will,” and then ask again, “Do we have free will? Yes or no?”

  A philosopher wiser yet may suspect that the confusion about “free will” shows the notion itself is flawed. So they pursue the Traditional Rationalist course: They argue that “free will” is inherently self-contradictory, or meaningless because it has no testable consequences. And then they publish these devastating observations in a prestigious philosophy journal.

  But proving that you are confused may not make you feel any less confused. Proving that a question is meaningless may not help you any more than answering it.

  The philosopher’s instinct is to find the most defensible position, publish it, and move on. But the “naive” view, the instinctive view, is a fact about human psychology. You can prove that free will is impossible until the Sun goes cold, but this leaves an unexplained fact of cognitive science: If free will doesn’t exist, what goes on inside the head of a human being who thinks it does? This is not a rhetorical question!

  It is a fact about human psychology that people think they have free will. Finding a more defensible philosophical position doesn’t change, or explain, that psychological fact. Philosophy may lead you to reject the concept, but rejecting a concept is not the same as understanding the cognitive algorithms behind it.