Rationality- From AI to Zombies

by Eliezer Yudkowsky

  The way it works in Western superhero comics is that the good guy gets bitten by a radioactive spider; and then he needs something to do with his powers, to keep him busy, so he decides to fight crime. And then Western superheroes are always whining about how much time their superhero duties take up, and how they’d rather be ordinary mortals so they could go fishing or something.

  Similarly, in Western real life, unhappy people are told that they need a “purpose in life,” so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes. You should be careful not to pick something too expensive, though.

  In Western comics, the magic comes first, then the purpose: Acquire amazing powers, decide to protect the innocent. In Japanese fiction, often, it works the other way around.

  Of course I’m not saying all this to generalize from fictional evidence. But I want to convey a concept whose deceptively close Western analogue is not what I mean.

  I have touched before on the idea that a rationalist must have something they value more than “rationality”: the Art must have a purpose other than itself, or it collapses into infinite recursion. But do not mistake me, and think I am advocating that rationalists should pick out a nice altruistic cause, by way of having something to do, because rationality isn’t all that important by itself. No. I am asking: Where do rationalists come from? How do we acquire our powers?

  It is written in The Twelve Virtues of Rationality:

  How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.” By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you think: “It may look like the sky is blue, but rationality is to believe the words of the Great Teacher,” you lose a chance to discover your mistake.

  Historically speaking, the way humanity finally left the trap of authority and began paying attention to, y’know, the actual sky, was that beliefs based on experiment turned out to be much more useful than beliefs based on authority. Curiosity has been around since the dawn of humanity, but the problem is that spinning campfire tales works just as well for satisfying curiosity.

  Historically speaking, science won because it displayed greater raw strength in the form of technology, not because science sounded more reasonable. To this very day, magic and scripture still sound more reasonable to untrained ears than science. That is why there is continuous social tension between the belief systems. If science not only worked better than magic, but also sounded more intuitively reasonable, it would have won entirely by now.

  Now there are those who say: “How dare you suggest that anything should be valued more than Truth? Must not a rationalist love Truth more than mere usefulness?”

  Forget for a moment what would have happened historically to someone like that—that people in pretty much that frame of mind defended the Bible because they loved Truth more than mere accuracy. Propositional morality is a glorious thing, but it has too many degrees of freedom.

  No, the real point is that a rationalist’s love affair with the Truth is, well, just more complicated as an emotional relationship.

  One doesn’t become an adept rationalist without caring about the truth, both as a purely moral desideratum and as something that’s fun to have. I doubt there are many master composers who hate music.

  But part of what I like about rationality is the discipline imposed by requiring beliefs to yield predictions, which ends up taking us much closer to the truth than if we sat in the living room obsessing about Truth all day. I like the complexity of simultaneously having to love True-seeming ideas, and also being ready to drop them out the window at a moment’s notice. I even like the glorious aesthetic purity of declaring that I value mere usefulness above aesthetics. That is almost a contradiction, but not quite; and that has an aesthetic quality as well, a delicious humor.

  And of course, no matter how much you profess your love of mere usefulness, you should never actually end up deliberately believing a useful false statement.

  So don’t oversimplify the relationship between loving truth and loving usefulness. It’s not one or the other. It’s complicated, which is not necessarily a defect in the moral aesthetics of single events.

  But morality and aesthetics alone, believing that one ought to be “rational” or that certain ways of thinking are “beautiful,” will not lead you to the center of the Way. It wouldn’t have gotten humanity out of the authority-hole.

  In Feeling Moral, I discussed this dilemma: Which of these options would you prefer?

  Save 400 lives, with certainty.

  Save 500 lives, 90% probability; save no lives, 10% probability.

  You may be tempted to grandstand, saying, “How dare you gamble with people’s lives?” Even if you, yourself, are one of the 500—but you don’t know which one—you may still be tempted to rely on the comforting feeling of certainty, because our own lives are often worth less to us than a good intuition.

  But if your precious daughter is one of the 500, and you don’t know which one, then, perhaps, you may feel more impelled to shut up and multiply—to notice that you have an 80% chance of saving her in the first case, and a 90% chance of saving her in the second.

  And yes, everyone in that crowd is someone’s son or daughter. Which, in turn, suggests that we should pick the second option as altruists, as well as concerned parents.

  My point is not to suggest that one person’s life is more valuable than 499 people. What I am trying to say is that more than your own life has to be at stake, before a person becomes desperate enough to resort to math.

  What if you believe that it is “rational” to choose the certainty of option 1? Lots of people think that “rationality” is about choosing only methods that are certain to work, and rejecting all uncertainty. But, hopefully, you care more about your daughter’s life than about “rationality.”

  Will pride in your own virtue as a rationalist save you? Not if you believe that it is virtuous to choose certainty. You will only be able to learn something about rationality if your daughter’s life matters more to you than your pride as a rationalist.

  You may even learn something about rationality from the experience, if you are already far enough grown in your Art to say, “I must have had the wrong conception of rationality,” and not, “Look at how rationality gave me the wrong answer!”

  (The essential difficulty in becoming a master rationalist is that you need quite a bit of rationality to bootstrap the learning process.)

  Is your belief that you ought to be rational more important than your life? Because, as I’ve previously observed, risking your life isn’t comparatively all that scary. Being the lone voice of dissent in the crowd and having everyone look at you funny is much scarier than a mere threat to your life, according to the revealed preferences of teenagers who drink at parties and then drive home. It will take something terribly important to make you willing to leave the pack. A threat to your life won’t be enough.

  Is your will to rationality stronger than your pride? Can it be, if your will to rationality stems from your pride in your self-image as a rationalist? It’s helpful—very helpful—to have a self-image which says that you are the sort of person who confronts harsh truth. It’s helpful to have too much self-respect to knowingly lie to yourself or refuse to face evidence. But there may come a time when you have to admit that you’ve been doing rationality all wrong. Then your pride, your self-image as a rationalist, may make that too hard to face.

  If you’ve prided yourself on believing what the Great Teacher says—even when it seems harsh, even when you’d rather not—that may make it all the more bitter a pill to swallow, to admit that the Great Teacher is a fraud, and all your noble self-sacrifice
was for naught.

  Where do you get the will to keep moving forward?

  When I look back at my own personal journey toward rationality—not just humanity’s historical journey—well, I grew up believing very strongly that I ought to be rational. This made me an above-average Traditional Rationalist a la Feynman and Heinlein, and nothing more. It did not drive me to go beyond the teachings I had received. I only began to grow further as a rationalist once I had something terribly important that I needed to do. Something more important than my pride as a rationalist, never mind my life.

  Only when you become more wedded to success than to any of your beloved techniques of rationality do you begin to appreciate these words of Miyamoto Musashi:1

  You can win with a long weapon, and yet you can also win with a short weapon. In short, the Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size.

  —Miyamoto Musashi, The Book of Five Rings

  Don’t mistake this for a specific teaching of rationality. It describes how you learn the Way, beginning with a desperate need to succeed. No one masters the Way until more than their life is at stake. More than their comfort, more even than their pride.

  You can’t just pick out a Cause like that because you feel you need a hobby. Go looking for a “good cause,” and your mind will just fill in a standard cliché. Learn how to multiply, and perhaps you will recognize a drastically important cause when you see one.

  But if you have a cause like that, it is right and proper to wield your rationality in its service.

  To strictly subordinate the aesthetics of rationality to a higher cause is part of the aesthetic of rationality. You should pay attention to that aesthetic: You will never master rationality well enough to win with any weapon if you do not appreciate the beauty for its own sake.

  *

  1. Musashi, Book of Five Rings.

  290

  When (Not) to Use Probabilities

  It may come as a surprise to some readers that I do not always advocate using probabilities.

  Or rather, I don’t always advocate that human beings, trying to solve their problems, should try to make up verbal probabilities, and then apply the laws of probability theory or decision theory to whatever number they just made up, and then use the result as their final belief or decision.

  The laws of probability are laws, not suggestions, but often the true Law is too difficult for us humans to compute. If P ≠ NP and the universe has no source of exponential computing power, then there are evidential updates too difficult for even a superintelligence to compute—even though the probabilities would be quite well-defined, if we could afford to calculate them.

  So sometimes you don’t apply probability theory. Especially if you’re human, and your brain has evolved with all sorts of useful algorithms for uncertain reasoning, that don’t involve verbal probability assignments.

  Not sure where a flying ball will land? I don’t advise trying to formulate a probability distribution over its landing spots, performing deliberate Bayesian updates on your glances at the ball, and calculating the expected utility of all possible strings of motor instructions to your muscles. Trying to catch a flying ball, you’re probably better off with your brain’s built-in mechanisms than using deliberative verbal reasoning to invent or manipulate probabilities.

  But this doesn’t mean you’re going beyond probability theory or above probability theory.

  The Dutch book arguments still apply. If I offer you a choice of gambles ($10,000 if the ball lands in this square, versus $10,000 if I roll a die and it comes up 6), and you answer in a way that does not allow consistent probabilities to be assigned, then you will accept combinations of gambles that are certain losses, or reject gambles that are certain gains . . .

  Which still doesn’t mean that you should try to use deliberative verbal reasoning. I would expect that for professional baseball players, at least, it’s more important to catch the ball than to assign consistent probabilities. Indeed, if you tried to make up probabilities, the verbal probabilities might not even be very good ones, compared to some gut-level feeling—some wordless representation of uncertainty in the back of your mind.

  There is nothing privileged about uncertainty that is expressed in words, unless the verbal parts of your brain do, in fact, happen to work better on the problem.

  And while accurate maps of the same territory will necessarily be consistent among themselves, not all consistent maps are accurate. It is more important to be accurate than to be consistent, and more important to catch the ball than to be consistent.

  In fact, I generally advise against making up probabilities, unless it seems like you have some decent basis for them. This only fools you into believing that you are more Bayesian than you actually are.

  To be specific, I would advise, in most cases, against using non-numerical procedures to create what appear to be numerical probabilities. Numbers should come from numbers.

  Now there are benefits from trying to translate your gut feelings of uncertainty into verbal probabilities. It may help you spot problems like the conjunction fallacy. It may help you spot internal inconsistencies—though it may not show you any way to remedy them.

  But you shouldn’t go around thinking that if you translate your gut feeling into “one in a thousand,” then, on occasions when you emit these verbal words, the corresponding event will happen around one in a thousand times. Your brain is not so well-calibrated. If instead you do something nonverbal with your gut feeling of uncertainty, you may be better off, because at least you’ll be using the gut feeling the way it was meant to be used.

  This specific topic came up recently in the context of the Large Hadron Collider, and an argument given at the Global Catastrophic Risks conference:

  That we couldn’t be sure that there was no error in the papers which showed from multiple angles that the LHC couldn’t possibly destroy the world. And moreover, the theory used in the papers might be wrong. And in either case, there was still a chance the LHC could destroy the world. And therefore, it ought not to be turned on.

  Now if the argument had been given in just this way, I would not have objected to its epistemology.

  But the speaker actually purported to assign a probability of at least 1 in 1,000 that the theory, model, or calculations in the LHC paper were wrong; and a probability of at least 1 in 1,000 that, if the theory or model or calculations were wrong, the LHC would destroy the world.

  After all, it’s surely not so improbable that future generations will reject the theory used in the LHC paper, or reject the model, or maybe just find an error. And if the LHC paper is wrong, then who knows what might happen as a result?

  So that is an argument—but to assign numbers to it?

  I object to the air of authority given to these numbers pulled out of thin air. I generally feel that if you can’t use probabilistic tools to shape your feelings of uncertainty, you ought not to dignify them by calling them probabilities.

  The alternative I would propose, in this particular case, is to debate the general rule of banning physics experiments because you cannot be absolutely certain of the arguments that say they are safe.

  I hold that if you phrase it this way, then your mind, by considering frequencies of events, is likely to bring in more consequences of the decision, and remember more relevant historical cases.

  If you debate just the one case of the LHC, and assign specific probabilities, it (1) gives very shaky reasoning an undue air of authority, (2) obscures the general consequences of applying similar rules, and even (3) creates the illusion that we might come to a different decision if someone else published a new physics paper that decreased the probabilities.

  The authors at the Global Catastrophic Risk conference seemed to be suggesting that we could just do a bit more analysis of the LHC and then switch it on. This struck me as the most disingenuous part of the argument. Once you admit the argument “Maybe the analysis could be wrong, and wh
o knows what happens then,” there is no possible physics paper that can ever get rid of it.

  No matter what other physics papers had been published previously, the authors would have used the same argument and made up the same numerical probabilities at the Global Catastrophic Risk conference. I cannot be sure of this statement, of course, but it has a probability of 75%.

  In general a rationalist tries to make their minds function at the best achievable power output; sometimes this involves talking about verbal probabilities, and sometimes it does not, but always the laws of probability theory govern.

  If all you have is a gut feeling of uncertainty, then you should probably stick with those algorithms that make use of gut feelings of uncertainty, because your built-in algorithms may do better than your clumsy attempts to put things into words.

  Now it may be that by reasoning thusly, I may find myself inconsistent. For example, I would be substantially more alarmed about a lottery device with a well-defined chance of 1 in 1,000,000 of destroying the world, than I am about the Large Hadron Collider being switched on.

  On the other hand, if you asked me whether I could make one million statements of authority equal to “The Large Hadron Collider will not destroy the world,” and be wrong, on average, around once, then I would have to say no.

  What should I do about this inconsistency? I’m not sure, but I’m certainly not going to wave a magic wand to make it go away. That’s like finding an inconsistency in a pair of maps you own, and quickly scribbling some alterations to make sure they’re consistent.

  I would also, by the way, be substantially more worried about a lottery device with a 1 in 1,000,000,000 chance of destroying the world, than a device which destroyed the world if the Judeo-Christian God existed. But I would not suppose that I could make one billion statements, one after the other, fully independent and equally fraught as “There is no God,” and be wrong on average around once.