Rationality- From AI to Zombies

by Eliezer Yudkowsky

  “But—but Occam’s Razor really is better than faith! That’s not like preferring a different flavor of ice cream! Anyone can see, looking at history, that Occamian reasoning has been far more productive than faith—”

  Which is all true. But beside the point. The point is that you, saying this, are rattling off a standard justification that’s already in your mind. The challenge of a crisis of faith is to handle the case where, possibly, our standard conclusions are wrong and our standard justifications are wrong. So if the standard justification for X is “Occam’s Razor!,” and you want to hold a crisis of faith around X, you should be questioning if Occam’s Razor really endorses X, if your understanding of Occam’s Razor is correct, and—if you want to have sufficiently deep doubts—whether simplicity is the sort of criterion that has worked well historically in this case, or could reasonably be expected to work, et cetera. If you would advise a religionist to question their belief that “faith” is a good justification for X, then you should advise yourself to put forth an equally strong effort to question your belief that “Occam’s Razor” is a good justification for X.

  (Think of all the people out there who don’t understand the Minimum Description Length or Solomonoff induction formulations of Occam’s Razor, who think that Occam’s Razor outlaws many-worlds or the Simulation Hypothesis. They would need to question their formulations of Occam’s Razor and their notions of why simplicity is a good thing. Whatever X in contention you just justified by saying “Occam’s Razor!,” I bet it’s not the same level of Occamian slam dunk as gravity.)

  If “Occam’s Razor!” is your usual reply, your standard reply, the reply that all your friends give—then you’d better block your brain from instantly completing that pattern, if you’re trying to instigate a true crisis of faith.

  Better to think of such rules as, “Imagine what a skeptic would say—and then imagine what they would say to your response—and then imagine what else they might say, that would be harder to answer.”

  Or, “Try to think the thought that hurts the most.”

  And above all, the rule:

  “Put forth the same level of desperate effort that it would take for a theist to reject their religion.”

  Because, if you aren’t trying that hard, then—for all you know—your head could be stuffed full of nonsense as ridiculous as religion.

  Without a convulsive, wrenching effort to be rational, the kind of effort it would take to throw off a religion—then how dare you believe anything, when Robert Aumann believes in God?

  Someone (I forget who) once observed that people had only until a certain age to reject their religious faith. Afterward they would have answers to all the objections, and it would be too late. That is the kind of existence you must surpass. This is a test of your strength as a rationalist, and it is very severe; but if you cannot pass it, you will be weaker than a ten-year-old.

  But again, by the time you know a belief is an error, it is already defeated. So we’re not talking about a desperate, convulsive effort to undo the effects of a religious upbringing, after you’ve come to the conclusion that your religion is wrong. We’re talking about a desperate effort to figure out if you should be throwing off the chains, or keeping them. Self-honesty is at its most fragile when we don’t know which path we’re supposed to take—that’s when rationalizations are not obviously sins.

  Not every doubt calls for staging an all-out Crisis of Faith. But you should consider it when:

  A belief has long remained in your mind;

  It is surrounded by a cloud of known arguments and refutations;

  You have sunk costs in it (time, money, public declarations);

  The belief has emotional consequences (note this does not make it wrong);

  It has gotten mixed up in your personality generally.

  None of these warning signs are immediate disproofs. These attributes place a belief at risk for all sorts of dangers, and make it very hard to reject when it is wrong. But they also hold for Richard Dawkins’s belief in evolutionary biology as well as the Pope’s Catholicism. This does not say that we are only talking about different flavors of ice cream. Only the unenlightened think that all deeply-held beliefs are on the same level regardless of the evidence supporting them, just because they are deeply held. The point is not to have shallow beliefs, but to have a map which reflects the territory.

  I emphasize this, of course, so that you can admit to yourself, “My belief has these warning signs,” without having to say to yourself, “My belief is false.”

  But what these warning signs do mark, is a belief that will take more than an ordinary effort to doubt effectively. So that if it were in fact false, you would in fact reject it. And where you cannot doubt effectively, you are blind, because your brain will hold the belief unconditionally. When a retina sends the same signal regardless of the photons entering it, we call that eye blind.

  When should you stage a Crisis of Faith?

  Again, think of the advice you would give to a theist: If you find yourself feeling a little unstable inwardly, but trying to rationalize reasons the belief is still solid, then you should probably stage a Crisis of Faith. If the belief is as solidly supported as gravity, you needn’t bother—but think of all the theists who would desperately want to conclude that God is as solid as gravity. So try to imagine what the skeptics out there would say to your “solid as gravity” argument. Certainly, one reason you might fail at a crisis of faith is that you never really sit down and question in the first place—that you never say, “Here is something I need to put effort into doubting properly.”

  If your thoughts get that complicated, you should go ahead and stage a Crisis of Faith. Don’t try to do it haphazardly, don’t try it in an ad-hoc spare moment. Don’t rush to get it done with quickly, so that you can say “I have doubted as I was obliged to do.” That wouldn’t work for a theist and it won’t work for you either. Rest up the previous day, so you’re in good mental condition. Allocate some uninterrupted hours. Find somewhere quiet to sit down. Clear your mind of all standard arguments, try to see from scratch. And make a desperate effort to put forth a true doubt that would destroy a false, and only a false, deeply held belief.

  Elements of the Crisis of Faith technique have been scattered over many essays:

  Avoiding Your Belief’s Real Weak Points—One of the first temptations in a crisis of faith is to doubt the strongest points of your belief, so that you can rehearse your good answers. You need to seek out the most painful spots, not the arguments that are most reassuring to consider.

  The Meditation on Curiosity—Roger Zelazny once distinguished between “wanting to be an author” versus “wanting to write,” and there is likewise a distinction between wanting to have investigated and wanting to investigate. It is not enough to say “It is my duty to criticize my own beliefs”; you must be curious, and only uncertainty can create curiosity. Keeping in mind Conservation of Expected Evidence may help you Update Yourself Incrementally: for every single point that you consider, and each element of new argument and new evidence, you should not expect your beliefs to shift more (on average) in one direction than another—thus you can be truly curious each time about how it will go.

  Original Seeing—Use Pirsig’s technique to prevent standard cached thoughts from rushing in and completing the pattern.

  The Litany of Gendlin and the Litany of Tarski—People can stand what is true, for they are already enduring it. If a belief is true you will be better off believing it, and if it is false you will be better off rejecting it. You would advise a religious person to try to visualize fully and deeply the world in which there is no God, and to, without excuses, come to the full understanding that if there is no God then they will be better off believing there is no God. If one cannot come to accept this on a deep emotional level, one will not be able to have a crisis of faith. So you should put in a sincere effort to visualize the alternative to your belief, the way that the best and highest skeptic wou
ld want you to visualize it. Think of the effort a religionist would have to put forth to imagine, without corrupting it for their own comfort, an atheist’s view of the universe.

  Make an Extraordinary Effort—See the concept of isshokenmei, the desperate convulsive effort to be rational, the effort that it would take to surpass the level of Robert Aumann and all the great scientists throughout history who never let go of their religions.

  The Genetic Heuristic—You should be extremely suspicious if you have many ideas suggested by a source that you now know to be untrustworthy, but by golly, it seems that all the ideas still ended up being right. (E.g., the one concedes that the Bible was written by human hands, but still clings to the idea that it contains indispensable ethical wisdom.)

  The Importance of Saying “Oops”—It really is less painful to swallow the entire bitter pill in one terrible gulp.

  Singlethink—The opposite of doublethink. See the thoughts you flinch away from, that appear in the corner of your mind for just a moment before you refuse to think them. If you become aware of what you are not thinking, you can think it.

  Affective Death Spirals and Resist the Happy Death Spiral—Affective death spirals are prime generators of false beliefs that it will take a Crisis of Faith to shake loose. But since affective death spirals can also get started around real things that are genuinely nice, you don’t have to admit that your belief is a lie, to try and resist the halo effect at every point—refuse false praise even of genuinely nice things. Policy debates should not appear one-sided.

  Hold Off On Proposing Solutions—Don’t propose any solutions until the problem has been discussed as thoroughly as possible. Make your mind wait on knowing what its answer will be; and try for five minutes before giving up, both generally, and especially when pursuing the devil’s point of view.

  And these standard techniques are particularly relevant:

  The sequence on The Bottom Line and Rationalization, which explains why it is always wrong to selectively argue one side of a debate.

  Positive Bias and motivated skepticism and motivated stopping, lest you selectively look for support, selectively look for counter-counterarguments, and selectively stop the argument before it gets dangerous. Missing alternatives are a special case of stopping. A special case of motivated skepticism is fake humility, where you bashfully confess that no one can know something you would rather not know. Don’t selectively demand too much authority of counterarguments.

  Beware of Semantic Stopsigns, Applause Lights, and your choice to Explain/Worship/Ignore.

  Feel the weight of Burdensome Details; each detail a separate burden, a point of crisis.

  But really there’s rather a lot of relevant material, here and on Overcoming Bias. The Crisis of Faith is only the critical point and sudden clash of the longer isshoukenmei—the lifelong uncompromising effort to be so incredibly rational that you rise above the level of stupid damn mistakes. It’s when you get a chance to use the skills that you’ve been practicing for so long, all-out against yourself.

  I wish you the best of luck against your opponent. Have a wonderful crisis!

  *

  130

  The Ritual

  The room in which Jeffreyssai received his non-beisutsukai visitors was quietly formal, impeccably appointed in only the most conservative tastes. Sunlight and outside air streamed through a grillwork of polished silver, a few sharp edges making it clear that this wall was not to be opened. The floor and walls were glass, thick enough to distort, to a depth sufficient that it didn’t matter what might be underneath. Upon the surfaces of the glass were subtly scratched patterns of no particular meaning, scribed as if by the hand of an artistically inclined child (and this was in fact the case).

  Elsewhere in Jeffreyssai’s home there were rooms of other style; but this, he had found, was what most outsiders expected of a Bayesian Master, and he chose not to enlighten them otherwise. That quiet amusement was one of life’s little joys, after all.

  The guest sat across from him, knees on the pillow and heels behind. She was here solely upon the business of her Conspiracy, and her attire showed it: a form-fitting jumpsuit of pink leather with even her hands gloved—all the way to the hood covering her head and hair, though her face lay plain and unconcealed beneath.

  And so Jeffreyssai had chosen to receive her in this room.

  Jeffreyssai let out a long breath, exhaling. “Are you sure?”

  “Oh,” she said, “and do I have to be absolutely certain before my advice can shift your opinions? Does it not suffice that I am a domain expert, and you are not?”

  Jeffreyssai’s mouth twisted up at the corner in a half-smile. “How do you know so much about the rules, anyway? You’ve never had so much as a Planck length of formal training.”

  “Do you even need to ask?” she said dryly. “If there’s one thing that you beisutsukai do love to go on about, it’s the reasons why you do things.”

  Jeffreyssai inwardly winced at the thought of trying to pick up rationality by watching other people talk about it—

  “And don’t inwardly wince at me like that,” she said. “I’m not trying to be a rationalist myself, just trying to win an argument with a rationalist. There’s a difference, as I’m sure you tell your students.”

  Can she really read me that well? Jeffreyssai looked out through the silver grillwork, at the sunlight reflected from the faceted mountainside. Always, always the golden sunlight fell each day, in this place far above the clouds. An unchanging thing, that light. The distant Sun, which that light represented, was in five billion years burned out; but now, in this moment, the Sun still shone. And that could never alter. Why wish for things to stay the same way forever, when that wish was already granted as absolutely as any wish could be? The paradox of permanence and impermanence: only in the latter perspective was there any such thing as progress, or loss.

  “You have always given me good counsel,” Jeffreyssai said. “Unchanging, that has been. Through all the time we’ve known each other.”

  She inclined her head, acknowledging. This was true, and there was no need to spell out the implications.

  “So,” Jeffreyssai said. “Not for the sake of arguing. Only because I want to know the answer. Are you sure?” He didn’t even see how she could guess.

  “Pretty sure,” she said, “we’ve been collecting statistics for a long time, and in nine hundred and eighty-five out of a thousand cases like yours—”

  Then she laughed at the look on his face. “No, I’m joking. Of course I’m not sure. This thing only you can decide. But I am sure that you should go off and do whatever it is you people do—I’m quite sure you have a ritual for it, even if you won’t discuss it with outsiders—when you very seriously consider abandoning a long-held premise of your existence.”

  It was hard to argue with that, Jeffreyssai reflected, the more so when a domain expert had told you that you were, in fact, probably wrong.

  “I concede,” Jeffreyssai said. Coming from his lips, the phrase was spoken with a commanding finality. There is no need to argue with me any further: you have won.

  “Oh, stop it,” she said. She rose from her pillow in a single fluid shift without the slightest wasted motion. She didn’t flaunt her age, but she didn’t conceal it either. She took his outstretched hand, and raised it to her lips for a formal kiss. “Farewell, sensei.”

  “Farewell?” repeated Jeffreyssai. That signified a higher order of departure than goodbye. “I do intend to visit you again, milady; and you are always welcome here.”

  She walked toward the door without answering. At the doorway she paused, without turning around. “It won’t be the same,” she said. And then, without the movements seeming the least rushed, she walked away so swiftly it was almost like vanishing.

  Jeffreyssai sighed. But at least, from here until the challenge proper, all his actions were prescribed, known quantities.

  Leaving that formal reception area, he passed to his arena, and ca
used to be sent out messengers to his students, telling them that the next day’s classes must be improvised in his absence, and that there would be a test later.

  And then he did nothing in particular. He read another hundred pages of the textbook he had borrowed; it wasn’t very good, but then the book he had loaned out in exchange wasn’t very good either. He wandered from room to room of his house, idly checking various storages to see if anything had been stolen (a deck of cards was missing, but that was all). From time to time his thoughts turned to tomorrow’s challenge, and he let them drift. Not directing his thoughts at all, only blocking out every thought that had ever previously occurred to him; and disallowing any kind of conclusion, or even any thought as to where his thoughts might be trending.

  The sun set, and he watched it for a while, mind carefully put in idle. It was a fantastic balancing act to set your mind in idle without having to obsess about it, or exert energy to keep it that way; and years ago he would have sweated over it, but practice had long since made perfect.

  The next morning he awoke with the chaos of the night’s dreaming fresh in his mind, and, doing his best to preserve the feeling of the chaos as well as its memory, he descended a flight of stairs, then another flight of stairs, then a flight of stairs after that, and finally came to the least fashionable room in his whole house.

  It was white. That was pretty much it as far as the color scheme went.

  All along a single wall were plaques, which, following the classic and suggested method, a younger Jeffreyssai had very carefully scribed himself, burning the concepts into his mind with each touch of the brush that wrote the words. That which can be destroyed by the truth should be. People can stand what is true, for they are already enduring it. Curiosity seeks to annihilate itself. Even one small plaque that showed nothing except a red horizontal slash. Symbols could be made to stand for anything; a flexibility of visual power that even the Bardic Conspiracy would balk at admitting outright.