Rationality- From AI to Zombies

by Eliezer Yudkowsky

  Or what about cryonics?

  Cryonics is an archetypal example of an extremely important issue (150,000 people die per day) that will have huge consequences in the foreseeable future, but doesn’t offer definite unmistakable experimental evidence that we can get right now.

  So do you say, “I don’t believe in cryonics because it hasn’t been experimentally proven, and you shouldn’t believe in things that haven’t been experimentally proven”?

  Well, from a Bayesian perspective, that’s incorrect. Absence of evidence is evidence of absence only to the degree that we could reasonably expect the evidence to appear. If someone is trumpeting that snake oil cures cancer, you can reasonably expect that, if the snake oil were actually curing cancer, some scientist would be performing a controlled study to verify it—that, at the least, doctors would be reporting case studies of amazing recoveries—and so the absence of this evidence is strong evidence of absence. But “gaps in the fossil record” are not strong evidence against evolution; fossils form only rarely, and even if an intermediate species did in fact exist, you cannot expect with high probability that Nature will obligingly fossilize it and that the fossil will be discovered.

  Reviving a cryonically frozen mammal is just not something you’d expect to be able to do with modern technology, even if future nanotechnologies could in fact perform a successful revival. That’s how I see Bayes seeing it.

  Oh, and as for the actual arguments for cryonics—I’m not going to go into those at the moment. But if you followed the physics and anti-Zombie sequences, it should now seem a lot more plausible that whatever preserves the pattern of synapses preserves as much of “you” as is preserved from one night’s sleep to morning’s waking.

  Now, to be fair, someone who says, “I don’t believe in cryonics because it hasn’t been proven experimentally” is misapplying the rules of Science; this is not a case where science actually gives the wrong answer. In the absence of a definite experimental test, the verdict of science here is “Not proven.” Anyone who interprets that as a rejection is taking an extra step outside of science, not a misstep within science.

  John McCarthy’s Wikiquotes page has him saying, “Your statements amount to saying that if AI is possible, it should be easy. Why is that?”1 The Wikiquotes page doesn’t say what McCarthy was responding to, but I could venture a guess.

  The general mistake probably arises because there are cases where the absence of scientific proof is strong evidence—because an experiment would be readily performable, and so failure to perform it is itself suspicious. (Though not as suspicious as I used to think—with all the strangely varied anecdotal evidence coming in from respected sources, why the hell isn’t anyone testing Seth Roberts’s theory of appetite suppression?2)

  Another confusion factor may be that if you test Pharmaceutical X on 1,000 subjects and find that 56% of the control group and 57% of the experimental group recover, some people will call that a verdict of “Not proven.” I would call it an experimental verdict of “Pharmaceutical X doesn’t work well, if at all.” Just because this verdict is theoretically retractable in the face of new evidence doesn’t make it ambiguous.

  In any case, right now you’ve got people dismissing cryonics out of hand as “not scientific,” like it was some kind of pharmaceutical you could easily administer to 1,000 patients and see what happened. “Call me when cryonicists actually revive someone,” they say; which, as Mike Li observes, is like saying “I refuse to get into this ambulance; call me when it’s actually at the hospital.” Maybe Martin Gardner warned them against believing in strange things without experimental evidence. So they wait for the definite unmistakable verdict of Science, while their family and friends and 150,000 people per day are dying right now, and might or might not be savable—

  —a calculated bet you could only make rationally.

  The drive of Science is to obtain a mountain of evidence so huge that not even fallible human scientists can misread it. But even that sometimes goes wrong, when people become confused about which theory predicts what, or bake extremely-hard-to-test components into an early version of their theory. And sometimes you just can’t get clear experimental evidence at all.

  Either way, you have to try to do the thing that Science doesn’t trust anyone to do—think rationally, and figure out the answer before you get clubbed over the head with it.

  (Oh, and sometimes a disconfirming experimental result looks like: “Your entire species has just been wiped out! You are now scientifically required to relinquish your theory. If you publicly recant, good for you! Remember, it takes a strong mind to give up strongly held beliefs. Feel free to try another hypothesis next time!”)

  *

  1. No longer on Wikiquotes, but included in McCarthy’s personal quotes page.

  2. Seth Roberts, “What Makes Food Fattening?: A Pavlovian Theory of Weight Control” (Unpublished manuscript, 2005), http://media.sethroberts.net/about/whatmakesfoodfattening.pdf.

  247

  Science Isn’t Strict Enough

  Once upon a time, a younger Eliezer had a stupid theory. Eliezer18 was careful to follow the precepts of Traditional Rationality that he had been taught; he made sure his stupid theory had experimental consequences. Eliezer18 professed, in accordance with the virtues of a scientist he had been taught, that he wished to test his stupid theory.

  This was all that was required to be virtuous, according to what Eliezer18 had been taught was virtue in the way of science.

  It was not even remotely the order of effort that would have been required to get it right.

  The traditional ideals of Science too readily give out gold stars. Negative experimental results are also knowledge, so everyone who plays gets an award. So long as you can think of some kind of experiment that tests your theory, and you do the experiment, and you accept the results, you’ve played by the rules; you’re a good scientist.

  You didn’t necessarily get it right, but you’re a nice science-abiding citizen.

  (I note at this point that I am speaking of Science, not the social process of science as it actually works in practice, for two reasons. First, I went astray in trying to follow the ideal of Science—it’s not like I was shot down by a journal editor with a grudge, and it’s not like I was trying to imitate the flaws of academia. Second, if I point out a problem with the ideal as it is traditionally preached, real-world scientists are not forced to likewise go astray!)

  Science began as a rebellion against grand philosophical schemas and armchair reasoning. So Science doesn’t include a rule as to what kinds of hypotheses you are and aren’t allowed to test; that is left up to the individual scientist. Trying to guess that a priori would require some kind of grand philosophical schema, and reasoning in advance of the evidence. As a social ideal, Science doesn’t judge you as a bad person for coming up with heretical hypotheses; honest experiments, and acceptance of the results, is virtue unto a scientist.

  As long as most scientists can manage to accept definite, unmistakable, unambiguous experimental evidence, science can progress. It may happen too slowly—it may take longer than it should—you may have to wait for a generation of elders to die out—but eventually, the ratchet of knowledge clicks forward another notch. Year by year, decade by decade, the wheel turns forward. It’s enough to support a civilization.

  So that’s all that Science really asks of you—the ability to accept reality when you’re beat over the head with it. It’s not much, but it’s enough to sustain a scientific culture.

  Contrast this to the notion we have in probability theory, of an exact quantitative rational judgment. If 1% of women presenting for a routine screening have breast cancer, and 80% of women with breast cancer get positive mammographies, and 10% of women without breast cancer get false positives, what is the probability that a routinely screened woman with a positive mammography has breast cancer? It is 7.5%. You cannot say, “I believe she doesn’t have breast cancer, because the experiment isn’t defi
nite enough.” You cannot say, “I believe she has breast cancer, because it is wise to be pessimistic and that is what the only experiment so far seems to indicate.” Seven point five percent is the rational estimate given this evidence, not 7.4% or 7.6%. The laws of probability are laws.

  It is written in the Twelve Virtues, of the third virtue, lightness:

  If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims. For you cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse. You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

  In Science, when it comes to deciding which hypotheses to test, the morality of Science gives you personal freedom of what to believe, so long as it isn’t already ruled out by experiment, and so long as you move to test your hypothesis. Science wouldn’t try to give an official verdict on the best hypothesis to test, in advance of the experiment. That’s left up to the conscience of the individual scientist.

  Where definite experimental evidence exists, Science tells you to bow your stubborn neck and accept it. Otherwise, Science leaves it up to you. Science gives you room to wander around within the boundaries of the experimental evidence, according to your whims.

  And this is not easily reconciled with Bayesianism’s notion of an exactly right probability estimate, one with no flex or room for whims, that exists both before and after the experiment. Bayesianism doesn’t match well with the ancient and traditional reason for Science—the distrust of grand schemas, the presumption that people aren’t rational enough to get things right without definite and unmistakable experimental evidence. If we were all perfect Bayesians, we wouldn’t need a social process of science.

  Nonetheless, around the time I realized my big mistake, I had also been studying Kahneman and Tversky and Jaynes. I was learning a new Way, stricter than Science. A Way that could criticize my folly, in a way that Science never could. A Way that could have told me what Science would never have said in advance: “You picked the wrong hypothesis to test, dunderhead.”

  But the Way of Bayes is also much harder to use than Science. It puts a tremendous strain on your ability to hear tiny false notes, where Science only demands that you notice an anvil dropped on your head.

  In Science you can make a mistake or two, and another experiment will come by and correct you; at worst you waste a couple of decades.

  But if you try to use Bayes even qualitatively—if you try to do the thing that Science doesn’t trust you to do, and reason rationally in the absence of overwhelming evidence—it is like math, in that a single error in a hundred steps can carry you anywhere. It demands lightness, evenness, precision, perfectionism.

  There’s a good reason why Science doesn’t trust scientists to do this sort of thing, and asks for further experimental proof even after someone claims they’ve worked out the right answer based on hints and logic.

  But if you would rather not waste ten years trying to prove the wrong theory, you’ll need to essay the vastly more difficult problem: listening to evidence that doesn’t shout in your ear.

  Even if you can’t look up the priors for a problem in the Handbook of Chemistry and Physics—even if there’s no Authoritative Source telling you what the priors are—that doesn’t mean you get a free, personal choice of making the priors whatever you want. It means you have a new guessing problem that you must carry out to the best of your ability.

  If the mind, as a cognitive engine, could generate correct estimates by fiddling with priors according to whims, you could know things without looking them, or even alter them without touching them. But the mind is not magic. The rational probability estimate has no room for any decision based on whim, even when it seems that you don’t know the priors.

  Similarly, if the Bayesian answer is difficult to compute, that doesn’t mean that Bayes is inapplicable; it means you don’t know what the Bayesian answer is. Bayesian probability theory is not a toolbox of statistical methods; it’s the law that governs any tool you use, whether or not you know it, whether or not you can calculate it.

  As for using Bayesian methods on huge, highly general hypothesis spaces—like, “Here’s the data from every physics experiment ever; now, what would be a good Theory of Everything?”—if you knew how to do that in practice, you wouldn’t be a statistician, you would be an Artificial General Intelligence programmer. But that doesn’t mean that human beings, in modeling the universe using human intelligence, are violating the laws of physics / Bayesianism by generating correct guesses without evidence.

  Nick Tarleton comments:

  The problem is encouraging a private, epistemic standard as lax as the social one.

  which pinpoints the problem I was trying to indicate much better than I did.

  *

  248

  Do Scientists Already Know This Stuff?

  poke alleges:

  Being able to create relevant hypotheses is an important skill and one a scientist spends a great deal of his or her time developing. It may not be part of the traditional description of science but that doesn’t mean it’s not included in the actual social institution of science that produces actual real science here in the real world; it’s your description and not science that is faulty.

  I know I’ve been calling my younger self “stupid,” but that is a figure of speech; “unskillfully wielding high intelligence” would be more precise. Eliezer18 was not in the habit of making obvious mistakes—it’s just that his “obvious” wasn’t my “obvious.”

  No, I did not go through the traditional apprenticeship. But when I look back, and see what Eliezer18 did wrong, I see plenty of modern scientists making the same mistakes. I cannot detect any sign that they were better warned than myself.

  Sir Roger Penrose—a world-class physicist—still thinks that consciousness is caused by quantum gravity. I expect that no one ever warned him against mysterious answers to mysterious questions—only told him his hypotheses needed to be falsifiable and have empirical consequences. Just like Eliezer18.

  “Consciousness is caused by quantum gravity” has testable implications: It implies that you should be able to look at neurons and discover a coherent quantum superposition whose collapse contributes to information-processing, and that you won’t ever be able to reproduce a neuron’s input-output behavior using a computable microanatomical simulation . . .

  . . . but even after you say “Consciousness is caused by quantum gravity,” you don’t anticipate anything about how your brain thinks “I think therefore I am!” or the mysterious redness of red, that you did not anticipate before, even though you feel like you know a cause of it. This is a tremendous danger sign, I now realize, but it’s not the danger sign that I was warned against, and I doubt that Penrose was ever told of it by his thesis advisor. For that matter, I doubt that Niels Bohr was ever warned against it when it came time to formulate the Copenhagen Interpretation.

  As far as I can tell, the reason Eliezer18 and Sir Roger Penrose and Niels Bohr were not warned is that no standard warning exists.

  I did not generalize the concept of “mysterious answers to mysterious questions,” in that many words, until I was writing a Bayesian analysis of what distinguishes technical, nontechnical and semitechnical scientific explanations. Now, the final output of that analysis can be phrased nontechnically in terms of four danger signs:

  First, the explanation acts as a curiosity-stopper rather than an anticipation-controller.

  Second, the hypothesis has no moving parts—the secret sauce is not a specific complex mechanism, but a blankly solid substance or force.

  Third, those who proffer the explanation cherish their ignorance; they speak proudly of how the phenomenon defeats ordinary science or is unlike merely mun
dane phenomena.

  Fourth, even after the answer is given, the phenomenon is still a mystery and possesses the same quality of wonderful inexplicability that it had at the start.

  In principle, all this could have been said in the immediate aftermath of vitalism. Just like elementary probability theory could have been invented by Archimedes, or the ancient Greeks could have theorized natural selection. But in fact no one ever warned me against any of these four dangers, in those terms—the closest being the warning that hypotheses should have testable consequences. And I didn’t conceptualize the warning signs explicitly until I was trying to think of the whole affair in terms of probability distributions—some degree of overkill was required.

  I simply have no reason to believe that these warnings are passed down in scientific apprenticeships—certainly not to a majority of scientists. Among other things, it is advice for handling situations of confusion and despair, scientific chaos. When would the average scientist or average mentor have an opportunity to use that kind of technique?

  We just got through discussing the single-world fiasco in physics. Clearly, no one told them about the formal definition of Occam’s Razor, in whispered apprenticeship or otherwise.

  There is a known effect where great scientists have multiple great students. This may well be due to the mentors passing on skills that they can’t describe. But I don’t think that counts as part of standard science. And if the great mentors haven’t been able to put their guidance into words and publish it generally, that’s not a good sign for how well these things are understood.