Rationality- From AI to Zombies

by Eliezer Yudkowsky

  Humans do not process evidence efficiently—our minds are so noisy that it requires orders of magnitude more extra evidence to set us back on track after we derail. Our collective, academia, is even slower.

  *

  1. Robert Matthews, “Do We Need to Change the Definition of Science?,” New Scientist (May 2008).

  251

  Faster Than Science

  I sometimes say that the method of science is to amass such an enormous mountain of evidence that even scientists cannot ignore it; and that this is the distinguishing characteristic of a scientist. (A non-scientist will ignore it anyway.)

  Max Planck was even less optimistic:1

  A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

  I am much tickled by this notion, because it implies that the power of science to distinguish truth from falsehood ultimately rests on the good taste of grad students.

  The gradual increase in acceptance of many-worlds in academic physics suggests that there are physicists who will only accept a new idea given some combination of epistemic justification, and a sufficiently large academic pack in whose company they can be comfortable. As more physicists accept, the pack grows larger, and hence more people go over their individual thresholds for conversion—with the epistemic justification remaining essentially the same.

  But Science still gets there eventually, and this is sufficient for the ratchet of Science to move forward, and raise up a technological civilization.

  Scientists can be moved by groundless prejudices, by undermined intuitions, by raw herd behavior—the panoply of human flaws. Each time a scientist shifts belief for epistemically unjustifiable reasons, it requires more evidence, or new arguments, to cancel out the noise.

  The “collapse of the wavefunction” has no experimental justification, but it appeals to the (undermined) intuition of a single world. Then it may take an extra argument—say, that collapse violates Special Relativity—to begin the slow academic disintegration of an idea that should never have been assigned non-negligible probability in the first place.

  From a Bayesian perspective, human academic science as a whole is a highly inefficient processor of evidence. Each time an unjustifiable argument shifts belief, you need an extra justifiable argument to shift it back. The social process of science leans on extra evidence to overcome cognitive noise.

  A more charitable way of putting it is that scientists will adopt positions that are theoretically insufficiently extreme, compared to the ideal positions that scientists would adopt, if they were Bayesian AIs and could trust themselves to reason clearly.

  But don’t be too charitable. The noise we are talking about is not all innocent mistakes. In many fields, debates drag on for decades after they should have been settled. And not because the scientists on both sides refuse to trust themselves and agree they should look for additional evidence. But because one side keeps throwing up more and more ridiculous objections, and demanding more and more evidence, from an entrenched position of academic power, long after it becomes clear from which quarter the winds of evidence are blowing. (I’m thinking here about the debates surrounding the invention of evolutionary psychology, not about many-worlds.)

  Is it possible for individual humans or groups to process evidence more efficiently—reach correct conclusions faster—than human academic science as a whole?

  “Ideas are tested by experiment. That is the core of science.” And this must be true, because if you can’t trust Zombie Feynman, who can you trust?

  Yet where do the ideas come from?

  You may be tempted to reply, “They come from scientists. Got any other questions?” In Science you’re not supposed to care where the hypotheses come from—just whether they pass or fail experimentally.

  Okay, but if you remove all new ideas, the scientific process as a whole stops working because it has no alternative hypotheses to test. So inventing new ideas is not a dispensable part of the process.

  Now put your Bayesian goggles back on. As described in Einstein’s Arrogance, there are queries that are not binary—where the answer is not “Yes” or “No,” but drawn from a larger space of structures, e.g., the space of equations. In such cases it takes far more Bayesian evidence to promote a hypothesis to your attention than to confirm the hypothesis.

  If you’re working in the space of all equations that can be specified in 32 bits or less, you’re working in a space of 4 billion equations. It takes far more Bayesian evidence to raise one of those hypotheses to the 10% probability level, than it requires to further raise the hypothesis from 10% to 90% probability.

  When the idea-space is large, coming up with ideas worthy of testing involves much more work—in the Bayesian-thermodynamic sense of “work”—than merely obtaining an experimental result with p < 0.0001 for the new hypothesis over the old hypothesis.

  If this doesn’t seem obvious-at-a-glance, pause here and review Einstein’s Arrogance.

  The scientific process has always relied on scientists to come up with hypotheses to test, via some process not further specified by Science. Suppose you came up with some way of generating hypotheses that was completely crazy—say, pumping a robot-controlled Ouija board with the digits of pi—and the resulting suggestions kept on getting verified experimentally. The pure ideal essence of Science wouldn’t skip a beat. The pure ideal essence of Bayes would burst into flames and die.

  (Compared to Science, Bayes is falsified by more of the possible outcomes.)

  This doesn’t mean that the process of deciding which ideas to test is unimportant to Science. It means that Science doesn’t specify it.

  In practice, the robot-controlled Ouija board doesn’t work. In practice, there are some scientific queries with a large enough answer space that, picking models at random to test, it would take zillions of years to hit on a model that made good predictions—like getting monkeys to type Shakespeare.

  At the frontier of science—the boundary between ignorance and knowledge, where science advances—the process relies on at least some individual scientists (or working groups) seeing things that are not yet confirmed by Science. That’s how they know which hypotheses to test, in advance of the test itself.

  If you take your Bayesian goggles off, you can say, “Well, they don’t have to know, they just have to guess.” If you put your Bayesian goggles back on, you realize that “guessing” with 10% probability requires nearly as much epistemic work to have been successfully performed, behind the scenes, as “guessing” with 80% probability—at least for large answer spaces.

  The scientist may not know they have done this epistemic work successfully, in advance of the experiment; but they must, in fact, have done it successfully! Otherwise they will not even think of the correct hypothesis. In large answer spaces, anyway.

  So the scientist makes the novel prediction, performs the experiment, publishes the result, and now Science knows it too. It is now part of the publicly accessible knowledge of humankind, that anyone can verify for themselves.

  In between was an interval where the scientist rationally knew something that the public social process of science hadn’t yet confirmed. And this is not a trivial interval, though it may be short; for it is where the frontier of science lies, the advancing border.

  All of this is more true for non-routine science than for routine science, because it is a notion of large answer spaces where the answer is not “Yes” or “No” or drawn from a small set of obvious alternatives. It is much easier to train people to test ideas than to have good ideas to test.

  *

  1. Max Planck, Scientific Autobiography and Other Papers (New York: Philosophical Library, 1949).

  252

  Einstein’s Speed

  In the previous essay I argued that the Powers Beyond Science are actually a standard and necessary part of the social process of scien
ce. In particular, scientists must call upon their powers of individual rationality to decide what ideas to test, in advance of the sort of definite experiments that Science demands to bless an idea as confirmed. The ideal of Science does not try to specify this process—we don’t suppose that any public authority knows how individual scientists should think—but this doesn’t mean the process is unimportant.

  A readily understandable, non-disturbing example:

  A scientist identifies a strong mathematical regularity in the cumulative data of previous experiments. But the corresponding hypothesis has not yet made and confirmed a novel experimental prediction—which their academic field demands; this is one of those fields where you can perform controlled experiments without too much trouble. Thus the individual scientist has readily understandable, rational reasons to believe (though not with probability 1) something not yet blessed by Science as public knowledge of humankind.

  Noticing a regularity in a huge mass of experimental data doesn’t seem all that unscientific. You’re still data-driven, right?

  But that’s because I deliberately chose a non-disturbing example. When Einstein invented General Relativity, he had almost no experimental data to go on, except the precession of Mercury’s perihelion. And (as far as I know) Einstein did not use that data, except at the end.

  Einstein generated the theory of Special Relativity using Mach’s Principle, which is the physicist’s version of the Generalized Anti-Zombie Principle. You begin by saying, “It doesn’t seem reasonable to me that you could tell, in an enclosed room, how fast you and the room were going. Since this number shouldn’t ought to be observable, it shouldn’t ought to exist in any meaningful sense.” You then observe that Maxwell’s Equations invoke a seemingly absolute speed of propagation, c, commonly referred to as “the speed of light” (though the quantum equations show it is the propagation speed of all fundamental waves). So you reformulate your physics in such fashion that the absolute speed of a single object no longer meaningfully exists, and only relative speeds exist. I am skipping over quite a bit here, obviously, but there are many excellent introductions to relativity—it is not like the horrible situation in quantum physics.

  Einstein, having successfully done away with the notion of your absolute speed inside an enclosed room, then set out to do away with the notion of your absolute acceleration inside an enclosed room. It seemed to Einstein that there shouldn’t ought to be a way to differentiate, in an enclosed room, between the room accelerating northward while the rest of the universe stayed still, versus the rest of the universe accelerating southward while the room stayed still. If the rest of the universe accelerated, it would produce gravitational waves that would accelerate you. Moving matter, then, should produce gravitational waves.

  And because inertial mass and gravitational mass were always exactly equivalent—unlike the situation in electromagnetics, where an electron and a muon can have different masses but the same electrical charge—gravity should reveal itself as a kind of inertia. The Earth should go around the Sun in some equivalent of a “straight line.” This requires spacetime in the vicinity of the Sun to be curved, so that if you drew a graph of the Earth’s orbit around the Sun, the line on the 4D graph paper would be locally flat. Then inertial and gravitational mass would be necessarily equivalent, not just coincidentally equivalent.

  (If that did not make any sense to you, there are good introductions to General Relativity available as well.)

  And of course the new theory had to obey Special Relativity, and conserve energy, and conserve momentum, et cetera.

  Einstein spent several years grasping the necessary mathematics to describe curved metrics of spacetime. Then he wrote down the simplest theory that had the properties Einstein thought it ought to have—including properties no one had ever observed, but that Einstein thought fit in well with the character of other physical laws. Then Einstein cranked a bit, and got the previously unexplained precession of Mercury right back out.

  How impressive was this?

  Well, let’s put it this way. In some small fraction of alternate Earths proceeding from 1800—perhaps even a sizeable fraction—it would seem plausible that relativistic physics could have proceeded in a similar fashion to our own great fiasco with quantum physics.

  We can imagine that Lorentz’s original “interpretation” of the Lorentz contraction, as a physical distortion caused by movement with respect to the ether, prevailed. We can imagine that various corrective factors, themselves unexplained, were added on to Newtonian gravitational mechanics to explain the precession of Mercury—attributed, perhaps, to strange distortions of the ether, as in the Lorentz contraction. Through the decades, further corrective factors would be added on to account for other astronomical observations. Sufficiently precise atomic clocks, in airplanes, would reveal that time ran a little faster than expected at higher altitudes (time runs slower in more intense gravitational fields, but they wouldn’t know that) and more corrective “ethereal factors” would be invented.

  Until, finally, the many different empirically determined “corrective factors” were unified into the simple equations of General Relativity.

  And the people in that alternate Earth would say, “The final equation was simple, but there was no way you could possibly know to arrive at that answer from just the perihelion precession of Mercury. It takes many, many additional experiments. You must have measured time running slower in a stronger gravitational field; you must have measured light bending around stars. Only then can you imagine our unified theory of ethereal gravitation. No, not even a perfect Bayesian superintelligence could know it!—for there would be many ad-hoc theories consistent with the perihelion precession alone.”

  In our world, Einstein didn’t even use the perihelion precession of Mercury, except for verification of his answer produced by other means. Einstein sat down in his armchair, and thought about how he would have designed the universe, to look the way he thought a universe should look—for example, that you shouldn’t ought to be able to distinguish yourself accelerating in one direction, from the rest of the universe accelerating in the other direction.

  And Einstein executed the whole long (multi-year!) chain of armchair reasoning, without making any mistakes that would have required further experimental evidence to pull him back on track.

  Even Jeffreyssai would be grudgingly impressed. Though he would still ding Einstein a point or two for the cosmological constant. (I don’t ding Einstein for the cosmological constant because it later turned out to be real. I try to avoid criticizing people on occasions where they are right.)

  What would be the probability-theoretic perspective on Einstein’s feat?

  Rather than observe the planets, and infer what laws might cover their gravitation, Einstein was observing the other laws of physics, and inferring what new law might follow the same pattern. Einstein wasn’t finding an equation that covered the motion of gravitational bodies. Einstein was finding a character-of-physical-law that covered previously observed equations, and that he could crank to predict the next equation that would be observed.

  Nobody knows where the laws of physics come from, but Einstein’s success with General Relativity shows that their common character is strong enough to predict the correct form of one law from having observed other laws, without necessarily having to observe the precise effects of the law.

  (In a general sense, of course, Einstein did know by observation that things fell down; but he did not get General Relativity by backward inference from Mercury’s exact perihelion advance.)

  So, from a Bayesian perspective, what Einstein did is still induction, and still covered by the notion of a simple prior (Occam prior) that gets updated by new evidence. It’s just the prior was over the possible characters of physical law, and observing other physical laws let Einstein update his model of the character of physical law, which he then used to predict a particular law of gravitation.

  If you didn’t have the concept of a �
��character of physical law,” what Einstein did would look like magic—plucking the correct model of gravitation out of the space of all possible equations, with vastly insufficient evidence. But Einstein, by looking at other laws, cut down the space of possibilities for the next law. He learned the alphabet in which physics was written, constraints to govern his answer. Not magic, but reasoning on a higher level, across a wider domain, than what a naive reasoner might conceive to be the “model space” of only this one law.

  So from a probability-theoretic standpoint, Einstein was still data-driven—he just used the data he already had, more effectively. Compared to any alternate Earths that demanded huge quantities of additional data from astronomical observations and clocks on airplanes to hit them over the head with General Relativity.

  There are numerous lessons we can derive from this.

  I use Einstein as my example, even though it’s cliché, because Einstein was also unusual in that he openly admitted to knowing things that Science hadn’t confirmed. Asked what he would have done if Eddington’s solar eclipse observation had failed to confirm General Relativity, Einstein replied: “Then I would feel sorry for the good Lord. The theory is correct.”

  According to prevailing notions of Science, this is arrogance—you must accept the verdict of experiment, and not cling to your personal ideas.

  But as I concluded in Einstein’s Arrogance, Einstein doesn’t come off nearly as badly from a Bayesian perspective. From a Bayesian perspective, in order to suggest General Relativity at all, in order to even think about what turned out to be the correct answer, Einstein must have had enough evidence to identify the true answer in the theory-space. It would take only a little more evidence to justify (in a Bayesian sense) being nearly certain of the theory. And it was unlikely that Einstein only had exactly enough evidence to bring the hypothesis all the way up to his attention.