Rationality- From AI to Zombies

by Eliezer Yudkowsky

  In the beginning, the Squishy Things had no fighter jets, no machine guns, no rifles, no swords. No bronze, no iron. No hammers, no anvils, no tongs, no smithies, no mines. All the Squishy Things had were squishy fingers—too weak to break a tree, let alone a mountain. Clearly not dangerous. To cut stone you would need steel, and the Squishy Things couldn’t excrete steel. In the environment there were no steel blades for Squishy fingers to pick up. Their bodies could not generate temperatures anywhere near hot enough to melt metal. The whole scenario was obviously absurd.

  And as for the Squishy Things manipulating DNA—that would have been beyond ridiculous. Squishy fingers are not that small. There is no access to DNA from the Squishy level; it would be like trying to pick up a hydrogen atom. Oh, technically it’s all one universe, technically the Squishy Things and DNA are part of the same world, the same unified laws of physics, the same great web of causality. But let’s be realistic: you can’t get there from here.

  Even if Squishy Things could someday evolve to do any of those feats, it would take thousands of millennia. We have watched the ebb and flow of Life through the eons, and let us tell you, a year is not even a single clock tick of evolutionary time. Oh, sure, technically a year is six hundred trillion trillion trillion trillion Planck intervals. But nothing ever happens in less than six hundred million trillion trillion trillion trillion Planck intervals, so it’s a moot point. The Squishy Things, as they run across the savanna now, will not fly across continents for at least another ten million years; no one could have that much sex.

  Now explain to me again why an Artificial Intelligence can’t do anything interesting over the Internet unless a human programmer builds it a robot body.

  I have observed that someone’s flinch-reaction to “intelligence”—the thought that crosses their mind in the first half-second after they hear the word “intelligence”—often determines their flinch-reaction to the notion of an intelligence explosion. Often they look up the keyword “intelligence” and retrieve the concept booksmarts—a mental image of the Grand Master chess player who can’t get a date, or a college professor who can’t survive outside academia.

  “It takes more than intelligence to succeed professionally,” people say, as if charisma resided in the kidneys, rather than the brain. “Intelligence is no match for a gun,” they say, as if guns had grown on trees. “Where will an Artificial Intelligence get money?” they ask, as if the first Homo sapiens had found dollar bills fluttering down from the sky, and used them at convenience stores already in the forest. The human species was not born into a market economy. Bees won’t sell you honey if you offer them an electronic funds transfer. The human species imagined money into existence, and it exists—for us, not mice or wasps—because we go on believing in it.

  I keep trying to explain to people that the archetype of intelligence is not Dustin Hoffman in Rain Man. It is a human being, period. It is squishy things that explode in a vacuum, leaving footprints on their moon. Within that gray wet lump is the power to search paths through the great web of causality, and find a road to the seemingly impossible—the power sometimes called creativity.

  People—venture capitalists in particular—sometimes ask how, if the Machine Intelligence Research Institute successfully builds a true AI, the results will be commercialized. This is what we call a framing problem.

  Or maybe it’s something deeper than a simple clash of assumptions. With a bit of creative thinking, people can imagine how they would go about travelling to the Moon, or curing smallpox, or manufacturing computers. To imagine a trick that could accomplish all these things at once seems downright impossible—even though such a power resides only a few centimeters behind their own eyes. The gray wet thing still seems mysterious to the gray wet thing.

  And so, because people can’t quite see how it would all work, the power of intelligence seems less real; harder to imagine than a tower of fire sending a ship to Mars. The prospect of visiting Mars captures the imagination. But if one should promise a Mars visit, and also a grand unified theory of physics, and a proof of the Riemann Hypothesis, and a cure for obesity, and a cure for cancer, and a cure for aging, and a cure for stupidity—well, it just sounds wrong, that’s all.

  And well it should. It’s a serious failure of imagination to think that intelligence is good for so little. Who could have imagined, ever so long ago, what minds would someday do? We may not even know what our real problems are.

  But meanwhile, because it’s hard to see how one process could have such diverse powers, it’s hard to imagine that one fell swoop could solve even such prosaic problems as obesity and cancer and aging.

  Well, one trick cured smallpox and built airplanes and cultivated wheat and tamed fire. Our current science may not agree yet on how exactly the trick works, but it works anyway. If you are temporarily ignorant about a phenomenon, that is a fact about your current state of mind, not a fact about the phenomenon. A blank map does not correspond to a blank territory. If one does not quite understand that power which put footprints on the Moon, nonetheless, the footprints are still there—real footprints, on a real Moon, put there by a real power. If one were to understand deeply enough, one could create and shape that power. Intelligence is as real as electricity. It’s merely far more powerful, far more dangerous, has far deeper implications for the unfolding story of life in the universe—and it’s a tiny little bit harder to figure out how to build a generator.

  *

  Part L

  The Simple Math of Evolution

  131

  An Alien God

  “A curious aspect of the theory of evolution,” said Jacques Monod, “is that everybody thinks he understands it.”

  A human being, looking at the natural world, sees a thousand times purpose. A rabbit’s legs, built and articulated for running; a fox’s jaws, built and articulated for tearing. But what you see is not exactly what is there . . .

  In the days before Darwin, the cause of all this apparent purposefulness was a very great puzzle unto science. The Goddists said “God did it,” because you get 50 bonus points each time you use the word “God” in a sentence. Yet perhaps I’m being unfair. In the days before Darwin, it seemed like a much more reasonable hypothesis. Find a watch in the desert, said William Paley, and you can infer the existence of a watchmaker.

  But when you look at all the apparent purposefulness in Nature, rather than picking and choosing your examples, you start to notice things that don’t fit the Judeo-Christian concept of one benevolent God. Foxes seem well-designed to catch rabbits. Rabbits seem well-designed to evade foxes. Was the Creator having trouble making up Its mind?

  When I design a toaster oven, I don’t design one part that tries to get electricity to the coils and a second part that tries to prevent electricity from getting to the coils. It would be a waste of effort. Who designed the ecosystem, with its predators and prey, viruses and bacteria? Even the cactus plant, which you might think well-designed to provide water and fruit to desert animals, is covered with inconvenient spines.

  The ecosystem would make much more sense if it wasn’t designed by a unitary Who, but, rather, created by a horde of deities—say from the Hindu or Shinto religions. This handily explains both the ubiquitous purposefulnesses, and the ubiquitous conflicts: More than one deity acted, often at cross-purposes. The fox and rabbit were both designed, but by distinct competing deities. I wonder if anyone ever remarked on the seemingly excellent evidence thus provided for Hinduism over Christianity. Probably not.

  Similarly, the Judeo-Christian God is alleged to be benevolent—well, sort of. And yet much of nature’s purposefulness seems downright cruel. Darwin suspected a non-standard Creator for studying Ichneumon wasps, whose paralyzing stings preserve its prey to be eaten alive by its larvae: “I cannot persuade myself,” wrote Darwin, “that a beneficent and omnipotent God would have designedly created the Ichneumonidae with the express intention of their feeding within the living bodies of Caterpillars, or th
at a cat should play with mice.”1 I wonder if any earlier thinker remarked on the excellent evidence thus provided for Manichaean religions over monotheistic ones.

  By now we all know the punchline: you just say “evolution.”

  I worry that’s how some people are absorbing the “scientific” explanation, as a magical purposefulness factory in Nature. I’ve previously discussed the case of Storm from the movie X-Men, who in one mutation gets the ability to throw lightning bolts. Why? Well, there’s this thing called “evolution” that somehow pumps a lot of purposefulness into Nature, and the changes happen through “mutations.” So if Storm gets a really large mutation, she can be redesigned to throw lightning bolts. Radioactivity is a popular super origin: radiation causes mutations, so more powerful radiation causes more powerful mutations. That’s logic.

  But evolution doesn’t allow just any kind of purposefulness to leak into Nature. That’s what makes evolution a success as an empirical hypothesis. If evolutionary biology could explain a toaster oven, not just a tree, it would be worthless. There’s a lot more to evolutionary theory than pointing at Nature and saying, “Now purpose is allowed,” or “Evolution did it!” The strength of a theory is not what it allows, but what it prohibits; if you can invent an equally persuasive explanation for any outcome, you have zero knowledge.

  “Many non-biologists,” observed George Williams, “think that it is for their benefit that rattles grow on rattlesnake tails.”2 Bzzzt! This kind of purposefulness is not allowed. Evolution doesn’t work by letting flashes of purposefulness creep in at random—reshaping one species for the benefit of a random recipient.

  Evolution is powered by a systematic correlation between the different ways that different genes construct organisms, and how many copies of those genes make it into the next generation. For rattles to grow on rattlesnake tails, rattle-growing genes must become more and more frequent in each successive generation. (Actually genes for incrementally more complex rattles, but if I start describing all the fillips and caveats to evolutionary biology, we really will be here all day.)

  There isn’t an Evolution Fairy that looks over the current state of Nature, decides what would be a “good idea,” and chooses to increase the frequency of rattle-constructing genes.

  I suspect this is where a lot of people get stuck, in evolutionary biology. They understand that “helpful” genes become more common, but “helpful” lets any sort of purpose leak in. They don’t think there’s an Evolution Fairy, yet they ask which genes will be “helpful” as if a rattlesnake gene could “help” non-rattlesnakes.

  The key realization is that there is no Evolution Fairy. There’s no outside force deciding which genes ought to be promoted. Whatever happens, happens because of the genes themselves.

  Genes for constructing (incrementally better) rattles must have somehow ended up more frequent in the rattlesnake gene pool, because of the rattle. In this case it’s probably because rattlesnakes with better rattles survive more often—rather than mating more successfully, or having brothers that reproduce more successfully, etc.

  Maybe predators are wary of rattles and don’t step on the snake. Or maybe the rattle diverts attention from the snake’s head. (As George Williams suggests, “The outcome of a fight between a dog and a viper would depend very much on whether the dog initially seized the reptile by the head or by the tail.”)

  But that’s just a snake’s rattle. There are much more complicated ways that a gene can cause copies of itself to become more frequent in the next generation. Your brother or sister shares half your genes. A gene that sacrifices one unit of resources to bestow three units of resource on a brother, may promote some copies of itself by sacrificing one of its constructed organisms. (If you really want to know all the details and caveats, buy a book on evolutionary biology; there is no royal road.)

  The main point is that the gene’s effect must cause copies of that gene to become more frequent in the next generation. There’s no Evolution Fairy that reaches in from outside. There’s nothing which decides that some genes are “helpful” and should, therefore, increase in frequency. It’s just cause and effect, starting from the genes themselves.

  This explains the strange conflicting purposefulness of Nature, and its frequent cruelty. It explains even better than a horde of Shinto deities.

  Why is so much of Nature at war with other parts of Nature? Because there isn’t one Evolution directing the whole process. There’s as many different “evolutions” as reproducing populations. Rabbit genes are becoming more or less frequent in rabbit populations. Fox genes are becoming more or less frequent in fox populations. Fox genes which construct foxes that catch rabbits, insert more copies of themselves in the next generation. Rabbit genes which construct rabbits that evade foxes are naturally more common in the next generation of rabbits. Hence the phrase “natural selection.”

  Why is Nature cruel? You, a human, can look at an Ichneumon wasp, and decide that it’s cruel to eat your prey alive. You can decide that if you’re going to eat your prey alive, you can at least have the decency to stop it from hurting. It would scarcely cost the wasp anything to anesthetize its prey as well as paralyze it. Or what about old elephants, who die of starvation when their last set of teeth fall out? These elephants aren’t going to reproduce anyway. What would it cost evolution—the evolution of elephants, rather—to ensure that the elephant dies right away, instead of slowly and in agony? What would it cost evolution to anesthetize the elephant, or give it pleasant dreams before it dies? Nothing; that elephant won’t reproduce more or less either way.

  If you were talking to a fellow human, trying to resolve a conflict of interest, you would be in a good negotiating position—would have an easy job of persuasion. It would cost so little to anesthetize the prey, to let the elephant die without agony! Oh please, won’t you do it, kindly . . . um . . .

  There’s no one to argue with.

  Human beings fake their justifications, figure out what they want using one method, and then justify it using another method. There’s no Evolution of Elephants Fairy that’s trying to (a) figure out what’s best for elephants, and then (b) figure out how to justify it to the Evolutionary Overseer, who (c) doesn’t want to see reproductive fitness decreased, but is (d) willing to go along with the painless-death idea, so long as it doesn’t actually harm any genes.

  There’s no advocate for the elephants anywhere in the system.

  Humans, who are often deeply concerned for the well-being of animals, can be very persuasive in arguing how various kindnesses wouldn’t harm reproductive fitness at all. Sadly, the evolution of elephants doesn’t use a similar algorithm; it doesn’t select nice genes that can plausibly be argued to help reproductive fitness. Simply: genes that replicate more often become more frequent in the next generation. Like water flowing downhill, and equally benevolent.

  A human, looking over Nature, starts thinking of all the ways we would design organisms. And then we tend to start rationalizing reasons why our design improvements would increase reproductive fitness—a political instinct, trying to sell your own preferred option as matching the boss’s favored justification.

  And so, amateur evolutionary biologists end up making all sorts of wonderful and completely mistaken predictions. Because the amateur biologists are drawing their bottom line—and more importantly, locating their prediction in hypothesis-space—using a different algorithm than evolutions use to draw their bottom lines.

  A human engineer would have designed human taste buds to measure how much of each nutrient we had, and how much we needed. When fat was scarce, almonds or cheeseburgers would taste delicious. But if you started to become obese, or if vitamins were lacking, lettuce would taste delicious. But there is no Evolution of Humans Fairy, which intelligently planned ahead and designed a general system for every contingency. It was a reliable invariant of humans’ ancestral environment that calories were scarce. So genes whose organisms loved calories, became more frequent. Like wat
er flowing downhill.

  We are simply the embodied history of which organisms did in fact survive and reproduce, not which organisms ought prudentially to have survived and reproduced.

  The human retina is constructed backward: The light-sensitive cells are at the back, and the nerves emerge from the front and go back through the retina into the brain. Hence the blind spot. To a human engineer, this looks simply stupid—and other organisms have independently evolved retinas the right way around. Why not redesign the retina?

  The problem is that no single mutation will reroute the whole retina simultaneously. A human engineer can redesign multiple parts simultaneously, or plan ahead for future changes. But if a single mutation breaks some vital part of the organism, it doesn’t matter what wonderful things a Fairy could build on top of it—the organism dies and the gene decreases in frequency.

  If you turn around the retina’s cells without also reprogramming the nerves and optic cable, the system as a whole won’t work. It doesn’t matter that, to a Fairy or a human engineer, this is one step forward in redesigning the retina. The organism is blind. Evolution has no foresight, it is simply the frozen history of which organisms did in fact reproduce. Evolution is as blind as a halfway-redesigned retina.

  Find a watch in a desert, said William Paley, and you can infer the watchmaker. There were once those who denied this, who thought that life “just happened” without need of an optimization process, mice being spontaneously generated from straw and dirty shirts.

  If we ask who was more correct—the theologians who argued for a Creator-God, or the intellectually unfulfilled atheists who argued that mice spontaneously generated—then the theologians must be declared the victors: evolution is not God, but it is closer to God than it is to pure random entropy. Mutation is random, but selection is non-random. This doesn’t mean an intelligent Fairy is reaching in and selecting. It means there’s a non-zero statistical correlation between the gene and how often the organism reproduces. Over a few million years, that non-zero statistical correlation adds up to something very powerful. It’s not a god, but it’s more closely akin to a god than it is to snow on a television screen.