Rationality- From AI to Zombies

by Eliezer Yudkowsky

  Around 50% of the total genome of maize consists of transposons, DNA elements whose primary function is to copy themselves into other locations of DNA. A class of transposons called “P elements” seem to have first appeared in Drosophila only in the middle of the twentieth century, and spread to every population of the species within 50 years. The “Alu sequence” in humans, a 300-base transposon, is repeated between 300,000 and a million times in the human genome. This may not extinguish a species, but it doesn’t help it; transposons cause more mutations which are as always mostly harmful, decrease the effective copying fidelity of DNA. Yet such cheaters are extremely fit.

  Suppose that in some sexually reproducing species, a perfect DNA-copying mechanism is invented. Since most mutations are detrimental, this gene complex is an advantage to its holders. Now you might wonder about beneficial mutations—they do happen occasionally, so wouldn’t the unmutable be at a disadvantage? But in a sexual species, a beneficial mutation that began in a mutable can spread to the descendants of unmutables as well. The mutables suffer from degenerate mutations in each generation; and the unmutables can sexually acquire, and thereby benefit from, any beneficial mutations that occur in the mutables. Thus the mutables have a pure disadvantage. The perfect DNA-copying mechanism rises in frequency to fixation. Ten thousand years later there’s an ice age and the species goes out of business. It evolved to extinction.

  The “bystander effect” is that, when someone is in trouble, solitary individuals are more likely to intervene than groups. A college student apparently having an epileptic seizure was helped 85% of the time by a single bystander, and 31% of the time by five bystanders. I speculate that even if the kinship relation in a hunter-gatherer tribe was strong enough to create a selection pressure for helping individuals not directly related, when several potential helpers were present, a genetic arms race might occur to be the last one to step forward. Everyone delays, hoping that someone else will do it. Humanity is facing multiple species-level extinction threats right now, and I gotta tell ya, there ain’t a lot of people steppin’ forward. If we lose this fight because virtually no one showed up on the battlefield, then—like a probably-large number of species which we don’t see around today—we will have evolved to extinction.

  Cancerous cells do pretty well in the body, prospering and amassing more resources, far outcompeting their more obedient counterparts. For a while.

  Multicellular organisms can only exist because they’ve evolved powerful internal mechanisms to outlaw evolution. If the cells start evolving, they rapidly evolve to extinction: the organism dies.

  So praise not evolution for the solicitous concern it shows for the individual; nearly all of your ancestors are dead. Praise not evolution for the solicitous concern it shows for a species; no one has ever found a complex adaptation which can only be interpreted as operating to preserve a species, and the mathematics would seem to indicate that this is virtually impossible. Indeed, it’s perfectly possible for a species to evolve to extinction. Humanity may be finishing up the process right now. You can’t even praise evolution for the solicitous concern it shows for genes; the battle between two alternative alleles at the same location is a zero-sum game for frequency.

  Fitness is not always your friend.

  *

  136

  The Tragedy of Group Selectionism

  Before 1966, it was not unusual to see serious biologists advocating evolutionary hypotheses that we would now regard as magical thinking. These muddled notions played an important historical role in the development of later evolutionary theory, error calling forth correction; like the folly of English kings provoking into existence the Magna Carta and constitutional democracy.

  As an example of romance, Vero Wynne-Edwards, Warder Allee, and J. L. Brereton, among others, believed that predators would voluntarily restrain their breeding to avoid overpopulating their habitat and exhausting the prey population.

  But evolution does not open the floodgates to arbitrary purposes. You cannot explain a rattlesnake’s rattle by saying that it exists to benefit other animals who would otherwise be bitten. No outside Evolution Fairy decides when a gene ought to be promoted; the gene’s effect must somehow directly cause the gene to be more prevalent in the next generation. It’s clear why our human sense of aesthetics, witnessing a population crash of foxes who’ve eaten all the rabbits, cries “Something should’ve been done!” But how would a gene complex for restraining reproduction—of all things!—cause itself to become more frequent in the next generation?

  A human being designing a neat little toy ecology—for entertainment purposes, like a model railroad—might be annoyed if their painstakingly constructed fox and rabbit populations self-destructed by the foxes eating all the rabbits and then dying of starvation themselves. So the human would tinker with the toy ecology—a fox-breeding-restrainer is the obvious solution that leaps to our human minds—until the ecology looked nice and neat. Nature has no human, of course, but that needn’t stop us—now that we know what we want on aesthetic grounds, we just have to come up with a plausible argument that persuades Nature to want the same thing on evolutionary grounds.

  Obviously, selection on the level of the individual won’t produce individual restraint in breeding. Individuals who reproduce unrestrainedly will, naturally, produce more offspring than individuals who restrain themselves.

  (Individual selection will not produce individual sacrifice of breeding opportunities. Individual selection can certainly produce individuals who, after acquiring all available resources, use those resources to produce four big eggs instead of eight small eggs—not to conserve social resources, but because that is the individual sweet spot for (number of eggs)×(egg survival probability). This does not get rid of the commons problem.)

  But suppose that the species population was broken up into subpopulations, which were mostly isolated, and only occasionally interbred. Then, surely, subpopulations that restrained their breeding would be less likely to go extinct, and would send out more messengers, and create new colonies to reinhabit the territories of crashed populations.

  The problem with this scenario wasn’t that it was mathematically impossible. The problem was that it was possible but very difficult.

  The fundamental problem is that it’s not only restrained breeders who reap the benefits of restrained breeding. If some foxes refrain from spawning cubs who eat rabbits, then the uneaten rabbits don’t go to only cubs who carry the restrained-breeding adaptation. The unrestrained foxes, and their many more cubs, will happily eat any rabbits left unhunted. The only way the restraining gene can survive against this pressure, is if the benefits of restraint preferentially go to restrainers.

  Specifically, the requirement is C∕B < FST where C is the cost of altruism to the donor, B is the benefit of altruism to the recipient, and FST is the spatial structure of the population: the average relatedness between a randomly selected organism and its randomly selected neighbor, where a “neighbor” is any other fox who benefits from an altruistic fox’s restraint.1

  So is the cost of restrained breeding sufficiently small, and the empirical benefit of less famine sufficiently large, compared to the empirical spatial structure of fox populations and rabbit populations, that the group selection argument can work?

  The math suggests this is pretty unlikely. In this simulation, for example, the cost to altruists is 3% of fitness, pure altruist groups have a fitness twice as great as pure selfish groups, the subpopulation size is 25, and 20% of all deaths are replaced with messengers from another group: the result is polymorphic for selfishness and altruism. If the subpopulation size is doubled to 50, selfishness is fixed; if the cost to altruists is increased to 6%, selfishness is fixed; if the altruistic benefit is decreased by half, selfishness is fixed or in large majority. Neighborhood-groups must be very small, with only around 5 members, for group selection to operate when the cost of altruism exceeds 10%. This doesn’t seem plausibly true of foxes restraining t
heir breeding.

  You can guess by now, I think, that the group selectionists ultimately lost the scientific argument. The kicker was not the mathematical argument, but empirical observation: foxes didn’t restrain their breeding (I forget the exact species of dispute; it wasn’t foxes and rabbits), and indeed, predator-prey systems crash all the time. Group selectionism would later revive, somewhat, in drastically different form—mathematically speaking, there is neighborhood structure, which implies nonzero group selection pressure not necessarily capable of overcoming countervailing individual selection pressure, and if you don’t take it into account your math will be wrong, full stop. And evolved enforcement mechanisms (not originally postulated) change the game entirely. So why is this now-historical scientific dispute worthy material for Overcoming Bias?

  A decade after the controversy, a biologist had a fascinating idea. The mathematical conditions for group selection overcoming individual selection were too extreme to be found in Nature. Why not create them artificially, in the laboratory? Michael J. Wade proceeded to do just that, repeatedly selecting populations of insects for low numbers of adults per subpopulation.2 And what was the result? Did the insects restrain their breeding and live in quiet peace with enough food for all?

  No; the adults adapted to cannibalize eggs and larvae, especially female larvae.

  Of course selecting for small subpopulation sizes would not select for individuals who restrained their own breeding; it would select for individuals who ate other individuals’ children. Especially the girls.

  Once you have that experimental result in hand—and it’s massively obvious in retrospect—then it suddenly becomes clear how the original group selectionists allowed romanticism, a human sense of aesthetics, to cloud their predictions of Nature.

  This is an archetypal example of a missed Third Alternative, resulting from a rationalization of a predetermined bottom line that produced a fake justification and then motivatedly stopped. The group selectionists didn’t start with clear, fresh minds, happen upon the idea of group selection, and neutrally extrapolate forward the probable outcome. They started out with the beautiful idea of fox populations voluntarily restraining their reproduction to what the rabbit population would bear, Nature in perfect harmony; then they searched for a reason why this would happen, and came up with the idea of group selection; then, since they knew what they wanted the outcome of group selection to be, they didn’t look for any less beautiful and aesthetic adaptations that group selection would be more likely to promote instead. If they’d really been trying to calmly and neutrally predict the result of selecting for small subpopulation sizes resistant to famine, they would have thought of cannibalizing other organisms’ children or some similarly “ugly” outcome—long before they imagined anything so evolutionarily outré as individual restraint in breeding!

  This also illustrates the point I was trying to make in Einstein’s Arrogance: With large answer spaces, nearly all of the real work goes into promoting one possible answer to the point of being singled out for attention. If a hypothesis is improperly promoted to your attention—your sense of aesthetics suggests a beautiful way for Nature to be, and yet natural selection doesn’t involve an Evolution Fairy who shares your appreciation—then this alone may seal your doom, unless you can manage to clear your mind entirely and start over.

  In principle, the world’s stupidest person may say the Sun is shining, but that doesn’t make it dark out. Even if an answer is suggested by a lunatic on LSD, you should be able to neutrally calculate the evidence for and against, and if necessary, un-believe.

  In practice, the group selectionists were doomed because their bottom line was originally suggested by their sense of aesthetics, and Nature’s bottom line was produced by natural selection. These two processes had no principled reason for their outputs to correlate, and indeed they didn’t. All the furious argument afterward didn’t change that.

  If you start with your own desires for what Nature should do, consider Nature’s own observed reasons for doing things, and then rationalize an extremely persuasive argument for why Nature should produce your preferred outcome for Nature’s own reasons, then Nature, alas, still won’t listen. The universe has no mind and is not subject to clever political persuasion. You can argue all day why gravity should really make water flow uphill, and the water just ends up in the same place regardless. It’s like the universe plain isn’t listening. J. R. Molloy said: “Nature is the ultimate bigot, because it is obstinately and intolerantly devoted to its own prejudices and absolutely refuses to yield to the most persuasive rationalizations of humans.”

  I often recommend evolutionary biology to friends just because the modern field tries to train its students against rationalization, error calling forth correction. Physicists and electrical engineers don’t have to be carefully trained to avoid anthropomorphizing electrons, because electrons don’t exhibit mindish behaviors. Natural selection creates purposefulnesses which are alien to humans, and students of evolutionary theory are warned accordingly. It’s good training for any thinker, but it is especially important if you want to think clearly about other weird mindish processes that do not work like you do.

  *

  1. David Sloan Wilson, “A Theory of Group Selection,” Proceedings of the National Academy of Sciences of the United States of America 72, no. 1 (1975): 143–146.

  2. Michael J. Wade, “Group selections among laboratory populations of Tribolium,” Proceedings of the National Academy of Sciences of the United States of America 73, no. 12 (1976): 4604–4607, doi:10.1073/pnas.73.12.4604.

  137

  Fake Optimization Criteria

  I’ve previously dwelt in considerable length upon forms of rationalization whereby our beliefs appear to match the evidence much more strongly than they actually do. And I’m not overemphasizing the point, either. If we could beat this fundamental metabias and see what every hypothesis really predicted, we would be able to recover from almost any other error of fact.

  The mirror challenge for decision theory is seeing which option a choice criterion really endorses. If your stated moral principles call for you to provide laptops to everyone, does that really endorse buying a $1 million gem-studded laptop for yourself, or spending the same money on shipping 5,000 OLPCs?

  We seem to have evolved a knack for arguing that practically any goal implies practically any action. A phlogiston theorist explaining why magnesium gains weight when burned has nothing on an Inquisitor explaining why God’s infinite love for all His children requires burning some of them at the stake.

  There’s no mystery about this. Politics was a feature of the ancestral environment. We are descended from those who argued most persuasively that the good of the tribe meant executing their hated rival Uglak. (We sure ain’t descended from Uglak.)

  And yet . . . is it possible to prove that if Robert Mugabe cared only for the good of Zimbabwe, he would resign from its presidency? You can argue that the policy follows from the goal, but haven’t we just seen that humans can match up any goal to any policy? How do you know that you’re right and Mugabe is wrong? (There are a number of reasons this is a good guess, but bear with me here.)

  Human motives are manifold and obscure, our decision processes as vastly complicated as our brains. And the world itself is vastly complicated, on every choice of real-world policy. Can we even prove that human beings are rationalizing—that we’re systematically distorting the link from principles to policy—when we lack a single firm place on which to stand? When there’s no way to find out exactly what even a single optimization criterion implies? (Actually, you can just observe that people disagree about office politics in ways that strangely correlate to their own interests, while simultaneously denying that any such interests are at work. But again, bear with me here.)

  Where is the standardized, open-source, generally intelligent, consequentialist optimization process into which we can feed a complete morality as an XML file, to find out what that mora
lity really recommends when applied to our world? Is there even a single real-world case where we can know exactly what a choice criterion recommends? Where is the pure moral reasoner—of known utility function, purged of all other stray desires that might distort its optimization—whose trustworthy output we can contrast to human rationalizations of the same utility function?

  Why, it’s our old friend the alien god, of course! Natural selection is guaranteed free of all mercy, all love, all compassion, all aesthetic sensibilities, all political factionalism, all ideological allegiances, all academic ambitions, all libertarianism, all socialism, all Blue and all Green. Natural selection doesn’t maximize its criterion of inclusive genetic fitness—it’s not that smart. But when you look at the output of natural selection, you are guaranteed to be looking at an output that was optimized only for inclusive genetic fitness, and not the interests of the US agricultural industry.

  In the case histories of evolutionary science—in, for example, The Tragedy of Group Selectionism—we can directly compare human rationalizations to the result of pure optimization for a known criterion. What did Wynne-Edwards think would be the result of group selection for small subpopulation sizes? Voluntary individual restraint in breeding, and enough food for everyone. What was the actual laboratory result? Cannibalism.

  Now you might ask: Are these case histories of evolutionary science really relevant to human morality, which doesn’t give two figs for inclusive genetic fitness when it gets in the way of love, compassion, aesthetics, healing, freedom, fairness, et cetera? Human societies didn’t even have a concept of “inclusive genetic fitness” until the twentieth century.