Rationality- From AI to Zombies

by Eliezer Yudkowsky

  134

  No Evolutions for Corporations or Nanodevices

  The laws of physics and the rules of math don’t cease to apply. That leads me to believe that evolution doesn’t stop. That further leads me to believe that nature—bloody in tooth and claw, as some have termed it—will simply be taken to the next level . . .

  [Getting rid of Darwinian evolution is] like trying to get rid of gravitation. So long as there are limited resources and multiple competing actors capable of passing on characteristics, you have selection pressure.

  —Perry Metzger, predicting that the reign of natural selection would continue into the indefinite future

  In evolutionary biology, as in many other fields, it is important to think quantitatively rather than qualitatively. Does a beneficial mutation “sometimes spread, but not always”? Well, a psychic power would be a beneficial mutation, so you’d expect it to spread, right? Yet this is qualitative reasoning, not quantitative—if X is true, then Y is true; if psychic powers are beneficial, they may spread. In Evolutions Are Stupid, I described the equations for a beneficial mutation’s probability of fixation, roughly twice the fitness advantage (6% for a 3% advantage). Only this kind of numerical thinking is likely to make us realize that mutations which are only rarely useful are extremely unlikely to spread, and that it is practically impossible for complex adaptations to arise without constant use. If psychic powers really existed, we should expect to see everyone using them all the time—not just because they would be so amazingly useful, but because otherwise they couldn’t have evolved in the first place.

  “So long as there are limited resources and multiple competing actors capable of passing on characteristics, you have selection pressure.” This is qualitative reasoning. How much selection pressure?

  While there are several candidates for the most important equation in evolutionary biology, I would pick Price’s Equation, which in its simplest formulation reads:

  ▵cov(vi, zi)

  change in average characteristic = covariance(relative fitness, characteristic).

  This is a very powerful and general formula. For example, a particular gene for height can be the Z, the characteristic that changes, in which case Price’s Equation says that the change in the probability of possessing this gene equals the covariance of the gene with reproductive fitness. Or you can consider height in general as the characteristic Z, apart from any particular genes, and Price’s Equation says that the change in height in the next generation will equal the covariance of height with relative reproductive fitness.

  (At least, this is true so long as height is straightforwardly heritable. If nutrition improves, so that a fixed genotype becomes taller, you have to add a correction term to Price’s Equation. If there are complex nonlinear interactions between many genes, you have to either add a correction term, or calculate the equation in such a complicated way that it ceases to enlighten.)

  Many enlightenments may be attained by studying the different forms and derivations of Price’s Equation. For example, the final equation says that the average characteristic changes according to its covariance with relative fitness, rather than its absolute fitness. This means that if a Frodo gene saves its whole species from extinction, the average Frodo characteristic does not increase, since Frodo’s act benefited all genotypes equally and did not covary with relative fitness.

  It is said that Price became so disturbed with the implications of his equation for altruism that he committed suicide, though he may have had other issues. (Overcoming Bias does not advocate committing suicide after studying Price’s Equation.)

  One of the enlightenments which may be gained by meditating upon Price’s Equation is that “limited resources” and “multiple competing actors capable of passing on characteristics” are not sufficient to give rise to an evolution. “Things that replicate themselves” is not a sufficient condition. Even “competition between replicating things” is not sufficient.

  Do corporations evolve? They certainly compete. They occasionally spin off children. Their resources are limited. They sometimes die.

  But how much does the child of a corporation resemble its parents? Much of the personality of a corporation derives from key officers, and CEOs cannot divide themselves by fission. Price’s Equation only operates to the extent that characteristics are heritable across generations. If great-great-grandchildren don’t much resemble their great-great-grandparents, you won’t get more than four generations’ worth of cumulative selection pressure—anything that happened more than four generations ago will blur itself out. Yes, the personality of a corporation can influence its spinoff—but that’s nothing like the heritability of DNA, which is digital rather than analog, and can transmit itself with 10-8 errors per base per generation.

  With DNA you have heritability lasting for millions of generations. That’s how complex adaptations can arise by pure evolution—the digital DNA lasts long enough for a gene conveying 3% advantage to spread itself over 768 generations, and then another gene dependent on it can arise. Even if corporations replicated with digital fidelity, they would currently be at most ten generations into the RNA World.

  Now, corporations are certainly selected, in the sense that incompetent corporations go bust. This should logically make you more likely to observe corporations with features contributing to competence. And in the same sense, any star that goes nova shortly after it forms, is less likely to be visible when you look up at the night sky. But if an accident of stellar dynamics makes one star burn longer than another star, that doesn’t make it more likely that future stars will also burn longer—the feature will not be copied onto other stars. We should not expect future astrophysicists to discover complex internal features of stars which seem designed to help them burn longer. That kind of mechanical adaptation requires much larger cumulative selection pressures than a once-off winnowing.

  Think of the principle introduced in Einstein’s Arrogance—that the vast majority of the evidence required to think of General Relativity had to go into raising that one particular equation to the level of Einstein’s personal attention; the amount of evidence required to raise it from a deliberately considered possibility to 99.9% certainty was trivial by comparison. In the same sense, complex features of corporations that require hundreds of bits to specify are produced primarily by human intelligence, not a handful of generations of low-fidelity evolution. In biology, the mutations are purely random and evolution supplies thousands of bits of cumulative selection pressure. In corporations, humans offer up thousand-bit intelligently designed complex “mutations,” and then the further selection pressure of “Did it go bankrupt or not?” accounts for a handful of additional bits in explaining what you see.

  Advanced molecular nanotechnology—the artificial sort, not biology—should be able to copy itself with digital fidelity through thousands of generations. Would Price’s Equation thereby gain a foothold?

  Correlation is covariance divided by variance, so if A is highly predictive of B, there can be a strong “correlation” between them even if A is ranging from 0 to 9 and B is only ranging from 50.0001 and 50.0009. Price’s Equation runs on covariance of characteristics with reproduction—not correlation! If you can compress variance in characteristics into a tiny band, the covariance goes way down, and so does the cumulative change in the characteristic.

  The Foresight Institute suggests, among other sensible proposals, that the replication instructions for any nanodevice should be encrypted. Moreover, encrypted such that flipping a single bit of the encoded instructions will entirely scramble the decrypted output. If all nanodevices produced are precise molecular copies, and moreover, any mistakes on the assembly line are not heritable because the offspring got a digital copy of the original encrypted instructions for use in making grandchildren, then your nanodevices ain’t gonna be doin’ much evolving.

  You’d still have to worry about prions—self-replicating assembly errors apart from the encrypted instructions, whe
re a robot arm fails to grab a carbon atom that is used in assembling a homologue of itself, and this causes the offspring’s robot arm to likewise fail to grab a carbon atom, etc., even with all the encrypted instructions remaining constant. But how much correlation is there likely to be, between this sort of transmissible error, and a higher reproductive rate? Let’s say that one nanodevice produces a copy of itself every 1,000 seconds, and the new nanodevice is magically more efficient (it not only has a prion, it has a beneficial prion) and copies itself every 999.99999 seconds. It needs one less carbon atom attached, you see. That’s not a whole lot of variance in reproduction, so it’s not a whole lot of covariance either.

  And how often will these nanodevices need to replicate? Unless they’ve got more atoms available than exist in the solar system, or for that matter, the visible Universe, only a small number of generations will pass before they hit the resource wall. “Limited resources” are not a sufficient condition for evolution; you need the frequently iterated death of a substantial fraction of the population to free up resources. Indeed, “generations” is not so much an integer as an integral over the fraction of the population that consists of newly created individuals.

  This is, to me, the most frightening thing about gray goo or nanotechnological weapons—that they could eat the whole Earth and then that would be it, nothing interesting would happen afterward. Diamond is stabler than proteins held together by van der Waals forces, so the goo would only need to reassemble some pieces of itself when an asteroid hit. Even if prions were a powerful enough idiom to support evolution at all—evolution is slow enough with digital DNA!—fewer than 1.0 generations might pass between when the goo ate the Earth and when the Sun died.

  To sum up, if you have all of the following properties:

  Entities that replicate;

  Substantial variation in their characteristics;

  Substantial variation in their reproduction;

  Persistent correlation between the characteristics and reproduction;

  High-fidelity long-range heritability in characteristics;

  Frequent birth of a significant fraction of the breeding population;

  And all this remains true through many iterations . . .

  Then you will have significant cumulative selection pressures, enough to produce complex adaptations by the force of evolution.

  *

  135

  Evolving to Extinction

  It is a very common misconception that an evolution works for the good of its species. Can you remember hearing someone talk about two rabbits breeding eight rabbits and thereby “contributing to the survival of their species”? A modern evolutionary biologist would never say such a thing; they’d sooner breed with a rabbit.

  It’s yet another case where you’ve got to simultaneously consider multiple abstract concepts and keep them distinct. Evolution doesn’t operate on particular individuals; individuals keep whatever genes they’re born with. Evolution operates on a reproducing population, a species, over time. There’s a natural tendency to think that if an Evolution Fairy is operating on the species, she must be optimizing for the species. But what really changes are the gene frequencies, and frequencies don’t increase or decrease according to how much the gene helps the species as a whole. As we shall later see, it’s quite possible for a species to evolve to extinction.

  Why are boys and girls born in roughly equal numbers? (Leaving aside crazy countries that use artificial gender selection technologies.) To see why this is surprising, consider that 1 male can impregnate 2, 10, or 100 females; it wouldn’t seem that you need the same number of males as females to ensure the survival of the species. This is even more surprising in the vast majority of animal species where the male contributes very little to raising the children—humans are extraordinary, even among primates, for their level of paternal investment. Balanced gender ratios are found even in species where the male impregnates the female and vanishes into the mist.

  Consider two groups on different sides of a mountain; in group A, each mother gives birth to 2 males and 2 females; in group B, each mother gives birth to 3 females and 1 male. Group A and group B will have the same number of children, but group B will have 50% more grandchildren and 125% more great-grandchildren. You might think this would be a significant evolutionary advantage.

  But consider: The rarer males become, the more reproductively valuable they become—not to the group, but to the individual parent. Every child has one male and one female parent. Then in every generation, the total genetic contribution from all males equals the total genetic contribution from all females. The fewer males, the greater the individual genetic contribution per male. If all the females around you are doing what’s good for the group, what’s good for the species, and birthing 1 male per 10 females, you can make a genetic killing by birthing all males, each of whom will have (on average) ten times as many grandchildren as their female cousins.

  So while group selection ought to favor more girls, individual selection favors equal investment in male and female offspring. Looking at the statistics of a maternity ward, you can see at a glance that the quantitative balance between group selection forces and individual selection forces is overwhelmingly tilted in favor of individual selection in Homo sapiens.

  (Technically, this isn’t quite a glance. Individual selection favors equal parental investments in male and female offspring. If males cost half as much to birth and/or raise, twice as many males as females will be born at the evolutionarily stable equilibrium. If the same number of males and females were born in the population at large, but males were twice as cheap to birth, then you could again make a genetic killing by birthing more males. So the maternity ward should reflect the balance of parental opportunity costs, in a hunter-gatherer society, between raising boys and raising girls; and you’d have to assess that somehow. But ya know, it doesn’t seem all that much more reproductive-opportunity-costly for a hunter-gatherer family to raise a girl, so it’s kinda suspicious that around the same number of boys are born as girls.)

  Natural selection isn’t about groups, or species, or even individuals. In a sexual species, an individual organism doesn’t evolve; it keeps whatever genes it’s born with. An individual is a once-off collection of genes that will never reappear; how can you select on that? When you consider that nearly all of your ancestors are dead, it’s clear that “survival of the fittest” is a tremendous misnomer. “Replication of the fitter” would be more accurate, although technically fitness is defined only in terms of replication.

  Natural selection is really about gene frequencies. To get a complex adaptation, a machine with multiple dependent parts, each new gene as it evolves depends on the other genes being reliably present in its genetic environment. They must have high frequencies. The more complex the machine, the higher the frequencies must be. The signature of natural selection occurring is a gene rising from 0.00001% of the gene pool to 99% of the gene pool. This is the information, in an information-theoretic sense; and this is what must happen for large complex adaptations to evolve.

  The real struggle in natural selection is not the competition of organisms for resources; this is an ephemeral thing when all the participants will vanish in another generation. The real struggle is the competition of alleles for frequency in the gene pool. This is the lasting consequence that creates lasting information. The two rams bellowing and locking horns are only passing shadows.

  It’s perfectly possible for an allele to spread to fixation by outcompeting an alternative allele which was “better for the species.” If the Flying Spaghetti Monster magically created a species whose gender mix was perfectly optimized to ensure the survival of the species—the optimal gender mix to bounce back reliably from near-extinction events, adapt to new niches, et cetera—then the evolution would rapidly degrade this species optimum back into the individual-selection optimum of equal parental investment in males and females.

  Imagine a “Frodo gene” that sacrifices
its vehicle to save its entire species from an extinction event. What happens to the allele frequency as a result? It goes down. Kthxbye.

  If species-level extinction threats occur regularly (call this a “Buffy environment”) then the Frodo gene will systematically decrease in frequency and vanish, and soon thereafter, so will the species.

  A hypothetical example? Maybe. If the human species was going to stay biological for another century, it would be a good idea to start cloning Gandhi.

  In viruses, there’s the tension between individual viruses replicating as fast as possible, versus the benefit of leaving the host alive long enough to transmit the illness. This is a good real-world example of group selection, and if the virus evolves to a point on the fitness landscape where the group selection pressures fail to overcome individual pressures, the virus could vanish shortly thereafter. I don’t know if a disease has ever been caught in the act of evolving to extinction, but it’s probably happened any number of times.

  Segregation-distorters subvert the mechanisms that usually guarantee fairness of sexual reproduction. For example, there is a segregation-distorter on the male sex chromosome of some mice which causes only male children to be born, all carrying the segregation-distorter. Then these males impregnate females, who give birth to only male children, and so on. You might cry “This is cheating!” but that’s a human perspective; the reproductive fitness of this allele is extremely high, since it produces twice as many copies of itself in the succeeding generation as its nonmutant alternative. Even as females become rarer and rarer, males carrying this gene are no less likely to mate than any other male, and so the segregation-distorter remains twice as fit as its alternative allele. It’s speculated that real-world group selection may have played a role in keeping the frequency of this gene as low as it seems to be. In which case, if mice were to evolve the ability to fly and migrate for the winter, they would probably form a single reproductive population, and would evolve to extinction as the segregation-distorter evolved to fixation.