Emergence

by Steven Johnson

  *

  You can think of Segel and Keller’s breakthrough as one of the first few stones to start tumbling at the outset of a landslide. Other stones were moving along with theirs—some of whose trajectories we’ll follow in the coming pages—but that initial movement was nothing compared to the avalanche that followed over the next two decades. At the end of its course, that landslide had somehow conjured up a handful of fully credited scientific disciplines, a global network of research labs and think tanks, and an entire patois of buzzwords. Thirty years after Keller challenged the pacemaker hypothesis, students now take courses in “self-organization studies,” and bottom-up software helps organize the Web’s most lively virtual communities. But Keller’s challenge did more than help trigger a series of intellectual trends. It also unearthed a secret history of decentralized thinking, a history that had been submerged for many years beneath the weight of the pacemaker hypothesis and the traditional boundaries of scientific research. People had been thinking about emergent behavior in all its diverse guises for centuries, if not millennia, but all that thinking had consistently been ignored as a unified body of work—because there was nothing unified about its body. There were isolated cells pursuing the mysteries of emergence, but no aggregation.

  Indeed, some of the great minds of the last few centuries—Adam Smith, Friedrich Engels, Charles Darwin, Alan Turing—contributed to the unknown science of self-organization, but because the science didn’t exist yet as a recognized field, their work ended up being filed on more familiar shelves. From a certain angle, those taxonomies made sense, because the leading figures of this new discipline didn’t even themselves realize that they were struggling to understand the laws of emergence. They were wrestling with local issues, in clearly defined fields: how ant colonies learn to forage and built nests; why industrial neighborhoods form along class lines; how our minds learn to recognize faces. You can answer all of these questions without resorting to the sciences of complexity and self-organization, but those answers all share a common pattern, as clear as the whorls of a fingerprint. But to see it as a pattern you needed to encounter it in several contexts. Only when the pattern was detected did people begin to think about studying self-organizing systems on their own merits. Keller and Segel saw it in the slime mold assemblages; Jane Jacobs saw it in the formation of city neighborhoods; Marvin Minsky in the distributed networks of the human brain.

  What features do all these systems share? In the simplest terms, they solve problems by drawing on masses of relatively stupid elements, rather than a single, intelligent “executive branch.” They are bottom-up systems, not top-down. They get their smarts from below. In a more technical language, they are complex adaptive systems that display emergent behavior. In these systems, agents residing on one scale start producing behavior that lies one scale above them: ants create colonies; urbanites create neighborhoods; simple pattern-recognition software learns how to recommend new books. The movement from low-level rules to higher-level sophistication is what we call emergence.

  Imagine a billiard table populated by semi-intelligent, motorized billiard balls that have been programmed to explore the space of the table and alter their movement patterns based on specific interactions with other balls. For the most part, the table is in permanent motion, with balls colliding constantly, switching directions and speed every second. Because they are motorized, they never slow down unless their rules instruct them to, and their programming enables them to take unexpected turns when they encounter other balls. Such a system would define the most elemental form of complex behavior: a system with multiple agents dynamically interacting in multiple ways, following local rules and oblivious to any higher-level instructions. But it wouldn’t truly be considered emergent until those local interactions resulted in some kind of discernible macrobehavior. Say the local rules of behavior followed by the balls ended up dividing the table into two clusters of even-numbered and odd-numbered balls. That would mark the beginnings of emergence, a higher-level pattern arising out of parallel complex interactions between local agents. The balls aren’t programmed explicitly to cluster in two groups; they’re programmed to follow much more random rules: swerve left when they collide with a solid-colored; accelerate after contact with the three ball; stop dead in their tracks when they hit the eight ball; and so on. Yet out of those low-level routines, a coherent shape emerges.

  Does that make our mechanized billiard table adaptive? Not really, because a table divided between two clusters of balls is not terribly useful, either to the billiard balls themselves or to anyone else in the pool hall. But, like the proverbial Hamlet-writing monkeys, if we had an infinite number of tables in our pool hall, each following a different set of rules, one of those tables might randomly hit upon a rule set that would arrange all the balls in a perfect triangle, leaving the cue ball across the table ready for the break. That would be adaptive behavior in the larger ecosystem of the pool hall, assuming that it was in the interest of our billiards system to attract players. The system would use local rules between interacting agents to create higher-level behavior well suited to its environment.

  Emergent complexity without adaptation is like the intricate crystals formed by a snowflake: it’s a beautiful pattern, but it has no function. The forms of emergent behavior that we’ll examine in this book show the distinctive quality of growing smarter over time, and of responding to the specific and changing needs of their environment. In that sense, most of the systems we’ll look at are more dynamic than our adaptive billiards table: they rarely settle in on a single, frozen shape; they form patterns in time as well as space. A better example might be a table that self-organizes into a billiards-based timing device: with the cue ball bouncing off the eight ball sixty times a minute, and the remaining balls shifting from one side of the table to another every hour on the hour. That might sound like an unlikely system to emerge out of local interactions between individual balls, but your body contains numerous organic clocks built out of simple cells that function in remarkably similar ways. An infinite number of cellular or billiard-ball configurations will not produce a working clock, and only a tiny number will. So the question becomes, how do you push your emergent system toward clocklike behavior, if that’s your goal? How do you make a self-organizing system more adaptive?

  That question has become particularly crucial, because the history of emergence has entered a new phase in the past few years, one that should prove to be more revolutionary than the two phases before it. In the first phase, inquiring minds struggled to understand the forces of self-organization without realizing what they were up against. In the second, certain sectors of the scientific community began to see self-organization as a problem that transcended local disciplines and set out to solve that problem, partially by comparing behavior in one area to behavior in another. By watching the slime mold cells next to the ant colonies, you could see the shared behavior in ways that would have been unimaginable watching either on its own. Self-organization became an object of study in its own right, leading to the creation of celebrated research centers such as the Santa Fe Institute, which devoted itself to the study of complexity in all its diverse forms.

  But in the third phase—the one that began sometime in the past decade, the one that lies at the very heart of this book—we stopped analyzing emergence and started creating it. We began building self-organizing systems into our software applications, our video games, our art, our music. We built emergent systems to recommend new books, recognize our voices, or find mates. For as long as complex organisms have been alive, they have lived under the laws of self-organization, but in recent years our day-to-day life has become overrun with artificial emergence: systems built with a conscious understanding of what emergence is, systems designed to exploit those laws the same way our nuclear reactors exploit the laws of atomic physics. Up to now, the philosophers of emergence have struggled to interpret the world. But they are now starting to change it.

  *
<
br />   What follows is a tour of fields that aren’t usually gathered between the same book jacket covers. We’ll look at computer games that simulate living ecologies; the guild system of twelfth-century Florence; the initial cell divisions that mark the very beginning of life; and software that lets you see the patterns of your own brain. What unites these different phenomena is a recurring pattern and shape: a network of self-organization, of disparate agents that unwittingly create a higher-level order. At each scale, you can see the imprint of those slime mold cells converging; at each scale, the laws of emergence hold true.

  This book roughly follows the chronology of the three historical phases. The first section introduces one of the emergent world’s crowning achievements—the colony behavior of social insects such as ants and termites—and then goes back to trace part of the history of the decentralized mind-set, from Engels on the streets of Manchester to the new forms of emergent software being developed today. The second section is an overview of emergence as we currently understand it; each of the four chapters in the section explores one of the field’s core principles: neighbor interaction, pattern recognition, feedback, and indirect control. The final section looks to the future of artificial emergence and speculates on what will happen when our media experiences and political movements are largely shaped by bottom-up forces, and not top-down ones.

  *

  Certain shapes and patterns hover over different moments in time, haunting and inspiring the individuals living through those periods. The epic clash and subsequent resolution of the dialectic animated the first half of the nineteenth century; the Darwinian and social reform movements scattered web imagery through the second half of the century. The first few decades of the twentieth century found their ultimate expression in the exuberant anarchy of the explosion, while later decades lost themselves in the faceless regimen of the grid. You can see the last ten years or so as a return to those Victorian webs, though I suspect the image that has been burned into our retinas over the past decade is more prosaic: windows piled atop one another on a screen, or perhaps a mouse clicking on an icon.

  These shapes are shorthand for a moment in time, a way of evoking an era and its peculiar obsessions. For individuals living within these periods, the shapes are cognitive building blocks, tools for thought: Charles Darwin and George Eliot used the web as a way of understanding biological evolution and social struggles; a half century later, the futurists embraced the explosions of machine-gun fire, while Picasso used them to re-create the horrors of war in Guernica. The shapes are a way of interpreting the world, and while no shape completely represents its epoch, they are an undeniable component of the history of thinking.

  When I imagine the shape that will hover above the first half of the twenty-first century, what comes to mind is not the coiled embrace of the genome, or the etched latticework of the silicon chip. It is instead the pulsing red and green pixels of Mitch Resnick’s slime mold simulation, moving erratically across the screen at first, then slowly coalescing into larger forms. The shape of those clusters—with their lifelike irregularity, and their absent pacemakers—is the shape that will define the coming decades. I see them on the screen, growing and dividing, and I think: That way lies the future.

  PART ONE

  African anthill (Courtesy of Corbis)

  Rise up, thou monstrous anthill on the plain

  Of a too busy world! Before me flow,

  Thou endless stream of men and moving things!

  Thy everyday appearance, as it strikes—

  With wonder heightened, or sublimed by awe—

  On strangers, of all ages; the quick dance

  Of colours, lights, and forms; the deafening din;

  The comers and the goers face to face,

  Face after face …

  —WORDSWORTH,

  “RESIDENCE IN LONDON”

  Cities have no central planning commissions that solve the problem of purchasing and distributing supplies… . How do these cities avoid devastating swings between shortage and glut, year after year, decade after decade? The mystery deepens when we observe the kaleidoscopic nature of large cities. Buyers, sellers, administrations, streets, bridges, and buildings are always changing, so that a city’s coherence is somehow imposed on a perpetual flux of people and structures. Like the standing wave in front of a rock in a fast-moving stream, a city is a pattern in time.

  —JOHN HOLLAND

  1

  The Myth of the Ant Queen

  It’s early fall in Palo Alto, and Deborah Gordon and I are sitting in her office in Stanford’s Gilbert Biological Sciences building, where she spends three-quarters of the year studying behavioral ecology. The other quarter is spent doing fieldwork with the native harvester ants of the American Southwest, and when we meet, her face still retains the hint of a tan from her last excursion to the Arizona desert.

  I’ve come here to learn more about the collective intelligence of ant colonies. Gordon, dressed neatly in a white shirt, cheerfully entertains a few borderline-philosophical questions on group behavior and complex systems, but I can tell she’s hankering to start with a hands-on display. After a few minutes of casual rumination, she bolts up out of her chair. “Why don’t we start with me showing you the ants that we have here,” she says. “And then we can talk about what it all means.”

  She ushers me into a sepulchral room across the hallway, where three long tables are lined up side by side. The initial impression is that of an underpopulated and sterilized pool hall, until I get close enough to one of the tables to make out the miniature civilization that lives within each of them. Closer to a Habitrail than your traditional idea of an ant farm, Gordon’s contraptions house an intricate network of plastic tubes connecting a dozen or so plastic boxes, each lined with moist plaster and coated with a thin layer of dirt.

  “We cover the nests with red plastic because some species of ants don’t see red light,” Gordon explains. “That seems to be true of this species too.” For a second, I’m not sure what she means by “this species”—and then my eyes adjust to the scene, and I realize with a start that the dirt coating the plastic boxes is, in fact, thousands of harvester ants, crammed so tightly into their quarters that I had originally mistaken them for an undifferentiated mass. A second later, I can see that the whole simulated colony is wonderfully alive, the clusters of ants pulsing steadily with movement. The tubing and cramped conditions and surging crowds bring one thought immediately to mind: the New York subway system, rush hour.

  At the heart of Gordon’s work is a mystery about how ant colonies develop, a mystery that has implications extending far beyond the parched earth of the Arizona desert to our cities, our brains, our immune systems—and increasingly, our technology. Gordon’s work focuses on the connection between the microbehavior of individual ants and the overall behavior of the colonies themselves, and part of that research involves tracking the life cycles of individual colonies, following them year after year as they scour the desert floor for food, competing with other colonies for territory, and—once a year—mating with them. She is a student, in other words, of a particular kind of emergent, self-organizing system.

  Dig up a colony of native harvester ants and you’ll almost invariably find that the queen is missing. To track down the colony’s matriarch, you need to examine the bottom of the hole you’ve just dug to excavate the colony: you’ll find a narrow, almost invisible passageway that leads another two feet underground, to a tiny vestibule burrowed out of the earth. There you will find the queen. She will have been secreted there by a handful of ladies-in-waiting at the first sign of disturbance. That passageway, in other words, is an emergency escape hatch, not unlike a fallout shelter buried deep below the West Wing.

  But despite the Secret Service–like behavior, and the regal nomenclature, there’s nothing hierarchical about the way an ant colony does its thinking. “Although queen is a term that reminds us of human political systems,” Gordon explains, “the queen is not a
n authority figure. She lays eggs and is fed and cared for by the workers. She does not decide which worker does what. In a harvester ant colony, many feet of intricate tunnels and chambers and thousands of ants separate the queen, surrounded by interior workers, from the ants working outside the nest and using only the chambers near the surface. It would be physically impossible for the queen to direct every worker’s decision about which task to perform and when.” The harvester ants that carry the queen off to her escape hatch do so not because they’ve been ordered to by their leader; they do it because the queen ant is responsible for giving birth to all the members of the colony, and so it’s in the colony’s best interest—and the colony’s gene pool—to keep the queen safe. Their genes instruct them to protect their mother, the same way their genes instruct them to forage for food. In other words, the matriarch doesn’t train her servants to protect her, evolution does.

  Popular culture trades in Stalinist ant stereotypes—witness the authoritarian colony regime in the animated film Antz—but in fact, colonies are the exact opposite of command economies. While they are capable of remarkably coordinated feats of task allocation, there are no Five-Year Plans in the ant kingdom. The colonies that Gordon studies display some of nature’s most mesmerizing decentralized behavior: intelligence and personality and learning that emerges from the bottom up.

  I’m still gazing into the latticework of plastic tubing when Gordon directs my attention to the two expansive white boards attached to the main colony space, one stacked on top of the other and connected by a ramp. (Imagine a two-story parking garage built next to a subway stop.) A handful of ants meander across each plank, some porting crumblike objects on their back, others apparently just out for a stroll. If this is the Central Park of Gordon’s ant metropolis, I think, it must be a workday.