Emergence

by Steven Johnson

  That promises a genuine revolution in what it means to be a media consumer, but it also demands a comparable revolution in the way businesses work. No company has more thoroughly explored the commercial possibilities of clusters and bottom-up organization than the celebrated auction site eBay. Since its launch in 1995, the site has been a virtual laboratory for experiments with clusters and self-regulating feedback. The “news” on eBay is almost entirely generated by the users of the service, and by the collective behavior of specific groups of users. The top auctions, the highly rated buyers-and-sellers lists, the user feedback, the communities formed around specific categories like Stamp Collecting or Consumer Electronics, the regional filters, the lists of new offerings from people you’ve bought from before—all of these are attempts to make patterns of group behavior transparent to individual users, the way a city neighborhood makes comparable patterns visible to its residents. EBay’s founder, Pierre Omidyar, originally created the site to enable his wife to trade Pez Dispensers with other Pez fanatics worldwide; six years later the site harbors thousands of similar microcommunities, united by shared interests. If eBay had restricted itself to showcasing the collector’s items that happened to be in vogue that month—Beanie Babies or PlayStation 2—the results wouldn’t have looked all that different from your traditional shopping mall. But they wisely allowed their site to splinter into thousands of smaller clusters, like little eddies in the group current of their customer activity. Skeptics used to argue that online auctions would never become a mainstream activity because the electronic medium would make it easy for scam artists to sell bogus merchandise. Those critics wildly underestimated the extent to which software can create self-regulating systems, systems that separate the scoundrels from the honest dealers, the way Slashdot’s quality filters separated quality from crap. Every seller on eBay has a public history of past deals; scam one buyer with a fake or broken item, and your reputation can be ruined forever. Like the public safety of Jacob’s sidewalks, the eBay population polices itself with almost unbelievable efficiency, which is why the site now attracts more than 30 million users. And unlike almost any other Web-based commerce site, eBay has been consistently profitable from its early days. The history of self-organizing clusters includes the silk and fine linens bought and sold on Florence’s Por Santa Maria or London’s Savile Row. But don’t underestimate the significance of Pez.

  Still, if eBay is a model for the way bottom-up systems can transform the relationship between buyer and seller, can the principles of emergence be usefully applied to the internal structure of organizations? Is it possible to build corporate systems that are more like ant colonies than command economies? Marketplaces—even those dominated by global megacorporations—tend to work in decentralized ways, but the internal structures of most corporations today rely on org charts that look more like feudal states than slime molds. The market may be bottom-up, but it is populated by chronically top-heavy agents. Decentralized production and development have done wonders for the world of Open Source software, where certain fundamental rights of ownership have been disavowed, but it remains a real question whether the more proprietary wing of late capitalism can model its internal organization after ant farms or neural nets. For one, the unpredictability of emergent systems makes them an ideal platform for book recommending or gameplaying, but no one wants a business that might spontaneously fire a phalanx of middle managers for no discernible reason. Controlled randomness is a brilliant recipe for city life and ant foraging, but it’s harder to imagine selling shareholders on it as a replacement for the CEO. Software designers like Danny Hillis or Oliver Selfridge leaned on evolutionary techniques to rein in their systems and to force them toward specific goals. But evolution requires many parallel generations to do its handiwork; no investor wants to wait around while her investment breeds a long-term strategy out of a million random business plans.

  Still, emergent systems can be brilliant innovators, and they tend to be more adaptable to sudden change than more rigid hierarchical models. Those qualities make the principles of bottom-up intelligence tantalizing ones for businesses struggling to keep up with the twenty-first-century rate of change. A number of companies, concentrated mostly in the high-tech industry, have experimented with neural-net-like organizational structures, breaking up the traditional system of insular and hierarchical departments and building a more cellular, distributed network of small units, usually about a dozen people in size. Units can assemble into larger clusters if they need to, and those clusters have the power to set their own objectives. The role of traditional senior management grows less important in these models—less concerned with establishing a direction for the company, and more involved with encouraging the clusters that generate the best ideas. Imagine a corporate system structured like the Slashdot quality filters: in a traditional company, the CEO composes the posts himself; in a Slashdot-style company, he’s merely tweaking the algorithm that promotes or demotes posts based on their quality. The vision for the company’s future comes from below, out of the ever-shifting alliances of smaller groups. Senior management simply provides the feedback mechanism—in the form of bonuses, options, or increased resources—ensuring that the most productive clusters thrive. CEOs still have a place in even the most distributed corporate structure, but they’re no longer allowed to be pacemakers. The Australian software company TCG, the Taiwanese Acer Group, and Sun Microsystems have all implemented cellular techniques with positive results. There’s even a management-theory journal devoted to these developing models. It is called, appropriately enough, Emergence.

  *

  If decentralized intelligence can transform the way businesses work, what can it do for politics? That many New Economy companies have been so quick to embrace the emergent worldview—in both their products and their internal structure—can sometimes make it seem as though emergence belongs squarely to the libertarian camp. Certainly the emphasis on local control and the resistance to command systems resonates with the Gingrichian call for anti-big-government devolution. But the politics of emergence are not so readily classified. The intelligence of ant colonies may be the animal kingdom’s most compelling argument for the power of the collective, and you can think of “local knowedge” as another way of talking about grassroots struggle. The libertarian right likes to rail against the centralized authority of the state, but at least most politicians in the world today are democratically elected, unlike the executives of most multinational corporations. The public sector has no monopoly on top-down systems, and there’s no reason why progressives shouldn’t also embrace decentralized strategies, even if those same strategies are being explored by right-wing think tanks and dot-coms. In fact, the needs of most progressive movements are uniquely suited to adaptive, self-organizing systems: both have a keen ear for collective wisdom; both are naturally hostile to excessive concentrations of power; and both are friendly to change. For any movement that aims to be truly global in scope, making it almost impossible to rely on centralized power, adaptive self-organization may well be the only road available.

  Nowhere are the progressive possibilities of emergence more readily apparent than in the anti-WTO protest movements, which have explicitly modeled themselves after the distributed, cellular structures of self-organizing systems. The Seattle protests of 1999 were characterized by an extraordinary form of distributed organization: smaller affinity groups representing specific causes—anti-Nike critics, anarchists, radical environmentalists, labor unions—would operate independently for much of the time, only coming together for occasional “spokescouncil” meetings, where each group would elect a single member to represent their interests. As Naomi Klein reported in The Nation, “At some rallies activists carry actual cloth webs to symbolize their movement. When it’s time for a meeting, they lay the web on the ground, call out ‘All spokes on the web,’ and the structure becomes a street-level boardroom.” To some older progressives, steeped in the more hierarchical tradition of past l
abor movements, those diverse “affinity groups” seemed hopelessly scattered and unfocused, with no common language or ideology uniting them. It’s almost impossible to think of another political movement that generated as much public attention without producing a genuine leader—a Jesse Jackson or Cesar Chavez—if only for the benefit of the television cameras. The images that we associate with the antiglobalization protests are never those of an adoring crowd raising their fists in solidarity with an impassioned speaker on a podium. That is the iconography of an earlier model of protest. What we see again and again with the new wave are images of disparate groups: satirical puppets, black-clad anarchists, sit-ins and performance art—but no leaders. To old-school progressives, the Seattle protesters appeared to be headless, out of control, a swarm of small causes with no organizing principle—and to a certain extent they’re right in their assessment. What they fail to recognize is that there can be power and intelligence in a swarm, and if you’re trying to do battle against a distributed network like global capitalism, you’re better off becoming a distributed network yourself.

  That is not a reason to embrace pure anarchy, of course. Ant colonies do not have leaders in any real sense, but they do rely heavily on rules: how to read patterns in the pheromone trail, when to change from foraging to nest-building, how to respond to other ants, and so on. An ant colony without local rules has no chance of creating a higher-level order, no chance of creating a collective intelligence. The antiglobalization movements are only beginning to figure out the proper rules for engagement between different cells. The spokescouncils of Seattle were a promising start, but learning how to cluster takes time. Klein writes, “What emerged on the streets of Seattle and Washington was an activist model that mirrors the organic, interlinked pathways of the Internet.” But as we’ve seen countless times over the preceding pages, even the Web itself—the largest and most advanced man-made self-organizing system on the planet—is only now becoming capable of true collective intelligence. By any measure, the Web’s mind-reading skills are embryonic at best, because we are still tweaking the rules of the system, still fiddling with how adaptive and intelligent clusters can prosper online. And if the Web’s collective intelligence is still in its infancy, think of how much room the new protest movements must have to grow. But thus far, their instincts have been sound ones. Beneath the window-smashing and the Rage Against the Machine concerts, the anti-WTO activists are doing something profound, even in these early days of their movement. They are thinking like a swarm.

  7

  See What Happens

  For years mathematicians have puzzled over a classic brainteaser known as the “traveling salesman problem.” Imagine you’re a salesman who has to visit fifteen cities during a business trip—cities that are distributed semirandomly across the map. What is the shortest route that takes you to each city exactly once? It sounds like a simple enough question, but the answer is maddeningly difficult to establish. Even with the number of cities set at a relatively modest fifteen, billions of potential routes exist for our traveling salesman. For complicated reasons, the traveling salesman problem is almost impossible to solve definitively, and so historically mathematicians—and traveling salesmen, presumably—have settled for the next best thing: routes that are tolerably short, but not necessarily the shortest possible.

  This might sound like an arcane issue, given the real-world decline of the traveling salesman, but the core elements of the problem lie at the epicenter of the communications revolution. Think of those traveling salesmen as bits of data, and the cities as Web servers and routers scattered all across the globe. Being able to calculate the shortest routes through that network would be a godsend for a massive distributed system like the Internet, where there may be thousands of “cities” on any given route, instead of just fifteen. The traveling salesman may finally have been killed off for good by online retailers like Amazon.com, but the traveling salesman problem has become even more critical to the digital world.

  In late 1999, Marco Dorigo of the Free University of Brussels announced that he and his colleagues had hit upon a way of reaching “near-optimal” solutions to the traveling salesman problem that was notably more time-efficient that any traditional approach. Dorigo’s secret: let the ants do the work.

  Not literal ants, of course. As we saw at the beginning of this book, ant colonies have an uncanny ability to calculate the shortest path to different food sources, using their simple language of pheromone trails. Dorigo’s insight was to solve the traveling salesmen problem the way an ant colony would: send out an army of virtual salesman to explore all possible routes on the map. When a salesman successfully completes a journey to all fifteen cities, he then traces his path back to the starting city, depositing a small amount of virtual “pheromone” along the way. Because the total amount of pheromone is finite, it is spread more thinly along longer routes, and more heavily along shorter ones. With thousands of ants exploring the map, some sections of shorter routes quickly accumulate thick layers of pheromone, while less efficient routes have almost no pheromone at all. After the first round of exploration, a new batch of virtual salesmen are released and encouraged to follow the routes that have been most heavily dosed with pheromones. After several repeated sessions, the salesmen swarm starts homing in on the shortest routes, reaching a near-optimal solution to the traveling salesman problem without using anything resembling traditional calculus or a central problem-solver. Since Dorigo’s announcement of his results, France Telecom, British Telecommunications, and MCI have applied antlike routing strategies to their telephone and data networks. Early studies show that Dorigo’s approach is significantly more efficient than the Open Shortest Path First routine used by the Internet to distribute data between nodes on the network. A few years from now, our online interactions may be sustained by the bottom-up power of swarm intelligence. And once again, the ants will have figured it out long before we did.

  What kind of data will those future networks transmit? Soon after this book’s publication, the Net will be teeming with the digital inhabitants of Will Wright’s latest creation, The Sims Online. A fusion of The Sims and SimCity, the game allows players to collectively build cities as part of a massive network collaboration. Unlike either previous game, all the citizens of the world are controlled by actual human beings, logged into the system from all around the world. As in The Sims, you can zoom into your own character’s living room, or visit a friend’s house down the street for a neighborhood barbecue. But you can also zoom out to see the entire landscape that the players have created. An early draft of the game that Wright showed me included a brilliant neighborhood-creation system that seemed straight out of the pages of Jane Jacobs. City neighborhoods are defined from the bottom up, as players establish their own homesteads in various regions of the virtual space. Any player can create his or her own private neighborhood, the way they can create a name for their virtual character on the screen. But you can also persuade your neighbors to adopt your neighborhood name as well. When a certain number of citizens have declared their allegiance to a specific neighborhood, the system officially recognizes that district and gives it a special sign, along with various tax breaks. The bigger the neighborhood, the bigger the sign, and the more lucrative the benefits. It’s a classic marriage of bottom-up growth and top-down management: let the neighborhoods come from below, but build incentives into the system to encourage their growth.

  Our newfound access to virtual cities on the computer screen hasn’t abated our appetite for real-world city living. Five years ago, most digital-savvy social critics predicted that the rise of the Web and various telecommuting appliances would deliver the death blow to city living, finishing a forty-year process that had begun with the surburban flight of the postwar years. We’d all be living on our Wyoming ranches in ten years, dialing into the office instead of straphanging on an overcrowded subway. Of all the lofty predictions of the midnineties, none have proven to be more misguided than those eulo
gies for urban life. The digital revolution has turned out to be a tremendous energizer for dense urban centers like San Francisco, New York, and Seattle—for reasons that date back to the guild system and trade clusters of twelfth-century Florence. Industries driven by ideas naturally gravitate toward physical centers of idea generation, even in an age of instant data transmissions. Bright minds with shared interests still flock together, even when they have wireless modems and broadband in their living rooms. Now smaller settlements are trying to learn from the street-centered dynamism of the traditional organic city: the New Urbanist movement has begun to transform America’s suburban development practices, by following the rules outlined by Jane Jacobs almost a half century ago: shorter blocks, livelier sidewalks, mixed-used zoning, and pedestrian-based transportation. Despite the Bengali typhoon of the digital revolution—or perhaps, in part, because of it—the old-style self-organizing city is today as vital and relevant as it has ever been.

  The vitality has its downsides, of course: rents, congestion, traffic jams. Even in the most sidewalk-centric cities, the flow of automobiles through the complex latticework of streets poses an organizational problem that rivals the traffic of information across the World Wide Web. For decades, urban engineers have built ever-more-complicated systems to direct the paths of automobiles through congested city streets, by observing the patterns of traffic and tweaking the stoplights and street directions where problems arose. But city traffic is a problem of organized complexity, and it is best tackled with bottom-up solutions, not top-down ones. Almost fifty years after he first came up with the idea for Pandemonium, Oliver Selfridge has embarked on a quest for exactly that solution, building a learning network of traffic lights that will find an optimal system in changing conditions. Selfridge wants to attack the problem of traffic the way Danny Hillis attacked the problem of number sorting: by giving the network the general goal of minimizing delays, but letting the overall system figure out the details, using the tools of feedback, neighbor interaction, and pattern recognition that are the hallmark of all self-organizing systems. Traffic jams themselves are a particularly crude form of emergent behavior, and for years we’ve been battling them with engineering solutions. Selfridge wants to take the master planners out of the equation. Make the traffic lights smart—by connecting them and feeding them information about backups or accidents—and you have a solution that can actually manage the immense and constantly changing problem of urban movement. You can conquer gridlock by making the grid itself smart.