Emergence

by Steven Johnson

  And if the Web would make a miserable city, it would do even worse as a brain. Here’s Steven Pinker, the author of How the Mind Works, in a Slate dialogue with Wright:

  The Internet is in some ways like a brain, but in important ways not. The brain doesn’t just let information ricochet around the skull. It is organized to do something: to move the muscles in ways that allow the whole body to attain the goals set by the emotions. The anatomy of the brain reflects that: it is not a uniform web or net, but has a specific organization in which emotional circuits interconnect with the frontal lobes, which receive information from perceptual systems and send commands to the motor system. This goal-directed organization comes from an important property of organisms you discuss: their cells are in the same reproductive boat, and thus have no “incentive” to act against the interests of the whole body. But the Internet, not being a cohesive replicating system, has no such organization.

  Again, the point here is that intelligent systems depend on structure and organization as much as they do on pure connectedness—and that intelligent systems are guided toward particular types of structure by the laws of natural selection. A latter-day Maxwell’s Demon who somehow manages to superglue a billion neurons to each other wouldn’t build anything like the human brain, because the brain relies on specific clusters to make sense of the world, and those clusters only emerge out of a complex interplay among neurons, the external world, and our genes (not to mention a few thousand other factors). Some systems, such as the Web, are geniuses at making connections but lousy with structure. The technologies behind the Internet—everything from the microprocessors in each Web server to the open-ended protocols that govern the data itself—have been brilliantly engineered to handle dramatic increases in scale, but they are indifferent, if not downright hostile, to the task of creating higher-level order. There is, of course, a neurological equivalent of the Web’s ratio of growth to order, but it’s nothing you’d want to emulate. It’s called a brain tumor.

  Still, in the midst of all that networked chaos, a few observers have begun to detect macropatterns in the Web’s development, patterns that are invisible to anyone using the Web, and thus mostly useless. The distribution of Web sites and their audiences appears to follow what is called a power law: the top ten most popular sites are ten times larger than the next hundred more popular sites, which are themselves ten times more popular than the next thousand sites. Other online cartographers have detected “hub” and “spoke” patterns in traffic flows. But none of these macroshapes, even if they do exist, actually makes the Web a more navigable or informative system. These patterns may be self-organizing, but they are not adaptive in any way. The patterns are closer to a snowflake’s intricacy than a brain’s neural net: the snowflake self-organizes into miraculously complicated shapes, but it’s incapable of becoming a smarter snowflake, or a more effective one. It’s simply a frozen pattern. Compare that to the living, dynamic patterns of a city neighborhood or the human brain: both shapes have evolved into useful structures because they have been pushed in that direction by the forces of biological or cultural evolution: our brains are masterpieces of emergence because large-brained primates were, on the whole, more likely to reproduce than their smaller-brained competitors; the trade clusters of the modern city proliferated because their inhabitants prospered more than isolated rural craftsmen. There is great power and creative energy in self-organization, to be sure, but it needs to be channeled toward specific forms for it to blossom into something like intelligence.

  But the fact that the Web as we know it tends toward chaotic connections over emergent intelligence is not something intrinsic to all computer networks. By tweaking some of the underlying assumptions behind today’s Web, you could design an alternative version that could potentially mimic the self-organizing neighborhoods of cities or the differentiated lobes of the human brain—and could definitely reproduce the simpler collective problem-solving of ant colonies. The Web’s not inherently disorganized, it’s just built that way. Modify its underlying architecture, and the Web might very well be capable of the groupthink that Teilhard envisioned.

  How could such a change be brought about? Think about Deborah Gordon’s harvester ants, or Paul Krugman’s model for edge-city growth. In both systems, the interaction between neighbors is two-way: the foraging ant that stumbles across the nest-building ant registers something from the encounter, and vice versa; the new store that opens up next to an existing store influences the behavior of that store, which in turn influences the behavior of the newcomer. Relationships in these systems are mutual: you influence your neighbors, and your neighbors influence you. All emergent systems are built out of this kind of feedback, the two-way connections that foster higher-level learning.

  Ironically, it is precisely this feedback that the Web lacks, because HTML-based links are one-directional. You can point to ten other sites from your home page, but there’s no way for those pages to know that you’re pointing to them, short of you taking the time to fire off an e-mail to their respective webmasters. Every page on the Web contains precise information about the other addresses it points to, and yet, by definition, no page on the Web knows who’s pointing back. It’s a limitation that would be unimaginable in any of the other systems that we’ve looked at. It’s like a Gap outlet that doesn’t realize that J.Crew just moved in across the street, or an ant that remains oblivious to the other ants it stumbles across in its daily wanderings. The intelligence of a harvester ant colony derives from the densely interconnected feedback between ants that encounter each other and change their behavior according to preordained rules. Without that feedback, they’d be a random assemblage of creatures butting heads and moving on, incapable of displaying the complex behavior that we’ve come to expect from the social insects. (The neural networks of the brain are also heavily dependent on feedback loops.) Self-organizing systems use feedback to bootstrap themselves into a more orderly structure. And given the Web’s feedback-intolerant, one-way linking, there’s no way for the network to learn as it grows, which is why it’s now so dependent on search engines to rein in its natural chaos.

  Is there a way around this limitation? In fact, a solution exists already, although it does nothing to modify the protocols of the Web, but rather ingeniously works around the shortcomings of HTML to create a true learning network that sits on top of the Web, a network that exists on a global scale. Appropriately enough, the first attempt to nurture emergent intelligence online began with the desire to keep the Web from being so forgetful.

  *

  You can’t really, truly understand Brewster Kahle until you’ve had him show you the server farm in Alexa Internet’s basement. Walk down a flight of outdoor steps at the side of an old military personnel-processing building in San Francisco’s Presidio, and you’ll see an entire universe of data—or at least a bank of dark-toned Linux servers arrayed along a twenty-foot wall. The room itself—moldy concrete, with a few spare windows gazing out at foot level—might have held a lawn mower and some spare file cabinets a few decades ago. Now it houses what may well be the most accurate snapshot of The Collective Intelligence anywhere in the world: thirty terabytes of data, archiving both the Web itself and the patterns of traffic flowing through it.

  As the creator of the WAIS (Wide Area Information Server) system, Kahle was already an Internet legend when he launched Alexa in 1996. The Alexa software used collaborative-filtering-like technology to build connections between sites based on user traffic. The results from its technology are showcased in the “related sites” menu option found in most browsers today. Amazon.com acquired Alexa Internet in 1999, but the company remains happily ensconced in its low-tech Presidio offices, World War II temporary structures filled with the smell of the nearby eucalyptus trees. “In just three years we got bigger than the Library of Congress, the biggest library on the planet,” Kahle says, arms outstretched in his basement server farm. “So the question is, what do we do now?”

&nb
sp; Obsessed with the impermanence of today’s datastreams, Kahle (and his partner, Bruce Gilliat) founded Alexa with the idea of taking “snapshots” of the Web and indexing them permanently on huge storage devices for the benefit of future historians. As they developed that project, it occurred to them that they could easily open up that massive database to casual Web surfers, supplementing their Web browsing experience with relevant pages from the archive. Anytime a surfer encountered a “404 Page Not Found” error—meaning that an old page had been deleted or moved—he or she could swiftly consult the Alexa archive and pull up the original page.

  To make this possible, Kahle and Gilliat created a small toolbar that launches alongside your Web browser. Once the application detects a URL request, it scurries off to the Alexa servers, where it queries the database for information about the page you’re visiting. If the URL request ends in a File Not Found message, the Alexa application trolls through the archives for an earlier version of the page. Kahle dubbed his toolbar a “surf engine”—a tool that accompanies you as you browse—and he quickly realized that he’d stumbled across a program that could do far more than just resuscitate old Web pages. By tracking the surfing patterns of its users, the software could also make connections between Web sites, connections that might otherwise have been invisible, both to the creators of those sites and the people browsing them.

  Two months after starting work on Alexa, Kahle added a new button to his toolbar, with the simple but provocative tag “What’s Next?” Click on the button while visiting a Marilyn Monroe tribute site, and you’ll find a set of links to other Marilyn shrines online; click while you’re visiting a community site for cancer survivors, and you’ll find a host of other like-minded sites listed in the pull-down menu. How are these connections formed? By watching traffic patterns, and looking for neighbors. The software learns by watching the behavior of Alexa’s users: if a hundred users visit FEED and then hop over to Salon, then the software starts to perceive a connection between the two Web sites, a connection that can be weakened or strengthened as more behavior is tracked. In other words, the associations are not the work of an individual consciousness, but rather the sum total of thousands and thousands of individual decisions, a guide to the Web created by following an unimaginable number of footprints.

  It’s an intoxicating idea, and strangely fitting. After all, a guide to the entire Web should be more than just a collection of handcrafted ratings. As Kahle says, “Learning from users is the only thing that scales to the size of the Web.” And that learning echoes the clustered neighborhoods of Florence or London. Alexa’s power of association—this site is like these other sites—emerges out of the desultory travels of the Alexa user base; none of those users are deliberately setting out to create clusters of related sites, to endow the Web with much-needed structure. They simply go about their business, and the system itself learns by watching. Like Gordon’s harvester ants, the software gets smarter, grows more organized, the more individual surfing histories it tracks. If only a thousand people fire up Alexa alongside their browsers, the recommendations simply won’t have enough data behind them to be accurate. But add another ten thousand users to the mix, and the site associations gain resolution dramatically. The system starts to learn.

  Let’s be clear about what that learning entails, because it differs significantly from the traditional sci-fi portraits of computerized intelligence, both utopian and dystopian. Alexa makes no attempt to simulate human intelligence or consciousness directly. In other words, you don’t teach the computer to read or appreciate Web site design. The software simply looks for patterns in numbers, like the foraging ants counting the number of fellow foragers they encounter per hour. In fact, the “intelligence” of Alexa is really the aggregated wisdom of the thousands—or millions—of people who use the system. The computer churns through the millions of ratings in its database, looks for patterns of likes and dislikes, then reports back to the user with its findings.

  It’s worth noting here that Alexa is not truly a “recommendation agent”; it is not telling you that you’ll like the five sites that it suggests. It’s saying that there’s a relationship between the site you’re currently visiting and the sites listed on the pull-down menu. The clusters that form via Alexa are clusters of association, and the links between them are not unlike the traditional links of hypertext. Think about the semantics of a hypertext link embedded in an online article: when you see that link, you don’t translate it as “If you like this sentence, you’ll like this page as well.” The link isn’t recommending another page; it’s pointing out that there’s a relationship between the sentence you’re reading and the page at the other end of the link. It’s still up to you to decide if you’re interested in the other sites, just as it’s up to you to decide which silk merchant you prefer on the Por Santa Maria. Alexa’s simply there to show you where the clusters are.

  Outside of the video-game world, Alexa may be the most high-profile piece of emergent software to date: the tool was integrated into the Netscape browser shortly after its release, and the company is now applying its technology to the world of consumer goods. But the genre is certainly diversifying. An East Coast start-up called Abuzz, recently acquired by the New York Times digital unit, offers a filtering service that enables people searching for particular information or expertise to track down individuals who might have the knowledge they’re looking for. A brilliant site called Everything2 employs a neural-net-like program to create a user-authored encyclopedia, with related entries grouped together, Alexa-style, based on user traffic patterns. Indeed, the Web industry is teeming with start-ups promising to bring like minds together, whether they’re searching for entertainment or more utilitarian forms of information. These are the digital-age heirs to the Por Santa Maria.

  Old-school humanists, of course, tend to find something alarming in the idea of turning to computers for expert wisdom and cultural sensibility. In most cases, the critics’ objections sound like a strangely inverted version of the old morality tales that once warned us against animating machines: Goethe’s (and Disney’s) sorcerer’s apprentice, Hoffmann’s sandman, Shelley’s Frankenstein. In the contemporary rendition, it’s not that the slave technology grows stronger than us and learns to disobey our commands—it’s that we deteriorate to the level of the machines. Smart technology makes us dumber.

  The critique certainly has its merits, and even among the Net community—if it’s still possible to speak of a single Net community—intelligent software remains much villified in some quarters. Decades ago, in a curiously brilliant book, God and Golem, Inc., Norbert Wiener argued that “in poems, in novels, in painting, the brain seems to find itself able to work very well with material that any computer would have to reject as formless.” For many people the distinction persists to this day: we look to our computers for number crunching; when we want cultural advice, we’re already blessed with plenty of humans to consult. Other critics fear a narrowing of our aesthetic bandwidth, with agents numbly recommending the sites that everyone else is surfing, all the while dressing their recommendations up in the sheep’s clothing of custom-fit culture.

  But it does seem a little silly to resist the urge to experiment with the current cultural system, where musical taste is usually determined by the marketing departments at Sony and Dreamworks, and expert wisdom comes in the form of Ann Landers columns and the Psychic Hotline. If the computer is, in the end, merely making connections between different cultural sensibilities, sensibilities that were originally developed by humans and not by machines, then surely the emergent software model is preferable to the way most Westerners consume entertainment: by obeying the dictates of advertising. Software like Alexa isn’t trying to replicate the all-knowing authoritarianism of Big Brother or HAL, after all—it’s trying to replicate the folksy, communal practice of neighbors sharing information on a crowded sidewalk, even if the neighbors at issue are total strangers, communicating to each other over the distributed ne
twork of the Web.

  *

  The pattern-seeking algorithms of emergent software are already on their way to becoming one of the primary mechanisms in the great Goldberg contraption of modern social life—as familiar to us as more traditional devices like supply and demand, representational democracy, snap polls. Intelligent software already scans the wires for constellations of book lovers or potential mates. In the future, our networks will be caressed by a million invisible hands, seeking patterns in the digital soup, looking for neighbors in a land where everyone is by definition a stranger.

  Perhaps this is only fitting. Our brains got to where they are today by bootstrapping out of a primitive form of pattern-matching. As the futurist Ray Kurzweil writes, “Humans are far more skilled at recognizing patterns than in thinking through logical combinations, so we rely on this aptitude for almost all of our mental processes. Indeed, pattern recognition comprises the bulk of our neural circuitry. These faculties make up for the extremely slow speed of human neurons.” The human mind is poorly equipped to deal with problems that need to be solved serially—one calculation after another—given that neurons require a “reset time” of about five milliseconds, meaning that neurons are capable of only two hundred calculations per second. (A modern PC can do millions of calculations per second, which is why we let them do the heavy lifting for anything that requires math skills.) But unlike most computers, the brain is a massively parallel system, with 100 billion neurons all working away at the same time. That parallelism allows the brain to perform amazing feats of pattern recognition, feats that continue to confound digital computers—such as remembering faces or creating metaphors. Because each individual neuron is so slow, Kurzweil explains, “we don’t have time … to think too many new thoughts when we are pressed to make a decision. The human brain relies on precomputing its analyses and storing them for future reference. We then use our pattern-recognition capability to recognize a situation as compatible to one we have thought about and then draw upon our previously considered conclusions.”