Tubes: A Journey to the Center of the Internet

by Andrew Blum

  Upstairs, the empty hallways smelled of mildew. We passed vacant offices, their doors cracked open. Auer’s space looked like it belonged to a private eye in a film noir. The three small rooms had linoleum floors and worn-out Venetian blinds. The double-hung windows were thrown wide open to the winter, the cheapest way to keep the machines cool. The only evidence of the building’s former opulence was a remnant scrap of mosaic floor tile, shattered in a corner like a broken mug. Auer’s piece of the Internet was set unceremoniously on a raised platform: two man-sized steel racks, filled with a half-dozen machines, snug in a nest of cables. The key piece of equipment was a black Cisco 6500 Series router, the size of a few stacked pizza boxes, its chassis tattooed with bar-coded inventory labels and poked through by blinking green LEDs.

  For the twenty-five thousand customers who relied on Auer’s company to connect to “the Internet,” this machine was the on-ramp. Its job was to read the destination of a packet of data and send it along one of two paths. The first path went upstairs to an equipment room belonging to Cogent, a wholesale Internet provider that serviced cities from San Francisco to Kiev. A yellow wire passed through a utility shaft, came through a wall, and plugged into Cogent’s equipment, itself connected to electronic colleagues in Chicago and Minneapolis. This building was Cogent’s only “point of presence” in all of Wisconsin, the only place Cogent’s express train stopped; that’s why Auer’s company was here, and all the others. The second cable went to Time Warner, whose wholesale Internet division provided an additional connection—a backup, plugging Auer’s piece of the Internet into all the rest.

  Taken as a whole, the building seemed a labyrinth, packed with a hundred years of twisted cables and broken dreams. Yet in its particularity, this part of the Internet—Auer’s part—was strikingly legible; it wasn’t an endless city at all but a simple fork in the road. I asked Auer what happened after here, and he shrugged. “I care about where we can talk to Cogent or Time Warner, which means this building. Once it’s here it’s really out of my hands.” For about twenty-five thousand Wisconsinites, this was the source. Their Internet went this way and it went that way: two yellow cables leading, eventually, to the world. Every journey—physical and virtual—begins with a single step.

  A few weeks later I went to Washington to visit TeleGeography’s offices, for a better sense of how Krisetya drew a clear map of the Internet’s mushy layer cake. But the night before I left, New York was hit by a blizzard, and I emailed Krisetya to let him know I’d be arriving later than expected. As the train moved south across New Jersey the snow began to dwindle, so that by the time we pulled into Washington the blanket of white I had left in New York had given way to clear gray sky and dry sidewalks. It was as if over the course of the ride the veil that had descended upon the landscape had just as quickly been lifted. Arriving in DC, I opened my laptop in the center of Union Station’s great neoclassical hall to log into a café’s wireless network and send off an email to California. A few minutes later, standing on the Metro platform, I thumbed a message to my wife saying that, despite New York being shut down by the snow, I had made it to Washington (and we’ll see about getting back).

  I share all these quotidian details of travel because on that day my senses were unusually attuned to the networks that surrounded me, both visible and invisible. Maybe it was the way the snow had drawn a new outline around the world’s familiar shapes, while slowing my progress past them. Or maybe it was just the early morning hour and the fact that I had maps on the brain. But as the train was sliding across the elbow of New Jersey, ducking out of the storm, I could imagine the emails following (albeit faster) along the same path. I had recently learned that many of the fiber-optic routes between New York and Washington were lain along the railroad tracks, and I could begin to imagine the route my email to California had taken: it might have shot back the way I’d come, to New York, before heading cross-country, or it could have continued farther west to Ashburn, Virginia, where there was an especially significant network crossroads. The exact route of that email didn’t matter; what did was that the Internet no longer seemed infinite. The invisible world was revealing itself.

  In a neighborhood of staid lobbyists and wood-paneled law firms, TeleGeography’s K Street office stands out for its lime-green walls, exposed ceilings, and translucent cubicle dividers. The front door pivoted creatively on its center point. Maps lined the walls, of course. On one, Spain had been adorned with a Groucho Marx mustache, a remnant of a recent holiday party. Krisetya welcomed me into his office, the desk piled high with books about information design. When he joined TeleGeography in 1999, he was put right to work on the company’s first big report, Hubs + Spokes: A TeleGeography Internet Reader. It was groundbreaking. Before, there were geographic maps showing the networks operated by individual corporations or government agencies, and there were “logical” diagrams of the whole Internet, like a subway map. Neither gave a strong sense of how the Internet adhered and diverged from the real-world geography of cities and countries. What places were more connected? Where were the hubs?

  Krisetya began looking at new ways of portraying that combination of the geopolitical and networked worlds. He blended the outlines of the continents with diagrams of the networks, “always layering something abstract on top of something that’s familiar, always looking to give it more meaning.” Other kinds of maps had long struggled with the same issues—like airline routes or subways. In both cases, the end points were more important than the paths themselves. They always had to balance the workings of the system internally with the external world it connected. London’s Tube map might be the height of the genre: a geographical fiction that pushes and pulls at real-world creations, leaving in its wake a kind of alternate city that’s become as real as the true one.

  On his maps, Krisetya portrayed this by showing the most heavily trafficked routes between cities, such as between New York and London, with the thickest lines—not because there were necessarily more cables there (or some single, superthick cable) but because that was the route across which the most data flowed. This was an insight that dated back to that first report. “If you look inside the Internet cloud a fairly distinct hub-and-spoke structure begins to emerge at both an operational (networking) and physical (geopolitical) level,” it explained. The Internet’s structure “is based upon a core of meshed connectivity between world cities on coastal shores—Silicon Valley, New York and Washington, DC; London, Paris, Amsterdam and Frankfurt; Tokyo and Seoul.” And it still is.

  Today’s version—the one TeleGeography calls the GIG—is the bible for big telecommunications companies. The key to its approach is still to look at Internet traffic as concentrated between powerful cities. TeleGeography breaks down the nebulous cloud into a clear system of point-to-point communications, of segments. Contrary to its ostensible fluidity, the geography of the Internet reflects the geography of the earth; it adheres to the borders of nations and the edges of continents. “That’s the nugget of our approach,” Krisetya explained to me in his office, sounding like a college tutor. “We always put much more emphasis on the actual geography than the connections in between. In the beginning, that’s what we were more familiar with. When the Internet was still very much abstract, we knew where the two end points were, even if we didn’t understand how this was all being built.”

  That had a certain clarity. The world is real; London is London, New York is New York, and the two usually had a lot to say to each other. But I was still hung up on what seemed a simple question: What, physically speaking, were all those lines? And where precisely did they run? If TeleGeography properly understood the Internet as being “point to point,” what and where were the points?

  For their part, TeleGeography’s analysts don’t go out into the world with a GPS and a sketchpad. They don’t attach sensors to the Internet to measure the speed of the bits passing by, like a water meter. Their process is quite low tech: they distribute a simple questionnaire to telecom execu
tives, requesting information about their networks in exchange for the promise to keep it confidential and to share the aggregated information with them. And then TeleGeography asks the Internet itself.

  To see how, Krisetya dropped me off at the tidy desk of Bonnie Crouch, the young analyst responsible for gathering and interpreting TeleGeography’s data on Asia. The diplomatic work of wrangling and cajoling the information from the telecom carriers was finished, and the responses loaded into TeleGeography’s database. Crouch’s job was to confirm what the carriers said, based on the Internet’s actual traffic patterns. Cartographers talk about “ground truth”: the in-person measurements used to check the accuracy of the “remote sensing”—which in contemporary mapmaking usually means aerial or satellite photographs. TeleGeography had its own way of checking the “ground truth” of the Internet.

  When I enter an address into my browser, a thousand tiny processes are set in motion. But in the most fundamental terms, I’m asking a computer far away to send information to a computer close by, the one in front of me. Browsing the web, that typically means a short command—“send me that blog post!”—is volleyed back with a far larger trove, the blog post itself. Behind the URL—say, www.mapgeeks.com—is a self-addressed envelope with the instructions that connect any two computers. Every piece, or “packet,” of data traveling across the Internet is labeled with its destination, known as an “IP” address. Those addresses are grouped into the equivalent of postal codes, called “prefixes,” given out by an international governing body, the Internet Assigned Numbers Authority. But the routes themselves aren’t assigned by anyone at all. Instead, each router announces the existence of all the computers and all the other routers “behind” it, as if posting a sign saying THIS SECTION OF THE INTERNET OVER HERE. Those announcements are then passed around from router to router, like a good piece of gossip. For example, Jon Auer’s router in Milwaukee is the doorway to his twenty-five thousand customers, grouped into just four prefixes. It announces its presence to the two neighboring routers, belonging to Cogent and Time Warner. Those two neighboring routers make a note of it, and then pass the word on to their neighbors—and so on, until every router on the Internet knows who’s behind whom. The complete aggregate list of destinations is known as the “routing table.” At the end of 2010 it had nearly four hundred thousand entries and was growing steadily. The whole thing is typically stored in the router’s internal memory, while a compact flash card, like the kind used by digital cameras, keeps the operating code. Auer buys his on sale at the local drugstore.

  Two things surprised me about this. The first is that every IP address is by definition public knowledge; to be on the Internet is to want to be found. The second is that the announcement of each route is based wholly on trust. The Internet Assigned Numbers Authority gives out the prefixes, but anyone can put up a sign pointing the way. And sometimes that does go horribly wrong. In one well-known incident in February 2008, the Pakistani government instructed all Pakistani Internet providers to block YouTube, because of a video it deemed offensive. But an engineer at Pakistan Telecom, receiving the memo at his desk, misconfigured his router, and rather than removing the announced path to YouTube, he announced it himself—in effect declaring that he was YouTube. Within two and a half minutes, the “hijacked” route was passed to routers across the Internet, leading anyone looking for YouTube to knock on Pakistan Telecom’s door. Needless to say, YouTube wasn’t in there. For most of the world, YouTube wasn’t available at all for nearly two hours, at which point the mess was sorted out.

  It sounds preposterously loose and informal. But it strikes at the core of the Internet’s fundamental openness. There’s a certain amount of vulnerability involved with being a network on the Internet. When two networks connect, they have to trust each other—which also means trusting everyone the other one trusts. Internet networks are promiscuous, but their promiscuity is out in the open. It’s free love. Jon Postel, the longtime administrator of the Internet Assigned Numbers Authority, put this into a koan, a golden rule for network engineers: “Be conservative in what you send, be liberal in what you accept.”

  For TeleGeography this means everything is out in the open, for those who know how to see it. The company uses a program called Traceroute, originally written in 1988 by a computer scientist at the Lawrence Berkeley National Laboratory. He had gotten fed up, as he put it in a mailing list message to his colleagues, trying to figure out “where the !?*! are the packets going?” and worked up a simple program that traced their paths. Enter in an IP address and Traceroute will feed back a list of the routers traversed to reach it, and the time (in milliseconds) elapsed in the journey between each one. TeleGeography then takes it one step further. It carefully selects fifteen locations around the world, looking especially for “dead-end” places with only a few paths out to the rest of the Internet—Denmark’s Faroe Islands, for example. It then searches for websites there hosting a copy of the Traceroute program (often a university computer science department), and directs those fifteen Traceroute hosts to query more than twenty-five hundred “destinations,” websites carefully chosen because they could reasonably be expected to actually live on a hard drive in the place where they say they live. Jagiellonian University in Poland, for example, is unlikely to host its website in, say, Nebraska. That meant TeleGeography in Washington was asking a computer science department in Denmark to show how it was connected to a university in Poland. It was like a spotlight in Scandinavia shining on twenty-five hundred different places around the world, and reporting back on the unique reflections. TeleGeography’s trick was finding real-world corners and dead ends, thereby minimizing the number of possible paths.

  Added all together, the fifteen hosts TeleGeography selected query twenty-five hundred destinations yielding more than twenty thousand journeys across the Internet—and, inherently, around the earth. Quite a few of these journeys are never completed; the traces conk out, lost in the ether. The whole set takes several days, not because TeleGeography has a slow computer, or even a slow Internet connection. Rather, those days represent the aggregate duration of all those thousands of trips, milliseconds piled upon milliseconds in which the explorer packets are crisscrossing the earth. And I don’t mean “crisscrossing” idly. These paths are by no means random or imaginary. Each packet—a clump of math, in the form of electrical signals or pulses of light—moves along very specific physical pathways. The whole point of each traceroute is to identify that specificity, that singular record of a journey. Theoretically you could divide up the task of querying each traceroute among multiple computers, but there’s no way to rush the traces themselves, no more than you can rush the speed of light. The time the packets take on their journey is the time they take. Each recorded journey is like a series of tiny postcards from around the globe. TeleGeography then layers the tens of thousands of them as if they were strands of papier-mâché, until the patterns emerge.

  Crouch and the other analysts then parse the routes by hand. “Any particular country of interest?” she asked me, with the geographic expansiveness I was quickly learning to love about Internet people. I told her to pick whichever she knew best, and she chose Japan—dodging the ambiguity of China’s networks. On her screen, a long list of jumbled letters and numbers, like a phone book without the names, scrolled down. Each grouping represented the results of a single trace—from the Faroes to Hokkaido, for example. Each individual line represented a single router: a lonely machine in a cold room, studiously forwarding packets. Over time the codes had become familiar to Crouch, like London’s streets are to a cabdriver. “You begin to get a feel for how companies name their routers,” she said. “Like that one’s going from SYD to HKG—the airport codes for Sydney to Hong Kong. And the carrier did tell us it’s running that route, so we don’t need to worry about it.” Her goal in reading these lists was to confirm that the carriers are operating the routes they say they are, and, with a more subjective eye, to make a judgment about the a
mount of traffic on that route. “Our research gives us all the pieces of the puzzle: the bandwidth, the Internet capacity, some of the pricing information. The gaps in between we can fill in with some reasonable accuracy.”

  It occurred to me that Crouch was part of the small global fraternity that knows the geography of the Internet the way most people know their hometowns. Her boss, a Texan named Alan Maudlin who improbably led TeleGeography’s analyst team from his home in Bratislava, possessed one of the best mental maps of the physical infrastructure of the Internet. I’d spoken with him before coming. “I don’t need to look at a map,” he told me over Skype. “I have in my mind, and I can almost note, which cables connect everywhere in the world.” Rather than maps of the Internet, his study in Slovakia was decorated with antique maps of Texas. “I suppose it is kind of like the Matrix, where you can see the code. I don’t even have to think about it anymore. I can just see where it’s going. I know what city the router is in, and where the packet’s going. It’s the weirdest thing. But it’s easy to just fly through it all, once you know what to look for.”

  Yet what’s so striking to me—and so often overlooked—is that each router is inherently present. Each router is a singular waypoint, a physical box, in a real place, on a packet’s journey across this real earth. Two billion people use the Internet from every country on earth; airplanes have Wi-Fi; astronauts browse the web from space. The question “Where is the Internet?” should seem meaningless, because where isn’t it? And yet, standing over Crouch’s shoulder, watching her identify the coded name of individual machines in a city on the other side of the world, the Internet didn’t seem infinite at all. It seemed like a necklace strung around the earth. Forming what pattern? Did it look like the route maps in the back of an airline magazine? Or was it more chaotic, like a bowl of spaghetti or the London Underground? Before, I’d imagined the Internet as something organic, beyond human design, like an ant colony or a mountain range. But now its designers seemed present, not an innumerable crowd but a tidy contact list on a laptop in Washington. So who were they then? Why did they lay their networks there? Where did it all begin?