Tubes: A Journey to the Center of the Internet

by Andrew Blum

  As we walked through the dim corridor between cages, the physical ramifications of all those connections were above us: thick rivers of cables in bundles the size of tires, laid in racks suspended from the ceiling, and then cascading down in “waterfalls,” as the techs call them, into each cage. The building buzzed with their energy. “You’re being irradiated as you speak,” Troyer said, only half-jokingly. “Jay’s already had three kids so he’s okay.” There were upward of ten thousand Internetwork connections, or “cross-connects,” in this building alone. This was the tangle of dusty cables hidden behind my couch inflated to the scale of a building—and it wasn’t any easier to organize.

  In the PAIX’s early days, “cable management” was a crucial technical challenge. The Internet was tangled. Experimenting with different ways of handling things, at one point Adelson and his staff tried prewiring different zones of the building to create fixed paths that could be patched together as needed, like an old-fashioned telephone switchboard. “But what we learned—or what poor Felix here learned—is that every time you did that you introduced a point of failure,” Adelson recalled, while Reyes shook his head at the memory. So they stuck to laying cable on an as-needed basis. A few years later, one particularly talented cable layer named John Pedro would earn US patent 6,515,224 for his technique: a “cascading cable tray system” with “pre-fabricated support structure.”

  As we walked between the cages filled with boxes glittering with blinking green lights, I had to remind myself to try to associate what I was seeing with its effect in the real world, in people’s lives—to confront, in the most basic terms, the way things move across the Internet. That required a leap of the imagination. Let’s say that yellow wire there belonged to eBay: Whose jade collectible teapot was zipping across it? Or what did a winemaker in New Zealand have to say to a sheikh in Qatar? My phone was on and getting emails. Were they passing through here? My niece lost a tooth—was the picture passing through Facebook’s cage over there?

  But the Internet as it surrounded me wasn’t a river I could dip a net in and pull up a sample to count the fish. To find the scale of information as we experience it each day—to find, say, a single email—would be more akin to counting the molecules of water. Each of those fiber-optic cables represented up to ten gigabits of traffic per second—enough to transmit ten thousand family pictures per second. The big router had up to 160 of those plugged in at once; and this building was filled with hundreds of those routers. Walking through the dimly lit aisles was like hacking through an underbrush of quadrillions, an unfathomable quantity of information.

  Yet for Adelson, there was a time when it was all personal. He saw a story in every corner. “Remember when we shut off Australia?” he excitedly asked the ad hoc tour group, stopping in front of one cage, a little emptier than the others. A router for the Australia Internet Exchange—“Ozzienet or something”—was installed in the building, but they weren’t paying their bills. Adelson remembers the phone call he received at home the evening they finally pulled the plug: “My wife was, like, ‘There’s somebody on the phone, they’re not happy, and it’s something about the Internet in Australia being down?’ I was, like, ‘Really? Give me that.’”

  In another cage was the onetime home of Danni’s Hard Drive, a prominent early pornography site—online home of Danni Ashe, who Guinness World Records once named the “Most Downloaded Woman” (a category they no longer track). One night in the late ’90s, Danni herself was purportedly discovered here in the basement, naked with her eponymous hard drive, in the midst of taking the “photo of the week.” The old-timers nodded at the memory, but later I’d hear the same legend repeated at other big Internet buildings, and when I eventually tracked down Ashe and her network engineer at the time, Anne Petrie, they placed the event not in Palo Alto but at MAE-West, the Silicon Valley cousin of MAE-East. “I am the woman formerly known as Danni Ashe,” she wrote to me. “Unfortunately, I don’t remember a whole lot of details from that day but I’m guessing the two engineers working for me at the time would.” Indeed, Petrie did. She had spent sixteen hours installing a pair of new SGI Origin servers, the state of the art at the time, and Ashe and her husband had come to see them. “Invariably whenever Danni was on television everything would go down, because it would just flood the servers with requests,” Petrie recalled. The photo shoot commemorated the occasion.

  We worked our way deeper into the building and back in time. Adelson stopped in front of a cage that was a bit larger than the others and pleaded with the guys, “Can we go in here? I won’t touch anything. I gotta go in here!” The space was more like a small office than a cubicle and was built into the corner of the building, so that two of its walls were solid, rather than the typical steel mesh. It was filled with ancient-looking equipment, pocked with little steel toggle switches and an old black telephone headset.

  “This is where one of the most important lies I’ve ever told took place,” Adelson announced, with mock gravity. From the beginning, the PAIX was “carrier neutral”—but at the beginning it was also “carrier free.” It was like that week after moving into a new home, before the cable guy arrived. It wasn’t connected. One of Adelson’s biggest challenges was convincing competing fiber-optic network owners to “pop” the building and establish a “point of presence”—a place to connect. But at the time the carriers just didn’t do it. They kept their own equipment in their own facilities and you came to them, paying an arm and a leg for the “local loop” required to do so. (This was the situation out of which MAE-East was born: its parent company, MFS, was in the local loop business, and MAE-East was essentially a very local loop.) If one carrier came into the PAIX, they knew others would. So Adelson lied. “I went to Worldcom and said, ‘Pacific Bell said they’re going to come in in about three weeks.’ And they’re, like, ‘Really?’ ” Then Adelson went to Pacific Bell (now AT&T) and said the same thing: “Guess who’s installing their fiber backbone in the basement…?” They panicked—their local loop monopoly was at stake. “They’re, like, ‘We’re coming in!’ We said we had orders—but we made the whole thing up.” Adelson pointed up at the ceiling, where a thick bundle of black cables disappeared into a dark hole. It was the sort of business decision—like bringing a hot dog vendor to a ball game—that one only needed to be convinced to try once; ever after, it would be recalled only as the smartest thing they’d ever done. After that, the building filled up so fast it became a struggle to keep up. Every available inch was taken over for equipment.

  “There were racks in the bathrooms!” Adelson said, as we continued upstairs to see what was once office space and now has been entirely converted for networking equipment. “How many times did we get to a point—just in the first two years—where we said, ‘It is not physically possible to do anything else in that building,’ and then a month later, be, like, ‘Okay, we found a way!’” They called the place “the Winchester Mystery House of Internet buildings,” an allusion to the haunted Silicon Valley mansion of the Winchester rifle heir, who for thirty-eight years obsessively added rooms in a desperate effort to evade the ghosts she believed her fortune had created. The PAIX was similarly an exercise in creative construction. Wedged as it was into a downtown Palo Alto block, there was no room for horizontal expansion. The city government had little appetite for the ever-increasing quantities of fuel needed to run the backup generators. Seismically, the structure was just barely up to snuff—for office space, not heavy computer equipment. Adelson shook his head at the memory. “You couldn’t have chosen a worse building.”

  But the Palo Alto Internet Exchange’s real trouble came from a different direction. At nearly the same moment the building became the Internet’s dominant switching point, in January 1998, Digital, its parent company, was acquired by the Compaq Corporation for $9.6 billion, in what was at the time the largest deal in the history of the computing industry. It was bad news on Bryant Street. As Compaq and Digital struggled to integrate, there was growing concer
n that the relatively small business of the PAIX would fall through the cracks—and at just the moment when the Internet needed it most. The PAIX had quickly set the standard for how, under a single roof, the network of networks that compose the Internet connect. But the PAIX’s success was also its Achilles heel: there wasn’t enough of it. The PAIX made it clear that carrier-neutral exchanges worked. But it also made it clear that they needed room to breathe—that an old building in a dense (and expensive) downtown wasn’t ideal.

  Adelson saw an opportunity. This was the moment when everybody and his mother had an idea for a “dot-com,” typically one that would use the Internet’s virtualizing power to transform an industry: from grocery delivery to auctions, movie listings to classified ads. But if most people saw the Internet as a means to leave the physical world behind by setting up virtual storefronts or auction halls, Adelson saw an unmet need in the opposite idea: all that virtual stuff needed a physical world to call its own.

  There would be Palo Alto Internet Exchanges everywhere. Adelson would be like the Conrad Hilton of the Internet, opening up a branded chain of “telecom hotels,” where network engineers could be assured of a consistent experience. Unlike the buildings owned by the big telecom carriers themselves—like Verizon or MCI—these would be neutral places where competing networks of all kinds could connect. Unlike MAE-East or, to a lesser degree, the PAIX, they would be built with the proper backup and security systems, and designed to make it as easy as possible for networks to connect with one another. And like a free morning Wall Street Journal at a business hotel, the exchanges would offer perks meant to specially appeal to their unique customers: the network engineers (and former network engineers), like Adelson himself. The challenge was figuring out where on the big planet Earth to put these places. How many of them did the Internet really need?

  Adelson relied on a crucial hunch about how the structure of the Internet would evolve: networks would need to interconnect at multiple scales. They had to not only occupy the same building but the same building in several different places around the world. The networks of the Internet would be global, but the infrastructure would always be local. In that sense, the Hilton analogy doesn’t require stretching at all: Equinix wasn’t trying to establish a single central point, but a short list of capitals in the most important markets—mirroring the tendency of big multinational corporations to have offices in the same short list of global cities, from New York to London to Singapore to Frankfurt. An Equinix “Internet Business Exchange” would be the same place everywhere. For me—a traveler to the Internet—this presented something of a paradox: Was an Equinix facility best understood as locally distinct and unique, or just one part of a continuous global realm, a wormhole across continents? Was an Equinix data center a place or placeless? Or both?

  When Adelson left Digital, he and a colleague, Al Avery, quickly raised $12.5 million in venture capital, primarily from some big names with a vested interest in the Internet’s growth, including Microsoft and Cisco, the router company. The “ix” in Equinix indicated an “Internet exchange”; the “equi,” their intent of being neutral and not competing with their customers. According to the arrangement made before Adelson left Digital, Equinix was to acquire the PAIX from Digital as its first location. But that never happened. In a turn of events that Adelson always thought of as a betrayal, the private deal turned into an open auction, and the PAIX slipped through his fingers.

  But the loss didn’t just change the fledgling Equinix’s business strategy, it indelibly shifted the geography of the Internet. On the assumption that Equinix had the West Coast covered, at least to start, Adelson focused his attention on Virginia, where MAE-East was still the traffic-snarled center. There were basic macro advantages to this move. The sheer geographic size of North America made it inefficient to send data back and forth across the country, especially multiple times. The fifty-millisecond trips added up, noticeably slowing things down. Compounding the problem, most intra-European Internet traffic was coming to the United States to move between networks; the regional centers were still in the future. The four thousand miles from Paris to Washington was far enough, without adding another twenty-five hundred across the continent. The East Coast needed a hub—and a more efficient one than MAE-East.

  Zooming in, Adelson saw that the obvious way to compete would have been to put a new Equinix building right into Tysons Corner. But that wasn’t an option. The place “had been in telco hell for too long already,” Adelson recalled. The surrounding roads had been dug up so many times that Fairfax County planning officials were sick of it. But Loudon County, farther from DC, was still mostly farmland in the shadow of Dulles airport. And county officials wanted in on the action. Adelson remembers the big poster in the lobby of the county office showing a fistful of telephone cables backlit with purple light and the hopeful slogan “Where the fiber is.” Fiber was what Equinix needed—lots of it, and from multiple carriers, as at the PAIX. It was the sunlight in the greenhouse. Loudon County officials were eager to help the company get it, even going so far as to offer Equinix the rights-of-way needed to “trench in”—literally dig a hole—to the front door of the building. And this time Adelson knew he wouldn’t have to lie to the carriers to get it. The PAIX had quickly become a gold mine for them, and Equinix was offering the same formula but on a larger scale. The timing couldn’t have been better. The broadband rush was on, with billions of dollars of investment being made to build multiple major new nationwide fiber-optic networks.

  For help selecting a site, Adelson hired a construction company fresh off one of those fiber builds and had its employees bring their maps along with them to the job. Together, they zeroed in on a small parcel of land wedged between Waxpool Road and the disused Washington & Old Dominion railroad tracks, about three miles from the tip of Dulles’s runways, in the unincorporated town of Ashburn. The fledgling Equinix bought the land outright. It had to own the dirt. The building Adelson had in mind could not be moved down the block a few years on. Once in place, it would be a delicate and immovable ecosystem, like a coral reef formed out of the steady accretion of networks.

  But at the time the place was just empty—or at least that’s how it felt to Adelson after the constraints of Palo Alto. The PAIX had been limited (and still is to this day) by its location downtown. But Ashburn amounted to a declaration of the Internet’s manifest destiny. The network of networks would no longer be beholden to legacy telephone infrastructure wedged into crowded cities. Instead, the Internet could expand into America’s virgin countryside, where the room for growth appeared limitless.

  Today, Ashburn, Virginia, is a small town that Internet people think of as a giant city. They toss around “Ashburn” as if it were London or Tokyo, and often in the same sentence. Equinix’s unmarked complex sits behind an Embassy Suites hotel, no larger or any less nondescript than the small warehouses and light industrial buildings up and down the block. On the hot June day I first visited, a maintenance worker wearing a surgical mask swept the empty sidewalk. Jetliners buzzed by low overhead. Heavy-duty power lines hemmed in the horizon. The surrounding neighborhood was so new that when I tried to drive around the block, the unblemished blacktop streets soon gave way to gravel. My GPS showed me to be crossing open ground. The map hadn’t caught up to the sprawl. On either side of the road, industrial-sized driveways led fifty feet into lush green meadows and then stopped, as if awaiting further instructions.

  On another visit a year later things had changed: the Embassy Suites was still there, along with the Christian Fellowship Church in a big-box building next door, like a Home Depot. But the empty meadows on the far side of the tight little Equinix campus had been filled in with what looked like two beached aircraft carriers. These were massive new data centers built by a competitor, DuPont Fabros, in an explicitly parasitic arrangement—a Burger King across the street from a McDonald’s. This was a clue as to Ashburn’s particular importance. It was the extreme logical opposite of the Inte
rnet’s standard proposition, and the evolution of Adelson’s initial insight: if most days we count on the Internet to let us be anywhere, Ashburn had indeed become an utterly unique place on earth—a place worthy of pilgrimage.

  When I showed up, I had trouble finding the door. Equinix had grown to fill six single-story buildings at the time I visited; by early 2012, four more had been added, totaling more than seven hundred thousand square feet—about the size of a twenty-story office building—all tightly arranged around a narrow parking lot. I saw no proper entrance to speak of and no signs, only blank steel doors that looked like fire exits. But the parking lot was full, and I followed a guy into the security lobby of what turned out to be the wrong building. When I finally found Dave Morgan, the director of operations for the complex, he saw no reason to apologize. On the contrary, confusion was his goal: customers are reassured by the anonymity of the place “except maybe on their first visit.” Then he shared a handy tip for the next time I found myself similarly lost on the way to the Internet: look for the door with the ashtray next to it.

  The lobby was dramatically lit with halogen spotlights. There was executive waiting room furniture, a pair of uniformed security guards nestled behind bulletproof glass, and a big TV tuned to CNN. Troyer was waiting inside. He’d flown in from California to give a proper tour. He had experienced this place in his previous job as a network technician at Cablevision, a New York–area cable company (and owner of my squirrel-chewed wires). Cablevision had always been ahead of the curve offering high speeds to its customers, which meant it had a lot of Internet traffic to move. It was Troyer’s task to move it as efficiently—and cheaply—as possible. He extended Cablevision’s backbone here from New York, to connect directly to as many different networks, and thereby reduce the amount the company was paying the middlemen, or “transit” networks, to do it for them. It was cheaper for Cablevision to lease its own “pipe” all the way to Virginia than to depend exclusively on the local options in New York (of all places). The Internet’s geography is indeed particular. (Not that there was any trench digging involved; the company merely leased capacity on an existing fiber pathway.) “That’s the point of view for most major network service providers,” Troyer explained. “‘Where can I send my network data physically—geographically—in order to get the most vectors?’ Or, to go back to the pipe analogy, ‘Where can I drag my data to where there are the most pipes available to send them on the shortest path possible?’ ” This was the identical issue faced by the guys at the Tortilla Factory (just down the road from here) when they decided to move into MAE-East. And it was the identical issue Adelson faced while at Netcom. For all of us sitting in front of our screens, the Internet only works because every network is connected, somehow, to every other. So where do those connections physically happen? More than most anywhere else, the answer is “Ashburn.”