Amazon Unbound

by Brad Stone

  As the project neared a hoped-for introduction in mid-2015, Amazon anonymously leased the ground floor of a new luxury apartment building in Seattle’s wealthy Capitol Hill neighborhood. Permits filed with the city included plans for a large produce department, dairy coolers, and an on-site kitchen for preparation of fresh foods. The Go team then sought Bezos’s sign-off. It would be a typical “Jeff meeting”—the documents endlessly rewritten and polished, the day choreographed to within an inch, and everyone holding their breath and hoping that things didn’t go haywire.

  To show off the concept, they leased a warehouse in south Seattle, near Starbucks headquarters, and converted part of the ground floor into a fifteen-thousand-square-foot mock supermarket. Plywood faux walls conveyed the perimeter, modular shelves could be moved around, and turnstiles mimicked technology that would scan shoppers’ smartphones when they walked in. Bezos and members of the S-team arrived and sat around an imported conference table to peruse the six-page document.

  Bezos is usually one of the slowest readers in these sessions; he seems to carefully consider every sentence. But this time, halfway through, he put down the document and said, “You know what, let’s go shop,” and led the S-team into the dummy store. They pushed grocery carts down the faux aisles with shelves stocked with canned food and plastic fruit and vegetables. Go employees posed as the baristas, butchers, and cheesemongers, taking orders and adding items to their imaginary bills.

  It seemed to go well, and afterward Bezos gathered the group in the makeshift conference room. He told them that while they all had done a fine job, the experience was too complicated. Customers would have to wait in line for meat, seafood, and fruit to be weighed and added to their bill, which contrasted with the store’s major selling point: the absence of time-wasting queues. He felt the magic was walking out without waiting—the physical equivalent of Amazon’s famous one-click ordering—and wanted to focus the effort on that, with a smaller and simpler experience. “It was one of those Amazon things, ‘We love it—let’s change everything!’ ” recalled Kristi Coulter, who worked on the project as a brand designer.

  Steve Kessel reconvened the Go team in their offices and broke the news: they would be losing the fresh produce, meat, and cheese, and pivoting to a much smaller convenience store format. For the next five years, the storefront on Capitol Hill, where they were going to build the midsized supermarket, would sit abandoned, in the heart of one of Seattle’s most well-trafficked neighborhoods, its windows mysteriously covered in brown paper.

  * * *

  By the start of 2016, the Amazon Go project had reached a critical juncture: the path ahead was going to be difficult and expensive. Kessel called another meeting of Go execs and asked whether they thought they should proceed, move the project into a gestational R&D phase, or cancel it. While a few execs were skeptical, the consensus was to push forward.

  Some of the engineers were relieved that they were reducing complexity in the stores by eliminating items with variable weights and prices, like steaks. Others were exhausted from two years of nonstop work and felt captive to an abrasive personality who pushed them relentlessly, generating numerous sprints toward artificial deadlines even as the race started to resemble a grueling marathon. Surprisingly, this time it was not Bezos but the executive who had worked by Bezos’s side and exhibited some of the trademark characteristics that executives often need to be successful at Amazon, Bezos’s former TA, Dilip Kumar.

  Kumar was from Salem, India, the son of a three-star general in the Indian Army. His family moved around when he was a child and spent about “two years everywhere,” as he put it. Kumar attended the prestigious Indian Institute of Technology and moved to the U.S. in ’94 to get a master’s degree in computer science and engineering from Penn State University and an MBA from Wharton. He joined Amazon in 2003, as it was still staggering after the dot-com bust. Over the years, Kumar relieved stress by locking himself in a conference room and teaching himself to juggle and later by performing stand-up comedy at local open-mic nights.

  Kumar inhabited a few aspects of the Amazonian leadership template forged by Bezos in his younger, more tempestuous years as CEO: hard-driving, maniacal about the customer, IQ over EQ, raw force of will over innate leadership ability. Colleagues said that Kumar had a remarkable memory and ability to recall even complex technical details. They also say he created an environment in which there were no other options but to succeed. Like every leader at Amazon, he could handily recite the company’s fourteen leadership principles—and like his boss, believed that the only way to get to good decisions was to passionately debate hard problems. “If I have to choose between agreement and conflict, I’ll take conflict every time,” Bezos often said. “It always yields a better result.”

  But unlike Bezos, Kumar was a skillful wielder of profanity in the workplace, colleagues said. They recall such epic shouting matches between him and Gianna Puerini that once Steve Kessel had to step in. Though instead of breaking up the dispute, Kessel asked them to fight more quietly. “If he was treating you nicely, it meant you were not important,” said a senior scientist on the project. Added Bali Raghavan, “He’s an intense guy to work for, and it used to drive me nuts. He also brings out the best in people.” Of course, the same thing was often said of Bezos as well.

  As Go execs decided to move forward, Kumar would need every bit of drive and prickly ingenuity. After the ill-fated demo to Bezos and the S-team, the Go group reduced the project to the size of a 7-Eleven-type convenience store, so that they could focus solely on the capabilities of the technology. Kumar’s engineers set up a secret lab on the ground floor of a new Amazon building, called Otter, on the corner of 5th Avenue and Bell Street. Workers could only access it from the inside, swiping their key cards to enter a pair of locked doors. The shelves were packed with fake food fashioned from clay and Styrofoam; shredded green construction paper stood in for lettuce.

  Kumar asked Go employees to visit and try to fool the overhead cameras and computer-vision algorithms. They wore heavy coats or walked in on crutches or pushed wheelchairs. One day, everyone was asked to bring in umbrellas to see if they would obscure the view of the cameras; on another, employees all wore Seattle Seahawks jerseys to confuse algorithms that partly distinguished shoppers using the color of their clothing.

  The challenge was that while the technology wasn’t fooled frequently, it was wrong often enough that it could create vexing problems if Amazon deployed it widely. Changes in lighting conditions and drifting shadows, the depth of a product’s placement on shelves, and hands and bodies that concealed customized stickers on products could easily confound the system. Toddlers were a special challenge—small, difficult for computers to distinguish from their parents, and the source of all kinds of anomalous mischief inside stores. Adults might also hoist them on their shoulders, for example, hold them in their arms, or push them in strollers, further confounding the customer identification algorithms, which determined which account would be charged.

  While Kumar and his engineers were addressing these challenges, Bezos and Kessel were getting impatient. Despite three years of work, Amazon hadn’t opened a single store. So in the peculiar fashion of invention at Amazon, they created separate teams to pursue the singular goal of bringing the company into the vast realm of physical retail. Bezos liked to say Amazon was “stubborn on vision, flexible on details,” and here was an illustration: groups working on parallel tracks would essentially compete to fulfill the “Just Walk Out” ideal and solve the problem of the cashierless store.

  Kumar’s group continued to develop a store with futuristic computer vision technology embedded in the ceilings and shelves. Meanwhile, Kessel asked Jeremy De Bonet, an Amazon technology director based in Boston, to form his own internal startup of engineers and computer vision scientists. They would end up flipping the problem around and integrating computer vision technology and sensors into a shopping cart, instead of blanketing them around the store. In
some ways, this was a harder problem. While the Go store could partially deduce the identity of an item based on where in the store it was located, a so-called “smart cart” would have to account for the possibility of a shopper selecting, say, a bag of oranges from the produce aisle but scanning them somewhere else in the store.

  This group’s efforts would also take years to develop and would culminate in several technologies that were integrated into the Go store, as well as the Amazon Dash Carts equipped with computer vision scanners and touchscreens that would allow customers to check out as they roamed the aisles of a supermarket.

  Bezos and Kessel formed a third team to pursue a more modest and immediately obtainable goal: opening bookstores with a more conventional checkout line. Books were the opposite of food—nonperishable, consistently priced, easy to stock—and, of course, the product category Amazon had pioneered online. Customers shopped for books less frequently than food, so waiting to pay was less of a hassle. And books could once again be bait, drawing in shoppers to experience devices like the Fire TV, the latest Kindles, and the new Amazon Echo.

  In the fall of 2015, as the company prepared its first Amazon Books outlet in an upscale mall in Seattle, speculation over how it would finally enter physical retail was so feverish that a reporter for the local tech blog GeekWire used a pole with a camera attached to it to peek inside. Around the same time, Bezos sneaked in through a back door to see the outlet for the first time and was delighted. He said he felt as if Amazon’s business was coming full circle.

  The store opened a few weeks later, on November 2, 2015. Employees who worked on the project got to contribute their favorite books to a “staff selections” bookshelf. Bezos himself selected three titles, and in a way, they foreshadowed an unexpected turn of events that lay ahead: Traps, a novel by his wife, MacKenzie Bezos; The 5 Love Languages, by Gary Chapman, about preserving romantic relationships; and The Gift of Fear, by his friend, the famed security consultant Gavin de Becker.

  To some longtime Go members, watching Amazon Books open within a few months was disheartening. But by the beginning of 2016, the team was finally preparing for their eventual launch. To decide on a formal name for the store, Puerini’s team conducted a series of branding exercises, brainstorming and writing tenets on what the name should communicate to the public. They settled on “Amazon Go” to convey speed. “Even the word itself is only two characters,” she said. “You can literally grab and go.”

  Inside the Otter lab, fake food was replaced with real items, and Go employees were asked to shop under specific scenarios. For example, Puerini recalled, “You’re running to a meeting: buy a salad and a drink for lunch,” or “You’re in a rush to pick up the kids from day care: grab milk, strawberries, and cereal for tomorrow morning.” Continuing to puzzle over the toddler challenge, they asked parents to bring in their kids, who fidgeted, ran around, and grabbed things, further stress-testing the system.

  Employees had mixed feelings about their progress. Many loved the convenience, reveling in the ability to run to the Otter lab before an afternoon meeting, grab a sandwich, and return, experiencing the just-walk-out magic they had once hypothesized about in the PR FAQs. But behind the scenes, the technology was not perfect and humans were needed to backstop it. Teams of employees were formed to review footage when the system wasn’t certain about a purchase, a so-called “low-confidence event.” The creation of these groups, the equivalent of the wizard-behind-the-curtain work being done by contractors that reviewed and improved Alexa’s responses, led some employees to question the entire effort. It “was a tricky thing,” said designer Kristi Coulter. “If we have an army of people looking at footage, is that scaling properly?”

  Humans had another role to play as well: to develop meal-kit recipes and prepare the daily lunch fare, such as lamb sandwiches and caprese salads. To get ready for the opening of an eighteen-hundred-square-foot prototype store on Amazon’s Seattle campus in late 2016, the company hired chefs and staff from industrial kitchens and chain restaurants. It then opened both a kitchen inside the prototype store and a commercial-grade cookhouse in south Seattle, to serve as a model for kitchens that the company planned to build around the country as part of a massive rollout of Go stores. Uncharacteristically, Amazon splurged, buying German commercial ovens that cost tens of thousands of dollars each.

  The kitchens brought with them another set of unexpected challenges. When something smelled off in the store’s kitchen, Amazon hired a pair of professional smellers to solve the mystery (the culprit: pickled daikon). Because food safety was a top priority, the commercial kitchen was kept bitterly cold, and Amazon initially declined requests from the hourly staff to install mats on the facility’s chilly concrete floor, one employee recalled. After a senior manager from headquarters spent a day observing operations at the cookhouse, the company issued the staff hoodies and other cold-weather gear and finally acceded on the mats. The people involved in the food service industry, it turned out, were proving as tricky to manage as Dilip Kumar’s algorithms.

  The original Amazon Go store opened to all employees in December 2016. The public opening was scheduled for a few weeks later but ended up being delayed a year after an entirely new series of problems surfaced. According to the Wall Street Journal, the system tended to freeze when twenty or more shoppers were in the store at the same time, and it lost track of products when shoppers picked them up and set them down on a different shelf. Clerks had to be notified to restock the items in the proper place. The system was also not perfectly accurate, even under the best of circumstances, and Amazon executives could not risk the possibility of it making mistakes, falsely charging shoppers and threatening customer trust.

  Shoppers themselves also tended to get confused by the novel format. “We noticed lots of customers hesitating at the exit, asking the entry associate if they really could leave,” Puerini said later. “In tests, we put up a big poster that said, ‘No, really, you can just walk out!’ ” A version of the sign is still there in the original store.

  When the first Amazon Go store finally opened to the public in January 2018, it was heralded as a peek into the future. (“The whole process was so quick and seamless, I almost forgot the items weren’t free,” wrote CNET.) But with the small size of the store, limited selection of items, and enormous expenses on salaries and operations, the numbers behind the project horrified finance execs. One told me that the original Go store, its adjoining kitchens and data center cost more than $10 million.

  “If you were a venture capitalist, this just did not make sense anymore,” said another executive privy to the decision-making. But Bezos wanted to forge ahead. “Jeff is master of ‘this isn’t working today, but could work tomorrow.’ If customers like it, he’s got the cash flow to fund it,” this exec said. In 2017, Amazon spent $22.6 billion on R&D, compared to Alphabet ($16.6 billion), Intel ($13.1 billion), and Microsoft ($12.3 billion). The tax-savvy CEO likely understood that these significant R&D expenses for projects like the Go store and Alexa were not only helping to secure Amazon’s future but could generate tax credits or be written off, lowering Amazon’s overall tax bill.

  Over the next few years, Amazon Go stores opened across Seattle, San Francisco, New York, and Chicago. Amazon closed its kitchens and instead bought food from the same vendors that make the middling salads and sandwiches for Starbucks and 7-Eleven. The pricey German ovens sat unused in the original store and the kitchen staff was dismissed.

  Disgruntled, the laid-off staff told tales of declining food quality and of giving away unsold meals to food banks and homeless shelters. “The only thing that’s fresh anymore is the vegetables,” groused one former employee. “It’s heartbreaking to see this project go to hell.”

  Bezos had envisioned thousands of Amazon Go stores, in urban areas around the country. Seven years into the project there were only twenty-six, hardly producing the financial results he’d had in mind when he conceived the concept. The stores also instig
ated a political backlash against Amazon for eliminating cashier jobs, the second most popular job in the country according to the U.S. Bureau of Labor Statistics. The stores also threatened to exclude low-income and older shoppers who did not have smartphones loaded with credit cards. Cities such as New York, Philadelphia, and San Francisco passed legislation mandating such stores accept cash.

  I spoke to Dilip Kumar in 2019, after he was promoted to run all of physical retail. Kumar alone remained from the original IHM trio—by then, Steve Kessel and Gianna Puerini had both retired. He insisted that it was “still early” for the Go project and noted that “customers love the experience” of walking out without stopping to pay. That, Kumar said, gave the project “a lot of latitude and degrees of freedom to be able to try other kinds of things.”

  One of those was moving the maturing technology back into midsized urban grocery stores. In 2020, right before the onset of the Covid-19 pandemic, Amazon opened its long-dormant site in Seattle’s Capitol Hill neighborhood, calling it Amazon Go Grocery. The cheese, meat, and seafood were back, and Kumar hinted that the cashierless system could work even in larger venues. “We’ve learned a lot,” he told the Wall Street Journal. “There’s no real upper bound. It could be five times as big. It could be ten times as big.”

  Amazon Go remained a money loser. But Bezos was still looking at it as a bet on computer vision and artificial intelligence, the kind of long-term, high-stakes experiment that was necessary to produce meaningful outcomes for large companies. As he wrote in his 2015 shareholder letter: