The Unicorn Project

by Gene Kim

  After all, it’s better to take their order and ship it slowly as opposed to giving them an error page.

  The team lead says, “Give us ten or fifteen minutes to push the code change out. I’ll keep you posted,” and runs from the room.

  Ten minutes later, Maxine is pacing, waiting for the fulfillment team to announce that they’re pushing their fix into production. When that happens, everyone will high-five each other and celebrate. She’s still waiting when someone yells out, “Wes! Web server page requests are timing out and front-end servers are crashing! These aren’t ‘404’ errors. Two servers are actually rebooting and clients are starting to get ‘unable to connect’ errors!”

  Maxine looks at the dashboards and is shocked at what she sees. The entire web server farm is pegged at a hundred percent CPU utilization, with some of them X’ed out because they have hard-crashed. Page load times have gone from 700 milliseconds to over twenty seconds, which is basically forever, and still climbing.

  This means some people going to their webpage won’t see anything at all because the requests for the page are not being fulfilled.

  Wes is staring at the graphs too, and then attempts to load the webpage on his phone. “Confirmed. Nothing is loading in my mobile browser. Web server team, what’s going on?” he hollers out.

  “They’re in the next room,” Kurt says. “I’ll go find out.” Maxine follows.

  In the next ten minutes, they learn just how bad the problem is. A record number of people are hitting the e-commerce site. They had anticipated this, which is why Brent had blasted their site with a homemade bot army, making sure they could actually handle such a heavy load.

  But apparently, they missed something important. They hadn’t tested for real customers coming to their site, who were presented product recommendations based on their customer profile. This was a new component they had created in the last week. The component wouldn’t render for bots, only for actual customers who were logged in.”

  As real customers hit their sites, this component made a bunch of database lookups from the front-end servers, which were never tested at this scale. Now those front-end servers are crashing under the load like a house of cards.

  “I need ideas on how to keep those front-end servers alive, I don’t care how crazy!” Wes says from the front of the room. The enormity of the problem is clear to everyone. Seventy percent of all incoming traffic is through the web. The largest portion of their order funnel is still the web, and if that stays down, all Black Friday goals will go down with it.

  “Get more servers into rotation?” someone says. Wes responds immediately, “Do it! No, Brent, not you, you stay here. Get someone assigned to it … Other ideas, people?”

  More ideas come out, and almost all of them are shot down. Brent says, “The recommendation component is what’s causing the unusual server load. Can we disable it until traffic dies down?”

  Maxine groans inwardly. They had worked so hard to get it working, and now they might have to tear it out to keep the site running.

  “Interesting. Well, can we or can’t we?” Wes says, asking the room.

  A group of managers and technical leads huddle with Maxine and Brent, and they quickly brainstorm ideas. They finally decide to just change the HTML page, commenting out the recommendations component. A brute force approach that Maxine appreciates, because no code changes are needed. The front-end team lead says, “We can change the HTML page and push it out to all the servers within ten minutes.”

  “Go!” Wes says.

  Maxine watches over the shoulders of two engineers as they carefully modify the HTML file. They’re careful because a mistake in the HTML can break the website as thoroughly as any code change. When he’s done, they review it together, commit the change into version control, and initiate the push into production.

  They’re surprised when there is no impact to the front-end performance, even after three minutes. They keep waiting to see a change, but servers keep dying. “What’s going on? What did we miss?” the engineer says, obviously trying to stay calm, confirming over and over that his modified HTML file is being loaded in the browser.

  “I can see your changes in the HTML served by the site,” Maxine announces loudly. “There must be another path that displays the recommendations component?”

  Wes is watching from behind them. “Everyone, the new HTML file has been pushed into production, but we’re still getting excessive CPU load. I need confirmation that recommendation component is still getting rendered somewhere. Give me hypotheses and ideas!”

  It takes four more minutes for them to discover that there is one more place where the component can be rendered. Maxine watches as they push another HTML file and is relieved to see that sixty seconds later the CPU load drops by thirty percent.

  “Congratulations, team,” Wes says, pausing to smile. He continues, “But that’s not enough to keep our servers up. What else can we do to reduce the load, people?”

  More ideas are proposed, more ideas are shot down, but some are jumped on immediately. The server load finally dips another fifty percent when the most common graphic images are offloaded from their local web servers and moved to a Content Distribution Network (CDN). This takes almost an hour to fully execute, but it’s enough to prevent the site from going down entirely.

  And so it goes for the rest of the day—hundreds of things going wrong, some big, some small, and never just one problem at a time. Like in their post mortems, they learn how imperfectly they understand this incredibly vast and complex system they’ve created and now must keep operating under extreme conditions.

  Hours fly by. There are moments of tired smiles and high fives as heroic acts keep everything running. The number of completed orders continues to climb, and Maxine is relieved that incoming order rates peak around three p.m., giving people a reason to hope that the worst might be behind them.

  Strangely, Maxine has a brief glimpse of Sarah looking sour on the sidelines—but even this doesn’t bother her. Maxine’s so proud of how well her teams did, handling every crisis that was thrown their way, quickly adapting and learning. And absolutely everyone knows that all this adversity is a great thing, because it is a consequence of the outrageous success of the Black Friday promotion made possible by the Unicorn Project.

  By four, it’s clear that the worst is behind them. Order traffic is still incredibly high, but off fifty percent from their peak earlier in the day. The number of failures and near-misses are down to a less bewildering rate, and people are actually starting to relax. As evidence of this, Wes is now wearing a Parts Unlimited baseball hat with a unicorn and large flames emblazoned on the side. He is laughing and joking with the people around him, handing out hats to everybody who walks by.

  Shortly before five, Maggie goes to the front of the room, and the champagne bottles and plastic cups are carried in by her staff. When everyone has a cup, she says, “What a day, everyone! We made it!”

  Everyone cheers, and Maxine drains her glass. She’s exhausted, but she can’t wait to hear the business results from everything they’ve done today.

  “This is the largest digital campaign that this company has ever done,” Maggie says. “We sent more emails today than ever. We pushed out more mobile app notifications than ever. We had the highest response rates. The highest conversion rates. We had higher e-commerce sales today than any other day in the company’s history. We will likely have the highest margin on sales today than any single other day. How’s that for the Unicorn Project pulling through?”

  Maxine laughs uproariously and cheers loudly, along with everyone around her.

  Maggie continues, “It’s going to take days to get final numbers, but you can see on the screen behind me that we’ve booked over $29 million in revenue today alone. We blew away last year’s sales record by a mile!”

  Maggie looks around the room for a moment, cheers again, and then says slowly, “This is a watershed moment for Parts Unlimited. This is what we’
ve been reaching for, for years. This shows that even horses can do unicorn-like things. Trust me, this will turn a lot of heads, and our job now is to start dreaming bigger. We’ve shown what an incredible business and technology team working together can do, and we’ve got to elevate the dreams, goals, and aspirations of our business leadership.

  “Bigger and better things are yet to come, everyone,” she says. “But in the meantime, we’ve all earned the right to celebrate. Uh, that is, when Wes says it’s safe for us to celebrate. Kurt and Maxine, get on up here and say a couple of words.”

  Kurt joins Maggie, laughing as he beckons to Maxine to join him at the front of the room. “Here’s to a kick-ass technology team that supported the Promotions team! We took a bunch of risks, and we did things that have never been done before in this company. As Maggie just said, we have a chance to make a material difference to the performance of the company.”

  Kurt turns to Maxine, obviously expecting her to say something. Maxine looks at everyone for a moment. “I’m so proud to be a part of this effort. Kurt is right in that we all took a bunch of risks to get here, and I think we’ve all learned so much in this journey. I can’t believe how much we’ve gotten done since I was first exiled to the Phoenix Project only a couple of months ago. Working on the Unicorn Project has been one of the most rewarding and fun things I’ve ever done, and I’ve never been as proud as I am today.

  “And I can’t wait to celebrate with all of you tonight, because I heard that Kurt is buying drinks at the Dockside. But there is one thing I need to say,” she says, waiting for people’s cheering to quiet down. “As awesome and amazing as what we pulled off today was, we’re a long way from being done. We’re basically Blockbuster, who just figured out how to do paper coupon promotions. If you think that’s enough to save Parts Unlimited, you must be smoking something.

  “Maggie’s right. We’re just at the beginning of our real fight. We haven’t blown up the Death Star yet. Not by a long shot. It’s still out there. What we did today was we finally figured out how to fly our X-wings. Our world is still in grave danger,” she continues. “But we finally have the tools, the culture, the technical excellence, and the leadership to win the fight. I can’t wait for the next chapter to prove that we’re not a Blockbusters or Borders, Toys“R”Us or Sears. We’re in it to win it, not to be another causality of the Retail Apocalypse!”

  Having said what she wanted to say, Maxine looks up and sees the shocked looks on everyone’s faces. Oops, Maxine thinks, realizing that she maybe should have saved that speech for a private conversation at the Dockside. Then she hears Maggie say, “Holy cow, Maxine is so right! I’m totally going to use that line with Steve and Sarah. I can’t wait for Round 2!”

  Everyone laughs and then people start applauding and cheering, Maggie loudest of them all. Although at the mention of Sarah, Maxine looks around, puzzled. She’s nowhere in sight. This is a very bad sign, Maxine thinks. Usually she’d be here to claim credit. Or pounce on someone if something went badly wrong. But Maxine is feeling too exhilarated to really care.

  Kurt and Maxine are the first ones at the Dockside. They push a bunch of tables together and pre-order a bunch of pitchers of beer for the large group that will be assembling there soon. Kurt looks squarely at Maxine. “By the way, this is a great time for me to tell you how much I appreciate everything you’ve done. We couldn’t have done this without you … the Rebellion changed when you arrived on the scene.”

  Hearing this, Maxine smiles. “You’re welcome, Kurt! We make a great team. And I’m so grateful that you sucked me into all of this.”

  She sits down as people start to file in and takes a sip of her wine, thoroughly enjoying it. She discovered a couple of weeks ago that Erik had instructed the bartender to always serve her from a special stash of wine from a vineyard owned by a friend of his.

  She looked into buying a bunch of bottles but balked when she found out how much it cost. Apparently, Erik drastically subsidizes the cost for her here. She bought one bottle for her and her husband to drink on a special occasion.

  As if he knew she was thinking about the wine, Erik arrives, grabbing the seat next to her. “Congratulations to you both—you did terrific today. Now, you need to show Steve and Dick how the future requires creating a dynamic, learning organization where experimentation and learning are a part of everyone’s daily work. It’s funny, when Steve was VP of manufacturing, he was very proud that hundreds of plant worker suggestions were put into production to improve safety, to reduce toil, to increase quality, and to increase flow. That too is also a form of continual experimentation. You now need it at a much larger level, liberated from the tyranny of project management and functional silos.

  “The Fifth Ideal is about a ruthless Customer Focus, where you are truly striving for what is best for them, instead of the more parochial goals that they don’t care about, whether it’s your internal plans of record or how your functional silos are measured,” he says. “Instead we ask whether our daily actions truly improve the lives of our customer, create value for them, and whether they’d pay for it. And if they don’t, maybe we shouldn’t be doing it at all.”

  Erik gets up, and one of the bartenders arrives with a newly opened bottle of wine. Erik takes it and places it in front of Maxine with a wink. “Congratulations, Maxine. Catch y’all later tonight!”

  He leaves just as six more of their teammates walk through the door. Maggie turns to Maxine and Kurt and asks, “What was that all about?”

  “I’m trying to figure that out myself,” Maxine says. “But it’s nothing that can’t wait until next week. Maybe we can find a moment to talk later tonight … But in the meantime, let’s celebrate!”

  The next morning when Maxine wakes up, her head is pounding. On top of the Dockside celebration, she and her husband had a couple more drinks while watching their favorite TV series late into the night. She didn’t actually remember falling asleep, such was her sudden exhaustion.

  She wants to go back to sleep on this Saturday morning, but she scans her phone. There is some chatter in the chatrooms about an ongoing issue in the stores. Apparently, store managers are having problems because of overwhelming demand for promoted items. They were completely out of stock, and it was taking them fifteen minutes per customer to create rain check orders, having to key each one into another clunky in-house ordering system.

  The in-store app teams were dispatched to the stores to figure out how to speed things up. Someone thinks they could write a simple tablet app to simplify the process. Maxine likes the idea, and she has full confidence that they’ll come up with a fix that will delight the in-store managers and staff.

  She smiles, satisfied that this problem could be solved without her. Over the past month, she has grown to trust and respect her teammates and appreciate what they’re doing.

  Maxine grins as she looks at the tickets to Comic-Con that Jake bought yesterday for her and the whole family.

  She smells bacon and eggs. Jake must be making breakfast, she thinks. Maybe she can go back to sleep after eating. Life keeps getting better.

  CHAPTER 16

  • Friday, December 5

  One week later, Maxine is sitting in the fanciest conference room she’s ever seen. She’s in Building 2, where all the top executives have their offices. The building is nearly seventy years old, one of the oldest and tallest on the corporate campus, complete with wood-paneled walls.

  For Maxine, it’s surreal to look at who’s sitting at the table. Never before has she been in a meeting with so many top executives. At the head of the table are Steve, Dick, Sarah, and three other executives she doesn’t recognize. This is the first time that she’s been in the same room with Steve and Dick, outside of the Town Halls.

  Maxine is surprised that Erik is sitting at the table too. Steve and Dick don’t give him much notice, as if they’re used to having him around.

  While Maggie is at the front of the room getting ready to present, Maxine s
tares around the room representing the opulence of a bygone era. She feels like she needs to tell Kurt not to touch any of the paintings or steal any of the fancy things hanging on the walls.

  It’s like the bridge crew decided to invite the red shirts from Engineering into the captain’s quarters for tea, asking for their advice on how to deploy their starship fleet.

  Which is, it occurs to her, what’s actually happening. Maggie’s briefing the top company executives on the blindingly awesome success of the Unicorn Project, the looming remaining threats that Maxine had warned about, and to present their proposals.

  When Steve nods, Maggie begins her presentation, reviewing the amazing Black Friday statistics. Even though Maxine has seen her present before, she’s still blown away by Maggie. She’s on fire, brilliantly describing what the team did and the amazing business outcomes they generated.

  “… All that activity resulted in what you’ve seen in these official revenue numbers. Because of the Black Friday campaign, we posted nearly $35 million of revenue on top of our run-rate business. Almost all of those orders came in through the web or our mobile apps,” she says. “For a variety of reasons, we believe this is mostly additive revenue. In other words, revenue that wouldn’t have been attained were it not for these campaigns. Which was the result of thousands of experiments, analyzing our customer base in ways we’ve never been able to before. This was made possible by the five incredible technology platforms we created and using all the data in our systems to make fantastic predictions on which promotions would most effectively drive sales.

  “We moved inventory that had been sitting on shelves, often for a year or more, freeing up badly needed working capital,” she says. “Looking forward, assuming we can generate some equally exciting promotions over the holidays, I’m thinking we could be looking at $70 million of upside revenue. That’s twenty percent above what we’ve been signaling to analysts and Wall Street.”