by Gene Kim
Randy turns to Josh. “Tell me more about the build failures.”
Maxine listens. All those stories about caveman technical practices in the Phoenix Project are actually true. She’s learned over her entire career that when people can’t get their builds going consistently, disaster is usually right around the corner.
She looks around at the entire floor. Over a hundred developers are typing away, working on their little piece of the system on their laptops. Without constant feedback from a centralized build, integration, and test system, they really have no idea what will happen when all their work is merged with everyone else’s.
Josh spins his chair around to Maxine. “Mrs. Chambers, I’ve got to go show Randy something, but I just emailed you what we’ve got in terms of documentation for new developers—there are wiki pages where I’ve assembled all of the release notes we’ve written and the documentation from the development teams. There’s also links to the stuff we know we need to write. Hopefully that will get you started?”
Maxine gives him a thumbs up. As they leave, she logs in with her new laptop and is able to get in and open her email, miraculously working the first time. But before looking at what Josh sent, she pokes around to see what else is on her new laptop.
Immediately, she is mystified. She finds links to HR systems, network shares to company resources, links to the expense reporting system, payroll, timecard systems … She finds Microsoft Word and Excel and the rest of the Office suite.
She frowns. This is fine for someone in finance, she thinks, but not a developer. There are no developer tools or code editors or source control managers installed. Opening up a terminal window she confirms that there aren’t any compilers, Docker, Git … nothing. Not even Visio or OmniGraffle!
Holy cow! What do they actually expect new developers to do? Read emails and write memos?
When you hire a plumber or a carpenter, you expect them to bring their own tools. But in a software organization with more than one developer, the entire team uses common tools to be productive. Apparently here on the Phoenix Project, the toolbox is empty.
She opens her email to see what Josh sent. It takes her to an internal wiki page, a tool many engineers use to collaborate on documentation. She tries to scroll up and down the wiki page, but the document is so short there isn’t even a scroll bar.
She stares at the nearly empty screen for several long moments. Screw you, Chris, she thinks.
Driven by morbid curiosity, Maxine spends the next half hour digging. She clicks around and finds only a handful of documents. She reads PowerPoint slides with architecture diagrams, lots of meeting notes and Agile sprint retrospectives, and a three-year-old product management requirements document. She is excited when she finds tantalizing references to some test plans, but when she clicks on the links, she is prompted by an authentication screen asking for her login and password.
Apparently she needs access to the QA servers.
She opens a new note file on her laptop and types a note to herself to find someone who can give her access.
Giving up on documentation for the moment, she decides to find the source code repositories. Developers write code, and code goes into source control repositories. There are developers working on Phoenix, ergo, there must be a Phoenix source code repo around here somewhere, she thinks.
To her surprise, despite almost ten minutes of searching, she can’t find it. She adds to her notes:
Find Phoenix source code repo.
She finds links to internal SharePoint documentation servers, which may have more clues, but she doesn’t have accounts on those servers either.
She types another note:
Get access to DEVP-101 SharePoint server.
For the next hour, so it goes—Search. Nothing. Search. Nothing. Search. Click. Authentication screen. Click. Authentication screen.
Each time, she adds more notes to her growing list:
Get access to QA-103 SharePoint Server.
Get access to PUL-QA-PHOENIX network share.
Get access to PUL-DEV-PHOENIX network share.
She adds more notes and to-dos, accumulating a list of more user accounts that she needs, adding the QA wiki server, the performance engineering wiki server, the mobile app team wiki, and a bunch of other groups with acronyms she doesn’t recognize.
She needs network credentials. She needs installers for all the tools that are mentioned. She needs license keys.
Maxine looks at her watch and is surprised to see that it’s nearly one o’clock. She’s achieved nothing in two hours except document thirty-two things she needs. And she still doesn’t know where the development tools or the source code repositories are.
If the Phoenix development setup were a product, it would be the worst product ever.
And now she needs food. She looks around, and seeing the nearly empty floor, realizes that she missed the lunch rush.
It would have been nice if she had followed them, but she had been too engrossed in digging through the labyrinth of Phoenix docs. Now she doesn’t know where people find food. She wonders if she should add that to her list too.
Right after “update and send out my résumé.”
From:
Alan Perez (Operating Partner, Wayne-Yokohama Equity Partners)
To:
Steve Masters (CEO, Parts Unlimited)
Cc:
Dick Landry (CFO, Parts Unlimited), Sarah Moulton (SVP of Retail Operations), Bob Strauss (Board Chair, Parts Unlimited)
Date:
6:07 a.m., September 4
Subject:
Go Forward Options, January Board Session **CONFIDENTIAL**
Steve,
Good seeing you two days ago in Elkhart Grove. As a newly elected board director, I’ve been learning a lot and appreciate the time being invested by the management team to get me up to speed. I’ve been especially impressed with Dick and Sarah (CFO and SVP Marketing, respectively).
Although I’m new, it’s clear that Parts Unlimited’s failed efforts to increase shareholder value have raised questions of confidence and created need for action. We must work together to break the string of broken promises repeated quarter after quarter.
Given how essential software is to your plans, your decision to replace your CIO and VP of IT Operations seems proper—hopefully this will restore accountability and increase urgency in execution.
To reiterate my motivation for reviewing strategic options at the board level: revenue growth isn’t the only way to reward shareholders—we’ve put so much focus into forcing Parts Unlimited to become a “digital company” that I believe we’ve lost sight of low risk ways to unlock value, such as restructuring the company and divesting non-core, poor-performing assets. These are just two obvious ways to increase profitability, which increases shareholder value and provides working capital for transformation.
We need to quickly assemble options for the board to review and consider. Given how much time management is spending on the current strategy, the board chair asked me to work with a few key members of the executive team to generate options for the board to discuss. I will work with Dick and Sarah, given their tenure and breadth of experience at the company. We’ll have bi-weekly calls to discuss and assess ideas, and we’ll be ready to present strategic options to the entire board in January.
Our firm bought a significant interest in Parts Unlimited because we believe there’s considerable shareholder value that can be unlocked here. I look forward to a productive working relationship and improved outcomes for Parts Unlimited that we can all be proud of.
Sincerely, Alan
CHAPTER 2
• Friday, September 5th
Maxine scans her to-do list, slowly shaking her head in frustration. It’s been two days and she is determined to perform a Phoenix build on her laptop, like any new developer should be able to do. This has become her mission. But according to her list, there are over a hundred items that she’s missing, and no one seems
to know where to find them.
She has done nothing on her list. Except for updating and sending out her résumé. Many friends replied to her right away, promising to look for positions she might be interested in.
Maxine asked her guide, Josh, about all those missing build items, but he didn’t know anything about them. The build team used to know these things, but the details are either out-of-date or missing entirely, the knowledge scattered across the entire organization.
She is frustrated, every turn she takes leads to a dead end. There is nothing fun about this challenge. What she is doing, she’s pretty sure, is the exact opposite of fun.
She’s an engineer at heart, and she loves challenges and solving problems. She’s been exiled smack in the middle of probably the most important project in the entire company’s history. And somewhere, there’s code—almost certainly millions of lines of code written by hundreds of developers over nearly three years. But she can’t find any of it.
Maxine loves coding and she’s awesome at it. But she knows that there’s something even more important than code: the systems that enable developers to be productive, so that they can write high-quality code quickly and safely, freeing themselves from all the things that prevent them from solving important business problems.
Which seems to be completely missing here. Maxine is one of the best in the game, but after four days she still has almost nothing to show for it. Just endless clicking around, reading documents, opening tickets, scheduling meetings with people to get things she needs, trapped in the worst scavenger hunt ever.
For a moment, Maxine wonders if she’s the only person having this problem. But she sees developers all around her struggling, so she quickly pushes away any feelings of self-doubt.
Maxine knows her kung fu is amazing. Many times in her career she’s had to solve problems that seemed hopeless and impossible. Often in the middle of the night. Sometimes without any documentation or source code. One of her most famous escapades is still known as the “Maxine Post-Holiday Save,” where all the in-store systems that handled refunds crashed spectacularly on the Friday after Christmas. It’s one of the busiest shopping days of the year as people come in to return gifts from loved ones so they can buy something they actually want.
With her team, Maxine worked into the wee hours of Saturday morning to fix a multi-threading deadlock in a database vendor’s ODBC driver. She had to manually disassemble the vendor library and then generate a binary patch. By hand.
Everyone said it couldn’t be done. But she pulled it off, to the amazement of the scores of people who had worked that outage for over seven hours. The database vendor professional services team was in awe and immediately offered her a job, which she politely declined.
The legends about her kept growing after that. She’s classically trained as a developer, and in her career, she’s written software to stitch together panoramic graphic images and chip layout algorithms for CAD/CAM applications, back-end servers for massive multi-user games, and, most recently, the ordering, replenishment, and scheduling processes that orchestrate thousands of suppliers into a plant production schedule for their MRP systems.
She routinely lives in the world of NP-complete problems that are so difficult to solve they can take more than polynomial time to complete. She loves the Papers We Love series, revisiting her favorite academic papers from mathematics and computer science.
But she has never seen her job as just writing application code, working only pre-deployment. In production, when theory meets reality, she’s fixed wildly misbehaving middleware servers, overloaded message buses, intermittent failures in RAID disk arrays, and core switches that somehow kept flipping into half-duplex mode.
She’s fixed technology components that were spilling out their guts in the middle of the night, having filled up every disk and log server, making it impossible for teams to understand what was actually happening. She led the effort to systematically isolate, diagnose, and restore those services based on decades of intuition and countless production battles.
She’s deciphered stack traces on application servers that were literally on fire, racing to get them safely backed up before the flooding water, halon extinguishers, and emergency power shutdowns destroyed everything.
But deep down, she’s a developer. She’s a developer who loves functional programming because she knows that pure functions and composability are better tools to think with. She eschews imperative programming in favor of declarative modes of thinking. She despises and has a healthy fear of state mutation and non-referential transparency. She favors the lambda calculus over Turing machines because of their mathematical purity. She loves LISPs because she loves her code as data and vice versa.
But hers is not merely a theoretical vocation—she loves nothing more than getting her hands dirty, creating business value where none thought it could be extracted, applying the strangler pattern to dismantle decades-old code monoliths and replacing them safely, confidently, and brilliantly.
She is still the only person who knows every keyboard shortcut from vi to the latest, greatest editors. But she is never ashamed to tell anyone that she still needs to look up nearly every command line option for Git—because Git can be scary and hard! What other tool uses SHA-1 hashes as part of its UI?
And yet, as awesome as she is, at the height of her soaring powers and skills honed over decades, here she sits in the middle of the Phoenix Project unable do a Phoenix build, even after two days. She found where two of the four source code repositories are, and she’s found the three installers for some of the proprietary source code management (SCM) tools and compilers.
However, she is still waiting for license keys for the SCM, and she doesn’t know who to ask to get license keys for the two other build tools. She needs credentials for three network shares and five SharePoints, and no one knows where to get the ten mysterious configuration files mentioned in the documentation. When she emailed the person who wrote the docs, it bounced. They had long left the organization.
She is stuck. No one responds quickly to her emails, her tickets, or her voicemails. She’s asked Randy to help, to escalate her requests, but everyone says it will take a couple of days because they’re so busy.
Of course, Maxine never just takes “no” for an answer. She has made it her mission to do whatever it takes to get a build running. She’s hunted down almost all of the people who have promised her something. She’s found out where they sit and has pestered them, even camping out at their desks, willing to stay there until they get her what she needs.
Sometimes she got what she needed: a URL, a SharePoint document, a license key, a configuration file. But more often than not, the person she hunted down didn’t have what she needed—they would have to ask somebody else, so they would open up a ticket on Maxine’s behalf. And now they were both waiting.
Sometimes, they had a promising lead or clue on who or where Maxine needed to go to next. Most times, though, it was just a dead end, and she was right back where she started.
Trying to get a Phoenix build going is like playing Legend of Zelda, if it were written by a sadist, forcing her to adventure far and wide to find hidden keys scattered across the kingdom and given only measly clues from uncaring NPCs. But when you finally finish the level, you can’t actually play the next level—you have to mail paper coupons to the manufacturer and wait weeks to get the activation codes.
If this really were a video game, Maxine would have already quit, because this game sucks. But Phoenix isn’t a game—Phoenix is important, and Maxine never quits or abandons important things.
Maxine sits at her desk looking at the calendar she’s printed out and pinned to the wall.
She turns back to her computer and runs her finger down her ever-growing list of to-dos again—each item a dependency she requires to get her build going.
She just added two more SharePoint credentials she needs to get from two different Dev managers who, for some reason, run their own A
ctive Directory domains. They’re rumored to contain some critical build documentation with some of the information she seeks.
Randy sent her a ton of Word docs, Visio diagrams, and marketing PowerPoint presentations, which she quickly skims for clues. They may be helpful to marketing people and architects, she supposes, but she’s an engineer. She doesn’t want to see brochures for the car they’ve promised to build—she wants to see the engineering plans and the actual parts that they’re going to assemble the car from.
These documents might be useful to someone, so she posted them on the wiki. Moments later, someone she doesn’t know asks her to take them down because they might contain confidential information.
Looking further down her to-do list, she reads:
Find someone who can give me access to Dev or Test environments.
These were referenced in some of the documentation she read yesterday, but she has no idea who to ask to get access.
She has crossed off one item:
Get account for integration test environment.
This was less satisfying than she had hoped. She poked around the environment for two hours, trying to gain an understanding of the giant application. But in the end, she found it too bewildering—it was like trying to picture the layout of an enormous building by crawling around air ducts without a map or a flashlight.
She types out a new to-do:
Find someone who’s actually doing integration testing so I can shoulder-surf while they work.
Watching someone use the Phoenix applications might help orient her. She’s baffled that no one knows of an actual person who uses Phoenix. Just who are they building all this code for?
Scanning her to-do list again, she confirms that she in fact has nothing to actually do—she has already pestered everybody today and now she’s just waiting for people to (not) get back to her.
It’s Friday, 1:32 p.m. Four and a half hours to go until five, when she can finally leave the building. She tries hard not to sigh again.
