by Gene Kim
Maxine feels her heart drop. “We can’t test it ourselves?”
“We used to be able to, before we got re-orged into Phoenix,” he says, wistfully. “The QA group took over the testing. And when they had some problems with different teams using the test environment at the same time, they took everyone’s access away. Now they’re the only ones who can log in, let alone run the tests.”
“Wait,” she says. “We write the tests but can’t run them?”
He laughs. “No, no. They write the tests. They don’t even let us see the test plans anymore.”
Maxine deflates even more, knowing where this is going. “And we can’t push it into production?”
Tom laughs again. “Nope, not any more. We used to be able to do that too. But now someone else deploys it for us. ‘Stay in your lane,’ they told us.” He shrugs his shoulders. Maxine is pretty sure she knows who said, “Stay in your lane.” That’d be Chris.
The joy that Maxine felt all day while working on the problem disappears. After all, fixing the code especially for features is just a fraction of the entire job. It’s not done until that customer can use what they’ve written. And even then, it’s probably still a work in progress, because we can always learn more about how to help that customer best achieve their goals.
“Crap,” she mutters. I’m back in the same place I was before, a long way away from the First Ideal. I still can’t actually do anything myself, Maxine thinks. Once again, she is dependent on others to create customer value.
Oblivious, Tom laughs and opens up a new window. “It’s not so bad. We just need to go into the ticketing system and mark this issue as ‘done.’ That lets the QA team know to test it, so it can be promoted into production.”
Tom looks at his watch and turns back to her, “That was great. We got a lot done today. Want to pick another defect to work on?” Maxine forces a smile and nods. This sucks, Maxine thinks. She likes finishing things, not just starting things.
Maxine continues to work with Tom all day, picking the next most urgent defect to be fixed. Tom once again compliments Maxine on how she thinks about problems. He’s impressed at how she writes unit tests that can be run without the need for a complex, integration test environment.
But there are limits—Data Hub’s job is to connect systems with each other. There’s only so much you can simulate on a single laptop. It would be nice to rearchitect Data Hub so you could, Maxine thinks wistfully.
Although she enjoys learning about Data Hub and the parts of the business it connects, there’s something about all this work that is deeply unsatisfying to her.
She thinks of Erik’s Second Ideal of Focus, Flow, and Joy. All the joy she felt vaporized when Tom told her that they had completed only a small portion of the work needed to create value. That’s just not good enough for her. In her MRP team, any developer could test their own code and even push code into production themselves. They didn’t have to wait weeks for other people to do that work for them. Being able to test and push code to production is more productive, makes for happier customers, creates accountability of code quality to the people who write it, and also makes the work more joyful and rewarding.
Maxine starts thinking about how to introduce some of the tools being built by the Rebellion. At the very minimum, we need to make standardized Dev environments available, so I can do builds on my laptop, she thinks. More things to talk about at the next Dockside meeting.
She continues to grind away, helping Tom with the work that he’s been assigned. Together they fix two defects and then tackle a crash-priority feature, this time to create some business rules around extended warranty plans, critical enough to be exempted from the feature freeze.
“Why is this so high priority?” Maxine asks Tom as she reads the ticket.
“This is hugely revenue-generating,” Tom explains. “One of the highest margin products are these new extended-warrantee plans. Customers loved the pilot warrantee program, especially for things like tires. Now in-store staff need a way to pull up this information, so they can do the repair work and file the claim with the third-party insurer.”
Tom continues, “Great for the customer, great for us, and a third-party insurer is taking all the financial risk.”
“Cool,” Maxine says, perking up. It’s features like this that support everything that Steve said in the Town Hall. It’s been a long time since Maxine’s done work on the revenue-generating side of the business.
Recommitting herself to feel relentlessly optimistic, Maxine and Tom start studying the feature, trying to figure out what is required to enable this important business capability. She tries not to think about how, even if they get it done today, it’ll just sit, waiting for the QA team to test it.
The next morning, Tom and Maxine are at a whiteboard, inventorying all the systems that they’ll need to change in order to enable extended warrantees. Two more engineers have joined them as the scope keeps increasing. And then they realize that they’ll need to talk with engineers from two other teams, as well. Maxine guesses that they’ll have to bring in six other teams because of how many business systems this affects.
Maxine is dismayed as the number of teams that need to be involved keeps growing. This is again the opposite of the First Ideal of Locality and Simplicity. Here, the changes that need to be made are not localized. Instead, their scattered across many, many different teams. This is not the famous Amazon ideal of the “two-pizza team,” where features can be created by individual teams that can be fed with two pizzas.
We’ll need a whole truckload of pizzas to ship this feature, Maxine thinks, watching as Tom draws another set of boxes on the whiteboard.
Kurt pokes his head into the conference room. “Hey, sorry for the interruption. Someone from Ops and the manager of the channel training management application are on a conference bridge. All their customer logins are failing. They say the connector has stopped working?”
“Not again,” says Tom. “Authentication has been flaky ever since the Phoenix deployment. We’re on it …”
“Roger that,” Kurt says, tapping something on his phone. “I just created a chat channel for all of us, okay?”
Maxine follows Tom back to his desk. As Tom opens up another browser window and types something, a login error appears on his screen.
“Okay, something’s definitely not working right. Let’s see if we can isolate why …” Tom mutters. “I doubt it’s actually a Data Hub connector. More likely it’s the enterprise customer authentication service or a problem in the network.”
Maxine nods, taking notes as more of the Data Hub universe comes into view. Skeptical, she offers, “Can’t we rule out network and authentication right away? If either of those were down, we wouldn’t even be able to get to the website, and authentication being down would take out every service …”
“Good point …” Tom says. “Could definitely still be networking, though … we’ve had a bunch of issues lately. Last week, the networking people accidentally blocked some internal IP addresses that caused us problems.”
“Networking. It’s always the networking people, right?” she says, smiling. “But if it’s always the networking people, why are they calling us?” Maxine asks.
“Yeah, well, all the users know is that they can’t connect to Data Hub,” he says. “We always explain that it’s not us; it’s something we need to connect to. But they don’t care.”
When Maxine sees Tom pull up the Ops ticketing system and create a new ticket, she asks, “What’s this for?”
“We need the production logs for Data Hub and its connectors to see if they’re handling traffic or if they’ve crashed,” he responds, filling out the numerous fields.
“We can’t directly access production logs?” Maxine asks, afraid of the answer.
“Nope. Ops people won’t let us,” he says, typing into the form.
“So, someone has to respond to the ticket and copy the logs off the server for us?” she asks in dis
belief.
“Yes,” he says, continuing to type, obviously very practiced at filling it out. He tabs between fields, types, mouses over to hit the drop-down box, hits the submit button, only to find that there’s still another required field that needs to be filled in.
Maxine groans. The Data Hub application that they’re working on might as well be running in outer space or at the bottom of a deep well. They can’t directly access it, they can’t see what it’s doing, and the only way they can understand what’s actually happening is to talk to someone in Operations through the ticketing system.
She wonders whether the ticket will get routed to her friend Derek at the helpdesk.
Tom finally succeeds in submitting the ticket. Satisfied, he says, “Now we wait.”
“How long does it usually take?” Maxine asks.
“For a Sev 2 incident? Not too bad—we’ll probably get it within a half hour. If it’s not related to an outage, it could take days,” Tom says. He looks at the clock. “What should we do while we wait?”
Even in the Data Hub team, she can’t escape the Waiting Place.
Four hours later, after reviewing the production logs, they confirm that the problem isn’t Data Hub. Two hours after that, everyone finally agrees. As Tom had suspected, it was an internal networking change that caused the problem.
Another round of intense finger-pointing ensues between Business Operations, Marketing, and within the technology organization. Eventually, Sarah gets involved and demands that there be severe consequences.
“Uh, oh,” says Tom, watching with Maxine from the far end of the table. “This can’t be good.”
From:
Wes Davis (Director, Distributed Operations)
To:
All IT Employees
Date:
7:50 p.m., September 25
Subject:
Personnel changes
Effective immediately, Chad Stone in network engineering is no longer with the company. Please direct all emails to his manager, Irene Cooper, or me.
For the love of all that is holy, please stop making mistakes so that I don’t have to write these stupid emails. (And if they fire me, direct your emails to Bill Palmer, VP, IT Operations.)
Thank you,
Wes
Finally, the day is over, which means another meeting at the Dockside Bar. They’ve invited the entire Data Hub team to join them. Maxine approves of being over-inclusive rather than accidentally leaving some worthy people out. Tom and three other engineers show up. Maxine is glad they’re here. After the last couple days, she’s eager to brainstorm ways to dramatically improve developer productivity on the Data Hub team.
Seeing everyone having fun, Maxine observes that this is a group of people who love hanging out with each other. Kurt stands up and addresses the group.
“Hello, new Rebellion teammates! Let me introduce everyone,” Kurt says. He introduces all the Rebellion members, as he did for Maxine and Kirsten. “And if you don’t mind, now that you’ve heard about some of the subversive things we’re working on to bring joy back to Parts Unlimited engineers, how about you tell us something that could make your lives a little easier?”
Tom’s two colleagues go first, introducing themselves and sharing their backgrounds. One has been on the Data Hub team, like Tom, for nearly a decade, but he doesn’t come up with anything to complain about, saying, “Life is okay, and I appreciate the invitation for drinks.”
When he clearly doesn’t have anything more to say, Tom starts. “Like my colleague, I’ve been on the Data Hub team for a long time. Back when it used to be called Octopus. We called it that because of how it connected to eight applications. Now it connects to over a hundred.
“I’ve been having a blast pair-programming with Maxine, and I still can’t believe we fixed a race condition bug! I’m delighted at her idea to get Data Hub Dev environments that we can all use,” he continues. “I’m not proud of this, but there have been times when we’ve hired new developers and six months later, they still can’t do a full build on their machines,” he says, shaking his head. “It wasn’t always like this. When I started, it was simpler. But over the years, we’ve hard-coded some things that we shouldn’t have, updated some things here, updated other things there, never quite documenting all of it … and now? It’s a mess.”
Looking up, he smirks at his teammates around the table, saying, “You know the developer joke of ‘it worked on my laptop’? Well, in Data Hub, we can’t even get it running on most people’s laptops.”
Everyone laughs. At one point or another, every developer on the planet has had this problem. It usually happens at the worst possible time, like when something crashes in production but mysteriously works perfectly on the developer’s laptop. Maxine remembers countless times when she’s had to painstakingly figure out what exactly was different between the developer’s laptop and the production environment.
“My pain points?” Tom muses. “It’s our environments. We used to have a good handle on this, but then we got moved into the Phoenix Project and they made us use environments from their centralized environments team.
“It’s crazy. We’re puny compared to the rest of Phoenix. To run Data Hub now, we have to install gigabytes of completely irrelevant dependencies,” he continues. “It takes forever to figure out how to get everything to run, and it’s so easy to break something by accident. No joke: I back up my work laptop every day because I’m so afraid that my builds will stop working and I’ll have to spend weeks figuring out how to fix them.”
Tom laughs, “Ten years ago, I lost my emacs configuration file and couldn’t find a recent backup. I just didn’t have it in me to recreate it. I finally gave up and switched editors.”
Everyone laughs, adding their own stories of loss, anguish, and grief of having to give up their most treasured tools.
Tom turns to Maxine. “I’d love to spend a couple of days exploring how we can make a Dev environment that all of us could use in our daily work. If we had a virtual machine image or a Docker image, any new team member could do a build on any machine, any time. That would be incredible.”
“You and I are definitely going to get along,” Maxine says, smiling. “We need developers to be able to focus their best energies on building features, not trying to get builds to work. I have a ton of passion for this too and would love your help.”
“That’s terrific,” Kurt says. “We all know how important environments are. For now, feel free to spend half your time on this—I’ll hide it in the timecarding system.”
Later in the evening, Kirsten shows up and pours herself a glass of beer from the pitcher on the table. Smiling, she says, “What did I miss?”
“Just plotting the inevitable toppling of the existing order, of course,” Kurt says. The new Data Hub team members openly stare at Kirsten as she takes a seat.
Kurt asks, “Kirsten, how’s Project Inversion going? The feature freeze? I heard that Bill Palmer convinced Steve to put all feature work on hold so everyone can pay down technical debt.”
“Confirmed,” she says. “Sarah Moulton is going ballistic, complaining how ‘all the idle developers’ are jeopardizing the promises the company has already made to customers and Wall Street. I still can’t believe she doesn’t get how this helps her. But Project Inversion is definitely happening: for thirty days, Ops is not doing anything except things to support Phoenix.”
“They’re not kidding around,” Brent says. “Bill has been awesome. He’s told me in no uncertain terms that I’m to work only on Phoenix-related things. He’s taken me off of pager rotation for basically everything. He’s even taken me off of every mailing list, had me turn off notifications from every chat room, and told me not to answer the phone for anyone. And best of all, he said to absolutely not show up for any outage calls. If I do, he’ll fire me.”
Hearing this, Maxine is shocked. Bill would fire Brent? Thinking of all the people who’ve been fired lately, Maxine can’t figure o
ut why Brent is smiling.
“It’s so fantastic,” Brent says, even appearing to be … tearing up? “Bill told me that he can’t fire the business unit executives or tell them what to do. He said that the only thing he can do is ensure that I’m not wasting time on those things. He said to tell anyone trying to reach me that I’ll be fired if I call them back.”
Brent laughs, obviously elated, finishing his beer and pouring himself another. “He’s assigned Wes to screen all my emails and phone calls and to yell at anyone trying to get ahold of me. Life is fantastic! Seriously, never better.”
Maxine smiles. She has seen how engineers can become the constraint many times in her career. It can be fun to be at the center of everything, but it’s certainly not sustainable. Down that road, only chronic wakeup calls, exhaustion, cynicism, and burnout await.
Kirsten smiles. “It’s working. Brent’s name shows up on more critical action items than anyone, and Bill has told everyone that their goal must be to protect his time.
“On the Development side, Chris promises that for thirty days, for all teams working on anything related to Project Phoenix, no new features,” Kirsten says, reading from her phone. “‘All teams need to be fixing high-priority defects, stabilizing the codebase, and doing whatever rearchitecting is needed to prevent another release disaster.’”
Maxine hears lots of excited murmurs from around the table. Maxine knows something like this is needed—and that this could be a fantastic opportunity for the Rebellion.
“There’s still a lot of disagreement among Chris’s direct reports on how to roll this out,” Kirsten continues. “They’ve spent so much time legislating what should and shouldn’t be worked on that we’ve already lost a week—lots of teams are still working on their features, business as usual. We’re going to need a lot more clarity from leadership on this—at this rate, the entire month will be gone, and we’ll have the same amount of technical debt as before, if not more.”
