by Gene Kim
Maxine gets goosebumps—not because the presenter mentioned her old MRP system, but because the presenter is clearly a brilliant engineer, fearlessly doing what she thinks needs to be done and not afraid to run things in production.
While the young engineer fields more questions from the LARB, Maxine texts the Rebellion in the chat channel:
Who is this engineer presenting? She’s awesome! She’s obviously Rebellion material. We should recruit her.
Adam texts back:
That’s Ellen. She’s one of the best Ops people around
Everyone nods at Maxine, agreeing with Adam’s assessment.
Brent adds in the chat channel:
Agreed. I had no idea she was working on this. This is great!
Maxine looks up when she hears Dwayne talking. “You have got to be kidding me. We created TEP-LARB to help evaluate new technologies. Apache Tomcat was created decades ago, and it’s either the second or third most widely used application server out there. If we aren’t brave enough to run Tomcat, we should get out of the technology game once and for all. I vote yes. And if you don’t, I think we all need to hear why.”
Someone from the Ops delegation says, “I don’t have anything against Tomcat. I’m just not comfortable with our ability to support this given our current staffing levels. We’re stretched thin as it is, and while I appreciate that this technology isn’t bleeding edge, we still need people to operate and maintain it …”
Dwayne interrupts, “But you just heard Ellen say that her team is willing to support it!”
Not even acknowledging Dwayne’s comment, the Security architect joins in, “And there’s the security risks. I’d like to get a historical report of Tomcat vulnerabilities, how quickly patches were made available, and any reported problems in patching. Maybe then we can come to a decision.”
Dwayne mutters, “For crying out loud. Ellen is the person who would write the security and the patching guidelines.”
“Thank you for your proposal. We look forward to this team presenting the requested information at our next meeting,” the Ops architect says, not looking up from the note he’s writing.
At the front of the room, Maxine sees Ellen and her teammates slump in exasperation. Ellen closes her laptop, nods respectfully to all the assembled architects, and takes a seat at the back of the room.
Maxine gives Ellen and her teammates the most enthusiastic thumbs up she can manage.
“Next up is Maxine and Adam on the proposal to move Data Hub into a new environment, running on containers, with automated code builds, tests, and deployments?” the Ops architect prompts.
Adam stands up, but after seeing the last presenter, Maxine already knows they’re sunk. No matter how well prepared they are, they’ll never be able to convince the LARB.
“… and to summarize, the urgent needs of the Promotions team requires us to get Data Hub functionality more quickly to our internal customers. We need a radically different way to store and retrieve data that allows us to be decoupled from the rest of the Phoenix teams,” Maxine concludes. “We’ve found a set of technologies that can help us achieve that, which have been battle-tested and used in production for over a decade at some of the largest internet properties on the planet: Google, Netflix, Spotify, Walmart, Target, Capital One, and many more. Based on our trials over the past several weeks, we are confident in our ability to support it, and we’re willing to support it ourselves if necessary.”
Brent, who joined them at the front of the room, adds, “The team supporting Data Hub production would be some of the most experienced people we have in Ops. Personally, I can’t even describe how excited I am by this effort. I think these technologies have applicability far beyond Data Hub and could really improve things for almost every application we support. We are willing to be available and responsible to resolve anything that goes wrong. Utilizing these techniques will help every Dev and Ops person at Parts Unlimited.”
Maxine sees Kurt smile at the team from the back of the room. Maxine is proud of everyone. It was a solid presentation. She sees Ellen grinning wildly, obviously impressed. But Maxine knows that it’s all for naught. The LARB was designed to be an organizational immune system to prevent dangerous changes—they are just too powerful and conservative.
Dwayne tries to rally support. “The LARB should foster innovative efforts like this, picking technologies that can help us win in the marketplace. We used to set the industry direction, making bold choices that left our competitors in the dust. People laughed when we created our own MRP system, saying we were idiots, but history has shown that that was the right thing to do. We were the first company in our industry to use thin clients in our factories, and because of that and hundreds of brilliant technology decisions, we became one of the most efficient and effective manufacturers in the country.”
Maxine looks around the room and sees some stirrings of excitement and renewed curiosity among the Dev architects. However, she sees all the Ops and Security architects shaking their heads. One of them says, “Dwayne, I appreciate what you’re saying, but we’ve never done anything even remotely similar to this. It’s embarrassing that we can’t even support Tomcat—but that shows you exactly why we can’t possibly support this. Unless there’s a group willing to volunteer to support this initiative as a side project, I think we need to table it.”
Dwayne speaks up, “Hell yes, I volunteer. And I’ll grab some people I know who would love to help the Data Hub team with the support responsibilities.”
“I’d love to help,” says Ellen from the back of the room. “I’ve been using Docker and the other tools you’ve mentioned for years. These are competencies we need at this company.”
“You’re in,” Maxine says to Ellen, smiling.
The Ops Chair looks surprised but says, “I appreciate your enthusiasm, but I’m afraid that we cannot support your initiative at this time. Let’s pick this up in six months and see if conditions have changed by then.”
Hearing enough, Kurt stands up and addresses the room. “Didn’t you hear the business context? Both Maggie Lee and Sarah Moulton have clearly stated that the company’s survival depends on this. This is so important that if you can’t support it we’re going to have to support it ourselves in Development.”
“We hear business people say things like that all the time,” the Ops Chair says. “We invite you back in six months to discuss it again. And now to other matters …”
Defeated, the team leaves the meeting, reassembling in a nearby conference room that Kurt booked in advance. Maxine invites Ellen and the three other engineers who presented the Tomcat proposal.
“Wow, that was so great. Are you really going to go rogue and run all this yourselves?” Ellen says, smiling ear to ear, not affected by the glum faces all around her. “If so, count me in. I’m Ellen, by the way,” she says, extending her hand to Maxine and then introducing her team.
“Good seeing you again, Ellen,” Adam says with a big smile. “Welcome to our merry band of rebels. If I’m reading the tea leaves right, I think we’re going to need your help soon.”
Ellen smiles. “The fact that you have Brent onboard is enough for me. What you presented was amazing. I had no idea anyone was working on these types of things here at Parts Unlimited.”
Brent smiles modestly, “But we still got our asses kicked, right?”
Kurt says, “We did indeed. But if all goes according to plan, by the end of the day there will be a memo going out from Chris and Sarah announcing a small re-org that will allow Data Hub to operate outside the conventional Ops and QA processes. That will be the official go-ahead to do whatever we need to do.”
Everyone on the Data Hub team cheers, surprised at the good news. Maxine hears Ellen mutter, “Wow. That’s some pretty powerful mojo you have on your side.”
Brent mutters back, “You have no idea. I’ll tell you later.” Adam laughs in agreement.
While almost everyone is celebrating, Dwayne is glum. When
Maxine asks why, Dwayne says, “I just can’t believe the LARB didn’t support these efforts. We let you all down. What was supposed to happen was that they would see the grave danger on the horizon. They were supposed to support our cause. They were supposed to help … Like Gandalf getting the support of the White Council in Lord of the Rings …”
Maxine is surprised when Dwayne puts both of his hands on his temples, groaning. After a minute, he finally says, “But it didn’t work out that way at all.”
Brent laughs. “You’ve got it wrong, Dwayne. The Fellowship of the Ring wasn’t ever officially sanctioned by the White Council. Gandalf warned everyone that the One Ring was at large, but Saruman refused to help because he was already working for the evil Sauron. So, Gandalf went rogue. He went it alone. Just like we’re going to do.”
“Damn right,” says Kurt. Turning to Ellen and her team, he says, “You all doing anything after work? There’s a bar that we go to …”
“What the hell have you gotten me into?!” Chris says, fuming at Kurt. “Maggie and Sarah tell me that you’ve proposed to create your own Ops organization inside of Dev?! And that you’ve gotten some sort of exception waiver to start running some new Tier 2 services in the cloud?! I don’t suppose you ever thought to ask me first?”
Maxine is in Chris’ office with Kurt, Dwayne, and Maggie. Chris is clearly not happy, but Maggie goes to extraordinary lengths to describe the business outcomes that need to be achieved and the grave consequences of not doing so.
Chris stares out his window for several moments and then turns to Maxine. “Do you think we really have the chops to keep all this from blowing up in our faces?”
“Absolutely, with the help of Dwayne and Brent from Ops,” she says with certainty. “I’ll do everything in my power to make sure things go smoothly. I really think we’ve got this, Chris. And I promise to take the blame for anything that happens.”
At the mention of Dwayne and Brent, a pained expression appears on Chris’ face. He looks at her, obviously thinking, What about ‘don’t rock the boat’ and ‘stay in your lane’ do you not understand?
Maxine shrugs. She knows that Chris supported mission-critical services early in his engineering career, over twenty years ago. But ever since then, he’s only been responsible for the code, no longer running the actual services that it enables. Maxine could almost see him tabulating all the inconveniences this could create, all the things that could go wrong, balancing it against what could happen if he refuses.
“Fine, fine, fine. I’ll do it,” he says reluctantly. “You people are going to give me a heart attack,” and then shoos them out of his office.
As promised, Chris sends out a memo to everyone announcing a re-org—the Data Hub team is now reporting directly to him, and as an experiment, they’ll be exempt from the normal rules and regulations around changes, able to test their own code, deploy it, and operate it in production themselves.
“The email just went out,” Kurt says, grinning wildly. “We’re in the deployment and operations business!”
“Wow, that’s incredible,” Maxine says, still staring at the email on her phone. “You know, despite everything we did, I was pretty sure it wasn’t really going to happen.”
Kurt laughs. “I don’t think Chris had that much choice in the matter. Both Maggie and Sarah took this all the way up to Steve.”
With the Data Hub re-org, the team is now committed. They are working furiously to automate the production deployments and to figure out how to do production operations without centralized Ops. To what extent they needed to really divorce themselves from Ops for things like backup was still unclear and being negotiated.
The enormity of the challenge is exhilarating. The goal is clear: enable fast and safe deployments into production, and for the first time in years, do it using the same environments across Dev, Test, and Production. And everyone wants to prove that they can get everything up and running before the rest of the Phoenix Project even finishes their testing cycle.
Once again, they are in an imaginary race against the lumbering Phoenix Project.
Maxine is working with Dwayne, Adam, Shannon, and Brent, making slow but sure progress on getting the Data Hub production services to run on something besides the fastest bare-metal servers that money can buy … a decade ago. Many things in Data Hub blew up when installed on a current OS version … from this decade. They found several binary executables that no one could find the source code for. Data Hub had become this fragile and irreproducible artifact. That’s great if you’re an art collector, Maxine thinks, but utterly unacceptable when you’re running a mission-critical service.
They work methodically to create a Test and Production service that behaves like the old one, but can be spun up instantly in a container. For days, she’s mired again in the messy world of infrastructure, dealing with Makefiles, YAML, and XML configuration files; Dockerfiles; purging secrets from their source code repositories; and using all her experience to speed up build and test times. This, unfortunately, required lots of Bash scripts.
Maxine remembers a quote from Jeffrey Snover, the inventor of PowerShell. He once said, “Bash is the disease you die with, but don’t die of.” Maxine shares this sentiment. Infrastructure is messy work, almost the opposite of the pure functional programming she loves—in infrastructure, almost everything you do has a side-effect that mutates the state of something in the environment, making it difficult to isolate and test changes, as well as diagnose problems when things go wrong.
But she knows how important this work is, and every bit of knowledge and expertise that she can put into these environments and CI/CD platforms will elevate the productivity of every engineer at Parts Unlimited.
Looking around, she realizes that now some of the best engineers in the company are working on making everyone else more productive. That’s the way it should be, she thinks.
By the next Thursday, Maxine is thrilled at how much they’ve been able to accomplish with all the restrictions lifted. But something strikes Maxine as odd. She notices that all the Data Hub engineers are pitching in. She certainly appreciates their help and she knows that Project Inversion was supposed to disallow feature work, but still, there’s almost always some urgent feature that needs working on.
Suspicious, she asks Tom what’s going on. He says, “This sounds strange, but technically there isn’t any feature work even ready. Believe it or not, every feature is waiting on something from Product Management,” he says. “It’s everything from a customer requirement that needs clarification, a question about a wireframe, a choice that needs to be made between different options or priorities … Sometimes it’s something small, like where a button should go. And sometimes it’s something big, like them not showing up to the demo to validate what we’ve built.” Tom laughs. “They think we’re the bottleneck, but we’re always waiting for them.”
“Can you show me?” says Maxine. None of the things Tom described sounded good, but the part about the product manager not showing up for the demo pisses her off. What a disrespectful thing to do to engineers who built what you asked them to.
She watches as Tom pulls up a tool she hasn’t worked with before, this one used by the product managers to capture ideas from customers: the ideal customer journey, value hypotheses, manage experiments, and so forth.
“What are all those blue cards?” she asks.
“Good eye. That’s exactly the problem,” he says. “Those are all the features that we’re working on, but we’re blocked because of something we need from Product Management. Like all those reasons I mentioned before. Oh, and here’s some yellow cards which are the features we’ve completed but that haven’t been accepted by the business stakeholders yet. This one has been waiting for forty days.”
Maxine feels her face turn red, indignant that as much as Product Management complains about the need to get features to market quickly, all these blue and yellow cards represent where they are in the way, not Development. How c
an we keep Product Management accountable? Maxine thinks. Time to bring in Kurt.
Ten minutes later, Kurt is with them, staring at the sea of blue cards. “I get it. This is not good, but I have an idea,” he says. “By the way, did you know that Sarah put a huge design agency on retainer, and now they’re flooding some of the other teams with wireframe diagrams that will probably never get worked on? And no matter how much the Dev managers ask them to stop sending wireframes, they still keep coming.”
“Why?” Maxine asks.
“I think it’s because Sarah needs to show off the apps she wants to build,” he says. “But what’s funny is that when the designers came here, the last thing they wanted to do was wireframes. They wanted to learn about our customers, and they did a bunch of exercises to better clarify goals around the personas we used. There was even one session where we all drew wireframes,” Kurt says, laughing.
Working with designers fascinates Maxine. Early in her career, the ratio of UX and designers to developers was 1:70. These days, great teams doing consumer-oriented products have ratios of 1:6 because it’s that important to create products that people love. Every consumer these days knows what a professional app feels like. Apps that don’t have great designers are often ridiculed as “enterprisey.”
She’s seen teams still waiting to be assigned designers, eventually making their own wireframes, HTML and CSS styling, and icons just to keep feature flow moving. These are the projects that teams are actually embarrassed to show other people, she thinks.
The good news is that Sarah got a bunch of great designers. The bad news is that she put them all where they weren’t needed and were actually slowing important development work down by flooding their backlogs with things that didn’t matter.
That evening after dinner, while her family played with Waffles, she opened up her laptop. Something about the sea of blue cards that Tom showed her earlier had been bothering her, and she’s determined to get to the bottom of it.
