by Gene Kim
Wes says he’s on it, and I ask, “Is there anything else Sarah could pop on us?”
He pauses to think and finally says, “No, I don’t think so. We have a pretty valid reason, with the payroll run failure, for why Brent wasn’t able to complete his work.”
I agree. Feeling like our asses are sufficiently covered, I say, “See you at ten.”
Less than an hour later, I’m walking to Building 9 in the hot sunshine, where many of the Marketing folks call home. To my surprise, I join a small army of it people walking the same way. Why?
Then it hits me. The majority of our marketing projects can’t be done without it. High touch marketing requires high tech. But if there’s so many of us assigned to these Marketing projects, shouldn’t they be coming to us?
I imagine that Sarah likes it this way, the spider sitting back, enjoying seeing all the company minions making their way to her lair.
I arrive and immediately see Kirsten Fingle, who runs the Project Management Office sitting at the head of the table. I am a big fan of hers. She is organized, levelheaded, and a stickler for accountability. When she first joined the company five years ago, she brought a whole new level of professionalism to our organization.
At her right, Sarah leans back in her chair, tapping away on her iPhone, oblivious to the rest of us.
Sarah is my age: thirty-nine. She’s very guarded about her age, always saying things in a way that would lead one to conclude she’s much older, but never actually lying.
Yet another maddening thing about Sarah.
There are about twenty-five people in the room. Many of the business line owners are here, some of whom work for Sarah. Chris Allers is also here. Chris is a little older than me and looks lean and fit. He’ll just as often be seen joking with someone as kicking their ass about missing a deadline. He has a reputation as a capable and no-nonsense manager. With nearly two hundred developers working for him, he needs to be.
To help with Phoenix, his team has grown by fifty people in the last two years, many through offshore development shops. Chris is constantly asked to deliver more features and do it in less time, with less money.
Several of his managers are in the room, too. Wes is also here, sitting right next to Chris. As I start to look for an open chair, I note how everyone seems unusually tense. And then I see why.
There, sitting right next to the only open seat at the table, is Steve.
Everyone seems to be going to great lengths to not stare at him. As I casually take my seat next to Steve, my phone vibrates. It’s a text message from Wes:
Shit. Steve has never attended a project management meeting. We are totally screwed.
Kirsten clears her throat. “First on our agenda is Phoenix. The news isn’t good. This project went from yellow to red about four weeks ago, and it’s my personal assessment that the deadline is in grave jeopardy.”
She continues in her professional voice, “To refresh your memory, last week there were twelve tasks in the critical path of Phoenix Phase 1. Only three of those tasks were completed.”
There is a collective groan in the room, and several people mutter to one another. Steve turns to look at me. “Well?”
I explain, “The critical resource in question is Brent, who has been one hundred percent utilized helping to recover from the payroll failure, which we all know about. This was a totally unforeseen emergency but obviously one that we had to handle. Everyone knows how important Phoenix is, and we are doing everything we can to make sure Brent can stay focused.”
“Thanks for that super creative explanation, Bill,” Sarah immediately responds. “The real issue here is that your people don’t seem to grasp how important Phoenix is to the company. Our competition is killing us in the market. You’ve all seen and heard the commercials about their new services. They’re beating us on innovation, both in the retail stores and online. They’ve already lured away some of our biggest partners, and our sales force is starting to panic. I’m not the type to say, ‘I told you so,’ but their latest product announcement shows why we can’t be acting as if this is just business as usual.”
She continues, “See, Bill, in order for us to increase market share, we must ship Phoenix. But for some reason, you and your team keep dragging your feet. Maybe you’re not prioritizing correctly? Or maybe you’re just not used to supporting a project this important?”
Despite all my mental preparation, I feel my face get hot with anger. Maybe it was the condescending way she was parroting Steve to me. Or how she wasn’t even looking at me while she was addressing me, instead looking at Steve to see how he reacts. Or the way she basically called me out-of-touch and incompetent.
Everyone is silent as I force myself to take a deep breath.
My anger dissipates. This is all just corporate theater. I don’t like it but accept it for what it is. I almost made the Marines my career when I was up for promotion to staff sergeant. You don’t become a senior nco in the Marines without being able to play politics.
“Interesting,” I say to Sarah. “You tell me which is more important: getting our factory employees paid or getting the Phoenix tasks done? Steve told me to resolve the payroll failure. How would you have prioritized this differently than Steve?”
At my mention of Steve, Sarah’s expression changes. “Well, maybe if it didn’t cause the failure in the first place, you wouldn’t have blown your commitments to us. I don’t think we can depend on you and your team.”
I nod slowly, not taking the bait. “I look forward to any suggestions you have to offer, Sarah.”
She looks at me, then at Steve. Apparently deciding there are no more points to be gained here, she rolls her eyes. I see Wes shaking his head in disbelief at this discussion, staying uncharacteristically quiet.
Sarah continues, “We’ve spent over $20 million on Phoenix, and we’re nearly two years late. We must get to market.” Looking over at Chris, she asks, “Given the delays from Bill’s group, when is the soonest we can go live?”
Chris looks up from his papers. “I’ve looked into this since we talked last week. If we expedite some things and if the virtualized environments from Bill’s team work as expected, we can go into production one week from Friday.”
I gape at Chris. He just made up an arbitrary date to go into production, with complete disregard for all the things we need to do before deployment.
I have a sudden flashback. In the Marines, we had a ritual for all the senior ncos. We’d hang out with beers and watch Star Wars: Return of the Jedi. Every time Admiral Ackbar would cry, “It’s a trap!” we’d all laugh uproariously, yelling for a replay.
This time, I’m not laughing.
“Now just wait a minute here!” Wes interjects, pounding the table. “What the hell are you trying to pull? We just found out two weeks ago about the specifics of the Phoenix deployment. Your guys still haven’t told us what sort of infrastructure we need, so we can’t even order the necessary server and networking gear. And by the way, the vendors are already quoting us three-week delivery times!”
He is now facing Chris, pointing at him angrily. “Oh, and I’ve heard that the performance of your code is so shitty, we’re going to need the hottest, fastest gear out there. You’re supposed to support 250 transactions per second, and you’re barely doing even four! We’re going to need so much hardware that we’ll need another chassis to put it all in and probably have to pay a custom-manufacturing fee to get it in time. God knows what this will do to the budget.”
Chris wants to respond, but Wes is relentless. “We still don’t have a concrete specification of how the production and test systems should be configured. Oh, do you guys not need a test environment anymore? You haven’t even done any real testing of your code yet, because that fell off the schedule, too!”
My heart lurches as all the implications sink in. I’ve seen this movie before. The plot is simple: First, you take an urgent date-driven project, where the shipment date cannot be delayed because
of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date.
The results are never pretty. Usually, the software product is so unstable and unusable that even the people who were screaming for it end up saying that it’s not worth shipping. And it’s always it Operations who still has to stay up all night, rebooting servers hourly to compensate for crappy code, doing whatever heroics are required to hide from the rest of the world just how bad things really are.
“Guys, I understand the desire to get Phoenix into production as quickly as possible,” I say to Steve and Chris as calmly as I can. “But based on what we’ve heard from Wes, I think it is incredibly premature to deploy. We still don’t know what equipment we need to hit the performance objectives, nor have we done any capacity testing to confirm our guesses. It’s unlikely we have adequate documentation to run this thing in production, let alone get everything monitored and backed up.”
In my most persuasive voice, I continue, “I want Phoenix in the market as badly as anyone else, but if the user experience is bad enough, we’ll end up driving our customers to the competition.”
I turn to Chris. “You can’t just throw the pig over the wall to us, and then high-five each other in the parking lot, congratulating yourselves on how you made the deadline. Wes is telling us that the pig will probably break its leg, and it’ll be my guys who work all-nighters and weekends to keep that pig alive.”
Chris replies hotly, “Don’t give me that bullshit about ‘throwing the pig over the wall.’ We invited your people to our architecture and planning meetings, but I can count on one hand the number of times you guys actually showed up. We routinely have had to wait days or even weeks to get anything we need from you guys!”
Then he just holds up his hands, as if everything is outside of his control. “Look, I’d like more time, too. But from the very beginning, we all knew that this was a date-driven project. That was a business decision we all made.”
“Exactly!” Sarah exclaims before I can respond. “This just shows how Bill and his team lack the necessary sense of urgency. Perfection is the enemy of good. Bill, we simply do not have the luxury of time to polish this to whatever gold standard you’re proposing. We need to create positive cash flow, and we cannot do that without taking back market share. And to do that, we need to deploy Phoenix.”
She looks over at Steve. “We understand risk, don’t we, Steve? You’ve been doing an absolutely amazing job selling this to analysts and even the guys on cnbc—I don’t think we want egg on our face by shipping even later than we already are.”
Steve nods his head and rubs his chin, rocking back and forth in his chair as he thinks. “Agreed,” he finally says, leaning forward. “We’ve made commitments to our investors and analysts that we were going to launch Phoenix this quarter.”
My jaw drops. Sarah has blunted all my arguments, leading Steve down a reckless, destructive path.
Exasperated, I say, “Does anyone think this is really odd? I’ve been in this room when we discussed installing new water fountains in the front of every store. We gave that team nine months to plan the rollout. Nine months! And all of us agreed that was reasonable.
“Now we’re talking about Phoenix, which impacts thousands of point of sale systems, and all of the back-office order entry systems. This is at least ten thousand times more complex than rolling out new water fountains, with way more risk to the business. And you’re only giving us one week to plan and execute the rollout?”
I throw my hands up, imploring Steve, “Doesn’t this seem a bit reckless and unfair?”
Kirsten nods, but Sarah says dismissively, “Bill, that’s a touching story but we’re not discussing water fountains, we’re discussing Phoenix. Besides, I believe the decision has already been made.”
Steve says, “Yes, it has. Thank you for sharing what you view as the risks, Bill.” He turns to Sarah. “When is the launch date?”
Sarah replies quickly, “Marketing launch is next Saturday, September 13. Phoenix will deploy at 5 p.m. the previous day.”
Steve writes the date in the back of his notebook and says, “Good. Keep me posted on progress, and let me know if there’s anything I can do to help.”
I look over at Wes, who mimes with his hands an airplane crashing into the table in front of him and bursting into flames.
In the hallway, Wes says, “I thought that went pretty well, boss.”
I’m not laughing. “What the hell happened in there? How did we get into this position? Does anyone know what’s required from us to support this launch?”
“No one has a clue,” he says, shaking his head in disgust. “We haven’t even agreed on how to do the handoff with Development. In the past, they’ve just pointed to a network folder and said, ‘Deploy that.’ There are newborn babies dropped off at church doorsteps with more operating instructions than what they’re giving us.”
I shake my head at his awful imagery, but he’s right. We’ve got a serious problem here.
He continues, “We’re going to have to assemble a huge team, including Chris’ guys, to figure out how we’re going to pull this off. We have problems at every layer: networking, servers, databases, operating systems, applications, Layer 7 switching—the whole wad of crap. It’s going to be late nights for all of us for the next nine days.”
I nod unhappily. This type of all-hands effort is just another part of life in it, but it makes me angry when we need to make some heroic, diving catch because of someone else’s lack of planning.
I say, “Get your team assembled, and ask Chris to assemble his respective team as well. Stop trying to do this by e-mail and in the ticketing system. We need everyone in the same room.”
“Speaking of commitments,” I say, “What was Chris referring to when he said that our guys never showed up to the Phoenix architecture and planning meetings? Is that true?”
Wes rolls his eyes in frustration. “Yeah, it’s true that his people would invite us at the last minute. Seriously, who can clear their calendar on less than a day’s notice?”
“Although, in fairness,” he says, after a moment, “we did get ample notice on a couple of the big planning meetings. And one of the most critical people who needed to be there wasn’t able to make it, due to escalations. You can probably guess who that is…”
I groan. “Brent?”
Wes nods, “Yep. He’s the guy we need at those meetings to tell those idiotic developers how things work in the real world and what type of things keep breaking in production. The irony, of course, is that he can’t tell the developers, because he’s too busy repairing the things that are already broken.”
He’s right. Unless we can break this cycle, we’ll stay in our terrible downward spiral. Brent needs to work with developers to fix issues at the source so we can stop fighting fires. But Brent can’t attend, because he’s too busy fighting fires.
I say, “We need our best minds to prepare for this deployment, so make sure Brent is there.”
Wes looks sheepish for a moment. I ask him, “What?”
“I think he’s working a network outage right now.” he replies.
“Not anymore,” I say. “They’re going to have to fix it without him. If someone has a problem with that, send them to me.”
“Okay, whatever you want, boss.” he says, shrugging his shoulders.
After the project management meeting, I’m in no mood to talk to anyone. I sit at my desk and grumble when my laptop doesn’t wake up. The disk drive light just keeps blinking. When nothing shows on the screen, I grab my empty mug that I keep on my desk by the picture of Paige and my two sons and walk to the coffee machine in the corner.
When I get back to my desk, a window on the screen tells me that it’s going to instal
l some critical new updates. I sit down, click “ok” and watch the status bar crawl across the screen. Suddenly, I see the dreaded “blue screen of death.” My laptop is now completely locked up and unusable.
It happens again even after I reboot. I mutter in frustration, “You’ve got to be kidding me!”
Just then, Ellen, my new assistant, pokes her head around the corner. Holding out her hand she says, “Good morning. Congratulations on the promotion, Bill!” Noticing my blue-screened laptop, she says sympathetically, “Ooh, that doesn’t look good.”
“Umm, thanks.” I say, reaching out to shake her hand. “Yeah, about this laptop, can you get a hold of someone in desktop support? There’s some serious crap headed our way from Phoenix, and I’m going to need it.”
“No problem,” she says, nodding with a smile. “I’ll tell them our new vp is hopping mad, demanding that his laptop get fixed. Of all people, you need a working computer, right?
“You know,” she adds, “I’ve heard that a bunch of other people are having problems like this today. I’ll make sure you get to the top of the list. You can’t afford to wait in line.”
More bricked laptops? This is surely evidence that the universe is out to get me today.
“By the way, I need some help coordinating some emergency Phoenix meetings. Has anyone granted you access to my calendar yet?” I ask.
She rolls her eyes. “No. That’s actually why I came down here. I wanted to see if you could print out your next couple of days. Obviously, that’s out of the question. I’ll have the desktop support person do that while he’s here. Sometimes it takes weeks for the e-mail administrators to get around to stuff like this.”
Weeks? That’s unacceptable. I quickly look at my watch and realize I’ll have to tackle this later. I’m already late.
“Do your best,” I say. “I’m off to Patty’s enterprise change management meeting. Call me if you need anything, okay?”
Being ten minutes late to Patty’s meeting, I hurry into the room, expecting to see either a bunch of people waiting for me impatiently or perhaps a meeting already underway.
