by Gene Kim
Exasperated, I shout, “Dang it, Wes. That was completely preventable! Get one of your junior guys to look at the logs every day for drive failures. Maybe even have him visually inspect the drives and count all the blinking lights. It’s called preventive maintenance for a reason! We need Brent on Phoenix, not piddly crap like this!”
Wes says defensively, “Hey, it’s actually a little more complicated than that. We put in the order for replacement drives, but they’ve been stuck in Procurement for weeks. We had to get one of our vendors to give it to us on credit. This wasn’t our fault.”
I lose my temper. “Wes, listen to me. i don’t care! I don’t care about Procurement. I don’t care how nice your vendors are. I need you to do your job. Make sure this doesn’t happen again!”
I take a deep breath. I realize my frustration is not because of the drive failure, but because we’re continually unable to stay focused on the things that matter most to the company.
“Look, let’s put this aside for now,” I say, looking back at Wes. “I’m serious about getting someone to look at that san daily, though. Set up a meeting sometime next week for you, me, and Patty to get to the bottom of these outages. We’ve got to figure out how to bring down the amount of break-fix work so we can get project work done. If we can’t get Phoenix work done, it’s jeopardizing the company.”
“Yeah, I got it. I’ll try to get it in before the Phoenix rollout.” Wes says, nodding sullenly. “And I’ll get on that san issue this afternoon.”
“Okay, back to the spreadsheet,” I say.
Patty observes glumly, “You’re right. The one consistent theme in the interviews was that everyone struggles to get their project work done. Even when they do have time, they struggle to prioritize all their commitments. People in the business constantly ask our staff to do things for them. Especially Marketing.”
“Sarah?” I ask.
“Sure, but it’s not only her,” she replies. “Practically every executive in the company is guilty of going directly to their favorite it person, either asking a favor or pressuring them to get something done.”
“How do we change the game here and get resourced to do all these projects properly?” I ask. “What should we be asking Steve for?”
Wes scrolls down his spreadsheet. “Based on our rough numbers, we’ll probably need to hire seven people: three database administrators, two server engineers, one network engineer, and one virtualization engineer. Of course, you know that it’ll take time to find these people and then another six to twelve months before they’re fully productive.”
Of course, I knew that new hires aren’t productive right away. But it was still dispiriting to hear Wes point out that real help was still a long way off, even if Steve approved the headcount.
Later that day, as I’m walking to our second cab meeting, I feel hopeful. If we can get our old change process going, we might be able to quickly resolve one of the largest audit issues and get some operational wins, as well.
I’m also pleased at how well Patty and Wes are working together.
As I near the conference room, I hear loud voices arguing.
“—then Patty got that engineer fired for doing his job. He was one of our best networking people. That wasn’t your call to make!”
No mistake. That’s Wes hollering. Then I hear Patty reply heatedly, “What? You signed off on that termination! Why is this suddenly my fault?”
I knew it was too good to be true.
I then hear John say, “That was the right call. We’re going into our third year of a repeat audit finding around change controls. That goes in front of the audit committee. Next time around, it probably won’t be just an engineer getting fired, if you get my drift.”
Wait. Who invited John to this meeting?
Before John can make things any worse, I quickly step through the door and say cheerfully, “Good afternoon, everyone! Are we ready to review some changes?”
Fourteen people turn to look at me. Most of the technical leads from the various groups are sitting at the table. Wes is standing up behind his chair, fuming, while Patty is standing in the front of the room, arms crossed.
John sits in the back of the room, with his three-ring binder open, very much an unwanted guest.
Using both hands, I set down my antique laptop. It hits the table with a thud and a clatter as the battery falls off, the tape no longer holding it in place, and then I hear a scratching sound as the disk drive spins down.
Wes’ angry expression disappears momentarily. “Wow, boss, nice gear. What is that, a Kaypro II? I haven’t seen one of those in about thirty years. Let me know if you need an 8-inch floppy to load CP/M on it—I’ve got one in my attic at home.”
Two of the engineers snicker and point. I smile briefly at Wes, grateful for the comic relief.
Remaining standing, I say to everyone, “Let me tell you why I assembled all of you here. Given the urgency of Phoenix, you can bet your ass that I wouldn’t waste your time if I didn’t think this was important.”
I continue, “First, the events that led to the san and payroll failure on Tuesday must not happen again. What started off as a medium-sized payroll failure snowballed into a massive friendly-fire san incident. Why? Because we are not talking to one another about what changes we’re planning or implementing. This is not acceptable.”
“Second, John is right. We spent yesterday morning with our auditors, discussing a bunch of deficiencies they found,” I continue. “Dick Landry is already crapping bricks because it could impact our quarterly financial statements. We need to tighten up our change controls, and as managers and technical leads, we must figure out how we can create a sustainable process that will prevent friendly-fire incidents and get the auditors off our back, while still being able to get work done. We are not leaving this room until we’ve created a plan to get there. Understood?”
When I’m satisfied that everyone has been properly cowed, I open it up for discussion. “So what’s preventing us from getting there?”
One of the technical leads quickly says, “I’ll start. That change management tool is impossible to use. There’s a million mandatory fields and most of the time, the drop down boxes for the ‘applications affected’ don’t even have what I need. It’s why I’ve stopped even putting in change requests.”
Another lead hollers out, “He’s not kidding. To follow Patty’s rules, I have to manually type in hundreds of server names in one of the text boxes. Most of the time, there’s not enough room in the field! A hundred server names are supposed to fit in a sixty-four-character text box? What idiot built that form?”
Again, more unkind laughter.
Patty is bright red. She shouts, “We need to use drop-down boxes so we can maintain data integrity! And I’d love to keep the application list up-to-date, but I don’t have the resources. Who’s going to keep the application catalog and change management database current? You think it just magically updates itself?”
“It’s not just the tool, Patty. It’s the entire broken process,” Wes asserts. “When my guys put in change requests, they have to wait a lifetime to get approvals, let alone get on the schedule. We have the business breathing down our neck to get crap done. We can’t wait for you to hem and haw, complaining that we didn’t fill out the form right.”
Patty snaps, “That’s crap, and you know it. Your people routinely break the rules. Like, say, when everyone marks all their change requests as an ‘urgent’ or ‘emergency change’. That field is only for actual emergencies!”
Wes retorts, “We have to do that, because marking them urgent is the only way to get your team to look at it! Who can wait three weeks for an approval?”
One of the lead engineers suggests, “Maybe we make another field called ‘extremely urgent?’”
I wait until the uproar quiets down. At this rate, we’re getting nowhere fast. Thinking furiously, I finally say, “Let’s take a ten-minute break.”
When we reconvene the me
eting, I say, “We are not leaving this meeting without a list of authorized and scheduled changes that we’re implementing in the next thirty days.
“As you can see, my assistant has brought in a pile of blank index cards. I want each group to write down every change they’re planning, one change per index card. I want three pieces of information: who is planning the change, the system being changed, and a one-sentence summary.
“I’ve drawn a calendar on the whiteboard where we will eventually post approved changes according to their scheduled implementation,” I continue. “Those are the rules. Short and simple.”
Wes picks up a pack of cards, looking at them dubiously. “Really? Paper cards, in this day and age? How about we use that laptop of yours, which probably even predates paper?”
Everyone laughs, but not Patty. She looks angry, obviously not pleased with the direction things are going.
“This isn’t like any change management process I’ve ever seen,” John says. “But I’ll put my changes on the board, like the upcoming firewall updates and monitoring changes that’re scheduled for the next couple of days.”
Surprisingly, John’s willingness to jump in inspires others, who begin writing their planned changes on their cards.
Finally, Wes says, “Okay, let’s try it. Anything is better than using that busted change management tool.”
One of the leads holds up a handful of cards. “I’m done with all the database changes we’re planning to make.”
When I nod for him to proceed, he quickly reads one of the cards: “Execute the vendor-recommended database maintenance script on Octave server xz577 to fix retail store pos performance issues. This affects the order entry database and applications. We’d like to do this next Friday evening at 8:30 p.m.”
I nod, pleased with the clarity of his proposed change. But Wes says, “That’s not a change! That’s just running a database script. If you were changing the script, then we’d have something to talk about. Next.”
The lead replies quickly, “No, it’s definitely a change. It temporarily changes some database settings, and we don’t know what production impact it could have. To me, it’s just as risky as a database configuration change.”
Is it a change or not? I can see both sides of the argument.
After thirty minutes of arguing, it’s still not clear that we know the definition of what a “change” should be.
Was rebooting a server a change? Yes, because we don’t want anyone rebooting servers willy-nilly, especially if it’s running a critical service.
How about turning off a server? Yes, for the same reason.
How about turning on a server? No, we all thought. That is, until someone came up with the example of turning on a duplicate dhcp server, which screwed up the entire enterprise network for twenty-four hours.
A half hour later, we finally write on the whiteboard: “a ‘change’ is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered.”
I look at my watch, alarmed that we’ve been in the room for nearly ninety minutes, and we still haven’t even approved our first change. I push us to move faster, but at the end of our two-hour meeting, we’ve only posted five changes on the whiteboard.
Surprisingly, no one else seems frustrated except me. Everyone is vigorously engaged in the discussion, even Patty. Everyone is discussing risks of the proposed changes, even discovering that one change wasn’t necessary.
Encouraged, I say, “We’ll pick this up on Monday. Get all your cards to Patty as soon as you can. Patty, what’s the best way for us to process all the cards?”
She says tersely, “I’ll set up a basket later today. In the meantime, pile them up at the front of the table.”
When we adjourn, several people tell me on their way out, “Great meeting,” and “I wish we had more time to discuss changes,” and “I’m looking forward to Monday.”
Only Patty has remained behind, arms crossed. “We spent a lot of blood, sweat, and tears creating our old change management policy, and everyone still blew it off. What makes you think this will be any different?”
I shrug. “I don’t know. But we’ll keep trying things until we have a system that works, and I’m going to make sure everyone keeps helping us get there. It’s not just to satisfy the audit findings. We need some way to plan, communicate, and make our changes safely. I can guarantee you that if we don’t change the way we work, I’ll be soon out of a job.”
Pointing at her old policy document, she says, “We shouldn’t just throw all this work out the window. We spent weeks designing it and hundreds of thousands of dollars with consultants, changing our tools around.”
She tears up slightly. I remind myself of how long she’s been trying to get this process integrated into the organization.
“I know that there was a lot of good work put into all this process,” I say sympathetically. “Let’s face it, though. No one was actually following it, as the auditors pointed out. We also know that people were gaming the system, just trying to get their work done.”
I say sincerely, “We may be starting over, but we need all your experience and skills to make this work. It’s still your process, and I know this is absolutely critical to our success.”
“Okay,” she says, sighing in resignation. “I suppose I care more about our survival than whether we use our old process or not.”
Her expression changes. “How about I write up the outputs of the meeting and the new instructions for submitting requests for changes?”
Later that afternoon, I’m back in the Phoenix war room when Patty calls. I run out to the hallway. “What’s up?”
She sounds stressed. “We’ve got a problem. I was expecting we’d have fifty changes for us to review next week. But we’re already up to 243 submitted changes. I keep getting e-mails from people saying to expect more cards over the weekend… I think we’re looking at over four hundred changes being made next week!”
Holy crap. Four hundred? How many of these four hundred changes are high risk, potentially affecting Phoenix, the payroll application, or worse?
I suddenly remember Rangemaster duty in the Marines. As Rangemaster, I was responsible for the safety of everyone on the firing range. I have a horrifying vision of a mob of four hundred unsupervised eighteen-year-olds jumping out of trucks, running to the firing range, firing their rifles into the air, hooting and hollering…
“Umm, at least people are following the process,” I say, laughing nervously.
I hear her laugh. “With all the change requests coming in, how are we going to get them all authorized by Monday? Should we put a temporary hold on changes until we get them all approved?”
“Absolutely not,” I say immediately. “The best way to kill everyone’s enthusiasm and support is to prevent them from doing what they need to do. I doubt we’ll get a second chance to get this right.
“Send out an e-mail telling everyone to submit any change for next week by Monday. Monday’s changes will not need to be authorized but changes for the remainder of the week will. No exceptions.”
I can hear Patty typing over the phone. “Got it. I’ll probably need to have some of my people help organize all of the change cards over the weekend. Frankly, I’m stunned by how many changes there are.”
So am I.
“Excellent,” I say, leaving my concerns unvoiced.
CHAPTER 7
• Friday, September 5
When I get back to my desk, I’m looking for the Advil I usually keep on my desk when my cell phone rings. “Palmer here,” I say, rummaging through my drawers.
“Hi, Bill. It’s Stacy—Steve’s assistant. I’m glad I caught you. There’s a potential new board member in town named Erik Reid who needs to talk with all the it executives. He’s wondering if you’re available for an hour right now.”
“Hang on a sec while I pull up my calendar,” I reply.
&nb
sp; The screen resolution on this old laptop is so low that the weekly view is unusable. I switch to the daily view, and the screen goes blank as the laptop chatters and whirs.
I give up waiting and say earnestly, “Look, I know this is important, but can’t this wait until Monday? You would not believe the day I’m having.”
She replies quickly, “I wish it could wait, but he’s only in town today. And from what I’ve seen, Bob Strauss, you know, the new company chairman, and Steve are in a tizzy because they’re worried that Erik may not accept our offer to join the board. He’s apparently some technology hotshot, and Bob and Steve managed to get him in town to woo him. He insists on meeting the it leadership team before he leaves.”
“Okay, I’m in,” I say, suppressing a sigh.
“Good. We have him set up in the conference room right by me. Come on over—there’s a great coffee and doughnut spread here.”
I laugh. “Well, that’s the first good news I’ve had all day. I’m on my way.”
As I walk into the conference room in Building 2, I wave to Stacy, pondering the strange world I’ve been pulled into. I’m not used to being thrown into the middle of board politics.
As promised, by the window is a large cart with four types of coffee and six boxes of Vandal Doughnuts, a place in town so famous, there’s a long line at almost all hours of the day.
A man in wrinkled khaki pants and an untucked denim button-down shirt is kneeling in front of the cart, unpacking the doughnuts onto two platters. I had no idea Vandal Doughnuts delivered.
I pick up a cup and start filling it with coffee, eyeing all the doughnuts. I say, “You know, my wife and I are huge fans of you guys. Back when we were dating, almost every Friday night we’d wait in line for twenty minutes to get our fix. Now that we have kids, she just sends me out to get them for her. Maybe I’ll take one home for her tonight.”
I grab a huge chocolate doughnut covered with Froot Loops, as well as a giant frosted glazed doughnut with bacon on it and three more that look tasty.
