by Gene Kim
When this happens now, we quickly jump on it to make sure that it doesn’t happen again.
And it’s not just Brent’s work that we’re improving. By reducing the number of projects in flight, we’re keeping clear lanes of work, so work can go from one work center to the other quickly, getting completed in record time.
We’ve all but emptied our ticketing system of outdated work. In one case, we even found a ticket that Wes put in over ten years ago as a junior engineer, referring to some task for a machine that has been long since decommissioned. Now we have confidence that all work in the system is important and actually has a prayer of being completed.
We are no longer the Bates Motel of work.
Against my staff’s expectations, we keep bumping up the number of projects we think we can handle concurrently. Because we have a better idea of what our flows of work are, and managing carefully which ones are allowed to go to Brent, we’re finding that we can keep releasing more projects without impacting our existing commitments.
I no longer think of Erik as a raving madman, but he’s eccentric, for sure. Now that I’ve seen the results with my own eyes in my own organization, I know that it Operations work is very similar to plant work. Erik has stated repeatedly that our improvements to date are only the tip of the iceberg.
Erik says that we are starting to master the First Way: We’re curbing the handoffs of defects to downstream work centers, managing the flow of work, setting the tempo by our constraints, and, based on our results from audit and from Dick, we’re understanding better than we ever have what is important versus what is not.
At the end, I led the retrospective portion, where we self-assessed how we did and the areas that we should improve. When someone mentioned that we should start inviting people from Development when we do our outage postmortem root cause analysis meetings, I realized that we are now also well on our way to understanding Erik’s Third Way, as well.
As Erik keeps reminding me, a great team performs best when they practice. Practice creates habits, and habits create mastery of any process or skill. Whether it’s calisthenics, sports training, playing a musical instrument, or in my experience, the endless drilling we did in the Marines. Repetition, especially for things that require teamwork, creates trust and transparency.
Last week, as I sat through our latest biweekly outage drill, I was very impressed. We were getting very good at this.
I feel certain that if the payroll failure that happened on my first day of the job happened now, we could complete the entire payroll run—not just the salaried staff, but the hourly staff, as well.
John quickly got the approval from Dick and Steve to have an outsourcer to take over the cafeteria pos systems and replace it with something commercially supported.
It was a fascinating exercise for Wes, Patty, and me to work with John to put together the outsourcing requirements for the cafeteria pos systems. As part of the due diligence process, we were going to hear from all the prospective outsourcers all the dogmas we used to believe before all our interactions with Erik. It will be interesting to see if we can retrain them.
It seems to me that if anyone is managing it without talking about the Three Ways, they are managing it on dangerously faulty assumptions.
As I’m pondering this, my phone rings. It’s John.
When I answer, he says, “My team discovered something troubling today. To prevent unauthorized black market it activities from cropping up, we’ve started routinely reviewing all the proposed projects coming into Kirsten’s Project Management Office. We also search all the corporate credit cards for recurring charges that might be for online or cloud services—which is just another form of unauthorized it. Some people are going around the project freeze. You have time to talk?”
“Let’s meet in ten minutes,” I say. “Don’t leave me hanging. Who’s trying to backdoor the system?”
I hear John laugh on the other end of the line. “Sarah. Who else?”
I invite Wes and Patty to the impromptu meeting but only Patty can make it.
John starts presenting what he found. Sarah’s group has four instances of using outside vendors and online services. Two are relatively innocuous but the others are more serious: she has contracted a vendor for a $200,000 project to do customer data mining and another vendor to plug into all our pos systems to get sales data for customer analytics.
“The first problem is that both projects violate the data privacy policy that we’ve given our customers,” John says. “We repeatedly promise that we will not share data with partners. Whether we change that policy or not is, of course, a business decision. But make no mistake, if we go ahead with the customer data mining initiative, we’re out of compliance with our own privacy policy. We may even be breaking several state privacy regulations that expose us to some liability.”
This doesn’t sound good, but John’s tone of voice suggests there’s worse to come. “The second problem is that Sarah’s vendor uses the same database technology that we used for our cafeteria pos system, which we know is virtually impossible to secure and maintain support for in production, if and when it becomes a part of daily operations.”
I feel my face get red hot. It’s not just about another cafeteria pos system that we’ll need to retrofit for production. It’s because applications like this contribute to our inaccurate sales order entry and inventory management data. We have too many cooks in the kitchen and no one accountable for maintaining the integrity of the data.
“Look, I don’t care about Sarah’s project management and invoicing tools—if it makes them more productive, let them use it,” I say. “It’s probably safe as long as it doesn’t interface with an existing business system, store confidential data, affect financial reporting, or whatever. But if it does, then we need to be involved and at least confirm that it doesn’t impact any of our existing commitments.”
“I agree,” John says, “Want me to take the first stab at that outsourced it service policy document?”
“Perfect,” I say. But with less certainty, I continue, “Although, what’s the right way to handle Sarah? I feel completely out of my league. Steve constantly protects her. How do we convey to him the potential mayhem she’s causing with her unauthorized projects?”
Making sure John’s office door is closed, I say to John and Patty “Guys, help me out. What does Steve see in her? How does she get away with so much crap? Over the past couple of weeks, I see how hard-nosed Steve can be, but Sarah routinely gets away with murder. Why?”
Patty snorts. “If Steve were a woman, I’d say that he’s attracted to dangerous men. A bunch of us have speculated about this for years. I’ve had a theory, which I must say, was pretty much validated in our last off-site.”
When she sees John and me both conspiratorially leaning forward, she smiles. “Steve prides himself on being an operations guy, and he’s admitted several times in company meetings that he doesn’t have a flair for strategy. I think that’s why he loved working with his old boss and our new chairman Bob so much. For a decade, Bob was the strategy guy, and all Steve had to do was execute the vision.
“For years, Steve searched for a strategy person to be his right-hand man. He went through quite a few people, even setting a couple executives against each other in this awful, drawn out competition. Pretty Machiavellian,” she continues. “And Sarah won. The word on the street was that there was a lot of backstabbing and underhanded tactics, but I suppose that’s what it takes to come out on top. Evidently, she has mastered how to whisper the right things in his ear, reinforcing his paranoia and aspirations.”
Patty’s explanation is so much more sophisticated than anything I’ve come up with. In fact, it sounds strikingly similar to what Paige would speculate when I got that distant, angry look at dinnertime.
John says awkwardly, “Umm, you don’t think there’s anything between them, do you? Like, anything…untoward?”
I raise my eyebrows. I wondered about that, to
o.
Patty just bursts out laughing. “I’m a pretty good judge of people. Both my parents were psychologists. I’d eat both of their diplomas if that were true.”
Seeing the expression on my face, she laughs even harder. “Look, not even Wes believes that, and there’s no one better at manufacturing drama than him. Sarah’s scared to death of Steve! You ever notice that when someone is talking, Sarah is always still looking at Steve, trying to gauge his reaction? It’s freakish, actually.”
She continues, “Steve has a blind spot for Sarah’s shortcomings, because she has something he needs and admires, which is the ability to come up with creative strategies, regardless of whether the strategy is good or bad. On the other hand, because Sarah is so insecure, she’ll do whatever it takes to not look bad.
“She simply doesn’t care about the body count she leaves in her wake, because she wants to be the next ceo of Parts Unlimited.” Patty says. “And apparently, Steve does too. He’s been grooming her as his successor for years.”
“What? She could be our next ceo?” I exclaim in shock, quickly wiping up the coffee I spit out onto John’s conference table.
“Wow, boss. You don’t hang out at the water cooler much, do you?” Patty says.
It’s Phoenix deployment day, and I’ve missed Halloween with my kids.
It’s already 11:40 p.m. As we’re standing once again around the noc conference table, I have an unsettling feeling of déjà vu. I count fifteen people here, including Chris and William.
Most people are tensely huddled around the table with laptops open, pizza boxes and candy wrappers piled behind them. Several other people are at the whiteboard, pointing at checklists or diagrams.
It took three hours longer than scheduled to migrate Phoenix into the qa test environment and get all the tests to pass. Although this is much better than the previous deployment, I thought we’d have fewer problems, given how hard we worked on improving the deployment process,
By 9:30 p.m., we were finally ready to do the migration into production. All the tests had finally passed, and Chris and William gave the thumbs-up to deploy. Wes, Patty, and I looked at the test reports, and gave the green light to start the deployment work.
Then all hell broke loose.
One of the critical database migration steps failed. We had only completed thirty percent of the deployment steps, and once again, we were dead in the water. Due to the database changes and the scripts already run, it wasn’t possible to roll back in the time remaining before the stores opened tomorrow morning.
Once again, we had to fight our way forward, trying to get to the next step so the deployment could resume.
Leaning against the wall, I watch everyone work, arms crossed, trying not to pace. It’s frustrating that once again we are grappling with another Phoenix deployment going bad, with potentially disastrous outcomes.
On the other hand, compared to last time, things are much calmer. While there is tension, and a lot of heated arguments, everyone is intensely focused on problem solving. We’ve already notified all the store managers about our progress, and they all have manual fallback procedures ready, just in case the pos systems are down when the stores open.
I see Wes say something to Brent, stand up to rub his forehead wearily, and then walk toward me. Chris and William also get up and follow him.
I meet them halfway. “Well?” I ask.
“Well,” Wes replies when he’s close enough to speak softly and be heard. “We’ve found our smoking gun. We just discovered that Brent made a change to the production database a couple of weeks ago to support a Phoenix business intelligence module. No one knew about it, let alone documented it. It conflicts with some of the Phoenix database changes, so Chris’ guys are going to need to do some recoding.”
“Shit,” I say. “Wait. Which Phoenix module?”
“It’s one of Sarah’s projects that we released after the project freeze was lifted,” he replies. “It was before we put the kanban around Brent. It was a database schema change that slipped through the cracks.”
I swear under my breath. Sarah again?
Chris has a pinched expression on his face. “This is going to be tricky. We’re going to have to rename a bunch of database columns, which will affect, who knows, maybe hundreds of files. And all the support scripts. This is going to be manual work and very error-prone.”
He turns to William. “What can we do to at least get some basic testing performed before we continue the deployment?”
William looks vaguely ill, wiping sweat from his face with his hands. “This is very, very…dicey… We can test, but we still may not find the errors until we hit those lines of code. That means we’re going to have failures in production, where the application just blows up. It may even take down the in-store pos systems, and that would be bad.”
He looks at his watch. “We’ve only got six hours to complete the work. Because we don’t have enough time to rerun all the tests, we’re going to have to take some shortcuts.”
We spend the next ten minutes sketching out a revised schedule that still completes by 6 a.m., enabling the stores to open normally with an hour to spare. As Chris and William head off to notify their teams, I indicate to Wes to remain behind.
“When we’re out of the woods,” I say, “we’ve got to figure out how to prevent this from happening again. There should be absolutely no way that the Dev and qa environments don’t match the production environment.”
“You’re right,“ Wes says, shaking his head in disbelief. “I don’t know how we’re going to do it. But you’ve got no argument from me.”
He looks behind him at Brent, incredulous. “Can you believe that Brent is at the center of all of this again?”
Much later, when the deployment is declared to be complete, everyone applauds. I look at my watch. It’s 5:42 a.m. on Saturday morning. The team spent the entire night working, completing the deployment twenty minutes early. That is, twenty minutes early according to the emergency schedule we hammered out. According to the original schedule, we finished almost six hours late.
William has confirmed that the test pos systems are working, as well as the e-commerce website and all the associated Phoenix modules.
Patty has started sending out notifications to all the store managers that the deployment was “successful.” She is attaching a list of known errors to look out for, an internal web page to get the latest Phoenix status, and instructions on how they can report any new problems. We’re keeping all the service desk people on standby, and both Chris and my teams are on-call to provide early life support. Basically, we’re all on standby to support the business.
With Wes and Patty handling the on-call schedule, I say “good job” to everyone and pack up my things. On my drive home, I wrack my brain, trying to think of how we can keep each Phoenix deployment from causing an emergency.
CHAPTER 29
• Monday, November 3
At 7:10 a.m. the following Monday, Chris, Wes, Patty, and John are again all seated with me in the boardroom. While we’re waiting for Steve, we talk about the aftermath of the second Phoenix deployment.
Erik is in the back of the room. In front of him is a bowl, an emptied instant oatmeal packet, and a French press full of greenish water, with leaves floating in it.
Seeing my puzzled expression, he says, “Yerba maté. My favorite drink from South America. I never travel without it.”
Steve walks through the door, still talking on his cell phone. “Look, Ron, for the last time, no! No more discounts—even if they were our last remaining customer. We’ve got to hold the line somewhere. Got it?”
He hangs up, exasperated, and finally sits down at the head of the table, muttering, “Sorry I’m late.” He opens up his folder, taking a moment to study something inside it.
“Despite how the Phoenix deployment went over the weekend, I’m extremely proud of everything you’ve have done over the last couple of weeks. Many people have told me about how
pleased they are with it. Even Dick,” he says incredulously. “He’s told me about how you’re helping to improve our key company performance measures, and he thinks it will be a game changer.”
He smiles. “I am very proud to be a part of this team that is obviously working together better than ever, trusting one another, and getting incredible results.”
He turns to John. “By the way, Dick has also told me that with your help, they’ve established that the financial restatement won’t be material.” Breaking into a smile, he says, “Thank God. I won’t be on the cover of Fortune magazine wearing handcuffs, after all.”
Just then, Sarah knocks on the door and enters the room.
“Good morning, Steve,” she says, as she walks in primly, sitting down besides Erik. “I take it you wanted to see me about my new marketing initiatives?”
“You mean, the unauthorized shifts of work you’re running inside the it factory, like some unscrupulous Chinese plant manager?” Erik asks.
Sarah looks Erik up and down, obviously sizing him up.
Steve indicates to John to present his findings. When he concludes, Steve says sternly, “Sarah, I issued a clear statement. No one is allowed to start any new it initiatives, internal or external, without my explicit approval. Please explain your actions.”
Sarah picks up her iPhone and angrily taps away for a couple of moments. Putting it down, she says, “Our competitors are kicking our ass. We need every advantage we can get. To achieve the stated objectives that you’ve laid out, I can’t wait for it. I’m sure they’re working very hard, doing their best with what they have and what they know—but it’s not enough. We need to be nimble, and sometimes we need to buy instead of build.”
Wes rolls his eyes.
I respond, “I know that it hasn’t always been able to deliver what you’ve needed in the past, and I know Marketing and Sales have gotten burned. We want the business to win as much as you do. The problem is that some of your creative initiatives are jeopardizing other important company commitments, such as complying with state laws and regulations on data privacy, as well as our need to stay focused on Phoenix.
