The Phoenix Project

by Gene Kim

  Before me is a scene of frenetic activity: There are tables covered with piles of incoming faxes from the stores and twelve people walk from one to the next. Each fax is a problem order waiting to be routed to an army of finance and customer service representatives who have been press-ganged into service. Their job was to either deduplicate or reverse every one of these transactions.

  In front of me, four finance people are sitting at another table, their fingers flying across ten-key calculators and open laptops. They’re manually tabulating the orders, trying to calculate the scale of the disaster and doing reconciliations to catch any mistakes.

  On the wall, they’re keeping track of the totals. So far, five thousand customers have had either duplicate payments or missing orders, and there are an estimated twenty-five thousand more transactions that still need to be investigated.

  I shake my head in disbelief. Steve is right. We massively screwed the customers this time. It’s downright embarrassing.

  On the other hand, I have to respect the operation the Finance people have put in place to handle the mess. It looks organized, with people doing what needs to get done.

  A voice next to me says, “Another Phoenix trainwreck, huh?”

  It’s John, taking in the scene like me. He’s not saying “I told you so,” but almost. With him, of course, is his ever-present black three-ring binder.

  John smacks his face with his palm. “If this were happening to our competitor, I’d be laughing my ass off. I told Chris over and over about this possibility, but he wouldn’t listen. We’re paying for it now.”

  He walks up to one of the tables and starts looking over people’s shoulders. I see his body suddenly tense as he picks a pile of papers up. He flips through the papers, his face ashen.

  He returns to where I’m standing, and whispers, “Bill, we’ve got a major problem. Outside. Now.”

  “Look at this order slip,” he hisses as we stand outside. “Do you see the problem here?”

  I look at the page. It’s a scanned order slip, slanted and low-resolution. It’s for a purchase of various auto parts, and the dollar amount seems reasonable at $53.

  I say, “Why don’t you just tell me?”

  John points to a handwritten number scrawled by the scanned credit card and customer signature. “That three-digit number is the cvv2 code on the back of the credit card. That’s there to prevent credit card fraud. Under the Payment Card Industry rules, we are not allowed to store or transmit anything on track 2 of the magnetic card stripe. Even possessing this is an automatic cardholder data breach and an automatic fine. Maybe even front-page news.”

  Oh, no. Not again.

  He continues, as if reading my mind, “Yeah, but worse this time. Instead of just being on the local news, imagine Steve being splashed on the front page of every market where we have customers and stores. And then flying to dc to be grilled by senators, on behalf of all their outraged constituents.”

  He continues, “This is really serious. Bill, we’ve got to destroy all of this information immediately.”

  I shake my head, saying, “No way. We’ve got to process every one of those orders, so that we don’t charge or even double-charge our customers. We’re obligated to do this, otherwise we’re taking money from them that we’ll eventually need to return.”

  John puts his hand on my shoulder, “That may seem important, but that’s only the tip of the iceberg. We’re already in deep shit because Phoenix leaked cardholder data. This may be just as bad. We get fined according to the number of cardholders affected.”

  He gestures at all the papers, saying, “This could more than double our fines. And you think our audits are bad now? This will make them ten times more painful, because they’ll classify us as a Level 1 merchant for the rest of eternity. They may even raise our transaction fees from three percent to—who knows how high? That could halve our retail store profit margins and—”

  He stops mid-sentence and opens up his three-ring binder to a calendar. “Oh, shit! The pci auditors are on-site today doing a business process walk-through. They’re on the second floor, interviewing the order administration staff about our operations. They’re even supposed to use this conference room!”

  “You’ve got to be kidding me,” I say as the feeling of panic starts to set in, which amazes me considering that it’s been three days of constant adrenaline.

  I turn to look through the window of the conference door and see very clearly all the finance people handling all the customer problem orders. Shit.

  “Look,” I say, “I know that sometimes people think you’re not on our side, but I really need your help. You’ve got to keep the auditors off this floor. Maybe even out of this building. I’ll put up some curtains on the windows, or maybe even barricade the door.”

  John looks at me and then nods. “Okay, I’ll handle the auditors. But I still don’t think you fully understand. As the custodians of cardholder data, we cannot allow hundreds of people to have access to it. The risk of theft and fraud is too high. We’ve got to destroy the data immediately.”

  I can’t help but laugh for a moment at the endless stream of problems.

  Forcing myself to focus, I say slowly, “Okay, I’ll make sure the Finance people understand this and handle it. Maybe we can get them all scanned and shipped to an offshore firm for them to enter.”

  “No, no, no. That’s even worse!” he says. “Remember, we’re not allowed to transmit it, let alone send it to a third party. Understand? Look, just so we can claim plausible deniability, I’m going to pretend I didn’t hear that just now. You’ve got to figure out how to destroy all this prohibited data!”

  I get pissed off at John’s mention of plausible deniability, regardless of whether it was well intentioned or not. I take a deep breath and say to him, “Keep those auditors off this floor, and I’ll worry about the card imprints. Okay?”

  He nods and says, “Roger. I’ll call you when I park the auditors somewhere safe.”

  As I watch him walk quickly down the hallway to the stairway, I keep thinking to myself, “He’s only doing his job. He’s only doing his job.”

  I swear under my breath and turn back around to look back at the conference room. And now I see the big printed sign hanging on the door, proclaiming “Phoenix pos Recovery War Room.”

  Suddenly, I feel like I’m in the movie Weekend at Bernie’s, where some teenage boys keep trying to hide or disguise a dead body from a hit man. Then I wonder if this is more like the massive around-the-clock shredding operation that allegedly happened at the offices of Arthur Andersen, the audit firm investigated after Enron failed. Am I complicit in destroying important evidence?

  What a mess. I shake my head, and walk back into the conference room to deliver the bad news.

  I finally get back down to the noc at 2:30 p.m., and survey the carnage as I make my way to my office. Seven extra tables have been set up to make more meeting spaces, and there are people assembled around each of them. Empty pizza boxes are piled up on many of the tables and in one corner of the room.

  I take a seat behind my desk, sighing in relief. I spent almost an hour with Ann’s team on the cardholder data issue, and then another half hour arguing with them that this is really their problem, not mine. I told them that I could help, but that my team was too tied up trying to keep Phoenix running to take any more responsibility.

  I realize with some amazement that this may have been the first time I’ve been able to say no to anyone in the company since I started in this role. I wonder if I could have done it if we weren’t the people almost single-handedly keeping our store order entry systems up.

  As I ponder this, my phone rings. It’s John. I answer quickly, wanting an update on the auditor issue. “Hey, John. How’s it going?”

  John replies, “Not terrible. I’ve got the auditors set up right next to me, here in Building 7. I’ve rearranged it so that all the interviews will be done here. They won’t go anywhere near the Phoenix war ro
om, and I’ve told the Building 9 security people explicitly not to let them past the front desk.”

  I chuckle at seeing John bend all the rules. “That’s great. Thanks for pulling all that together. Also, I think Ann could use your help figuring what exactly it takes to stay in compliance with the cardholder data regulations. I helped as best I could, but…”

  John says, “No problem. I’m happy to assist.”

  He hesitates for a couple of moments. “I hate to bring it up now, but you were supposed to give internal audit the sox-404 response letter today. How is that coming along?”

  I burst out in laughter. “John, our plan was to get that report done over the weekend after the Phoenix deployment. But, as you know, things didn’t quite go as planned. I doubt anyone has worked on it since Friday.”

  In a very concerned voice, John says, “You know that the entire audit committee looks at this, right? If we blow this deadline, it’s like a red flag to everyone that we have severe control issues. This could drive up the length of the external audit, too.”

  I say as reasonably as I can, “Trust me, if there was anything I could do, I would. But right now, my entire team has been working around-the-clock to support the Phoenix recovery efforts. Even if they completed the report, and all I had to do was bend over and pick it up, I couldn’t. We’re that far underwater.”

  As I’m talking, I realize how liberating it is to state that my team is absolutely at capacity and that there aren’t any calories left over for any new tasks, and people actually believe me.

  I hear John say, “You know, I could free two engineers up. Maybe they could help do some of the legwork around estimating the remediation effort? Or if you need it, we could even put them into the technical resource pool to help with recovery. They’re both very technical and experienced.”

  My ears perk up. We’ve got everyone deployed doing all sorts of things that this emergency requires and most have pulled at least one all-nighter. Some are monitoring fragile services and systems, others are helping field phone calls from the store managers, others are helping qa build systems and write tests, some are helping Development reproduce problems.

  I say immediately, “That would be incredibly helpful. Send Wes an e-mail with a couple of bullet points on each of your engineers. If he doesn’t have an urgent need for their skills, I’ll task them on generating the remediation estimates, as long as it doesn’t require interrupting anyone working Phoenix.”

  “Okay, great,” John says. “I’ll send the info to Wes later today, and I’ll let you know what he and I decide to do.”

  He signs off, and I consider the potential stroke of good fortune that someone could be working on the audit response.

  I then wonder if the fatigue is getting to me. Something is really screwy in the world when I’m finding reasons to thank Development and Security in the same day.

  CHAPTER 14

  • Tuesday, September 16

  By late Monday night, we had stabilized the situation. Working with Chris’ team, the stores finally had working cash registers again, but everyone knows it’s only a temporary fix. At least we don’t need to keep sensitive cardholder data anymore, much to John’s relief.

  It’s 10:37 a.m., and I’m standing outside of Steve’s office with Chris. He’s leaning against the wall, looking pensively at the floor. Ann, John, and Kirsten are also here, waiting for their turn, like penitent students waiting outside the principal’s office in elementary school.

  The door to Steve’s office opens and Sarah walks out. She looks ashen faced and on the verge of tears. She was the first to go in, and her session didn’t even take the whole ten minutes.

  She closes the door behind her, blows out her breath, and says to Chris and me, “You’re next.”

  “Here goes nothing…,” I say, opening the door.

  Steve stands by the window, looking out over the corporate campus. “Take a seat, gentlemen.”

  When we’re sitting down, Steve starts to pace in front of us. “I’ve talked with Sarah. As the project leader, I’m holding her responsible for the success or failure of Phoenix. I have no idea if I have a leadership problem or if she just has the wrong people on the bus.”

  My jaw drops. Did Sarah somehow manage to weasel her way out of her part in this disaster? This whole thing is her fault!

  Steve turns to Chris. “We put over $20 million into this project, and the lion’s share went to your team. From where I’m standing, we’d be better off if we had nothing to show for it. Instead, I have half my company scrambling to pick up the wreckage from the damage you caused.”

  Turning back to both of us, he continues, “In the good years, we were a five percent net margin company. That meant to make $1 million, we had to sell $20 million in products. Who knows how many sales we lost over the weekend and how many loyal customers we’ve lost forever.”

  He starts pacing again. “We’ve done a terrible disservice to our customers. They’re the people who need to fix their cars to get to work. They’re fathers working on projects with their kids. We’ve also screwed some of our best suppliers and clients.

  “To appease the people who actually used Phoenix, Marketing is now giving away $100 vouchers, which could cost us millions of dollars. Come on! We’re supposed to take money from customers, not the other way around!”

  As a former sergeant, I know there’s a time and place for dressing someone down. But this is too much. “No offense, sir, but this is supposed to be news to me? I called you, explaining what would happen, asking you to delay the launch. You not only blew me off, you told me to try to convince Sarah. Where’s your responsibility in all of this? Or have you outsourced all your thinking to her?”

  As I’m talking, I realize I may be making a big mistake by saying what I really think. Maybe it’s from weeks of crisis-fueled adrenaline, but it feels good rattling Steve’s cage. Really good.

  Steve stops pacing, pointing his finger at my forehead. “I know more about responsibility than you may learn in your entire lifetime. I’m tired of your Chicken Little routine, screaming that the sky is falling and then happily saying ‘I told you so’ afterward. I need you to come to me with some actual solutions.”

  Leaning into him, I say, “I told you exactly what was going to happen when your sidekick Sarah brought up this crazy plan almost two weeks ago. I proposed to you a timeline that would have prevented all this from happening. You tell me that I could have done better? I’m all ears.” With exaggerated respect, I add, “Sir.”

  “I’ll tell you what I need from you,” he replies calmly. “I need the business to tell me it’s no longer being held hostage by you it guys. This has been the running complaint the entire time I’ve been ceo. it is in the way of every major initiative. Meanwhile, our competitors pull away from us, leaving us in the dust. Dammit, we can’t even take a crap without it being in the way.”

  He takes a deep breath. “None of this is why you’re here today. I called you in to tell you two things. First, thanks to this latest it screwup, the board has insisted that we investigate splitting up the company. They think the company is worth more sold off in pieces. I’m against this, but they’ve already got consultants in our panties investigating its feasibility. There’s nothing I can do about that.

  “Second, I’m done playing Russian roulette with it. Phoenix just shows me that it is a competency that we may not be able to develop here. Maybe it’s not in our dna. I’ve given Dick the green light to investigate outsourcing all of it and asked him to select a vendor in ninety days.”

  Outsourcing all of it. Holy shit.

  That means everyone in my entire department may not have jobs anymore.

  That means that I may not have a job anymore.

  In a sudden and sobering instant, I realize that the feeling of elation and confidence I felt rattling Steve’s cage was only an illusion. He has all the power. With a wave of his pen, he could outsource all of us to the lowest-cost bidder from some random corne
r of the planet.

  I glance over at Chris, and he looks as shaken as I feel.

  Steve continues, “I expect you’ll give Dick all the help he requires. If you can pull off some sort of miracle during the next ninety days, we’ll consider keeping it in-house.”

  “Thank you, gentlemen. Please send in Kirsten,” he says with finality.

  “Sorry I’m late,” I say, slumping down in the booth across from Chris.

  Shell-shocked after our meeting with Steve, he and I decided to meet for lunch. In front of him is some sort of fruity drink with an umbrella. I always figured him to be a blue-collar drinker—more like Pabst Blue Ribbon, not some bachelorette party mixed drink.

  He laughs humorlessly. “Trust me. You showing up ten minutes late is the least of my problems. Get yourself a drink.”

  Paige tells me repeatedly that I shouldn’t trust this guy. She has a great instinct for people, but when it comes to me, she’s embarrassingly protective, which makes me laugh. I’m the ex-Marine, after all. She’s just a “nice nurse.”

  “Any pilsner you have on tap, please,” I say to the waitress. “And a scotch and water, too. It’s been a rough day.”

  “That’s what I heard. No problem at all, sweetie,” she replies, laughing. To Chris, she asks, “Another mai tai for you?”

  He nods, handing her his empty glass. So that’s what a mai tai looks like. I’ve never tried one. We Marines are very self-conscious about what we’re seen drinking.

  Chris raises his water glass and says, “To having a common death sentence.”

  I smile wanly and raise my glass. Feeling obligated to inject some optimism, I say, “And here’s to figuring out how we get a stay of execution from the governor.”

  We clink glasses.

  “You know, I’ve been thinking,” Chris says. “Maybe my group being outsourced wouldn’t be the worst thing in the world. I’ve been in software development for virtually my entire career. I’m used to everyone demanding miracles, expecting the impossible, people changing requirements at the last minute, but, after living through this latest nightmare project, I wonder if it might be time for a change…”