A weary voice sounded over the phone. “I think we need to take ARC’s findings seriously,” Burke said, breaking his long silence. “The lack of radar detection is not definitive for a few reasons. For one, Russia and China could have launched nuclear-armed hypersonic missiles which cannot be reliably detected by our radar. Second, radar systems are not infallible. That’s why we use dual confirmation from satellites to detect nuclear attacks. And then there’s the issue with someone taking our satellite early warning system offline. Wouldn’t they also attempt to interfere with our radars? There is a chance that the ARC system knows something we don’t. I’d rather not bet against it in favor of your gut feeling, Madam President.”
Susan took a deep breath as heat flushed through her body. A few of her advisors and military commanders nodded in agreement. She couldn’t fathom how they could have more trust in a machine than their own common sense.
She cleared her throat. “What do we know about the missiles?”
“Ma’am, the three unidentified objects launched from Siberia took off about four minutes ago and are above the horizon,” the NORAD commander said. “From our sea-based X-band radar sensor data, their signatures suggest that they are space rockets rather than ballistic missiles. The rockets appear to be on a direct ascent pathway into low earth orbit at the moment.”
“You’re saying the three so-called missiles from Russia currently displayed in red on ARC’s dashboard are actually rockets travelling into space?” Susan asked.
“Yes, ma’am,” the NORAD commander said.
“That appears to confirm my theory,” David said. “These rockets are sending microsatellites into space for Russia.”
“Then why does ARC view these launches as threats?” Susan asked, scrunching her face.
No one answered. Because no one had the answer. None of her advisors truly understood what went on inside the deep neural network. And yet, they would rather place their trust in that thing than in her leadership.
“What about the red missile indicators over the Northwest Passage?” Burke asked over the phone. “We spent much of this morning discussing our concerns about China’s two lost submarines. Granted, the ARC system seems a bit off, but it has detected a nuclear attack. What if it has misread the threat posed by the Russian missiles but gotten the threat of the Chinese missiles right?”
“Sir, we’re not picking up any signals on our radars for those missiles,” the NORAD commander said.
“So, we’re not certain where they are headed,” Burke said.
“They’re probably part of China’s war game,” David said. “Just like the three earlier launches.”
“I’m not sure I feel comfortable making such blind assumptions,” Burke said.
“Why would China first launch three hypersonic missiles as part of its war game to put us on alert and then launch a surprise nuclear attack from its concealed submarines?” Elizabeth asked.
No one spoke up with an answer. Because it didn’t make any sense.
“Ma’am, should we give the Chinese ambassador a call?” Elizabeth asked. “Maybe if Chen understands the situation, he could give us a straight answer about what’s going on.”
Susan eyed the clock again. Five minutes and twelve seconds. “No. We don’t have time for a phone call. By the time you get him on the phone and explain the situation, the clock will have run out.”
“It’s possible that China noticed our increase in alert status to DEFCON 3,” Burke said. “That could explain their changing course of action.”
“Or what if Morgan was right about the defection scenario this morning?” David blurted.
Susan swung her head in his direction, her eyes wider than before.
“You’re suggesting a rogue submarine commander might have launched nuclear weapons at us?” Elizabeth asked.
“It’s possible,” David said. “And it could explain why the ARC system has gone completely haywire.”
“But we have still detected no missiles approaching our airspace,” Elizabeth said, her eyes round and pupils dilated. “Ma’am, I’m worried this is a false alarm.”
Out of the corner of her eye, the clock now read four minutes and fifty-one seconds.
“Does any of this really matter?” Susan asked briskly. “We need to make a decision. Here. Now. With the limited information we have on hand.” She paused to take a deep breath. “The ARC system has issued a launch order for an all-out nuclear attack against Russia and China. If, as I suspect, the ARC system is wrong, and we allow the launch to proceed, then we will have accidentally killed millions of innocent people. We will also have started a nuclear war, which will lead to a retaliatory strike, devastating U.S. cities. If the ARC system is right, then we will absorb an initial blow from Russia and/or China, and our ability to retaliate may be compromised. Either way, millions of Americans will die. Issuing the rescind order is the only scenario that has the potential to avoid such a catastrophe altogether.”
There was silence in the room. Most of her advisors stared pale-faced at the clock, which read four minutes and five seconds. Susan glanced at Elise, who gave her a grim nod.
Susan reached for the keypad, grabbed the nuclear code card, and entered the 24-digit number for the rescind order. Before pressing the enter key, she compared the entry screen to the number on her card to confirm she’d entered the right number. She inhaled deeply and pressed the button. Then she turned her attention to the ARC dashboard, the sound of her heart pounding in her ears.
The ticker at the bottom of the hologram dashboard announced that the rescind order had been received and acknowledged by the ARC system. Exhaling sharply, Susan leaned back in her chair and stared numbly at the table, wondering if she had mere minutes to live. Her eyes flickered back to the countdown clock, and she was jolted out of her seat.
The clock continued its countdown, second by second.
Three minutes and fifty-eight seconds.
51
Countdown
GRACE
1703
National Military Command Center
The Pentagon
Arlington, Virginia
Did the president make the right call?
Grace breathed steadily, consciously moving air in and out of her lungs in an attempt to calm her nerves. Her muscles remained tense, and she sat stiffly along the back wall of the war room. To her left, Arjun cowered in the corner with his face scrunched up. Grace wasn’t sure if he was more horrified of what might come or that he’d helped to bring it about.
General Hawkins stood in front of the videoconference screen in the National Military Command Center, awaiting the president’s next order. In the absence of the SecDef and chairman, he’d taken charge of operations in the war room. Surrounded by senior civilian and military leaders of the Pentagon, his eyes remained stubbornly stuck on the countdown clock. Grace suspected that General Hawkins, like everyone else, was waiting to learn the answer to her question.
Did the president make the right call?
Grace jerked her head toward the clock when she thought she saw the number of seconds change. She blinked her eyes a few times just to be sure. There were three minutes and fifty-eight seconds left. Then, three minutes and fifty-seven seconds. Three minutes and fifty-six seconds. She tilted her head and frowned.
Wait. Isn’t the countdown clock supposed to stop now?
The ticker at the bottom of ARC’s dashboard revealed an ominous announcement, followed by the same airy, female voice from before.
ARC has canceled the rescind order issued by the President
In three minutes and fifty-five seconds, ARC will transmit the launch order to U.S. nuclear forces
the nuclear attack will commence as planned
A few people gasped audibly. Others looked to be in a state of shock. There were terrified faces everywhere she looked. The room felt as if it had dropped a few degrees in temperature. Even General Hawkins appeared visibly shaken by the news.
No o
ne had seen this coming. No one had anticipated that the autonomous system would learn how to strip its designers of their autonomy. If someone had any inkling at all that something like this were possible with ARC, the Department of Defense would never have launched the system.
Grace sat stunned for a moment, trying to grasp the terrible truth.
The rescind order has been rescinded. We’re going to start a nuclear war.
“Ma’am,” General Hawkins said, “should we initiate emergency procedures for devolution and delegation of authority to ensure the line succession in the—”
“No,” Tolley said. “General Burke will be arriving at Raven Rock shortly and will fill in for us if need be.”
We’re probably going to die here if we don’t stop ARC.
Grace sank into her chair, leaned forward, and rested her face in her hands. She started going back through everything she knew about ARC’s code in her head. But every time she followed a potential line of reasoning, she came to the same dead end.
There’s no way out of this.
“Let’s focus on stopping the ARC system from carrying out the launch,” Tolley said on the video screen. “General Hawkins, I want to shut it down now.”
“Ma’am, that’s no longer possible,” General Hawkins said, his chin dipping.
“What do you mean, no longer possible?” Tolley asked.
“Once the ARC system gives the launch order, the option to shut it down is disabled,” he said. “The system engineers designed it this way for our own protection. The rescind order is the only way to stop ARC from carrying out the launch. And…” General Hawkins went oddly silent.
For our own protection?
Grace shook her head in disgust and glanced at the clock.
Four minutes and ten seconds.
Adrenaline pumping through her veins, she searched her mind frantically for a possible way out.
Could Arjun tweak the algorithms again?
“That has worked out really well for us, hasn’t it?” Tolley seemed to finish the general’s thought for him and then shook her head in apparent disbelief. When the president looked up again, she appeared to be looking intently at the screen as if she were looking for someone or something. “Where is Major Lim?”
Grace snapped her head up, her heart pounding even harder than before. She furrowed her brow.
The president is asking for me?
“Ma’am?” General Hawkins asked.
“I’m looking for Major Grace Lim,” Tolley said. “Is she in the room with you or not?”
General Hawkins nodded and turned his head toward Grace, his hardened eyes landing squarely on her face. With an eyebrow raised, he motioned for her to hurry up and join him at the center of the room.
While the rest of the senior Pentagon leadership stared at her in surprise, Grace pushed her heavy legs forward as fast as she could toward him. When she came into view of the camera, the president gave her a half smile.
“Ma’am, how can I be of help?” Grace asked, her voice shaking. She stole another glance at the clock and gulped.
Three minutes and fifty seconds.
“Morgan mentioned you might know a way to get us out of this mess. She said you understand what’s happening with ARC?”
She did? But why would she say that?
Grace’s mind searched desperately for the reason. With everything that had happened, her conversation with Morgan felt so long ago. Then it came to her like a bolt out of the blue. She remembered Morgan’s theory about ARC and quickly realized how that might explain their current predicament.
“Ma’am, ARC has overridden the rescind order because it defies the first rule of the system,” Grace said.
“The first rule?” Tolley asked.
“Yes, ma’am, as its first rule, ARC is designed to optimize its ability to retaliate after a nuclear attack. The system engineers designed it that way because we believe the fear of retaliation has prevented nuclear war for decades.”
Tolley rubbed her chin for a moment and said, “Then ARC must have perceived a severe threat to its ability to retaliate. Otherwise, it would not have launched this attack.”
Grace nodded. “Yes, ma’am, that’s what I’m thinking. It’s possible that ARC encountered new technological and geopolitical developments for which there was no matching training data to guide its interpretations within the model of deterrence. As a result, ARC has improvised in order to uphold its first rule and adopted an offensive posture. If ARC is able to take out the nuclear arsenals of our adversaries with a massive first strike, it will be better positioned to retaliate in the next moves.”
Tolley nodded eagerly. “ARC might not have to retaliate at all because Russia and China will be rendered incapable of launching a massive counterstrike. This would be the optimal situation from ARC’s perspective.”
“Exactly,” Grace said. “But not so optimal for us.”
“Then how do we fix this?” Tolley asked. “Is there a way to adjust ARC’s programming?” She paused for a moment. “General Hawkins, you said we can no longer shut ARC down. Are we also unable to make changes?”
General Hawkins cleared his throat. “Ma’am, unfortunately, now that ARC has given the launch order, we are completely shut out of the system until the order is carried out.”
“Actually, I may have an idea to get around that,” Grace said tentatively. She turned to look at Arjun, who was now sitting up straight in his chair and appeared to be on alert. The urgent look on his face suggested he’d just had exactly the same thought she did. Turning back to the president, Grace said, “Ma’am, Centoreum Tech created an invisible back door to the ARC system to make changes to the algorithms. By invisible, I mean that it’s undetectable to us at the Pentagon, but also to the ARC system itself. We could try using this back door to somehow modify the first rule of the system.”
“You think that would convince ARC to back down and rescind its own launch order?” Tolley asked.
Grace nodded. “If we change the first rule of ARC from optimizing its ability to retaliate to something like preventing nuclear war instead, I believe it would cancel the launch and retreat into a defensive stance.”
“Do it,” Tolley said firmly.
Grace peeked at the clock and shuddered. Three minutes and two seconds.
“Um, ma’am. Unfortunately, I can’t do it myself. I need help from Centoreum Tech’s systems engineer. And he’s right here in the room with us.”
Tolley glared at the video screen, her face muscles tightening. “Is this the same engineer who cheated on ARC’s tests?”
Grace grimaced. She craned her neck and saw Arjun squirming uncomfortably. “Yes, ma’am, but we don’t have time to worry about his mistakes right now. Please… he’s the only one who intimately knows his way around ARC’s code.”
Two minutes and fifty seconds.
“Fine.”
Grace and Arjun hurried over to a terminal. From a standing position, she typed in her login, password, and authorization code to open a non-classified connection to the internet. Then Arjun sat down in front of the computer, and Grace hovered behind him. General Hawkins came up behind them, his arms crossed, a scowl on his face. Arjun began typing quickly. The other civilian and military leaders formed a tight huddle around them.
“I first need to get through the backdoor without being detected by ARC,” Arjun said.
“But I thought ARC couldn’t detect the back door,” Grace said.
“We’ve now entered wartime operations, and ARC is programmed to prevent any type of intrusion,” Arjun said. “With the front door disabled, it will be easier for ARC to detect us if we’re not careful.” He typed so fast, she couldn’t keep track of what he was doing.
Grace glanced at the clock.
Two minutes and twenty-eight seconds.
“You need to hurry up,” she said. “We have less than three minutes left.”
“I’m going as fast as I can,” Arjun said, wiping sweat from his
brow. His fingers raced across the keys for several seconds. “There. I’ve entered the ARC system through the backdoor.” He looked up at the hologram dashboard.
“What are you looking for?” Grace asked.
“I’m just checking for an alert message from ARC about a possible intrusion, but it looks like we’re good.”
When Grace glimpsed the countdown clock, she put her hands on her cheeks and bit her lip.
One minute and forty seconds.
“Now I need to find the right location in ARC’s source code,” Arjun said, staring back at the screen full of computer code. His eyes darted back and forth as he read each line.
Grace closed her eyes and focused on her breathing.
“Found it,” Arjun said. “I found the code for the first rule.”
Her eyes fluttered open. “Do you know how to change it?” Grace asked.
“Not sure,” Arjun said. “I need to think for a moment. I’m not sure how to express a new first rule.”
“Don’t think for too long,” Grace said. She stared at the screen and tried to make sense of the code. “If there’s a first rule, wouldn’t there also be a second rule? Maybe we could just cancel the first one.”
“Yes, but it’s not that simple. ARC’s system is based on the model of deterrence, which consists of a number of rules that are considered together in order to prevent nuclear war. And the notion of preventing nuclear war might be too vague to change anything. The ARC system already thinks it’s preventing nuclear war by removing the capability of our adversaries to fight nuclear war in the first place.”
Grace wrinkled her brow. “Is there a way to limit ARC’s use of nuclear weapons?” Her eyes darted to the clock, and her heart sank. “What if we code the first rule to prohibit ARC from using nuclear weapons except after getting hit by a nuclear attack?”
Rescind Order Page 34
