Rescind Order

by Natasha Bajema

  Drew’s face flushed. “Okay, sorry. It’s just that most people get the wrong impression from how artificial intelligence is portrayed in the movies. Many assume that AI-enabled machines are as intelligent as people, or in some cases, even more intelligent than humans. But we’re not even close to building machines anywhere near that. What makes today’s AI machines powerful is their ability to learn and adapt to their environment by analyzing massive volumes of data. It’s the data that enables them to be autonomous. And that’s much different from human intelligence.”

  “So, you’re saying that machines learn, but not at all like humans do,” Tori said. “Can you explain that?”

  “Sure.” Drew scratched his head, thinking for a moment. “Sometimes it helps to compare how machines and children learn about the world around them. A child learns the difference between a cat and a dog from just a few examples. Once children know what cats and dogs are, they can identify them in any position, from any angle, and in any type of lighting. A machine, on the other hand, needs to see millions of labeled images of cats and dogs in different angles, positions, and types of lighting to determine whether something is indeed a cat or a dog. That’s because a machine doesn’t actually know what a cat or a dog is… rather, a machine detects tiny patterns in an image and compares them to a massive set of images they’ve been trained on. Then a machine determines the answer based on probability. That means there is always a small margin of error in the outcomes.”

  “Are you saying that humans learn faster than machines?” Tori asked, wrinkling her nose.

  Drew shook his head. “Not faster. It’s true that machines can process millions of images in a few seconds. But humans learn an incredible amount from far fewer examples. Humans are also capable of transferring knowledge from one task to another. Even if they don’t know the exact species, most children can tell you if something is an animal or not. They know this because they have seen other animals and understand intuitively what features animals have and how they behave and relate to us.”

  “And they don’t have to see millions of images of animals to do it,” Tori said, her face lighting up with understanding.

  “Correct. By the time a young child learns to identify dogs, cats, and other domesticated animals, they may be able to learn the names of wild animals, such as elephants, with only a single example.”

  “And a machine would still need millions of images to learn this?”

  “Yep,” Drew said.

  “How does this apply to autonomous weapons, then?” Tori asked.

  “Okay, so far, we’ve been talking about visual object recognition,” Drew said. “This is an important feature for developing autonomous weapons systems. In the same way that a machine can learn the difference between cats and dogs, it must now learn how to accurately detect, identify, and eliminate specific targets without human assistance and intervention. This is a far more complex task. For example, an autonomous system must distinguish between an enemy tank and a friendly tank in mere seconds.”

  “But don’t tanks have different designs and distinct features?” Tori asked.

  Drew smiled, realizing he’d finally come full circle. “They do, but that’s where the data problem comes in. Take Russian tanks, for example. Our intelligence services have pictures of different models of Russian tanks, but not millions of images of every model from all angles, positions, and in all types of lighting. That means that autonomous weapons systems have far fewer examples to learn from. Their margins of error will be greater and have life-or-death consequences.”

  Tori’s eyes lit up. “So, you’re saying that autonomous weapons are bound to make some pretty costly mistakes.”

  “Exactly,” Drew said, grinning from ear to ear. “But the data issue gets even more complex when adversaries attempt to trick each other.”

  “How so?” Tori asked.

  Drew cocked his head, enjoying a new surge in his confidence. He was on a roll. “If an adversary wants to fool our autonomous weapons systems, they could simply modify their tanks to look like ours. Or change them in such a way that the system would fail to identify them as enemy targets. Advanced adversaries might also use metamaterials to cloak their tanks, making them invisible to certain wavelengths on the electromagnetic spectrum. They might also engage in cyberattacks against the deep neural networks inside our autonomous weapons systems and feed them false data to corrupt their outcomes. The false data would look normal to human eyes so we’d never know the system was corrupted.”

  Drew stopped, suddenly feeling self-conscious when he saw Tori’s face contort. Did I get too technical again?

  Tori scratched her cheek and smiled. “Well, this has been incredibly fascinating, but we’re running out of time. I do have one more question for you this morning,” she said, giving him a big smile.

  “Sure, no problem,” Drew exhaled, relieved that he hadn’t put her off.

  “If you could say anything to President Tolley right now, what would you want her to know?” Tori asked.

  Drew jerked his head slightly. He wasn’t expecting this sort of question after their extended conversation about data. He paused for a moment to collect his thoughts and stared at the ground, searching his brain for the best catchphrase. If Drew wanted to have impact, it had to be something that people understood and feared. Something that could go viral.

  Then he lifted his chin to the camera and said, “Today, President Tolley wants to authorize the Pentagon to unleash the Terminator. But we already know how those movies end. Humans die en masse. It’s time for her to reconsider America’s future and the risks of artificial intelligence on the battlefield.”

  “Thanks so much,” Tori said, motioning to her cameraman to stop rolling. “I think that’s a wrap.”

  “No problem,” Drew said, glancing at his watch. He was too nervous to ask her when the video might air. Fumbling with his hands, he waved goodbye to Tori as he hurried back toward the umbrella. When it came into view, Drew saw that two of his friends had already arrived. They were standing in his spot and glaring at him for shirking his duty.

  11

  Cats versus Dogs

  GRACE

  0640

  National Military Command Center

  The Pentagon

  Arlington, Virginia

  Grace could feel the heat rise behind her ears. She’d been trying to describe the problem with the ARC system to Colonel Martinez for the past twenty minutes. And he wasn’t getting any of it. She took a deep breath and stretched her neck to release some tension.

  Martinez gave her an apologetic look. “Sorry I’m so daft.”

  “Don’t worry about it. It’s a complex topic,” Grace said, trying her best to hide her frustration. “We’ll start from the beginning again.”

  For a moment, she considered the best approach. If she couldn’t get the colonel to grasp the basics, the risk she was taking by circumventing her chain of command would all be for naught.

  “Okay, you understand the basics of how the ARC system functions, right?” Grace asked.

  Martinez gave her a half shrug. “I know that ARC’s deep neural network consists of layered algorithms and that the network learns patterns from massive volumes of data. But I’m not sure what that means.”

  “During the training phase, we fed the ARC system billions of data points, and the algorithms learned to produce specified outcomes,” Grace said.

  Martinez’s eyes lit up as if he remembered something. “If I recall correctly from the command brief I received last month, ARC consists of four major components that are linked together and activate in a phased timeline, each phase taking place before the first nuclear detonation on our soil.”

  Grace bobbed her head and gave him an encouraging smile.

  Martinez grinned back and began to count them off on his hand. “As its first component, ARC operates using a big data-driven intelligence, surveillance, and reconnaissance (ISR) deep neural network. That’s the critical starting
point, right?”

  “Yes, by feeding ARC’s network new ISR data each month, the system can determine potential changes in the behavior of our nuclear adversaries and the implications of new technologies for deterrence. The ISR component leverages data inputs from all government agencies and provides strategic warning to our operations centers for any emergent nuclear threats.”

  Martinez lifted another finger and looked at her. “The early warning system represents the second component and provides reliable and early detection of nuclear attacks. It gathers sensor data in real time from radars and satellites to detect launches of nuclear weapons.”

  Grace nodded. “The sensor data are fed into ARC, and the system issues a nuclear attack warning or no warning.”

  Martinez raised his third finger. “Then in the event of a nuclear attack by an adversary, the third component, our command and control system enables the prompt authorization of retaliation before the enemy’s nuclear weapons reach U.S. soil. It determines the most effective retaliatory nuclear attack and decides how many missiles to launch, how many bombs to deliver, what targets to hit, and in what order.” He raised his fourth finger. “As the fourth component, ARC provides a rapid dissemination of the launch order to our nuclear forces and delivers the weapons to targets in the attacking country. The main idea behind the ARC system is to accelerate and improve nuclear decision-making and ensure effective retaliation. That’s about all I know.”

  Grace smiled, flashing her teeth. “Awesome. That’s something, at least. But we still have to start from the beginning. Or else you won’t understand the nature of the problem I’ve identified.” She paused for a moment to make sure he was in agreement.

  Martinez gave her a thumbs-up. “Sure, go for it.”

  She took another deep breath. “Okay. Across all of its components, ARC operates using several of the most advanced deep neural networks that exist today, and they are tightly linked together. The outputs of one component within the ARC system feed into the next one as inputs. Each of these networks was designed to mimic the deep neural network of the human brain. To achieve the complexity of the brain’s architecture, a team of mathematicians, engineers, and computer scientists developed a series of complex algorithms that function as networked layers of artificial neurons, just like the billion neurons in your brain.”

  “Huh?” Martinez said, rubbing the back of his neck. “You totally lost me on the brain stuff.”

  Not again.

  Grace suppressed a sigh. “Okay, let me try a different tack. Think of each algorithm in the network as a separate mathematical formula analyzing one piece of a larger equation. Each algorithmic layer within the deep neural network performs a different calculation. As the input data travels through each layer, a mathematical formula interprets the input data to produce a probabilistic outcome using advanced statistics, with an extremely small error rate.”

  Martinez made an apologetic face. “Still not getting it. Sorry. Math wasn’t my best subject.”

  Grace put her hand on her forehead. “Maybe I should try an example so that you can visualize what I’m saying?”

  Martinez nodded eagerly.

  “You’ve heard of the cats-versus-dogs explanation for training a deep neural network?” Grace asked. “It’s pretty famous.”

  Martinez bobbed his head. “I think so. That’s for a deep neural network that specializes in visually identifying objects, right?”

  Grace nodded eagerly. “Yes, data scientists train a deep neural network to recognize objects by starting with something simple like images of cats and dogs. But we need to feed it a massive dataset of images of cats and dogs, each of which would have to be labeled accordingly. Remember the algorithmic layers I was talking about?”

  “Yep… then you said something about a mathematical formula and lost me.” He grinned.

  Grace smirked. “I’ll skip that part this time. Just assume that your deep neural network reads millions of images labeled with cat or dog and then develops a set of rules for relationships between the different features of dogs and cats to predict an outcome about the content of the image. Make sense?”

  “Still with you.”

  “Let’s say we want the algorithm to determine whether the image contains a dog. The first layer of the deep neural network might focus on a single characteristic such as the number of ears. The algorithm analyzes the input data, in this case an image of a dog, and predicts whether the image contains a cat or a dog based on that feature. Since both cats and dogs have two ears, that layer would not be definitive. Rather, it would produce a small percentage chance that the image specifically contains a dog. The next layer might address the shape of the ear. Cats generally have pointy ears that stick up, whereas dogs can have many different ear shapes. Since this is a more defining feature, the absence of pointy ears would increase the percentage chance that the image contains a dog. Does that make sense?”

  “Yeah, it’s starting to…” Martinez furrowed his brow. “In other words, the deep neural network doesn’t actually know what dogs and cats look like. It just makes probabilistic predictions about what the image contains. If the animal has two floppy ears, then it’s a dog at this probability. If pointy ears, then it’s a cat at that probability.”

  And he’s finally getting it…

  “Exactly. And—”

  Martinez rubbed his chin. “But all of this has me wondering what kind of dataset we used to train ARC. I mean, we’re not talking about the simple task of identifying dogs or cats. We’re talking about analyzing intelligence to predict emerging nuclear threats, detect nuclear attacks, decide on an effective nuclear war fighting strategy, and launch the order. Do we even have enough of the right data to train the algorithm to do all of that effectively?”

  Ding, ding, ding.

  “Well, that’s another longer conversation. The short answer is no. The ARC system relies primarily upon collected data from peacetime operations.”

  “But what about the Cuban Missile Crisis?” Martinez asked. “Could that data help ARC identify red flags for emerging threats?”

  Grace shook her head. “That’s not enough data for a few reasons. First, the conflict was with a single country during a very short time period—thirteen days, to be exact. Now remember, back then, we didn’t have the internet or social media. When you think about a tweet or post going viral in a matter of minutes today, news didn’t travel quickly back then. Newspapers and old-school news reporting were the primary source of information about world events. Second, we didn’t see that threat emerge until it was practically too late, and the nuclear-armed missiles were located on Cuba. Although we have more satellite coverage today than we did back then, it’s still possible to miss things happening on the ground. Third, the crisis remained a peacetime operation in that no shots were fired. We increased our alert status and used some intense signaling to communicate our intentions to the Soviet Union. You can probably guess why it’s dangerous to extrapolate from such a small and narrow dataset to predict what might happen after nuclear war begins.”

  Martinez nodded. “But that’s why we run these monthly tests, right?”

  “Yes, but the tests are somewhat limited in that they produce synthetic data about nuclear conflict. We can’t extract real-world wartime behavior from peacetime operations or from data based on a simulation. Centoreum Tech used the model of nuclear deterrence to keep ARC on track in the absence of sufficient data. Do you want me to further explain the data problem?”

  Martinez glanced at his watch. “No, we don’t have much time. Just give me the bottom-line up front. I want to understand the problem with the ARC system. We can discuss the rest during the pre-brief meeting with the chairman this morning.”

  Um, that sounds… like a terrible idea.

  Grace’s face blanched at the notion of trying to teach a four-star general and other high-level military officers about the topic of data bias.

  “Don’t worry… I’ll be there to help you thro
ugh it,” Martinez said. “As a leader, the chairman is extremely thoughtful. He will want to understand this.”

  Grace exhaled sharply, her pulse elevating rapidly. She knew Martinez was trying his best to reassure her, but every time he mentioned the chairman, she imagined herself telling him about cats and dogs while trying to keep a straight face. Her stomach flipped a few times.

  “Okay, the bottom line…” She paused for a moment, attempting to settle her nerves. “Centoreum Tech claims that ARC operates with an error rate of about 0.1 percent, making it the most accurate deep neural network in the world after NASA’s rocket software. Of course, any automated system that interacts with nuclear weapons should have the smallest margin of error possible. But the important point is this. There’s still a small margin for error.”

  “Okay, you’re saying that ARC can still make mistakes. So, has it made a mistake?” Martinez asked.

  Grace took a deep breath. “Well, I’m not sure. It depends on what you’d consider a mistake.”

  “What?” Martinez asked. “I thought you said there was a problem with ARC.”

  “I did. And there is. But it’s not what you think it is.” Grace grimaced.

  “Explain, please.”

  “Earlier this morning, we exposed the ARC’s deep neural network to a new round of ISR data in preparation for today’s test.”

  “Yup.”

  “Well, I checked the outcomes of the ISR data infusion around 4 a.m. this morning. ARC interpreted China’s forward deployment of submarines as an offensive move rather than a defensive one.”

  Martinez squished his eyebrows together.

  “And for this reason, the ARC system has recommended increasing the alert status of U.S. nuclear forces to DEFCON 3 and undertaking a countermove against Chinese submarines in the Northwest Passage.”

  Martinez’s eyes widened.

  “The ARC system could be producing a false positive. If it has, then the system is making recommendations based on a misinterpretation of the data.” Grace paused for a moment, waiting for the full implications of her finding to sink in.