AI and the Trolley Problem

by Pat Cadigan

  Begin Reading

  Table of Contents

  About the Author

  Copyright Page

  Thank you for buying this

  Tom Doherty Associates ebook.

  To receive special offers, bonus content,

  and info on new releases and other great reads,

  sign up for our newsletters.

  Or visit us online at

  us.macmillan.com/newslettersignup

  For email updates on the author, click here.

  The author and publisher have provided this e-book to you without Digital Rights Management software (DRM) applied so that you can enjoy reading it on your personal devices. This e-book is for your personal use only. You may not print or post this e-book, or make this e-book publicly available in any way. You may not copy, reproduce, or upload this e-book, other than to read it on one of your personal devices.

  Copyright infringement is against the law. If you believe the copy of this e-book you are reading infringes on the author’s copyright, please notify the publisher at: us.macmillanusa.com/piracy.

  The wind was blowing sharply from the east, across the north European plain from Siberia to the flatlands of East Anglia. Despite that, Helen Matthias was perspiring through her running suit by the time she finished her usual morning circuit of the Lakenwell Airbase perimeter. After two years, she was getting used to the winters here. They felt harsher than the ones she remembered as a kid in Massachusetts, and the snow usually came later, after the turn of the year. This morning she thought she could detect the faint scent of ice in the air. Was that a little hello from Siberia? Prasanna would have told her it was all in her head; if so, her imagination was especially strong today.

  Maybe she should ask the donkeys, Helen thought, waving to the one plodding toward her on the perimeter road. When she had passed it earlier, it had automatically moved to one side, putting itself between her and the electrified fence to reduce her risk of accidentally stumbling into it; safety first. The donkey was still keeping to one side; maybe Felipe Dos had told it to expect more traffic.

  Why people called them donkeys was a mystery to Helen; they looked more like a collection of welded-together toolboxes on four legs. There was no head; front and back were determined by their direction of travel. The roboticists claimed it was a matter of convenience. Helen told them robots that lacked the concept of backwards as humans understood it was one of those supposedly little things that could very well bite them in the ass later.

  The roboticists were skeptical but curious, and asked her to explain her thinking in detail, and in writing, thank you. What they really wanted, she knew, was a formal proof, but they’d settle for a well-reasoned hypothesis. Over the last several days, she’d been setting her thoughts down, and as often happened at Lakenwell, found she was having a hard time seeing the trees for the forest. Which was actually a jungle. As one of her philosophy professors had liked to say, Oh, what a tangled web we weave when first we practice to perceive. But perceiving was what they paid her to do.

  “Hey, T-1,” she said to the donkey as it drew nearer, making a sound that wasn’t quite like a horse’s clip-clop. It was painted in spiraling red and white stripes that widened in the center of its body and narrowed at either end. No front, no back. “How ya doin’?”

  “Can’t complain, but I always do,” it replied in a slight Texas twang. “Y’all stay safe now, and don’t pee on the fence.”

  Helen gave a surprised laugh. That was new, she thought, staring after it. Apparently someone had expanded the database of responses, not to mention accents. T-1 was short for Thing One. Thing Two was on the opposite side of the camp. There were two others—Hop-A-Long and Bob—all of them wired into Felipe Dos, who ran most functions on the base.

  According to the Lakenwell handbook, the donkeys were part of the security system. The bright colors made them easy to spot. They were armed with live ammunition, and they would shoot. Despite their clunky appearance, they could not be knocked over, and they could easily outrun a human over virtually any terrain. Any unauthorized attempt to access their software would cause them to self-destruct in a way the handbook described as “unpleasant and potentially life-changing to anybody nearby.”

  Helen suspected the donkeys had more to do with surveillance than protection, but when she shared this thought with Prasanna, her friend laughed. “The base has full-saturation surveillance, but people spend hardly any time around the donkeys,” Prasanna said. “What could they possibly pick up that the outdoor monitoring system wouldn’t already have?”

  Helen had been about to say the outdoor monitoring system wasn’t as comprehensive as the one indoors, so people tended to be less guarded, even around the donkeys. Then she thought better of it; they were, after all, indoors. “You’re right,” she said. “Maybe living under one-hundred-percent surveillance is making me paranoid.”

  “I stopped noticing it a lot faster than I thought I would,” Prasanna told her. “Maybe I just like getting so much attention, even if most of the time it’s from Felipe Dos.”

  Her heart rate had returned to normal after her run, but Helen stayed a few moments longer, breathing deeply and looking around. Lakenwell had been largely abandoned after the Cold War, and the British government had been happy to let the Americans set up a research lab with both civilian and military personnel. Now they all rattled around like too few peas in a too-large pod. After spending most of her adult life in urban environments by choice, Helen had been surprised at how easily she had adjusted to all the empty space and the isolated location. Maybe she’d simply needed the change.

  A strong chill swept over her, and she remembered what the base commander, Gillian Wong, had told her: If you want to freeze to death, start by getting sweaty. Wong knew what she was talking about—she’d been with a number of Special Forces units before assuming command of the base. Helen wondered how she felt about getting such a tame assignment. Maybe Wong had needed a change, too.

  Helen liked her, as did just about everyone on the civilian staff. She was good company and easy to talk to, always keeping things light and divulging little about herself. None of the civilians knew where she stood in terms of politics, religion, or sexual orientation. Helen supposed it was to do with her being the base commander. And she was always the base commander; she never seemed to be off duty, and no one had ever seen her out of uniform. None of them had seen any of the military personnel out of uniform. Prasanna joked that they probably all had special military pajamas, fatigue onesies. Helen thought if the heat in the military quarters was cranked up as high as everywhere else on the base, they probably slept in the raw.

  All the military staff were pretty nice, if a bit more standoffish than their CO. According to Ybanez in systems engineering, they’d been handpicked by Wong personally. Helen was intrigued; a veteran from Special Forces and her handpicked unit watching over a long-disused airbase full of assorted engineers, roboticists, and AI researchers—complete and utter nerds, herself included—somewhere off a less-traveled road in the British East Midlands. What kind of trouble were they expecting?

  After two years, she was pretty sure it was cabin fever. Everyone was confined to the base most of the time. The government provided plenty of compensation in the way of entertainment—an extensive library of books, movies, TV shows, and video games, not to mention full access to the web, not to mention a gym that would have made Helen’s old aerobics instructor weep with joy, although there were usually more soldiers in it than engineers. The onsite chef changed every four months—apparently food preparation was an industry that attracted people who never took a deal without parole.

  Helen’s own employment contract had another year to r
un; after that, they’d either ask her to stay or invite her to leave. Unless she screwed the pooch in a particularly egregious way, she was pretty certain it would be the former. Specialists in machine ethics were still very thin on the ground; not many jobs for them, either, and the few that were available tended to be a lot more technical than what she was doing here.

  She felt another, more intense chill and started toward the main residential building. Just before she reached the entrance, the door banged open and Cora Jordan bounded down the cement steps in mismatched sweats and a bright blue scarf the same shade as her bright blue hair.

  “Hey, how the Helen are ya?” she said loudly, running in place.

  Cora Jordan was a firm believer in overdoing every joke, especially if it were too boring and unfunny to actually be a joke. Helen made herself smile. “I’m good. You?”

  “Completely fit for anything, of course,” Cora replied heartily. Her eyes looked a little too shiny. If she’d been anyone else, Helen would have been sure she was pumped up on something. In Cora’s case, however, it was more likely she hadn’t taken anything, probably for days.

  “You want to join me for breakfast?” Helen asked her. “It’s so cold—”

  “Can’t eat till after, I’ll puke,” Cora said, still running in place. “How is it this morning, cold?”

  “Uh, yeah. How about a hot drink? Coffee, herbal tea—”

  “No, I’ll puke,” Cora said impatiently. “You see anybody else on the track?”

  “Not a soul. Unless you count Thing One.”

  “Oh, great! I love those stripes, they’re so trippy. You talk to ’im? What’d he say?”

  “I asked him how he was and he said he couldn’t complain but he always does,” Helen said, thinking that if she kept Cora engaged, she could distract her and get her back inside. “He also told me to stay safe and not pee on the fence.”

  Cora screamed with laughter, jumping up and down as if this were the funniest thing she’d ever heard in her life, while Helen wondered why the sound hadn’t made anyone rush outside to see who’d been hurt. “Seriously?! Oh my God, that’s priceless! Don’t pee on the fence, Jesus! You think he’d say that to me if I asked him?” Before Helen could answer, Cora galloped away across the scrubby dead grass, her scarf flying.

  Helen stared after her and sighed. “I’ve gotta report this,” she said aloud. “She could hurt herself. It’s not snitching. Friends don’t let friends drive drunk, friends don’t let friends run away from their meds.”

  Except she already had.

  * * *

  “I called Medical and left a message before I took a shower,” Helen told Prasanna in the cafeteria. They were sitting at Prasanna’s usual table by the windows, looking out at the windswept runways to the east.

  “Then you’ve done your duty,” Prasanna said, smiling. She was one of the few Brits on the team, a software engineer with dark brown skin and shiny black hair she wore in a single braid over her shoulder. Today she was dressed in a dark green pullover and black trousers; she always looked to Helen as if she were going somewhere special, even when she wasn’t wearing any makeup. “So why do you look like you did something wrong? It’s not that being a snitch thing, is it?”

  Helen shook her head. “I should’ve made her come inside with me instead of telling her the latest cute thing T-1 said.”

  “What did he say?” asked Prasanna. Helen told her and she laughed. “That is a good one. One of the guys must’ve come up with that—you have to warn guys not to do things like that. Mother Nature saved us from that kind of foolishness.”

  “Unless we’re off our meds,” Helen said.

  “Cora is an adult,” Prasanna said firmly. “She’s not legally incompetent even when she is off her meds. Which means not only are you not her keeper, you have no right to force her to do anything. You told Medical, it’s in their hands.”

  Helen shook her head again. “I should have done something more. I don’t know how I’m supposed to give a machine ethics when my own need some work.”

  “Jeez, give it a rest, will ya?” Prasanna said, and nodded at Helen’s chunky black watch. “How many calories did you burn on your run?”

  Helen tapped the screen, waited, then took the watch off and gave it a hard shake. “I guess that’s classified,” she said, showing her friend the message on the tiny screen.

  “‘Data unavailable’ again?” Prasanna made a tsk sound. “Third day in a row, isn’t it?”

  “Yeah. It’s worse than useless,” Helen said. “I might as well have a Magic 8-Ball strapped to my wrist. It’s almost that big. I keep banging it on things.”

  “Maybe that’s the problem.” Prasanna spooned up some grits. Her fondness for them was a recent development. Helen, who had studiously avoided consuming any herself during the few years she had lived in the Kansas City area, had never imagined she’d have to avoid them in the UK. “Maybe you broke it.”

  “Nah, it’s shockproof. You can drop it off the roof and then kick a field goal with it and it keeps on ticking. Or humming.”

  “Then maybe Felipe Dos thinks you’re too obsessed with calories.”

  “That’s not as far-fetched as you’d think,” Helen said, smiling, “if a bit more advanced than I would expect. Although seeing as how I’m competent with or without calories, I don’t think Felipe has any right to—” She cut off, staring open-mouthed at the unbelievable sight visible through the windows behind Prasanna.

  The other woman twisted around to look. “Oh my God. Tell me I’m not seeing that.”

  “No can do,” Helen said weakly. “Cora really is riding Thing One like a—a—”

  “Like a donkey,” Prasanna finished for her. She started to laugh, then quickly smothered it. “I’m sorry, that’s not funny, is it?”

  “Actually, it is,” Helen said. The two women got up and went to the window.

  Cora had tied her scarf around the donkey’s midsection and entwined her legs in it on either side to keep herself from falling off. The donkey made an awkward steed, giving Cora a bumpy ride as it headed for the main building over the scrubby, colorless grass. Cora slapped its would-be flank and hollered for it to run the other way. To Helen and Prasanna’s collective astonishment, it did—but without turning around, so that Cora was suddenly riding backwards.

  She yelled for it to stop and turn around, and it obeyed, making a full, three-hundred-sixty-degree turn.

  “Dammit, one-eighty!” Cora yelled. “One-eighty turn!”

  The donkey started to do as it was told, then turned back. Cora kept yelling orders at it and it would start to obey, then reverse itself. “It’s like it’s confused,” Prasanna said to Helen.

  “Felipe’s telling it to come in the way it’s supposed to when there’s a malfunction,” Helen said. “But for some reason, there’s a conflict because of Cora, and there shouldn’t be.”

  “Maybe she’s sitting on an alt-delete button,” Prasanna said, unable to keep from giggling.

  Helen shook her head. “Robots like this have been used in combat to carry weapons,” she said. “But this one’s a lot more sophisticated. It shouldn’t be doing that.”

  “Maybe it likes her?” Prasanna was holding her middle now.

  Cora had finally aimed the donkey away from the building. “Okay, let’s go! Head for the road! Giddyap! Mush! Andelay! Get the lead out!” The donkey suddenly took off at a gallop in the direction of the main gate, and by some miracle, Cora managed to hold on. Four soldiers in a golf cart came around the side of the building and gave chase.

  “Took them long enough,” Helen muttered.

  “The guards at the gate’ll stop her, won’t they?” Prasanna said, still laughing a little.

  “Don’t ask me,” Helen said. “I just work here.” The watch she was holding chimed loudly. The message on the screen said 666.

  Prasanna laughed some more. “You’re holding it upside down!”

  “Nothing would surprise me.” Helen tucke
d the watch in her pocket and headed for Gillian Wong’s office.

  * * *

  Helen was somewhat alarmed to find two guards outside the commander’s office, both armed not with the usual pistols but with automatic weapons.

  “Are we under attack?” Helen asked.

  “Not that we know of, ma’am,” said the ranking soldier politely.

  Helen’s jaw dropped. The last time Sergeant Kara Arendse had called her ma’am had been the day she’d arrived. Every couple of weeks they took turns beating each other at table tennis. Although, now that Helen was thinking of it, not lately; it had been at least a month since their last game, maybe longer.

  “What’s going on?” Helen asked tensely.

  “The commander will explain everything,” Arendse told her, her face expressionless. “Sergeant Martinez will escort you.”

  “Follow me, ma’am,” said Martinez. Helen hesitated; Arendse stood at attention, pointedly staring straight ahead, giving no sign that she even knew Helen was still there.

  “Please,” Martinez added. “This way.”

  Helen kept quiet as she followed him through the main residential building to a stairwell on the opposite side. Once the door closed behind them, however, she started bombarding him with questions.

  “Ma’am, I have no answers for you,” he said, talking over her as they started down the stairs toward the basement. “Only Commander Wong can tell you what you want to know.”

  “Okay, just tell me one thing. Just one.” Helen stopped and grabbed the metal railing with both hands. “I’m not going another step until you do.”

  Martinez looked up at her unhappily. “What is it?”

  “Am I in trouble? Are you taking me down to the brig?”

  The soldier’s features seemed to relax slightly. “The brig is a separate building. If you were ‘in trouble,’ you would be escorted there in restraints.”

  “Then where are we going?” Helen demanded.

  “You said just one question. That makes two.”