Three Laws Lethal

by David Walton

  She had barely thought of Penn. In fact, she had intentionally shielded herself from such memories, surrounding herself with new places and sights and smells. She hadn’t logged on to her Penn account in months. The autocar simulations she never wanted to see again, but they had been little more than a driving course with roads and traffic lights and simple rules anyway. Her true experiments had been the worlds in which her Mikes lived and died. As far as she knew, all of those simulations were still loaded on her cluster. In fact, they were probably still running.

  The last time she had worked on that software, she had built the evolutionary algorithm directly into the worlds, so that new Mikes would keep spawning and fighting for resources indefinitely. The months since then would have been like millennia to them—tens of thousands of generations would have spawned and fought and died.

  That night, she skipped her usual wandering and returned to her apartment. She lay back on her bed with her glasses and her game console and typed the password to access the cluster of virtual machines allocated to her on the university cloud. After logging in to the game, she was presented with a Realplanet avatar and a view of the sparkling lake and distant mountains of the Mikes’ world. Everything looked much as it had before.

  Then she pulled up an info panel to review the details of the world, and discovered the impossible. The original world she’d created covered about a dozen square miles and was limited to a hundred Mikes. It couldn’t be much bigger than that, because the more Mikes she added, and the larger the world, the more computing capacity it required, and her university account was limited to the cluster of virtual machines and associated memory she had available.

  The info panel, however, showed that the world had grown a thousandfold. Instead of a dozen square miles, it now covered the equivalent of the area of Maryland. Instead of a hundred Mikes, it now supported a population of ten million.

  Ten million. It wasn’t possible. The computing capacity required to manage a simulation of that size would far outstrip her university allotment. Besides, the world was a fixed size. It wasn’t designed to grow larger. Realplanet could handle larger simulations, of course, but only when given the instructions and server capacity to do so. How was this happening?

  There was only one explanation, crazy as it sounded. The Mikes had done it. Which meant they were reaching out of the confines of the Realplanet simulation. Somehow, they had evolved to use the tools inside their world to affect things outside of it. The environment to which they were adapting, after all, was not just the simulation but the whole computing system that hosted it. If some Realplanet bug allowed changes to the file system or runtime environment, and exploiting that bug gave them a survival advantage, then given enough time, the Mikes would find it.

  It was the power of evolution at work. Over time, any mechanism that improved their ability to produce offspring would survive into the next generation. She didn’t know how they were doing it yet, but she would find out. The real question was, if they could do this, what more could they learn to accomplish?

  CHAPTER 12

  Of the four of them, Tyler was the only one to attend commencement and receive his diploma. He attended the undergraduate ceremony, hoping to see Naomi, but she didn’t come. Brandon, true to his word, didn’t show up for their graduate school commencement either. Tyler felt the eyes of the other students on him as he waited for his turn at the front. They all knew what had happened.

  He felt like a fraud, wearing the mortarboard and receiving his diploma as if he had accomplished something. His grandmother took him for dinner afterwards. He didn’t feel like celebrating, and she understood that, but he tried to smile and eat for her sake. The steak tasted delicious, but it was all he could do not to spit it out. Even enjoying a meal seemed wrong. It should have been all four of them, celebrating together. It should have been Abby.

  Finally, his grandmother dropped him off in front of his apartment building and drove away. Some friends from the engineering school had invited him to go drinking, but he didn’t feel like joining them. He had never been much of a drinker anyway, and this would be a celebration binge. As lonely as the apartment would be, he preferred it to cheerful company.

  To his surprise, a woman waited for him outside his apartment. She was tall and blonde, in her thirties, and strikingly pretty in a hard-edged kind of way. She wore flowing black dress pants, a sleeveless black shirt, and low-cut dress boots. Her hair was short and professionally styled in a kind of uneven, curled bob, and she wore bright red lipstick. She leaned against his door with her arms crossed, all sharp angles and hard lines. She could have been posing for a photograph, except that she didn’t smile.

  He slowed to a stop, key card in hand.

  “Tyler Daniels?” she asked.

  “Yes.”

  She pushed off from the door with one boot and held out a hand. He shook it warily. Her hand was dry and slim, no rings. He got an unexpected scent from her, not perfume, something like wood varnish and old books.

  “Congratulations on your graduation, Mr. Daniels. I’d like to buy you a drink.”

  He thought, reporter, though she didn’t give him that vibe. Could she be a cop? FBI? She looked like female feds did on crime shows, though he didn’t see a gun or a badge.

  “Who are you?” he asked.

  She flashed him a smile that seemed more classy and professional than warm. “Lauren Karelis. I represent Andrea Copeland, the motorcyclist whose husband was killed by a Mercedes autocar back in March.”

  “You’re a lawyer?”

  She shrugged. “Guilty as charged.”

  “What do you want with me, Ms. Karelis?”

  “Please, call me Lauren. I want to hire you.”

  “I don’t need a job,” he said, though it was an outright lie. He hadn’t even applied anywhere, and he couldn’t just live in his parents’ basement and play video games. He was going to have to find work.

  “I’m hoping I can change your mind,” she said. “But it’s a long story. Could we do this over a drink instead of in the hall? I flew in from Seattle this afternoon, and I’m a bit tired.”

  “The local bars are going to be crowded tonight.”

  “I saw a café with a white dog on the sign on my way in— looked like a nice place. Do they serve drinks, or is it BYOB?”

  Tyler grinned. “Pretty sure they serve drinks. A bit pricey, though.”

  Lauren waved away his concern. “The expense account can handle it.”

  For the second time that year, Tyler found himself at the White Dog Café, treated to a meal by a woman who had a lot more money than he did. This time, they sat inside, at a small table against a wall filled with paintings of dogs. The dogs came in all different colors, not just white. It was an eclectic collection, in a variety of frames and painting styles.

  “I thought this case would have been over by now,” Tyler said. “It’s been two months since it was in the news.”

  Lauren shook her head. “Not even close. The wheels of justice turn slowly. Even the initial complaint filing takes weeks. Then there are pleadings with counterclaims and cross-claims, motions to dismiss and motions for judgment, scheduling conferences and attempts to settle out of court. Right now, we’re still in the discovery process, with another two months to go until the court date.”

  “Where do I fit in?”

  She leaned forward, arms resting on the table. “An important claim of our case is that the software driving the Mercedes-Benz chose to hit my client’s husband’s motorcycle on purpose,” she said. “We believe the car’s sensors would have detected his presence, but that the software determined his life to be less valuable than the occupants of the car, even though they were better protected in a heavy luxury vehicle. In short, we believe the car’s software intentionally killed Mr. Copeland.”

  “Of course it did,” Tyler said.

  Lauren leaned back, breathed a sigh of relief. “I’m glad to hear you say that,” she said. “I was afraid
you’d disagree.”

  “I don’t think there’s any question. For you or me, it would just be a reflex. There’s no time for us to do anything more than react. To a computer, though, it’s plenty of time. It’s as if we put the crash on slow motion, then got out of the car, looked around, made some careful measurements, thought about it awhile, and finally made a decision. There’s enough time to consider all the options and choose one.”

  “The problem we have is that Mercedes denies it,” Lauren said.

  “Seriously?”

  “They claim it was accidental. That there was only time to act to save Ms. Brighton and her children, and that Mr. Copeland’s presence in the next lane was unfortunate and unavoidable.”

  “Unavoidable, maybe,” Tyler said. “But not accidental. It’s possible their software can’t distinguish between a human obstacle and a non-human obstacle, and just chose the smaller one. Regardless, it made a choice.”

  “They further claim that their software is merely an aid to human drivers, just like cruise control, and that the human is always responsible to take control in emergencies. That’s in the small print you sign when you purchase one of their cars, of course. But it’s also completely at odds with how they market the car. They don’t want humans taking control, and they don’t encourage it. They tell people they’re safer in an accident if they let the car drive. And maybe they are. Annabelle Brighton and her girls walked away without a scratch. But that doesn’t mean you’re safer in the next lane over. In fact, their car just might choose to take you out.”

  “So, what, you want me as an expert witness?” Tyler asked. She made a wry face. “Not exactly.”

  Tyler wasn’t really surprised. He was a fresh graduate with no experience in autocar software beyond getting a friend killed. He wasn’t much of an expert in anything. “What then?”

  “You’re one of the few software engineers in the world with a deep understanding of autocar technology who is not employed by one of the major companies,” she said.

  He made a face. There was a reason he wasn’t employed in the industry, and he didn’t want to be reminded of it.

  “I know what happened to your friend,” she said. “I’m sorry.”

  “Thanks,” Tyler said. “But you were getting press for your software even before that unfortunate tragedy. You know this stuff. The guy I hired first is a forensic software analyst, used to intellectual property infringement cases. He isn’t getting anywhere on this. You, on the other hand, know self-driving cars. You’ve written software to run them, so you know how they work.” She leaned toward him and held his gaze with very blue eyes. “I need you to reverse-engineer the Mercedes software and prove that it intentionally murdered Hal Copeland.”

  The waiter brought their drinks. Tyler was glad, since it gave him a moment to collect his thoughts before answering. He didn’t think she would like what he had to say.

  “You should understand, I don’t actually agree with your cause,” he told her. She kept looking at him, waiting, so he went on. “Self-driving cars are the future. We need them, and we need them as quickly as they can be fielded. For every person killed by one, whether by choice or by software error, a dozen more are saved because the cars’ owners aren’t driving drunk or texting or falling asleep at the wheel. I don’t want to work against that advance. Self-driving cars, even bad ones, are a lot better than what we’ve got without them.”

  “I agree,” Lauren said.

  Tyler had been building up to a rant, but her quiet assent interrupted him. He threw up his hands. “How can you represent the plaintiff in this case, and still agree with what I just said?”

  “Because they should be as good as possible. They’re the future, as you said. There’s no stopping them, and I wouldn’t want to. But neither do I want big companies setting the rules in secret. This software can take lives. I don’t want the wealthy barreling down highways in giant SUVs without concern for others because their software, tuned to drive aggressively, is free from public scrutiny. There’s a huge transition going on here, and finding the right balance of freedom and protection is a process. It’s a process that involves the law and the courts and includes lawsuits like this one.”

  Tyler felt a little sheepish. “I can see that,” he said. “In fact, I wrote an article for an autocar magazine that argued that all the core, decision-making algorithms should be made available to the world as open source software. Not only would that make the basis of the cars’ choices public, but also it would help with hacking, since the software would get worldwide scrutiny, and any vulnerabilities would be found quickly.”

  Her red lips turned upwards in a subtle smile. “So you see why this case is important. It’s the first one of its kind, which means it’s establishing precedent. Future cases will look back to this one to influence how these decisions are made.”

  “But why reverse-engineer the software? Don’t they have to give it to you? It’s pretty crucial to the case.”

  “Mercedes claims it’s proprietary.”

  “But that’s crazy. They’ve got to show you. It’s like saying you’re not allowed to examine the murder weapon to see if it has blood on it.”

  “Preaching to the choir,” she said. “The problem is, a lot of the software that runs in your car is proprietary. There’s precedent for ruling that way, despite the same argument being made.”

  “Can’t the judge force them?”

  She sighed. “Well, and that’s where it gets complicated. I filed a motion, and the judge ruled that it was a discovery item, and they had to turn it over. They pled some nonsense about finding the exact version that had been installed in the car, and he gave them a week to prepare.”

  “Okay. But you should get it then, right?”

  “Already did. The problem is, they spent that week obfuscating.”

  “Oh no.” Tyler knew exactly what that meant. “Oh, yes. They dumped the code into one giant file, removed all the comments, and changed the variable names to random letters of the alphabet. It compiles and runs, but it’s completely unreadable.”

  “So when you say ‘reverse-engineer’ . . .”

  “I mean I want you to make sense of their obfuscated source code and establish the basis on which it chose to swerve into Mr. Copeland’s lane and take his life.”

  Tyler took a sip of his drink, barely tasting it. “You’ve got to be able to fight this,” he said. “Obfuscated code is, as you said, unreadable. They’re not acting in good faith.”

  “And I’m working that angle, too. It’s a clear Rule 26 violation.”

  “Uh, yeah, sure,” Tyler said. “Probably Rules 39 and 118.6c, too.”

  She laughed. “Sorry. Rule 26 of the Federal Rules of Civil Procedure is what prohibits lawyers from giving discovery responses that intentionally harass, cause unnecessary delay, or needlessly increase the cost of litigation. This pretty much does all three of those. I complained to the judge that it’s like translating documents into Arabic before turning them over. The information is there, yes, but they sure didn’t use them that way, and it’s going to be a lot more effort and cost on our part to understand them. I think we’ll win the argument. I hope we’ll win it. But it’ll take more motions and arguments and experts to make our point. And in the meantime, we need to do what we can.”

  “This whole thing is insane.”

  She grinned. “That’s lawyers for you. Seriously, though, the problem is that the people who understand the law don’t generally understand technology. They’re making important determinations about issues that require some amount of technical expertise to rule on competently, but the technology changes faster than anybody can keep up with it.”

  “So that’s what you want me to do. Wade through impossibly obfuscated code and try to make sense of it.”

  “Not exactly. I’ve got somebody working on that already. I want you to look at what he’s deciphered, see if it makes sense and what you can understand from it. Then, when we get the real cod
e—hopefully soon—you can dive in and tell us what you find.”

  Tyler looked up at the wall behind her, where two dozen dogs of various breeds looked down at him. He didn’t exactly have any job prospects lined up. This was temporary, but it would, at least, let him interact in the autocar world a little longer. There were plenty of software companies in Seattle, so he could look for a job at the same time, maybe start a new life in the Pacific Northwest. “I’ll do it,” he said.

  “One more thing.” She winced a little, as if delivering bad news. “You’ll have to sign an NDA.”

  “Meaning I agree not to disclose the source code to anybody else?”

  “Meaning you can’t benefit financially from what you see.” He nodded slowly. “I guess I can do that.”

  “Glad to hear it.”

  “Speaking of finances, how are you paying for all this?” Tyler asked. “Is Andrea Copeland that rich?”

  “Nope. But Mercedes-Benz most definitely is. Enough so that my firm considers the chance of significant profit worth the risk of litigation. Besides, we’re also getting a considerable investment in the case from the Carmichael Group.”

  Tyler frowned. “The Carmichael Group is anti-automation. They don’t think computers should be involved in any important decisions. If it were up to them, we’d carry on as we are, killing 1.3 million people every year.”

  She held up her hands in a gesture of surrender. “This is an expensive case. We’ll take any help we can get.”

  “Okay,” Tyler said. “I’m in. When do I start?”

  The move to Seattle required depressingly little effort. Most of the furniture in the apartment had been Brandon’s, and most of the electronics as well. Tyler’s collection of paperback science fiction books still gathered dust in his parents’ basement, and it didn’t seem practical to move them across the country. In the end, he had two suitcases of clothes and three boxes of other possessions, and that was it. He hired a company to sell everything he had left behind and climbed on a plane bound for Seattle–Tacoma International.