“No, it didn’t.” David shook his head. He raised his voice defensively. “Look, I didn’t think that you would fly off to Wisconsin based only on an email.”
“Hey, I had just heard that my father had a heart attack. I was freaking out. And it’s your damn fault I had to go through that.” Mike was yelling now, and David leaned away in his seat.
“OK, chill out guys. If the two of you are fighting, you are not going to be thinking straight.” Christine said, also yelling to get their attention. “Look, what happened when you got back Mike? Did you look into ELOPe?” She spoke calmly, placatingly.
“I tried, but I couldn’t get access to either the source code or the system logs,” Mike said, resigned. “I assumed David had locked down access so that no one would find what he had done. And I didn’t want to say anything to any of the rest of the team, because I didn’t want to raise suspicions. I was still trying to cover for you, David.” Mike glared at him.
“I’m sorry, Mike,” David said gently. “I shouldn’t have said that about you and your dad. I’m really sorry you flew halfway across the country, but I’m glad your father was fine.”
Mike hesitated a minute, then nodded slightly, accepting the apology. “Well, did you do anything to lock down the system?” he asked.
“No. In retrospect, it sounds like a great idea to have locked down access to ELOPe, but I didn’t do it.”
“Shit, then somehow ELOPe has removed my access to the servers and the code.”
“Ugh, guys, that seems impossible.” Christine said. “Even if you are right, Mike, and ELOPe is somehow originating emails on its own, it seems preposterous to think that ELOPe could social engineer you to leave town. And how is ELOPe going to get your access rights removed? If all it can do is send emails, you can’t send an email to revoke someone’s access. I assume you guys have some kind of internal web application that handles access control. I think you’ve somehow become paranoid about David’s deceit being discovered, and now your imagination is running away with you.”
“No, Christine,” Mike said. “I’ve thought about this for days now, and it is possible. Let’s say ELOPe didn’t want to be turned off. It knows that I can turn it off. Now it has to figure out how to ensure I won’t do it. If it analyzes enough emails, it could figure out that people don’t do work when they leave town. If it can figure that out, then it can also determine that people leave town for family medical emergencies. If it had enough emails about medical emergencies, it could figure out that messages about family emergencies usually come from family members. My own email history would show who my parents are, and their email addresses, and that I’ve flown to visit them before. If it put all those things together in one long chain of deductions, it could figure out to fabricate an email from my mother saying that my father is sick. I know it sounds farfetched, but this is all within the design parameters of ELOPe.”
“Are you saying that this thing is reasoning and thinking like a human being?” Christine asked, shaking her head. “Because no matter how smart you guys are, I’m having a hard time believing that some code you wrote is suddenly developing a mind of its own.”
“It’s not thinking,” David said. “ELOPe is just analyzing emails, figuring out what language will optimize the success of the primary goal I entered, which was to maximize success of the ELOPe project. It’s a straightforward process; goal, analysis, language optimization, in response to inputs. It can chain goals together. It is not independent thought, but it can have the appearance of independent thought.”
Mike raised his hand up. “Look, here’s an analogy I thought of while I was waiting for your plane. Imagine that you’ve got all the pieces of all the jigsaw puzzles in the world. Now imagine you have a computer that is patient enough to try every possible combination of every possible puzzle. Given enough time, it could make any arbitrary picture it wanted out of those pieces. And that’s what emails are to ELOPe—puzzle pieces. It looks at the millions of emails in its library of emails, figures out all the components of them, and then figures out new ways to piece them together.”
“ELOPe, the computer system that ran away with itself.” David laughed nervously. “Well, we got the name right.”
“So is it an artificial intelligence? Is it thinking for itself, or isn’t it?” Christine spoke softly, half to herself.
“I don’t know, hon,” David said. “I don’t see how it could be capable of free-form thinking, which is what most people would think of as an AI. But it is pretty sophisticated when it comes to goal analysis and synthesis. We couldn’t hardwire goals into ELOPe and have it meet the design objectives. We had to let it discover people’s goals. So we gave it the ability to contextually determine goals by parsing emails.”
“Then it tries to make sense of those goals in terms of other goals it understands,” Mike added. “We implemented two approaches to learning about new goals. First, it can see whether goals might be similar based on language analysis. For example, a ‘break from work’ is semantically similar to a ‘vacation’, and a simple dictionary lookup can figure that out. Second, it can guess where one goal might be an extension of another. If it thinks one goal might be an extension of another, it will predict what people’s responses will be, and then test to see if the predictions match actual historical responses. For example, if I simply said I wanted to have fun today, then ELOPe might be able to extrapolate that activities such as playing a game, miniature golf, or going to see a band are fun.”
David nodded. “So when I added code to create an overriding goal to maximize the success of the ELOPe program, it’s hard to know what it might consider. The more emails it analyzes, the broader the definition of ‘success’ it might have. Up until the last couple of weeks, it had never had such a large base of emails to analyze, nor such a large number of servers to do the analysis on.”
“From what you’re saying, the more emails it analyzes, not only do the possibilities for what constitutes success get broader, but the system would also discover more methods to accomplish those goals,” Christine said. “What it really sounds like you’ve built is an expert system for social engineering. You know what I mean by social engineering?”
Mike nodded his head yes, but David had a puzzled look on his face, and shook his head.
“Social engineering is the name given to techniques for tricking people into giving you information or making changes to information systems,” Christine said. “Social engineering was popularized by hackers in the nineteen eighties. And by hackers, I don’t mean the good guy hackers like Richard Stallman. I’m thinking of folks like the Kevins.”
Mike nodded his head again, but David looked even more puzzled, and turned around to look at his wife.
“Honey, how can you be married to me, and not know this stuff? You know I was a total online geek as a kid, yes?”
“What can I say?” David sighed. “Please go ahead.”
“Okay, look. The eighties and nineties were the heyday of hacking. Folks like Kevin Mitnick and Kevin Poulsen were able to get access to all kinds of computer systems, phone company records, credit card company records. I think it was Kevin Poulsen who said that it was easier to trick someone into giving you a password than to brute force hack it. The classic example would be someone who was trying to get access to a company’s internal phone system. She might call the front desk of the company, and tell them, ‘Hi, I’m your AT&T rep. I’m stuck on a pole down the street troubleshooting your system. I need you to punch a few buttons on your end.’”
“And?” David asked.
“And the buttons the hacker would ask the operator to press might be a key sequence that would forward all incoming calls to an outside line. Then the hacker could impersonate an employee of the company from their home phone, so they could do even more social engineering. The point is, simply by knowing the lingo, giving plausible reasons, knowing what motivates people, a hacker can gain information or get people to do things by cleverly manipula
ting the human tendency to trust other humans. Since you’ve built a system that learns lingo, language nuances, and motivations, and can evaluate what will be most effective to the receiver, it is, by definition, an expert system for social engineering.”
David looked flabbergasted by this explanation. “How do you know all this?”
“You know, books and stuff,” Christine said, with a sarcastic smile.
“This is pretty much what I concluded when I was with my parents,” Mike added. “We never explored how far the system could go on its own.” He paused and looking meaningful at David. “So, what do we do?”
“I think you and I have to go right to Gary Mitchell, and tell him the truth. Even if that means I lose my job as the result of this. We have to get Gary to approve an immediate outage with hard power down. Then we can pull ELOPe off the system, even if that means we rebuild those servers from the ground up. Forget home. Mike, please go straight to the site.”
“On my way,” Mike replied, as he sped down Alberta Avenue, bypassing the turnoff for David’s block, and heading for the highway to downtown.
* * *
“Gary Mitchell is still gone. His admin says he was supposed to take a vacation over the holiday break, but he should have been back by now. Tahiti, in case you were wondering.” David had just returned from the building across the street where Gary’s office was located, while Mike and Christine waited in David’s office.
“I’m picturing him laying on a beach, a cigar in one hand, and a whiskey in the other.” Mike shook his head.
“I know,” David said, laughing, “I don’t think his admin meant to tell me where he was, but I was a bit demanding.”
“No word from him?”
“Nothing. He should have been back in town a couple of days ago. His admin has paperwork waiting for him to sign. She’s sent him emails and left voicemails, but nothing.”
Mike grunted.
“While you were over there, I spoke to Richard, who was the only member of our development team that worked through the holiday break,” Mike explained. “Remember that email you told me about just after we received the additional servers? You mentioned that we were assigned a team of top notch performance optimization experts to help us work on performance improvements to drive down resource utilization?”
David nodded. “Yeah?”
“According to Richard, that team showed up on the Monday after Christmas,” Mike said. “They were a bunch of subcontractors he had never met before. They had two guys onsite for the entire Christmas break, and another dozen offsite. He says he had an email from you telling him to grant them access, so he did.”
“Let me see what changes they made.” David turned to his computer, attempting to pull down the latest source code. He stared in frustration as an error dialog popped up, with an ‘access denied’ message. He tried again, and then pounded the keyboard in frustration.
“I’m in the same boat you’re in,” he finally said to Mike. “My access to the source code has been removed. I can’t pull down a copy of the latest code to see what they changed. Do you have any ideas?”
“Well, Richard kept an eye on them the first few days, and then he left for a skiing trip over New Year. If you look in your email inbox, the contractors emailed us both a written report of what they did. It was sent Friday morning, so they finished up just before either of us got back to town. According to the email, they made some major improvements in performance, mostly related to the Bayesian network. Melanie was here at work yesterday, pulled down the latest code and ran the performance tests against it. The new code seems to have taken the import of new emails from x squared to x log x, and quartered the evaluation time.”
“Woah, are you guys saying you originally had an exponential resource utilization curve?” Christine thought about her own work on massively multiplayer games. She knew that, in the ideal case, when you add users to an internet application of any kind, you want the application to scale linearly. She shook her head and turned to look at David and Mike. “How the hell did you ever expect this to scale?”
“Scaling the resources has been the major bottleneck all along. It’s why we ran into so many resource constraints, and why the project was in danger. It didn’t appear that there was any way to scale without requiring a massive number of servers.” David shrugged. “I just kept hoping that as long as we kept the project alive, we’d find some way to overcome that limitation. Now it seems that someone has.”
David turned back to his computer. He was still trying to coerce his computer into giving him access to the code. “Damn, how did our project access get revoked? I don’t understand how email could interface with the access rights.” He turned back to Mike. “Do we have any idea what else these contractors did?”
“I might.”
Everyone turned around to look at the door. There stood an older unshaven man, dressed in rumpled clothes, carrying bundles of paper under his arms.
“Gene Keyes, Controls and Compliance.” He spoke in a deep rumble. “I’m here to save your ass.”
* * *
Over the course of the next hour, Gene briefed the others on what he had found during his investigation. Like them, he had tried to reach Gary Mitchell, with no success. He had uncovered that while there were unusual charges across the company, the only consistent patterns of unusual behavior were found in three departments. As Gene spoke, he laid printed reports across David’s desk.
The first, of course, was the R&D department in which David’s ELOPe project was housed. According to Gene’s print outs, it had paid for several small allotments of servers and subcontractors to make modifications to ELOPe. The nature of the modifications was not specified, but the budget amounts were sufficient for dozens of engineer-months of work from an outside vendor.
In fact, it was the expenditures in David’s department that had led Gene to them. David’s legitimate order several months earlier of a pool of high performance servers had clued Gene in that all of the later purchases might be somehow tied to the project that first needed additional servers.
“So does this mean we’re under suspicion?” Mike asked meekly.
“No, I can see the problem is bigger than you boys. In fact, I can see the problem is bigger than just people,” Gene said, the cryptic answer more puzzling.
“But where did the money come from?” David asked. “I exhausted our budget weeks ago.”
“Transferred in from other departments,” Gene answered. “Namely, Gary Mitchell’s Ops group.”
Gene went on to explain that the second department that contained the unusual pattern of purchases was Gary Mitchell’s department, which was responsible for operations for the company’s communication products, including AvoMail, Avogadro Voice, and Avogadro Chat. By virtue of the size of its business, Mitchell’s department had a vast operational budget compared to the relatively small R&D budget for the ELOPe project. Gene’s printed records showed that Gary’s department purchased tremendous quantities of servers, had servers reallocated from other projects, paid for a variety of subcontractors to do programming work, and finally transferred substantial funds to both the ELOPe project and the Offshore Data Center department.
“Offshore Data Center department? What do they do?” asked Christine.
“They took the data center in a box concept, which is a standard shipping container filled with racks of computers, and put it on a seaworthy barge,” David answered. “Then they connect the data-center-boxes to wave-action electrical generators. The whole thing is connected back to the data grid with fiber optic cables. Avogadro calls them ODCs for short.”
“Well, anyone care to guess the third department with the same unusual pattern of purchases?” Gene asked.
“The Offshore Data Center department?” Mike volunteered.
“Bingo,” Gene answered, pointing to Mike. “Not every purchase order has all of the details, but from the ones I’ve been able to track down, I’ve found that the data centers ha
ve been augmented with satellite communication capability, and line of sight microwave transmission, the kind used for traditional phone and data communications before fiber optics.”
“If ELOPe has somehow gotten onto an Offshore Data Center,” Mike mused, as he paced across the room, “we wouldn’t be able to kill its communication channel by simply cutting off the fiber optic line. We’d actually have to go out to the ODC and turn off the computers by hand.”
Gene laughed out loud, a harsh, barking noise that startled the others, sending Christine off her perch on David’s desk.
“What’s so funny?” David asked.
“Nobody is shutting those machines off,” Gene growled, his face now stern.
“Why not?”
“Because you didn’t let me finish. According to the purchase orders, the offshore data centers are armed. They have autonomous robots for self-defense,” Gene said, slowing down and exaggerating each word. “Apparently to protect them from pirates. I heard what you boys were discussing when I came in, and I came to the same conclusion myself: there’s an artificial intelligence in the computer making these purchases. And now the AI has armed itself. So there’s no way we’re going to just walk on board the ODC and turn off the computers. We’re going to have to blow them all up.”
David slumped down in his seat. “Holy shit, holy shit, holy shit. How did we get ourselves into this mess?”
“You kids trusted the computer with everything,” Gene said, grumbling, “and worse yet, you put no controls in place. No leash, no way to shut it down.”
“Wait, I don’t understand,” Mike said, pacing by the window again. “How did you conclude it was a computer program making the purchases, and not a person?” He turned and stared hard at Gene.
“One benefit of being in the audit department — I can access anyone’s emails. And there’s some mighty funny emails going out.” Gene paused, and pulled out another sheaf of papers, this one almost an inch thick. He took a few pages off the top and put them on the desk.
Avogadro Corp. s-1 Page 10
