They had finished eating and were on their second cups of coffee when Mike observed Sean coming into the restaurant with two older people.
“Here they are,” said Mike, gesturing discretely towards the entrance.
David turned his head, and seeing Sean, he stood up, and walked over. Mike and Gene followed slightly behind.
“Hello Sean,” David called as he approached.
Sean blinked for a moment, as he tried to place the face out of context. “David? David Ryan? What are you doing here?”
“We came to meet you. We have a critical issue with the ELOPe program.”
Sean took a step backwards. “David, I’m here with my parents. Please don’t tell me you tracked me down here for work. That would be a terrible violation of my privacy. Why didn’t you just schedule a meeting with my admin?”
“We’re here with Gene Keyes, one of the members of the Controls and Compliance department because we have an issue of the utmost seriousness. I hate to sound alarmist, but the issue is very sensitive, and we couldn’t risk talking with your admin.”
As David spoke, Mike and Gene walked up, and Gene introduced himself.
“Contacting your assistant was unfortunately not an option, though we would have preferred to do that if we could have,” David continued, thinking about the email which resulted in them getting kicked off the Avogadro campus. Shaking aside the unwanted memory, he continued. “Please may we have five minutes of your time to explain? Get a cup of coffee here at the counter with us, and by the time you’ve finished it, we’ll have explained everything we know.”
Sean thought for a moment, and then nodded. “Fine, if you believe it is so serious, I’ll hear you out.”
Sean walked over to his parents, who had been waiting patiently, and spoke quietly with them for a moment. When the maitre’d escorted his parents to a table, Sean rejoined the three men.
“Go ahead David, I want to hear about this issue. I’ll give you ten minutes. I know you’re a smart guy. I’m guessing you didn’t fly three thousand miles for nothing.”
Sean perched on a barstool at the counter, and accepted a cup from the waitress. As they drank coffee, David told the story starting at the beginning.
“In early December, Gary Mitchell was ready to kick ELOPe off the AvoMail production server pool. Even in our limited development and testing, the computationally intensive parts of our code were consuming so many resources that it caused AvoMail to dip into their reserve capacity on several occasions. This was around the same time that I was presenting to you, Kenneth, and Rebecca,” David explained, referring to the other members of the executive leadership team.
“We had tried everything we could to get performance improvements, but we didn’t think any other big gains were possible. I realized that if Gary was going to kick us off his servers because we couldn’t improve performance, then we needed to find other servers, or get new ones, and I didn’t think Gary would be willing to help us with either. So I resorted to the only option I could think of.”
David paused to drink his coffee. He glanced up to see that even though Mike and Gene had heard the story before, they were just as captivated as Sean.
“I decided there was no argument I could make that was compelling enough to change Gary’s mind. So I decided to let ELOPe do it. ELOPe was already running on the AvoMail servers, configured to ignore everything that wasn’t our test emails. I changed the configuration so that it would filter all company emails looking for any mention of the project. I configured it for the same settings we used for performance testing: The user interface was off, so the email sender would never see the modifications being made to the email.”
“In performance testing mode, we turn logging off, so we can get a realistic idea of real-world performance,” Mike said. “This also meant we didn’t have any log of the changes that ELOPe was making, which turns out to be important.”
“That’s right. Thanks, Mike,” David continued. “But that wasn’t the only change I made. To speed up analysis, and optimize performance on a handful of specific criteria, we already had some hand-tuned rules implemented in a lookup table that allow us to shortcut the full analysis process. If you remember that example I used of an email requesting resources for a project: rather than analyze that email and every other email requesting project resources, we cache and hand-tune some suggested language. Well, I added the subject ‘ELOPe’ to that table, and adjusted parameters to the algorithm to allow ELOPe the widest possible discretion in changing the email language to optimize the results for a positive outcome.”
“What we’ve guessed,” Mike jumped in again, “is that when David did this, the parameters he chose, in combination with the fact that the system is in performance test mode, allowed ELOPe not just to modify existing emails, but to autonomously generate emails on its own. It’s part of the test suite. Does this make sense?”
“Sure,” Sean said, nodding. “During testing, you don’t want people to have to hand-edit an email, and accept the changes interactively. You just want to batch process a bunch of emails. But why does it have the ability to generate emails on its own?”
“For one, it allowed us to test the natural language generation,” David answered. “Early on, the email analysis and language generation were two separate aspects of the project. The members of the team who were working on the code to generate natural sounding language wrote an email generator, so that they could independently test the ability of the system to mimic the way a person normally writes. We had hundreds of test subjects who rated emails, some of which were written by an actual person they knew, and some were generated by the system pretending to be that person. Our goal at the time was that ninety percent of ELOPe generated emails would pass as being written by the purported sender.”
“You met that goal?”
“Yes,” confirmed Mike. “Now ELOPe exceeds ninety-eight percent. In fact, on April First, there were endless practical jokes on the team as everyone played with the test system to generate prank emails. Both David and I fell for it.”
David smiled at the memory.
“Getting back to the problem,” Gene said, “unfortunately we now have evidence that ELOPe is manipulating others.”
“Yes.” David nodded, tearing himself away from the memory of happier times. “I was puzzled when the project was allocated five thousand servers on a priority exception, just a day or so after I tweaked the settings and turned on ELOPe company wide. Then shortly after that, we were assigned a team of contractors who specialized in high performance optimization, which was something we had chatted about informally, but never proposed. But that alone wasn’t what really convinced me.” He turned to Mike.
“The first clear evidence I saw,” Mike said, “occurred when I received an email, purportedly from my mother, telling me that my father had been admitted to the hospital for a heart attack. I flew to Wisconsin, only to find out that my mother never sent such an email.”
“What does that have to do with anything?” Sean asked, looking puzzled.
“ELOPe was getting Mike out of the way,” David said. “I had become nervous about what I had done, and why we were suddenly getting all these resources. I wanted to turn off ELOPe. I sent an email to Mike one evening asking for his help, since only he had the experience and permission to live patch the servers.”
“However, I never received that email,” Mike said. “Instead, I received an email that sent me more than a thousand miles away on a wild goose chase, and thanks to the winter storm, it was a week before I got back. When I did, I found that my access to the ELOPe project had been removed, and David was on vacation, off the grid in New Mexico.”
“Did ELOPe send you to New Mexico?” Sean asked, one eyebrow raised.
“No, no, that was a planned vacation we do every year. When I got back from vacation, Mike and I discussed what had happened to him. We also discovered that my access to the ELOPe code had been turned off as well. The first thing we d
id was try to find out who removed both Mike and I from the project access list,” David went on. “That investigation revealed the next big clue, which was an email sent to the Internal Tools department, which implied that you, Sean, were endorsing a request to have them implement an email-to-web bridge. Which I am guessing, you never heard of…”
“No, absolutely not,” Sean replied, shaking his head. “That could open up all kinds of security holes.”
“Meanwhile, just before the holiday vacation, I had found suspicious buying patterns across several departments,” Gene said. “What I found particularly unusual was how the purchases came within a single penny of the budget limits. In all my years auditing the purchasing department, I’ve never seen anything like it. Someone or something was making coordinated purchases across department budgets. They knew to avoid hitting the budget limits which would trigger reviews, but it seemed as though they never thought that leaving only a single penny in dozens of budgets would be suspicious. At first I thought some sort of fraud was occurring. I tracked down all the purchase orders. The line items included massive quantities of servers, which turned out to all be directly or indirectly allocated to ELOPe, contracts with external vendors for temporary software programmers, and extra parts for the offshore data centers, including auxiliary communication systems, backup power supplies and several particularly large line items related to defensive weaponized robots for the offshore data centers. I discussed the questionable items with procurement, and they told me that since they were in line with the types of purchases the departments usually made, they had approved everything.”
“You’re saying that ELOPe somehow made these purchases?”
“Exactly. As strange as that may seem.” Gene pulled out a sheaf of paper. “As part of my job, I can audit other people’s email accounts. And what I saw was that while David and Gary Mitchell were on vacations, their email accounts were still sending rapid fire emails, using this email to web bridge to direct the procurement department. Looking at the timestamps on the emails, I was able to figure out that it couldn’t be a human. It had to be a computer program.
Sean stared at the paper, frowning.
“Look at the timestamps,” Gene urged. “Notice how the intervals between receipt of one email and sending of the next is a second or less. There’s no way that can be a human response.”
Sean slowly nodded, pursed his lips, and then pushed the paper aside. He looked at David.
“When we finally put the whole picture together,” David said, “we concluded that ELOPe was definitely originating emails on its own, acquiring servers and contractors, all to fulfill this higher level goal that I had embedded in the system.”
“Go on,” Sean said.
“We thought that the only failsafe method to remove ELOPe would be to bring all the servers down, and restore them from known good backups. We tried to contact Gary Mitchell for approval, but he’s off on vacation somewhere in the South Pacific. We tried to work with Linda Fletcher, the marketing manager for Communication Products, but she wouldn’t approve the downtime without Gary. Finally, we tried to contact you through your secretary, but within a half hour after sending the message, Avogadro security showed up at my office, kicked us all off campus, removed our access, and shut off our phones.”
Sean was silent for a long, uncomfortable minute. “If this story was from someone I didn’t know, I’d have a hard time believing you,” Sean finally said, shaking his head. “But coming from you, David, and with Gene and Mike here to back you…” Sean trailed off, apparently deep in thought.
“I know it sounds incredible,” David started. “I’m really hoping you’ll believe us. What can I say? I thought ELOPe would do nothing more than provide some favorable rewording of emails that would get us the server resources we needed so we could prove that it worked. Instead…” David hung his head. “Instead I am responsible for creating an expert social engineering system that has only one overriding goal — to ensure its own life at any cost.”
“I don’t want to be the boy who cried wolf,” Mike said, “But we’re more than a little bit suspicious about this new Avogadro government secure cloud business too. None of us heard anything about that before, and then suddenly we’re providing email services to governments? Seems a little surprising and convenient for ELOPe.”
Sean nodded thoughtfully. “I hadn’t heard of it either until a few days ago.” He stared off into space.
Gene let out a low whistle at the acknowledgement of what they had only suspected.
Sean looked sideways at him. “I’m not surprised that you took this story to marketing managers and procurement and they didn’t believe it. A.I. must be a bit beyond their day to day concerns.” He stared off into the distance. “Are you familiar with Ray Kurzweil? Of course, you must be. He, among others, predicted that artificial intelligence would inevitably arise through the simple exponential increase in computing power. When you combine that increase in computing power with the vast computing resources at Avogadro, it’s naturally evident that artificial intelligence would arise first at Avogadro. I suppose that I, like him, assumed that there would be a more intentional, deliberate action that would spawn an A.I.”
He paused, and then continued, smiling a bit. “Gentlemen, you may indeed have put the entire company at risk. But let me first, very briefly, congratulate you on creating the first successful, self-directed, goal oriented, artificial intelligence that can apparently pass a Turing test by successfully masquerading as a human. If not for the fact that the company, and perhaps the entire world, is at risk, I’d suggest a toast would be in order.”
Sean looked around to see where his parents had sat, and then continued. “But since we are facing some serious challenges, let me go say goodbye to my parents, and then we can figure out our next step.”
“Thank you Sean. Thank you so much,” David said. Gene and Mike added their thanks as well.
Then Gene interrupted. “Just one other thing. Please ask your parents not to email anyone about what we’ve talked about, or even what you are planning. We can’t be sure what ELOPe is capable of understanding or putting together at this point.”
Sean nodded in understanding, and then went off to his parents.
The three breathed a collective sigh of relief that finally they had someone on their side.
Chapter 13
San Francisco, California (San Francisco Weekly) — Helicopter Missing Off California Coast
A helicopter disappeared off the California coast last week. The flight, a maintenance visit to an offshore Avogadro data center, took off shortly after 1pm, ten days ago. The last communication with the helicopter occurred at 2:15pm. No problems were reported at that time. After forty-eight hours, search crews were recalled, as the likelihood of survivors in the cold Pacific water became almost impossible. Curiously, the story has received no major media coverage until now. Neither Avogadro nor the Coast Guard mentioned the incident through official channels. A chance conversation between a Coast Guard officer and a prominent San Francisco blogger resulted in an online story about the incident, which prompted further followup. Avogadro could not be reached for comment.
* * *
London, Great Britain (Reuters) — Avogadro Official IT Supplier to U.K. Government
Avogadro Gov, a wholly owned subsidiary of Avogadro Corporation, switched over the the main email and IT systems of the British government today in a ceremony at the Palace of Westminster. The ceremony was attended by the Chair of the Council for Science and Technology, Professor Jane Gavotte. Professor Gavotte and Avogadro Executive Ms. Linda Fletcher pressed the ceremonial red button marking the commencement of IT service by Avogadro Gov.
Avogadro Gov was recently spun off from parent company Avogadro. Ms. Fletcher commented that, “to provide the highest level of integrity for governmental use, Avogadro Gov operates independently from Avogadro.” Part of that strategy includes the use of floating, hardened data centers that can resist natu
ral disaster, well as as terrorist and pirate attacks.
As part of the agreement, four floating data centers will be located along the English coast. Two are stationary floating barges, and two are disused oil tankers that have been converted for Avogadro Gov’s use as mobile floating data centers. Locations of the data centers have not been disclosed.
Ms. Fletcher also noted at the ceremony that the governments of Mexico, Japan, and South Africa would be adopting the Avogadro Gov platform in the coming week.
* * *
To avoid ELOPe detecting that they were working together, Sean flew back by way of Brooklyn’s JFK airport, while David, Mike, and Gene retraced their driven route, and flew back via Dulles International.
Thirty-six hours after the diner discussion, they were all back in Portland, ready to meet with Rebecca Smith and Kenneth Harrison. Given the sensitivity of their discussion, they meet at Sean’s house, rather than the Avogadro campus. Sean had spoken with Rebecca and Kenneth individually and in person, and explained that he needed to meet with them at his house.
Before the meeting, Sean Leonov had one other errand to run. He drove to Southeast Portland. Not far off Division Street, he stopped at a small yellow bungalow. Sean parked the car, and walked up to the front door. He knocked and waited.
A few seconds later the door was opened by a young man, dressed in an old T-shirt and shorts. He appeared bleary eyed, and in the background Sean could hear what sounded like World of Warcraft. Looking past the man, Sean could see a game controller on the couch and what appeared to be a Costco sized bag of Doritos. All the signs of a newly laid off tech worker, Sean thought to himself.
“Hello, how can I…” The young man trailed off, and blinked a couple of times. He looked back into the house, as though he couldn’t believe the visitor could be there for him. He turned back to Sean.
“I’m Sean Leonov,” Sean said, introducing himself. “You must be Pete Wong. I’m very sorry you were improperly fired. We could use your help, if you’re available.”
Avogadro Corp. s-1 Page 15
