by Isaac Asimov
“As you see fit, sir. If you think it a wise idea to go immediately.”
Damned busybody nursemaid of a robot. Kresh indulged himself with a brief flash of temper. If he waited around until the weather was just right, would Oberon then hint that he should wait until he had had a full meal and a nice long nap before setting out on the arduous thirty-second journey across the parking lot? They were on the clock here, and he had already been worrying that he had wasted too much time.
“I think it’s a wise idea, all right,” Kresh growled. “In fact I find it downright brilliant.” He undid his seat restraint, got up, and grabbed his rain poncho from the seat opposite, where he had tossed it down after coming aboard. The thing was still a trifle damp, but no matter. He pulled it on over himself, adjusted the hood, and glared at Oberon. “I’d suggest you stay here for the time being,” he said, “unless you think it a wise idea to get in my way.”
Plainly, Oberon did not think it a wise idea to reply to that. Kresh turned his back on the robot, grabbed the hatch handle, and yanked up on it. The hatch unlatched, and Kresh gave it a good hard shove. It swung open and he stepped out into the roaring weather.
The driving rain caught him full in the face, coming down cold and hard. Kresh held up his hand to shield his face, and squinted through the downpour. He walked around to the opposite side of the ship, and then straight ahead, toward the entrance to the Terraforming Center. The wind grabbed at his poncho, blowing it flat against his body and sending its hem flapping and slapping wildly behind him. He leaned into the wind, struggling to hold the poncho hood on top of his head as the wind did its best to pull it off, and the rain blew in regardless.
A pair of big double glass doors, the sort that opened at the center, formed the main entrance of the Terraforming Center. Kresh got to them and almost grabbed at the handles before he realized that wouldn’t work. He wasn’t going to get in unless he followed the rules—rules he had approved himself. “VOICEPRINT!” he shouted above the noise of the storm.
“Auto-voiceprint system ready,” an utterly depersonalized voice replied from nowhere in particular. Even though Kresh had been expecting a reply, it still startled him. The voice was clearly artificial—calm, emotionless, bloodless.
Kresh answered back in a somewhat lower tone of voice. If he could hear the voiceprint, probably it could hear him. “Name—Governor Alvar Kresh,” he said. “Password—Terra Grande.”
“Identity confirmed, clearance to enter confirmed,” the voice replied. The doors unlatched. Kresh, impatient and eager to get out of the rain, grabbed the handles of both doors and pulled them a bit too hard. The wind caught at the left side door and yanked it out of his hand, bouncing it against the left-side wall before it swung back. There was a second, inner pair of doors that swung inward, and Kresh shoved them out of his way without breaking stride.
He had not been here in a long time, but he still knew his way around. He turned left and marched down the main hallway toward the third set of doors. The first two doorways in the hallway were perfectly ordinary affairs, but not the entrance to Room 103. It was a huge, armored steel hatch that more closely resembled the doors of a vault than anything else. The door was locked down and secured, as it should have been, but there was a palmprint button by the side of the door. Kresh slapped his hand down on it. After a moment, there was a bump, a clunk, and a thud and the massive door swung outward.
Kresh ducked inside the moment the door was open wide enough to do so. A startled-looking middle-aged woman in a lab coat was working at a desk just inside the door. She stared open-mouthed at the intruder, then got to her feet. She seemed about to protest, and two or three of the robots took a step or two closer, as if they feared that the intruder might intend harm to the woman. But then Kresh threw back the hood of his poncho. It was clear that the woman and the robots recognized him instantly—but knowing who he was only seemed to increase their sense of bewilderment.
But Alvar Kresh was not much interested in the emotional state of the swing-shift technical staff. He barely looked at them. He looked around until he spotted two huge and gleaming hemispherical enclosures, each about five meters across, each sitting on a plinth or thick pillar, about the diameter of the hemisphere on top of it. The pillars raised the bases of the hemispheres up to just about eye level. One of the hemispheres was a smooth and perfectly rounded dome, the other a geodesic form, made up of flat panels, with all manner of complicated devices and cables and conduits hanging off it at every angle. Kresh nodded at the two machines, and spoke.
“I want to talk to the twins,” he said.
10
DR. LESCHAR SOGGDON opened her mouth and shut it, then opened it again and left it that way for a moment before she found her voice. “You’re—you’re Governor Kresh,” she said at last.
“Yes,” her visitor replied testily. “I know I am. And I need to talk to the twins concerning some climate projections. Now.”
Soggdon was now at even more of a loss. “Sir, it doesn’t work that way. You can’t just come in and—”
“I can,” Kresh said. “I should know. I wrote the regulations.”
“Oh, yes, yes, sir, of course. I wasn’t suggesting that you were not allowed to come here. It is merely a question of having the training and the understanding of our procedures here. It would probably be wiser for you to submit your questions in writing to the General Terraforming Committee and then—”
“Who are you?” Kresh asked, interrupting her. “What is your position here?”
Soggdon flushed and drew herself up to her full height, bringing her eyes roughly level With the base of Kresh’s neck. “I am Dr. Leschar Soggdon,” she said with as much dignity as she could muster. “I’m the night shift supervisor here.”
“Very well, Dr. Soggdon. Please listen carefully. I have come here precisely for the reason that I want—I need—to avoid that sort of delay and caution. I am here on a matter of the greatest urgency and importance, and I must be certain I am getting my information direct from the source. I cannot take the chance of some expert misinterpreting my questions or the answers from the twins. I cannot wait for the General Committee to have a conference and debate the merits and the meaning of my questions. I have to ask my questions now, and get an answer now. Is that clear? Because if it is not, you’re fired.”
“I ah—ah—ah, sir, I ah—”
“Yes? Do you have other job prospects?”
She swallowed hard and started again. “Very well,” Soggdon said at last. “But, sir, with all due respect, I would ask that you sign a statement that you proceeded against my advice and specifically ordered me to cooperate.”
“I’ll sign whatever you like,” Kresh said. “But right now let me talk to the twins.” The governor peeled off his poncho and handed it to the nearest robot. He walked to the far side of the huge room, where the two massive hemispherical enclosures sat. Inside were the two Terraforming Control Centers, one a Spacer-made sessile robotic unit, and the other a Settler-made computational system.
A sort of combination desk and communications console sat facing the two machines. Governor Kresh pulled out the chair and sat down at it. “All right, then,” he said. “What do I do?”
Soggdon was severely tempted simply to show the man the proper controls to operate and let him charge ahead as directly as he liked. But she knew just how much damage even a minor slip of the tongue could produce. The idea of having Unit Dee caught in a major First Law conflict just because Kresh wanted to have his own way was too much for her. She had to speak up. “Sir,” she said, “I’m sorry, but you have to understand a few things before you start, and I’m going to make sure you understand them, even if it means I lose my job. Otherwise you could cause any amount of damage to Unit Dee.”
Kresh looked up at her in annoyed surprise, but then his expression softened, just a bit. “All right,” he said. “I always tell myself that I prefer it when people stand up to me. I guess this is my big chance to prove
it. Tell me what I should know, but don’t take too long about it. You can start by telling me what ‘D’ means.”
His question took her by surprise. Soggdon looked at him carefully before she spoke. How could a man who didn’t even know what—or who—Unit Dee was expect to barge in here and take over? “I didn’t mean the letter ‘D,’ sir. I meant Unit Dee. That’s what we call the robotic terraforming control unit. Unit Dee.”
Kresh frowned and looked over at the two units, and seemed to notice for the first time the two neatly lettered signs, one attached to each of the two hemispheres. The sign on the front of the rounded-off dome read Unit Dee, and the one on the angular geodesic dome read Unit Dum.
“Ah. I see,” he said. “I confess I don’t know much about how you run things here. I visited here once or twice during construction, but not since you’ve been operational. I know the code name for the two Control Units is still ‘the twins’—but not much else. I suppose those names stand for something. Acronyms?”
Soggdon frowned. For someone determined to charge in here and take over, he certainly was ready to get distracted by side issues. “I believe the name Unit Dee referred to the fourth and final design considered. From there it seemed to develop into a sort of private joke among the day shift staff,” she said. “I must confess I never bothered to find out what the joke was. It might have something to do with Unit Dum being, well, dumb, nonsentient, but I’ve never understood the exact significance of Unit Dee.” Soggdon shrugged. She had never been much known for her sense of humor.
“All right,” said the governor. “All that to one side,” he went on, “what do I need to know to avoid producing damage to the twins?”
“Well, Unit Dee is the only one likely to suffer damage. Unit Dum is a non sentient computation device, not a robot. He has a pseudo-self-aware interface that allows him to converse, to a limited extent, but he’s not a robot and he’s not subject to the Three Laws. Unit Dee is a different story. She’s really not much more than an enormous positronic brain hooked up to a large number of interface links. A robot brain without a conventional robot body—but she is, for all intents and purposes, a Three-Law robot. Just one that can’t move.”
“So what is the difficulty?” Kresh demanded, clearly on the verge of losing his patience again.
“That should be obvious,” Soggdon replied, realizing just a second too late how rude a thing that was to say. “That is—well, my apologies, sir, but please consider that Unit Dee is charged with remaking an entire planet, a planet that is home to millions of human beings. She was designed to be capable of processing truly huge amounts of information, and to make extremely long-range predictions, and to work at both the largest scale and the smallest level of detail.”
“What of it?”
“Well, obviously, in the task of remaking a planet, there are going to be accidents. There are going to be people displaced from their homes, people who suffer in floods and droughts and storms deliberately produced by the actions and orders of these two control systems. They will, inevitably, cause some harm to some humans somewhere.”
“I thought that the system had been built to endure that sort of First Law conflict. I’ve read about systems that dealt with large projects and were programmed to consider benefit or harm to humanity as a whole, rather than to individuals.”
Soggdon shook her head. “That only works in very limited or specialized cases—and I’ve never heard of it working permanently. Sooner or later, robotic thinking machines programmed to think that way can’t do it anymore. They burn out or fail in any of a hundred ways—and the cases you’re talking about are robots who were expected to deal with very distant, abstract sorts of situations. Unit Dee has to worry about an endless series of day-by-day decisions affecting millions of individual people—some of whom she is dealing with directly, talking to them, sending and receiving messages and data. She can’t think that way. She can’t avoid thinking about people as individuals.”
“So what is the solution?” Kresh asked.
Soggdon took a deep breath and then went on, very quickly, as if she wanted to get it over with as soon as possible. She raised her hand and made a broad, sweeping gesture. “Unit Dee thinks this is all a simulation,” she said.
“What?” Kresh said.
“She thinks that the entire terraforming project, in fact the whole planet of Inferno, is nothing more than a very complex and sophisticated simulation set up to learn more in preparation for a real terraforming project some time in the future.”
“But that’s absurd!” Kresh objected. “No one could believe that.”
“Well, fortunately for us all, it would seem that Unit Dee can.”
“But there’s so much evidence to the contrary! The world is too detailed to be a simulation!”
“We limit what she can see, and know, very carefully,” Soggdon replied. “Remember, we control all of her inputs. She only receives the information we give her. In fact, sometimes we deliberately introduce spurious errors, or send her images and information that don’t quite make sense. Then we correct the ‘mistakes’ and move on. It makes things seem less real—and also establishes the idea that things can go wrong. That way when we do make mistakes in calculations, or discover that we’ve overlooked a variable, or have just plain let her see something she shouldn’t have, we can correct it without her getting suspicious. She thinks Inferno is a made-up place, invented for her benefit. So far as she knows, she is actually in a laboratory on Baleyworld. She thinks the project is an attempt to learn how to interact with Settler hardware for future terraforming projects.” Soggdon hesitated for a moment, and then decided she might as well give him the worst of the bad news all at once. “In fact, Governor, she believes that you are part of the simulation.”
“What!”
“It was necessary, believe me. If she thought you were a real person, she would of course wonder what you were doing in the made-up world of her simulation. We have to work very hard to make her believe the real world is something we have made up for her.”
“And so you had to tell her that I did not really exist.”
“Precisely. From her point of view, sapient beings are divided into three groups—one, those who exist in the real world, but don’t have anything to do with her; two, real-world people here in the lab and in the field who talk with her and interact with her—and three, simulants, simulated intelligences.”
“Simulants,” Kresh said, very clearly not making it into a question. He was ordering her to explain the term, not asking her to do so.
“Ah, yes, sir. That’s the standard industry term for the made-up humans and robots placed in a simulation. Unit Dee believes that the entire population of Inferno is really nothing more than a collection of simulants—and you are a member of that population.”
“Are you trying to tell me I can’t talk to her because she’ll realize that I’m not made-up?” Kresh asked.
“Oh, no, sir! There should be no problem at all in your talking with Unit Dee. She talks every day with ecological engineers and field service robots and so on. But she believes them all to be doing nothing more than playing their parts. It is essential that she believe the same thing about you.”
“Or else she’ll start wondering if her simulated reality is actually the real world, and start wondering if her actions have caused harm to humans,” said Kresh.
“She has actually caused the death of several humans already,” Soggdon replied. “Unavoidably, accidentally, and only to save other humans at other times and places. She has dealt reasonably well with those incidents—but only because she thought she was dealing with simulants. And, I might add, she does have a tendency to believe in her simulants, to care about them. They are they only world she’s ever known.”
“They are the only world there is,” said Kresh. “Her simulants are real-life people.”
“Of course, of course, but my point is that she knows they are imaginary, and yet has begun to believe in the
m. She believes in them in the way one might care about characters in a work of fiction, or the way a pet owner might talk to her nonsentient pet. On some level Unit Dee knows her simulants are not real. But she still takes a genuine interest in them, and still experiences genuine, if mild, First Law conflict when one of them dies and she might, conceivably, have prevented it. Causing the death of simulants has been extremely difficult for her.
“If she were to find out she had been killing real people—well, that would be the end. She might simply experience massive First Law conflict and lock up altogether, suffer brainlock and die. Or worse, she might survive.”
“Why would it be worse if she survived?” Kresh asked.
Soggdon let out a long weary sigh and shook her head. She looked up at the massive hemisphere and shook her head. “I don’t know. I can guess. At best, I think she would find ways to shut down the whole operation. We’d try to stop her, of course, but she’s too well hooked in, and she’s awfully fast. I expect she’d order power shutdowns, find some way to deactivate Unit Dum so he couldn’t run the show on his own, erase computer files—that sort of thing. She’d cancel the reterraforming project because it could cause injury to humans.”
“The best sounds pretty bad. And at worst?”
“At worst, she would try to undo the damage, put things back the way they were.” Soggdon allowed herself a humorless smile. “She’d set to work trying to un-reterraform the planet. Galaxy alone knows what that would end up like. We’d shut her down, of course, or at least try to do so. But I don’t need to exaggerate the damage she could do.”
Kresh nodded thoughtfully. “No, you don’t,” he said. “But I still need to talk with her—and with Unit Dum. You haven’t said much about him, I notice.”
Soggdon shrugged. “There’s not much to say. I suppose we shouldn’t even call him a he—he’s definitely an it, a soulless, mindless, machine that can do its job very, very, well. When you speak with him, you’ll really be dealing with his pseudoself-aware interface, a personality interface—and, I might add, it is quite deliberately not a very good one. We don’t want to fool ourselves into thinking Unit Dum is something he is not.”
