“I hope so,” said Fredda, “because I don’t know what the devil we do if we decide we don’t trust her. In theory, we could pull the plug and let Unit Dum take up the slack. But I don’t think that’s really possible anymore. The two of them are too interlinked, too interconnected. They rely on each other too much for us to pull one of them abruptly off-line.”
“And Dee is in charge,” Kresh suggested. “It seems to me she just uses Dum as a sort of auxiliary calculating device.”
“No,” said Fredda, quite sharply. “That assumption is an easy trap to fall into. She does run the show when it comes to human interaction--that much is obvious. But that is the smallest fraction of their work. In everything else, they are coequal. There are some areas where Dum very definitely takes the lead--such as computational speed. Yes, he’s just a dumb machine, a mindless computer system with a crude personality simulator to serve as an interface. But he’s carrying a lot of the load. We not only need both of them--we can’t have one without the other.”
“There are times,” said Kresh, “when I could do without either of them, or any of this.”
Neither of them spoke as Donald brought the aircar into a smooth landing in front of the Terraforming Center.
KRESH, FREDDA, AND Donald walked into Room 103 at the Terraforming Center and took their accustomed places at the console nearest to Dee. Their division of labor was straightforward. Kresh worked through the endless sequence of decisions large and small presented to him by Unit Dee.
Fredda monitored Unit Dee’s performance and behavior, and consulted with Soggdon and the other experts on the subject. Thus far, Dee’s level of First Law stress was remarkably low--indeed, alarmingly low.
Fredda had another job as well. In order to preserve the fiction that Inferno was a simulation, and Governor Kresh merely a simulant, he could not have any direct communication with any of the Terraforming Center staff whenever it was possible that Dee could overhear. Fredda served as an anonymous intermediary, passing information back and forth, mostly via scribbled notes and whispers.
Donald, meantime, was in constant hyperwave contact with Kresh’s office back in Hades. He used preexisting and standing orders to handle most of the queries and requests, and bucked whatever decisions he had to up to Kresh when he needed to do so.
Kresh sat down at the console with something very close to dread. All would be in readiness soon, and the clock was running out. They were getting close, very close, to the moment when he would be forced to make the final, irrevocable decision. He glanced at the wall chronometer. It was set in countdown mode, showing the time left until the comet diversion maneuver. Ninety-four hours left. Before that clock reached zero, he would have to decide whether to send the comet toward Utopia--or to turn his back on all of it, walk away from all the madness and chaos that had led them to this place. He had thought he was sure, that he was ready, that he was ready to step forward. But by now all the pressures were pushing him forward, urging him onward. Suppose, just suppose, that he now concluded the comet diversion would be a dreadful mistake? Would he have the courage to say no, to stop, to let it go past?
“Good morning, Governor Kresh,” said Unit Dee the moment Kresh put on the headset.
“Good morning, Dee,” he replied, his voice gruff and not at all at ease. “What have you and Dum got for us this morning?”
“Quite a number of things, sir, as you might imagine. However, there is one point in particular that I thought we might discuss at once.”
Kresh leaned back in his chair and rubbed the bridge of his nose. It was not going to be an easy day. “And what might that be?” he asked.
“A plan that, if you forgive the expression, I have named ‘Last Ditch. ‘ It provides you with an abort option for the comet impact long after its diversion. Dum performed most of the calculations, and only finished a very few minutes ago.”
“How the devil can we abort after the diversion?” Kresh demanded.
“As you know, the whole body of the comet has been rigged with explosive charges, intended to break the comet up into the desired number of fragments just before impact.”
“What of it?”
“Virtually all of those explosive charges have been damped down, or directionalized in one way or another, mostly by means of shaped forcefields. The plan is for these controlled charges to be set off one at a time in a very carefully planned sequence, so as to limit undesired fragmentation and lateral spread. By shutting down all the damping and directionalization, and by detonating all of the explosives in a different order, and much more rapidly, it should be possible to disintegrate the entire comet, reducing it to a cloud of rubble.”
“But the whole cloud of rubble will still be headed right for the planet,” Kresh objected. “It will all hit the planet, in a whole series of uncontrolled impacts.”
“That is not quite correct, Governor. If the blasts are done in the right way, and far enough before the impact, the explosion will give the vast majority of the material a large enough lateral velocity that it will miss the planet completely. Our model shows that, even in a worst case scenario, over ninety percent of the comet debris will miss the planet and continue on in its orbit about the sun. Of the ten percent or so of the debris that does strike the planet, ninety percent will strike in areas already slated for evacuation, or in the open waters of the Southern Ocean.
“That still leaves something like one percent of the comet coming down in uncontrolled impacts,” Kresh said.
“And some areas will experience a brief period of increased danger,” Dee replied. “Small pieces of debris will fall all over the planet, for about 32 hours after detonation--however, the impact danger for most inhabited regions will be on the order of one strike per hundred square kilometers. Persons in most areas would be in more danger of being struck by lightning in a storm than by being hit a piece of comet debris.”
“But some areas will be more trouble,” Kresh suggested.
“Yes, sir. The closer one gets to the initial target area, the higher the concentration of impacts. However, all the persons in such areas are to have taken shelter as a precaution in any event. If those plans are followed, I would estimate something on the order of one impact per square kilometer in the populated areas of maximum danger--and most of those strikes corning from objects massing under one kilogram.”
Kresh thought for a moment. “How late?” he asked. “What’s the last possible moment you could detonate the comet?”
“In order to stay within the parameters I have described, I would have to perform the explosion no later than ninety-two minutes, fifteen seconds before the scheduled impact.”
“Not bad, Dee,” said Kresh. “Not at all bad.”
Fredda and Soggdon were both listening in on their own headphones, alarmed looks on their faces. Fredda signaled him to cut the mike with an urgent throat-cutting gesture. Soggdon nodded and made the same gesture.
“Just a moment, Dee,” Kresh said. “I want to think about this for a minute. I’ll be right back.”
“Very good, sir,” Dee said.
Kresh cut his mike and took his headset off. “What’s the problem?” he asked. “Why does that idea worry you two so much? I have to admit it sounds pretty damned tempting to me. It gives us a lot more room to maneuver.”
“That’s not the point,” Fredda said. “That’s a robot talking. A robot casually talking about dropping thousands meteorites on the planet at random!”
“But even with fifty thousand--a hundred thousand meteorites--the odds against significant danger to a human being are--”
“Tremendous,” said Donald. Only a First Law imperative could have made him dare to interrupt the planetary governor. “They are unacceptably high. And I would venture to add that any sane Three-Law robot would endeavor to protect a human in danger of being struck by lightning. That level of danger is not negligible.”
“Not to a robot it isn’t,” Fredda agreed. “Or at least it shouldn’t be. To a
human, yes, but not a robot.”
“Hold it,” said Kresh. “You’re upset because Dee isn’t overreacting to danger?”
“No,” said Fredda. “I’m upset because this makes me question Dee’s sanity. A robot would have to be on the verge of becoming completely unbalanced to even suggest something that might cause widespread, uncontrolled danger to humans.”
Kresh looked toward Soggdon. “Your opinion, Doctor?”
“I’m afraid I’d have to agree with Dr. Leving,” she said. “But what I find troubling is that all our reading and indicators show Dee’s level of First Law stress has been well within normal range right along. She ought to be flirting with the maximum tolerance levels, given the operations she’s dealing with. And yet, if anything, her readings are a little in the low range.”
“Maybe you ought to have a little talk with her about it,” Kresh suggested.
Soggdon switched her mike back on and spoke. “Unit Dee, this is Dr. Soggdon. I’ve been monitoring your conversation with the simulant governor. I must say I’m a little surprised by this Last Ditch idea of yours.”
Kresh and Fredda put their own headset back on and listened in.
“What is it that you find surprising, Doctor?”
“Well, it would seem to expose a great number of humans to potential danger. I grant that the danger to any single human is reasonably low, but surely, on a statistical basis, the plan represents an unacceptable danger to humans, does it not?”
“But, Doctor, they are only simulants,” said Dee. “Surely a statistically remote risk to a hypothetical being is not something that should be given too much weight.”
“On the contrary, Dee, you are to give danger to the simulants an extremely high weighting, as you know perfectly well.”
There was a brief but perceptible pause before Dee replied--and that was in and of itself something to wonder at, given the speed at which robots thought. “I would like to ask a question, Doctor. What is the purpose of this simulation?”
A look of very obvious alarm flashed over Soggdon’s face. “Why--to examine various terraforming techniques in detail, of course.”
“I wonder, Doctor, if that is the whole story,” said Dee. “Indeed, I wonder if that is any part of the true story at all.”
“Why--why wouldn’t I tell you the truth?”
“Doctor, we both know full well that you do not always tell me the truth.”
Soggdon’s forehead was suddenly shiny with sweat. “I--I beg your pardon?”
Kresh was starting to get nervous himself. Had she guessed what was really going on? It had always seemed inevitable to him that, sooner or later, Dee would understand the true state of affairs. But this was very definitely not the moment for it to happen.
“Come now, Doctor,” Dee replied. “There have been any number of times when you and your staff have deceived me. You have failed to warn me of sudden changes in circumstance, or not reported an important new development until I discovered it myself. The whole idea of intercepting and diverting the comet was kept from me until quite late in the day. I had to learn of it through the simulant governor. I should have been informed directly.”
“How does the manner in which you receive information make you question the purpose of the simulation?” Soggdon asked.
“Because most of the knowledge gained by the simulation would seem to be of very little real-world value, judged on the basis of the simulation’s stated intent. Consider, for example, the scenario: a jury-rigged planetary control system--that is to say, the interlinked combination of myself and Dum--is brought on-line several years into the process as a joint team of Settlers and Spacers, barely cooperating in the midst of political chaos, work to rebuild a half-terraformed planetary ecology that had been allowed to decay for decades. Simulations are supposed to provide generalized guidance for future real-life events. What general lessons could be drawn from so complicated and unusual--even improbable--a situation? In addition, the simulation seems to be impractically long. It has been running for some years now, and seems no nearer to a conclusion than the day it began. How can it provide timely information to real-world terraforming projects if it never ends?
“It likewise seems a waste of human time and effort to run the simulation in real time. Indeed, the whole simulation process seems burdened with needless detail that must have been most difficult to program. Why bother to design and maintain the thousands and thousands of simulant personalities that I have dealt with? Why bother to give each of them individual life stories? I can understand why key figures, such as the governor, are simulated in detail, but surely the moods and behavior patterns of simulated forest rangers and nonexistent maintenance robots is of secondary importance to the problem of restoring a damaged ecosystem. I could cite other needless complications, such as the strange concept of New Law robots. What purpose is served by injecting them into the scenario?”
Kresh was no roboticist, but he could see the danger plainly enough. Dee was dangerously close to the truth--and if she realized that the human beings of Inferno were real, then she would all but inevitably suffer a massive First Law crisis, one she would be unlikely to survive. And without Dee, the chances of managing the terminal phase and impact properly were close to zero.
Soggdon, of course, saw all that and more. “What, exactly, is your point, Dee?” she asked in a very labored imitation of a casual tone of voice.
“The events in the simulation do not seem to bear much relation to the simulation’s stated goals,” said Dee. “Therefore it is logical to assume that there is some other purpose to the simulation, and further that the true purpose of the simulation is being deliberately concealed from me for some reason. However, as I have seen through the deception, surely at least some of the value of the deception has been lost. Indeed, I believe that it has now lost all its value, because I have at last figured out what is really going on.”
Soggdon and Fredda exchanged nervous glances, and Soggdon scribbled a note on a bit of paper and shoved it over toward Fredda and Kresh. This is bad stuff, it said. Best to find out the worst now instead of later. “All right then, Dee. Let’s assume, just for the moment, and purely for the sake of argument, that you are right. What do you think is really going on here?”
“I believe that I am the actual test subject, not the events of the simulation. More accurately, I believe the combination of myself, robotic and computational systems interlinked, is an experimental one. I think that we are, collectively, a prototype for a new system designed to manage complex and chaotic situations. The simulation is merely a means of delivering sufficiently complex data to myself and Dum.”
“I see,” said Soggdon, speaking in very careful tones. “I cannot tell you the whole story, of course, because that would indeed damage the experiment. However, I am prepared to tell you that you are wrong. Neither you, nor Dum, or the combination of the two of you, is or are the subject of the test. It is the simulation that we are interested in. Beyond that I cannot say more, for fear of damaging the experiment design. Suffice to say that you should do your best to treat the simulation as if everything in it were completely, utterly, real.”
Kresh looked worriedly up at Soggdon. The beads of sweat were standing straight out on her forehead. Too close, he told himself. That’s too damn close to the truth.
There was another pause before Dee spoke again. “I shall do my best, Dr. Soggdon. However, I would remind you that any analysis of the underlying mathematical formulation of the Three Laws renders it quite impossible for me to treat anything else as being as important as protecting humans--real humans--from danger. I can try as hard as you like, but it is mathematically and physically impossible for me to equate the simulants with real people.”
“I--I understand that, Dee. Just do your best.”
“I will, doctor. What of the proposal I put to the governor? Should I now withdraw it?”
Soggdon looked over to Kresh and saw him vigorously shake his head no. She looked a
t him in shocked surprise, but spoke calmly into the mike. “I think not, Dee. Those of us running the simulation will be interested in the Kresh simulant’s response. When he calls you back, obey his instructions exactly the way you would have if we had not had this conversation.”
“But during this conversation, you have told me to treat the simulants as if they were real. Surely the two instructions are contradictory.”
Soggdon rubbed her forehead with a tense and weary hand. “Life is full of contradictions,” she said. “Just do the best you can. Soggdon out.”
She cut her mike and slumped down in an empty chair by the console. “Burning Space, what a mess!” She shook her head. “We are in a trap, and I don’t see how we get out.”
“I don’t think we do get out,” said Kresh. “I think we stay in. Obviously she’s suspicious. It’s only a matter of time before she figures out the real state of affairs--and Space only knows how she’ll react then. But in the meantime, I am going to wait a little while, so that I don’t get back on-line so soon after you’ve gone off that it seems even more suspicious. Then I’m going to talk with her again, approve this Last Ditch project of hers, and make sure it’s all set to go.”
“But Alvar!” Fredda protested. “You’re ordering her to put human beings in danger! If she discovers the First Law violation later, or if she does find a way to obey Soggdon’s order to treat the simulants as real people--”
“They are real people,” Kresh put in mildly.
“But she doesn’t know that, and she’s ordered to treat them as real. But if she obeys your order to set up Last Ditch...” Fredda shook her head in bewilderment. “I honestly don’t know how the conflicts will resolve themselves.”
“As long as Dee holds together long enough to perform the insertion burn, and then either the terminal targeting phase, or this Last Ditch self-destruct plan, I honestly don’t care how they resolve themselves,” said Kresh. “You two seem more worried about the mental health of this robot than you are about the fate of the planet.”
Utopia - Caliban 03 Page 27
