“Which brings us to a second scenario,” Dyer carried on. “We take active steps to deny it access to the resources it wants. The system retaliates in kind by denying us the resources that we want, say by progressively shutting down energy plants, grounding air traffic, blacking out cities . . . all kinds of things.” He raised his hands to stifle the objections that appeared written across several faces. “Don’t forget, I am not postulating that the system has any concept of Man or sees its behavior in the same terms as we would. But this is a powerful learning machine! All it knows is that certain events in the environment around it are capable of obstructing its goals, and that certain coordinated actions on its part have the effect of stopping those events from happening. It’s like a dog scratching. It just feels uncomfortable and learns that doing certain things makes it feel better. The dog doesn’t have to be an entomologist or know that it’s fleas that are causing the discomfort.”
“But how could it possibly know that cutting off power to cities or anything like that would help?” Gary Corbertson, Director of Software Engineering from Datatrex Corporation, shook his head in disbelief. “I thought you said it wouldn’t know anything about people. How could it figure out how to blackmail them if it didn’t even know about them? That doesn’t make sense.”
“It wouldn’t have to figure out why it worked,” Dyer replied. “All it would need to know is that it did. Suppose it decided that it wanted a Jupiter probe all to itself, but we tried to take the probe away and it responded by shutting down cities at the rate of ten per night. Suppose also that we knew why it was doing it. What do you think we’d do?” He nodded slowly around the room. “It’d get its Jupiter probe pretty soon, wouldn’t it?”
“Mmm . . . I think I see the point,” Schroder said slowly. “All a baby has to know about the world is that when it screams loud enough it gets what it wants.”
“Good analogy,” Dyer agreed. “I’m not suggesting the system would do anything as sophisticated as that to start with, but like a baby it would experiment, observe, connect and hypothesize. Pretty soon it would have a fair grasp of what actions resulted in what effects.
“And now take our supposition one step further,” he went on. “What if the coordinated actions that it learned amounted not merely to blackmail but to overt aggression? As far as the machine’s concerned there’d be no difference—certain actions simply makes the discomfort go away or the comfort increase. That’s where the fact that it doesn’t possess any human values or concepts at all becomes really worrying. Another scenario—it discovers that it gets far faster and more positive results when it doesn’t stop at threats; it carries them out. Now it’s exhibiting open hostility as far as we’re concerned, but it doesn’t know it.
“So, without invoking any human attributes at all, we’ve just taken it through the whole spectrum from indifference to hostility—a perfectly plausible simulation of behavior that we thought we wouldn’t have to worry about because it couldn’t evolve the emotions that normally accompany it. But now we see that it wouldn’t have to evolve any such emotions.”
As Dyer sat down, Richter, now looking less disgruntled, leaned toward him.
“Christ, Ray,” he said over the hubbub of voices that broke out on every side. “Is FISE really capable of going all the way to that extent?”
“Not one of them in a lab,” Dyer told him. “But what happens when you connect thousands of them up together? Would you want to put money on it?” Richter sat back, shaking his head slowly and frowning to himself. The meeting subsided to silence again as Campbell Roberts, Muller’s representative from Australia/New Zealand, began to speak.
“I still think we’re exaggerating the whole thing,” he declared loudly. “So there are risks. Nobody ever said there weren’t. All through history men have taken risks where the benefits they stood to gain justified them. But as we said earlier on, if the system starts doing things we don’t like, we can always pull the plug on it. If we have to, we can always take the bloody thing to bits again. Why in hell’s name are we getting so hung up about some lousy machine developing a mind of its own? We’ve got minds too, dammit, and we’ve been around a lot longer. If it wants to play survival games I reckon we could teach it a thing or two. I say put FISE in and make damn sure it never forgets who’s boss. Homo sapiens have had plenty of practice at that!”
“Maybe it won’t let you pull the plug,” Fierney pointed out.
“That’s bloody ridiculous!”
“I’m not so sure it is,” Muller commented. “Even now TITAN controls its own power sources and the manufacture of most of its own parts. If current forecasts are anything to go by, it will soon control everything related to its own perpetuation—from surveying sources of raw material to installing extensions to itself and carrying out one-hundred-percent self-repair. On top of that it controls other machinery of every description. It might not reach the point of becoming incapable of being switched off, but it could conceivably make the job of switching it off an extremely difficult and possibly costly undertaking.”
“But why should it want to do that in the first place if it doesn’t have a survival instinct?” Roberts objected.
“What have you got to say to that, Dr. Dyer?” Schroder invited.
“The same thing applies as before,” Dyer said without hesitation. “If the system evolved some overriding purpose that its programming compelled it to strive to achieve, it wouldn’t take it long to figure out that its own continued existence was an essential prerequisite to being sure of achieving it. Its own observations would tell it that its existence could not be guaranteed as things stood, so its immediate response would be to experiment in order to find out what it could do to remedy the situation. The rest follows logically from there. In other words, here we have a mechanism via which something tantamount to a survival instinct could emerge without the need for any survival-dominated origins at all. And as I said before, once you’ve got a survival instinct established, all the emotions that go with it will follow in the course of time.”
Dyer paused to allow his words time to sink in and then summarized his view on the things that had been said.
“If the system started to exhibit any of the traits we’ve been talking about, that in itself wouldn’t add up to an insurmountable problem because, as Campbell says, we can always pull the plug. As long as that’s true, the benefits outweigh the risks; and if that was all there were to it, my vote would be to upgrade the net. But that isn’t all there is to it. If the system were to evolve a survival drive, logically we would expect it to attempt making itself an unpullable plug. Even that, in itself, wouldn’t be a problem if it didn’t succeed. After all, it wouldn’t matter much what the system wanted as long as it was incapable of doing much about it. If we could guarantee that, I’d still say upgrade the net. But we can’t.
“It all boils down to two questions. One: Could the system evolve a survival instinct? Two: If it did, what could it do about it? The second is really the key. Until we can find some way of answering that with confidence, I can’t see our way clear to taking things further.”
A long silence followed Dyer’s words. Then Schroder took up the debate.
“I’m inclined to agree that we can’t recommend putting FISE into the net at this stage. As to the question of continuing with FISE research, that’s a funding issue that doesn’t concern this meeting. But something else bothers me. Everything that has been said this morning has assumed that we’ve been talking about a supercomplex that includes FISE machines. But the business at Maskelyne happened with the system as it is now. Even with just HESPER, TITAN showed itself to be capable of integrating its activities to a degree that nobody thought possible.” He gestured vaguely toward the door. “Out there is a world that’s being run by a super-complex of HESPERS. What guarantee do we have that the kinds of behavior you’ve described can’t happen even today with the system we’ve got?”
Dyer had been expecting the question. He held Schr
oder’s eye and replied simply. “None.”
Schroder considered the answer for a long time. At last he sighed and stretched his arms forward across the table in front of him.
“The objective of the meeting was to agree what to do about HESPER,” he reminded them all. “We have three choices: Allow TITAN to grow further, freeze it where it is now, or downgrade it by taking HESPER out. We can’t allow it to grow further until we have some way of obtaining guaranteed answers to Dr. Dyer’s two key questions. If we leave it as it is, we risk a repetition of the Maskelyne kind of accident but maybe on a catastrophic scale, which would clearly be totally unacceptable. Therefore, as I see it, the only choice open to us is the third. Does anybody here disagree?”
“Except you’re only going to have to cross the bridge sometime later anyway,” Richter threw in. “It doesn’t matter what kind of machines you develop in labs after you downgrade the net, the only way you’ll ever know how a planetwide complex of them will perform is by building it. In the end you’d still wind up in the same position.”
“But we don’t know what’s on the other side of this particular bridge,” Schroder pointed out. “It may lead to somewhere we don’t want to go, and if it does it will only be one-way. The key question is: Could the system make itself invulnerable? The only way we could answer that for certain would be if it did. That is obviously unacceptable because once it had reached that point it could do anything it pleased afterward. As Dr. Dyer says, we can’t proceed further until we know the answer. On the other hand, the only way we can find the answer is by getting there. It’s a vicious circle. The only alternative is not to try getting there at all but stay where we are. But the present position is unstable because of HESPER and Maskelyne, so the only way open is back.”
“So civilization levels out on a plateau,” Richter objected. “And from that line of argument there’s no way past it. What happens when we find we need something heftier than TITAN?”
“Let’s try some positive thinking,” Fritz Muller suggested. “Everybody is saying we can’t go further because we don’t know the answers to Dr. Dyer’s two questions. That’s negative. Let us say instead that we could go further if we knew the answers. Positive.”
“I like it,” Richter said immediately. He looked at Schroder. “That’s what you should put in your recommendation! Recommend to CIM and to the World Council that we do something to get some answers instead of backing off.”
The meeting concurred and Schroder duly entered the point in his notes.
“Exactly what are we recommending them to do?” he inquired, looking up at Richter as he finished writing. Richter hadn’t really thought about it. He blinked, frowned to himself, rubbed the tip of his nose, and at last looked back at Schroder.
“I don’t know,” he confessed simply.
Schroder shifted his eyes inquiringly toward Dyer.
“I don’t know,” Dyer told him.
The meeting went on to examine the problem from a dozen different angles, but at the end of the session it was obvious that nobody else knew either.
CHAPTER NINE
“Vince talked to Schroder sometime this morning about it,” Richter said from the screen in Dyer’s office. “The bean counters at CIM are still grumbling but it sounds as if your point yesterday about postponing any upgrade to the net and discontinuing the FISE project being two separate issues was well taken. It’s still in the balance, but Schroder seemed to be starting to bend your way. I thought you’d like to know.”
“Thanks,” Dyer acknowledged. “Did Vince have any idea how long it might take before we know for sure?” Richter shook his head and showed his hands in front of him on the screen. “How about this chance you mentioned of Vince maybe putting internal funding into FISE?” Dyer asked. “Any more on that?” Richter shrugged and shook his head again.
“We didn’t go into that.”
Dyer made a face and nodded resignedly. He was naturally anxious to see the FISE project saved from the axe, especially after the amount of work that he and his team had put into it and following the encouraging developments that had taken place that week, but at a more personal level there was more at stake. HESPER had been his contribution to making the world a generally better place; to him, the continuing national and international support for the FISE project symbolized the world’s acceptance and approval of both him and his work. Withdrawal from HESPER and abandonment of FISE would be tantamount to a vote of no confidence and the implications bothered him more than he was prepared to admit even to himself.
“So we’ll just have to wait and see what next week brings,” he said. “Unless somebody can come up with some answers to those questions, there’s not a lot we can do. Had any more thoughts about it, Ted? Did you get a chance to talk to Sigmund about it?”
“No,” Richter replied. “I’ve been thinking about it but it just keeps going around the same circle. Sigmund’s not in today.” Of course, Dyer thought to himself. It was Friday—another yachting weekend. “Anyhow,” Richter went on, “as you say, we haven’t got much choice but to leave it at that for the time being. I’ll keep you posted if anything happens. Okay?”
“Okay Ted, I’d appreciate it,” Dyer said. “See you around.”
“See ya, Ray.”
Dyer cut off the screen and remained staring at it for a long time. The meeting in Washington had debated the issue into the evening and he had talked to Richter about nothing else on the way back to New York, but always they had come back to the same impasse. If TITAN ever developed the equivalent of a survival drive, what could it do about it? The only way anybody would know would be when it happened. The risks implied by that would be totally out of the question. There had to be another way of getting the answers they needed.
At length he sighed, shook his head and leaned back in his chair. As he did so he noticed the wad of equipment maintenance approval forms that he had finished signing just as Richter called. He scooped them up and walked out of the office to give them to Betty, at the same time wondering to himself why it was that in an age when everything from tax returns to personal letters had become electronic, things as mundane as interdepartmental formalities still required pieces of paper in triplicate.
“These are okay,” he said as he dropped the forms on Betty’s desk. “Signed, sealed, stamped and approved. You can get rid of ’em.”
“Thanks,” Betty said. “Oh. Frank Wescott called from CIT while you were talking to Ted. He just left a message . . .” She checked a note pad by her elbow. “It said, ‘How do you deal with a time bomb that’s wired to a doomsday machine?’ ” Betty looked up curiously. “He said you’d know what it meant.”
“It’s okay,” Dyer replied, smiling. “Just something we talked about in Washington yesterday.” Betty shook her head and looked nonplused.
“Fancy that. I always wondered what you people talked about all day at those meetings. Now I know. Isn’t it nice to know there are people in the world who worry about the time bombs and the doomsday machines for us. There—that’s one more thing I won’t have to lose any more sleep wondering about.”
“Glad to hear it,” Dyer told her. “Life would sure be dull without them.”
At that moment Ron’s voice rose from behind the partitions that separated them from the lab area.
“Hell, I keep tellin’ ya, it’s obvious! We go after them with the tanks!”
“No, shut up a second, Ron,” Chris’s voice replied. “I don’t like it. There’s something funny about—”
“But they’re wide open in the center there. If we go in fast we’ll bust ’em wide—”
“Shut up, Ron!”
“But I’m tellin’ ya we’ve got ’em licked!”
“And I’m saying it’s a trap. They want us to go after them in the center. It was only a token fight and they’re pulling out too fast.”
“Sounds as if they’re fighting World War II again,” Dyer grinned. Betty raised her hands and shrugged. Dyer strolle
d through to the lab to see what was going on.
It was as he expected. The image in the holo-tank was a miniature 3-D landscape made up of wooded hills, tracts of bare, rolling plain, rivers and forests, complete with towns, roads and bridges. Formations of mixed squares; circles, triangles and other symbols glowed superimposed on the terrain to divide it roughly into two halves, one dominated by red and the other by blue, although in places the two became intermixed and disordered, giving the whole thing the appearance of something like a general’s battle map. Ron was sitting at the console and looking impatient while Chris stood peering thoughtfully down into the top of the tank.
“What is it?” Dyer inquired as he drew up alongside Chris and began examining the situation.
“Battle of Kursk, 1943,” Chris replied absently. “Germans and Russians. We’re the reds . . . Zukhov.”
“Who’s on the other end?” Dyer asked.
“Mike and Dave at Cornell,” Chris told him without looking up. “I think they’re trying to pull a fast one here. The sods have done it before.”
The battle-simulation games available from the network library were Chris and Ron’s latest craze. Domestic holo-tanks to replace conventional flat-screen displays were expensive and one of Ron’s first improvements on the FISE display had been to build an interface to hook it into the net.
“They’ve just broken off from a punch-up with tanks in the middle there by the river,” Chris said, gesturing vaguely. “Now they’re pulling back and Ron wants to go after them. I think it’s fishy.”
“How come?” Dyer asked.
“Doesn’t feel right,” Chris said. “I reckon they’ve got something hidden up behind that ridge that’s waiting to cut us off if we cross the river.” Dyer surveyed the scene for a few seconds and then nodded slowly. There was a long, shallow ridge overlooking the approach to the river crossing and the ground behind the ridge was out of the direct line-of-sight from any of the red symbols as positioned. The displays presented to the two playing teams would not be identical; with the opposing forces deployed as shown, only the enemy “generals” and the computers that functioned as umpire would know whether or not the ridge concealed additional hostile units.
Cyber Rogues Page 10
