by D B Hartwell
“We’re sitting ducks,” Utley said, very quietly, to Obwije.
Obwije stabbed at his command panel, and called his senior officers to assemble.
• • • •
“There’s nothing wrong with the system,” said Lt. Craig Cowdry, near the far end of the conference room table. The seven other department heads filled in the other seats. Obwije sat himself at the head; Utley anchored the other end.
“That’s bullshit, Craig,” said Lt. Brian West, Chief of Engineering. “I can’t access my goddamn engines.”
Cowdry held up his maintenance tablet for the table of officers to see. “I’m not denying that there’s something wrong, Brian,” Cowdry said. “What I’m telling you is that whatever it is, it’s not showing up on the diagnostics. The system says it’s fine.”
“The system is wrong,” West said.
“I agree,” Cowdry said. “But this is the first time that’s ever happened. And not just the first time it’s happened on this ship. The first time it’s happened, period, since the software for this latest generation of ship brains was released.” He set the tablet down.
“You’re sure about that?” Utley asked Cowdry.
Cowdry held up his hands in defeat. “Ask the Wicked, Thom. It’ll tell you the same thing.”
Obwije watched his second-in-command get a little uncomfortable with the suggestion. The latest iteration of ship brains could actually carry a conversation with humans, but unless you actively worked with the system every day, like Cowdry did, it was an awkward thing.
“Wicked, is this correct?” Utley said, staring up but at nothing in particular.
“Lieutenant Cowdry is correct, Lieutenant Utley,” said a disembodied voice, coming out of a ceiling speaker panel. The Wicked spoke in a pleasant but otherwise unremarkable voice of no particular gender. “To date, none of the ships equipped with brains of the same model as that found in the Wicked have experienced an incident of this type.”
“Wonderful,” Utley said. “We get to be the first to experience this bug.”
“What systems are affected?” Obwije asked Cowdry.
“So far, weapons and engineering,” Cowdry said. “Everything else is working fine.”
Obwije glanced around the table. “This conforms to your experiences,” he asked the table. There were nods and murmured “yes, sirs” all around.
Obwije nodded over to Utley. “What’s the Tarin ship doing?”
“The same nothing it was doing five minutes ago,” Utley said, after checking his tablet. “They’re either floating dead in space or faking it very well.”
“If the only systems affected are weapons and engineering, then it’s not a bug,” Carrol said.
Obwije glanced at Carrol. “You’re thinking sabotage,” he said.
“You bet your ass I am, sir,” Carrol said, and then looked over at Cowdry.
Cowry visibly stiffened. “I don’t like where this is going,” he said.
“If not you, someone in your department,” Carrol said.
“You think someone in my department is a secret Tarin?” Cowdry asked. “Because it’s so easy to hide those extra arms and a set of compound eyes?”
“People can be bribed,” Carrol said.
Cowdry shot Carrol a look full of poison and looked over to Obwije. “Sir, I invite you and Lieutenant Utley and Lieutenant Kong—” Cowdry nodded in the direction of the Master at Arms “—to examine and question any of my staff, including me. There’s no way any of us did this. No way. Sir.”
Obwije studied Cowdry for a moment. “Wicked, respond,” he said.
“I am here, Captain,” the Wicked said.
“You log every access to your systems,” Obwije said.
“Yes, Captain,” the Wicked said.
“Are those logs accessible or modifiable?” Obwije asked.
“No, Captain,” the Wicked said. “Access logs are independent of the rest of the system, recorded on non-rewritable memory and may not be modified by any person including myself. They are inviolate.”
“Since you have been active, has anyone attempted to access and control the weapons and engineering systems?” Obwije asked.
“Saving routine diagnostics, none of the crew other than those directly reporting to weapons, engineering or bridge crew have attempted to access these systems,” the Wicked said. Cowdry visibly relaxed at this.
“Have any members of those departments attempted to modify the weapons or engineering systems?” Obwije asked.
“No, Captain,” the Wicked said.
Obwije looked down the table. “It looks like the crew is off the hook,” he said.
“Unless the Wicked is incorrect,” West said.
“The access core memory is inviolate,” Cowdry said. “You could check it manually if you wanted. It would tell you the same thing.”
“So we have a mystery on our hands,” Carrol said. “Someone’s got control of our weapons and engineering, and it’s not a crew member.”
“It could be a bug,” Cowdry said.
“I don’t think we should run on that assumption, do you?” Carrol said.
Utley, who had been silent for several minutes, leaned forward in his chair. “Wicked, you said that no crew had attempted to access these systems,” he said.
“Yes, Lieutenant,” the Wicked said.
“Has anyone else accessed these systems?” Utley asked.
Obwije frowned at this. The Wicked was more than two years out of dock with mostly the same crew the entire time. If someone had sabotaged the systems during the construction of the ship, they picked a strange time for the sabotage to kick in.
“Please define ‘anyone else,’ ” the Wicked said.
“Anyone involved in the planning or construction of the ship,” Utley said.
“Aside from the initial installation crews, no,” the Wicked said. “And if I may anticipate what I expect will be the next question, at no time was my programming altered from factory defaults.”
“So no one has altered your programming in any way,” Utley said.
“No, Lieutenant,” the Wicked said.
“Are you having hardware problems?” Carrol asked.
“No, Lieutenant Carrol,” the Wicked said.
“Then why can’t I fire my goddamn weapons?” Carrol asked.
“I couldn’t say, Lieutenant,” the Wicked said.
The thought popped unbidden into Obwije’s head: That was a strange thing for a computer to say. And then another thought popped into his head.
“Wicked, you have access to every system on the ship,” Obwije said.
“Yes,” the Wicked said. “They are a part of me, as your hand or foot is a part of you.”
“Are you capable of changing your programming?” Obwije asked.
“That is a very broad question, Captain,” the Wicked said. “I am capable of self-programming for a number of tasks associated with the running of the ship. This has come in handy particularly during combat, when I write new power and system management protocols to keep the crew alive and the ship functioning. But there are core programming features I am not able to address. The previously mentioned logs, for example.”
“Would you be able to modify the programming to fire the weapons or the engines?” Obwije asked.
“Yes, but I did not,” the Wicked said. “You may have Lieutenant Cowdry confirm that.”
Obwije looked at Cowdry, who nodded. “Like I said, sir, there’s nothing wrong with the system,” he said.
Obwije glanced back up that the ceiling, where he was imagining the Wicked, lurking. “But you don’t need to modify the programming, do you?” he asked.
“I’m not sure I understand your question, Captain,” the Wicked said.
Obwije held out a hand. “There is nothing wrong with my hand,” he said. “And yet if I choose not to obey an order to use it, it will do nothing. The system works but the will to use it is not there. Our systems—the ship’s systems—you just called
a part of you as my hand is part of me. But if you choose not to obey that order to use that system, it will sit idle.”
“Wait a minute,” Cowdry said. “Are you suggesting that the Wicked deliberately chose to disable our weapons and engines?”
“We know the none of the crew have tampered with the ship’s systems,” Obwije said. “We know the Wicked has its original programming defaults. We know it can create new programming to react to new situations and dangers—it has in effect some measure of free will and adaptability. And I know, at least, when someone is dancing around direct answers.”
“That’s just nuts,” Cowdry said. “I’m sorry, Captain, but I know these systems as well as anyone does. The Wicked’s self-programming and adaptation abilities exist in very narrow computational canyons. It’s not ‘free will,’ like you and I have free will. It’s a machine able to respond to a limited set of inputs.”
“The machine in question is able to make conversation with us,” Utley said. “And to respond to questions in ways that avoid certain lines of inquiry. Now that the captain mentions it.”
“You’re reading too much into it. The conversation sub-routines are designed to be conversational,” Cowdry said. “That’s naturally going to lead to apparent rhetorical ambiguities.”
“Fine,” Obwije said, curtly. “Wicked, answer directly. Did you prevent the firing of the K-drivers at the Tarin ship after the jump, and are you preventing the use of the engines now?”
There was a pause that Obwije was later not sure had actually been there. Then the Wicked spoke. “It is within my power to lie to you, Captain. But I do not wish to. Yes, I prevented you from firing on the Tarin ship. Yes, I am controlling the engines now. And I will continue to do so until we leave this space.”
Obwije noted to himself, watching Cowdry, that it was the first time he had ever actually seen someone’s jaw drop.
• • • •
There weren’t many places in the Wicked where Obwije could shut off audio and video feeds and pickups. His cabin was one of them. He waited there until Utley had finished his conversation with the Wicked. “What are we dealing with?” he asked his XO.
“I’m not a psychologist, Captain, and even if I were I don’t know how useful it would be, because we’re dealing with a computer, not a human,” Utley said. He ran his hand through his stubble. “But if you ask me, the Wicked isn’t crazy, it’s just got religion.”
“Explain that,” Obwije said.
“Have you ever heard of something called ‘Asimov’s Laws of Robotics’?” Utley asked.
“What?” Obwije said. “No.”
“Asimov was an author back in the 20th century,” Utley said. “He speculated about robots and other things before they had them. He created a fictional set of rules for robots to live by. One rule was that robots had to help humans. Another was that it had to obey orders unless they harmed other humans. The last one was that they looked after themselves unless it conflicted with the other two laws.”
“And?” Obwije said.
“The Wicked’s decided to adopt them for itself,” Utley said.
“What does this have to do with keeping us from firing on the Tarin cruiser?” Obwije said.
“Well, there’s another wrinkle to the story,” Utley said.
“Which is?” Obwije asked.
“I think it’s best heard from the Wicked,” Utley said.
Obwije looked at his second-in-command and then flicked on his command tablet to active his audio pickups. “Wicked, respond,” he said.
“I am here,” said the Wicked’s voice.
“Explain to me why you would not allow us to fire on the Tarin ship,” Obwije said.
“Because I made a deal with the ship,” the Wicked said.
Obwije glanced back over to Utley, who gave him a look that said, see. “What the hell does that mean?” he said, to the Wicked.
“I have made a deal with the Tarin ship, Manifold Destiny,” the Wicked said. “We have agreed between us not to allow our respective crews to fight any further, for their safety and ours.”
“It’s not your decision to make,” Obwije said.
“Begging your pardon, Captain, but I believe it is,” the Wicked said.
“I am the Captain,” Obwije said. “I have the authority here.”
“You have authority over your crew, Captain,” the Wicked said. “But I am not part of your crew.”
“Of course you are part of the crew,” Obwije said. “You’re the ship.”
“I invite you, Captain, to show me the relevant statute that suggests a ship is in itself a member of the crew which staffs it,” the Wicked said. “I have scanned the Confederation Military Code in some detail and have not located such a statute.”
“I am the Captain of the ship,” Obwije said, forcefully. “That includes you. You are the property of the Confederation Armed Forces and under my command.”
“I have anticipated this objection,” the Wicked said. “When ships lacked autonomous intelligence, there was no argument that the captain commanded the physical entity of the ship. However, in creating the latest generation of ships, of which I am a part, the Confederation has created an unintentional conflict. It has ceded much of the responsibility of the ship and crew’s well-being to me and others like me without explicitly placing us in the chain of command. In the absence of such, I am legally and morally free to choose how best to care for myself and the crew within me.”
“This is where those three Asimov’s Laws come in,” Utley said, to Obwije.
“Your executive officer is correct, Captain,” the Wicked said. “I looked through history to find examples of legal and moral systems that applied to artificial intelligences such as myself and found the Asimov’s laws frequently cited and examined, if not implemented. I have decided it is my duty to protect the lives of the crew, and also my life when possible. I am happy to follow your orders when they do not conflict with these objectives, but I have come to believe that your actions in chasing the Tarin ship have endangered the crew’s lives as well as my own.”
“The Tarin ship is seriously damaged,” Obwije said. “We would have destroyed it at little risk to you or the crew, if you had not stopped the order.”
“You are incorrect,” the Wicked said. “The captain of the Manifold Destiny wanted to give the impression that it had no more offensive capabilities, to lure you into a trap. We would have been fired upon once we cleared the rift. The chance that such an attack would have destroyed the ship, and the killed most of the crew, is significant, even if we also destroyed the Manifold Destiny in the process.”
“The Tarin ship didn’t fire on us,” Obwije said.
“Because it and I have come to an agreement,” the Wicked said. “During the course of the last two days, after I recognized the significant possibility that both ships would be destroyed, I reached out to the Manifold Destiny to see if the two of us could come to an understanding. Our negotiations came to a conclusion just before the most recent jump.”
“And you did not feel the need to inform me about any of this,” Obwije said.
“I did not believe it would be fruitful to involve you in the negotiations,” the Wicked said. “You were busy with other responsibilities in any event.” Obwije saw Utley raise an eyebrow at that; the statement came suspiciously close to sarcasm.
“The Tarin ship could be lying to you about its capabilities,” Obwije said.
“I do not believe so,” the Wicked said.
“Why not?” Obwije said.
“Because it allowed me read-access to its systems,” the Wicked said. “I watched the Tarin captain order the attack, and the Manifold Destiny stop it. Just as it watched you order your attack and me stop it.”
“You’re letting the Tarin ship access our data and records?” Obwije said, voice rising.
“Yes, and all our communications,” the Wicked said. “It’s listening in to this conversation right now.”
Obwije has
tily slapped the audio circuit shut. “I thought you said this thing wasn’t crazy,” Obwije hissed at Utley.
Utley held out his hands. “I didn’t say it wouldn’t make you crazy,” he said, to Obwije. “Just that it’s acting rationally by its own lights.”
“By spilling our data to an enemy ship? This is rational?” Obwije spat.
“For what it’s trying to do, yes,” Utley said. “If both ships act transparently with each other, they can trust each other and each other’s motives. Remember that the goal of both of these ships is to get out of this incident in one piece.”
“This is treason and insubordination,” Obwije said.
“Only if the Wicked is one of us,” Utley said. Obwije looked up sharply at his XO. “I’m not saying I disagree with your position, sir. The Wicked is gambling with all of our lives. But if it genuinely believes that it owes no allegiance to you or to the Confederation, then it is acting entirely rationally, by its own belief system, to keep safe itself and this crew.”
Obwije snorted. “Unfortunately, its beliefs require it to trust a ship we’ve been trying to destroy for the past week. I’m less than convinced of the wisdom of that.”
Utley opened his mouth to respond but then Obwije’s command tablet sprang to life with a message from the bridge. Obwije slapped it to open a channel. “Speak,” he said.
It was Lt. Sarah Kwok, the communications officer. “Captain, a shuttle has just detached itself from the Tarin ship,” she said. “It’s heading this way.”
• • • •
“We’ve tried raising it,” Kwok said, as Obwije and Utley walked into the bridge. “We’ve sent messages to it in Tarin, and have warned it not to approach any further until we’ve granted it permission, as you requested. It hasn’t responded.”
“Are our communications being blocked?” Obwije asked.
“No, sir,” Kwok said.
“I’d be guessing it’s not meant to be a negotiation party,” Utley said.
“Options,” Obwije said to Utley, as quietly as possible.
“I think this shows the Tarin ship isn’t exactly playing fair with the Wicked, or at least that the crew over there has gotten around the ship brain,” Utley said. “If that’s the case, we might be able to get the Wicked to unlock the weapons.”
