2312

by Kim Stanley Robinson

  A clear rectangular pavilion tent had been set up on one corner of the plaza. Inside it stood a smaller cloth tent, like a yurt or a partially deflated buckyball, resting on the bigger tent’s floor. Wahram led Swan and the inspector through the locks of the tent, then into the inner yurt. There they found a small group of people sitting on cushions on the floor, in a rough circle.

  They all stood to greet the newcomers. There were about a dozen or fifteen of them there. Obviously most of them already knew Wahram and Genette, and Swan was introduced to more people than she could remember.

  When the introductions were done and they were all seated on the floor, Wahram turned to Swan. “Swan, we would like to have a conversation with you that does not include Pauline. We hope you will again agree to turn her off.”

  Swan hesitated, but something in Wahram’s face—some strangled inarticulate entreaty, as one might see on the face of Toad when trying to convince Rat and Mole to join him in something he thought terribly important—caused her to say, “Yes, of course. Turn everything off, Pauline.” And after she heard the click Pauline used to announce her sleep, Swan pushed the button behind her ear for good measure.

  “She’s off,” Swan said. She turned off Pauline all the time, but she didn’t like other people asking her to do it.

  Inspector Genette jumped up and stood on the table before her; they were almost exactly at eye level. “We’re also wondering if we can make a check to be sure that Pauline is completely inactivated. Sometimes the human host cannot be sure. You’ll notice that I left Passepartout back in the city, for instance.”

  “It could be recording you from a distance, no?” she said.

  Genette looked doubtful. “I don’t think so, but it’s precisely to forestall any such eavesdropping that we are inside this confidential space. We have black-boxed ourselves. But we’d like to make sure the interior is clear by running some tests on you.”

  “All right,” Swan said, as huffy as Pauline would have been. “Check her out, but I’m sure she’s asleep.”

  “Sleeping people still hear things. We want her off. And indeed, may I recommend to you the advantages of keeping your qube separated from your own body.”

  “Impolite people have often suggested that,” Swan replied.

  The tests of Pauline’s level of activity were run by way of wands placed against her neck; then Swan was asked to briefly don a flexible mesh cap.

  “All right,” Wahram said when one of his colleagues nodded confirmation. “We are alone here now, and this conversation is not being recorded. We must all agree to keep what is said here secret. Will you do that?” he said to Swan.

  “I will,” Swan said.

  “Good. Alex started these meetings, along with Jean here. She felt there were problems developing that should be discussed outside the realm of the system’s artificial intelligences. And one problem was a new kind of qube appearing on the scene. Inspector?”

  Inspector Genette said to Swan, “You recall those supposed people on the Inner Mongolia? In a way they passed the Turing test, or the Swan test, I guess you could call it, in that you thought they were people putting on an act. People do that sometimes, and in many ways it’s a likelier explanation than the existence of a completely achieved humanoid.”

  “I still think they were people,” Swan said. “Do you know any different?”

  “Yes. Those were three of the humanoid qubes that we have discovered. There are about four hundred of them. Most of them act very much like people, and keep a low profile. A few act very strangely. The three you met were some of the strange ones. Another one made that break-in attempt at Wang’s station on Io. We recovered the remains of it from the lava, and the quantum dot framework was still detectable.”

  Swan shook her head. “Those three I met seemed a little too foolish to be machines, if you see what I mean.”

  “Maybe you’re just used to Pauline,” the inspector suggested.

  Swan said, “But she’s often foolish. Nothing new there. Although I admit, she surprises me quite a lot. More than most people surprise me.”

  “You are always saying otherwise to her,” Wahram noted with a curious glance.

  “Yes. I like to tease her.”

  Genette nodded. “But you programmed Pauline to have a bold character—a conversationalist, directed to respond to things at a slant. There’s some recursive programming in her that gives more weight to associative and metaphorical thinking than to logical if-thens.”

  “Well, but that’s only part of it. Deduction is logical, supposedly, and she has a strong deduction program. But deduction turns out to be almost as metaphorical as free association. In the end it’s wild what she will say.”

  Wahram said to the entire group, “The question of programming lies at the heart of today’s meeting. There’s clear evidence to suggest that some qubes are actively self-programming, in particular the ones involved with assembling these humanoids with qubes for brains. We don’t know that any humans asked them to do that, and we don’t know why they’re doing it. So—the first questions concern what they are, and who’s making them. We know that they can’t communicate internally with each other because of the decoherence issues. They’re not some kind of entangled group mind, in other words. But they can communicate just like we do, by talking with each other, using all the ways we ourselves communicate. But in their case, when they employ quantum encryption, it’s not possible to break their codes. Robin here”—who was the person on the other side of Wahram, and who nodded at Swan—“has been coordinating the recording of their conversations over radio and the cloud, and even in some direct vocal communications. While we can’t crack their codes, we can see that they’re talking.”

  Swan said, “But go back a bit—how could they self-program? I’ve heard that recursive self-programming does nothing but speed up operations they already know.”

  “Well, but if they were instructed to try to make something, for instance, then it might lead to some odd results. Pushing at ways to make something work could have initiated other ideas in them. It might be much like the way they play a game of chess. They’re given a task, which is to win, and they’re told to figure out ways to do it, and then, in their usual testing of all possible options, they might have had certain unexpected successes at modeling effective courses of action to get what they want. That wouldn’t be exactly a higher-order process, but it still might do the job, and lead to new algorithms. And that could then feed back into trying more things. At some point in trying to self-program for more effectiveness, they might have stumbled into consciousness, or something like it. Or the process might just have resulted in some strange new behaviors, even destructive behaviors. This, anyway, is the theory we’ve been pursuing.”

  “Do the original qube programmers think this kind of process could go very far? I mean, wouldn’t the qubes still be stuck in algorithms?”

  “As it turns out, the programmers who first built quantum computers used differing structures, and they ended up creating several different internal operating architectures. So really there are different kinds of qubes, each kind with different forms of cognition—different protocols, algorithms, neural networks. They have brain imitations of various sorts—aspects of what you might call self-awareness, and many other features of consciousness. They’re not simply one design, and in terms of their mentation, they may have started speciating.”

  Inspector Genette took over: “We’re seeing clear signs of self-programming in the qubes. Where that may have led is hard to say. But we’re worried, because they don’t have the brain architecture and chemistry that makes us think the way we do. We think very emotionally. Our emotions are crucial to decision making, long-term thinking, memory creation—our overall sense of meaning. Without these abilities we wouldn’t be human. We wouldn’t be able to function as individuals in groups. And yet the qubes don’t have emotions, but are instead thinking by way of different architectures, protocols, physical methods. Th
us they have mentalities that are not at all human, even if they are in some sense conscious. And we can’t even be sure that they resemble each other in the ways they’ve emerged into this new state. We don’t know if they think in math or in logic terms, or in a language like English or Chinese. Or if different qubes aren’t different in that way too.”

  Swan nodded as she thought it over. If the silly girls had been qubes—the lawn bowler also—that was rather amazing, just in terms of morphology. As to mentation, none of this particularly surprised her. “I talk to Pauline about these issues all the time,” she told them. “But what’s clear to me from those conversations is how crippled the qubes are by these mental absences you speak of. Maybe it is the lack of emotions. There’s so much they can’t do.”

  “So it has seemed,” Wahram said after a silence. “But now it looks like they may be generating goals for themselves. Maybe there are some pseudo-emotions there; we don’t know. Probably they still aren’t very wise—more like crickets than dogs. But, you know—we don’t know how our own minds work, in terms of creating the higher levels of consciousness. Since we can’t get inside the qubes to see what’s happening in them, we’re even less sure of them than we are of us. So… it’s a problem.”

  “Have you taken some of them apart to see?”

  “Yes. But the results are ambiguous. It’s curiously similar to trying to study our own brains—it’s the moment of thought that you want to study, but even if you can find where in the thinking mechanism the thoughts are happening, you can’t be sure what exactly is causing those thoughts, or how they are experienced from the inside. In both cases they involve quantum effects that can’t easily be tracked to a physical source or action.”

  “There’s some worry that we set a bad example by doing too much of this kind of thing,” Genette added. “What if they get the idea that it’s all right for them to study us in the same way?”

  Swan nodded unhappily, recalling the look in the eyes of the lawn bowler—even in the eyes of the silly girls, now that she was reconsidering them. They had had a look that said they would do almost anything. Or that they didn’t understand what they were saying.

  But people had that look all the time.

  “So,” Wahram said. “You see our problem. And now it’s getting more urgent, because there’s solid evidence that these qube humanoids were ordered up by other qubes—qubes in boxes or robots, or asteroid frameworks, as was usual.”

  “Why would they do that?” Swan asked.

  Wahram shrugged.

  “Is it bad?” Swan asked, thinking it over. “I mean, they can’t band together into some kind of hive mind creature, because of decoherence. And so ultimately they’re just people with qube minds.”

  “People without emotions.”

  “There have always been people like that. They get by.”

  Wahram squinted. “Actually, they don’t. But look, there’s more.” He looked at Genette, who said to Swan, “The attacks we’ve been investigating, on Terminator and the Ygassdril, both had a qubical involvement. Also, I had that photo you gave me of your lawn bowler couriered to Wang, and he went through his unaffiliated files, and though he couldn’t ID the bowler, he had photos that show your person at a meeting Lakshmi organized in Cleopatra in the year 2302. That’s significant, because the reports of strange behaviors began to appear throughout the system in the years right after that. When all the sightings are correlated and analyzed, they converge back in time and space to that meeting on Venus. We also find that the organization in Los Angeles that ordered the pebble-launching ship is entirely qubical, with the only humans involved located in a kind of board of directors. We also found qubes involved with the construction of the launch mechanism, which we now suspect was built in an unaffiliated shipyard trailing the Vesta group. We found the print order. There are very few humans in those particular shipyards anymore; they’re almost entirely robotic. So it’s at least possible that all this has been done by qubes, with no humans involved at all.”

  “Maybe so,” Swan said, “but I have to say right now, that lawn bowler had emotion. It was burning a hole in me with its look! It wanted me to know something. Otherwise why even approach me, why make those incredible shots? It wanted me to know it was there. And desire is definitely an emotion.”

  The others there considered this.

  Swan went on: “Why do you think it has to be that emotions are biochemical? Couldn’t you have emotions without hormones or blood or anything? Some new affect system that is electrical, or quantum?”

  Genette raised a hand as if to stop her: “We don’t know. All we can say is we don’t know what kind of intentionality they have now, because their intentions were very limited when they started. Read the input, run it through algorithms, present the output—that was AI intention before this. So now that it appears that they are intending things, we have to be on our guard. Not only on general principles, as with any new unknown thing, but because some of them are acting bizarrely, while others have already made attacks on us.”

  One of the group, a Dr. Tracy, Swan seemed to recall, said, “Maybe living in humanoid bodies has made these qubes emotional by definition. Embodied mind is emotional, let us say—and now they are embodied minds.”

  A woman as small as Inspector Genette stood on her chair and said, “I’m still not convinced the qubes have any higher-order thinking, including things like intentionality and emotion, which derive from consciousness itself. Despite their incredible calculating speeds, they are still operating by algorithms we gave them, or else derivable subsequent algorithms. Recursive programming can only refine these. They are simple algorithms. Consciousness is so much more complex a field than that. They can’t build from algorithms to consciousness—”

  “Are you sure?” Genette interjected.

  The small woman tilted her head in just the way Swan had seen Genette do it. “I think so. I don’t see how the higher levels of complexity could evolve from the algorithms they have. They can’t make metaphors; they can barely understand them. They can’t read facial expressions. In skills like these a four-year-old is vastly ahead of them, and an adult human simply a different order of being altogether.”

  “This is what we were taught when we were young,” Genette said. “And more importantly, when the qubes were young.”

  “But also it’s what we have studied all our lives, and seen with our own two eyes,” the small woman replied somewhat sharply. “And programmed.”

  Despite these truths, no one there looked particularly comforted.

  “What about the facility where these humanoids are made, or decanted or whatnot?” Wahram asked Genette. “Can we shut it down?”

  “When we find it,” the inspector said grumpily.

  “Could we round up all the humanoids you’ve identified?”

  “I think so,” Genette said. “We’ve had to do some scrambling there, because Alex was central to this effort, and we’ve had to reestablish our team by shaking the network pretty hard. So we managed that, and the team has relinked around her absence. They have identified and are following about four hundred of these things, as I said. Our scan of the system has been fine enough that we don’t think there are any more hiding in any settlement we have access to. I can’t be positive about the unaffiliateds, but we’re looking in all of them. While we do that, we’re keeping our distance from the humanoids we have under surveillance, and they don’t seem to know they’re tagged. Very few of them act as strange as those three in the Inner Mongolia, or the one that burned up on Io. They tend to try to blend in. I don’t know how to interpret that. It’s as if they’re waiting for something. It makes me feel like we’re not seeing the whole picture, and so I don’t want to wait much longer before we act. But it would be nice to think we understood the total situation before doing so.”

  Genette had been walking around on the table while speaking, and now stopped before Swan, as if making a case specifically to her: “These organisms, thes
e qubical humanoids, exist. And in some respects their pattern of behavior so far hasn’t been what I would call sane. Some have attacked us, and we don’t know why.”

  After a silence Wahram added, “So we have to act.”

  Lists (15)

  health, social life, job, house, partners, finances; leisure use, leisure amount; working time, education, income, children; food, water, shelter, clothing, sex, health care; mobility; physical safety, social safety, job security, savings account, insurance, disability protection, family leave, vacation; place tenure, a commons; access to wilderness, mountains, ocean; peace, political stability, political input, political satisfaction; air, water, esteem; status, recognition; home, community, neighbors, civil society, sports, the arts; longevity treatments, gender choice; the opportunity to become more what you are

  that’s all you need

  EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE MOBILE

  The spaceliner ETH Mobile was not a hollowed asteroid but rather one of the very large manufactured ships built in lunar orbit in the previous century. Made by Swiss universities and engineering firms that continued to operate them, they were combinations of glassy metals, bioceramics, aerogels, and water both frozen and liquid. They were extremely fast; frequent small fission explosions firing behind a pusher plate at the rear of the ship accelerated it at a one-g equivalent for those inside, and this very rapid rate of acceleration was typically maintained to the midpoint of a trip, at which point the ship was going so fast that it was necessary for it to turn and decelerate at the same rate. But even decelerating for half of each trip, the average speeds were so high that relatively short transit times were possible all over the solar system, and the longer the trip, the faster the top speeds became, so it was not a linear thing: Earth to Mercury took three and a half days; Saturn to Mercury, eleven days; across the Neptune orbit (“width of solar system”), sixteen days.