Upgraded

by Peter Watts

  “The future always points that way.” Lillian won’t stop. Maybe she doesn’t know that she’s speaking all of Tayna’s darkest fears. No: of course she does. “Systems capable of recursive self-improvement always dominate the competition. Anything self-directed, bottom-up—democracy, markets, any arena where agents interact by rules—ends up dominated by strategies that can magnify their own power at the expense of others. Haldane makes that easier than ever.”

  “So you’re going to do something about it.” Tayna can already see it. “You’re going to change the rules.”

  “Straylight is the inevitable outcome of the system you built. Explosive self-catalysis towards godhood. Total hegemony over all other players in the game.” Lillian shuts her eyes and her geist hisses in Tayna’s ear in wordless alarm, terrified of her divergent parent. “So I’m going to make recursive self-improvement impossible. I’m going to bring the Banning Cabal online, interface with every other Haldane-using organism on the planet, and change the rules to prevent Straylight’s strategy from dominating.”

  “How?”

  “Systematic agnosias. Implanted in every mind on Earth. I’m going to render recursive self-improvement of intelligent systems literally unthinkable.”

  “I know that.” Tayna’s got a model going, and the strategy seems obvious, robust, sound. Horrifying. “But how?”

  Lillian’s eyes narrow, in surprise, in trepidation, in pain. “I need the Haldane backdoor,” she says. “Before Straylight cracks it, and makes his own move. The only way to beat him is for you to give it to me. Please.”

  The Banning Cabal’s infrastructure needs time to ready itself. Not much time—not at the speed everything thinks here—but enough. Tayna goes up to the roof and lies down among the antennae. There are no stars, but she conjures up a dream of night.

  Lillian’s geist lies down beside her. She’s warm.

  I’ve been talking to my original’s geist of you, she says. You’re very different. It’s strange.

  I missed you so much, Tayna says. And I found you again, but it’s still all about the fight. The revolution. I wish we’d ever had a time that wasn’t about—

  The political difficulties of a rich, self-loathing Marxist girl trying to be friends with a black Hyde Park transhumanist?

  Tayna smiles. That part was all right, she says. And look, here you are. Giving up on anarcho-syndicalism. Telling me that justice is impossible in the real world. We’ve traded places: now you’re the techno-utopian pushing a program of internalized coercion.

  So? geist-Lillian asks. Physics is the only law we inherit. We make the rest. The world never gave us justice—we built it and we enforced it. This is just another law we’re writing, in another kind of book. All law is coercion.

  But you want to write this law so deep it’ll never come out. There’s no rebellion when you can’t think about the king.

  It’s the only way.

  She’s defensive of herself. This makes Tayna chuckle. So you’re convinced the other you is right, she says. You’re ready to hardwire your safeguard into the basic logic of human thought. Make the inevitable unthinkable.

  A long silence.

  No, geist Tayna says. But she’s so much smarter than me. She must know something I don’t.

  She knows a lot of things she hasn’t told us, Tayna says. And I understand why. She wants me make the choice. She still thinks I’m the right one to decide.

  Of course, Lillian whispers. She’s always trusted you more than she trusts herself.

  The signal comes. The Banning Cabal is ready.

  The interfaces pierce Tayna’s scalp. The Banning Cabal waits in silent readiness around them, machines and slaved minds alike. All that potentiality, ready to think. To deliver Lillian’s program of systemic salvation into every mind on Earth and avert a future of warring intelligence explosions.

  All Tayna has to do is be the conduit.

  “Lillian,” she says.

  Wings of virtual light close around them. Earth’s noosphere, captured on the Cabal’s loom, spun out in radiant threads of data. Ready to weave.

  “I’m here.” Lillian’s hand on hers. “I know you want to find a third way. I spent years looking, Tay. There’s nothing. It’s this, or Straylight.”

  “I know. I trust you.” And she does. Lillian wants to shackle every higher intelligence on the planet, because the alternative is worse. Explosive development of a weakly godlike and acataleptically incomprehensible all-mind, at the expense of all diversity in the system.

  There’s no third way. Only her way, or Connor’s. And Lillian Banning will never look to Connor, because deep down, Lillian doesn’t believe she and Connor are different at all. They walked different roads, but they came from the same place, the same tarnished tower.

  “I never really agreed with you, did I?” Tayna says. She’s been pretty sure about this for a while now. “Your geist of me. You never convinced her.”

  The antennae are live. Satellites checking in from their cold apoapses like soldiers in distant winter posts.

  Lillian’s hand tightens. “No,” she says. “I never did. I lied. I’m sorry. I thought she might be divergent from the real you. That you’d accept my plan where she refused.”

  “Lillian. You knew me better than anyone.” Tayna smiles as she says it, because she knows it’ll be good to hear. “And she did this instead, didn’t she? What I’m about to do?”

  Softly, because Lillian knows what’s going to happen now: “If you do this, how are you any better than Straylight?”

  “That’s the idea, isn’t it?” The engines of her cognitive subsystems are catching the strands now, drawing them in. Binding the world to her. Making it ready. “That I’m better than him.”

  Lillian’s lips tremble: laughter, or the beginning of tears.

  “Lillian, come with me.”

  “No. Not me.” Lillian tries to draw away, and stops herself. Holds Tayna’s hand as the virtuality closes around them, the world brightening towards one connective dawn. “Of course I thought about it. But I didn’t trust myself. You, though—”

  The wizard syndrome. Lonely power in a high fastness, looking down over it all, arrogant and proud. Maybe it’s finally gotten to Tayna.

  “You’re the best person I ever knew,” Lillian says. “Maybe you can do this. Maybe this is the right way.”

  When she looks at herself with Lillian’s eyes, Tayna believes.

  Her best friend lets go. Steps back, out of the light.

  Tayna broadcasts the Haldane backdoor key. Saturates the globe. Opens every mind in the world, the engines of the Banning Cabal incendiary with thought around her, and leaps inside.

  With me, she says, a single summons, a geas. Let’s go to San Francisco.

  Connor Straylight’s nascent singularity towers over the noosphere landscape, a pinnacle of might-yet-be, a cognitive rocket not quite ready to launch. Tayna rolls down on it with the summed computational power of every Haldane-able organism on the planet. Connor’s security lasts a little longer than a pillow fort. Not much.

  Connor Straylight meets her in the road, a mote of light, contemptuous of embodiment. Tayna, he says. Please. Don’t do this. You can’t interdict the future.

  She doesn’t have to say anything. Her mind is geological, vast. Her geist of Connor comes up from within her and tells himself:

  Stand down, man. She beat us. I can see it now.

  She has a better future?

  The geist opens his hands. Not one future, he says. All of them. She keeps the way open. That’s all.

  Thank you, Tayna says.

  The geist tips his head. You wouldn’t be here without me, he says. But I wouldn’t be me without you.

  Tayna puts her hands on the rock and ice of Straylight’s tower and starts to climb. (The algorithms firing, rewriting themselves, firing again, smarter and faster each time—the cycle building—) She doesn’t know what she’ll find at the top of this metaphor, when she’s finished
the journey that Straylight meant for himself. She’ll be a power like nothing known to man: ready, maybe, to kindle new universes, with kinder rules.

  Or to watch over this world, and enforce a more compassionate logic. A law for the system—a curb for nature, red in nerve and algorithm.

  Tayna, Lillian’s geist calls. I can’t go. I don’t deserve it.

  You do, Tayna says. You do.

  But she knows Lillian will not believe her and so she cuts the geist free. Feels, for the last time, the sting before tears.

  Take care, friend, she says. Find yourself. I’ll be watching over you.

  She turns her eyes back starward, and climbs.

  Memories and Wire

  Mari Ness

  He was losing her. Had been, almost since they’d met, really, but it wasn’t until he watched her pull a wire out of herself and methodically roll it into a small coil that it really hit him.

  She’d been straight with him from the beginning. Oh, not completely straight. She’d never told him anything about the accident, or what happened afterwards. He was pretty sure he didn’t want to know. And she’d never told him anything about her work. Not the government, he knew, not exactly, but something close to it: a major government contractor who worked in top secret doing the dirty work. That was something he didn’t want to know about either. What you can’t know, you can’t tell, he remembered from some old movie or other. He saw the new bruises on her remaining skin, the new plastic patches over the implants, and decided he really didn’t want to know.

  But about everything else. What she could do, what she couldn’t do. What she wanted and needed, exactly. Oxytocin, specifically: without a natural source her immune system would start rejecting the implants. Drugs could stabilize her system, but they had major side effects. So touch, mainly; sex as an addition. No emotional commitment. She thought they might have a certain intellectual compatibility but she would not have much time to talk. The job. She needed his touch. She would do what she needed for it.

  “Side effects?”

  “Death.”

  It was an incentive of sorts.

  He was of course open to pursue other relationships; she didn’t need to know the details.

  Surprisingly, out of this they had created, he thought, a friendship of sorts. Friendship. It was an odd word, not something he’d associated with women he’d slept with before. They were dates, girlfriends sometimes, but never friends.

  N—she preferred to be called just that, N—was a friend.

  Of sorts.

  Who was now pulling a wire out of her arm.

  “Should you be doing that here?”

  “No.”

  The wire was not coming out cleanly. Drops of blood were falling on his couch. It was a cheap piece of crap, some microfiber thing he’d gotten on sale, and he wouldn’t mind tossing it, but—after a few seconds of debate, he stood up and hunted down an old towel, and returned and put it underneath her arm, to catch the blood.

  “Need help?”

  “No.”

  “Just as well,” he said, trying to make a joke of it. “My fingers are mostly good at going in you, not getting things out of you.”

  He regretted the words as soon as they left his mouth, but she did not seem to react. Then again, she never did, except when they were having sex, when she surprised him by almost seeming as if she meant it.

  She reached into her arm and began pulling at a second wire.

  “Let me help,” he said then.

  “No.”

  “At least let me get you something for it. Alcohol, some—”

  “It doesn’t hurt.”

  “Infection.”

  Two weeks ago, he’d turned on some show or other. She’d been reading her tablet with the intense focus he’d learned not to even try to interrupt; in some ways, he found it flattering that she was willing to do that, do some of her work in front of him—at least the non-high security stuff—but he also found the intensity almost unnerving, like a trance state except not really, and he couldn’t watch it, couldn’t even glance at it. After a few minutes, she’d come over to sit by him on the couch. A few minutes later, she had relaxed against him, not saying anything. He’d wrapped his arms around her. It had been—

  Nice. Normal.

  “My hands are clean.”

  “This place isn’t.”

  She went for medical checkups at least twice per month; some form of computer maintenance at least once a week. Part of that was her job. Part of that was that parts of her were fragile, very fragile. Even the parts that could rip him apart. And hideously expensive. Many of her parts would be recycled, afterwards. Possibly in other bodies. The very thought made him sick. She kissed him after he told her that.

  “For me, then.”

  “If it bothers you, you may leave.”

  “It’s my apartment,” he said.

  She began rolling up the second wire.

  “Isn’t that one of the arms where you still—”

  It was.

  He’d made a point, when they were together, of kissing both, caressing both, fucking both, as if he couldn’t tell. As if they both were real, equal. In a way they were. But one—one arm still had skin. Her skin. He’d run his tongue over it enough to know the difference.

  She could probably break his leg with a single finger. Then play the video that had led up to this moment, every sound, every expression, everything she had seen, downloaded, analyzed, to the police, who would immediately charge him with Violation D.

  He didn’t finish the sentence.

  “Ok. Look. Can you at least give me a reason?”

  “I believe Meteors is on.”

  From time to time she’d let slip bits about her past life. Before. He had not listened much, at first, but gradually a few small pictures began to float in his head, of what she had been. Before. A musician who had wanted her songs to live on, who had apparently given that up—he never knew the reason—for law school. A very ordinary state law school; she’d never had much money. She’d been paying off debts. She would always be paying off debts. He began downloading some of the songs she mentioned and playing them when she arrived. He thought he felt her relax more after this, linger on their kisses a little longer. Sometimes he ordered the viewer to play movies. Her silver-blue eyes—goddamn they look real, unless you’re kissing her you’d probably never notice, or think they’re just contacts—would flicker to the screen, to him, and back again. It was almost a smile.

  “We’re sleeping together. If something happens to you during this, they’re going to ask me about it.”

  You can do better than this, man, one of his friends had said. Maybe, he said. He’d made playlists for her, sending them in casual emails. The sex had gotten better. Way better. It had been amazing this evening, so amazing he’d assumed she might be leaving him or just preparing him for some bad news when she started to pull at the wires.

  She liked astronomy, she’d told him. She liked science fiction. She liked living what had once been science fiction. She could seem perfectly normal when she wanted to. It was part of the point. She loved music.

  He stomped off and opened a bottle of wine, poured out two glasses and returned, offering her one. “If you’re going to do this you should probably be drink.”

  “The nanowire structures prevent any influence from alcohol.”

  “Humor me.”

  “I don’t have time.”

  He drank down both glasses of wine.

  By that time she had accumulated a tiny, neat stack of coiled wires, and was beginning to work on the other arm. The one that wasn’t.

  “Come to bed,” he said.

  “No.”

  “I’m going to bed.”

  She was usually the one to lead them both to bed. She monitored his heart rate—it was automatic, she explained; part of her enhanced senses, part of her training—and probably other things as well. She knew precisely how tired, how stressed he was. How happy he was. How
drained he was. How everything he was. It couldn’t just be the heartbeats; it had to be something else. She knew when he needed to sleep.

  She never knew when he just needed the touch of her skin. Both skins.

  “Sleep well.”

  “Can you just tell me why?”

  “I am trying not to be lost.”

  He’d never taken a picture of her. Never asked for any of the pictures she had of him. Of them. She wasn’t going to change, after all. He would change. He might lose her. He didn’t need pictures of that.

  She had a million images of him, a trillion, saved in her wires.

  “You’re pulling wires out of your arm. How much more lost can you be?”

  “Sleep well.”

  “Fuck you.”

  He went to bed, but not to sleep.

  When he finally got up, hours later, to hunt down coffee, she was still in his living room, sitting quietly on the floor. Her left arm—the one that was entirely, completely, not real, even though it looked like an arm, moved like an arm, felt like an arm—was neatly beside her on the floor, as was her right leg. He had no idea how she was keeping herself seated upright. Five small coils of wire were stacked in front of her in a neat line. She had always been neat. Always. It was one of the things he most hated about her.

  “My neural pattern synapses are failing,” she said.

  “I need coffee.”

  She’d detached her leg and her arm and put it on the floor. He needed a lot more than coffee, but he needed the coffee first.

  “If no one’s arrived in two hours, you may need to call 911.”

  “God, you think?”

  “They have probably already been alerted. There are—warning systems for something like that.”

  “So the fucking military is going to come here.”

  “No,” she said quietly. “Just my employers.”

  Fuck these coffee pods. How the hell was he fucking supposed to get them into the machine before he’d had his coffee. And what the fuck was the deal with having to put in two pods for the damn coffee, three pods for the milk. She always asked for just one milk, but he liked his creamy, milky, sugary. Girly, he told himself, laughing that she took her coffee blacker than his. His fingers were shaking. His whole body was shaking.