by Scott Sibary
Another smile crept over his face, and she said no more.
“Well, are you hungry?” He stood up.
“Yes, I am always hungry.”
The private booths in the dining room were defined by fabric hanging down from the ceiling to drape behind benches on either side of the tables. The floor was carpeted, and sounds and light were softened. Even voices seemed lowered. One was left to focus on the aromas and flavors of the food. And one’s companion.
AnDe narrated facts and stories about the food they were served. Time seemed to ease, slow, and be suspended in the idle comfort of their setting. Then the meal was over, the food remnants taken away, and they were left with just their glasses of wine.
Solveig took a sip from the last tablespoon in her glass, leaving half. AnDe motioned to a waitress, then gestured to the glass.
“No, thank you. I wasn’t even going to have one,” Solveig said.
He said something to the waitress and turned back to Solveig. “I just ordered a bottle of soda water.”
“Thanks.” She tapped her phone sitting on the table then looked up at him. “My translator got that one right.”
He looked so relaxed that another glass of wine might put him to sleep. She let the quiet play and felt herself merging into the cushioned ambience surrounding them. Equanimity poured over her, until the seed for a counter-feeling grew in its wake. She examined him again, then ventured.
“Let me admit something I feel here and now. In fact, it’s what motivates me more than anything else.” AnDe looked attentive. “You’re familiar with the Asilomar Principles?”
“Of course. They make the case for the kind of controls you work on.”
“And are not taken seriously by world leaders.”
“I don’t know. Some governments began to fund research on AI ethics, or agreed to cooperate on the topic.”
“It hasn’t made much difference, until now, and only because of the plague and the Collapse. Designing directives and imperatives is like designing the first rockets. They weren’t familiar with the potential power of the explosive forces, and the parts they were using were designed for non-explosive propulsion, so they didn’t know how their safety devices would hold up if things started to go bad. The point is, we need to develop the safeguards first. When we eventually create superintelligence, we’d better have done it right the first time.”
AnDe seemed to protest. “We always try to get it right the first time.”
“Not just trying; succeeding. There won’t be a second time. Current designs—including yours—strictly program the means for running systems, like managing traffic or utility infrastructure. Yet, by definition, strict programming won’t allow re-examination of the means chosen.”
He looked impatient. “That’s not quite fair. Self-improving systems naturally tend to clarify their goals, or utility functions, and we’ll have an opportunity to monitor the progress of their development.”
“Like a clarification that it needs to protect itself in order to fulfill its objectives? That could blow up in our faces. And a clarification that it should acquire resources? That could destroy us. And I’m talking about superintelligence designed to be beneficial.”
AnDe leaned forward and his words burst out. “So am I.” He glanced around, then seemed restless even when he sat back.
“What about the military AI that keeps getting more powerful? Does it have the goal of being all-powerful? And there’s this competition, and fear, between the militaries and the private market, each seeking to create powerful AGI. Those developers aren’t prioritizing the safeguards that we are!”
AnDe offered her only a tight face with a blurry gaze.
“The threat of dangerous AI is drawing closer,” she said. “But we have an opportunity to create something that has the necessary controls, yet could prove powerful enough to stop malicious AI.”
He raised his eyebrows. “You think you can save humanity?”
“No. We can.”
“What you’re suggesting goes beyond our mandate.”
She heard his foot tapping rapidly on the floor. “It can be accomplished within our project. It must be,” she said.
“I don’t know.” He stared at her briefly, then said, “Excuse me for a moment while I use the restroom.”
What timing. Damn.
When AnDe returned to the table, he was grinning like a happy drunk. “I just don’t know. I should call you ‘Champion.’ You sound so ardent.”
“I believe in what I’m saying.”
He looked down at the table, as if pondering a chess move.
Her breathing went tight. “Sorry for talking so much about the job right now—here. It’s a lovely place.” She cleared her throat.
“Communication is necessary. You just told me you can’t pull off something like this alone.” He looked up at her. “Don’t we then need everyone in the profession, even in society? If everyone has a stake, everyone has a role to play. Working together, not unilaterally.”
“As much as we are prepared to. There’s the question of timing and how much to . . .” Her voice trailed off into silent debate, then came back assertively. “It’s not my practice to make leaps of faith.”
“What if you have no choice?”
“You always have a choice.”
They stared like two wrestlers, separated but watching the moves of the other. Then AnDe gave half a shake of his head. “I will say, I admire your devotion. Do you approach your avocations the same way?”
“Project Manager Deng, I sure can’t predict your next move—I mean, direction in the conversation.”
“Maybe you can. I started asking you about fiction. What authors do you like?”
“Well . . .” She contemplated the chimerical figure across from her. “Do you know Virginia Woolf? Quite different from Asimov.”
His face went blank as he leaned back and looked again at the ceiling. “No, I don’t know her, only the name from a film. Maybe she was a character played by Elizabeth Taylor in a very old movie. I love old movies.”
Solveig let out a laugh and marked the common interest for when she needed a safe topic. “No, that wasn’t about Virginia Woolf, unless you’re confusing Taylor with Nicole Kidman in The Hours. Woolf’s writing is full of metaphors, allegories, and sometimes moves like a stream of consciousness. I find it stimulating.” She paused, then concluded abruptly, “It’s difficult to explain until you try it.”
“Perhaps I’ll try one of her stories. I’m always up for a challenge.”
“They can be slow going, so try one of her shorter works. Otherwise, it could take you years.”
His face replied with curiosity, and she added, “One of her longer works is titled The Years.”
She sighed, wondering whether her stay in Beijing would be counted in years, and whether she’d go the way of Virginia Woolf if it did. She noticed him watching her closely.
“Let’s see if we can accomplish this project in less than that,” he said.
They stood outside the entrance to the residential compound, talking about diplomatic etiquette. AnDe suggested dispensing with the use of titles between the two of them.
“Yes, of course, I would be honored. I’m quite used to going by first name. So, good night, AnDe.”
“Good night, Solveig.” He waited beside the car while she walked to the entrance of the residential compound.
Her gait had eased from when she’d first appeared in the tight dress. She walked through the security port and to the entrance of her building, then glanced back. AnDe was still watching as she let the door to the building swing shut behind her.
In the hallway she sensed a phantom lurking, as if waiting to tackle her in private. Her feet slowed to a tentative, wary pace, letting the sound of her footsteps be muffled by the carpet. She slipped through her doorway with the quiet of a ghost. Then with a foreboding click, her apartment door closed behind her, and she turned to find the menacing figment in her foyer. It
stood faceless as a telepathic robot, and it was signaling to her brain how dangerous it could be.
What if they got their way, those politicians who wanted an AGI that could fake its true nature? And what if it were superior to human intelligence?
She trembled before the specter of a monster she might help create, another premonition that might come to be. She waved an arm at the light controls, brightening the room and dispelling the phantom. She could reject it as only an hypothesis, yet it was close enough to reality. She took slow, deep breaths, feeling the tightness of the silk fabric around her chest. She must do whatever she could to end the threat. And, realizing that, she resolved she could never cease to engage in the development of new intelligence.
Now you’ve gone and opened up to him, she told herself. You said more than you needed to, without first knowing his thinking! Be more careful. Don’t rely on someone else unless it’s necessary. And that’s what you need to figure out.
Chapter Eleven
Her nighttime journey had led her to a world filled with human-sized electronic towers: servers that made no sound other than a faint, cold hum. She found herself working alone in a place where humans seemed obsolete. A meaningless world.
Solveig strained her eyes, rebelling against her undirected imagination. The clock on her bamboo nightstand read one minute before the usual six o’clock alarm. She flipped back the duvet covering her bed and shuddered as she lost the enfolding pretense of companionship. She drew away from the pillow tucked behind her lumbar and the one that cradled her head. Feathers packed into clouds of bleached cotton might offer comfort and protection while flying in the dream world. No need for them now.
Yet her thoughts while lying awake in the middle of the night could be more piercing than any nightmare. Meanings, confusion, and doubts about the nature of existence. Such neuroses would never gain hold during the bright hours of the day but were alive and menacing during the dark hours of the soul. In those moments, facile conclusions were a useless defense against undeniable mortality. To get out of bed cost too much sleep and dulled her mind the following day. And to what end? Like the dark universe not visible behind the bright daylight, plaguing thoughts could come again in a matter of hours.
A bottle of sleeping pills with one tablet sitting beside it stood near the far edge of the nightstand. With a smirk she dropped the pill back into the bottle and the bottle into the nightstand drawer. Another night accomplished without a crutch.
She sat on the edge of the bed, contemplating the prospect of another Saturday at her desk, in her apartment, in the compound, working. She could call Lars, and they might again play tennis, take a sauna, have a drink, watch a movie, or just talk. But going alone beyond the compound was not an option. The protocol for her as leader required that she not go out unescorted. She could have a local escort, whether an approved professional guide or someone sent by the Chinese Foreign Ministry. Or she could go with a colleague, even a Chinese colleague, as she had to dinner with AnDe two weeks earlier.
Then she remembered today’s arrangement. AnDe was to pick her up at nine o’clock to take her to the Forbidden City. She sprang to her bedroom window. The chill and grey of the March morning would soon vanish, and she would feel the drugging sunshine of premature spring.
Six minutes before nine, she received a message from AnDe that he was in an agency shuttle and about to arrive.
She exited the building and found him waiting for her at the outer gate. From there, he escorted her to the car.
“You should be proud of me,” he said.
Human intentions: not as easy to decipher as computer algorithms. Does he actually mean I should feel pride in him? How presumptuous. Oh, forget it, Solveig.
Watching her contemplate, he said, “Time’s up! I’ll tell you: I’m reading To the Lighthouse.”
She raised her eyebrows. “You are tackling—I mean, reading—Virginia Woolf? Find it challenging?”
His words sang with humor: “Easier than our project.” Then more seriously, he said, “But I’ll admit I’m totally lost.”
I’m proud all right, she thought, but of myself. “Then you’re making a good start at finding your way. I think that’s the theme of the story.” She poked him gently in the shoulder. Then she caught herself, a quivering of surprise shooting through her. Don’t be dismissive of your responsibilities, came the neurotic voice, and her muscles tightened in a wave from head to foot.
Approaching the car, she saw a reflection of a conflicted mind in the dark window glass of the door. With her head dimly portrayed against the backdrop of a bright sky, her changing emotions flew like clouds across her face and revealed the darkness of a subconscious rarely perceived in daylight.
As she paused, AnDe reached in front of her and swung the door open. Solveig’s shadow-self flew away, pursued by the reflection of the man she was with. And then the dark interior, awaiting her acquiescence.
Good lord, said the sporting voice, let the day unfold on its own. She slid into the rear seat, and AnDe got in beside her.
The shuffling foot traffic slowed to a clot at the ticketing gate. A chilly gust blew in their faces and helped her breathe. Her hair lifted off her shoulders, blowing back. She began to reach into her pocket for an elastic hair band, but her elbow bumped the person to her right. She let her hair go.
They approached the outer gateway of the Forbidden City. Solveig stopped to take in the massive cliff-like wall with its dwarfed opening. AnDe walked on for several paces before turning back to look for her. He presented a quixotic figure: the innocuous expression, the pale-green silk scarf tied in a loose knot around his neck, the bright maroon shoes that had just come into fashion for single men, and the overly relaxed posture. She caught up with him.
As they strolled around the monuments of power, AnDe escorted her in whichever direction her interest strayed. At times, her sudden turn to investigate something would send him dancing around other tourists just to catch up with her. Despite her mass-produced attire, she presumed the role of foreign dignitary, mostly observing, occasionally inquiring, and usually listening. Hours passed.
Eventually, they found themselves three stories up on the west side of the Meridian Gate. To the south was the main moat and two more gates before reaching the modern city. Turning to the north, across the largest courtyard in the complex, they faced the Gate of Supreme Harmony, where honor guards had paraded in obeisance to the imperial family and its ingrained system of privilege.
Whose supreme harmony? Solveig asked silently.
Gazing at the same scene, AnDe sounded elated by the formerly excluded commoners milling about within the palace complex. He suggested their presence showed a movement towards greater harmony.
Solveig turned away.
He probed. “Big, eh? The palace, and the city?”
“It overwhelms me, despite my travels. I suppose you’re used to this. Maybe you see an irony in my being here.” She pointed her chin at him, lips closed.
His face looked quizzical.
“It must seem strange, out of balance to you,” she said, “working on this great project with someone from a tiny country.”
“Hmm,” he muttered, his gaze moving over the crowds and congestion, and on to the giant city. “Not as amazing as a population of fewer than five hundred producing one of the talents of our project.”
“Oh?”
“My elementary school produced me!”
She conceded a diplomatic chuckle.
He extended his arms wide. “It’s simply a matter of drawing lines where you want to.”
“Very clever.” She dismissed his joke with a shake of her head. “I was thinking about this architecture.” She gestured to the Gate of Supreme Harmony and the Hall of Supreme Harmony beyond. “A society that could erect such buildings should structure itself to be stable for as long as the buildings themselves. But they didn’t, did they?”
“In China, we’re used to thinking in terms of cycles. Dynasties r
ise and eventually fall. Should we expect that to correlate with the durability of the building technology?”
“I suppose not.” She examined the buildings again. “In Europe, the picture isn’t as clear. There were families that rose and fell out of power in less than a century, and others that lasted much longer. The buildings of the Roman Empire and the Roman Catholic Church stood like proof of long-lasting institutions. But their reigns were far more limited.”
“I want to visit some of the great cathedrals,” he said, sounding jovial. “Holographic theaters don’t create the full experience.”
“Well, from my viewpoint, the churches have also been instruments for control, and an example of the direction we’re trying to avoid.”
“We’re not creating a spiritual institution.”
“I meant when the churches have been involved in political or social policy.” She threw up her hands. “When I was in high school, I had to write a book report for my history and society class. I chose The Name of the Rose. The book was long, and I got lazy and simply watched the movie. My paper was returned with the comment, ‘Resubmit after reading the book.’
“I was ashamed. I sat down to read, starting from page one. I’d thought it was a medieval murder mystery, only, the narrator begins by saying he’d just returned from Prague in the spring of 1968! Luckily, something needled me to research that, and the theme of institutional control over freedom of thought and access to ideas opened up for me. It’s a theme I’ve come to take very personally.”
“And you don’t want to be an agent of thought control.”
“Of course not.”
“Your story, you know, is really funny.” His laugh received a fiery look. “No, listen. I had a similar experience. In my foreign history class, I got to choose almost any book to write a paper about. I searched for something very short, and found a tiny book on the history of the Catholic church in the Middle Ages. I was happy until I got into it. Every sentence had to be read carefully. It was extremely slow going, even with the translator. I became irritated and wrote that its theme might apply to our government. You see, as far as I understood it, it said the success of the church depended on how well it balanced competing needs: the conservative need to maintain values against the need for radical ideas to spur innovation. My teacher wrote, “Very interesting. Now apply this to yourself.”
