by Scott Sibary
“My mind’s always jumping around. Can’t break the habit. Maybe it’s my nature.”
“Can you quiet your mind, let it be absorbed in observing the moment?”
“Not often—pretty much only while meditating. I’m afraid if I sit back and just observe, one voice will dominate and that’ll be my ego, with all its pride and insecurities.”
“I don’t mean the ego; I mean just your quiet awareness doing the observing.”
“Oh, I’m afraid I might lose motivation if my mind becomes too quiet. I have this jaundiced image of becoming a cloned monk, uncreative, not developing. But then I get these results, good and bad.” She slumped back, making the bench groan. “You see, a wandering mind can be useful. I use arguments that pop into my mind to debate questions from different perspectives. It can help with tough problems.”
“How do you do that—gain different perspectives—if it’s all coming from you?”
“It’s not entirely. Can you honestly say there’s a core, fixed you? A true self that can prove to the other internal voices that it’s more genuine and authentic, that it should be the one to lead, to decide?”
“Maybe, although, I haven’t found it yet.” His body vibrated with silent laughter.
“I know you understand this. If you’ve got a headache and you’re also listening to music while eating something, why shouldn’t that inspire several separate thought processes, along with any reflections on those or associated memories? Those are different perspectives.”
“Always moderated by my unconscious system of processing thoughts.”
“By your brain in its current state, yes. But what about the content of thoughts? We’ve all picked up ideas from people we’ve known, or from brilliant minds whose lives ended years or centuries ago, who still speak to us through their music, art, or writings. So the thoughts we play with can be like a dialogue with other minds.”
“OK, Socra-veig.” He gave her a teasing grin. “Go let your head get stuck in the clouds with too many thoughts going on. And no strong monitor to help you focus. Me, I love getting absorbed in just one concentrated flow of activity, like trying to navigate my way past a problem in software design, or dancing around the tennis ball and returning one of your serves for a winner.”
Monitor. The term resonated on different levels. He was right: something like a monitor was needed to help the mind focus. For humans to monitor AI developments, the AI needed an internal monitor to communicate and report to the users. Then there was this new responsibility, to monitor the current behavior of her colleagues, and even of herself. She had barely caught herself before mentioning the issue to Lars.
“Monitor,” she mused. “Sounds like adding one more voice into the mix. So while you think about a monitor, I’ll see if I can ace you again.”
“Forget about results,” he said, shaking his head, “or you won’t get them.”
They’d rinsed off after tennis, yet she could detect the scent from Lars’s body. Looking down, her hair hanging on either side of her face, she watched the first drop of sweat run down her nose. It paused there, swelling, before breaking free to fall on the inside of her upper thigh.
They hadn’t spoken since entering the sauna. The intimate, dimly lit, wood-paneled room quietly wrapped the two naked figures in an almost-suffocating heat. Solveig let her mind roam with worrisome thoughts arising as copiously as the beads of sweat. Like the way the sweat gathered at her chin and nose, swelling to a tickling before letting go, she would shake off one thought and there would soon come another.
The parching air burned in her lungs as she inhaled before asking her question.
“Lars, what’s your prediction? Will the system we’re building further entrench power? I mean, giving more to the countries already most powerful. Might it increase their advantage over the rest of the world? Will it be destabilizing? Or, are they the best countries to lead?”
“It’s a tool that could work in either direction, so I’m not sure. Its advice is supposed to be public. It all depends on how the Council uses it.” He shook his head, casting drops of sweat in both directions. “But ‘best countries’? How would you define that?”
“Oh, yeah, that’s me: needing to define key values. How about saying we measure ‘best’ by the way a society approaches gender issues? A society that promotes equality between the sexes and does not discriminate based on gender preference should be one that generally treats people well, and supports human advancement. So that might be a rough gauge.”
“Yeah, you certainly cover the two of us with that approach.” He laughed.
“How would you define it?”
His eyes aimed sideways towards her as he formulated his words: “I think you might look at how much trust people are able to show each other.”
Catching his expression, she wondered whether her recent behavior had evinced a new mistrust or suspicion of her colleagues. She imagined she could feel every drop of sweat beading on her face, tickling her contentiously. Her fingers rubbed along the surface of the wooden bench, her nails tracking in the grooves of the grain. “And yet they say our generation has an epidemic of narcissism. Doesn’t that mean people will trust less?”
“Oh, come on! You’re talking about an attitude of expecting others to do things for you. Yeah, maybe earlier generations had less of that. I’m talking about trusting people not to do anything against you, and to fulfill promises or legal responsibility.”
“For example?”
Lars rattled off what sounded like a list of irritations: “Like having turnstiles as well as tickets for public transportation. Is it a society where turnstiles are needed? Is it a society that needs to monitor students taking exams? In communication systems, how much does the government or its contractors eavesdrop?”
Solveig continued to work the grain of the wood, tracking in her mind the implications of his examples. “Like here?”
“Pervasive monitoring is an extreme example of lack of trust.”
“Good. We’ve got that clear, Lars. What other examples?”
“How much security protection do people feel they need for their homes? How much dealing is by lengthy contract? In other words, how much are people able to rely on others behaving with the consideration they agree they should? How much can people rely on the expectations others give them? Societies are held together by common beliefs. And those beliefs don’t work unless people have faith in the them. That’s a kind of trust.”
“Trust sounds like an all-or-nothing approach. How about level of confidence? You have more confidence in some people than in others, don’t you?”
“Sure. But you asked me about entire societies. And in aggregate, it will appear as a level of trust, and it says a lot about the society. The more people and systems are able to trust, the less they need to spend tax money on patrolling and incarcerating, or on attacking other societies preemptively. Then more public resources can be applied to making things better.
“And consider this: add up the value of all the resources—time, materials, and lost opportunities—spent on mistrusting. I’ll bet it adds up to way more than half the value of human productivity. That ought to be unnecessary. And without needing to mistrust, people could put more psychological energy into enjoying and fulfilling their lives rather than worrying about their neighbors or strangers. A society based on fear can’t advance.”
“What about those who cheat whenever they believe they can get away with it? Doesn’t a trusting system actually encourage exploitation by sociopaths? Some people are unable to feel guilt or empathy—and there’ve been enough of them, sometimes in high offices, to ruin societies. They’ve led the worst atrocities in history, and it’s scary to imagine we could be creating something like that: a tool without the feelings to restrain its own actions. It’s scary that people could start trusting AI simply because they’ve started to rely on it.”
Lars looked lost in thought as drips of sweat fell off his chin.
�
�Oh, I didn’t mean to scare you,” Solveig said, giving him a wide-eyed look and barely concealed smile. “You look like you’re sweating with fear!”
“Hey, you’re pretty sweaty yourself!” He gestured at her front. “Those are tough questions. Maybe the way a society approaches sociopaths is another indicator of how advanced it is. To the extent that sociopaths are a product of their environment, society played a role in that. And to the extent they’re that way by nature, well, the question becomes, are people well-informed enough to avoid being exploited by sociopaths and to keep them out of power. Maybe in a trusting society, some exploitation is unavoidable. Then we should look at how well a society minimizes that. You’ve got me grasping for ideas. My point is, the harm from trusting must be less than the gigantic costs society incurs by failing to build trust.”
She sighed and the hot air burned in her lungs. “I just hope what we’re creating won’t make these problems worse.”
“It won’t.” Lars grinned. “Trust me!”
Chapter Fourteen
Again, the system has failed. Already May, Solveig thought, and every attempt at a merger has met a roadblock.
The nineteen computer scientists and engineers had gathered in the central meeting room. The Chinese contingent sat at two of the tables, with AnDe standing between them. Sitting with her team at the third table, Solveig contemplated which component of the interface might need changing, as if she didn’t know the search would be fruitless. She ignored the invisible spotlight she felt burning on her.
The Chinese operating system had refused to accept the Protection Lock and vital codes, except in a few instances where it would not follow the imperatives. In those test runs, the AI complied with orders by recommending action that it should have predicted could have unacceptable consequences. The first imperative should have required it to be disobedient, and the AI should have alerted them to the likely problem. When given conflicting orders, it complied with one and violated the other, again without alerting them to the conflict.
No one suggested there were bugs in the Norwegian software. They all knew that it had worked effectively within a less powerful operating system before the mission began. But if the problem lay with the Chinese operating system, then that might indicate the failures were no accident, that the operating system was not designed to allow a successful interface, and that the Chinese government was not ready for the project to succeed. And as the failures continue, Solveig thought, such a grim explanation becomes more likely.
Then there was the other explanation: that someone—someone in that room—was sabotaging the project and causing the failures. One face came to her darkened mind. As the room went quiet she looked up, and was startled to see that face staring directly at her, his expression no longer congenial and patient.
AnDe seemed to be waiting for her attention before addressing the entire assembly. “May I begin by saying how disappointed we are in this most recent failure. After months of cooperation, we have nothing to show those awaiting our success. We begin to wonder if we’re doing all we can, and thinking of all we can, to make our work succeed. I’m not saying we’re not working hard enough, only that we might speak a little more openly on what we think might be causing these failures.” He turned again to the Norwegians. “Your group has been well-informed about our system, but we have much less information about yours. This trial-and-error approach might take more time than we have.”
His team members looked up at him, avoiding confrontational glances at their Norwegian colleagues.
“You know why that is,” Solveig said, speaking slowly. “It was part of the agreement between our governments, and also the understanding of all of those who await an operational WEA. There’s no reason to be secretive about the general aspects of the operating system, but there are good reasons to be secretive about the systems protecting the imperatives.” As she employed his tactic of stating the obvious, her mind spun on his change in mannerism. She glanced at her team and caught a look of What have I been saying? from Per. The others sat stone-faced.
“All we’re asking,” said AnDe, “is that you reconsider which aspects must remain confidential from us and which need not, and what disclosures might make the difference to whether this project continues to fail. Systems being developed in other countries might instead be chosen as the future World Electronic Analyst.”
“Fine,” she said, shrugging her shoulders. “If they do come up with something that works, let them present it as an alternative. The two countries represented here are not trying to gain an advantage over the rest of the world. Or are we?” She aimed at him a saucy glare.
AnDe softened his look. “No,” he said, turning to address the entire assembly. “I hope that never becomes the case.”
As with a crowd hit by a disheartening announcement, no voices projected above the murmur as they broke for lunch. The noise reminded Solveig of the low-pitched rumbling and rattling made by the cooling system for the computer room: sounds that suggested the attachments securing that system were degrading.
Solveig stood up quickly, then took a large sip from her water glass, swishing out the bitter taste in her mouth. She took three more gulps. Then a wave of dizziness caught up with her, like a flood of doubts. It surged around her legs, undermining her footing and threatening to drag her down to an underworld of suspicions. The fingers of her free hand flexed as if reaching for something to grasp, to keep her from descending to a place where frustration would lead to despondency. Her hand found the back of her chair as the physical sensation dissipated and the mental one lingered. She heard approaching steps.
“Solveig?” came a hopeful-sounding feminine voice. The face came into focus as that of Wu Mishuang, or Miriam, as she was called in English. The personification of politeness waited with a patient face.
“Yes, Miriam?” Mishuang stood two steps away, deferential as ever to Solveig as having the higher status. “Not an uplifting discussion, eh?”
Mishuang shook her head, then nodded. “It worries me. I think it worries everyone.” She paused, as if checking Solveig’s reaction. “But we can figure it out; I think we will be successful soon.” Her voice tapered off.
“I’m glad you have determination.”
“Thank you.” Mishuang seemed to hesitate. “If I may, I would like to invite you to go out with me, if you are free on Saturday or Sunday? I thought you and I might enjoy an afternoon together.” She nodded to her own words and then seemed to sag. “I’ve been a little tired and stressed recently. I’d like to do something refreshing. My face is showing it, I think.” She wiped one hand lightly over her face, a hopeful smile appearing from behind. “Would you like to join me for a facial treatment? I know a spa I like very much. They have many services and facilities. We might go for a meal afterwards.”
Two months earlier, Mishuang had invited Solveig for an outing with her husband and son. Solveig had declined as it conflicted with other plans, though she could easily have changed them. No further invitation came until now. Looking at Mishuang’s face, Solveig saw only an enviably clear complexion. She squeezed her own face between the thumb and fingers of her left hand, recalling the deteriorating complexion she’d noticed in the mirror that morning. Stressed; underwater. She remembered Reidar’s advice, from when he confronted her in her apartment. Then she looked again at the woman of about her own age.
“Well, Miriam, that makes two of us. Yeah, I would like to. I think this Sunday might work. I’ll let you know for sure after lunch. And thank you very much for the invitation.”
“Great! I am happy.” Mishuang took one step backwards before turning away.
One can never tell when they’ll magically appear, Solveig said to herself.
At the Wednesday lunch of the Norwegian team only Per acted unperturbed. He seemed to view the confrontation by AnDe as a challenge he was eager to take up. At the end of the meal he renewed his dinner invitation to Solveig. She accepted.
The door to Per’s
apartment stood ajar that evening when Solveig knocked. She heard him chanting to himself the words of an old folksong.
“Come in!” came the boisterous response.
His quarters were noticeably smaller than hers, and his dining area was simply a table within the kitchen. He had two pans cooking on the stove, and the room was filled with the fragrance of potatoes with dill, cabbage steamed with caraway, and, she thought, maybe cheese. Plain as could be, it hit her with a wave of nostalgia that made her sink in her steps.
Then she noticed candles and a bottle of Sancere on the table, and started to wonder.
“You’ve made a lovely table for a weekday supper,” she said.
“Always nice to have a visit from a lady! Besides, it’s your first time to dine here. I know you like wine, so . . . a little splurge after a stressful day.”
“Yeah, I guess I will.”
“I could have ordered from the cafeteria, but they don’t make what I wanted: old-fashioned cooking. I’m not much of a cook, as you can see. Most of what I know I learned from my grandmother one summer I stayed with her. I have to admit, she wasn’t a fancy cook either.” He shrugged. “But I tried to make something colorful with the salad.”
She looked at the bowl containing lettuce and arugula mixed with slices of radicchio, red bell pepper, fennel, and orange. From it rose the aroma of a lemon vinaigrette. “Beautiful.”
“The food is set, so why don’t we sit in the next room with a glass of wine. There are hors d’oeuvres in there.” He lifted the bottle from the table and led her into the sitting room where he’d set out almonds, peanuts, celery, carrots, and quartered-pieces of smørbrød with cheese.
“I know you’re more or less vegan,” he said, “but I also know with cheese you’ll cheat a little.”
“Occasionally, depending.”
His voice sounded as though he was reciting an advertisement. “Both the Danish blue and the Jarlsberg are organic and sustainably produced under certified animal-friendly standards.” He poured two crystal goblets half-full, then looked her in the eyes as he handed her one. “Well, skål to getting through the battle today. You did a superb job of fending off the ambush.”
