Conception: Book One of Human Dilemma

by Scott Sibary

  “Your crew, hmm . . . You said you’re like a captain, and Rolv like the executive officer of a ship; do you see it as a merchant or a naval ship?” Jaw set, he fixed on her face without staring into her eyes.

  “I’m sorry, I mixed my metaphors. By captain I did not mean of a ship, but of a sports team, like a football team.”

  “Ah. Yes. And I could say that I see myself as the head coach of my team rather than as captain. I’m not a star player.”

  Oh, you’re very much a player, she thought. “But yes, Rolv is a lot like an officer on a merchant ship. Or maybe a little fishing boat with a crew of six? He likes to look very serious, the way he furrows his brows, but he’s often teasing.”

  “I can see that, and I really like it. You are fortunate to have someone with his humor and his experience!” He grinned widely.

  The hand on her lap clenched while she showed him a blank expression. And what are you implying, she asked silently, about my skills?

  At their fourth Friday lunch in the same private banquet room, a trickier subject was brewing in Solveig’s mind. She considered her timing. Four small platters of food steamed on the table, most of the aromas unfamiliar to her. She sampled two of the dishes, then ventured into the subject troubling her.

  “Project Manager Deng, please bear with me for a few minutes while I explain how I view our situation. You know we must prevent human users from fooling around with the imperatives in the vital codes. We must also prevent an AI, as it self-improves, from making changes to the core ideas in the imperatives. The strides you’ve made here in endowing AI with self-programming abilities are impressive. The unfortunate consequence is that they all too readily conflict with my Protection Lock. It makes meshing the two systems more difficult. The going could be slow.”

  “We understand.”

  “The primary imperative of altruism cannot work as a single broad statement with clarifications. Instead, it must be a process that requires an AI to think and behave in compliance with numerous guidelines on the essence of what humans value. Since there’s nowhere near complete agreement about values, the analyses it offers to users might be limited, and laden with qualifications. The WEA could turn out to be of little use to human policy makers. And if it’s given even limited operational control over any system—like utilities, transportation, communication—it might jam up when presented with conflicts that human managers routinely work through.

  “As I understand it, your work on vital codes has prioritized preventing collateral damage from blindly pursuing an objective. Your work seems modeled on how to keep humans from making that kind of mistake. We go beyond. We provide a mechanism for addressing value conflicts. What’s important to achieve in one culture might be important to avoid in another.”

  “For example?”

  “Play in your mind the values of tolerance and cultural diversity against the countervailing need to protect cultural integrity and continuity. A culture lives on only if those living in a community or society adhere to its norms. So then, how does one define the boundaries where cultures change? What about communities where there’s a mosaic of ideas and practices? The United States and Canada, for example. Then there’s the tendency for even those places to meld over time into more or less uniform cultures with their own traditions.

  “And what about respecting human rights? There’s no general agreement on what these should include. We create access to works of philosophy and religion, and—don’t hold your breath—even academic treatises.”

  “Most of what humans naturally want and value can be figured out scientifically,” AnDe said.

  “What? You think it’s that simple? Are you aware that by law in some nations, corporations are considered persons, and entitled to human rights? How is an AI going to deal with that?”

  “The greater the intelligence, the tougher the problems it can solve. We enhance intelligence by increasing its ability to optimize among available choices.”

  “That only highlights the difficulty.”

  “The AGI considers optimization across several domains: different contexts with different variables.”

  “Of course,” she said. “And when contexts suggest different conclusions, an AI will get stuck.”

  “People usually go back to relying on the highest principle. So could an AGI.”

  “What makes it the highest? In our approach, the solution proposed by the AI must lie within the boundaries we set. It’s similar to playing with terminology.”

  AnDe didn’t show much reaction, so she ventured further. “Meanwhile, you call this new system ‘general,’ but I wouldn’t.”

  “Why not?”

  “It lacks a system that integrates the machine’s many different narrow AIs into one coherent mind. You created a simpler system: one that merely coordinates them and chooses which single, narrow AI capability to use for analysis, adding in data and suggestions from other AI sources.”

  “Which means it’s not narrow.”

  “It lacks true judgment. It seems to me that it identifies simple attributes of the problem posed to it to select which narrow AI component to employ. I think the coordinating algorithms need to be problem-solving and learning-capable, and take over much of the function of what is currently left to the narrow AI components.”

  His voice suggested curiosity. “You worked on such systems in India, didn’t you?”

  “Yes. It’s something I might return to: making a genuine AGI.”

  “What would you call ours?”

  “I suppose one could call it, quasi-general AI.”

  “Hmmm. I can accept that, just between us. But officially, our project is referring to it as ‘general.’ OK?”

  She shrugged. “OK.”

  AnDe sighed and looked up at the ceiling. “Maybe it’s part of the cultural differences you mentioned.”

  “That’s another assignment I give myself: learning new perspectives, because I never know what route my unconscious mind will take to come up with an answer. I’ll tell you, in India I read about the country’s religions, history, etc., to get an introduction to what I’d called ‘Eastern thinking.’ I thought it might help me understand how their engineers approached problem-solving. This must sound naïve to you, but at first I didn’t understand that Chinese and Indian perspectives could be so different from each other.”

  She offered an apologetic smile, but saw only a distant expression on his face. “I’m sorry if I’m boring you.”

  “Not at all.” He perked up. “Yes, very different. Buddhism was brought from India and Nepal about 300 BC, and we traded goods even earlier, but that hasn’t drawn us together.”

  “You said ‘BC’ rather than ‘BCE.’ You know that means ‘before Christ’? Did you use that because I come from a traditionally Christian country?”

  “I’ve noticed your English is better than mine. And you pick up quickly on terminology. But consider this: what makes it a ‘common era’ in BCE and CE? Only Jesus. That’s a strong value judgment . . . Jesus as something the whole world has in common. I guess some people want us all to think that way. ‘Before Christ’ is a fact. The choice of a reference point reflects a cultural bias, yet it’s a data point anyone can refer to.” His voice switched from serious to jovial and his eyes twinkled. “First I thought ‘AD’ meant ‘after death.’ Now I use it because it reminds me of my initials!” He laughed freely. “But you were talking about India, the new leader of the English-speaking world.”

  “Seriously? You can’t mean that.” He makes a joke out of a possible disagreement, she thought, then throws at me a shocking viewpoint.

  “The US government has lost the credibility advantage it used to have.”

  She tilted her head and gave him an examining look. “As I see it, the US military is still in many ways most powerful. And their economy is recuperating rapidly. India seems separate and much weaker to me.”

  “The Collapse did little damage to the Indian economy,” he said.


  Solveig pictured a student of martial arts making challenges at the master and being parried with seemingly no effort or movement. Eventually, the master spins the energy of a challenge back onto the student, who finds herself tumbling to the mat.

  “What about,” she said, her voice not hiding the challenge, “the international team that is supposed to inspect and approve our product before it can be formally selected as the World Electronic Analyst? Almost half of them are from the US, and they have credibility. How do you explain that? If they don’t approve our system, then systems from other countries will be considered. And the US has three.”

  “Which aren’t quite ready. The Indian version is.”

  “It’s not a true AGI either, and it was a premature release, with unresolved problems. I know.” Or instead of martial arts, is he more the artist of illusions? she wondered.

  “The Indian diaspora live throughout the English-speaking world and must have an influence. Maybe it’s a different way to view leadership?”

  Leadership again. She wondered if he felt threatened by them, and if underneath he was pursuing an agenda of control. She could not make her part succeed, or make the mission successful, on her own. He must believe as well. She searched for less disheartening interpretations. Perhaps he was afraid the Indian AI would ultimately prove more powerful than the one being developed in China. Maybe that was why the Chinese were eager to have her.

  AnDe tilted his head to the side and raised his eyebrows, as if to let her talk.

  I might as well bring it up now, she thought. “From my understanding of Chinese society, it seems there’s a contrast between the patience people show when they need to wait for something that’s not under their control, and the hurried approach I see when people are anxious to get a job done. You don’t even mind calls about work in the middle of the night.”

  “Yes, we value both patience and hard work. Don’t you?”

  “Yes,” she replied, “but hard doesn’t always mean fast. The quality must be there, or there’s no point in doing the job. Too many technological developments have been hurriedly implemented. Right?”

  He showed no reaction.

  “With nuclear power and fracking, obvious problems were disregarded so they could start selling the product. Our motivations are different, but my point is the same. The unaddressed issues will come back to haunt us—haunt everyone.”

  “Excuse me, but are you saying you think we’re rushing to the point of sacrificing quality?” His voice carried a higher pitch.

  Solveig tightened her grip on her teacup, the way someone under assault might grasp for something. She caught his eyes noticing. “If rushing undermines Norwegian control of the Protection Lock, yes. The eventual World Electronic Analyst has to be a system that world leaders can trust. If they see it as an entirely Chinese product, operated under Chinese influence, they won’t do that, will they?” Her fingers pulled away from the cup.

  He answered slowly, as if moving his words through a mine field. “This is a tricky subject. I believe most within our government want to promote long-term world stability.”

  “Meaning?”

  “World Council members will have to be cooperative with each other, especially if they are to control operational programs the way many treaty organizations currently do. If any one country or bloc dominates, then the World Council will be a failure and we might get a return to the violent cycles of history. So we’re not trying to rush the project or dominate the end product. And we’re serious about trying to make your approach work. We just don’t know that it will. And we don’t have unlimited time to find out.”

  “It’s done well in our preliminary testing. Although I’ll admit, our operating system is simpler.”

  “Good. We are committed to trying it out.” He raised his teacup and took a slow sip, as though it needed to be savored. “So where are you in your reading?”

  “Last night I finished an overview of all the dynasties since the Han.”

  “I’m impressed! You could be relaxing with your colleagues or catching up with friends and family. You honor us with your efforts.”

  He sounds like he means it, she thought. But he is part diplomat.

  “Except for staying in touch with my mother and a very few relatives and friends, I’m pretty much a loner. I take my work seriously.”

  “Clearly. So let me help with your cultural study.” He spoke in a warmer tone. “We have a great variety of cuisines in this country, maybe more than you suspect. The food says much about the region it comes from: the climate, the geography, the people.” He gestured at the platters, making faces to convey his opinions. “I noticed you like the Sichuan peppers. You even ate some separately, like you were curious. They’re supposed to help one survive the summer heat. Too spicy for me!”

  “You’d get used to it after two years in India!” She rocked her head sideways in a manner she’d picked up while living in Bangalore—a handy habit for softening one’s words.

  He smiled at the gesture. “Now you want to learn about China, which means we must have more time to do this seriously. Only at a restaurant with regional cuisine will you experience the context of the food. Its place of origin will come to life. How about next time we have dinner? It could serve as your evening’s study.” His tone seemed to plead.

  She curled up her mouth on one side. “OK.” She nodded and thought, and I’ll remember Stig’s warning.

  The residential compound that Solveig and her team had moved into spread, like a small university campus, over four city blocks. It contained extensive facilities for foreigners who might wish to live and work as securely as possible from exterior monitoring: housing, conference rooms for meetings and recreation, a gym and outdoor athletic facilities, a cafeteria and restaurant, and a few shops. For Solveig it felt comfortable enough.

  That evening she spent reading and checking on blogs and comments about the mission. Her eyes were drawn to negative comments like flies to feces. “Fools” was used to label her team in one comment, “traitors” in another. And her own name was targeted.

  “If Kleiveland is as bad a mission leader as she was an instructor, we should all kiss our asses goodbye!”

  And another: “True genius is rare. Pretended genius is common. If she had a real brain, she’d learn to just shut up!”

  She searched for more new references to her personally. Many came up, some negative, and the occasional one just as nasty. Her vision clouded dark, and she closed her eyes, recalling phrases from an underworld of anonymous comments: “Tall doesn’t mean pretty. Coy doesn’t mean precious. Fast and strong don’t mean virtuous. In her case, it all spells ‘fraud.’ ” The memories surrounded her like a posse of ghoulish witches. With a screeching cackle they closed in. “That iceberg needs a good humping to shoot some warmth into her. Maybe she’d even calve, though I doubt it, ha! ha!”

  Gasping, she opened her eyes and changed the screen.

  Her heart still racing, she read over the summary notes for the week, and then focused on arranging the issues that lay before her team. Eventually, she felt drowsy.

  In bed, hoping to glide into sleep, she dropped her guard . . . and the questions pounced. She writhed like a dolphin caught in a tuna net. The net closed in on her, tightening around her wrists, her ankles, her waist, and her neck, slowly, surely.

  Just lie here and do nothing, and you’ll be trapped in the next cataclysm along with all the other victims. You agreed to participate, to be involved. Even to lead. How could you refuse? Not after living through that human-caused holocaust. It’s your responsibility, damn it!

  Her fists broke free and shook in the dim light.

  What if there is no way of knowing what this will lead to?

  She rolled out of bed and stood, legs apart. She raised her arms and leaned slowly over to one side, then to the other. She tried this and other standing yoga poses. She tried to meditate, but still the monkey-mind bounded around the room, taunting her.


  Holy shit! A bureaucrat: that’s what he is! Someone not sharp enough to invent, to create, to ideate. Someone steady and amiable, who follows his orders without questioning. Someone who won’t rock the boat, or the warship. Her silent words rolled out with a lilt to mock her naïve behavior at lunch.

  He just sits and listens, lets you show your cards, hoping you’ll reveal your secrets.

  Still standing, she pulled her legs together and raised a pointing arm.

  He throws you off course with his off-beat comments and solicitous offers. I’m not fooled. Better not sail his ship across my gun sights!

  Her gesticulating finger fell to her side.

  Oh, damn it! We’re all on that ship.

  She dropped to the carpet, crossed her legs and stretched, twisting her torso around to one side and then to the other.

  No, Stig was wrong. Deng AnDe is no Casanova. And no, not a dull bureaucrat. He’s more complex, has more faces than he’s showing. Maybe hidden genius. But what if there’s more to Stig’s parallel than simply the amorous exploits? The historical Casanova: wasn’t he also famous as a financial schemer, promoter of a lottery? That devious way to allocate resources! And wasn’t he supposed to be a genius at cracking puzzles and escaping traps?

  Chapter Eight

  Solveig continued on her rectangular path, one deliberate step after another. Within the sharp lines and smooth surfaces of the box-shaped conference room sat five more humans, leaning back or slumping. From the surface of their work table rose a small city of screens, keyboards, touchpads, and personal devices. A few pens and sheets of paper landscaped the open areas. All so that real minds might have their commands to the artificial translated into a binary logic.

  “What I’m trying to say is this.” She stopped pacing when she saw she had their full attention. She aimed her gaze at the large electronic chalkboard on the back wall and over the tops of the five heads turned towards her. “It needs to be accepted as a sealed box.” Her raised hand waved in a circling gesture at the walls.