Rationality- From AI to Zombies

Home > Science > Rationality- From AI to Zombies > Page 148
Rationality- From AI to Zombies Page 148

by Eliezer Yudkowsky


  Tsuyoku naritai is more stressful than being content with who you are. Isshokenmei calls on your willpower for a convulsive output of conventional strength. “Make an extraordinary effort” demands that you think; it puts you in situations where you may not know what to do next, unsure of whether you’re doing the right thing. But “Shut up and do the impossible” represents an even higher octave of the same thing, and its cost to its employer is correspondingly greater.

  Before you the terrible blank wall stretches up and up and up, unimaginably far out of reach. And there is also the need to solve it, really solve it, not “try your best.” Both awarenesses in the mind at once, simultaneously, and the tension between. All the reasons you can’t win. All the reasons you have to. Your intent to solve the problem. Your extrapolation that every technique you know will fail. So you tune yourself to the highest pitch you can reach. Reject all cheap ways out. And then, like walking through concrete, start to move forward.

  I try not to dwell too much on the drama of such things. By all means, if you can diminish the cost of that tension to yourself, you should do so. There is nothing heroic about making an effort that is the slightest bit more heroic than it has to be. If there really is a cheap shortcut, I suppose you could take it. But I have yet to find a cheap way out of any impossibility I have undertaken.

  There were three more AI-Box experiments besides the ones described on the linked page, which I never got around to adding in. People started offering me thousands of dollars as stakes—“I’ll pay you $5,000 if you can convince me to let you out of the box.” They didn’t seem sincerely convinced that not even a transhuman AI could make them let it out—they were just curious—but I was tempted by the money. So, after investigating to make sure they could afford to lose it, I played another three AI-Box experiments. I won the first, and then lost the next two. And then I called a halt to it. I didn’t like the person I turned into when I started to lose.

  I put forth a desperate effort, and lost anyway. It hurt—both the losing, and the desperation. It wrecked me for that day and the day afterward.

  I’m a sore loser. I don’t know if I’d call that a “strength,” but it’s one of the things that drives me to keep at impossible problems.

  But you can lose. It’s allowed to happen. Never forget that, or why are you bothering to try so hard? Losing hurts, if it’s a loss you can survive. And you’ve wasted time, and perhaps other resources.

  “Shut up and do the impossible” should be reserved for very special occasions. You can lose, and it will hurt. You have been warned.

  . . . but it’s only at this level that adult problems begin to come into sight.

  *

  311

  Final Words

  Sunlight enriched air already alive with curiosity, as dawn rose on Brennan and his fellow students in the place to which Jeffreyssai had summoned them.

  They sat there and waited, the five, at the top of the great glassy crag that was sometimes called Mount Mirror, sometimes Mount Monastery, and more often simply left unnamed. The high top and peak of the mountain, from which you could see all the lands below and seas beyond.

  (Well, not all the lands below, nor seas beyond. So far as anyone knew, there was no place in the world from which all the world was visible; nor, equivalently, any kind of vision that would see through all obstacle-horizons. In the end it was the top only of one particular mountain: there were other peaks, and from their tops you would see other lands below; even though, in the end, it was all a single world.)

  “What do you think comes next?” said Hiriwa. Her eyes were bright, and she gazed to the far horizons like a lord.

  Taji shrugged, though his own eyes were alive with anticipation. “Jeffreyssai’s last lesson doesn’t have any obvious sequel that I can think of. In fact, I think we’ve learned just about everything that I knew the beisutsukai masters knew. What’s left, then—”

  “Are the real secrets,” Yin completed the thought.

  Hiriwa and Taji and Yin shared a grin, among themselves.

  Styrlyn wasn’t smiling. Brennan suspected rather strongly that Styrlyn was older than he had admitted.

  Brennan wasn’t smiling either. He might be young, but he kept high company, and had witnesssed some of what went on behind the curtains of the world. Secrets had their price, always; that was the barrier that made them secrets. And Brennan thought he had a good idea of what this price might be.

  There was a cough from behind them, at a moment when they had all happened to be looking in any other direction but that one.

  As one, their heads turned.

  Jeffreyssai stood there, in a casual robe that looked more like very glassy glass than any proper sort of mirrorweave.

  Jeffreyssai stood there and looked at them, a strange abiding sorrow in those inscrutable ancient eyes.

  “Sen . . . sei,” Taji started, faltering as that bright anticipation stumbled over Jeffreyssai’s return look. “What’s next?”

  “Nothing,” Jeffreyssai said abruptly. “You’re finished. It’s done.”

  Hiriwa, Taji, and Yin all blinked, a perfect synchronized gesture of shock. Then, before their expressions could turn to outrage and objections—

  “Don’t,” Jeffreyssai said. There was real pain in it. “Believe me, it hurts me more than it hurts you.” He might have been looking at them; or at something far away, or long ago. “I don’t know exactly what roads may lie before you—but yes, I know you’re not ready. I know I’m sending you out unprepared. I know that everything I taught you is incomplete. That what I said is not what you heard. I know that I left out the one most important thing. That the rhythm at the center of everything is missing and astray. I know that you will harm yourself in the course of trying to use what I taught; so that I, personally, will have shaped, in some fashion unknown to me, the very knife that will cut you . . .

  “. . . that’s the hell of being a teacher, you see,” Jeffreyssai said. Something grim flickered in his expression. “Nonetheless, you’re done. Finished, for now. What lies between you and mastery is not another classroom. We are fortunate, or perhaps not fortunate, that the road to power does not wend only through lecture halls. Or the quest would be boring to the bitter end. Still, I cannot teach you; and so it is a moot point whether I would. There is no master here whose art is all inherited. Even the beisutsukai have never discovered how to teach certain things; it is possible that such an event has been prohibited. And so you can only arrive at mastery by using to the fullest the techniques you have already learned, facing challenges and apprehending them, mastering the tools you have been taught until they shatter in your hands—”

  Jeffreyssai’s eyes were hard, as though steeled in acceptance of unwelcome news.

  “—and you are left in the midst of wreckage absolute. That is where I, your teacher, am sending you. You are not beisutsukai masters. I cannot create masters. I cannot even come close. Go, then, and fail.”

  “But—” said Yin, and stopped herself.

  “Speak,” said Jeffreyssai.

  “But then why,” she said helplessly, “why teach us anything in the first place?”

  Brennan’s eyelids flickered just the tiniest amount.

  It was enough for Jeffreyssai. “Answer her, Brennan, if you think you know.”

  “Because,” Brennan said, “if we were not taught, there would be no chance at all of our becoming masters.”

  “Even so,” said Jeffreyssai. “If you were not taught—then when you failed, you might simply think you had reached the limits of Reason itself. You would be discouraged and bitter amid the wreckage. You might not even realize when you had failed. No; you have been shaped into something that may emerge from the wreckage of your past self, determined to remake your art. And then you will remember much that will help you. If you had not been taught, your chances would be—less.” His gaze passed over the group. “It should be obvious, but understand that the moment of your crisis cannot be provoked artific
ially. To teach you something, the catastrophe must come as a surprise.”

  Brennan made the gesture with his hand that indicated a question; and Jeffreyssai nodded in reply.

  “Is this the only way in which Bayesian masters come to be, sensei?”

  “I do not know,” said Jeffreyssai, from which the overall state of the evidence was obvious enough. “But I doubt there would ever be a road that leads only through the monastery. We are the heirs in this world of mystics as well as scientists, just as the Competitive Conspiracy inherits from chess players alongside cagefighters. We have turned our impulses to more constructive uses—but we must still stay on our guard against old failure modes.”

  Jeffreyssai took a breath. “Three flaws above all are common among the beisutsukai. The first flaw is to look just the slightest bit harder for flaws in arguments whose conclusions you would rather not accept. If you cannot contain this aspect of yourself then every flaw you know how to detect will make you that much stupider. This is the challenge that determines whether you possess the art or its opposite: intelligence, to be useful, must be used for something other than defeating itself.

  “The second flaw is cleverness. To invent great complicated plans and great complicated theories and great complicated arguments—or even, perhaps, plans and theories and arguments which are commended too much by their elegance and too little by their realism. There is a widespread saying which runs: ‘The vulnerability of the beisutsukai is well-known; they are prone to be too clever.’ Your enemies will know this saying, if they know you for a beisutsukai, so you had best remember it also. And you may think to yourself: ‘But if I could never try anything clever or elegant, would my life even be worth living?’ This is why cleverness is still our chief vulnerability even after its being well-known, like offering a Competitor a challenge that seems fair, or tempting a Bard with drama.

  “The third flaw is underconfidence, modesty, humility. You have learned so much of flaws, some of them impossible to fix, that you may think that the rule of wisdom is to confess your own inability. You may question yourself so much, without resolution or testing, that you lose your will to carry on in the Art. You may refuse to decide, pending further evidence, when a decision is necessary; you may take advice you should not take. Jaded cynicism and sage despair are less fashionable than once they were, but you may still be tempted by them. Or you may simply—lose momentum.”

  Jeffreyssai fell silent then.

  He looked from each of them, one to the other, with quiet intensity.

  And said at last, “Those are my final words to you. If and when we meet next, you and I—if and when you return to this place, Brennan, or Hiriwa, or Taji, or Yin, or Styrlyn—I will no longer be your teacher.”

  And Jeffreyssai turned and walked swiftly away, heading back toward the glassy tunnel that had emitted him.

  Even Brennan was shocked. For a moment they were all speechless.

  Then—

  “Wait!” cried Hiriwa. “What about our final words to you? I never said—”

  “I will tell you what my sensei told me,” Jeffreyssai’s voice came back as he disappeared. “You can thank me after you return, if you return. One of you at least seems likely to come back.”

  “No, wait, I—” Hiriwa fell silent. In the mirrored tunnel, the fractured reflections of Jeffreyssai were already fading. She shook her head. “Never . . . mind, then.”

  There was a brief, uncomfortable silence, as the five of them looked at each other.

  “Good heavens,” Taji said finally. “Even the Bardic Conspiracy wouldn’t try for that much drama.”

  Yin suddenly laughed. “Oh, this was nothing. You should have seen my send-off when I left Diamond Sea University.” She smiled. “I’ll tell you about it sometime—if you’re interested.”

  Taji coughed. “I suppose I should go back and . . . pack my things . . .”

  “I’m already packed,” Brennan said. He smiled, ever so slightly, when the other three turned to look at him.

  “Really?” Taji asked. “What was the clue?”

  Brennan shrugged with careful carelessness. “Beyond a certain point, it is futile to inquire how a beisutsukai master knows a thing—”

  “Come off it!” Yin said. “You’re not a beisutsukai master yet.”

  “Neither is Styrlyn,” Brennan said. “But he has already packed as well.” He made it a statement rather than a question, betting double or nothing on his image of inscrutable foreknowledge.

  Styrlyn cleared his throat. “As you say. Other commitments call me, and I have already tarried longer than I planned. Though, Brennan, I do feel that you and I have certain mutual interests, which I would be happy to discuss with you—”

  “Styrlyn, my most excellent friend, I shall be happy to speak with you on any topic you desire,” Brennan said politely and noncommitally, “if we should meet again.” As in, not now. He certainly wasn’t selling out his Mistress this early in their relationship.

  There was an exchange of goodbyes, and of hints and offers.

  And then Brennan was walking down the road that led toward or away from Mount Monastery (for every road is a two-edged sword), the smoothed glass pebbles clicking under his feet.

  He strode out along the path with purpose, vigor, and determination, just in case someone was watching.

  Some time later he stopped, stepped off the path, and wandered just far enough away to prevent anyone from finding him unless they were deliberately following.

  Then he sagged wearily back against a tree-trunk. It was a sparse clearing, with only a few trees poking out of the ground; not much present in the way of distracting scenery, unless you counted the red-tinted stream flowing out of a dark cave-mouth. And Brennan deliberately faced away from that, leaving only the far gray of the horizons, and the blue sky and bright sun.

  Now what?

  He had thought that the Bayesian Conspiracy, of all the possible trainings that existed in this world, would have cleared up his uncertainty about what to do with the rest of his life.

  Power, he’d sought at first. Strength to prevent a repetition of the past. “If you don’t know what you need, take power”—so went the proverb. He had gone first to the Competitive Conspiracy, then to the beisutsukai.

  And now . . .

  Now he felt more lost than ever.

  He could think of things that made him happy. But nothing that he really wanted.

  The passionate intensity that he’d come to associate with his Mistress, or with Jeffreyssai, or the other figures of power that he’d met . . . a life of pursuing small pleasures seemed to pale in comparison, next to that.

  In a city not far from the center of the world, his Mistress waited for him (in all probability, assuming she hadn’t gotten bored with her life and run away). But to merely return, and then drift aimlessly, waiting to fall into someone else’s web of intrigue . . . no. That didn’t seem like . . . enough.

  Brennan plucked a blade of grass from the ground and stared at it, half-unconsciously looking for anything interesting about it; an old, old game that his very first teacher had taught him, what now seemed like ages ago.

  Why did I believe that going to Mount Mirror would tell me what I wanted?

  Well, decision theory did require that your utility function be consistent, but . . .

  If the beisutsukai knew what I wanted, would they even tell me?

  At the Monastery they taught doubt. So now he was falling prey to the third besetting sin of which Jeffreyssai had spoken: lost momentum, indeed. For he had learned to question the image that he held of himself in his mind.

  Are you seeking power because that is your true desire, Brennan?

  Or because you have a picture in your mind of the role that you play as an ambitious young man, and you think it is what someone playing your role would do?

  Almost everything he’d done up until now, even going to Mount Mirror, had probably been the latter.

  And when he blanked out
the old thoughts and tried to see the problem as though for the first time . . .

  . . . nothing much came to mind.

  What do I want?

  Maybe it wasn’t reasonable to expect the beisutsukai to tell him outright. But was there anything they had taught him by which he might answer?

  Brennan closed his eyes and thought.

  First, suppose there is something I would passionately desire. Why would I not know what it is?

  Because I have not yet encountered it, or ever imagined it?

  Or because there is some reason I would not admit it to myself?

  Brennan laughed out loud, then, and opened his eyes.

  So simple, once you thought of it that way. So obvious in retrospect. That was what they called a silver-shoes moment, and yet, if he hadn’t gone to Mount Mirror, it wouldn’t ever have occurred to him.

  Of course there was something he wanted. He knew exactly what he wanted. Wanted so desperately he could taste it like a sharp tinge on his tongue.

  It just hadn’t come to mind earlier, because . . . if he acknowledged his desire explicitly . . . then he also had to see that it was difficult. High, high, above him. Far out of his reach. “Impossible” was the word that came to mind, though it was not, of course, impossible.

  But once he asked himself if he preferred to wander aimlessly through his life—once it was put that way, the answer became obvious. Pursuing the unattainable would make for a hard life, but not a sad one. He could think of things that made him happy, either way. And in the end—it was what he wanted.

  Brennan stood up, and took his first steps, in the exact direction of Shir L’or, the city that lies in the center of the world. He had a plot to hatch, and he did not know who would be part of it.

  And then Brennan stumbled, when he realized that Jeffreyssai had already known.

  One of you at least seems likely to come back . . .

  Brennan had thought he was talking about Taji. Taji had probably thought he was talking about Taji. It was what Taji said he wanted. But how reliable of an indicator was that, really?

 

‹ Prev