Such dire intimations were frequently to be encountered in the pages of broadsheet newspapers, as often as not illustrated by an apocalyptic image from the Terminator films—by a titanium-skulled killer robot staring down the reader with the glowing red points of its pitiless eyes. Elon Musk had spoken of AI as “our greatest existential threat,” of its development as a technological means of “summoning the demon.” (“Hope we’re not just the biological boot loader for digital superintelligence,” he’d tweeted in August of 2014. “Unfortunately, that is increasingly probable.”) Peter Thiel had announced that “People are spending way too much time thinking about climate change, way too little thinking about AI.” Stephen Hawking, meanwhile, had written an op-ed for The Independent in which he’d warned that success in this endeavor, while it would be “the biggest event in human history,” might very well “also be the last, unless we learn to avoid the risks.” Even Bill Gates had publicly admitted his disquiet, speaking of his inability to “understand why some people are not concerned.”
Was I myself concerned? Yes and no. For all that they appealed to my inner core of pessimism, I was not much convinced by these end-time auguries, in large part because they seemed to me to be the obverse of Singulatarian prophecies about AI ushering in a new dispensation, in which human beings would ascend to unimaginable summits of knowledge and power, living eternally in the undimming light of Singularity’s dawn. But I understood that my skepticism was more temperamental than logical, and that I knew almost nothing about the possibly excellent reasons for these fears, or the speculative technology that provoked them. And even if I couldn’t quite bring myself to believe it, I was helplessly, morbidly fascinated by the idea that we might be on the verge of creating a machine that could wipe out our entire species, and by the notion that capitalism’s great philosopher-kings—Musk, Thiel, Gates—were so publicly exercised about the Promethean dangers of that ideology’s most cherished ideal. These dire warnings about AI were coming from what seemed the most unlikely of sources: not from Luddites or religious catastrophists, that is, but from the very people who seemed to most neatly personify our culture’s reverence for machines.
One of the more remarkable phenomena in this area was the existence of a number of research institutes and think tanks substantially devoted to raising awareness about what was known as “existential risk”—the risk of absolute annihilation of the species, as distinct from mere catastrophes like climate change or nuclear war or global pandemics—and to running the algorithms on how we might avoid this particular fate. There was the Future of Humanity Institute in Oxford, and the Centre for Study of Existential Risk at the University of Cambridge, and the Machine Intelligence Research Institute in Berkeley, and the Future of Life Institute in Boston, which latter outfit featured on its board of scientific advisors not just prominent figures from science and technology like Musk and Hawking and the pioneering geneticist George Church, but also, for some reason, the beloved film actors Alan Alda and Morgan Freeman.
What was it that these people were referring to when they spoke of existential risk? What was the nature of the threat, the likelihood of its coming to pass? Were we talking about a 2001: A Space Odyssey scenario, where a sentient computer undergoes some malfunction or other and does what it deems necessary to prevent anyone from shutting it down? Were we talking about a Terminator scenario, where a Skynettian matrix of superintelligent machines gains consciousness and either destroys or enslaves humanity in order to further its own particular goals? Certainly, if you were to take at face value the articles popping up about the looming threat of intelligent machines, and the dramatic utterances of savants like Thiel and Hawking, this would have been the sort of thing you’d have had in mind. They may not have been experts in AI, as such, but they were extremely clever men who knew a lot about science. And if these people were worried—along with Hawkeye from M*A*S*H and the guy who had played, among other personifications of serene wisdom, a scientist trying to prevent the Singularity in the 2014 film Transcendence—shouldn’t we all be worrying with them?
One figure who loomed especially large over this whole area, its eschatologist-in-chief, was Nick Bostrom, the Swedish philosopher who, before he became known as the world’s foremost prophet of technological disaster, had been one of the most prominent figures in the transhumanist movement, a cofounder of the World Transhumanist Association. In late 2014, Bostrom, director of the Future of Humanity Institute, had published a book called Superintelligence: Paths, Dangers, Strategies, in which he outlined the nature of the AI peril. For an academic text that made no significant gesture toward the casual reader, the book had been selling in unexpectedly high quantities, even to the point of appearing on the New York Times bestseller list. (The surge in sales was partly due to Elon Musk sternly advising his Twitter followers to read it.)
Even the most benign form of AI imaginable, the book suggested, could conceivably lead to the destruction of humanity. One of the more extreme hypothetical scenarios the book laid out, for instance, was one in which an AI is assigned the task of manufacturing paper clips in the most efficient and productive manner possible, at which point it sets about converting all the matter in the entire universe into paper clips and paper-clip-manufacturing facilities. The scenario was deliberately cartoonish, but as an example of the kind of ruthless logic we might be up against with an artificial superintelligence, its intent was entirely serious.
“I wouldn’t describe myself these days as a transhumanist,” Nick told me one evening over dinner at an Indian restaurant near the Future of Humanity Institute. Though he was married, his wife and young son were based in Canada, and he lived alone in Oxford. The arrangement involved frequent transatlantic flights, and regular Skype check-ins; regrettable though it was from a work-life balance point of view, it allowed him to focus on his research to a degree that would not otherwise have been possible. (He ate at this particular restaurant so frequently that the waiter had brought him a chicken curry without his having to make any explicit request for same.)
“I mean,” he said, “I absolutely believe in the general principle of increasing human capacities. But I don’t have much connection with the movement itself anymore. There is so much cheerleading of technology in transhumanism, so much unquestioning belief that things will just exponentially get better, and that the right attitude is just to let progress take its course. These are attitudes I have distanced myself from over the years.”
Nick had become a kind of countertranshumanist of late; you couldn’t reasonably accuse him of being a Luddite, but to the extent that he had made a name for himself, both within and outside academia, it was on the strength of his detailed forewarnings about where we might be bringing technology, and where it might take us.
“I still think,” he said, “that within a few generations it will be possible to transform the substrate of our humanity. And I think artificial superintelligence will be the engine that drives that.”
Like many transhumanists, Nick was fond of pointing out the vast disparity in processing power between human tissue and computer hardware. Neurons, for instance, fire at a rate of 200 hertz (or 200 times per second), whereas transistors operate at the level of gigahertz. Signals travel through our central nervous systems at a speed of about 100 meters per second, whereas computer signals can travel at the speed of light. The human brain is limited in size to the capacity of the human cranium, where it is technically possible to build computer processors the size of skyscrapers.
Such factors, he maintained, created the conditions for artificial superintelligence. And because of our tendency to conceive of intelligence within human parameters, we were likely to become complacent about the speed with which machine intelligence might exceed our own. Human-level AI might, that is, seem a very long way off for a very long time, and then be surpassed in an instant. In his book, Nick illustrates this point with a quotation from the AI safety theorist Eliezer Yudkowsky:
AI might make an appa
rently sharp jump in intelligence purely as the result of anthropomorphism, the human tendency to think of “village idiot” and “Einstein” as the extreme ends of the intelligence scale, instead of nearly indistinguishable points on the scale of minds-in-general. Everything dumber than a dumb human may appear to us as simply “dumb.” One imagines the “AI arrow” creeping steadily up the scale of intelligence, moving past mice and chimpanzees, with AI’s still remaining “dumb” because AI’s cannot speak fluent language or write science papers, and then the AI arrow crosses the tiny gap from infra-idiot to ultra-Einstein in the course of one month or some similarly short period.
At this point, the theory goes, things would change quite radically. And whether they would change for the better or for the worse is an open question. The fundamental risk, Nick argued, was not that superintelligent machines might be actively hostile toward their human creators, or antecedents, but that they would be indifferent. Humans, after all, weren’t actively hostile toward most of the species we’d made extinct over the millennia of our ascendance; they simply weren’t part of our design. The same could turn out to be true of superintelligent machines, which would stand in a similar kind of relationship to us as we ourselves did to the animals we bred for food, or the ones who fared little better for all that they had no direct dealings with us at all.
About the nature of the threat, he was keen to stress this point: that there would be no malice, no hatred, no vengeance on the part of the machines.
“I don’t think,” he said, “that I’ve ever seen a newspaper report on this topic that has not been illustrated by a publicity still from one of the Terminator films. The implication of this is always that robots will rebel against us because they resent our dominance, that they will rise up against us. This is not the case.”
And this brought us back to the paper-clip scenario, the ridiculousness of which Nick freely acknowledged, but the point of which was that any harm we might come to from a superintelligent machine would not be the result of malevolence, or of any other humanlike motivation, but purely because our absence was an optimal condition in the pursuit of its particular goal.
“The AI does not hate you,” as Yudkowsky had put it, “nor does it love you, but you are made out of atoms which it can use for something else.”
One way of understanding this would be to listen to a recording of, say, Glenn Gould playing Bach’s Goldberg Variations, and to try to experience the beauty of the music while also holding simultaneously in your mind a sense of the destruction that was wrought in the creation of the piano it is being played on: the trees felled, the elephants slaughtered, the human beings enslaved and killed in the interest of the ivory traders’ profits. Neither the pianist nor the maker of the piano had any personal animosity toward the trees, the elephants, the enslaved men and women; but all were made of atoms that could be used for specific ends, for the making of money, the making of music. Which is to say that this machine which so terrifies a certain contingent of rationalists is perhaps not so different from us after all.
There is, among computer scientists working on AI, a particular reluctance to make predictions about how soon we might expect anything like a superhuman intelligence to emerge—even among those, by no means a majority, who believe such a prospect to be realistic. This has partly to do with a basic disinclination, among scientists in general, to make insufficiently substantiated claims that may wind up making them look foolish. But the reluctance also has a lot to do with the specific history of a field littered with conspicuous examples of people breezily underestimating the challenges involved. In the summer of 1956, before ideas about intelligent machines had begun to coalesce into anything like a discipline, a small group of scientists—leading figures in mathematics, cognitive science, electrical engineering, and computer science—gathered for a six-week-long workshop at Dartmouth College. The group included Marvin Minsky, Claude Shannon, and John McCarthy, men now seen as founders of AI. In a proposal to the Rockefeller Foundation, which funded the workshop, the group supplied the following rationale for its convening:
We propose that a 2 month, 10 man study of artificial intelligence be carried out….The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines that use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.
This sort of hubris has been an intermittent characteristic of AI research, and has led to a series of “AI winters”—periods of drastically decreased funding following bursts of intense enthusiasm about the imminent solution to some or other problem which then turned out to be much more complicated than imagined.
Repeated patterns, through the decades, of overpromising and underdelivering had led to a culture within AI whereby researchers were reluctant to look too far ahead. And this in turn has led to a difficulty in getting the field to engage seriously with the question of existential risk. Most developers working on AI did not want to be seen as making immoderate claims for the technology they were working on.
And whatever else you thought of it, this particular claim about the annihilation of humanity did lay itself open to the charge of immoderation.
—
Nate Soares raised a hand to his close-shaven head and tapped a finger smartly against the frontal plate of his monkish skull.
“Right now,” he said, “the only way you can run a human being is on this quantity of meat.”
We were talking, Nate Soares and I, about the benefits that might come with the advent of artificial superintelligence. For Nate, the most immediate benefit would be the ability to run a human being—to run, specifically, himself—on something other than this quantity of neural meat to which he was gesturing.
He was a sinewy, broad-shouldered man in his mid-twenties, with an air of tightly controlled calm; he wore a green T-shirt bearing the words “NATE THE GREAT,” and as he sat back in his office chair and folded his legs at the knee, I noted that he was shoeless, and that his socks were mismatched, one plain blue, the other white and patterned with cogs and wheels.
The room was utterly featureless save for the chairs we were sitting on, and a whiteboard, and a desk, on which rested an open laptop and a single book, which I happened to note was a hardback copy of Bostrom’s Superintelligence. This was Nate’s office at the Machine Intelligence Research Institute, in Berkeley. The bareness of the space was a result, I assumed, of the fact that Nate had only just assumed his new role as executive director, having left a lucrative career as a software engineer at Google the previous year, and having subsequently risen swiftly up the ranks at MIRI. Nate’s job had formerly been held by Eliezer Yudkowsky—the AI theorist Bostrom had cited on the quantum leap “from infra-idiot to ultra-Einstein”—who had founded MIRI in 2000. (It was originally called the Singularity Institute for Artificial Intelligence; the name was changed in 2013 to avoid confusion with Singularity University, the Silicon Valley private college founded by Kurzweil and Peter Diamandis.)
I knew that Nate conceived of his task, and the task of MIRI, in starkly heroic terms, because I had read a number of articles he had written for the rationalist website Less Wrong in which he discussed his long-held desire to save the world from certain destruction. In one of these articles, I had read about his strict Catholic upbringing, his break with the faith in his teens, and his subsequent investment of his energies into “the passion, the fervor, the desire to optimize the future” through the power of rationality. Nate’s rhetoric in these writings seemed to me a heightened performance of the Silicon Valley house style, in which every new social media platform or sharing economy start-up was announced with the fervent intention to “change t
he world.”
At fourteen, he wrote, he became aware of the chaos of all human things, of a world around him “that couldn’t coordinate,” and he made a promise to himself. “I didn’t promise to fix governments: that was a means to an end, a convenient solution for someone who didn’t know how to look further out. I didn’t promise to change the world, either: every little thing is a change, and not all changes are good. No, I promised to save the world. That promise still stands. The world sure as hell isn’t going to save itself.”
I was intrigued by the tone of Nate’s writing, by the way in which it merged the language of logic with a kind of terse geek-romanticism: a strange, contradictory register which seemed to capture something essential about the idealization of pure reason that was such a prominent aspect of not just transhumanism, but of a broader culture of science and technology—something I had begun to think of as magical rationalism.
He spoke, now, of the great benefits that would come, all things being equal, with the advent of artificial superintelligence. By developing such a transformative technology, he said, we would essentially be delegating all future innovations—all scientific and technological progress—to the machine.
These claims were more or less standard among those in the tech world who believed that artificial superintelligence was a possibility. The problem-solving power of such a technology, properly harnessed, would lead to an enormous acceleration in the turnover of solutions and innovations, a state of permanent Copernican revolution. Questions that had troubled scientists for centuries would be solved in days, hours, minutes. Cures would be found for diseases that currently obliterated vast numbers of lives, while ingenious workarounds to overpopulation were simultaneously devised. To hear of such things is to imagine a God who had long since abdicated all obligations toward his creation making a triumphant return in the guise of a software, an alpha and omega of zeros and ones.
To Be a Machine Page 9
