The Trouble With Physics: The Rise of String Theory, The Fall of a Science, and What Comes Next

by Lee Smolin

  This hurts science, because it chokes off the investigation of alternative directions, some of them very promising. Despite the inadequate investment in these approaches, a few have moved ahead of string theory to the point of suggesting definite predictions for experiments, which are now in progress.

  How is it possible that string theory, which has been pursued by more than a thousand of the brightest and best-educated scientists, working in the best conditions, is in danger of failing? This has puzzled me for a long time, but now I think I know the answer. What I believe is failing is not so much a particular theory but a style of doing science that was well suited to the problems we faced in the middle part of the twentieth century but is ill suited to the kinds of fundamental problems we face now. The standard model of particle physics was the triumph of a particular way of doing science that came to dominate physics in the 1940s. This style is pragmatic and hard-nosed and favors virtuosity in calculating over reflection on hard conceptual problems. This is profoundly different from the way that Albert Einstein, Niels Bohr, Werner Heisenberg, Erwin Schrödinger, and the other early-twentieth-century revolutionaries did science. Their work arose from deep thought on the most basic questions surrounding space, time, and matter, and they saw what they did as part of a broader philosophical tradition, in which they were at home.

  In the approach to particle physics developed and taught by Richard Feynman, Freeman Dyson, and others, reflection on foundational problems had no place in research. This freed them from the debates over the meaning of quantum physics that their elders were embroiled in and led to thirty years of dramatic progress. This is as it should be: Different styles of research are needed to solve different kinds of problems. Working out the applications of established frameworks requires very different kinds of thinking—and thinkers—than inventing those frameworks in the first place.

  However, as I will argue in detail in the pages to come, the lesson of the last thirty years is that the problems we’re up against today cannot be solved by this pragmatic way of doing science. To continue the progress of science, we have to again confront deep questions about space and time, quantum theory, and cosmology. We again need the kinds of people who can invent new solutions to long-standing foundational problems. As we shall see, the directions in which progress is being made—which are taking theory back into contact with experiment—are led by people who have an easier time inventing new ideas than following popular trends and for the most part do science in the reflective and foundational style of the early-twentieth-century pioneers.

  I want to emphasize that my concern is not with string theorists as individuals, some of whom are the most talented and accomplished physicists I know. I would be the first to defend their right to pursue the research they think is most promising. But I am extremely concerned about a trend in which only one direction of research is well supported while other promising approaches are starved.

  It is a trend with tragic consequences if, as I will argue, the truth lies in a direction that requires a radical rethinking of our basic ideas about space, time, and the quantum world.

  I

  THE UNFINISHED REVOLUTION

  1

  The Five Great Problems in Theoretical Physics

  FROM THE BEGINNING of physics, there have been those who imagined they would be the last generation to face the unknown. Physics has always seemed to its practitioners to be almost complete. This complacency is shattered only during revolutions, when honest people are forced to admit that they don’t know the basics. But even revolutionaries still imagine that the big idea—the one that will tie it all up and end the search for knowledge—lies just around the corner.

  We live in one of those revolutionary periods, and have for a century. The last such period was the Copernican revolution, beginning in the early sixteenth century, during which Aristotelian theories of space, time, motion, and cosmology were overthrown. The culmination of that revolution was Isaac Newton’s proposal of a new theory of physics, published in his Philosophiae Naturalis Principia Mathematica in 1687. The current revolution in physics began in 1900, with Max Planck’s discovery of a formula describing the energy distribution in the spectrum of heat radiation, which demonstrated that the energy is not continuous but quantized. This revolution has yet to end. The problems that physicists must solve today are, to a large extent, questions that remain unanswered because of the incompleteness of the twentieth century’s scientific revolution.

  The core of our failure to complete the present scientific revolution consists of five problems, each famously intractable. These problems confronted us when I began my study of physics in the 1970s, and while we have learned a lot about them in the last three decades, they remain unsolved. One way or another, any proposed theory of fundamental physics must solve these five problems, so it’s worth taking a closer look at each.

  Albert Einstein was certainly the most important physicist of the twentieth century. Perhaps his greatest work was his discovery of general relativity, which is the best theory we have so far of space, time, motion, and gravitation. His profound insight was that gravity and motion are intimately related to each other and to the geometry of space and time. This idea broke with hundreds of years of thinking about the nature of space and time, which until then had been viewed as fixed and absolute. Being eternal and unchanging, they provided a background, which we used to define notions like position and energy.

  In Einstein’s general theory of relativity, space and time no longer provide a fixed, absolute background. Space is as dynamic as matter; it moves and morphs. As a result, the whole universe can expand or shrink, and time can even begin (in a Big Bang) and end (in a black hole).

  Einstein accomplished something else as well. He was the first person to understand the need for a new theory of matter and radiation. Actually, the need for a break was implicit in Planck’s formula, but Planck had not understood its implications deeply enough; he felt that it could be reconciled with Newtonian physics. Einstein thought otherwise, and he gave the first definitive argument for such a theory in 1905. It took twenty more years to invent that theory, known as the quantum theory.

  These two discoveries, of relativity and of the quantum, each required us to break definitively with Newtonian physics. However, in spite of great progress over the century, they remain incomplete. Each has defects that point to the existence of a deeper theory. But the main reason each is incomplete is the existence of the other.

  The mind calls out for a third theory to unify all of physics, and for a simple reason. Nature is in an obvious sense “unified.” The universe we find ourselves in is interconnected, in that everything interacts with everything else. There is no way we can have two theories of nature covering different phenomena, as if one had nothing to do with the other. Any claim for a final theory must be a complete theory of nature. It must encompass all we know.

  Physics has survived a long time without that unified theory. The reason is that, as far as experiment is concerned, we have been able to divide the world into two realms. In the atomic realm, where quantum physics reigns, we can usually ignore gravity. We can treat space and time much as Newton did—as an unchanging background. The other realm is that of gravitation and cosmology. In that world, we can often ignore quantum phenomena.

  But this cannot be anything other than a temporary, provisional solution. To go beyond it is the first great unsolved problem in theoretical physics:

  Problem 1: Combine general relativity and quantum theory into a single theory that can claim to be the complete theory of nature.

  This is called the problem of quantum gravity.

  Besides the argument based on the unity of nature, there are problems specific to each theory that call for unification with the other. Each has a problem of infinities. In nature, we have yet to encounter anything measurable that has an infinite value. But in both quantum theory and general relativity, we encounter predictions of physically sensible quantities becomin
g infinite. This is likely the way that nature punishes impudent theorists who dare to break her unity.

  General relativity has a problem with infinities because inside a black hole the density of matter and the strength of the gravitational field quickly become infinite. That appears to have also been the case very early in the history of the universe—at least, if we trust general relativity to describe its infancy. At the point at which the density becomes infinite, the equations of general relativity break down. Some people interpret this as time stopping, but a more sober view is that the theory is just inadequate. For a long time, wise people have speculated that it is inadequate because the effects of quantum physics have been neglected.

  Quantum theory, in turn, has its own trouble with infinities. They appear whenever you attempt to use quantum mechanics to describe fields, like the electromagnetic field. The problem is that the electric and magnetic fields have values at every point in space. This means that there are an infinite number of variables (even in a finite volume there are an infinite number of points, hence an infinite number of variables). In quantum theory, there are uncontrollable fluctuations in the values of every quantum variable. An infinite number of variables, fluctuating uncontrollably, can lead to equations that get out of hand and predict infinite numbers when you ask questions about the probability of some event happening, or the strength of some force.

  So this is another case where we can’t help but feel that an essential part of physics has been left out. There has long been the hope that when gravity is taken into account, the fluctuations will be tamed and all will be finite. If infinities are signs of missing unification, a unified theory will have none. It will be what we call a finite theory, a theory that answers every question in terms of sensible, finite numbers.

  Quantum mechanics has been extremely successful at explaining a vast realm of phenomena. Its domain extends from radiation to the properties of transistors and from elementary-particle physics to the action of enzymes and other large molecules that are the building blocks of life. Its predictions have been borne out again and again over the course of the last century. But some physicists have always had misgivings about it, because the reality it describes is so bizarre. Quantum theory contains within it some apparent conceptual paradoxes that even after eighty years remain unresolved. An electron appears to be both a wave and a particle. So does light. Moreover, the theory gives only statistical predictions of subatomic behavior. Our ability to do any better than that is limited by the uncertainty principle, which tells us that we cannot measure a particle’s position and momentum at the same time. The theory yields only probabilities. A particle—an atomic electron, say—can be anywhere until we measure it; our observation in some sense determines its state. All of this suggests that quantum theory does not tell the whole story. As a result, in spite of its success, there are many experts who are convinced that quantum theory hides something essential about nature that we need to know.

  One problem that has bedeviled the theory from the beginning is the question of the relationship between reality and the formalism. Physicists have traditionally expected that science should give an account of reality as it would be in our absence. Physics should be more than a set of formulas that predict what we will observe in an experiment; it should give a picture of what reality is. We are accidental descendants of an ancient primate, who appeared only very recently in the history of the world. It cannot be that reality depends on our existence. Nor can the problem of no observers be solved by raising the possibility of alien civilizations, for there was a time when the world existed but was far too hot and dense for organized intelligence to exist.

  Philosophers call this view realism. It can be summarized by saying that the real world out there (or RWOT, as my first philosophy teacher used to put it) must exist independently of us. It follows that the terms by which science describes reality cannot involve in any essential way what we choose to measure or not measure.

  Quantum mechanics, at least in the form it was first proposed, did not fit easily with realism. This is because the theory presupposed a division of nature into two parts. On one side of the division is the system to be observed. We, the observers, are on the other side. With us are the instruments we use to prepare experiments and take measurements, and the clocks we use to record when things happen. Quantum theory can be described as a new kind of language to be used in a dialogue between us and the systems we study with our instruments. This quantum language contains verbs that refer to our preparations and measurements and nouns that refer to what is then seen. It tells us nothing about what the world would be like in our absence.

  Since quantum theory was first proposed, a debate has raged between those who accept this way of doing science and those who reject it. Many of the founders of quantum mechanics, including Einstein, Erwin Schrödinger, and Louis de Broglie, found this approach to physics repugnant. They were realists. For them quantum theory, no matter how well it worked, was not a complete theory, because it did not provide a picture of reality absent our interaction with it. On the other side were Niels Bohr, Werner Heisenberg, and many others. Rather than being appalled, they embraced this new way of doing science.

  Since then, the realists have scored some successes by pointing to inconsistencies in the present formulation of quantum theory. Some of these apparent inconsistencies arise because, if it is universal, quantum theory should also describe us. Problems, then, come from the division of the world required to make sense of quantum theory. One difficulty is where you draw the dividing line, which depends on who is doing the observing. When you measure an atom, you and your instruments are on one side and the atom is on the other side. But suppose I watch you working through a videocam I have set up in your laboratory. I can consider your whole lab—including you and your instruments, as well as the atoms you play with—to constitute one system that I am observing. On the other side would be only me.

  You and I hence describe two different “systems.” Yours includes just the atom. Mine includes you, the atom, and everything you use to study it. What you see as a measurement, I see as two physical systems interacting with each other. Thus, even if you agree that it’s fine to have the observers’ actions as part of the theory, the theory as given is not sufficient. Quantum mechanics has to be expanded, to allow for many different descriptions, depending on who the observer is.

  This whole issue goes under the name the foundational problems of quantum mechanics. It is the second great problem of contemporary physics.

  Problem 2: Resolve the problems in the foundations of quantum mechanics, either by making sense of the theory as it stands or by inventing a new theory that does make sense.

  There are several different ways one might do this.

  Provide a sensible language for the theory, one that resolves all puzzles like the ones just mentioned and incorporates the division of the world into system and observer as an essential feature of the theory.

  Find a new interpretation of the theory—a new way of reading the equations—that is realist, so that measurement and observation play no role in the description of fundamental reality.

  Invent a new theory, one that gives a deeper understanding of nature than quantum mechanics does.

  All three options are currently being pursued by a handful of smart people. There are unfortunately not many physicists who work on this problem. This is sometimes taken as an indication that the problem is either solved or unimportant. Neither is true. This is probably the most serious problem facing modern science. It is just so hard that progress is very slow. I deeply admire the physicists who work on it, both for the purity of their intentions and for their courage to ignore fashion and attack the hardest and most fundamental of problems.

  But despite their best efforts, the problem remains unsolved. This suggests to me that it’s not just a matter of finding a new way to think about quantum theory. Those who initially formulated the theory were not realists. They did not believe th
at human beings were capable of forming a true picture of the world as it exists independent of our actions and observations. They argued instead for a very different vision of science: In their view, science can be nothing but an extension of the ordinary language we use to describe our actions and observations to one another.

  In more recent times, that view looks self-indulgent—the product of a time we hope we have advanced beyond in many respects. Those who continue to defend quantum mechanics as formulated, and propose it as a theory of the world, do so mostly under the banner of realism. They argue for a reinterpretation of the theory along realist lines. However, while they have made some interesting proposals, none has been totally convincing.

  It is possible that realism as a philosophy will simply die off, but this seems unlikely. After all, realism provides the motivation driving most scientists. For most of us, belief in the RWOT and the possibility of truly knowing it motivates us to do the hard work needed to become a scientist and contribute to the understanding of nature. Given the failure of realists to make sense of quantum theory as formulated, it appears more and more likely that the only option is the third one: the discovery of a new theory that will be more amenable to a realist interpretation.

  I should admit that I am a realist. I side with Einstein and the others who believe that quantum mechanics is an incomplete description of reality. Where, then, should we look for what is missing in quantum mechanics? It has always seemed to me that the solution will require more than a deeper understanding of quantum physics itself. I believe that if the problem has not been solved after all this time, it is because there is something missing, some link to other problems in physics. The problem of quantum mechanics is unlikely to be solved in isolation; instead, the solution will probably emerge as we make progress on the greater effort to unify physics.