How Not to Be Wrong : The Power of Mathematical Thinking (9780698163843)

by Ellenberg, Jordan

  What can this mean? Not, I hurry to say, that Leo Tolstoy composed his novel with the names of rabbis concealed therein, designed to be uncovered only once modern Hebrew was developed and classic works of world literature translated into it. Rather, McKay and Bar-Natan are making a potent point about the power of wiggle room. Wiggle room is what the Baltimore stockbroker has when he gives himself plenty of chances to win; wiggle room is what the mutual fund company has when it decides which of its secretly incubating funds are winners and which are trash. Wiggle room is what McKay and Bar-Natan used to work up a list of rabbinical names that jibed well with War and Peace. When you’re trying to draw reliable inferences from improbable events, wiggle room is the enemy.

  In a later paper, McKay and Bar-Natan asked Simcha Emanuel, a Talmud professor then at the University of Tel Aviv, to draw up another list of appellations, this one not designed for compatibility with either the Torah or War and Peace. On this list, the Torah did only a little better than chance. (How Tolstoy did is left unreported.)

  It is very unlikely that any given set of rabbinic appellations is well matched to birth and death dates in the book of Genesis. But with so many ways of choosing the names, it’s not at all improbable that among all the choices there would be one that made the Torah look uncannily prescient. Given enough chances, finding codes is a cinch. It’s especially easy if you use Michael Drosnin’s less scientific approach to code-finding. Drosnin said of code skeptics, “When my critics find a message about the assassination of a prime minister encrypted in Moby Dick, I’ll believe them.” McKay quickly found equidistant letter sequences in Moby Dick referring to the assassination of John F. Kennedy, Indira Gandhi, Leon Trotsky, and, for good measure, Drosnin himself. As I write this, Drosnin remains alive and well despite the prophecy. He is on his third Bible code book, the last of which he advertised by taking out a full-page ad in a December 2010 edition of the New York Times, warning President Obama that, according to letter sequences hidden in Scripture, Osama bin Laden might already have a nuclear weapon.

  Witztum, Rips, and Rosenberg insist they weren’t like the masters of the incubator funds, displaying to the public only the experiments that gave the best possible results; their precise list of names was chosen in advance, they say, before running any tests. And that may well be true. But even if it is, it casts the miraculous success of the Bible codes in a very different light. That the Torah, like War and Peace, can successfully be mined for some version of the rabbis’ names is not surprising. The miracle, if there is one, is that Witztum and his colleagues were moved to choose precisely those versions of the names on which the Torah scores best.

  There’s one loose end that should trouble you, though. McKay and Bar-Natan made a compelling case that the wiggle room in the design of Witztum’s experiment was enough to explain the Bible codes. But the Witztum paper was carried out using standard statistical tests, the same ones scientists use to judge claims about everything from medicines to economic policies. It wouldn’t have been accepted in Statistical Science otherwise. If the paper passed that test, shouldn’t we have accepted its conclusions, however otherworldly they may have seemed? Or, to put it another way: if we now feel comfortable rejecting the conclusions of the Witztum study, what does that say about the reliability of our standard statistical tests?

  It says you ought to be a little worried about them. And it turns out that, without any input from the Torah, scientists and statisticians have already been worrying about them for quite some time.

  SEVEN

  DEAD FISH DON’T READ MINDS

  Because here’s the thing: the Bible code kerfuffle is not the only occasion on which the standard statistical tool kit has been used to derive a result that sounds like magic. One of the hottest topics in medical science is functional neuroimaging, which promises to let scientists see your thoughts and feelings flickering across your synapses in real time through ever-more-accurate sensors. At the 2009 Organization for Human Brain Mapping conference in San Francisco, UC Santa Barbara neuroscientist Craig Bennett presented a poster called “Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction.” It takes a second to unwrap the jargony title, but when you do, the poster announces pretty clearly the unusual nature of its results. A dead fish, scanned in an fMRI device, was shown a series of photographs of human beings, and was found to have a surprisingly strong ability to correctly assess the emotions the people in the pictures displayed. That would be impressive enough for a dead person or a live fish—for a dead fish, it’s Nobel Prize material!

  But the paper, of course, is a deadpan gag. (And a well-executed one: I especially like the “Methods” section, which starts “One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. . . . Foam padding was placed within the head coil as a method of limiting salmon movement during the scan, but proved to be largely unnecessary as subject motion was exceptionally low.”) The joke, like all jokes, is a veiled attack: in this case, an attack on sloppy methodology among those neuroimaging researchers who make the mistake of ignoring the fundamental truth that improbable things happen a lot. Neuroscientists divvy up their fMRI scans into tens of thousands of small pieces, called voxels, each corresponding to a small region of the brain. When you scan a brain, even a cold dead fish brain, there’s a certain amount of random noise coming through on each voxel. It’s pretty unlikely that the noise will happen to spike exactly at the moment that you show the fish a snapshot of a person in emotional extremity. But the nervous system is a big place, with tens of thousands of voxels to choose from. The odds that one of those voxels provides data matching up well with the photos is pretty good. That’s exactly what Bennett and his collaborators found; in fact, they located two groups of voxels that did an excellent job empathizing with human emotion, one in the salmon’s medial brain cavity and the other in the upper spinal column. The point of Bennett’s paper is to warn that the standard methods of assessing results, the way we draw our thresholds between a real phenomenon and random static, come under dangerous pressure in this era of massive data sets, effortlessly obtained. We need to think very carefully about whether our standards for evidence are strict enough, if the empathetic salmon makes the cut.

  The more chances you give yourself to be surprised, the higher your threshold for surprise had better be. If a random Internet stranger who eliminated all North American grains from his food intake reports that he dropped fifteen pounds and his eczema went away, you shouldn’t take that as powerful evidence in favor of the maize-free plan. Somebody’s selling a book about that plan, and thousands of people bought that book and tried it, and the odds are very good that, by chance alone, one among them will experience some weight loss and clear skin the next week. And that’s the guy who’s going to log in as saygoodbye2corn452 and post his excited testimonial, while the people for whom the diet failed stay silent.

  The really surprising result of Bennett’s paper isn’t that one or two voxels in a dead fish passed a statistical test; it’s that a substantial proportion of the neuroimaging articles he surveyed didn’t use statistical safeguards (known as “multiple comparisons correction”) that take into account the ubiquity of the improbable. Without those corrections, scientists are at serious risk of running the Baltimore stockbroker con, not only on their colleagues but on themselves. Getting excited about the fish voxels that matched the photos and ignoring the rest is as potentially dangerous as getting excited about the successful series of stock newsletters while ignoring the many more editions that blew their calls and went in the trash.

  REVERSE ENGINEERING, OR, WHY ALGEBRA IS HARD

  There are two moments in the course of education where a lot of kids fall off the math train. The first comes in the elementary grades, when fractions are introduced. Until that moment, a n
umber is a natural number, one of the figures 0, 1, 2, 3 . . . It is the answer to a question of the form “how many.”* To go from this notion, so primitive that many animals are said to understand it, to the radically broader idea that a number can mean “what portion of,” is a drastic philosophical shift. (“God made the natural numbers,” the nineteenth-century algebraist Leopold Kronecker famously said, “and all the rest is the work of man.”)

  The second dangerous twist in the track is algebra. Why is it so hard? Because, until algebra shows up, you’re doing numerical computations in a straightforwardly algorithmic way. You dump some numbers into the addition box, or the multiplication box, or even, in traditionally minded schools, the long-division box, you turn the crank, and you report what comes out the other side.

  Algebra is different. It’s computation backward. When you’re asked to solve

  x + 8 = 15

  you know what came out of the addition box (namely, 15) and you’re being asked to reverse-engineer what, along with the 8, went in.

  In this case, as your seventh-grade math teacher no doubt once told you, you can flip things over to get things right-side round again:

  x = 15 − 8

  at which point you can just toss 15 and 8 in the subtraction box (making sure now to keep track of which one you toss first . . .) and find that x must be 7.

  But it’s not always so easy. You might need to solve a quadratic equation, like

  x2 − x = 1.

  Really? (I hear you cry.) Might you? Other than that your teacher asked you to, why would you?

  Think back to that missile from chapter 2, still traveling furiously toward you:

  Maybe you know that the missile launched from 100 meters above ground level, with upward velocity of 200 meters per second. If there was no such thing as gravity, the missile would just keep on rising along a straight line in accordance with Newton’s laws, getting 200 meters higher each second, and its height after x seconds would be described by the linear function

  height = 100 + 200x.

  But there is such a thing as gravity, which bends the arc and forces the missile to curve back toward earth. It turns out that the effect of gravity is described by adding a quadratic term:

  height = 100 + 200x − 5x2

  where the quadratic term is negative just because gravity pushes missiles down, not up.

  There are a lot of questions you might ask about a missile heading toward you, but one of particularly great import is: When will it land? To answer this is just to answer the question: When will the height of the missile be zero? That is, for what value of x is it the case that

  100 + 200x − 5x2 = 0?

  It is by no means clear how you’re supposed to “flip” this equation around and solve for x. But maybe you don’t have to. Trial and error is a very powerful weapon. If you plug x = 10 into the above formula, to see how high the missile is after 10 seconds, you get 1,600 meters. Plug in x = 20 and you get 2,100 meters, so it looks like the missile may still be rising. When x = 30, you get 1,600 again: promising; we must be past the peak. At x = 40 the missile is once again just 100 meters above the ground. We could walk forward 10 more seconds, but when we’re so close to impact already that’s surely overdoing it. If you plug in x = 41 you get −105 meters, which doesn’t mean you’re predicting the missile has actually begun burrowing under the earth’s surface, but rather that impact has already happened, so that your nice, clean model of the missile’s motion is, as we say in ballistics, no longer operative.

  So if 41 seconds is too long, what about 40.5? That gives −1.25 meters, just a little bit below 0. Turn back the clock a little to 40.4, and you get 19.2m, so impact hasn’t happened yet. 40.49? Very close, just 0.8m above the ground. . . .

  You can see that by playing the trial and error game, carefully turning the time knob back and forth, you can approximate the time of impact as closely as you like.

  But have we “solved” the equation? You’re probably hesitant to say you have—after all, even if you keep fine-tuning your guesses until you get the time of impact pinned down to

  40.4939015319 . . .

  seconds after launch, you don’t know the answer, but just an approximation of the answer. In practice, though, it doesn’t help you to time the impact to the millionth of a second, does it? Probably just saying “about 40 seconds” is enough. Try to generate an answer any more precise than that and you’re wasting your time, and you’ll probably be wrong, besides, because our very simple model of the missile’s progress fails to take into account many other factors, like air resistance, the variation in air resistance coming from weather, the spin of the missile itself, and so on. These effects may be small, but they’re surely big enough to keep you from knowing down to the microsecond when the projectile will show up for its appointment with the ground.

  If you want a satisfyingly exact solution, never fear—the quadratic formula is here to help. You may well have memorized this formula once in your life, but unless you have an unusually gifted memory or you are twelve, you don’t have it in mind just at the moment. So here it is: if x is a solution to

  c + bx + ax2 = 0

  where a, b, and c are any numbers whatsoever, then

  In the case of the missile, c = 100, b = 200, and a = −5. So what the quadratic formula has to say about x is that

  Most of the symbols in there are things you could type in your calculator, but there’s one funny outlier, the ±. It looks like a plus sign and a minus sign that love each other very much, and this isn’t so far off. It indicates that, although we started our mathematical sentence, all confidence, with

  x =

  we end up in a state of ambivalence. The ±, something like a blank Scrabble tile, can be read as either a + or a −, as we choose. Each choice we make produces a value of x that makes the equation 100 + 200x − 5x2 = 0 hold. There is no single solution to this equation. There are two.

  That there are two values of x which satisfy the equation can be made apparent to the eye, even if you long ago forgot the quadratic formula. You can draw a graph of the equation y = 100 + 200x − 5x2 and get a nice upside-down parabola, like this:

  The horizontal line is the x-axis, those points on the plane whose y-coordinate is 0. When the curve y = 100 + 200x − 5x2 meets the x-axis, it must be the case both that y is 100 + 200x − 5x2 and that y = 0; so 100 + 200x − 5x2 = 0, precisely the equation we were trying to solve, now given geometric form as a question about the intersection between a curve and a horizontal line.

  And geometric intuition demands that if such a parabola noses its way above the x-axis at all, it must strike the x-axis in exactly two places, no more, no fewer. In other words, there are two values of x such that 100 + 200x − 5x2 = 0.

  So what are these two values?

  If we choose to read ± as +, we get

  x = 20 + 2√105

  which is 40.4939015319 . . . , the same answer we came up with by trial and error. But if we choose −, we get

  x = 20 − 2√105

  which is −0.4939015319 . . .

  As an answer to our original question, this is somewhat nonsensical. The answer to “When is that missile going to hit me?” can’t be “Half a second ago.”

  Yet this negative value of x is a perfectly good solution to the equation, and when math tells us something we should at least try to listen. What does the negative number mean? Here’s one way to understand it. We said the missile was launched from 100 meters off the ground, at a velocity of 200 meters per second. But all we really used was that, at time 0, the missile was traveling upward at that velocity from that position. What if that wasn’t actually the launch? Maybe the launch took place, not at time 0, from 100 meters up, but at some earlier time, directly from the ground. What time?

  The computation tells us: there are exactly two times when the missile is at ground level. On
e time is 0.4939 . . . seconds ago. That’s when the missile was launched. The other time is 40.4939 . . . seconds from now. That’s when the missile lands.

  Perhaps it doesn’t seem so troubling, especially if you’re used to the quadratic formula, to get two answers to the same question. But when you’re twelve it represents a real philosophical shift. You’ve spent six long years in grade school figuring out what the answer is, and now, suddenly, there is no such thing.

  And those are just quadratic equations! What if you have to solve

  x3 + 2x2 − 11x = 12?

  This is a cubic equation, which is to say it involves x raised to the third power. Fortunately, there is a cubic formula that allows you to figure out, by a direct computation, what values of x could have gone in the box to make 12 fall out when you turn the crank. But you didn’t learn the cubic formula in school, and the reason you didn’t learn it in school is that it’s kind of a mess, and wasn’t worked out until the late Renaissance, when itinerant algebraists roamed across Italy, engaging each other in fierce public equation-solving battles with money and status on the line. The few people who knew the cubic formula kept it to themselves or wrote it down in cryptic rhymed verse.

  Long story. The point is, reverse engineering is hard.

  The problem of inference, which is what the Bible coders were wrestling with, is hard because it’s exactly this kind of problem. When we are scientists, or Torah scholars, or toddlers gaping at the clouds, we are presented with observations and asked to build theories—what went into the box to produce the world that we see? Inference is a hard thing, maybe the hardest thing. From the shape of the clouds and the way they move we struggle to go backward, to solve for x, the system that made them.