LESSON: Extraordinary Claims Need Extraordinary Evidence
AS THIS TABLE ILLUSTRATES, THE DRAMATIC CLAIM THAT ABORTIONS had suddenly begun to increase—after twenty years of almost uninterrupted annual decreases—begged for confirmation.
Before accepting any such claims, it is wise to first look carefully at the source. In this case, the claim that the trend had suddenly reversed came from a critic of Republican policies who had no track record of evaluating abortion statistics and no special expertise in the field; in addition, the claim was based on fragmentary information.
Stassen got the data wrong for two reasons. First, he tried to project a national trend from an unscientific, nonrandom sample. He used the first sixteen states to report their official abortion data to the federal Centers for Disease Control, without waiting for the other thirty-four states to report. That’s like trying to predict the outcome of a presidential election after the first sixteen states close their polls and report their results. Second, in two of the states among Stassen’s sixteen, the Guttmacher Institute found the reporting system unreliable. In Colorado, where Stassen claimed that rates “skyrocketed 111 percent,” the reporting procedure had been changed in order to compensate for historic underreporting. What Stassen thought was an increase in the number of abortions really reflected an improvement in procedures for counting them.
CASE STUDY: Is Cold-Eeze “Clinically Proven?”
EVEN A GOOD STUDY FROM A REPUTABLE SOURCE CAN BE MISLEADING, if the results can’t be replicated. If you’ve ever gotten the sniffles and shopped for a drugstore cold remedy, you’ve probably noticed a product called Cold-Eeze, which contains a zinc compound and claims to be “clinically proven to cut colds by nearly half.” Science has proven this zinc stuff works! Or has it? A close look at the evidence shows that the Cold-Eeze claim—indeed, the entire company—is based largely on a single study from 1996. But several other studies have produced starkly different results.
The original study, still cited on the Cold-Eeze website as we write this, was done at the Cleveland Clinic Foundation by Dr. Michael Macknin. That study and Macknin’s many subsequent news interviews sent Cold-Eeze flying off drugstore shelves. He told Barbara Walters on ABC’s 20/20 in January 1997 that he “got goose bumps” when he tabulated the data, and added, “here was something that actually seemed like it was helping the common cold, and nothing had really worked like this before.” The price of stock in the Quigley Corporation, which sells Cold-Eeze, soared, going from about $2 a share before the study was published to a high of $37 at the time of the 20/20 broadcast.
The first study seemed solid enough. It was published in a reputable, peer-reviewed journal, Annals of Internal Medicine. And it was a double-blind study: forty-nine subjects got the Cold-Eeze lozenges, fifty got a placebo, and neither the subjects nor the persons dispensing the lozenges knew which was which until the time came to tally the results. But when Macknin conducted a second study, of 249 suburban Cleveland students in grades one through twelve, he found that the kids who received the placebo got over colds just as quickly as those getting Cold-Eeze. The lozenges “were not effective in treating cold symptoms in children and adolescents,” said the report, published in The Journal of the American Medical Association in June 1998. It added that further research was needed “to clarify what role, if any, zinc may play in treating cold symptoms.” Quigley’s stock plunged. The company had actually paid for Macknin’s second study, but it doesn’t display that one on its website.
Since that 1996 study that launched Cold-Eeze sales, the evidence has continued to be mixed. Over the years some scientific studies have found that zinc gluconate (the featured ingredient of Cold-Eeze) seemed to reduce the duration and severity of cold symptoms, but several other scientific studies have found no such effect. The National Institutes of Health have concluded: “Additional research is needed to determine whether zinc compounds have any effect on the common cold.” Zinc might work, or it might not.
So Cold-Eeze turns out to be another brand built on spin, like Listerine. In 1999, the Federal Trade Commission accused Quigley of false advertising for claiming on the QVC shopping network that its lozenges could actually prevent colds. The FTC said Quigley had no reasonable evidence to support such claims, and Quigley settled the case by agreeing to stop making them. Also false, in our view, is the company’s biggest selling point, its claim that Cold-Eeze is “clinically proven” to cut cold symptoms by 42 percent. The best the company can say truthfully is that Cold-Eeze has been clinically tested, with inconclusive results.
Non-Evidence: Linus Pauling and Bruce Willis
We’ve shown that anecdotes can mislead us, and that sloppy, biased, or made-up studies can masquerade as evidence, and that even good studies can fail to hold up when set beside later research. Here are a few other things that do not count as evidence.
APPEALS TO AUTHORITY
Linus Pauling won two Nobel Prizes, one for chemistry and the other for peace, but he had no particular expertise in medicine. Nevertheless, millions of people swallowed his claims that high doses of vitamin C could cut the incidence of the common cold and might even be effective against cancer. In fact, at least sixteen controlled experiments, some involving thousands of volunteers, have failed to show that vitamin C has any effect on either condition. Dr. Stephen Barrett of Quackwatch.org reports that “no responsible medical or nutrition scientists share [Pauling’s] views.”
* * *
FactCheck.org’s Guide to Testing Evidence
At FactCheck.org, we find ourselves asking a few basic questions again and again in evaluating evidence. Here’s a short list of tests we have found useful:
• Is the source highly regarded and widely accepted? There are a number of longstanding organizations we know we can count on for reliable, unbiased information. For job statistics, the Bureau of Labor Statistics is every economist’s basic source. For hurricane statistics, the National Hurricane Center is considered the authority. For determining when business recessions begin and end, the private, nonprofit National Bureau of Economic Research is an authority cited in economic history books. And for abortion statistics, the Guttmacher Institute is accepted by all sides as trustworthy.
• Is the source an advocate? Claims made by political parties, candidates, lobbying groups, salesmen, and other advocates may be true but are usually self-serving and as a result may be misleading; they require special scrutiny. Always compare their information with other sources. The National Research Council—chartered by Congress and able to call on the nation’s most eminent experts—is a better source on whether gun-control laws cut crime than is a group devoted to lobbying for or against such laws. The Guttmacher Institute does advocate “reproductive freedom,” but we accept its numbers not only because the institute is trusted by both sides of the debate, but also because in this case they support a president who is on the other side of the issue. The institute’s report that the number of abortions continues to diminish under Bush is the opposite of what we would expect if it were allowing ideological bias to color its findings.
• What is the source’s track record? Look for previous experience. In our abortion example, Stassen had no record of conducting studies on abortion statistics. In contrast, the Guttmacher Institute’s surveys of abortion providers go back to 1973.
• What method is used? Mitch Snyder’s estimate of the number of homeless people turned out to depend on a collection of guesses, at best. The Urban Institute’s estimates, though they also involve some assumptions and guesswork, are based on U.S. Census data gathered in a uniform way from a very large, random sample.
• Does the source “show its work”? Good researchers always explain how they arrived at their numbers and conclusions. Daniel Cristol described exactly how he and his colleagues conducted their 400 timed observations of crows, and he published the results. Good research methods are transparent.
• Is the sample random? News organizations and websites
are fond of conducting “unscientific” polls. Viewers or visitors are asked to express a preference, and the results are reported. This is just a marketing method designed to draw interest; the results are utterly meaningless because the sample is self-selected, not random. Some such polls have even been intentionally rigged. At best, they are like Stassen’s nonrandom sixteen-state sample, which turned out not to reflect the situation in all fifty states.
• Is there a control group? Good scientific procedure requires a “control” to provide a valid basis for comparison. A crow dropping nuts in front of a car proves nothing. Cristol watched crows when cars were coming, but he also watched crows when cars were notcoming, and he observed the difference (none). In tests of new drugs one group gets a placebo, with no active ingredients, to provide a point of comparison with the group that gets the actual drug.
• Does the source have the requisite skill? A trained epidemiologist should be trusted more than a newspaper headline writer to evaluate whether a cluster of cancer cases was caused by something in the water, or was just a statistical fluke.
• Have the results been replicated, or contradicted? Sometimes one study tells a story that isn’t backed up by later research. Have the results been repeated in similar studies? Do other researchers agree, or do they come up with contrary findings? The Cold-Eeze story shows how cherry-picked studies can mislead us.
* * *
The lesson here is that somebody who is an authority in one field isn’t necessarily qualified in another. Sam Waterston plays a smart, tough prosecutor on television’s Law & Order series, but so far as we know he has no special expertise as a financial adviser. So why should we give any special weight to his TV commercials praising the brokerage firm TD Waterhouse? Bruce Willis endorsed President George H. W. Bush in 1992 and supported the current war in Iraq, but his portrayal of action heroes on the screen is no reason to give his political or military views any more weight than your next-door neighbor’s. The same goes for Martin Sheen’s endorsement of Howard Dean in the 2004 Democratic presidential primary. Playing a president on TV is about as valid a qualification for making political judgments as playing a doctor on TV might be in recommending decaffeinated coffee, the way Robert Young did in the 1970s, after starring in Marcus Welby, MD on television.
Before relying on any authority, ask yourself, “Is this source competent? Does he know what he’s talking about? Does she have any real evidence? Do other authorities in the same field agree?”
APPEALS TO POPULARITY
Advertisers use these all the time. A typical example: a hospital in Saginaw, Michigan, says it is “preferred two to one.” Does having a large number of patients mean that a hospital provides the best care? Not necessarily. It might simply have better parking or be located more conveniently.
Other examples to look out for: “top-selling”; “number one”; “preferred over…” In politics the “front runner” is often the candidate to watch, but is not necessarily the best one or even the one destined to win, as onetime front runner Howard Dean discovered in the 2004 Democratic primary races. And in February 2006, both Ford and General Motors were claiming to be the top-selling brand of automobiles in the United States, as figures on 2005 vehicle registrations trickled in. In fact, both automakers had been losing ground for years, falling from a combined 60 percent of the U.S. market in 1986 to about 45 percent in 2005. Each might just as easily have said it was “preferred by fewer and fewer.”
Popularity may settle elections, but it doesn’t settle questions of fact. Ask yourself, “Is this thing popular because it’s good, or for some other reason, such as a big advertising budget?”
FAULTY LOGIC
Whole books and several websites have been devoted to the question of logical fallacies. One that trips up many people is the idea that when two events happen, the first one has caused the second. In Latin, this is called the post hoc, ergo propter hoc fallacy, meaning “after this, therefore because of this.”
The post hoc fallacy is seductive because of what we observe to be true in our daily lives. We step on the car’s accelerator; the car moves forward. We flip on a light switch; the light comes on. We touch a lit match to kindling; the kindling catches fire. We may be disappointed, however, if we assume that the shirt we put on before a successful fishing trip is therefore a “lucky” shirt that will magically produce fish the next time. That’s just superstition. When we fall prey to the post hoc fallacy, we’re like the rooster who thought his crowing made the sun rise.
Consider a statement being made in early 2006 by the Brady Campaign, formerly known as Handgun Control, Inc. The campaign stated that passage of the Brady Law in 1993 (imposing background checks for handgun purchasers) and the federal “assault weapons” ban in 1994 (which banned the sale of certain semi-automatic weapons) were followed by years of declines in violent crime. This the group cited as proof that “gun laws work.”
It’s true that crime rates plunged throughout the 1990s, starting just about the time the two gun laws were enacted. But that doesn’t mean that the gun laws caused the decline. Criminologists, economists, and just about everybody else are still debating over what did. Proposed causes include increased numbers of police, the practice of “community policing,” “zero-tolerance” policies, and even the legalization of abortion two decades earlier. In late 2004, the Committee on Law and Justice of the National Research Council examined the question whether gun laws affect crime rates, and concluded that a connection couldn’t be shown: “In summary, the committee concludes that existing research studies and data…do not credibly demonstrate a causal relationship between the ownership of firearms and the causes or prevention of criminal violence or suicide.”
Be careful about jumping to conclusions. Always ask, “Are these facts really connected?”
And—always—keep asking, “What’s the evidence?”
Chapter 7
Osama, Ollie, and Al
The Internet Solution
SO FAR WE’VE POINTED OUT HOW TO RECOGNIZE SPIN AND MISINFORMATION, explained some of the tricks that spinners use to mislead us, and described the psychological traps that too often make us accomplices in our own deception. We’ve said that staying unspun can save us money, embarrassment, and perhaps even our lives, but that it also requires us to adjust our mental habits so that we look actively for facts that might disprove whatever we happen to believe at the moment, rather than giving in to our hardwired human tendency to see only supporting evidence. And we’ve discussed the basics of how to tell good evidence from random anecdotes. Now it’s time to talk about where, and how, to find the solid facts you need.
The solution to spin is the Internet, if you use it very carefully.
The Wall Street Journal’s personal-technology columnist, Walt Mossberg, may have put it best: “The World Wide Web is a marvelous thing. Because it exists, more people have direct access to more knowledge than at any time in history.” That’s true—and there’s more reliable information being added every day. Furthermore, much of this information is available to everyone, for only the price of an Internet connection.
Unfortunately, as you probably know, the Web is also a conduit for new gushers of toxic informational sludge as well. Anybody can say anything they want on the Internet, regardless of whether it is true, and people can post anonymously or under a false identity. We’ve already mentioned websites that tout fraudulent products, and con artists who use mass e-mailings to reach their victims. The trick is to sort the gold from the dross. We’ll show you how to do that, and perhaps even have some fun along the way.
To illustrate, we offer the story of an Internet hoax that was swallowed by untold thousands of gullible believers—and we show how to find the facts.
Osama, Ollie, and Al
Within weeks after the calamity of September 11, 2001, an e-mail began to circulate containing what the anonymous author described as “stunning” information. He (or she) claimed that Oliver North had warned Co
ngress as far back as 1987 that Osama bin Laden is “the most evil man alive” and had said, “I would recommend that an assassin team be formed to eliminate him and his men from the face of the earth.” Furthermore, this message stated, the senator questioning North was Albert Gore of Tennessee, the future vice president and Democratic candidate for president.
This message was red meat to a lot of conservatives. At a time when President Bush was being criticized for ignoring warnings of a possible terrorist attack, the idea that Al Gore could have prevented 9/11 if only he had listened to a former Ronald Reagan aide was irresistible. The message was forwarded and reforwarded countless times.
The message referred to North’s televised testimony before a Senate committee investigating the Iran-Contra affair. North, now a conservative political commentator, was then a lieutenant colonel in the U.S. Marine Corps and had played a key role in the scandal as a White House military aide involved in secretly aiding the Contra rebels of Nicaragua in their attempt to overthrow the leftist president of Nicaragua, Daniel Ortega. But the message was totally false. In 1987, bin Laden was in Afghanistan fighting the Soviet Union, not the United States. He didn’t form al Qaeda until the following year. Gore didn’t question North: he wasn’t even a member of the Iran-Contra investigating committee. The man questioning North was John Nields, the investigating committee’s lawyer. The security system cost $13,800 (according to North’s subsequent indictment) and not $60,000.
unSpun Page 12
