Why Trust Science?

by Naomi Oreskes

  Mood changes are admittedly a difficult thing to study and almost impossible to quantify. Feelings are, by definition, subjective, and depression cannot be measured in the sort of way that cholesterol or high blood pressure can be. But consider this: in 2016, a clinical trial of a male hormonal contraceptive injection in 320 men was abandoned after the men taking part reported increased incidences of adverse effects, including changes in libido and mood disorders. In fact, more than 20% reported mood disorders. One man developed severe depression; another tried to commit suicide. Because of the adverse effects, the trial was halted—even though the rate of pregnancy suppression was more than 98%. The researchers reported:

  The study regimen led to near-complete and reversible suppression of spermatogenesis. The contraceptive efficacy was relatively good compared with other reversible methods available for men. The frequencies of mild to moderate mood disorders were relatively high.128

  The male hormonal contraceptive injection was shown to work as well as the Pill, yet the clinical trial was stopped because of adverse effects, one of which was a dramatic increase in mood disorders.129 If you are wondering how the researchers measured this, the answer is: self-reporting.

  This result could have been predicted, not only because similar effects were seen in women, but because there is a mechanism that explains why hormonal contraceptives have this effect. It is the link between reproductive hormones and serotonin.

  Low levels of serotonin, a neurotransmitter in the brain, have been linked to depression. High levels of estrogen, as in first-generation [oral contraceptives], and progestin, as in some progestin-only contraceptives, have been shown to lower the brain serotonin levels by increasing the concentration of a brain enzyme that reduces serotonin.130

  The converse is also true: anti-depressant drugs that target serotonin uptake, such as Prozac and Zoloft, are known to have an adverse effect on libido.131 They can also cause erectile dysfunction and anorgasmia; one study in the 1990s found that 45% of female patients on SSRIs (selective serotonin reuptake inhibitors) experienced drug-induced sexual dysfunction and some studies suggest even higher rates.132 This occurs because drugs that stimulate serotonin uptake can interfere with the uptake of hormones involved in sexual desire and reproduction, like dopamine.133 In other words, the issue cuts both ways: drugs that are or target hormones involved in sex can cause depression; drugs that treat depression can affect the hormones involved in sex.

  We have known for fifty years that the Pill can cause mood disorders in women. We know that drugs that treat mood disorders can affect hormones involved with libido, and scientists know at least one mechanism by which this occurs. And a recent study was stopped because hormonal contraceptive caused mood disorders in male subjects. A reasonable person might therefore ask: what was left to be established? Or as my daughter put it, why was the finding that the Pill causes depression in women viewed as news?

  Let us return to the Denmark study. It did not find that previous studies of oral contraceptives had shown that hormonal contraception did not cause mood changes. Rather, it concluded that “inconsistent research methods and lack of uniform assessments [made] it difficult to make strong conclusions about which … users are at risk for adverse mood effects.”134 In other words, it suggested that until now, we didn’t know enough to draw a firm conclusion.

  These researchers took the conventional approach of assuming no effect and requiring statistical proof at a specific significance level to say that an effect had been detected—and was therefore known. So did the various studies that preceded them. There’s nothing particularly shocking about this; it is common statistical practice. But it says, in effect, that if evidence is not available that meets that standard, we must conclude that our results are inconclusive—or in lay terms, that we just don’t know.

  There are two problems with this approach. The first, which is a general one, is that a negative finding is often taken as indicating “no effect,” when in fact it simply means that the researchers have not been able to detect the effect, at least not at a level that achieves statistical significance. (Many negative studies actually do see effects, but not ones that pass the bar of statistical significance at the 95% level.)135 It is the classic conflation of absence of evidence with evidence of absence, and it can lead to false negative conclusions. Still, if enough good studies are done that consistently fail to find an effect (or one really large one with great statistical power), we might fairly conclude that the effect really isn’t there.

  But what if there is evidence from non-statistical sources, such as patient reports, that there may well be an effect? What if there is a theoretical reason (as there is here) to think that an effect is in fact likely? In that case, why are we assuming that there is none? Why are researchers playing dumb? If we know or have reason to suspect that something is a risk, it may be warranted to flip the null and use a default assumption of “effect” rather than “no effect,” or to accept a lower level of statistical significance. (This has sometimes been done, as when the Environmental Protection Agency accepted some studies of the impacts of secondhand smoke at a confidence level of 90% rather than 95% on the grounds that the same chemicals that were known to cause cancer in primary smoke were also present in secondhand smoke.)136 After decades of case reports, and with a mechanism to explain why it might be so, researchers should have accepted the null hypothesis that the Pill could cause depression and sought statistical evidence to disprove that hypothesis.

  The second problem relates to how we think about causation. The classic argument that correlation is not causation is misleading. What we should say is that correlation is not necessarily causation. Many things are correlated that are not causally related. But if we have an observed correlation between two phenomena, and we are aware of a mechanism that explains how one of them can be caused by the other, and if that mechanism is known to be present, then the logical conclusion is the observed correlation is caused by the known mechanism. Under these conditions, correlation is causation. Or at least, it is likely to be.

  A classic example is the correlation between shark attacks and ice cream sales. Statisticians love to use this as an example to prove how correlations can be misleading: both are related to warm weather, when people swim in the ocean and eat ice cream. Neither one causes the other. But what if we had independent evidence that the smell of ice cream attracted sharks? Then it might be the case that there was a causal relation. Now suppose that the correlation did not achieve statistical significance at the 95% level. Would we conclude that there was no relation between the ice cream and the attacks? Under currently prevailing norms, we would. And we would be wrong. We need to pay attention to mechanisms.137

  Consider another example. When the United States reduced the speed limit on interstate highways to fifty-five miles per hour, traffic fatalities dropped dramatically. The motivation for the speed limit change was to save fuel, not lives, so one might initially suppose that this correlation was just coincidental. In fact, driving at lower speeds reduces the chance of an accident and the likelihood that any accident that occurs will be fatal. Because we understand this, we rightly conclude that lowering the speed limit caused a decrease in traffic fatalities.

  Playing dumb makes sense when we have no reason to suspect that phenomena are linked, or have affirmative reason to suppose they are not. If we knew nothing about hormones and mental health we might have rightly said that we needed more evidence to conclude that the Pill might cause depression. But we know that hormones affect brain chemistry. This is one reason why manufacturers have worked to decrease estrogen levels in oral contraceptives.

  Women have always known that we sometimes get moody and depressed right before our periods. Indeed, popular lore makes us unreliable—as scientists, as political leaders, as CEOs—because of this. Stereotypes typically draw the wrong conclusions from the evidence on which they are based, but that in and of itself is not a refutation of the evidence. Hormones affect our
moods. This is true for men and women.

  Doctors who have not warned their patients of this risk during the past thirty years have been ignoring evidence. Public health officials who have downplayed the risk by discounting evidence—in this case, reams of it collected over more than three decades—because it did not meet certain methodological preferences have done women a grave disservice. Had doctors and public health professionals paid more attention to “iffy” case reports instead of discounting them, they would not simply have come to a better conclusion, epistemologically. They would have done their jobs better—and served their patients well—by not discounting a real and troubling side effect of an otherwise desirable medication. Given that the Pill has been implicated in suicidal ideation, they might even have saved lives.

  Example 5: Dental Floss

  My final case involves a very grave public health issue: dental floss.

  Many people have recently heard that flossing your teeth doesn’t do you any good. In August 2016 there was a flurry of coverage saying so. The New York Times asked, “Feeling Guilty about Not Flossing? Maybe There Is No Need.”138 The Los Angeles Times reassured its readers that if they didn’t floss, they needn’t feel bad because it probably doesn’t work anyway.139 So did Mother Jones, which ran the headline, “Guilty No More: Flossing Doesn’t Work.”140 Newsweek asked, “Has the Flossing Myth Been Shattered?”141

  These various reports were based on an article by the Associated Press (AP) that claimed that there is “little proof that flossing works.” The AP quoted National Institutes of Health dentist Tim Iafolla, acknowledging “that if the highest standards of science were applied in keeping with the flossing reviews of the past decade, ‘then it would be appropriate to drop the floss guidelines.’ ”142 The Chicago Tribune linked this latest reversal in scientific fortune to previous (alleged) reversals on salt and fat.143 Evidently, we can add dental floss to the list of issues on which scientists have “got it wrong.”

  It was not just the alleged lack of evidence that caught reporters’ attention; there was also a suggestion of incompetence or even malfeasance. The New York Times suggested that the federal government may have violated the law that stipulates that federal dietary guidelines must be based on scientific evidence. So too the AP: “The federal government has recommended flossing since 1979, first in a surgeon general’s report and later in the Dietary Guidelines for Americans issued every five years. The guidelines must be based on scientific evidence, under the law.”144 The Week ran the story under the headline “Everything You Believed about Flossing Is a Lie.”145 The Detroit News referred to the defenders of floss as the “floss-industrial complex.”146 One website called it “The Great Dental Floss Scam.”147

  Many reports contained an element of schadenfreude: some journalists seemed practically gleeful that journalists had one-upped scientists.148 WRVO, an NPR affiliate in Oswego, New York, ran the story under the headline “How a Journalist Debunked a Decades-Old Health Tip.”149 The report claimed that the story began when AP reporter Jeff Donn learned from his “son’s orthodontist … that there was in fact no good evidence that dental floss helps prevent cavities and gum disease.”150 Poynter.org labeled the story “How a Reporter Took Down Flossing.”151 A website promoting collective consciousness and natural living ran the story under the headline “The Deceit of the Dental Health Industry,” stating that “flossing has been shown to be almost useless in terms of its purported benefits” and suggesting that “most of your oral health is determined by your diet and nutrition.”152

  On the face of it, this certainly appears to be a case of scientists having “got it wrong.” Dentists and public health officials, including those in positions of governmental authority, have been instructing us about something that now we are told is not the case. We have wasted time and money on something useless. And this bears directly on the issue of trust, because if scientists have been wrong for decades about dental floss—as well as perhaps fat and sugar—then what else have they been wrong about? Will they tell us next that it is all right to smoke? Or that climate change is a hoax? Scientists might be tempted to respond that the fracas over flossing is just the messy work of science correcting itself, as a major scientific study revealed the weaknesses in previous work. But that is not what transpired. In fact, this is not a case of scientists getting it wrong at all. It’s a case of journalists getting it wrong, and scientists getting blamed.

  The “scientific” finding was not a scientific finding at all, but the result of an investigation by one reporter for the AP.153 The source of the media story was the media itself. According to their own reporting, which they filed under the rubric “The Big Story,” the AP “looked at the most rigorous research conducted over the past decade, focusing on 25 studies that generally compared the use of a toothbrush with the combination of toothbrushes and floss. The findings? The evidence for flossing is ‘weak, very unreliable,’ of ‘very low’ quality, and carries ‘a moderate to large potential for bias.’ ‘The majority of available studies fail to demonstrate that flossing is generally effective in plaque removal,’ said one review conducted last year. Another academic review, completed in 2015, cites ‘inconsistent/weak evidence’ for flossing and a ‘lack of efficacy.’ ”154

  The New York Times, seemingly staying close to the facts, informed readers that “A review of 12 randomized controlled trials published in The Cochrane Database of Systematic Reviews in 2011 found only ‘very unreliable’ evidence that flossing might reduce plaque after one and three months. Researchers could not find any studies on the effectiveness of flossing combined with brushing for cavity prevention.” (We will address the distinction between gum health and cavity prevention in a moment.) But, as the Times rightly noted, that study was done in 2011, so how and why did this become a story in 2016?

  According to the AP, their investigation of the matter was triggered by a decision by the US government to drop flossing from federal dietary guidelines (and not by Jeff Donn’s conversation with his son’s orthodontist, raising further questions about the whole story). This led them to ask the question: “What were those guidelines based on in the first place?”155 While it was later revealed that the guideline change was the result of a decision to focus the dietary guidelines on diet—i.e., food—rather than other health practices, the cat was out of the bag.156 The “finding” that flossing does not work was all over the news. As for Donn, he was quoted in a subsequent interview: “I think the best science indicates that [by flossing] I’m not doing anything beneficial for my health.”157

  Let us step back from the media coverage to ask: what scientific evidence exists to support or refute the claim that dental floss is of value? Donn is neither a scientist nor a dentist, and in fact his claim is not correct. The available science does not indicate that by flossing we are “not doing anything beneficial” for our health.

  The most well-known and respected source of information on the state of the art in biomedicine is the Cochrane group, a nonprofit collaboration that bills itself as “representing an international gold standard for high quality, trusted information.” The collaboration claims thirty-seven thousand participants from more than 130 countries who “work together to produce credible, accessible health information that is free from commercial sponsorship and other conflicts of interest.”158 As the New York Times correctly reported, in 2011, the collaboration issued a report from its oral health group reviewing existing clinical trials examining the benefits of regular use of dental floss.159

  The report was based on a review of twelve trials, with 582 subjects in flossing-plus-toothbrushing groups and 501 participants in toothbrushing-alone groups. The report summary reads as follows:

  There is some evidence from twelve studies that flossing in addition to tooth-brushing reduces gingivitis compared to tooth-brushing alone. There is weak, very unreliable evidence from 10 studies that flossing plus tooth-brushing may be associated with a small reduction in plaque at 1 and 3 months. No
studies reported the effectiveness of flossing plus tooth-brushing for preventing dental caries [tooth decay].160

  That part of the summary was reported, wholly or in part, in many of the media reports. But the report also said:

  Flossing plus tooth-brushing showed a statistically significant benefit compared to tooth-brushing in reducing gingivitis at the three time points studied [although the effect size was small].161 The 1-month estimate translates to a 0.13 point reduction on a 0 to 3 point scale for … gingivitis … and the 3 and 6 month results translate to 0.20 and 0.09 reductions on the same scale.162

  This additional information refutes much of the media presentation. The crux of the news coverage was that many existing studies are weak, involving small numbers of people or very short periods of time. That is true. But it is not the same as demonstrating that flossing has no benefit. On the contrary, if the Cochrane review is correct, these studies indicate that, over the time period of the study, small but statistically significant reduction of gingivitis was observed in patients who flossed along with brushing.