2008 - Bad Science

by Ben Goldacre

  How can we explain, then, the apparent fact that industry funded trials are so often so glowing? How can all the drugs simultaneously be better than all of the others? The crucial kludge may happen after the trial is finished.

  Publication bias and suppressing negative results

  ‘Publication bias’ is a very interesting and very human phenomenon. For a number of reasons, positive trials are more likely to get published than negative ones. It’s easy enough to understand, if you put yourself in the shoes of the researcher. Firstly, when you get a negative result, it feels as if it’s all been a bit of a waste of time. It’s easy to convince yourself that you found nothing, when in fact you discovered a very useful piece of information: that the thing you were testing doesn’t work.

  Rightly or wrongly, finding out that something doesn’t work probably isn’t going to win you a Nobel Prize—there’s no justice in the world—so you might feel demotivated about the project, or prioritise other projects ahead of writing up and submitting your negative finding to an academic journal, and so the data just sits, rotting, in your bottom drawer. Months pass. You get a new grant. The guilt niggles occasionally, but Monday’s your day in clinic, so Tuesday’s the beginning of the week really, and there’s the departmental meeting on Wednesday, so Thursday’s the only day you can get any proper work done, because Friday’s your teaching day, and before you know it, a year has passed, your supervisor retires, the new guy doesn’t even know the experiment ever happened, and the negative trial data is forgotten forever, unpublished. If you are smiling in recognition at this paragraph, then you are a very bad person.

  Even if you do get around to writing up your negative finding, it’s hardly news. You’re probably not going to get it into a big-name journal, unless it was a massive trial on something everybody thought was really whizzbang until your negative trial came along and blew it out of the water, so as well as this being a good reason for you not bothering, it also means the whole process will be heinously delayed: it can take a year for some of the slacker journals to reject a paper. Every time you submit to a different journal you might have to re-format the references (hours of tedium). If you aim too high and get a few rejections, it could be years until your paper comes out, even if you are being diligent: that’s years of people not knowing about your study.

  Publication bias is common, and in some fields it is more rife than in others. In 1995, only 1 per cent of all articles published in alternative medicine journals gave a negative result. The most recent figure is 5 per cent negative. This is very, very low, although to be fair, it could be worse. A review in 1998 looked at the entire canon of Chinese medical research, and found that not one single negative trial had ever been published. Not one. You can see why I use CAM as a simple teaching tool for evidence-based medicine.

  Generally the influence of publication bias is more subtle, and you can get a hint that publication bias exists in a field by doing something very clever called a funnel plot. This requires, only briefly, that you pay attention.

  If there are lots of trials on a subject, then quite by chance they will all give slightly different answers, but you would expect them all to cluster fairly equally around the true answer. You would also expect that the bigger studies, with more participants in them, and with better methods, would be more closely clustered around the correct answer than the smaller studies: the smaller studies, meanwhile, will be all over the shop, unusually positive and negative at random, because in a study with, say, twenty patients, you only need three freak results to send the overall conclusions right off.

  A funnel plot is a clever way of graphing this. You put the effect (i.e., how effective the treatment is) on the x-axis, from left to right. Then, on the y-axis (top-to-bottom, maths-skivers) you put how big the trial was, or some other measure of how accurate it was. If there is no publication bias, you should see a nice inverted funnel: the big, accurate trials all cluster around each other at the top of the funnel, and then as you go down the funnel, the little, inaccurate trials gradually spread out to the left and right, as they become more and more wildly inaccurate—both positively and negatively.

  If there is publication bias, however, the results will be skewed. The smaller, more rubbish negative trials seem to be missing, because they were ignored—nobody had anything to lose by letting these tiny, unimpressive trials sit in their bottom drawer—and so only the positive ones were published. Not only has publication bias been demonstrated in many fields of medicine, but a paper has even found evidence of publication bias in studies of publication bias. Here is the funnel plot for that paper. This is what passes for humour in the world of evidence-based medicine.

  The most heinous recent case of publication bias has been in the area of SSRI antidepressant drugs, as has been shown in various papers. A group of academics published a paper in the New England Journal of Medicine at the beginning of 2008 which listed all the trials on SSRIs which had ever been formally registered with the FDA, and examined the same trials in the academic literature. Thirty-seven studies were assessed by the FDA as positive: with one exception, every single one of those positive trials was properly written up and published. Meanwhile, twenty-two studies that had negative or iffy results were simply not published at all, and eleven were written up and published in a way that described them as having a positive outcome.

  This is more than cheeky. Doctors need reliable information if they are to make helpful and safe decisions about prescribing drugs to their patients. Depriving them of this information, and deceiving them, is a major moral crime. If I wasn’t writing a light and humorous book about science right now, I would descend into gales of rage.

  Duplicate publication

  Drug companies can go one better than neglecting negative studies. Sometimes, when they get positive results, instead of just publishing them once, they publish them several times, in different places, in different forms, so that it looks as if there are lots of different positive trials. This is particularly easy if you’ve performed a large ‘multicentre’ trial, because you can publish overlapping bits and pieces from each centre separately, or in different permutations. It’s also a very clever way of kludging the evidence, because it’s almost impossible for the reader to spot.

  A classic piece of detective work was performed in this area by a vigilant anaesthetist from Oxford called Martin Tramer, who was looking at the efficacy of a nausea drug called ondansetron. He noticed that lots of the data in a meta-analysis he was doing seemed to be replicated: the results for many individual patients had been written up several times, in slightly different forms, in apparently different studies, in different journals. Crucially, data which showed the drug in a better light were more likely to be duplicated than the data which showed it to be less impressive, and overall this led to a 23 per cent overestimate of the drug’s efficacy.

  Hiding harm

  That’s how drug companies dress up the positive results. What about the darker, more headline-grabbing side, where they hide the serious harms?

  Side-effects are a fact of life: they need to be accepted, managed in the context of benefits, and carefully monitored, because the unintended consequences of interventions can be extremely serious. The stories that grab the headlines are ones where there is foul play, or a cover-up, but in fact important findings can also be missed for much more innocent reasons, like the normal human processes of accidental neglect in publication bias, or because the worrying findings are buried from view in the noise of the data.

  Anti-arrhythmic drugs are an interesting example. People who have heart attacks get irregular heart rhythms fairly commonly (because bits of the timekeeping apparatus in the heart have been damaged), and they also commonly die from them. Anti-arrhythmic drugs are used to treat and prevent irregular rhythms in people who have them. Why not, thought doctors, just give them to everyone who has had a heart attack? It made sense on paper, they seemed safe, and nobody knew at the time that they would actually increase the
risk of death in this group—because that didn’t make sense from the theory (like with antioxidants). But they do, and at the peak of their use in the 1980s, anti-arrhythmic drugs were causing comparable numbers of deaths to the total number of Americans who died in the Vietnam war. Information that could have helped to avert this disaster was sitting, tragically, in a bottom drawer, as a researcher later explained:

  When we carried out our study in 1980 we thought that the increased death rate…was an effect of chance…The development of [the drug] was abandoned for commercial reasons, and so this study was therefore never published; it is now a good example of ‘publication bias’. The results described here…might have provided an early warning of trouble ahead.

  That was neglect, and wishful thinking. But sometimes it seems that dangerous effects from drugs can be either deliberately downplayed or, worse than that, simply not published.

  There has been a string of major scandals from the pharmaceutical industry recently, in which it seems that evidence of harm for drugs including Vioxx and the SSRI antidepressants has gone missing in action. It didn’t take long for the truth to out, and anybody who claims that these issues have been brushed under the medical carpet is simply ignorant. They were dealt with, you’ll remember, in the three highest-ranking papers in the BMJ’s archive. They are worth looking at again, in more detail.

  Vioxx

  Vioxx was a painkiller developed by the company Merck and approved by the American FDA in 1999. Many painkillers can cause gut problems—ulcers and more—and the hope was that this new drug might not have such side-effects. This was examined in a trial called VIGOR, comparing Vioxx with an older drug, naproxen: and a lot of money was riding on the outcome. The trial had mixed results. Vioxx was no better at relieving the symptoms of rheumatoid arthritis, but it did halve the risk of gastrointestinal events, which was excellent news. But an increased risk of heart attacks was also found.

  When the VIGOR trial was published, however, this cardiovascular risk was hard to see. There was an ‘interim analysis’ for heart attacks and ulcers, where ulcers were counted for longer than heart attacks. It wasn’t described in the publication, and it overstated the advantage of Vioxx regarding ulcers, while understating the increased risk of heart attacks. ‘This untenable feature of trial design,’ said a swingeing and unusually critical editorial in the New England Journal of Medicine, ‘which inevitably skewed the results, was not disclosed to the editors or the academic authors of the study.’ Was it a problem? Yes. For one thing, three additional myocardial infarctions occurred in the Vioxx group in the month after they stopped counting, while none occurred in the naproxen control group.

  An internal memo from Edward Scolnick, the company’s chief scientist, shows that the company knew about this cardiovascular risk (‘It is a shame but it is a low incidence and it is mechanism based as we worried it was’). The New England Journal of Medicine was not impressed, publishing a pair of spectacularly critical editorials.

  The worrying excess of heart attacks was only really picked up by people examining the FDA data, something that doctors tend—of course—not to do, as they read academic journal articles at best. In an attempt to explain the moderate extra risk of heart attacks that could be seen in the final paper, the authors proposed something called ‘the naproxen hypothesis’: Vioxx wasn’t causing heart attacks, they suggested, but naproxen was preventing them. There is no accepted evidence that naproxen has a strong protective effect against heart attacks.

  The internal memo, discussed at length in the coverage of the case, suggested that the company was concerned at the time. Eventually more evidence of harm emerged. Vioxx was taken off the market in 2004; but analysts from the FDA estimated that it caused between 88,000 and 139,000 heart attacks, 30 to 40 per cent of which were probably fatal, in its five years on the market. It’s hard to be sure if that figure is reliable, but looking at the pattern of how the information came out, it’s certainly felt, very widely, that both Merck and the FDA could have done much more to mitigate the damage done over the many years of this drug’s lifespan, after the concerns were apparent to them. Data in medicine is important: it means lives. Merck has not admitted liability, and has proposed a $4.85 billion settlement in the US.

  Authors forbidden to publish data

  This all seems pretty bad. Which researchers are doing it, and why can’t we stop them? Some, of course, are mendacious. But many have been bullied or pressured not to reveal information about the trials they have performed, funded by the pharmaceutical industry.

  Here are two extreme examples of what is, tragically, a fairly common phenomenon. In 2000, a US company filed a claim against both the lead investigators and their universities in an attempt to block publication of a study on an HIV vaccine that found the product was no better than placebo. The investigators felt they had to put patients before the product. The company felt otherwise. The results were published in JAMA that year.

  In the second example, Nancy Olivieri, director of the Toronto Haemoglobinopathies Programme, was conducting a clinical trial on deferiprone, a drug which removes excess iron from the bodies of patients who become iron-overloaded after many blood transfusions. She was concerned when she saw that iron concentrations in the liver seemed to be poorly controlled in some of the patients, exceeding the safety threshold for increased risk of cardiac disease and early death. More extended studies suggested that deferiprone might accelerate the development of hepatic fibrosis.

  The drug company, Apotex, threatened Olivieri, repeatedly and in writing, that if she published her findings and concerns they would take legal action against her. With great courage—and, shamefully, without the support of her university—Olivieri presented her findings at several scientific meetings and in academic journals. She believed she had a duty to disclose her concerns, regardless of the personal consequences. It should never have been necessary for her to need to make that decision.

  The single cheap solution that will solve all of the problems in the entire world

  What’s truly extraordinary is that almost all of these problems—the suppression of negative results, data dredging, hiding unhelpful data, and more—could largely be solved with one very simple intervention that would cost almost nothing: a clinical trials register, public, open, and properly enforced. This is how it would work. You’re a drug company. Before you even start your study, you publish the ‘protocol’ for it, the methods section of the paper, somewhere public. This means that everyone can see what you’re going to do in your trial, what you’re going to measure, how, in how many people, and so on, before you start.

  The problems of publication bias, duplicate publication and hidden data on side-effects—which all cause unnecessary death and suffering—would be eradicated overnight, in one fell swoop. If you registered a trial, and conducted it, but it didn’t appear in the literature, it would stick out like a sore thumb. Everyone, basically, would assume you had something to hide, because you probably would. There are trials registers at present, but they are a mess.

  How much of a mess is illustrated by this last drug company ruse: ‘moving the goalposts’. In 2002 Merck and Schering Plough began a trial to look at Ezetimibe, a drug to reduce cholesterol. They started out saying they were going to measure one thing as their test of whether the drug worked, but then announced, after the results were in, that they were going to count something else as the real test instead. This was spotted, and they were publicly rapped. Why? Because if you measure lots of things (as they did), some might be positive simply by chance. You cannot find your starting hypothesis in your final results. It makes the stats go all wonky.

  Adverts

  ‘Clomicalm tablets are the only medication approved for the treatment of separation anxiety in dogs.’

  There are currently no direct-to-consumer drug adverts in Britain, which is a shame, because the ones in America are properly bizarre, especially the TV ones. Your life is in disarray, your restless legs⁄migrai
ne⁄cholesterol have taken over, all is panic, there is no sense anywhere. Then, when you take the right pill, suddenly the screen brightens up into a warm yellow, granny’s laughing, the kids are laughing, the dog’s tail is wagging, some nauseating child is playing with the hose on the lawn, spraying a rainbow of water into the sunshine whilst absolutely laughing his head off as all your relationships suddenly become successful again. Life is good.

  Patients are so much more easily led than doctors by drug company advertising that the budget for direct-to-consumer advertising in America has risen twice as fast as the budget for addressing doctors directly. These adverts have been closely studied by medical academic researchers, and have been repeatedly shown to increase patients’ requests for the advertised drugs, as well as doctors’ prescriptions for them. Even adverts ‘raising awareness of a condition’ under tighter Canadian regulations have been shown to double demand for a specific drug to treat that condition.

  This is why drug companies are keen to sponsor patient groups, or to exploit the media for their campaigns, as has been seen recently in the news stories singing the praises of the breast cancer drug Herceptin, or Alzheimer’s drugs of borderline efficacy.

  These advocacy groups demand vociferously in the media that the companies’ drugs be funded by the NHS. I know people associated with these patient advocacy groups—academics—who have spoken out and tried to change their stance, without success: because in the case of the British Alzheimer’s campaign in particular, it struck many people that the demands were rather one-sided. The National Institute for Clinical Excellence (NICE) concluded that it couldn’t justify paying for Alzheimer’s drugs, partly because the evidence for their efficacy was weak, and often looked only at soft, surrogate outcomes. The evidence is indeed weak, because the drug companies have failed to subject their medications to sufficiently rigorous testing on real-world outcomes, rigorous testing that would be much less guaranteed to produce a positive result. Does the Alzheimer’s Society challenge the manufacturers to do better research? Do its members walk around with large placards campaigning against ‘surrogate outcomes in drugs research’, demanding ‘More Fair Tests’? No.