by Ben Goldacre
Then, almost two years into this process, the EMA changed tack. Suddenly, it began to argue that study reports contain personal data about individual patients. This argument hadn’t been raised by the EMA before, but it’s also not true. There may have been some information in some whole sections of the study reports that gave detail on some individual participants’ odd presentations, or on possible side effects, but these were all in the same appendix, and could easily be removed.
The EU Ombudsman’s conclusions were clear: the EMA had failed in its duty to give an adequate or even coherent explanation of why it was refusing access to this important information. He made a preliminary finding of maladministration. After doing that, he was under no obligation to offer any further opinion on the weak excuses offered by the EMA, but he decided to do so anyway. His report is damning. The EMA had flatly failed to address a serious charge that its withholding information about these trials was against the public interest, and exposed patients to harm. The Ombudsman also described how he had gone through the study reports in detail himself, and had found that they contained neither any commercially confidential information, nor any details about the commercial development of the drugs. The EMA’s claims that answering the request would have put it under an inappropriate administrative burden were untrue, and it had overestimated the work this would have involved: specifically, he explained, removing any personal data, where it did sometimes appear, would be easy.
The Ombudsman told the EMA to hand over the data, or produce a convincing explanation of why it shouldn’t. Amazingly, the EMA, the drugs regulator covering the whole of Europe, still refused to hand over the documents. During this delay, people certainly suffered unnecessarily, and some possibly also died, simply for want of information. But the behaviour of the EMA then deteriorated even further, into the outright surreal. Any scrap of the company’s thinking about how to run the trial, it argued, which could be intuited from reading the study reports and protocols, was commercially sensitive with respect to its thoughts and plans. This was true, the EMA said, even though the drugs were already on the market, and the information was from final clinical trials, at the very end of the commercial drug-development process. The researchers responded that this was perverse: they knew that withheld data is usually negative, so if anything, any other company seeing negative data about these drugs would be less likely to try to get a competitor to market, if it appeared that the benefits of the drugs were more modest than had been thought.
That wasn’t the end of it. The EMA also grandly dismissed the idea that lives were at risk, stating that the burden of proof was on the researchers to demonstrate this. To me this is – if you can forgive me – a slightly contemptuous attitude, especially given what happens in the next paragraph. It’s plainly true that if doctors and patients are unable to see which is the best treatment, then they will make worse decisions, exposing patients to unnecessary harm. Furthermore, it is obvious that larger numbers of academics making transparent judgements about publicly accessible trial data are a much more sensible way of determining the risks and benefits of an intervention than a brief blanket ‘yes or no’ edict and summary from a regulator. This is all true of drugs like orlistat and rimonabant, but it’s also true of any drug, and we will see many cases where academics spotted problems with drugs that regulators had missed.
Then, in 2009, one of the two drugs, rimonabant, was taken off the market, on the grounds that it increases the risk of serious psychiatric problems and suicide. This, at the same time that the EMA was arguing that researchers were wrong to claim that withholding information was harming patients.
And then the EMA suddenly claimed that the design of a randomised trial itself is commercially confidential information.
In case it needs reiterating, let me remind you that the first trial appears in the Bible, Daniel 1:12, and although the core ideas have certainly been refined over time, all trials are essentially identical experiments, generic across any field, with the basics of a modern trial sketched out at least half a century ago. There is absolutely no earthly sense in which anyone could realistically claim that the design of a randomised controlled trial is a commercially confidential or patentable piece of intellectual property.
This was now a farce. The researchers opened all barrels. The EMA was in breach of the Declaration of Helsinki, the international code of medical ethics, which states that everyone involved in research has a duty to make trial findings public. The researchers knew that published papers give a flattering subset of the trial data, and so did the EMA. Patients would die if the EMA continued to withhold data. There was nothing of serious commercial value in there. The EMA’s brief public summaries of the data were inaccurate. The EMA was complicit in the exploitation of patients for commercial gain.
It was now August 2009, and the researchers had been fighting for more than two years for access to data on two widely prescribed drugs, from the very organisation that is supposed to be protecting patients and the public. They weren’t alone. The French ‘prescribers’ bulletin’ Prescrire was trying at the same time to get the EMA’s documents on rimonabant. It was sent some unhelpful documents, including the rather remarkable ‘Final Assessment Report’, from the Swedish agency that had handled the drug’s approval much earlier. You can read this PDF in full online. Or rather, you can’t. You can see below exactly what the scientific analysis of the drug looked like – the document the EMA sent to one of France’s most respected journals for doctors.69 I think it tells a rather clear story, and to add to the insult, there are sixty pages of this in total.
In the meantime, the Danish Medical Authority had handed over fifty-six study reports to Cochrane (though it still needed more from the EMA); a complaint from the drug company about this had been rejected by the Danish government, which had seen no problem with commercial information (there was none), nor administrative hassle (it was minimal), nor the idea that the design of a randomised trial is commercial information (which is laughable). This was chaos. The EMA – which you may remember was responsible for EudraCT, the transparency tool that is held in secret – was going out on a very peculiar limb. It seemed that it would do anything it could to withhold this information from doctors and patients. As we shall see, sadly, this level of secrecy is its stock in trade.
But now we come to the end of this particular road for the EMA. It handed the full, final study reports over to the Ombudsman, reminding him that even the table of contents for each was commercial. Once he had had these documents in his hand, his final opinion came swiftly. There was no commercial data here. There was no confidential patient information here. Hand it over to the public, now. The EMA, at glacial pace, agreed to set a deadline for delivering the data to the researchers, doctors and patients who needed it. The Ombudsman’s final ruling was published at the end of November 2010.70 The initial complaint was made in June 2007. That is a full three and a half years of fighting, obstruction and spurious arguments from the EMA, during which one of the drugs had to be removed from the market because it was harming patients.
After this precedent had been set, the Cochrane researchers realised they were in a good position to apply for more study reports, so they set about doing so. The first area in which they tried to get more documents was antidepressants. It’s a good place to start, since these drugs have been the focus of some particularly bad behaviour over the years (though we should remember that missing trial data pervades every corner of medicine). What happened next was even more bizarre than the EMA’s three-year battle to withhold information on orlistat and rimonabant.
The researchers put in their request to the EMA, but were told that the drugs had been approved back in the era when marketing authorisations were given out by individual countries, rather than by the EMA centrally. These local authorisations were then ‘copied’ to all other nations. The MHRA, the UK drugs regulator, held the information the researchers wanted, so they would have to approach it for a copy. The researchers d
utifully wrote to the MHRA, asked for the reports on a drug called fluxetine, and then waited patiently. Finally the answer came: the MHRA explained that it would be happy to hand over this information, but there was a problem.
The documents had all been shredded.71
This was in keeping, it explained, with the agency’s retention policy, which was that such documents were only kept if they were of particular scientific, historical or political interest, and the files didn’t meet those criteria. Let’s just take a moment to think through what the criteria must be. The SSRI antidepressant drugs have seen many scandals on hidden data, and that should be enough on its own, but if you think back to the beginning of this chapter, one of them – paroxetine – was involved in an unprecedented, four-year-long investigation into whether criminal charges could be brought against GSK. That investigation into paroxetine was the largest investigation that the MHRA has ever conducted into drug safety: in fact, it was the largest investigation the MHRA has ever conducted, of any kind, ever. Quite apart from that, these original study reports contain vitally important data on safety and efficacy. But the MHRA shredded them all the same, feeling that they were not of sufficient scientific, historical or political interest.
I can only leave you there.
Where are we so far?
The missing data saga is long and complex, and involves patients around the world being put at risk, and some major players which have let us down to an extraordinary extent. Since we are almost at the end of it, this is a good moment to gather together what we’ve seen so far.
Trials are frequently conducted, but then left unpublished, and so are unavailable to doctors and patients. Only half of all trials get published, and those with negative results are twice as likely to go missing as those with positive ones. This means that the evidence on which we base decisions in medicine is systematically biased to overstate the benefits of the treatments we give. Since there is no way to compensate for this missing data, we have no way of knowing the true benefits and risks of the drugs doctors prescribe.
This is research misconduct on a grand, international scale. The reality of the problem is universally recognised, but nobody has bothered to fix it:
Ethics committees allow companies and individual researchers with a track record of unpublished data to do more trials on human participants.
University administrators and ethics committees permit contracts with industry that explicitly say the sponsor can control the data.
Registers have never been enforced.
Academic journals continue to publish trials that were unregistered, despite pretending that they won’t.
Regulators have information that would be vital to improve patient care, but are systematically obstructive:
they have poor systems to share poor summaries of the information that they do have;
and they are bizarrely obstructive of people asking for more information.
Drug companies hold trial results that even regulators don’t see.
Governments have never implemented laws compelling companies to publish data.
Doctors’ and academics’ professional bodies have done nothing.
What to do about all this?
You will need to wait, because in the next section there are even greater horrors to come.
Trying to get trial data from drug companies: the story of Tamiflu
Governments around the world have spent billions of pounds on stockpiling a drug called Tamiflu. In the UK alone we have spent several hundred million pounds – the total figure is not yet clear – and so far we’ve bought enough tablets to treat 80 per cent of the population if an outbreak of bird flu occurs. I’m very sorry if you have the flu, because it’s horrid being ill; but we have not spent all this money to reduce the duration of your symptoms in the event of a pandemic by a few hours (though Tamiflu does do this, fairly satisfactorily). We have spent this money to reduce the rate of ‘complications’: a medical euphemism, meaning pneumonia and death.
Lots of people seem to think Tamiflu will do this. The US Department of Health and Human Services said it would save lives and reduce hospital admissions. The European Medicines Agency said it would reduce complications. The Australian drugs regulator said so too. Roche’s website said it reduces complications by 67 per cent.73 But what is the evidence that Tamiflu really will reduce complications? Answering questions like this is the bread and butter of the Cochrane Collaboration, which you will remember is the vast and independent nonprofit international collaboration of academics producing hundreds of systematic reviews on important questions in medicine every year. In 2009 there was concern about a flu pandemic, and an enormous amount of money was being spent on Tamiflu. Because of this, the UK and Australian governments specifically asked the Cochrane Respiratory Diseases Group to update its earlier reviews on the drug.
Cochrane reviews are in a constant review cycle, because evidence changes over time as new trials are published. This should have been a pretty everyday piece of work: the previous review, in 2008, had found some evidence that Tamiflu does indeed reduce the rate of complications. But then a Japanese paediatrician called Keiji Hayashi left a comment that would trigger a revolution in our understanding of how evidence-based medicine should work. This wasn’t in a publication, or even a letter: it was a simple online comment, posted underneath the Tamiflu review on the Cochrane website.
You’ve summarised the data from all the trials, he explained, but your positive conclusion is really driven by data from just one of the papers you cite, an industry-funded meta-analysis led by an author called Kaiser. This, ‘the Kaiser paper’, summarises the findings of ten earlier trials, but from these ten trials, only two have ever been published in the scientific literature. For the remaining eight, your only information comes from the brief summary in this secondary, industry source. That’s not reliable enough.
In case it’s not immediately obvious, this is science at its best. The Cochrane review is readily accessible online; it explains transparently the methods by which it looked for trials, and then analysed them, so any informed reader can pull the review apart, and understand where the conclusions came from. Cochrane provides an easy way for readers to raise criticisms. And, crucially, these criticisms did not fall on deaf ears. Tom Jefferson is the head of the Cochrane Respiratory Group, and the lead author on the 2008 review. He realised immediately that he had made a mistake in blindly trusting the Kaiser data. He said so, without any defensiveness, and then set about getting the information in a straight, workpersonlike fashion. This began a three-year battle, which is still not resolved, but which has thrown stark light on the need for all researchers to have access to clinical study reports on trials wherever possible.
First, the Cochrane researchers wrote to the authors of the Kaiser paper, asking for more information. In reply, they were told that this team no longer had the files, and that they should contact Roche, the manufacturer of Tamiflu. So naturally they wrote to Roche and asked for the data.
This is where the problems began. Roche said it would hand over some data, but the Cochrane reviewers would need to sign a confidentiality agreement. This was an impossibility for any serious scientist: it would prevent them from conducting a systematic review with any reasonable degree of openness and transparency. More than this, the proposed contract also raised serious ethical issues, in that it would have required the Cochrane team to actively withhold information from the reader: it included a clause saying that on signing it, the reviewers would never be allowed to discuss the terms of this secrecy agreement; and more than that, they would be forbidden from ever publicly acknowledging that it even existed. Roche was demanding a secret contract, with secret terms, requiring secrecy about trial data, in a discussion about the safety and efficacy of a drug that has been taken by hundreds of thousands of people around the world. Jefferson asked for clarification, and never received a reply.
Then, in October 2009, the company changed tack: they would li
ke to hand the data over, they explained, but another meta-analysis was being conducted elsewhere. Roche had given them the study reports, so Cochrane couldn’t have them. This was a simple non-sequitur: there is no reason why many groups shouldn’t all work on the same question. In fact, quite the opposite: replication is the cornerstone of good science. Roche’s excuse made no sense. Jefferson asked for clarification, but never received a reply.
Then, one week later, unannounced, Roche sent seven short documents, each around a dozen pages long. These contained excerpts of internal company documents on each of the clinical trials in the Kaiser meta-analysis. This was a start, but it didn’t contain anything like enough information for Cochrane to assess the benefits, or the rate of adverse events, or fully to understand exactly what methods were used in the trials.
At the same time, it was rapidly becoming clear that there were odd inconsistencies in the information on this drug. Firstly, there was considerable disagreement at the level of the broad conclusions drawn by people who apparently had access to different data. The FDA said there were no benefits on complications, while the Centers for Disease Control and Prevention (in charge of public health in the USA – they wear nice naval uniforms in honour of their history on the docks) said it did reduce complications. The Japanese regulator made no claim for complications, but the EMA said there was a benefit. In a sensible world, all these organisations would sing from the same hymn sheet, because all would have access to the same information.
Reflecting this pattern, Roche’s own websites said completely different things in different jurisdictions, depending on what the local regulator had said. It’s naïve, perhaps, to expect consistency from a drug company, but from this and other stories it’s clear that industry utterances are driven by the maximum they can get away with in each territory, rather than any consistent review of the evidence.
