by Ben Goldacre
If you ever want to spend a chilling afternoon living in the head of an industry whose product has been proven to kill a third of its customers, this is the place for you. ‘The importance of younger adults’ is a tobacco industry paper that uses financial modelling to explain the importance of recruiting teenage smokers to replace the dying older ones before it’s too late, and explains that ‘repeated government studies have shown less than one third of smokers start after age 18 [and] only 5 per cent of smokers start after age 24’. ‘Youth cigarette – new concepts’ from Marketing Innovations Inc. takes these ideas further, into cola- and apple-flavour cigarettes, because ‘apples connote goodness and freshness’.
How much did it matter if the researchers had worked for the tobacco companies? A lot: the risks of Alzheimer’s associated with smoking reported by these papers were on average about a third lower than in those conducted by other researchers, and they produced many papers showing cigarettes were actively protective. If you exclude the eleven papers by researchers associated with the tobacco industry, and look only at the remaining thirty-two, your chances of getting Alzheimer’s as a smoker are vastly higher: for the gamblers out there, comparing a smoker against a non-smoker, the odds of getting Alzheimer’s are higher by 1.72 to 1.
So does that mean we can comfortably ignore all research that comes from people who disgust us? In the 1930s, identifying toxic threats in the environment became an important feature of the Nazi project to build a master race through ‘racial hygiene’. Two researchers, Schairer and Schöniger, were working on biological theories of degenerate behaviour under Professor Karl Astel, a scientist who helped organise the vile ‘euthanasia’ operation that murdered 200,000 mentally and physically disabled people.
In 1943 they published a well-conducted case-control study demonstrating a relationship between smoking and lung cancer, almost a decade before any other researchers elsewhere. Their paper wasn’t mentioned in the classic Doll and Bradford Hill paper of 1950, and if you check in the Science Citation Index, it was referred to only four times in the 1960s, once in the 1970s, and then not again until 1988, despite providing a valuable early warning on a killer that would cause 100 million early deaths in the twentieth century. It’s not obvious what you should do with evidence from untrustworthy sources, but it’s always worth appraising its untrustworthiness with the best tools available.
Foreign Substances in Your Precious Bodily Fluids
Guardian, 9 February 2008
You’ll find fluoride in tea, beer and fish, which might sound like a balanced diet to you. This week the Health Secretary Alan Johnson announced a major new push for putting it in all tap water, with some very grand promises, and in the face of serious opposition.
In Stanley Kubrick’s film Dr Strangelove, General Jack D. Ripper first developed his theories about environmental poisoning and bodily fluids when he experienced impotence, fatigue, and a pervasive sense of emptiness during the physical act of love. He instantly identified the cause: a communist plot to pollute our precious bodily fluids with fluoride.
Bill Etherington MP calls it a ‘poison’. Campaigners say Nazis used it to subdue people in concentration camps. According to the Guardian’s own (sadly departed) alternative health columnist, fluoride is ‘in the same league as lead and arsenic’.
The reality is that anybody making any confident statement about fluoride – positive or negative – is speaking way beyond the evidence. In 1999 the Department of Health commissioned the Centre for Reviews and Dissemination at York University to carry out a systematic review of the evidence on the benefits of fluoridation for dental health, and to look for evidence of harm. Little new work has been done since.
They found 3,200 research papers, mostly of very poor quality. The ones which met the minimum quality threshold suggested that there was vaguely, possibly, around a 15 per cent increase in the number of children without dental caries in areas with fluoridated water, but the studies generally couldn’t exclude other explanations for the variance. Of course, the big idea with fluoride in water is that it can reduce social inequalities in dental health, because everyone drinks it: but there isn’t much evidence on that either – the work is of even poorer quality, and the results are inconsistent.
So when the British Dental Association says there is ‘overwhelming evidence’ that adding fluoride to water helps fight against tooth decay, they’re with General Ripper. And when Alan Johnson says: ‘Fluoridation is an effective and relatively easy way to help address health inequalities, giving children from poorer backgrounds a dental health boost that can last a lifetime,’ he’s really just pushing an admirably old-fashioned line that complex social problems can be addressed with £50 million worth of atoms. The people behind the York review have had to spend a fair amount of time pointing out that people are misrepresenting their work.
But since I’m in the mood for some scaremongering, let’s not forget the potential harms. Fluoridation will give around one in eight people mottled teeth (‘fluorosis’). And there’s something else to worry about, if you like worrying. An observational study from Taiwan found a high incidence of bladder cancer in women from areas where the natural fluoride content in water was high. It might easily have been a chance finding – the study in question measured lots of variables, and if you measure enough things, then some of them are bound to come out positive, just by chance. But it could be real.
The problem here is one of small effect sizes. You don’t need a careful designed study to show that falling out of a plane will probably kill you, but finding a link between fluoride and bladder cancer would be a pig to research, because the effect size is small, the exposure is spread over half a century, and the outcome – bladder cancer – takes a lifetime to reveal itself. Welcome to the finer details behind ‘more research is needed’.
And the fascinating thing about public health is that, with population effects, the numbers can start to get very scary, very quickly: in the UK, for example, just a tiny 10 per cent increase in risk would give you one thousand extra new cases of bladder cancer every year. Fear. Actually, I enjoyed that. Maybe I should move to the Mail.
How Myths Are Made
Guardian, 8 August 2009
Much of what we cover in this column revolves around the idea of a ‘systematic review’, where the literature is surveyed methodically, following a predetermined protocol, to find all the evidence on a given question. As we saw in another column,1 for example, the Soil Association would rather have the freedom to selectively reference only research that supports their case, rather than the totality of the evidence.
Two disturbing news stories demonstrate how this rejection of best practice can also cut to the core of academia.
Firstly, the Public Library of Science in the US this week successfully used a court order to obtain a full trail of evidence showing how pharmaceutical company Wyeth employed commercial ‘ghost writers’ to produce what were apparently academic review articles, published in academic journals, under the names of academic authors. These articles, published between 1998 and 2005, stressed the benefits of taking hormones to protect against problems like heart disease, dementia and ageing skin, while playing down the risks. Stories like this, sadly, are commonplace; but to understand the full damage that these distorted reviews can do, we need to understand a little about the structure of academic knowledge.
In a formal academic paper, every claim is referenced to another academic paper: either an original research paper, describing a piece of primary research in a laboratory or on patients; or a review paper which summarises an area. This convention gives us an opportunity to study how ideas spread, and myths grow, because in theory you could trace who references what, and how, to see an entire belief system evolve from the original data. Such an analysis was published this month in the British Medical Journal, and it is quietly seminal.
Steven Greenberg from Harvard Medical School focused on an arbitrary hypothesis: the specifics are irrelevan
t to us, but his case study was the idea that a protein called β amyloid is produced in the skeletal muscle of patients who have a condition called ‘inclusion body myositis’. Hundreds of papers have been written on this, with thousands of citations between them. Using network theory, Greenberg produced a map of interlocking relationships, to demonstrate who cited what.
By looking at this network of citations he could identify the intersections with the most incoming and outgoing traffic. These are the papers with the greatest ‘authority’ (Google uses the same principle to rank webpages in its search results). All of the ten most influential papers expressed the view that β amyloid is produced in the muscle of patients with IBM. In reality, this is not supported by the totality of the evidence. So how did this situation arise?
Firstly, we can trace how basic laboratory work was referenced. Four lab papers did find β amyloid in IBM patients’ muscle tissue, and these were among the top ten most influential papers. But looking at the whole network, there were also six very similar primary research papers, describing similar lab experiments, which are isolated from the interlocking web of citation traffic, meaning that they received no or few citations. These papers, unsurprisingly, contained data that contradicted the popular hypothesis. Crucially, no other papers refuted or critiqued this contradictory data. Instead, those publications were simply ignored.
Using the interlocking web of citations, you can see how this happened. A small number of review papers funnelled large amounts of traffic through the network, with 63 per cent of all citation paths flowing through one review paper, and 95 per cent of all citation paths flowing through just four review papers by the same research group. These papers acted like a lens, collecting and focusing citations – and scientists’ attention – on the papers supporting the hypothesis, in testament to the power of a well-received review paper.
But Greenberg went beyond just documenting bias in what research was referenced in each review paper. By studying the network, in which review papers are themselves cited by future research papers, he showed how these reviews exerted influence beyond their own individual readerships, and distorted the subsequent discourse, by setting a frame around only some papers.
And by studying the citations in detail, he went further again. Some papers did cite research that contradicted the popular hypothesis, for example, but distorted it. One laboratory paper reported no β amyloid in three of five patients with IBM, and its presence in only a ‘few fibres’ in the remaining two patients; but three subsequent papers cited these data, saying that they ‘confirmed’ the hypothesis. This is an exaggeration at best, but the power of the social network theory approach is to show what happened next: over the following ten years, these three supportive citations were the root of 7,848 supportive citation paths, producing chains of false claim in the network, amplifying the distortion.
Similarly, many papers presented aspects of the β amyloid hypothesis as a theory – but gradually, through incremental mis-statement, in a chain of references, these papers came to be cited as if they proved the hypothesis as a fact, with experimental evidence, which they did not.
This is the story of how myths and misapprehensions arise. Greenberg might have found a mess, but instead he found a web of systematic and self-reinforcing distortion, resulting in the creation of a myth, ultimately retarding our understanding of a disease, and so harming patients. That’s why systematic reviews are important, that’s why incremental mis-statement matters, and that’s why ghost writing should be stopped.
Publish or Be Damned
Guardian, 4 August 2005
I have a very long memory. So often with ‘science by press release’, newspapers will cover a story even though the scientific paper doesn’t exist, assuming it’s around the corner. In February 2004 the Daily Mail was saying that cod liver oil is ‘nature’s superdrug’. The Independent wrote: ‘They’re not yet saying it can enable you to stop a bullet or leap tall buildings, but it’s not far short of that.’ These glowing stories were based on a press release from Cardiff University, describing a study looking at the effect of cod liver oil on some enzymes – no idea which – that have something to do with cartilage – no idea what. I had no way of knowing whether the study was significant, valid or reliable. Nobody did, because it wasn’t published. No methods, results, conclusions to appraise. Nothing.
In 1998 Dr Arpad Pusztai announced through the telly that genetically modified potatoes ‘caused toxicity to rats’. Everyone was extremely interested in this research. So what had he done in his lab? What were they fed? What had he measured? A year later the paper was published, and it was significantly flawed. Nobody had been able to replicate his data and verify the supposed danger of GM, because we hadn’t seen the write-up, the academic paper. How could anyone examine, let alone have a chance to rebut, Pusztai’s claims? Peer review is just the start; then we have open scrutiny by the scientific community, and independent replication.
So anyway, I wrote at the time that these cod liver oil people at Cardiff University were jolly irresponsible, that patients would worry, GPs would have no answers for them, and so on. This week I contacted Cardiff and said: This is what I said last year, now where’s the paper? Prof John Harwood responded through the press office: ‘Mr Goldacre is quite right in asserting that scientists have to be very certain of their facts before making public statements or publishing data.’ I’m a doctor, but it’s good to know we agree. If puzzling.
Bill the pelican being given a dose of cod liver oil by his keeper
‘Because of that,’ continued Prof Harwood, ‘Professor Caterson and my laboratory are continuing to work on samples.’ Right … ‘I’m afraid this takes a long time and much longer than journalists or public relations firms often realise. So, I regret he will have to be patient before Professor Caterson or myself are prepared to comment in detail.’ How kind. And only slightly patronising. I don’t want them to comment on fish oil. It’s seventeen months after ‘nature’s superdrug’: I just want to know where the published paper is.
In 2014, after being patient for a decade as requested, I contacted Prof Caterson and the Cardiff Press Office again. They confirmed that the research has never been published in a journal. Nobody can read or critique the methods or results, and the only public trace is a skeletal description describing a brief conference presentation. This document is four paragraphs long. The press release was seventeen paragraphs long. I’ll try them again in a decade.1
Academic Papers Are Hidden from the Public. Here’s Some Direct Action
Guardian, 3 September 2011
This week George Monbiot won the internet with a long Guardian piece on academic publishers. For those who didn’t know: academics, funded mostly by the public purse, pay for the production and dissemination of academic papers; but for historical reasons, these are published by private organisations who charge around $30 per paper, keeping out any reader who doesn’t have access through their institution.
This is a barrier to the public understanding of science, and also to ongoing scholarship by people who’ve wandered away from institutional academia. There are open-access alternatives, where academics pay up-front and the paper is free to all readers, but these are patchy, and require your funder to pay £1,000 per paper. If the journal your work is best suited for doesn’t do open access, then you might reasonably accept a closed-access journal.
The arguments are big. What I find interesting is the recent rise of direct action on this issue.
Aaron Swartz is a fellow at Harvard’s Center for Ethics, and a digital activist. He has been accused of intellectual property theft on a grand scale, and the federal indictment document, available in full online, describes an inspiringly nerdy game of cat and mouse.
Swartz denies all charges. Allegedly, he bought a laptop to harvest academic papers from the website JSTOR. Using a guest login at MIT – they last fourteen days – he set a program running to download papers in bulk. JSTOR and MIT smelt a rat: t
hey blocked access to whole ranges of computers in MIT, creating havoc. Swartz set two computers on the job, running so fast that several JSTOR servers stopped working.
So then, allegedly, he tried a slower approach. You’ll have seen racks of flashing network equipment in office buildings. He opened one up in a quiet basement, plugged in a laptop with some external hard drives, hid them under a box, and left this package quietly downloading papers by the million. Months later he was seen returning, peering cautiously through cracks in doors, carrying his bicycle helmet over his face and looking through the ventilation holes. He was arrested and bailed for $100,000: he had downloaded 4.8 million academic papers.
It’s hard not to be impressed, and this is not the first time Swartz has taken public data access into his own hands. In the US, court records are available online, but at a cost, in a scheme generating a $150 million budget surplus. When free access was given at seventeen libraries, Swartz set up a script to harvest the lot. He got 19,856,160 pages before the system was shut down.
Now, the US government alleges that Swartz intended to release his vast academic paper stash for free on file-sharing websites. This may be true, but he did not do so. Shortly after his arrest, however, a posting appeared on the Pirate Bay website, declaring the release of an immense file, free for download. It contains thirty-three gigabytes’ worth of academic papers from the UK journal Philosophical Transactions of the Royal Society. The release of this file, explained the poster, was an act of protest at Swartz’s arrest. The papers in it range from the seventeenth century up to 1923, and are mostly out of copyright.
These are, in some respects, remarkable tales of Robin Hood behaviour. JSTOR expended huge effort on scanning these Royal Society papers in the 1990s, when scanning was tougher than it is now, and it should be thanked. But it’s hard to believe we can’t find any better way of allowing public access to such documents: JSTOR sells each paper for between $8 and $19, while the Royal Society estimates that the pay-per-view income from the public accessing them is half of 1 per cent of its journal income.
