2008 - Bad Science

by Ben Goldacre

  You might feel, after the ‘trial’ nonsense, that we should be cautious about accepting Durham and Equazen’s appraisal of their own work, but I’d be suspicious of the claims from a lot of serious academics in exactly the same way (they would welcome such suspicion, and I’d be able to read the research evidence they were drawing on). I asked Equazen for their twenty positive studies, and was told I would have to sign a confidentiality agreement to see them. That’s a confidentiality agreement to review the research evidence for widely promoted claims, going back years, in the media and by Durham Council employees, about a very controversial area of nutrition and behaviour, of huge interest to the public, and about experiments conducted—forgive me if I’m getting sentimental here—on our schoolchildren. I refused.*

  ≡ Although you wouldn’t know if I had signed, since I wouldn’t be able to tell you.

  Meanwhile, all over the newspapers and television, everywhere you looked, going back to at least 2002, there were reports of positive fish-oil trials in Durham, using Equazen products. There seemed to have been half a dozen of these trials, in various locations, performed by Durham Council staff on Durham state school children, ytt there was no sign of anything in the scientific literature (beyond one study by a researcher from Oxford, which was on children with developmental coordination disorder). There were wildly enthusiastic Durham Council press releases that talked about positive trials, sure. There were Madeleine Portwood interviews with the press, in which she talked enthusiastically about the positive results (and talked too about how the fish oil was improving children’s skin conditions, and other problems)—but no published studies.

  I contacted Durham. They put me on to Madeleine Port-wood, the scientific brains behind this enormous and enduring operation. She appears regularly on telly talking about fish oils, using inappropriately technical terms like ‘limbic’ to a lay audience. ‘It sounds complicated,’ say TV presenters, ‘but the science says…’ Portwood is evidently very enthusiastic about talking to parents and journalists, but she did not return my calls. The press office took a week to reply to my emails. I asked for details of the studies they had performed, or were performing. The responses appeared to be inconsistent with the media coverage. At least one trial seemed to be missing. I asked for methodological details of the studies they were doing, and results of the ones that were completed. Not until we publish, they said.

  Equazen and Durham Council had coached, preened and spoon-fed a huge number of journalists over the years, giving them time and energy; as far as I can see there is only one difference between me and those reporters: from what they wrote, they clearly know very little about trial design, whereas I know, well, a fair bit (and now so do you).

  Meanwhile I kept being referred to the durhamtrials.org website, as if this contained useful data. It had evidently bamboozled many journalists and parents before me, and it’s linked to by many news stories and Equazen adverts. But as a source of information about the ‘trials’, this site is a perfect illustration of why you should publish a trial before you make any dramatic claims about the results. It’s hard to tell what’s there. The last time I looked, there was some data borrowed from a proper trial published elsewhere by some Oxford researchers (which happened to be done in Durham), but other than that, no sign of Durham’s own placebo-controlled trials which kept appearing in the news. There were plenty of complicated-looking graphs, but they seem to be on special Durham ‘trials’, with no placebo-control group. They seem to describe improvements, with sciencey-looking graphics to illustrate them, but there are no statistics to say if the changes were statistically significant.

  It’s almost impossible to express just how much data is missing from this site, and how useless that renders what is there. As an example, there is a ‘trial’ with its results reported in a graph, but nowhere on the entire site, as far as I can see, does it tell you how many children were in the study reported. It’s hard to think of a single more basic piece of information than that. But you can find plenty of gushing testimonials that wouldn’t be out of place on an alternative therapists’ website selling miracle cures. One child says: ‘Now I am not so interested in the TV. I just like reading books. The best place in all the world is the library. I absolutely love it.’

  I felt the public deserved to know what had been done in these trials. This was probably the most widely reported clinical trial of the past few years, it was a matter of great public interest, and the experiments were performed on children by public servants. So I made a Freedom of Information Act request for the information you would need to know about a trial: what was done, who were the children, what measurements were taken, and so on. Everything, in fact, from the standardised and very complete ‘CONSORT’ guidelines which describe best practice in writing up trial results. Durham Council refused, on grounds of cost.

  So I got readers of the column to ask for little bits of information, so that none of us was asking for anything very expensive. We were accused of running a vexatious ‘campaign’ of ‘harassment’. The head of the council complained about me to the Guardian. I was eventually told that my questions might be answered, if I travelled 275 miles north to Durham in person. Various readers appealed; they were told that much of the information they had been refused did not in any case exist.

  Finally, in February 2008, after a disappointing fall in the rate of improvement in GCSE results, the council announced that there had never been any intention of measuring exam performance. This surprised even me. To be scrupulously precise, what they said, in answer to a written question from an indignant retired headmaster, was this: ‘As we have said previously it was never intended, and the County Council never suggested, that it would use this initiative to draw conclusions about the effectiveness or otherwise of using Fish Oil to boost exam results.’

  To say that this contradicts their earlier claims would be something of an understatement. In a Daily Mail article from 5 September 2006 headlined ‘Fish Oil Study Launched to Improve GCSE Grades’ Dave Ford, the council’s Chief Schools Inspector, said: ‘We will be able to track pupils’ progress and measure whether their attainments are better than their predicted scores.’ Dr Madeleine Portwood, senior educational psychologist running the ‘trial’, said: ‘Previous trials have shown remarkable results and I am confident that we will see marked benefits in this one as well.’

  Durham county council’s own press release from the beginning of the ‘trial’ reads: ‘Education chiefs in County Durham are to mount a unique back-to-school initiative today which they believe could result in record GCSE pass levels next summer.’ It reports that children are being given pills ‘to see whether the proven benefits it has already brought children and young people in earlier trials can boost exam performances too’. The council’s Chief Schools Inspector is ‘convinced’ that these pills ‘could have a direct impact on their GCSE results…the county-wide trial will continue until the pupils complete their GCSE examinations next June, and the first test of the supplement’s effectiveness will be when they sit their ‘mock’ exams this December’. ‘We are able to track pupils’ progress and we can measure whether their attainments are better than their predicted scores,’ says Dave Ford in the press release for the trial which, we are now told, was not a trial, and was never intended to collect any data on exam results. It was with some astonishment that I also noticed that they had changed their original press release on the Durham website, and removed the word ‘trial’.

  Why is all this important? Well, firstly, as I have said, this was the most well-reported trial of that year, and the fact that it was such a foolish exercise could only undermine the public’s understanding of the very nature of evidence and research. When people realise that they are flawed by design, then exercises like this undermine the public’s faith in research: this can only undermine willingness to participate in research, of course, and recruiting participants into trials is difficult enough at the best of times.

  More than tha
t, there are also some very important ethical issues here. People volunteer their bodies—and their children’s bodies—to participate in trials, on the understanding that the results will be used to improve medical and scientific knowledge. They expect that research performed on them will be properly conducted, informative by design, and that the results will be published in full, for all to see.

  I have seen the parent-information leaflets that were distributed for the Durham project, and they are entirely unambiguous in promoting the exercise as a scientific research project. The word ‘study’ is used seventeen times in one of these leaflets, although there is little chance that the ‘study’ (or ‘trial’, or ‘initiative’) can produce any useful data, for the reasons we have seen, and in any case it has now been announced that the effect on GCSE results will not be published.

  For these reasons the trial was, in my opinion, unethical.*

  ≡ While we’re on the subject of ethics, Durham have claimed that to give placebo to half the children would itself be unethical: this is another very basic misunderstanding on their part. We do not know if fish oils are beneficial or not. That would be the point of doing some proper research into the area.

  You will have your own view, but it is very hard to understand what justification there can be for withholding the results of this ‘trial’ now that it has concluded. Educationalists, academic researchers, teachers, parents and the public should be permitted to review the methods and results, and draw their own conclusions on its significance, however weak the design was. In fact, this is the exact same situation as the data on antidepressants’ efficacy being withheld by the drug companies, and a further illustration of the similarities between these pill industries, despite the food supplement pill industry’s best efforts to present itself as somehow ‘alternative’.

  The power is in the pill?

  We should be clear that I’m not—and I’m quite entitled to say this—myself very interested in whether fish-oil capsules improve children’s IQ, and I say this for a number of reasons. Firstly, I’m not a consumer journalist, or a lifestyle guru, and despite the infinitely superior financial rewards, I am absolutely very much not in the business of ‘giving readers health advice’ (to be honest, I’d rather have spiders lay eggs in my skin). But also, if you think about it rationally, any benefit of fish oil for school performance will probably not be all that dramatic. We do not have an epidemic of thick vegetarians, for example, and humans have shown themselves to be as versatile as their diets are diverse, from Alaska to the Sinai desert.

  But more than anything, at the risk of sounding like the most boring man you know, again: I wouldn’t start with molecules, or pills, as a solution to these kinds of problems. I can’t help noticing that the capsules Durham is promoting cost 80p per child per day, while it spends only 65p per child per day on school meals, so you might start there. Or you might restrict junk-food advertising to children, as the government has recently done. You might look at education and awareness about food and diet, as Jamie Oliver recently did very well, without recourse to dodgy pseudoscience or miracle pills.

  You might even step away from obsessing over food—just for once—and look at parenting skills, teacher recruitment and retention, or social exclusion, or classroom size, or social inequality and the widening income gap. Or parenting programmes, as we said right at the beginning. But the media don’t want stories like that. ‘Pill solves complex social problem’ feels much more like a news story than anything involving a boring parenting programme.

  This is partly due to journalists’ own sense of news value, but it’s also a question of how stories are pushed. I’ve not met Hutchings et al, the authors of the parenting study that kicked off this chapter—and I’m quite prepared to be told that they are in Soho House until 2 a.m. every night, schmoozing broadcast media journalists with champagne and nibbles—but in reality I suspect they are quiet, modest academics. Private companies, meanwhile, have top-dollar public-relations firepower, one single issue to promote, time to foster relationships with interested journalists, and a wily understanding of the desires of the public and the media, our collective hopes and consumer dreams.

  The fish-oil story is by no means unique: repeatedly, in a bid to sell pills, people sell a wider explanatory framework, and as George Orwell first noted, the true genius in advertising is to sell you the solution and the problem. Pharmaceutical companies have worked hard, in their direct-to-consumer advertisements and their lobbying, to push the ‘serotonin hypothesis’ for depression, even though the scientific evidence for this theory is growing thinner every year; and the nutrition supplements industry, for its own market, promotes dietary deficiencies as a treatable cause for low mood (I myself do not have a miracle cure to offer, and repetitively enough, I think that the social causes of these problems are arguably more interesting—and possibly even more amenable to intervention).

  These fish-oil stories were a classic example of a phenomenon more widely described as ‘medicalisation’, the expansion of the biomedical remit into domains where it may not be helpful or necessary. In the past, this has been portrayed as something that doctors inflict on a passive and unsuspecting world, an expansion of the medical empire: but in reality it seems that these reductionist biomedical stories can appeal to us all, because complex problems often have depressingly complex causes, and the solutions can be taxing and unsatisfactory.

  In its most aggressive form, this process has been characterised as ‘disease-mongering’. It can be seen throughout the world of quack cures—and being alive to it can be like having the scales removed from your eyes—but in big pharma the story goes like this: the low-hanging fruit of medical research has all been harvested, and the industry is rapidly running out of novel molecular entities. They registered fifty a year in the 1990s, but now it’s down to twenty a year, and a lot of those are just copies. They are in trouble.

  Because they cannot find new treatments for the diseases we already have, the pill companies instead invent new diseases for the treatments they already have. Recent favourites include Social Anxiety Disorder (a new use for SSRI drugs), Female Sexual Dysfunction (a new use for Viagra in women), night eating syndrome (SSRIs again) and so on: problems, in a real sense, but perhaps not necessarily the stuff of pills, and perhaps not best conceived of in reductionist biomedical terms. In fact, refraining intelligence, loss of libido, shyness and tiredness as medical pill problems could be considered crass, exploitative, and frankly disempowering.

  These crude biomedical mechanisms may well enhance the placebo benefits from pills, but they are also seductive precisely because of what they edit out. In the media coverage around the rebranding of Viagra as a treatment for women in the early noughties, and the invention of the new disease Female Sexual Dysfunction, for example, it wasn’t just the tablets that were being sold: it was the explanation.

  Glossy magazines told stories about couples with relationship problems, who went to their GP, and the GP didn’t understand their problem (because that is the first paragraph of any medical story in the media). Then they went to the specialist, and he didn’t help either. But then they went to a private clinic. They did blood tests, hormone profiles, esoteric imaging studies of clitoral bloodflow, and they understood: the solution was in a pill, but that was only half the story. It was a mechanical problem. Rarely was there a mention of any other factors: that she was feeling tired from overwork, or he was exhausted from being a new father, and finding it hard to come to terms with the fact that his wife was now the mother of his children, and no longer the vixen he first snogged on the floor of the student union building to the sound of ‘Don’t You Want Me Baby?’ by the Human League in 1983: no. Because we don’t want to talk about these issues, any more than we want to talk about social inequality, the disintegration of local communities, the breakdown of the family, the impact of employment uncertainty, changing expectations and notions of personhood, or any of the other complex, difficult factors that play in
to the apparent rise of antisocial behaviour in schools.

  But above all we should pay tribute to the genius of this huge fish-oil project, and every other nutritionist who has got their pills into the media, and into schools, because more than anything else, they have sold children, at the most impressionable time of their lives, one very compelling message: that you need to take pills to lead a healthy normal life, that a sensible diet and lifestyle are not enough in themselves, and that a pill can even make up for failings elsewhere. They have pushed their message directly into schools, into families, into the minds of their worried parents, and it is their intention that every child should understand that you need to eat several large, expensive, coloured capsules, six of them, three times a day, and this will improve vital but intangible qualities: concentration, behaviour and intelligence.

  This is the greatest benefit to the pill industry, of every complexion. I would prefer fish-oil pills to ritalin, but fish-oil pills are being marketed at every child in the country, and they have undoubtedly won. Friends tell me that in some schools it is considered almost child neglect not to buy these capsules, and its impact on this generation of schoolchildren, reared on pills, will continue to bear rich fruit for all the industries, long after the fish-oil capsules have been forgotten.

  Calming down: the apothecary industrial complex

  Generating news coverage as a means of increasing brand awareness for a commercial product is a well-trodden path (and it also drives those contentless ‘scientists have found the equation for…’ stories that we shall see in a later chapter). PR companies even calculate something called ‘advertising equivalents’ for the exposure your brand gets for free, and in a period when more newsprint is generated by fewer journalists, it’s inevitable that such shortcuts to colourful copy are welcomed by reporters. News and features stories about a product also carry far more weight in the public imagination than a paid advert, and are more likely to be read or watched.