2008 - Bad Science

by Ben Goldacre

  More importantly, why is everyone talking about omega-3? On to the next chapter.

  8 ‘Pill Solves Complex Social Problem’

  Medicalisation—or ‘Will fish-oil pills make my child a genius?’

  In 2007 the British Medical Journal published a large, well-conducted, randomised controlled trial, performed at lots of different locations, run by publicly funded scientists, and with a strikingly positive result: it showed that one treatment could significantly improve children’s antisocial behaviour. The treatment was entirely safe, and the study was even accompanied by a very compelling cost-effectiveness analysis.

  Did this story get reported as front-page news in the Daily Mail, the natural home of miracle cures (and sinister hidden scares)? Was it followed up on the health pages, with an accompanying photo feature, describing one child’s miraculous recovery, and an interview with an attractive relieved mum with whom we could all identify?

  No. The story was unanimously ignored by the British news media, despite their preoccupation with both antisocial behaviour and miracle cures, for one simple reason: the research was not about a pill. It was about a cheap, practical parenting programme.

  At the same time, for over five years now, newspapers and television stations have tried to persuade us, with ‘science’, that fish-oil pills have been proven to improve children’s school performance, IQ, behaviour, attention, and more. In fact, nothing could be further from the truth. We are about to learn some very interesting lessons about the media, about how not to conduct a trial, and about our collective desire for medicalised, sciencey-sounding explanations for everyday problems. Do fish-oil pills work? Do they make your child cleverer and better-behaved? The simple answer is, at the moment, nobody could possibly know. Despite everything you have been told, a trial has never been done in mainstream children.

  The newspapers would have you believe otherwise. I first became aware of ‘the Durham trials’ when I saw on the news that a trial of fish-oil capsules was being planned in 5,000 children. It’s an astonishing testament to the news values of the British media that this piece of research remains, I am quite prepared to suggest, probably the single most well-reported clinical trial of the past few years. It was on Channel 4 and ITV, and in every national newspaper, sometimes repeatedly. Impressive results were confidently predicted.

  This rang alarm bells for two reasons. Firstly, I knew the results of the previous trials offish-oil capsules in children—I’ll describe them in due course—and they weren’t very exciting. But more than that, as a basic rule, I would say this: whenever somebody tells you that their trial is going to be positive before they’ve even started it, then you know you’re onto an interesting story.

  Here is what they were planning to do in their ‘trial’: recruit 5,000 children in their GCSE year, give them all six fish-oil capsules a day, then compare how they did in their exams with how the council had estimated they should do without the capsules. There was no ‘control’ group to compare against (like the Aqua Detox bath with no feet in it, or the ear candle on the picnic table, or a group of children taking placebo capsules without fish oil in them). Nothing.

  By now you might not need me to tell you that this is a preposterous and above all wasteful way to go about doing a study on a pill that is supposed to improve school performance, with £1 million worth of generously donated capsules and 5,000 children at your disposal. But humour an old man, and let me flesh out your hunches, because if we cover the theoretical issues properly first, then the ‘researchers’ at Durham become even more amusingly absurd.

  Why you have a placebo group

  If you divide a group of kids in half, and give a placebo capsule to one group, and the real capsule to the other group, you can then compare how well each group does, and see whether it was the ingredients in the pill that made the difference to their performance, or just the fact of taking a pill, and being in a study. Why is this important? Because you have to remember that whatever you do to children, in a trial of a pill to improve their performance, their performance will improve.

  Firstly, children’s skills improve over time anyway: they grow up, time passes, and they get better at stuff. You might think you’re clever, sitting there with no nappy on, reading this book, but things haven’t always been that way, as your mother could tell you.

  Secondly, the children—and their parents—know that they are being given these tablets to improve their performance, so they will be subject to a placebo effect. I have already harped on about this at phenomenal length, because I think the real scientific story of the connections between body and mind are infinitely more interesting than anything concocted by the miracle-cure community, but here it is enough to remind you that the placebo effect is very powerful: consciously or unconsciously, the children will txpect themselves to improve, and so will their parents and their teachers. Children are exquisitely sensitive to our expectations of them, and anyone who doubts that fact should have their parenting permit revoked.

  Thirdly, children will do better just from being in a special group that is being studied, observed and closely attended to, since it seems that the simple fact of being in a trial improves your performance, or recovery from illness. This phenomenon is called the ‘Hawthorne effect’, not after a person, but after the factory where it was first observed. In 1923 Thomas Edison (he of the lightbulb) was chairing the ‘Committee on the Relation of Quality and Quantity of Illumination to Efficiency in the Industries’. Various reports from several companies had suggested that better lighting might increase productivity, so a researcher called Deming went with his team to test the theory at Western Electric’s Hawthorne plant at Cicero, Illinois.

  I will give you the simplified ‘myth’ version of the findings, as a rare compromise between pedantry and simplicity. When the researchers increased light levels, they found that performance improved. But when they reduced the light levels, performance improved then, too. In fact, they found that no matter what they did, productivity increased anyway. This finding was very important: when you tell workers they are part of a special study to see what might improve productivity, and then you do something…they improve their productivity. This is a kind of placebo effect, because the placebo is not about the mechanics of a sugar pill, it is about the cultural meaning of an intervention, which includes, amongst other things, your expectations, and the expectations of the people tending to you and measuring you.

  Beyond all this technical stuff, we also have to place the GCSE results—the outcome which was being measured in this ‘trial’—in their proper context. Durham has a very bad exam record, so its council will be battling hard in every way it possibly can to improve school performance, with all manner of different initiatives and special efforts, and extra funding, running simultaneously with the fish-oil ‘trials’.

  We should also remember that bizarre English ritual whereby GCSE results get better every year, yet anyone who suggests that the exams are getting easier is criticised for undermining the achievement of the successful candidates. In fact, taking the long view, this easing is obvious: there are forty-year-old O-level papers which are harder than the current A-level syllabus; and there are present-day university finals papers in maths that are easier than old A-level pipers.

  To recap: GCSE results will get better anyway; Durham will be desperately trying to improve its GCSE results through other methods anyway; and any kids taking pills will improve their GCSE results anyway, because of the placebo effect and the Hawthorne effect.

  This could all be avoided by splitting the group in half and giving a placebo to one group, separating out what is a specific effect of the fish-oil pills, and what is a general effect of all the other stuff we’ve described above. That would give you very useful information.

  Is it ever acceptable to do the kind of trial that was being carried out in Durham? Yes. You can do something called an ‘open trial’, without a placebo group, and this is an accepted kind of research. In fac
t, there is an important lesson about science here: you can do a less rigorous experiment, for practical reasons, as long as you are clear about what you are doing when you present your study, so other people can make their own minds up about how they interpret your findings.

  But there is an important caveat. If you do this kind of ‘compromise’ study, without a placebo group, ‘open label’, but in the hope of getting the most accurate’ possible picture of the treatment’s benefits, then you do it as carefully as you can, while being fully aware that your results might be distorted by expectation, by the placebo effect, by the Hawthorne effect, and so on. You might sign up your kids calmly and cautiously, saying in a casual, offhand fashion that you’re doing a small informal study on some tablets, you don’t say what you expect to find, you hand them out without fanfare, and you calmly measure the results at the end.

  What they did in Durham was the polar opposite. There were camera crews, soundmen and lighting men flooding the classrooms. Children were interviewed for radio, for television, for the newspapers; so were their parents; so were their teachers: so were Madeleine Portwood, the educational psychologist who was performing the trial, and Dave Ford, Head of Education, talking—bizarrely—about how they confidently expected positive results. They did literally everything which, in my view, would guarantee them a false positive result, and ruin any chance of their study giving meaningful and useful new information. How often does this happen? In the world of nutritionism, sadly, it seems to be standard research protocol.

  We should also remember that these fish-oil ‘trials’ were measuring some highly volatile outcomes. Performance at school in a test, and ‘behaviour’ (a word with a large semantic footprint, if ever I saw one) are huge, variable, amorphous things. More than most outcomes, they will change from moment to moment, with different circumstances, frames of mind, and expectations. Behaviour is not like a blood haemoglobin level, or even height, and nor is intelligence.

  Durham Council and Equazen were so successful in their publicity drive, whether through an uncontainable enthusiasm for a positive result or simple foolishness (I really don’t know which) that they effectively sabotaged their ‘trial’. Before the first fish-oil capsule was swallowed by a single child, the Eye Q branded supplement and trial had received glowing publicity in the local papers, the Guardian, the Observer, the Daily Mail, The Times, Channel 4, the BBC, ITV, the Daily Express, the Daily Mirror, the Sun, GMTV, Woman’s Own, and many more. Nobody could claim that the children weren’t well primed.*

  ≡ In fact, it’s hard to overstate quite how large the fish-oil circus became, over the many years it ran. Professor Sir Robert Winston himself, moustachioed presenter of innumerable ‘science’ programmes for the BBC, personally endorsed a competing omega-3 product, in an advertising campaign which was ultimately terminated by the ASA since it breached their codes on truthfulness and substantiation.

  You’re not an educational psychologist. You’re not the Head of Education at a council. You’re not the long-standing MD of a multi-million-pound pill business running huge numbers of ‘trials’. But I am quite sure that you understand very clearly all of these criticisms and concerns, because this isn’t rocket science.

  Durham defend themselves

  Being a fairly innocent and open-minded soul, I went to the people behind the trial, and put it to them that they had done the very things which would guarantee that their trial would produce useless results. That is what anyone would do in an academic context, and this was a trial after all. Their response was simple. ‘We’ve been quite clear,’ said Dave Ford, Chief Schools Inspector for Durham, and the mastermind behind the project to give out the capsules and measure the results. ‘This is not a trial.’

  This felt a bit weak. I call up to suggest that they’re doing a badly designed piece of research, and suddenly everything’s OK, because it’s not actually a ‘trial’? There were other reasons for thinking this was a fairly implausible defence. The Press Association called it a trial. The Daily Mail called it a trial. Channel 4 and ITV and everyone covering it all presented it, very clearly, as research (you can see the clips at badscience.net). More importantly, Durham Council’s own press release called it a ‘study’ and a ‘trial’, repeatedly.*

  ≡ As a testament to the astonishing foolishness of Durham Council, they’ve now even gone to the trouble of changing the wording of this press release on their website, as if that might address the gaping design flaws.

  They were giving something to schoolchildren and measuring the result. Their own descriptive term for this activity was ‘trial’. Now they were saying it wasn’t a trial.

  I moved on to Equazen, the manufacturer which is still being lauded throughout the press for its involvement in these ‘trials’ which were almost guaranteed—by virtue of the methodological flaws we have already discussed—to produce spurious positive results. Adam Kelliher, chief executive of the company, clarified further: this was an ‘initiative’. It was not a ‘trial’, nor was it a ‘study’, so I could not critique it as such. Although it was hard to ignore the fact that the Equazen press release talked about giving a capsule and measuring the results, and the word which the company itself used to describe this activity was: ‘trial’.

  Dr Madeleine Portwood, the senior educational psychologist running the study, called it a ‘trial’ (twice in the Daily Mail). Every single write-up described it as research. They were giving ‘X’ and measuring change ‘Y’. They called it a trial, and it was a trial—but a stupid trial. Simply saying, ‘Ah, but this is not a trial’ didn’t strike me as an adequate—nor indeed a particularly adult—defence. They didn’t seem to think a trial was even necessary, and Dave Ford explained that the evidence already shows fish oils are beneficial. Let’s see.

  The fish-oil evidence

  Omega-3 oils are ‘essential fatty acids’. They’re called ‘essential’ because they’re not made by the body (unlike glucose or vitamin D, for example), so you have to eat them. This is true of a lot of things, like many vitamins, for example, and it’s one of the many reasons why it’s a good idea to eat a varied diet, pleasure being another.

  They are found in fish oils, and—in slightly different form—in evening primrose oil, linseed oil and other sources. If you look at the flow charts in a biochemistry textbook you will see that there is a long list of functions which these molecules perform in the body: they are involved in constructing membranes, and also some of the molecules that are involved in communication between cells, for example during inflammation. For this reason some people think it might be useful to eat them in larger amounts.

  I’m open to the idea myself, but there are good reasons to be sceptical, because there is a lot of history here. In the past, decades before the Durham ‘trials’, the field of essential fatty acid research has seen research fraud, secrecy, court cases, negative findings that have been hushed up, media misreporting on a massive scale, and some very striking examples of people using the media to present research findings direct to the public in order to circumvent regulators. We’ll come back to that later.

  There have been—count them—six trials to date on fish oil in children. Not one of these trials was done in ‘normal’ mainstream children: all of them have been done in special categories of children with one diagnosis or another—dyslexia, ADHD, and so on. Three of the trials had some positive findings, in some of the many things they measured (but remember, if you measure a hundred things in a study, a few of them will improve simply by chance, as we will see later), and three were negative. One, amusingly, found that the placebo group did better than the fish-oil group on some measures. They are all summarised online at badscience.net.

  And yet: ‘AH of our research, both published and unpublished, shows that the Eye Q formula can really help enhance achievement in the classroom,’ says Adam Kelliher, CEO of Equazen. All of it.

  To take a statement like this seriously, we would have to read the research. I am not for one nanosecond
accusing anybody of research fraud: in any case, if anyone did suspect fraud, reading the research would not help, because if people have faked their results with any enthusiasm then you need a forensic statistician and a lot of time and information to catch them out. But we do need to read published research in order to establish whether the conclusions drawn by the stakeholders in the research are valid, or whether there are methodological problems that make their interpretation the product of wishful thinking, incompetence, or perhaps even simply a judgement call with which you would not concur.

  Paul Broca, for example, was a famous French craniologist in the nineteenth century whose name is given to Broca’s area, the part of the frontal lobe involved in the generation of speech (which is wiped out in many stroke victims). Among his other interests, Broca used to measure brains, and he was always rather perturbed by the fact that the German brains came out a hundred grams heavier than French brains. So he decided that other factors, such as overall body weight, should also be taken into account when measuring brain size: this explained the larger Germanic brains to his satisfaction. But for his prominent work on how men have larger brains than women, he didn’t make any such adjustments. Whether by accident or by design, it’s a kludge.

  Cesare Lombroso, a nineteenth-century pioneer of ‘biological criminology’, made similarly inconsistent fixes in his research, citing insensitivity to pain among criminals and ‘lower races’ as a sign of their primitive nature, but identifying the very same quality as evidence of courage and bravery in Europeans. The devil is in the detail, and this is why scientists report their full methods and results in academic papers, not in newspapers or on television programmes, and it is why experimental research cannot be reported in the mainstream media alone.