by Ben Goldacre
Kids Who Spot Bullshit, and the Adults Who Get Upset About It
Guardian, 28 May 2011
If you can tear yourself away from Ryan Giggs’s penis for just one moment, I have a different censorship story.
Brain Gym is a schools programme I’ve been writing about since 2003. It’s a series of elaborate physical movements with silly pseudoscientific justifications: you wiggle your head back and forth, because that gets more blood into your frontal lobes for clearer thinking; you contort your fingers together to improve some unnamed ‘energy flow’. They’re keen on drinking water, because ‘processed foods’ – I’m quoting the Brain Gym Teacher’s Manual – ‘do not contain water’. You pay hundreds of thousands of pounds for Brain Gym, and it’s still done in hundreds of state schools across the UK.
This week I got an email from a science teacher about a thirteen-year-old pupil. Both have to remain anonymous. This pupil wrote an article about Brain Gym for her school paper, explaining why it’s nonsense: the essay is respectful, straightforward, and factual. But the school decided this article couldn’t be printed, because it would offend the teachers in the junior school who use Brain Gym.
Now, this is weak-minded, and perhaps even vicious. More interesting, though, is how often children are able to spot bullshit, and how often adults want to shut them up.
Emily Rosa is the youngest person ever to have published a scientific paper in the Journal of the American Medical Association, one of the most influential medical journals in the world. At the age of nine she saw a TV programme about nurses who practise ‘Therapeutic Touch’, claiming they can detect and manipulate a ‘human energy field’ by hovering their hands above a patient.
For her school science-fair project, Rosa conceived and executed an experiment to test if they really could detect this ‘field’. Twenty-one experienced practitioners put their palms on a table, behind a screen. Rosa flipped a coin, hovered her hand over the therapist’s left or right palm accordingly, and waited for them to say which it was: the therapists performed no better than chance, and with 280 attempts there was sufficient statistical power to show that these claims were bunk.
Therapeutic Touch practitioners, including some in university posts, were deeply unhappy: they insisted loudly that JAMA was wrong to publish the study.
Closer to home is Rhys Morgan, a schoolboy with Crohns Disease. Last year, chatting on www.crohnsforum.com, he saw people recommending ‘Miracle Mineral Solution’, which turned out to be industrial bleach, sold with a dreary conspiracy theory to cure Aids, cancer, and so on.
At the age of fifteen, he was perfectly capable of exploring the evidence, finding official documents, and explaining why it was dangerous. The adults banned him. Since then he’s got his story on The One Show, while the Chief Medical Officer for Wales, the Food Standards Agency and Trading Standards have waded in to support him.
People wring their hands over how to make science relevant and accessible, but newspapers hand us one answer on a plate every week, with their barrage of claims on what’s good for you or bad for you: it’s evidence-based medicine. If every school taught the basics – randomised trials, blinding, cohort studies, and why systematic reviews are better than cherry-picking your evidence – it would help everyone navigate the world, and learn some of the most important ideas in the whole of science.
But even before that happens, we can feel optimistic. Information is more easily accessible now than ever before, and smart, motivated people can sidestep traditional routes to obtain knowledge and disseminate it. A child can know more about evidence than their peers, and more than adults, and more than their own teachers; they can tell the world what they know, and they can have an impact.
So the future is bright. And if you’re one of the teachers who stopped a child’s essay from being published because it dared to challenge your colleagues for promoting the ludicrousness of Brain Gym, then really: shame on you.
Existential Angst About the Bigger Picture
Guardian, 21 May 2011
Here’s no surprise: beliefs which we imagine to be rational are bound up in all kinds of other stuff. Political stances, for example, correlate with various personality features. One major review in 2003 looked at thirty-eight different studies, containing data on 20,000 participants, and found that overall, political conservatism was associated with things like death anxiety, fear of threat and loss, intolerance of uncertainty, a lack of openness to experience, and a need for order, structure and closure.
Beliefs can also be modified by their immediate context. One study from 2004, for example, found that when you make people think about death (‘Please briefly describe the emotions that the thought of your own death arouses in you’) they are more likely to endorse an essay discussing how brilliant George W. Bush was in his response to 9/11.
A new study looks at intelligent design, the more superficially palatable form of creationism, promoted by various religious groups, which claims that life on earth is too complex to have arisen through evolution and natural selection. Intelligent design implies a reassuring universe, with a supernatural creator, and it turns out that if you make people think about death, they’re less likely to approve of a Richard Dawkins essay, and more likely to rate intelligent design highly.
So that’s settled: existential angst drives us into the hands of religion. Rather excellently, the effect was partially reversed when people also read a Carl Sagan essay about how great it is to find meaning in the universe for yourself using science. It’s perfect. I love this stuff. I love social science research that reinforces my prejudices. Everybody does.
But that’s where I start to fall down. If I like these results, then lots of other people will like them too, whether it’s the academic psychologists doing the research, the statisticians they collaborate with, the academic journal editors and reviewers who decide whether or not the paper gets an easy ride into print, the press officers who decide whether or not to shepherd its findings towards the public, or even, finally, the bloggers and journalists who write about it. At every step, there is room for fun results to get through, and for unwelcome results to fall off the radar.
This isn’t a criticism of any individual study. Rather, it’s the angst-inducing context that surrounds every piece of academic research that you read: a paper can be perfect, brilliantly well-conducted, and yet there’s no way of knowing how many negative findings go missing. For all we know, we’re just seeing the lucky times the coin landed heads up.
The scale of the academic universe is dizzying, after all. Our most recent estimate is that there are over 24,000 academic journals in existence, 1.3 million academic papers published every year, and over 50 million papers have been published since scholarship began.
And for every one of these 50 million papers there will be unknowable quantities of blind alleys, abandoned experiments, conference presentations, work in progress seminars, and more. Look at the vast number of undergraduate and masters dissertations that had an interesting finding, and got turned into finished academic papers; and then think about the even vaster number that didn’t.
In medicine, where the stakes are tangible, systems have grown up to try to cope with this problem: trials are supposed to be registered before they begin, so we can notice the results that get left unpublished. But the systems are imperfect, and pre-registration is very rarely done, even in medical research, for anything other than trials.
We are living in the age of information, and vast tracts of data are being generated around the world, on every continent and on every question. A £200 laptop will let you run endless statistical analyses. The most interesting questions aren’t around individual nuggets of data, but rather how we can corral it to create an information architecture which serves up the whole picture.
The Glorious Mess of Real Scientific Results
Guardian, 6 November 2010
Popular science is often triumphalist, presenting research as a set of completed answers, whe
n in reality much of what gets published makes a glorious, necessary mess.
Here is an example. Solomon Asch’s legendary studies from the 1950s on conformity are among my favourite experiments of all time. Some people in a room are asked to judge the length of a line; all but one are stooges, and they unanimously assert what is obviously an incorrect answer. The one true, unsuspecting experimental subject conforms to the majority view, despite knowing that it’s incorrect, about a third of the time.
This is a chilling result that feels just right, and over the past half-century researchers have replicated the study over a hundred times in seventeen countries, allowing hints of patterns to be spotted in the results. One analysis of US studies found that conformity has declined since the 1950s. Another found that ‘collectivist’ countries tend to show higher levels of conformity than individualist ones.
This month the International Journal of Psychology published a new variant. Instead of one real subject in a room full of stranger stooges, they used polarising glasses – the same technology used to present a different image to the left and right eye for 3D films – to show participants different images on the same screen, at the same time, in the same room. This meant that friends could disagree, legitimately, and so exert social pressure, but without faking it.
The results were problematic. Overall, sometimes the minority people did conform to peer pressure, giving incorrect answers. But when the results were broken down, women did conform, a third of the time, but men did not. This poses a problem. Why were the results of this study different from the original study?
It could be that the subjects were different. The Asch experiments were only conducted in men, and they did conform. Perhaps modern Japanese undergraduates are different from 1950s US undergraduates (although cultural and generational differences have not previously been shown to be so large that they abolish the conformity effect completely).
It could be that the original task, where subjects had to judge the length of a line, was slightly different. But if anything, the task in the new experiment was harder than in the original, because the polarising glasses required that extra visual noise be added in; and if judgements were trickier, and therefore closer calls, then you might expect that conformity would increase, rather than decrease.
Or it could be that the relationships were different. Perhaps conforming effects are less pronounced among people who know each other compared to an experiment with a room full of stranger stooges: perhaps you feel more comfortable disagreeing with friends. This would be an important answer, if true, because when we extrapolate from the lab to the everyday, we’re probably more interested in conformity effects among acquaintances, because that’s what happens in a real community.
Maybe these questions will be resolved with a new experiment – you could probably design one yourself that would discriminate between the different possible explanations – but that will depend on whether someone is interested enough, and whether they can get the money and the time. Perhaps the paper will sink like a stone, and be ignored or overlooked, as sometimes happens with uncomfortable data.
But what you should know is this: alongside the triumphalism, and the answers, in reality, grey and conflicting results like these run deep in the research literature. They’re not an aberration, or a disappointment; in fact they are arguably the glorious norm, in the noise of over 20,000 academic journals, publishing well over a million articles every year. Alongside the giants, and the clean easy answers, challenging and ambiguous findings like these are what science is really made of.
Nullius in Verba1
Not in the Guardian, 26 June 2010
Here is some pedantry: I worry about data being published in newspapers rather than academic journals, even when I agree with its conclusions. Much like Bruce Forsyth, the Royal Society has a catchphrase: Nullius in verba, or ‘On the word of nobody’. Science isn’t about assertions on what is right, handed down from authority figures. It’s about clear descriptions of studies, and the results that came from them, followed by an explanation of why they support or refute a given idea.
Last week the Guardian ran a major series of articles on the mortality rates after planned abdominal aortic aneurysm repair in different hospitals. Like many previously published academic studies on the same question, they discovered that hospitals which perform the operation less frequently have poorer outcomes. I think this is a valid finding.
The Guardian pieces aimed to provide new information, in that they did not use the Hospital Episodes Statistics, which have been used for much previous work on the topic (and on the NHS Choices website, where they are used to rate hospitals for the public). Instead they approached each hospital with a Freedom of Information Act request, asking the surgeons themselves for the figures on how many operations they performed, and how many people died.
Many straightforward academic papers are built out of this kind of investigative journalism work, from early epidemiology research into occupational hazards, through to the famous recent study hunting down all the missing trials of SSRI antidepressants that companies had hidden away. It’s not clear whether this FOI data will be more reliable than the Hospital Episodes numbers – ‘Discuss the strengths and weaknesses of the HES dataset’ is a standard public health exam question – and reliability will probably vary from hospital to hospital. One unit, for example, reported a single death after ninety-five emergency AAA operations on FOI request, when on average about one in three people in the UK die during this procedure, and that suggests to me that there may be problems in the data. But there’s no doubt that this was a useful thing to do, and there’s no doubt that hospitals should be helpful and share this information.
So what’s the problem? It’s not the trivial errors in the piece, although they were there. The main Guardian article says there are ten hospitals with over 10 per cent mortality, but in the data there are only seven. It says twenty-three hospitals do over fifty operations a year, but looking at the data there are only twenty-one.
But here’s what I think is interesting. This analysis was published in the Guardian, not an academic journal. Alongside the articles, the Guardian published its data, and as a long-standing campaigner for open access to data, I think this is exemplary. I downloaded it, as the Guardian webpage invited, and did a quick scatter plot, and a few other things. I couldn’t see the pattern for greater mortality in hospitals that did the procedure infrequently. It wasn’t barn door. Others had the same problem. I received a trickle of emails from readers who also couldn’t find the claimed patterns (including a Professor of Stats, if that matters to you). Jon Appleby, Chief Economist on Health Policy at the King’s Fund, posted on Guardian Comment is Free saying that he couldn’t find the pattern either.
The journalists were also unable to tell me how to find the pattern. They referred me instead to Peter Holt, an academic surgeon who’d analysed the data for them. Eventually I was able to piece together a rough picture of what had been done, and after a few days, more details were posted online. It was a pretty complicated analysis, with safety plots and forest plots. I think I buy it as fair.
So why does it matter, if the conclusion is probably valid? Because science is not a black box. There is a reason why people generally publish results in academic journals instead of newspapers, and it’s got little to do with ‘peer review’ and a lot to do with detail about methods, which tell us how you know if something is true. It’s worrying if a new data analysis is published only in a newspaper, because the details of how the conclusions were reached are inaccessible. This is especially true if the analysis is so complicated that the journalists themselves did not know about it, and could not explain it, and this transparency is especially important if you’re seeking to influence policy. The information needs to be somewhere.
Open data – people posting their data freely for all to re-analyse – is the big hip new zeitgeist, and a vitally important new idea. But I was surprised to find that the thing I’ve a
dvocated for wasn’t enough: open data is sometimes no use, unless we also have open methods.
Is It OK to Ignore Results from People You Don’t Trust?
Guardian, 6 March 2010
If the media were actuarial about drawing our attention to the causes of avoidable death, your newspapers would be filled with diarrhoea, Aids and cigarettes every day. In reality we know this is an absurd idea. For those interested in the scale of our fascination with rarity, one piece of research looked at a three-month period in 2002 and found that 8,571 people had to die from smoking to generate one story on the subject from the BBC, while there were three stories for every death from vCJD.
So you’ve probably heard that smoking might prevent Alzheimer’s. It comes up in the papers, sometimes to say it’s a true finding, sometimes to say it’s been refuted. Maybe you think it’s a mixed bag, that the research is contradictory, that ‘experts are divided’. Perhaps you smoke, and joke about how it’ll stop you losing your marbles, at least.
This month, Janine Cataldo and colleagues publish a systematic review on the subject, but with a very interesting twist. First, they found all the papers ever published on smoking and Alzheimer’s, using an explicit search strategy which they describe properly in the paper – because they’re scientists, not homeopaths – to make sure that they found all of the evidence, rather than just the studies they already knew about, or the ones which flattered their preconceptions.
They found forty-three in total, and overall, smoking significantly increases your risk of Alzheimer’s. But they went further. Eleven of the studies were written by people with affiliations to the tobacco industry. This wasn’t always declared, so to double check, the researchers searched on the University of California’s Legacy Tobacco Documents Library, a vast collection of scanned material that has been gathered over decades of legal action.
