by Ben Goldacre
Their quoted source was no less impeccable than a BBC Radio 4 documentary presented by Felicity Finch (her what plays Ruth Archer), broadcast on Monday. ‘The number of babies with Down syndrome has steadily fallen, that is until today, when for the first time ever that number is higher than before, when testing was introduced.’ I see. ‘I’m keen to find out why more parents are making this decision.’ They’re not. ‘I was so intrigued by these figures that I’ve been following some parents to find out what lies behind their choice.’ Felicity, they’re not. The entire founding premise of your entire twenty-seven-minute documentary is wrong.
There has indeed been a 4 per cent increase in Down’s syndrome live births in England and Wales from 1989 to 2006 (717 and 749 affected births in the two years, respectively). However, since 1989 there has also been a far greater increase in the number of Down’s syndrome foetuses created in the first place, because people are getting pregnant much later in life.
What causes Down’s syndrome? We don’t really know, but maternal age is the only well-recognised association. Your risk of a Down’s syndrome pregnancy below the age of twenty-five is about one in 1,600. This rises to about one in 340 at thirty-five, and one in forty at the age of forty-three. In 1989, 6 per cent of pregnant women were over thirty-five years of age. By 2006 it was 15 per cent.
The National Down Syndrome Cytogenetic Register holds probably the largest single dataset on Down’s syndrome, with over 17,000 anonymous records collected since 1989, making it one of the most reliable resources in the search for patterns and possible causal factors. They have calculated that if you account for the increase in the age at which women are becoming pregnant, from 1989 to 2006 the number of Down’s syndrome live births in the UK would have increased not by 4 per cent, but from 717 to an estimated 1,454, if screening and subsequent termination had not been available.
Except, of course, antenatal screening is widely available, it is widely taken up, and contrary to what every newspaper told you this week, it is widely acted upon. More than nine out of ten women who have an antenatal diagnosis of Down’s syndrome decide to have a termination of the pregnancy. This proportion has not changed since 1989. This is the ‘decision’ that Felicity Finch, Radio 4, the Mail, The Times, the Mirror and the rest are claiming more parents are taking: to carry on with a Down’s syndrome pregnancy. This is what they are taking as evidence of a more caring society. But the figure has not changed.
Since we’ve now established beyond any doubt that the team behind this documentary got their numbers – and therefore their whole factual premise – entirely wrong, I think we’re also entitled to engage with their crass moral judgements. If I terminate a Down’s syndrome pregnancy, is that proof that society is not a warm, caring place, and that I am not a warm, caring person? For many parents the decision to terminate will be a difficult and upsetting one, especially later in life, and stories like this create a pretty challenging backdrop for making it. This would have been true even if the programme-makers had got their figures absolutely perfect, but as is so often the case for those with spare flesh to wave at strangers, their facts and figures are simply wrong.
The National Down Syndrome Cytogenetic Register felt obliged to issue a thorough clarification. The thoroughly brilliant ‘Behind the Headlines’ service on the NHS Choices website took the story to pieces, as it so often does, in its daily round-up of the real evidence behind the health news (disclosure: I had a trivially tiny hand in helping to set this service up).
Everybody ignored them, nobody has clarified, and Born With Down’s remains ‘Choice of the Day’ on the Radio 4 website.
Drink Coffee, See Dead People
Guardian, 17 January 2009
‘Danger from just 7 cups of coffee a day’, said the Express on Wednesday. ‘Too much coffee can make you hallucinate and sense dead people say sleep experts. The equivalent of just seven cups of instant coffee a day is enough to trigger the weird responses.’ The story appeared in almost every national newspaper.
This was weak observational data. That’s just the start of our story, but you should know exactly what the researchers did. They sent an email inviting students to fill out an online survey, and 219 agreed.
The survey is still online (I just clicked answers randomly to see the next question until I got to the end). It asks about caffeine intake in vast detail, and then uses one scale to measure how prone you are to feeling persecuted, and another, the ‘Launay-Slade Hallucination Scale’, sixteen questions designed to measure ‘predisposition to hallucination-like experiences’.
Some of these questions are about having hallucinations and seeing ghosts, but some really are a very long way from there. Heavy coffee drinkers could have got higher scores on this scale by responding affirmatively to statements like: ‘No matter how hard I try to concentrate on my work, unrelated thoughts always creep into my mind’; ‘Sometimes a passing thought will seem so real that it frightens me’; or ‘Sometimes my thoughts seem as real as actual events in my life.’ That’s not seeing ghosts or hearing voices.
And of course, this was weak observational data, and there could have been alternative explanations for the observed correlation between caffeine intake and very slightly higher LSHS scores. Maybe some students who drink a lot of coffee are also sleep-deprived, and marginally more prone to hallucinations because of that. Maybe they are drinking coffee to help them get over last night’s massive marijuana hangover.
Maybe the kinds of people who take drugs instrumentally to have fun and distort their perceptions also take drugs like caffeine instrumentally to stay alert. You can think of more, I’m sure. The researchers were keen to point out this shortcoming in their paper. The Express and many others didn’t seem to care.
Then, if you read the academic paper, you find that the associations reported are weak. For the benefit of those who understand ‘regression’ (and it makes anybody’s head hurt), 18 per cent of the variance in the LSHS score is explained by gender, age and stress. When you add in caffeine to those three things, 21 per cent of the variance in the LSHS score is explained: only an extra 3 per cent, so caffeine adds very little. The finding is statistically significant, as the researchers point out, so it’s unlikely to be due to chance, but that doesn’t change the fact that it’s still weak, and it still explains only a tiny amount of the overall variance in scores on the ‘predisposed-to-hallucinations’ scale.
Lastly, most newspapers reported a rather dramatic claim, that seven cups of coffee a day is associated with a three times higher prevalence of hallucinations. This figure does not appear anywhere in the paper. It seems to be an ad hoc analysis done afterwards by the researchers, and put into the press release, so you cannot tell how they did it, or whether they controlled appropriately for problems in the data, like something called ‘multiple comparisons’.
Here is the problem. Apparently this three-times-greater risk is for the top 10 per cent of caffeine consumers, compared with the bottom 10 per cent. They say that heavy caffeine drinkers were three times more likely to have answered affirmatively to just one LSHS question: ‘In the past, I have had the experience of hearing a person’s voice and then found that no one was there.’
Now, this poses massive problems. Imagine that I am stood facing a barn, holding a machine gun, blindfolded, firing off shots whilst swinging my whole body from side to side and laughing maniacally. I then walk up to the barn, find three bullet holes which happen to be very close together, and draw a target around them, claiming I am an excellent shot.
You can easily find patterns in your data once it’s collected. Why choose 10 per cent as your cut-off? Why not the top and bottom quarters? Maybe they have accounted for this problem. You don’t know. I don’t know. They say they have, to me, in emails, but it wasn’t in the paper, and we can’t all see the details. I don’t think that’s satisfactory for a headline finding, and the first claim of a press release.
Then there is one final problem: putting a finding in the press rel
ease but not into the paper is a subversion of the peer-review process. People will read this coverage, they will be scared, and they will change their behaviour. But the researchers’ key reported claim, with massive popular impact, was never peer reviewed, and crucially the technical details behind it are not in the public domain.
I’m sorry to see academics unblameless in this dreary situation.
Voices of the Ancients
Guardian, 16 January 2010
Every now and then you have to salute a genius. Both the Daily Mail and the Metro report new research analysing the positions of Britain’s ancient sites, and the results are startling: primitive man had his own form of ‘sat nav’. Researcher Tom Brooks analysed 1,500 prehistoric monuments, and found them all to be on a grid of isosceles triangles, each pointing to the next site, allowing our ancestors to travel between settlements with pinpoint accuracy. The papers even carried an example of his map work, which I have reproduced here.
That this pattern could occur simply because one site was on the way to the next was not considered. Mr Brooks has proven, he explains, that there were keen mathematicians here 5,000 years ago, millennia before the Greeks invented geometry: ‘Such is the mathematical precision, it is inconceivable that this work could have been carried out by the primitive indigenous culture we have always associated with such structures … all this suggests a culture existing in these islands in the past quite outside our expectation and experience today.’ He does not rule out extraterrestrial help.
In the Metro Tom Brooks is a researcher. To the Daily Mail he is a researcher, a historian and a writer. I hope it’s not rude or unfair for me to add ‘retired marketing executive of Honiton, Devon’.
Matt Parker, his nemesis, is based in the School of Mathematical Sciences at Queen Mary, University of London. He has applied the same techniques used by Mr Brooks to another mysterious and lost civilisation.
‘We know so little about the ancient Woolworths stores,’ he explains, ‘but we do still know their locations. I thought that if we analysed the sites we could learn more about what life was like in 2008 and how these people went about buying cheap kitchen accessories and discount CDs.’
The results revealed an exact and precise geometric placement of the Woolworths locations. ‘Three stores around Birmingham formed an exact equilateral triangle (Wolverhampton, Lichfield and Birmingham stores), and if the base of the triangle is extended, it forms a 173.8-mile line linking the Conwy and Luton stores. Despite the 173.8-mile distance involved, the Conway Woolworths store is only forty feet off the exact line and the Luton site is within thirty feet. All four stores align with an accuracy of 0.05 per cent.’
Matt Parker used an ancient technique: he found his patterns in eight hundred ex-Woolworths locations by ‘skipping over the vast majority, and only choosing the few that happen to line up’.
With 1,500 locations, Mr Brooks had almost twice as much data to work with, and on this issue Parker is clear: ‘It is extremely important to look at how much data people are using to support an argument. For example, the case for global warming covers vast amounts of comprehensive evidence, but it is still possible for people to search through the data and find a few isolated examples that appear to show otherwise.’
BIG DATA
There’s Something Magical About Watching Patterns Emerge from Data
Guardian, 11 June 2011
We all know that one atom of experience isn’t enough to spot a pattern: but when you put lots of experiences together and process that data, you get new knowledge. This might sound obvious, but following it through – watching patterns emerge from the noise – still gives me a sense of beauty and awe.
A paper in the British Medical Journal this week is a perfect example. Medicine is an imperfect art, so it’s inevitable that healthcare workers will make some suboptimal decisions: not so much the dramatic stuff – injecting people with the wrong drug – but more the marginal decisions, at the edges of the tweaks in a patient’s journey, affecting outcomes in ways that are harder to predict.
These kinds of complex decisions will inevitably be affected by context, and one example of that context is the franticness of A&E. Waiting times are a problem in a lot of countries. In the UK we introduced a four-hour ceiling as our target, and most hospitals met it. Abolishing that four-hour target was one of the coalition government’s first NHS reforms. But do waiting times matter?
Some researchers in Canada decided to find out. They gathered data from all the people who visited any A&E department in Ontario over a five-year period: this gave them data on a dizzying 22 million visits. Of these, 14 million resulted in the patient being seen and then sent home. Then they followed these patients up to see what happened, and specifically, to see if they died.
They also had another piece of information: for each patient they knew, from internal hospital data, what the average waiting time in A&E was at the time they arrived. This means that they were able to compare the odds of death for patients discharged when the average wait in A&E was less than four hours (or more) against the odds of death for patients discharged when the wait was less than one hour. Remember, this isn’t the time that individual patient waited, it’s the average wait in the department, as a proxy for how frantic things were.
The results were as you might fear. For patients sent home who attended an A&E department when the average wait there was more than six hours, the odds of death were almost twice those of patients sent home when the wait was less than one hour. This odds ratio was similar for patients measured as high or low urgency at triage, so it’s true for patients with both serious and less serious presentations.
Even more starkly, there’s a very clear trend in the data, where each step up in waiting time results in a higher risk of death. This becomes statistically significant when average waits reach just three hours. For those who care about saving money, the odds of being admitted – and so taking up an expensive hospital bed – also rose dramatically as average wait time increased.
However important you might find those specific results, the methodological issues are much more interesting, and they all arise because of the big numbers involved. We would never have discovered any of this without huge numbers of patients’ records, because the outcomes involved are rare: you only see a handful of deaths out of every 10,000 people sent home from A&E.
What’s more, because they had so many patients’ data, the researchers were able to see an effect even within hospitals, over time: so it wasn’t just that crap hospitals overall had longer waits, and higher death rates. What’s more, amazingly, they didn’t lose a single patient during follow-up: the death – or otherwise – of every single patient who was sent home from A&E could be tracked through their notes.
No individual patient or doctor could possibly have shown with any certainty, from their own personal experience of any one adverse outcome, that long waiting times in A&E are dangerous. This study is a remarkable testament to the power of good-quality computerised health records, and the kinds of new knowledge you can generate from interrogating them. It’s also, I’ll agree, a pretty frightening result.
Give Us the Data
Guardian, 7 October 2011
Bad things happen when problems are protected by a force field of tediousness. Here is an example. Data is the fabric of the modern world: just as we walk down pavements, so we trace routes through data, and build knowledge and products out of it. The government has lots of data that has already been collected, because it was needed to run the country properly: simple stuff like maps, postcode areas, land ownership, procurement data, endless weather readings, and so on.
Right now a fight is happening in Whitehall between two factions in government: one group thinks we should give this data away for free, as a matter of principle, because it will make good things happen; the other thinks we should restrict access, and sell it. A consultation is under way. Despite a positive ministerial introduction, each of the three options it gives
for releasing data is foolishly restrictive. Here’s why that’s a problem.
As things stand, much everyday government data is locked down so hard that nerds are forbidden to repurpose it. You could have a map of who owns what in your town, on your screen, at a click. You could find out what company boards someone sits on, and map their relationships and overlaps with all the other directors in the country. You could download transcripts of court proceedings that affect you. All this is blocked by the government’s restrictive data policies.
There are areas where access has been won by the shame of a simple moral argument. Hansard is a record of everything that happens in Parliament. TheyWorkForYou.com is a repurposing of that data which adds huge value, not just by being more usable than Hansard, but by identifying patterns in MPs’ voting behaviour. When it first came out, Hansard argued – embarrassingly – that this was an illegal breach of copyright.
But there are also straight commercial applications. If you’re making services or things that you sell to government, then seeing what they use and need helps you sell them stuff. That data is even internally useful: if you can see what everyone else is paying for toilet paper, you might get a better deal for your own department.
All this data has to be created, regardless of whether or not it gets sold, simply in order to run the country. You could ‘sweat the asset’, and charge money for access; but if you release it for free, at barely any cost to yourself, without fiddliness, in its raw form, the benefits are potentially huge.
This becomes especially clear when you notice how the restrictions extend beyond specific realms of data, and into the kind of core structural information that is needed as a civic skeleton for simple, everyday activity. The Royal Mail still owns all our postcode information, and you can’t get the house-number boundaries of each specific postcode without paying. All the most interesting data projects involve linking one dataset with another, and for addresses, that often means using postcodes, as a commonly used structural spine (I’m willing to bet that you don’t know your house’s latitude and longitude). This kind of framework data is the pavement of data space, and if you’re not allowed to use it, projects go unmade.
