Not much in life is this certain and breast cancer certainly isn't. Still, ludicrous implausibility didn't stop the claim making the headlines. With luck, viewers were less easily taken in than the many journalists who would have had a hand in bringing the report to air, since this is a passing piece of innumeracy that should have been easy to spot.
So what was the real meaning of the study so mangled in the news? It was true that research had shown a link between alcohol and breast cancer, but the first thing to do when faced with an increase in risk is to try to quell the fear and concentrate on the numbers. And the first question to ask about this number couldn't be simpler, that old favourite: 'how big is it?'
Cancer Research UK, which publicised the research – a perfectly professional piece of work, led by a team at Oxford University – announced the results as follows:
A woman's risk of breast cancer increases by 6 per cent for every extra alcoholic drink consumed on a daily basis, the world's largest study of women's smoking and drinking behaviour reveals.
It added that for two drinks daily, the risk rises by 12 per cent. This was how most other news outlets reported the story. So there was something in this 6 per cent after all, but it was 6 per cent if you had one drink every day of your adult life, not 6 per cent for every single drink, a small change of wording for a not-so-subtle difference of meaning.
This at least is accurate. It is also still meaningless. What's missing, yet again, is how big the risk was to start with. Until we know this, being told only by how much it has changed is no help at all.
Imagine two people running a race. One is twice as fast as the other. 'Fred twice as fast as Eric,' says the headline. But is Fred actually any good (is the risk high)? If the difference between them is all you know, then you know next to nothing of Fred's ability. They might both be hopeless, with Fred plodding but Eric staggering, with a limp, and blisters, on no training and an eight-pint hangover. Or Eric might be a respectable runner and Fred a world record holder. If all we know is the relative difference (twice as good as the other one), and we don't know how good the other one was, then we're ignorant of most of what matters. Similarly, if the only information we have about two risks is the difference between them (one is 100 per cent more risky than the other, without even knowing the other), we know nothing useful about either.
It is plain to see that the same percentage rise in risk can lead to a very different number at the end depending on the number at the beginning. It is astonishing how often news reports do not tell us the number at the beginning, or at the end, but only the difference.
'Stop-and-search of Asians up fourfold'; 'Teenage pregnancy up 50 per cent in London borough'. But have the stop-and-search numbers for the area in question gone up from one person last quarter to four people this quarter, and plausibly within the sort of variation that might ordinarily be expected, or up from 100 to 400 and indicative of a dramatic and politically charged change in police tactics? Are the teenage pregnancy rates up from two last year to three this year, or from 2,000 to 3,000? Viewers of the TV quiz show Who Wants to be a Millionaire? know that doubling the money with the next correct answer varies hugely depending how much you have won already. 'Contestant doubles her money' tells us nothing.
So why does the news sometimes report risk with only one number, the difference between what a risk was and what it becomes? 'Risk for drinkers up 6 per cent!' Six per cent of what? What was it before? What is it now? These reports, whatever their authors think they are saying, are numerical pulp.
How do we make the numbers for breast cancer meaningful? First, once more, the formal way, which you can ignore again if necessary. We need to know the baseline risk – the risk of contracting the disease among women who do not drink. About 9 per cent of women will have breast cancer diagnosed by the time they are 80. Knowing this, we can get an idea of how much more serious it is for drinkers, and because the baseline risk is fairly small, an increase of 6 per cent in this risk will still leave it relatively small. Like the slow runner, a 6 per cent increase in speed will not make him a contender.
(To do the calculation properly, take the roughly 9 per cent risk we started with and work out what 6 per cent of that would be. Six per cent of 9 per cent is about 0.5 per cent. That is the additional risk of drinking one unit daily, a half of one per cent, or for two drinks daily about 1 per cent. But that is still not as intuitively easy to grasp as it could be. Many people struggle to understand percentages in any form. In a survey, 1,000 people were asked what '40 per cent' meant: (a) one quarter, (b) 4 out of 10, or (c) every 40th person. About one third got it wrong. To ask, as we just did: 'what is 6 per cent of 9 per cent?' will surely baffle even more.)
Two days after the study made so many headlines Sarah Boseley, the Guardian newspaper's level-headed health editor, wrote an article entitled 'Half a Pint of Fear': 'How many of us poured a glass of wine or a stiff gin last night without wondering, if only briefly, whether we might be courting breast cancer? … There will undoubtedly be women who turned teetotal instantly.'
Now why would they do that? Only, as she went on to suggest, because they had been panicked by the 6 per cent figure when in truth the difference a drink a day makes could have, and should have, been presented in a less alarming light. That is, unless creating alarm was the whole point.
Our interest is not advocacy, but clarity. So let us start again without a practice that makes the risk appear as big as possible. Let's do away with percentages altogether and speak, once more as journalists should, of people.
Here is the simpler way to describe what the reports horribly failed to do, looking at the effect of two drinks a day, rather than one, to keep the numbers round.
'In every 100 women, about nine will typically get breast cancer in a lifetime. If they all had two extra drinks every day, about ten would.'
And that's it.
Again, you can quickly see that in a hundred women having two alcoholic drinks every day there would be about one extra case of cancer.
Though 1 woman in 100 is a small proportion, because the British population is large this would still add up to quite a few cases of breast cancer (if all women increased their drinking by this amount). Our point is not to make light of a frightening illness, nor to suggest that the risk of cancer is safely ignored. It is because cancer is frightening that it is important to make sense of the risks in a way most people can understand. Otherwise we are all at the mercy of news reports which appear like bad neighbours, leaning over the fence as we go about our lives and saying with a sharp intake of breath: 'You don't want to be doing that.' And maybe you don't. But let us take the decision on the basis of the numbers presented in a way that makes intuitive human sense.
The number of people affected in every hundred is known by statisticians as a natural frequency. It is not far from being a percentage, but is a little less abstract, and that helps. For a start, it is how people normally count, so it feels more intuitively intelligible. It also makes it much harder to talk about relative differences, harder to get into the swamp of talking about one percentage of another. By the way, if you think we exaggerate the difficulty for many people of interpreting percentages, we will see in a moment how even physicians who have been trained in medical statistics make equally shocking and unnecessary mistakes when interpreting percentages for their patients' test results.
Natural frequencies could easily be adopted more widely, but are not, so tempting the conclusion that there is a vested interest both for advocacy groups and journalists in obscurity. When the information could be conveyed in so plain a fashion, why do both often prefer to talk about relative percentage risks without mentioning the absolute risk, all in the most abstract terms? The suspicion must be that this allows the use of 'bigger' numbers ('six' per cent is big enough perhaps to be a scare, the absolute change 'half of one per cent', or even 'one woman in every 200' is probably less disturbing). Bigger numbers win research grants and sell causes, as well as newspap
ers.
One standard defence of this is that no one is actually lying. That is true, but we would hope for a higher aspiration from cancer charities or serious newspapers and broadcasters than simply getting away with it. The reluctance to let clarity get in the way of a good story seems particularly true of health risks, and isn't only a habit of some journalists. There are, in fact, international guidelines on the use of statistics that warn against the use of unsupported relative risk figures. Cancer Research UK in this case seems either to have been unaware of or to have ignored these guidelines in this case in favour of a punchier press release. When we have talked to journalists who have attended formal training courses in the UK, not one has received any guidance about the use of relative risk figures.
In January 2005 the president of the British Radiological Protection Board announced that risks revealed in new medical research into mobile phones meant children should avoid them. The resulting headlines were shrill and predictable.
He issued his advice in the light of a paper from the Karolinska Institute in Sweden that suggested long-term use of mobiles was associated with a higher risk of a brain tumour known as an acoustic neuroma. But how big was the risk? The news reports said that mobile phones caused it to double.
Once again, almost no one reported the baseline risk, or did the intuitively human thing and counted the number of cases implied by that risk; the one honourable exception we found – all national newspapers and TV news programmes included – being a single story on BBC News Online. A doubling of risk sounds serious, and it might be. But as with our two men in a race, it could be that twice as big, just like twice as good, or twice as bad, doesn't add up to much in the end.
With mobile phones you could begin with the reassurance that these tumours are not cancerous. They grow, but only sometimes, and often slowly or not at all after reaching a certain size. The one problem they pose if they do keep growing is that they put pressure on surrounding brain tissue, or the auditory nerve, and might need cutting out. You might also note that the research suggested these tumours did not appear until after at least ten years of at least moderate mobile phone use. And you might add to all those qualifications the second essential question about risk: how big was it? That is, what was the baseline?
When we spoke to Maria Feychting of the Karolinska Institute, one of the original researchers, a couple of days after the story broke, she told us that the baseline risk was 0.001 per cent or, expressed as a natural frequency, i.e. people, about 1 in 100,000. This is how many would ordinarily have an acoustic neuroma if they didn't use a mobile phone. With ten years regular phone use, the much-reported doubling took this to 0.002 per cent, or 2 people in every 100,000 (though it was higher again if you measured by the ear normally nearer the phone). So regular mobile phone use might cause the tumours in 0.001 per cent of those who used them, or one extra person in every 100,000 in that group.
Would Maria Feychting stop her own children using mobile phones? Not at all: she would rather know where they were and be able to call them. She warned that the results were provisional, the study small, and quite different results might emerge once they looked at a larger sample. In fact, it was usually the case, she said, that apparent risks like this seemed to diminish with more evidence and bigger surveys. Two years later the worldwide research group looking into the health effects of mobile phones – Interphone – of which the Karolinska Institute team was a part, did indeed produce another report drawing on new results from a much larger sample. It now said there was no evidence of increased risk of acoustic neuroma from mobile phones, the evidence in the earlier study having been a statistical fluke, a product of chance, that in the larger study disappeared.
A great many percentage rises or falls, in health statistics, crime, accident rates and elsewhere, are susceptible to the same problem and the same solution. So we can make this the standard response to any reported risk: imagine that it is the difference between those two men in a race. When you are told that one runner/risk is – shock! – far ahead of the other, more than, bigger than the other, always ask: but what's the other one/risk like? Do not only tell me the difference between them. Better still, reports of risks could stick to counting people, as people instinctively do, keep percentages to a minimum and use natural frequencies. Press officers could be encouraged to do the same, and then we could all ask: how many extra people per 100 or per 1000 might this risk affect?
Risk is one side of uncertainty. There is another, potentially as confusing and every bit as personal.
Imagine that you are a hardworking citizen on the edge, wide-eyed, sleep-deprived, while nearby in the moonlight someone you can barely now speak of in civil terms has a faulty car alarm. It's all right; we understand how you feel, even if we must stop short, naturally, of condoning your actions as you decide that if he doesn't sort out the problem, right now, you might just remember where you put the baseball bat.
The alarm tells you with shrill self-belief that the car is being broken into; you know from weary experience that the alarm in question can't tell theft from moped turbulence. You hear it as a final cue to righteous revenge; a statistician, on the other hand, hears a false positive.
False positives are results that tell you something important is afoot, but are wrong. The test was done, the result came in, it said 'yes', but mistakenly, for the truth was 'no'. All tests have a risk of producing false positives.
There is also a risk of error in the other direction, the false negative. False negatives are when the result says 'no', but the truth is 'yes'. You wake up to find, after an uninterrupted night's sleep, that the car has finally been stolen. The alarm – serves him right – had nothing to say about it.
There are a hundred and one varieties of false positive and negative; the cancer clusters in Chapter 3 on chance are probably a false positive, and it is in health generally that they tend to be a problem, when people have tests and the results come back saying they either have or haven't got one condition or another. Some of those test results will be wrong. The accuracy of the test is usually expressed as a percentage: 'The test is 90 per cent reliable.' It has been found that doctors, no less than patients, are often hopelessly confused when it comes to interpreting what this means in human terms.
Gerd Gigerenzer is a psychologist, Director of the Centre for Adaptive Behaviour and Cognition at the Max Planck Institute in Berlin. He asked a group of physicians to tell him the chance of a patient truly having a condition (breast cancer) when a test (a mammogram) that was 90 per cent accurate at spotting those who had it, and 93 per cent accurate at spotting those who did not, came back positive.
He added one other important piece of information: that the condition affected about 0.8 per cent of the population for the group of 40–50-year-old women being tested. Of the twenty-four physicians to whom he gave this information, just two worked out correctly the chance of the patient really having the condition. Two others were nearly right, but for the wrong reasons. Most were not only wrong, but hopelessly wrong. Percentages confused the experts like everyone else.
Quite a few assumed that, since the test was 90 per cent accurate, a positive result meant a 90 per cent chance of having the condition, but there was a wide variety of opinion. Gigerenzer comments: 'If you were a patient, you would be justifiably alarmed by this diversity.' In fact, more than nine out of ten positive tests under these assumptions are false positives, and the patient is in the clear.
To see why, look at the question again, this time expressed in terms that make more human sense, natural frequencies.
Imagine 1,000 women. Typically, eight have cancer, for whom the test, a fairly but not perfectly accurate test, comes back positive in 7 cases with one result wrong. The remaining 992 do not have cancer, but remember that the test can be inaccurate for them too. Nearly 70 of them will also have a positive result. These are the false positives, people with positive results that are wrong.
Now we can see easily that there will be
about 77 positive results in total (the true positives and the false positives combined) but that only about 7 of them will be accurate. This means that for any one woman with a positive test, the chance that it is accurate is low and not, as most physicians thought, high.
The consequences, as Gigerenzer points out, are far from trivial: emotional distress, financial cost, further investigation, biopsy, even, for an unlucky few, unnecessary mastectomy. Gigerenzer argues that at least some of this is due to a false confidence in the degree of certainty conferred by a test that is 'at least 90 per cent accurate'. If positive tests were reported to patients with a better sense of their fallibility, perhaps by speaking of people not percentages, it would ease at least some of the emotional distress. But how does that false confidence arise? In part, because the numbers used are not in tune with the human instinct to count.
Uncertainty is a fact of life. Numbers, often being precise, are sometimes used as if they overcome it. A vital principle to establish is that many numbers will be uncertain, and we should not hold that against them. Even 90 per cent accuracy might imply more uncertainty than you would expect. The human lesson here is that since life is not certain, and since we know this from experience, we should not expect numbers to be any different. They can clarify uncertainty, if used carefully, but they cannot beat it.
Having tried to curb the habit of over-interpretation, we need to restrain its opposite, the temptation to throw out all such numbers. Being fallible does not make numbers useless, and the fact that most of the positives are false positives does not mean the test is no good. It has at least narrowed the odds, even if with nothing like 90 per cent certainty. Those who are positive are still unlikely to have breast cancer, but they are a little more likely than was thought before they were tested. Those who are negative are now even less likely to have it than was thought before they were tested.
The Tiger That Isn't Page 11
