As the prisons overflow and the effectiveness of strategies against reoffending seems in universal doubt, often owing to a failure to measure their effects carefully, this strategy – cheap, potentially transformative, carefully measured – remained neglected. Isn't this a little odd? Of course, there is still a possibility that the results fell victim to a rogue, confounding factor or measurement error, but the process seems to have been responsible enough. Nine years on, when we first came across these results, the failure to pursue the findings, to try to replicate the experiment to test again whether the results were chance, was mystifying. Weak numbers and bogus numbers are hurled about with abandon in many comparisons. Here, where the numbers are strongly suggestive and responsibly used, they were ignored. The new trial will be fascinating.
Finally, as promised, the answer to the question of where Britain ranks internationally. Taking the more serious rankings together, it is, according to Christopher Hood … in the bottom third of OECD countries; actually 11th out of 13. But you are sublimely indifferent to such a Byzantine comparison now, aren't you?
12
Correlation: Think Twice
This causes that. Press the remote and the channel changes. Plant the seed and it grows. When the sun shines, it is warmer. Sex makes babies.
Human (and sometimes animal) ability to see how one thing leads to another is prodigious – thank goodness, since it is vital to survival.
But it also goes badly wrong. From applying it all the time, people acquire a headstrong tendency to see it everywhere, even where it isn't. We see how one thing goes with another – and quickly conclude that it causes the other, and never more so than when the numbers or measurements seem to agree.
This is the oldest fallacy in the book, that correlation proves causation, and also the most obdurate. And so it has recently been observed by smart researchers that overweight people live longer than thin people, and therefore concluded that being overweight causes longer life. Does it? We will see.
How do we train the instinct that serves us so well most of the time for the occasions when it doesn't? Not by keeping it in check – it is genius at work – but by refusing to let it stop. Do not rest on the first culprit, the explanation nearest to hand or most in mind. Do not, like Pavlov's dogs, credit the bell with bringing the food, feel your mouth begin to water, and leave it at that. Keep the instinct for causation restless in its search for explanations and it will serve you well.
Loud music causes acne. How else to explain the dermatological disaster area witnessed in everyone wired to headphones audible at fifty paces?
That is a cheap joke. There are many possible causes of acne, even in lovers of heavy metal, the likelier culprits being teenage hormones and diet. Correlation – the apparent link between two separate things – does not prove causation: just because two things seem to go together doesn't mean one brings about the other. This shouldn't need saying, but it does, hourly.
Get this wrong – mistake correlation for causation – and we flout one of the most elementary rules of statistics. When we spot a logical fallacy of this kind lurking behind a claim, we cannot believe anyone could have fallen for it. That is, until tomorrow, when we miss precisely the same kind of fallacy and then see fit to say the claim is supported by compelling evidence. It is frighteningly easy to think in this way. Time and again someone measures a change in A, notes another in B, and declares one the mother of the other.
Try the logic of this argument: the temperature is up, the Norfolk coastline is eroding; therefore global warming is eroding the Norfolk coastline.
Or this: the temperature is up, a species of frog is dying out; therefore global warming is killing the frogs.
Or how about this: the temperature is up, there are more cases of malaria in the East African highlands, global warming is causing more malaria in the East African highlands. QED.
Convinced by these news stories from respectable broadcasters and newspapers? You shouldn't be; they are all causation/ correlation errors, made harder to spot by plausibility (at least to some). Plausibility is often part of the problem, encouraging us to skip more rigorous proof and allowing the causation instinct to settle too quickly: it sounds plausible, so it must be right. Right? Wrong.
In all these cases, environmental campaigners noted that as one measurement – average global temperature – moved, so did another: the position of the coastline, the number of frogs, cases of malaria. They put these facts together and confidently assumed they had added two and two to make four and produce what they called compelling evidence but we might prefer to call classic cases of logical hopscotch, fit for, if not derision, at least serious scepticism. All these claims have been vigorously and credibly challenged, as we shall see.
It is worth saying here and now that this chapter is no exercise in climate change denial. We need to beware another fallacy, namely that because campaigners sometimes make false claims about the effects that therefore no effects exist. That doesn't follow either. And we can note in passing that some critics of global warming have been equally guilty of spectacular numerical sophistry. The point is that even with strong cases, perhaps especially with strong cases of which people are devoutly convinced, they get carried away.
So this is a guide to a certain variety of failed reasoning, but it is a frequent failure in hard cases, given great impetus by numbers. If we state that a rare frog is dying out as a result of global warming, it sounds OK, but lacks beef; it would be more powerful if we could throw in some measurements and say that researchers believe the past decade of record temperatures has led to a 60 per cent decline in the population of red-spotted tree frogs.
Put aside, if you can, your own convictions, and follow us to the point: we are concerned here only with how to avoid mistaking correlation for a causal relationship. Learn how and, whatever side you are on, you can get closer to something more important than conviction: understanding.
It is a peculiar hazard, this tendency to confuse causation and correlation, which is (a) well known and well warned against, yet (b) simultaneously repeated ad nauseam, making it tempting to say that (a) causes (b). It is also a hazard far more widespread than in debate about climate, if more easily spotted, it must be said, in examples like these:
People with bigger hands have better reading ability; so we should introduce hand-stretching exercises in schools.
In Scandinavia, storks are more likely to be seen on the rooftops where larger families live. Therefore storks cause babies.
Less obvious here:
Children who come further down the birth order tend to do less well in school tests. Therefore birth order determines intelligence.
Downright controversial here:
People with multiple sclerosis have lesions in the brain; so if we stop the lesions, we can stop the disease (i.e. the lesions cause the problem).
And here:
Girls at single-sex schools do better than girls in mixed schools, therefore single-sex schools are better for girls.
What seems often to determine how easily we spot causation/ correlation errors is how fast a better explanation comes to mind: thinking of decent alternatives slows conclusions and sows scepticism. Once again, imagination can take you far (though a thirst for more data also helps).
Good prompts to imagination are these straightforward questions: what else could be true of the group, the place, the numbers we are interested in? What other facets do they share, what else do we know that might help to explain the patterns we see? This is where the instinct for seeing causation can be put to good use, by stretching it further than the first answer to hand.
Where shall our imagination begin? With the most comical of our examples. The statement that hand size in children correlates with reading ability is true; but true because …?
Because we generally read better as we grow up, for reasons mostly to do with maturing intelligence and education, and as we grow older our hands grow bigger. Bigger hands are a correla
te of better reading, but the cause lies elsewhere; no case, then, for hand-stretching.
Next, storks and babies. This is harder, since the true explanation is less easy to guess, and there really are more storks on homes with larger families. But where does the causation truly lie? Perhaps because the house tends to be bigger as family size increases, and with more roof space …
In each case there is a third factor that proves to be the genuine explanation: age in the first, house size in the second. That is one typical way for causation/correlation error to creep in. Two things change at the same time, but the reason lies in a third.
Now we begin to see how it works, what about the others, all of which have made the news?
Sufferers of multiple sclerosis have lesions in the brain. The more advanced the illness, the worse the lesions. But do the lesions cause the progressive disability characteristic of that illness? It is plausible – for many years it was thought true – and when a drug called beta interferon was discovered that seemed to arrest the lesions, it was used in the fervent hope that it would slow the disease.
There was only one way to verify the hypothesis, and that was by studying patients over many years to see how fast the illness progressed, relative to the number of lesions and the use of beta interferon. The results, when finally produced in 2005, were bitterly depressing: patients who have taken beta interferon do have fewer lesions, but are no better on average than others who have not. The progressive worsening of other symptoms seems to continue at the same rate in both groups. The lesions were found to be an effect, not a cause, of multiple sclerosis, and beta interferon was, said researchers, to use a mordant analogy, no more than a sticking plaster that failed to treat the cause.
Birth order and intelligence is also tricky. It is, once again, true that the further along the birth order you are, the worse you tend to do in IQ tests: first-borns really do perform best, second-borns next best, and so on, not every time, but more often than not, and there is a plausible explanation (watch out!) that goes like this: the more children a family has, the less parental attention each receives: the first has lots, the second maybe half as much, and so on. This is believable, but does that make it true?
Let's try the imagination test: what else could be true as you pass along the birth order? At third or fourth, or even sixth or seventh, what is plainly true is that we are now looking at a big family. What do we know about big families? One thing we know is that they tend to be of lower socio-economic status. Poorer people tend to have more children, and we also know that the children of poorer families tend, for various reasons, to do less well. So the further down the birth order you are, the more likely you are to come from a poorer family; not always, of course, but is this true often enough to be the explanation for what happens on average?
The evidence is not conclusive, but the answer is 'probably', since it also turns out, when looking at the birth order of children from the same family, that no one has found a significantly consistent pattern of performance; the last born within the same family is, as far as we know, just as likely to do best in an IQ test as the first.
The causation/correlation mistake here has been to try to explain what happens across many families (richer, smaller ones tend to do better; larger, poorer ones not so well) and claim that it applies to birth order within any one family. It looks plausible (that word again) and it seems commonly believed, but it is probably wrong.
Next, gender and school performance. It is true that girls attending single-sex schools do better academically than girls who don't. But does this prove causation, i.e. does it prove that it's single-sex education that produces the better examination results? (We'll put aside the question of what it does to their social education, which is too normative a concept to measure.) Is it, in short, the lack of boys what done it?
Again, we must use our imaginations to ask what else is true of girls in single-sex schools. We must be restless in the search for causation and not settle on the obvious correlation as our culprit. The first thing that's true is that they have relatively wealthy parents; most of these schools are fee-paying. And what do we now know from the previous example about socio-economic status and academic performance? Wealthier families tend to have, for whatever reason, academically higher performing children. Second, single-sex schools are more often selective, so that there will be a tendency to take the more able girls to begin with. So it is not surprising that single-sex schools do better: they take more able girls than other schools and these girls are usually from wealthier families. We have established that they ought to do better for all sorts of reasons, and this without taking into account any consideration of the effect of single-sex teaching.
Spare a thought for the statistician who, asked to settle this question, has to find a way of distilling school results to rid them of the effect of socio-economic background or pupil selection by ability so as to isolate the gender effect. As far as they have been able to do that, the balance of statistical opinion, once they have made these allowances, is that there is no difference.
There is evidence that girls tend to make slightly less inhibited choices of subject in a single-sex school, and it will almost certainly suit some pupils – which may be reasons enough for wanting your daughter to attend – but it cannot be expected to secure better exam results in general than the girls would achieve had they attended a mixed school that took pupils of similar ability.
Perhaps now, if the sensitivity of these subjects hasn't created so much hostility that we've lost our readers, we can turn to one of the most sensitive of all: climate change.
First, malaria in East Africa. It has been known for some time that malaria in highland areas is hindered by low temperatures. These inhibit the growth of the parasites in the mosquito. The Tear Fund was one of several charities to produce evidence of an increased incidence of malaria in the East African highlands and to attribute it to climate change.
There were anecdotes: for example of the man from the highlands who had become landless and was living in poverty because he was bitten, got malaria, couldn't sustain work on the land, lost the land and was forced to work in bonded labour.
But when researchers looked closely at the records, they found no support for the argument. One of them, Dr David Hay, a zoologist from Oxford University, said of the records for that specific area in contrast to global averages, that: 'The climate hasn't changed, therefore it can't be responsible for changes in malaria.' His colleague David Rogers, a professor of ecology, said that some groups responded to this by accepting that there was no change in average climate but arguing that there had been a change in variability of the climate. That is an intelligent proposition, knowing, as we now do, that averages can conceal a lot of variation. So they looked – and found no significant change there either. The researchers concluded that an increase in drug resistance is a more likely explanation for the observed increase in malaria – in this instance. This was a case where there was not even a correlation at the local level, but an assumed connection between what was happening to climate globally and disease locally.
Mary Douglas, an anthropologist, wrote that people are in the habit of blaming natural disasters on things they do not like. But the loose conjunction in the back of the mind of two things both labelled 'bad' does not mean that one causes the other.
The so-called first victim of climate change was the South American golden toad. 'It is likely,' said one campaigner, 'that the golden toad lives only in memory.'
J. Alan Pounds, from the Golden Toad Laboratory for Conservation in Costa Rica, acknowledges that the toads have been badly affected by a disease called the chytrid fungus, but argues: 'Disease is the bullet, climate change is the gun.'
In fact, the fungus does not need high temperatures, and was deadly to the toads anywhere between 4°C and 23°C. Alan Pounds is not convinced: 'We would not have proposed the hypothesis we did if there was not such a strong pattern,' he said. It is quite likely, most scientists b
elieve, that climate change will wipe out some species. It is not at all clear in this case that it has already done so.
Climate change is confidently expected to result in a rise in sea levels. Rising sea levels may well cause coastal erosion. The temptation is to observe coastal erosion and blame climate change, as several TV news reports have done, accompanied by dramatic shots of houses teetering on clifftops. The campaign group Friends of the Earth is more careful. Though deeply concerned about the future effects of climate change on coastal erosion, it says there has been erosion at the rate of about 1 metre a year for the last 400 years in parts of East Anglia: 'This erosion over the centuries is a result of natural processes and sea-level rise from land movements. However, in recent years the rate of erosion appears to have increased at some points along the coast. The causes are poorly understood but in addition to natural processes and sea-level rise, the effects of hard coastal defences are thought to play an important role. Ironically, our attempts to defend against sea-level rise may actually add to coastal erosion.' When Friends of the Earth is cautious, reporters might also think twice.
Untangling climatic causation from correlation is fiendishly hard, and though climate change may make coastal erosion seriously worse in future, it is hard to claim it has made a difference yet (though some argue that coastal storms have worsened as a result of climate change and that these have accelerated coastal erosion). In fact, the rate of sea-level rise was faster in the first half of the twentieth century than in the second.
When a tit-bit of evidence seems to our taste, the temptation is to swallow it. For the non-committed too, this kind of deduction has an appeal – to laziness. It doesn't demand much thought, the nearest suspect saves time, the known villain might as well do. Even with intellectual rigour, mistakes happen, as with beta interferon and multiple sclerosis. During medical trials of new drugs, it used to be customary to record anything that happened to a patient taking an experimental drug and say the drug might have caused it: 'side effects', they were called, as it was noted that someone had a headache or a runny nose and thereafter this 'side effect' was printed forever on the side of the packet. Nowadays these are referred to as 'adverse events', making it clear they might have had nothing to do with the medication.
The Tiger That Isn't Page 19
