That is more like it: 'The most likely results' may still be wrong, as all predictions may be wrong, but at least they represent the balance of the evidence from the experiment, not the most outlying part of it.
None of this, incidentally, implies comfort to climate-change deniers: 4°C, even 3°C, would still bring dramatic consequences, though the case would arguably be strengthened – if more modest – by using this overwhelmingly likely result (according to this experiment), rather than one that could be dismissed as scaremongering. It is always worth asking if the number we are presented with is a realistic possibility, or the Papal one.
'It could be you', say adverts for the National Lottery, which is true, though we all know what kind of truth it is. In any case, responsible news reporting owes us something better than a lottery of possibilities. And yet there is an inbuilt bias to news reporting which actually runs in favour of outliers. 'What's the best line?' every news editor asks, and every journalist knows that the bigger or more alarming the number, the more welcomed it will be by the boss. In consequence, the less likely it is that a result is true, the more likely it is to be the result that is reported. If that makes you wonder what kind of business news organisations think they are in, often the answer is the one that interests and excites readers and viewers most. It seems that consumers of news, no less than producers, like extreme possibilities more than likely ones. No wonder we are so bad at probabilities. 'Your daughter could be Pope', says the number in the headline on the news-stand. 'Gosh, really?' say readers, reaching for the price of a copy.
It is a question continually to ask of surprising numbers: if this is new and different, or does the very fact that it is new and different invite caution? Do the numbers mark a paradigm-shift or a rogue result? The answer in the climate-change example is clear-cut, we think. Even some of those involved in the research came to regret the prominence they gave to a freak number.
For a trickier and more colourful case that brings out the judgement between oddity and novelty, try the Hobbit.
Some 18,000 years old, with the consistency of thick mashed potato or blotting paper, they turned up in a sodden cave (described by the journal Nature as 'a kind of lost world') and became a worldwide sensation when news of the find was reported in March 2003. The skeletal remains were soon nicknamed Hobbit Man, though the most complete of the skeletons might have been – it is still disputed – an approximately 30-year-old woman, given the name Little Lady of Flores, or Flo. They were discovered in the Liang Bua cave on the island of Flores in Indonesia; hence the scientific name for what was hailed as an entirely new species of human: Homo floresiensis.
At about 1 metre (39 inches) tall, the 'Hobbits' were shorter than the average adult height of even the smallest modern humans, such as African pygmies (pygmies are defined as having average male adult height of 1.5m or less) and it was this that grabbed the popular imagination.
They also reportedly had strikingly long arms and a small brain. The joint Australian–Indonesian team of palaeoanthropologists had been looking for evidence of the original human migration from Asia to Australia, not for a new species. But these little people seemed to have survived until more recently than any sub-species of human apart from our own and may, some believe, be consistent with sightings of little people known locally as Ebu Gogo, reported until the nineteenth century. Some wonder if they live on in an isolated jungle in Indonesia.
The bones were certainly extremely unusual, apparently like no other discovered remains of any branch of the genus Homo: not Homo erectus, nor modern-day Homo sapiens, nor the Neanderthals. But were they a new species?
Two reasons to pause before judgement came together on 10 February 1863 in New York City – at a wedding: Charles Stratton married Lavinia Warren in a ceremony attended, as reports of the time captured that famous day, by the 'haut ton' of society. At the time, Stratton was 33 inches tall, Warren 34 inches (approximately 85cm). A few inches shorter even than the Hobbits, but intellectually unimpaired and leading a full life, they had a fine, custom-built home paid for by their many admirers and Stratton's wealth from years as a travelling exhibit for P. T. Barnum, for whom he appeared under the stage name General Tom Thumb.
The usually staid New York Observer reported the wedding as the event of the century, if not unparalleled in history:
We know of no instance of the kind before where such diminutive and yet perfect specimens of humanity have been joined in wedlock. Sacred as was the place, and as should be the occasion, it was difficult to repress a smile when the Rev. Mr. Willey, of Bridgeport, said, in the ceremony – 'You take this woman,' and 'You take this man' &c.
Stratton and Warren were obviously the same species as the rest of us, perfect specimens according to the Observer. Their four parents and nine of their brothers and sisters were a more typical height, though the last of Lavinia's sisters, Minnie, was shorter even than she. Their existence within one species tells us, if we didn't know it already, that the human genetic programme can produce extreme variation.
The world's smallest man on record was Che Mah at 2ft 2in. Robert Wadlow, the world's tallest, was 8ft 11in, more than four times as tall.
So imagine that our newlyweds had set up home on an island in Indonesia, started a family, and that their remains went undiscovered until the twenty-first century. How might we describe them? Would we recognise them for what they were, outliers in the wonderfully wide spectrum of human variation, or wonder perhaps if they too were a separate species?
The argument about whether Hobbit Man was a different kind, or simply an extreme variation of an already known kind, still rages. Most recently it has been argued that it was indeed a new species, following scans of the remains of the skull and a computer-generated image of the shape of the brain by a team at Florida State University.
The team was trying to determine whether the cave dwellers from Flores could be explained as the result of disease (microencephaly – having an exceptionally small brain – is a condition we know today), though there may have been some other, unknown syndrome which caused their particular constellation of physical characteristics. Professional rivalries and fights over access to the remains by eager researchers have complicated the investigation. 'We have no doubt it is not a microcephalic,' said Dean Falk, a palaeoanthropologist who worked with the team that discovered the remains. 'It doesn't look like a pygmy either.' The journal Nature, which first published news of the Hobbit discovery, headlined the latest research: 'Critics silenced by scans of Hobbit skull'.
That assessment may prove optimistic. Robert Eckhardt of Penn State University and a sceptic, reportedly said he was not convinced: 'We have some comprehensive analyses underway which I really think will resolve this question. The specimen has multiple phenomena that I would characterise as very strange oddities and probably pathologies.' In 2008, Dr Peter Obendorf from the School of Applied Science at RMIT University, Melbourne, and colleagues, also said they believed the little people were not a new species, but had developed a dwarfism condition because of severe nutritional deficiencies.
Fortunately, it is not necessary to settle the argument to make the point: unexpected and extreme findings might tell us something new and extraordinary; they might also be curious, but irrelevant to the bigger picture. The fact that they are extreme and unexpected is as much a reason for caution as excitement. Statisticians know them as 'outliers'.
Imagine that the heights of every adult human in the country were plotted on a graph. Most would bunch within 10cm or so of the average for men of about 1.75m (5ft 10in) and 1.63m (5ft 4in) for women (Health Survey for England 2004). Some, but not so many, would be further along the graph in one direction or another, at which point we might begin to call them tall or short. A very few, including Robert Wadlow, Che Mah, Lavinia Warren and Charles Stratton, would be at the far ends. The problem with the Hobbit is to decide whether it is simply a human of a kind we already know, from the extreme end of the graph, an outlier with an ill
ness, or if it tells us that we need to create a new graph entirely, and change our assumptions about human evolution. The argument continues.
Remember our three possibilities when faced with a shock figure – an amazing story, a duff number, or a misinterpretation – and then apply them to the discovery of an adult skeleton three feet tall identified as a new species. Of the three, which is most likely?
Is the Hobbit genuinely new, different and warranting a change to the whole map of human evolution? Or is it, like Charles Stratton perhaps, one of Homo sapiens' very own, but slightly erratic, tee shots?
If the number is a genuine Hobbit, fine, we will have to change our thinking. But strange as Tom Thumb is, well, we knew already that the results of human reproduction can be surprising from time to time, that there will always be outliers, in all sorts of ways, but they remain, quite obviously, human. Statistics teaches us that outliers are to be expected, and so there is nothing abnormal about them, but they are certainly atypical. If that's all that was discovered in Liang Bua cave, an atypical human, it doesn't tell us anything new at all; in the scatter plot of human life, it's one wayward dot, and might be a guide to nothing more than its unusual self.
In fact, outliers are usually far less interesting than Tom Thumb. The real Charles Stratton fascinates; most statistical outliers on the other hand are not human but products of an experiment, survey or calculation, observations which are (by definition) atypical, and one of the first principles of statistics is that suspicious data or outliers may need to be rejected, not least because there's every chance that the outlier is simply wrong – measured or recorded inaccurately. Tom Thumb at least was real, even if on the fringes of the distribution. Giving credence to statistical outliers from forecasts, on the other hand, indulges fantasy.
Three years ago, a company produced five scenarios for house prices, four suggesting modest rises, one a sharp fall. Naturally, the headlines said: 'House prices could plunge'. In fact, for two more years, they rose steadily. In all forecasts, timing matters. Like a stopped clock, many will prove accurate, eventually, but that doesn't make them wise or useful at the time.
Some time after the period of this forecast, at the time of writing, UK house prices did seem to be falling, slightly, and the consensus was that they might fall a little further. Once again, there are extreme predictions receiving abundant coverage, of falls of 40 per cent, for example. But we would tend to side more with the majority of tee shots than those to the car park.
The point is that outliers can be a routine fluke of the system, an erratic moment, and there will always be such blips in any distribution of height, house prices, weather forecasts or whatever. They need not be a cause for alarm. They are not necessarily a revelation.
If your business is catching drug cheats in sport, the natural variability that means some people will always be outliers is a headache. Many of the drugs that people take to make them faster or stronger are substances that are already in their system naturally. Athletes want a little more of what seems to make them good.
So cheats are detected by looking for those with extreme levels of these substances in their body. In essence, we identify the outliers and place them under suspicion. The hormone testosterone, for example, occurs naturally, and is typically found in the urine in the ratio of one part testosterone to one part epitestosterone, another hormone. The World Anti Doping Agency says there are grounds for suspicion that people have taken extra testosterone in anyone found with a ratio of 4 parts testosterone to 1 part epitestosterone. The threshold used to be 6:1, but this was considered too lax.
There are two problems. First, there have been documented cases of outlying but entirely innocent and natural testosterone to epitestosterone (T/E) ratios of 10 or 11 to 1, well above the level at which the authorities become suspicious. Second, there are whole populations, notably in Asia, with a natural T/E ratio below 1:1, who can take illegal testosterone with less danger of breaching the 4:1 limit. In short, there is abundant variation.
The first person to be thrown out of the Olympics for testosterone abuse was a Japanese fencer with an astronomical T/E ratio of about 11:1. The Japanese sporting authorities were so scandalised they put him in hospital, under virtual arrest, where his diet and medication could be strictly controlled. They found that his T/E ratio remained unchanged; he was a natural outlier. It took some believing that such odd results could mean nothing, but the evidence was incontrovertible. According to Jim Ferstle, a journalist we interviewed who has followed the campaign against drug taking in sport for twenty years, the man never received so much as an apology.
Until the late 1990s this was the only test for testosterone abuse, even though we have little idea what proportion of the population would have been naturally above the 6:1 ratio which then applied, nor an accurate idea how many would be naturally above the 4:1 ratio that applies now. To add to the problem, it is well known that alcohol can temporarily raise T/E ratios, particularly among women. So there is a danger that if the outliers are simply rounded up and called cheats, there will be honest people unjustly accused.
Fortunately, there is now a second test, which detects whether the testosterone in a suspect sample is endogenous (originating within the body), or exogenous (originating outside). This test, too, has failed in experiments to catch every known case of doping (a group of students was given high levels of testosterone by a Swiss sports-medicine clinic and then tested, but not all were identified by the test as having been doped). This second test has not yet been shown falsely to accuse the innocent, but it can produce an 'inconclusive' result. One athlete, Gareth Turnbull, an Irish 1500-metre runner, told us that he had spent about 100,000 euros on lawyers defending himself against a charge of illicit testosterone use after an adverse T/E test result and an 'inconclusive' second test for the source of the testosterone, all following a night's heavy drinking. He was eventually cleared in October 2006, and in 2008 agreed a financial settlement with his national athletics authority, but not before, as he puts it, it became impossible to Google his name without coming across the word 'drugs'.
As with height and the Hobbit, we need to remember that abnormal is normal, there are always outliers, and we should fully expect to see the computer spit out a figure of 11°C or higher – sometimes; but we should also acknowledge that it may tell us nothing. And if we wish to slip in a change of definition at some point, at which it is decreed that normal stops and suspicion starts, in order to label everyone beyond that point a potential cheat, or a new species, we have to be very sure indeed that redefinition is warranted. If those outliers are produced by forecasts or computer simulations, we might want to discard them altogether, or at least qualify them with the thought of the Pope taking a wayward tee shot.
Certain words invite suspicion that an outlier is in play. When we see phrases of the kind 'may reach', 'could be as high as', 'potentially affect', it's worth wondering whether this is the most likely, or the most extreme possibility (and therefore one of the most unlikely), and then to ask how far adrift it is from something more plausible. Outliers will always pop up from time to time, particularly in forecasts, but these forecasts seldom come to pass. It's a mildly entertaining game, each time we see the words 'could potentially affect' or similar, to supplement them with the mental parentheses: 'but probably won't'.
11
Comparison: Mind the Gap
If we compare thee favourably to a summer's day, you might accept it as a compliment, but not the basis for a league table. People and weather are categorically different, obviously. Such comparisons are impossible – with apologies to Shakespeare – without a good deal of definitional fudge. In a sonnet, we approve, and call it metaphor; but in politics …
And yet politics loves comparison. It is the staple of argument. Every 'this-is-better-than-that' attempts it. Lately, that enthusiasm has spilled into the way we're encouraged to think of schools, hospitals, crime, and much else, through the device of league tables or rankin
gs, and their raw material, the performance assessment: how one compares with another, who's up, who's down, who's top, who's bottom, who's good, bad, 'bog standard' or failing, who shows us 'best practice'. Comparison has become the supreme language of government. It is now, in many ways, the crux of public policy, in the ubiquitous name of offering informed choice.
But politics has precisely that bad habit of fudging the people/ weather problem, and of overlooking definitional differences. To detect this, the principle to keep in mind is one everyone already knows, but which has grown stale with overuse. It is as true and relevant as ever, however disguised in league tables or performance indicators, and it is this: is the comparison of like with like?
The former Presidential contender and mayor of New York, Rudolph Giuliani, has survived prostate cancer. In August 2007, as the campaign for the Presidential primaries kicked off, he used that experience in a radio advertisement to make this dazzling political comparison: 'I had prostate cancer, five, six years ago,' he said. 'My chances of surviving prostate cancer – and thank God I was cured of it – in the United States, are 82 per cent. My chances of surviving prostate cancer in England, are only 44 per cent under socialised medicine.'
It was devastating, if true. The British healthcare system, so often compared favourably to America's, was in fact only half as good at curing cancer. There was even a (somewhat-dated) bar graph from the Commonwealth Fund, a think tank, that seemed to support the argument: the UK's National Health Service, relatively speaking, was a killer.
Was Rudy right? Is the comparison fair? The Commonwealth Fund disowned his interpretation of its data, shown opposite, and a simple question shows why: is it likely that prostate cancer is 2.8 times as common in the United States as in Britain – some protective elixir in British beer perhaps? That is what these figures suggest: incidence of 136 cases for every 100,000 men in the US, but only 49 cases per 100,000 men in the UK. If he is right, we have to explain why American men are falling to this illness dramatically more often than men from any other developed country in the world for which we have data.
The Tiger That Isn't Page 16
