Bad Science
Page 42
* Iain Chalmers was knighted for setting up the Cochrane Collaboration. Being very practical people, the researchers at Cochrane wanted to know if there was any real value in this, so they ran a randomised trial. Were recipients of letters from ‘Iain Chalmers’ more or less likely to respond, they wondered, if he signed his name ‘Sir Iain Chalmers’? A simple system was set up, and just before they were posted, outgoing letters were randomly signed ‘Sir Iain Chalmers’ or just ‘Iain Chalmers’. The researchers then compared the number of replies to each signature: and the ‘Sir’ made no difference at all. This study is published in full – despite reporting a negative result – in the Journal of the Royal Society of Medicine, and it is not a flippant subject for research. There are many knights in medicine; there are troubling things you can do to increase your chance of becoming one; and lots of people think, ‘If I was a knight, people would take my good ideas much more seriously.’ The paper is titled ‘Yes Sir, No Sir, not much difference Sir’.54 After reading it, you can relax your ambitions.
* Setting out a simple list of trials is also important for other reasons, including something called ‘duplicate publication’. A British anaesthetist called Martin Tramèr conducted a review on the efficacy of a nausea drug called ondansetron, and noticed that lots of the data seemed to be replicated. On closer inspection, it turned out that lots of trials had been conducted in lots of different places, and then bundled up into multi-centre trials.77 But the results for many individual patients had been written up several times over, bundled up with other data, in different journals and different papers. Crucially, data which showed the drug in a better light were more likely to be duplicated than the data which showed it to be less impressive; overall this led to a 23 per cent overestimate of the drug’s efficacy.
* If this is of particular interest to you, it’s covered at length in my previous book, Bad Science.
* This is tough, but here’s how to think through the derivation of the 3/n rule, if you’re statistically inclined. Let’s say we’re eating week-old chicken, and the probability of death is 0.2, so the probability of no death is (1–0.2), which is 0.8. If we have two observations – I eat mouldy chicken twice – then the probability of ‘no death’ is less: it’s 0.8 x 0.8, or 0.64 (so, my chances of death are rising with every meal of mouldy chicken I eat). If I eat mouldy chicken n times, the probability of no death is 0.8^n, or, to go back to where the 0.8 came from, that’s (1–0.2)^n, or more generally (1–risk)^n. Now we want to sit at the other end of the telescope. We want to know the maximum possible risk of something happening that is compatible with never having seen it happen, after n observations (or mouldy-chicken meals), with at most a 5 per cent margin of error. In equation terms, we would say that (1–risk)^n equals 0.05, or rather, since we’re not interested in (1–risk), but in (1–maximum risk), we’d say (1–maximum risk)^n=0.05. Now we just have to rearrange that equation, to make it give us maximum risk when we know n. The calculus-wrestling goes: 1–maximum risk = 0.05^(1/n), and for n greater than 30 that’s approximately the same as 1–maximum risk = 1–(3/n). We’re nearly there: take away the ‘one minus’ on both sides and you have maximum risk = 3/n. That may have been a bit tougher than your average Vorderman maths session, but it is much more useful. ‘I’ve Never Met a Nice South African’ is a racist song about racists. Now you know to ask: ‘How many have you met?’
* Was that a mistake? The ‘transparency guidelines’ themselves are weak and confused: they give no deadline for reporting, for example. The EMA objected strongly when HAI’s report was published in 2010. You are mistaken, it said: patient groups tell us about their funding, they don’t tell the public, and neither do we. I think it’s fair to say that this is indicative of the EMA’s approach to transparency more broadly. The one thing that made these organisations declare their funding was HAI calling up and asking questions about it. That goes to show the power of embarrassment as a public-policy tool.
* The other reason is that court cases set precedents, and make future cases against a company much easier to fight. Because of this, companies settle before cases come to court when they think the result might go against them – meaning that they control the public legal discourse as well as the public academic discourse. A paper on this subject, called ‘Why the Haves Come Out Ahead’, is one of the most widely cited in legal academia.79