by Andrew Leigh
What are the odds of success? A recent US study found that if you started with ten drugs, four would be knocked out by Phase I trials. Four more would flunk Phase II trials. Of the remaining two drugs, one would either fail Phase III trials or get rejected by the Food and Drug Administration.37 In other words, only one in ten drugs that look promising in laboratory and animal tests ends up finding its way onto the market. For drugs used to treat cancer and heart disease, the odds of a drug making its way from lab to market are lower still.
In each case, those taking the new drug are compared against people taking a fake drug. The word ‘placebo’ comes from the Latin placere, meaning ‘to please’. It reflects the fact that people can respond differently when they receive what they believe is an effective treatment. When medical researchers see a change in outcomes among people who have only taken sugar pills, they call it ‘the placebo effect’.
Early research on the placebo effect turns out to have overstated the power of placebos, wrongly conflating the natural tendency of patients to recover with the impact of placebos. Modern researchers now doubt that the placebo effect actually helps our bodies heal faster. But it does seem to affect self-reported impacts, such as pain.38 For alleviating discomfort, the placebo effect works in surprising ways. For example, placebo injections produce a larger effect than placebo pills.39 Even the colour of a tablet changes the way in which patients perceive its effect. Thanks to randomised trials, we know that if you want to reduce depression, you should give the patient a yellow tablet.40 For reducing pain, use a white pill. For lowering anxiety, offer a green one. Sedatives work best when delivered in blue pills, while stimulants are most effective as red pills. The makers of the movie The Matrix clearly knew this when they devised a moment for the hero to choose between a blue pill and a red pill. Blue would wipe his mind and make him happy; red would show him how truly terrifying the world is.
If we simply compare patients who take a pill with those who do not, then we might wrongly attribute the impact entirely to the active ingredients in the tablet. By contrast, a well-designed randomised trial strips out the placebo effect – for example, by comparing pain levels among patients who are given white sugar pills with pain levels among patients who are given identical-looking white aspirin tablets.
*
Patients with severe emphysema used to be treated with lung volume reduction surgery, until a randomised trial showed that it significantly increased the risk of death.41 After minor strokes, neurosurgeons would once routinely perform an extracranial to intracranial bypass (connecting an artery outside the skull with one inside the skull). The surgery was supported by case studies, but a randomised trial showed that it produced worse outcomes.42 For a patient whose bowel is caught up in scar tissue, experts used to favour laparoscopic surgery to ‘unpick’ the adhesions – until a randomised trial showed that this kind of surgery did not reduce pain or improve quality of life.43 Beta-blockers, which had previously been thought to endanger patients with heart disease, have now been shown in randomised trials to lower the chance of death.44
Among postmenopausal women, early studies of those who chose to take hormone therapy suggested that the treatment might reduce the incidence of cardiovascular disease. By the turn of the twenty-first century, around 90 million hormone therapies were being prescribed for newly postmenopausal American women. Then randomised controlled trials showed that hormone therapy had only negative impacts: raising the risk of stroke and the risk of obstruction of a vein by a blood clot.45 For doctors, changing the advice they gave to patients wasn’t easy. As Chicago physician Adam Cifu describes his experience, ‘I had to basically run back all those decisions with women. And, boy, that really sticks with you, when you have patients saying, “But I thought you said this was the right thing.”’46
Medical ethics dictate that researchers should stop a trial if there is evidence of harm. Until the early 2000s, it was normal to treat severe head injuries with steroid injections. Then a trial in Glasgow began randomising patients to receive either a steroid injection or a placebo injection. Halfway through, researchers saw that the death rate among those who received the steroid was 21 per cent: considerably higher than the 18 per cent death rate among those who received a placebo.47 The results were conclusive enough to stop the study and publish the results. Head injury patients no longer get steroid injections as a routine matter.
Randomised trials have also helped doctors do a better job of screening. For years, doctors have responded to patients with non-specific back pain by ordering CT scans, MRIs or even X-rays. In recent years, randomised trials have shown that the results of such tests don’t help medical professionals treat pain.48 In fact, patients with back pain who were randomly assigned to get an X-ray ended up with a higher level of self-reported pain and more frequent follow-up visits to the doctor.49
An even tougher area is cancer screening. If screening were error-free, it would be straightforward to roll it out. But it turns out that screening comes with costs as well as benefits. In a systematic review of randomised trials of breast cancer screening, Cochrane concluded that ‘for every 2000 women invited for screening throughout 10 years, one will avoid dying of breast cancer and 10 healthy women, who would not have been diagnosed if there had not been screening, will be treated unnecessarily. Furthermore, more than 200 women will experience important psychological distress including anxiety and uncertainty for years because of false positive findings.’50
The European Randomized Study of Screening for Prostate Cancer, covering men in the Netherlands, Belgium, Sweden, Finland, Italy, Spain and Switzerland, is now able to compare mortality rates thirteen years after blood-test screening. With more than 160,000 men in the trial, the death rate is slightly lower for men who have been screened for prostate cancer (commonly known as ‘PSA screening’). But the difference is small – one death averted for every 781 men who are screened – and the researchers are not yet confident that they have enough evidence to justify prostate cancer screening for every man aged over fifty.51
*
For my own part, randomised trials have helped shape how I look after my health. I used to take a daily multivitamin tablet, until I read a study that drew together all the available randomised trials of vitamins A, C and E, beta carotene and selenium.52 The study found that for otherwise healthy people, there is no evidence that extra vitamins make you live longer. If anything, those who took vitamin supplements seemed to live shorter lives. Not wanting to send myself to an early grave, I stopped taking multivitamin tablets.
The same goes for fish oil. Based on a 2002 study, millions of people in advanced countries began popping pills made from mushed-up sardines and anchovies.53 Yet a decade later, a much larger, systematic review of randomised studies found no evidence that omega-3 supplements prevented heart attacks.54 So I dropped the fish oil tablet too.
As for the rest, I can’t help thinking of Tim Minchin’s beat poem ‘Storm’ every time I accidentally wander into the ‘herbal remedies’ section of the supermarket. In it, Tim imagines himself responding to an advocate of natural medicine:
‘By definition’, I begin
‘Alternative medicine’, I continue,
‘Has either not been proved to work,
Or been proved not to work.
Do you know what they call “alternative medicine”
that’s been proved to work? “Medicine”.’
I love running, so I’m always on the lookout for randomised trials on exercise science. After reading randomised trials, I’ve opted to choose my running shoes based on comfort, moving away from the ‘stability’ models that I’d been wearing for many years.55 After a marathon, I’ll wear compression socks, since an Australian trial showed that they significantly boost recovery.56 When training, I’ll try to include some high-intensity bursts, based on a randomised trial that found that the cardiovascular benefits of sprint training are five times greater than for moderate exercise.57
Ar
ound home, when I have to remove a bandaid from one of my sons, I’ll remind them that a randomised trial by James Cook University researchers found that the fast approach was less painful than the slow approach.58 When I sip my morning brew, I take pleasure in the randomised evidence showing that coffee protects against DNA breaks.59 And after reading the evidence on annual medical check-ups, I’m persuaded that they do not reduce my chance of falling ill, but do add to the cost of the health care system. For example, in the United States, annual physicals account for one-tenth of doctor visits, despite expert bodies recommending against them for people who aren’t showing any symptoms of illness.60
*
Medical researchers were among the earliest pioneers of randomised trials. Indeed, I’ve chosen to start this book with health care precisely because it’s so far ahead of many other fields. One of the reasons that modern medicine is saving more lives than ever before in human history is its willingness to test cures against placebos or the best available alternative. If it works, we use it; if not, it’s back to the lab. In just one field – strokes and neurological disorders – there are around 50,000 Americans alive today thanks to recent randomised trials.61 For every new treatment – AIDS drugs, the human papillomavirus vaccine, magnetic resonance imaging, genetic testing – medicine has discarded old ones – bloodletting, gastric freezing, routine circumcision and tonsillectomy.62
But there is still more that medicine can do to benefit from randomised trials.63 As we have seen, surgical randomised trials remain comparatively rare, with hospitals conducting tens of thousands of medical procedures every year that are not supported by good evidence. Surgeon Ian Harris gives the example of spine fusion surgery for back pain: an operation now performed on 1 in 1000 Americans each year, even though randomised trials show no better results than for intensive rehabilitation.64 Harris notes that ‘the more you know, the harder it gets . . . a conflict develops between what you understand to be true, based on scientific research . . . and what everyone else is doing.’65 Surgeon and author Atul Gawande argues that ‘pointless medical care’ costs hundreds of billions of dollars annually.66 Each year, one in four Americans receives a medical test or treatment that has been proven through a randomised trial to be useless or harmful.67 An Australian study identified over 150 medical practices that are commonly used despite being unsafe or ineffective.68 The randomistas not only need to produce more evidence, they also need to do a better job at publicising what they know already.
3
DECREASING DISADVANTAGE, ONE COIN TOSS AT A TIME
The first time Daniel’s mum threw him out of home, he was thirteen.1 He’d grown up around drugs, alcohol and dysfunction, and thought it was ‘fun . . . a big adventure’. Through his teenage years, he describes a cycle of his mother saying, ‘Come back’, ‘Get out’, ‘Come back’. When Mum didn’t want him at home, Daniel couch-surfed with friends or slept on the streets. Pretty soon, he says, ‘I didn’t worry too much about where I was, I would just get really drunk and lie down.’ Daniel stumbled from alcohol to marijuana, pills and ice. He began stealing from strangers and family. Because of the methamphetamines, Daniel’s teeth had holes in them, and his face was covered in scabs. He lost 25 kilograms.
When Daniel’s brother offered a few words of advice, Daniel took a swing at him with a machete. The blade missed, and the incident finally made Daniel realise the extent to which his life had got out of control. He stopped the drinking and drugs, found a place to live and finally got a job.
Hard-core poverty almost always involves more than one source of stress. Daniel’s case involved crime, drugs and unemployment, compounded by poor education, few friends and bad health. And yet he was lucky because he managed to get help at the age of twenty-one. What can we do to help people who have been homeless for decades?
In Melbourne, the Sacred Heart Mission has been working closely with long-term homeless people since 1982. A few years ago, the organisation proposed to trial a new intensive casework program, targeted at people who had been sleeping rough for at least a year. When they pitched the idea to their philanthropic partners, one donor urged that it be evaluated through a randomised trial.2
Guy Johnson, who worked in community housing and would eventually help conduct the research, was pretty sceptical at first.3 People in the community sector, he told me, ‘freak out at the word experimental’, and prefer to select participants based on need, not chance. But Johnson came to regard randomisation not only as the most rigorous method for evaluating the program, but also the fairest way of deciding who got the service.
Discussing the prospect of such an experiment with homeless people, the research team were reminded that missing out is a regular part of these people’s lives. Social service agencies sometimes ‘cherrypick’ clients or reject those who have been difficult in the past. People with complex needs can miss out altogether. When staff at an agency know they are being externally evaluated, they can engage in ‘soft targeting’ to make their programs look better.
A randomised experiment is different. Anyone who satisfies the basic entry criteria has the same probability of getting into the program. This means that everyone starts with an equal chance. Participants may not be happy when they find themselves in the control group: one person’s response was a simple ‘fuck you’. But the people conducting the study found that most homeless people understood the importance of having a credible comparison group. ‘Rather than being cruel and unjust,’ wrote researchers Guy Johnson, Sue Grigg and Yi-Ping Tseng, ‘we felt that randomisation was the most fair, equitable and transparent means of allocating places in, and evaluating the impact of, social welfare programs.’4
The ‘Journey to Social Inclusion’ experiment was Australia’s first randomised trial of a homelessness program. The intervention lasted for three years.5 For the forty or so people in the treatment group, it provided intensive support from a social worker, who was responsible for only four clients. This caseworker might help them find housing, improve their health, reconnect with family and access job training. Another forty people in the control group did not receive any extra support. Both groups were paid $30 to answer surveys every six months.
What might we expect from the program? If you’re like me, you’d have hoped that three years of intensive support would see all participants healthy, clean and employed. But by and large, that’s not what the program found. Those who were randomly selected into the program were indeed more likely to have housing, and less likely to be in physical pain. But Journey to Social Inclusion had no impact on reducing drug use or improving mental health. In fact, those who received intensive support were more likely to be charged with a crime (perhaps because having stable housing made it easier for the police to find them). At the end of three years, just two people in the treatment group had a job – the same number as in the control group.6
While it’s disappointing that the program didn’t bring most participants back into mainstream society, it’s less surprising once you begin to learn about the people it seeks to assist. In many cases, they were abused in childhood (the mother of one participant used to put Valium in the child’s breakfast cereal). Most had used drugs for decades, and they were used to sleeping rough. Few had completed school or possessed the skills to hold down a regular job. If they had children of their own, more often than not they had been taken away by child protection services.
The Journey to Social Inclusion program is a reminder of how hard it is to turn around the living standards of the most disadvantaged. If you’ve been doing drugs for decades, your best hope is probably a stable methadone program. If you’re in your late forties with no qualifications and no job history, a stable volunteering position is a more realistic prospect than a steady paycheck. If all your friends have criminal histories, it’ll take more than a day a week with a social worker for you to build a stable social network. Change is possible, but it is more likely to be incremental than immediate. Hollywood loves to depict overnigh
t transformations, but the more common trajectory for someone recovering from deep trauma looks more like two steps forward and one step back.
Unless we properly evaluate programs designed to help the long-term homeless, there’s a risk that people of goodwill – social workers, public servants and philanthropists – will fall into the trap of thinking it’s easy to change lives. There are plenty of evaluations of Australian homelessness programs that have produced better results than this one. But because none of those evaluations was as rigorously conducted as this one, there’s a good chance they’re overstating their achievements.
*
In Los Angeles, Maricela Quintanar said the biggest difference was the quiet. Gone was the party music, the sound of drug deals, the gunshots.7 ‘The only noise here is the cars,’ she told a reporter.
It was 1997, and 28-year-old Maricela had moved with her family from a public housing project on the east side of the city to a privately rented apartment on the west side. Five years earlier, the videotaped beating of Rodney King had led to riots that left more than fifty people dead. Now, Maricela’s family was part of an experiment to test one of the biggest questions in social science: how much do neighbourhoods matter?
For decades, scholars had argued over the causes of poverty, debating the effect of factors such as money, motivation and race. Many also believed that bad neighbourhoods kept poor people poor. If true, this had profound implications for poverty, suggesting that governments should worry not only about the adequacy of the social safety net, but also about geography.
