by Andrew Leigh
The best non-government organisations are always looking for ways to put their programs to the test, and so improve what they do. They know that a lousy outcome for a program is ultimately a great result for the community, because it means we can stop spending money in ways that don’t work.
Randomised trials are at their best when the results make us a little uncomfortable. As someone who tends to regard exam results as a useful indicator of student aptitude, I was troubled when I read a study carried out in Chicago that provided primary school students with an unexpected incentive just before the test started.56 Immediately before the exam began, some students were told that if they beat their score on the last test, they would receive a prize. Even small rewards increased students’ scores by the equivalent of several months’ learning. The youngest students were most susceptible to a cheap trophy, while older students responded to $20 in cash. The incentives were especially effective if the researchers placed the prize on the student’s desk and said that it would be taken away if they failed. Conversely, prizes had little impact if the researchers promised to hand them over in a month’s time. Since students only learned about the prizes a few minutes before the exam, the sizable results suggest that most Chicago school students aren’t trying their hardest on the typical test. In turn, this implies that test scores are a rougher measure of ability than researchers had previously thought.
Carrying out randomised evaluations can even be done on the largest of programs. As we’ll see in Chapter 7, researchers in developing countries have used randomised trials to test the benefits of changes as massive as opening new schools in a village or doubling teacher pay in a school.
But sample size isn’t the only thing that matters – it’s also whether you’re testing something that might reasonably be expected to work at all. In New York, a randomised trial across more than 600 schools set out to test whether teacher merit pay was effective. But rather than rewarding individual teachers, the system depended on the performance of an entire school. Given that the typical New York school has sixty teachers, it was very unlikely that any single teacher’s efforts would affect their school’s final results. If that wasn’t enough, the school outcome was based on a complex formula that was extremely difficult for most teachers to understand.57 Unsurprisingly, the experiment found no impact of merit pay on test scores.58 This is an accurate assessment of that kind of merit pay plan, but should not be extrapolated to conclude that merit pay never works.59
A similar problem arose with the Project STAR experiment, a randomised trial of class size reductions. Conducted in Tennessee in the late 1980s, the experiment showed that pupils scored better in smaller classes.60 However, it has been suggested that this impact was due to the fact that the treatment group teachers knew that if the study showed a positive effect, then class sizes would be reduced permanently across the state.61 If this critique is true, then this suggests that class size reductions do have the potential to produce learning gains, but only if teachers face strong incentives to produce better outcomes.
*
At the college level, randomised experiments are proliferating. In Ohio and North Carolina, researchers worked with tax preparation company H&R Block to identify low-income families with a child just about to finish high school.62 Half of these families were randomly offered assistance in completing a financial aid application. From the perspective of the tax preparer, this process took about eight minutes and cost less than $100 (including the cost of the software). But it made a significant difference to the families. Two years later, the children of those who had received help applying for financial aid were one-quarter more likely to be enrolled at university.
Because children whose parents did not attend university often lack basic information about the college application process, modest interventions can have large impacts. In Ontario, a three-hour workshop for Year 12 students raised college attendance rates by one-fifth, relative to a randomised control group.63 In regional Massachusetts, peer support provided by text message raised the odds that Year 12 students would enrol in college.64 In Chile, a randomised trial that provided information about earnings made poor students more likely to enrol in courses with higher earning potential.65 Among low-income, high-achieving US high school seniors, an experiment that gave students more information about college quality markedly increased the odds that they chose an institution to which they were well-matched.66
Randomistas have even shed light on the direct benefits of attending university. Under normal circumstances, it’s impossible to know the counterfactual to attending college. But in the Netherlands, an unusual randomised trial provided one answer. With medical schools massively oversubscribed, the Dutch government decided to allocate places based on a lottery. This makes it possible to compare the earnings of the few fortunate students who are admitted with their less gelukkig counterparts. It turns out that a spot at medical school boosts earnings by 50 per cent, equivalent to around €1 million in lifetime earnings.67
Admission isn’t all that counts. Fail to finish a degree, and you’re less likely to enjoy its benefits. Across advanced countries, only four out of ten bachelor’s students graduate on time.68 Even three years after they were supposed to have walked across the stage, just seven out of ten have a bachelor’s degree. Is there anything randomised trials can teach us about improving university completion rates?
As it turns out, there is. In one experiment, personalised coaching increased completion rates for at-risk students.69 Another showed that a combination of academic support services and financial incentives reduced dropout rates – though only for female students.70 Given that college dropout is expensive for institutions and for individuals, both these programs seem to represent good value for money.
There is something fitting about applying the philosophy of ‘test, learn, adapt’ to improving education. But can the same philosophy that helped make Sesame Street one of the world’s most effective educational television programs be applied to police and prisons?71 Is it feasible or practical to reduce crime and incarceration through randomised trials?
6
CONTROLLING CRIME
A house in suburban Canberra had been the target of multiple burglaries. For the sixth time, someone had broken in through a window, and stolen items from the bedroom of the family’s nine-year-old son.1 But this time the culprit had been caught in the act. It was the nine-year-old boy from next door – nabbed with a pillowcase full of Lego.
When police officer Rudi Lammers was called to the scene, he decided not to simply follow the usual processes for dealing with young offenders. Instead, he sat down with the two nine-year-olds and asked the victim, ‘What do you think we should do?’
The reply surprised him. The victim tipped out half the Lego from the pillowcase, and gave the rest to the thief. Then he said, ‘Any time you want to play Lego, come over. But can you come through the front door? Because Dad gets really cranky when you come through the window.’
Decades later, Lammers was approached by a man in a Canberra club who whispered in his ear, ‘Do you know who I am? I’m the Lego boy – that experience changed my life.’ The former child thief had stopped stealing after that incident, and now ran a building company.
In an informal sense, Lammers was practising ‘restorative justice conferencing’ – bringing offender and victim together to discuss what the perpetrator should do to repair the harm they have caused. Restorative justice is common in traditional societies, including among the Maori in New Zealand, Native Americans and Indigenous Australians. Since the late 1980s, criminologists have argued that by engendering shame and recompense, restorative justice could be a better deterrent than fines or jail time. But when it first began, many regarded restorative justice as silly or soft.
Since the late 1990s, places as diverse as Indianapolis, London and Canberra have been running experiments in which offenders were randomly directed either into restorative justice or the traditional judicial proces
s. Some kinds of cases – such as family violence or fraud – aren’t suitable for restorative justice, but the experiments covered a wide range of other crimes, including assault, robbery and car theft.
Combining the results of ten restorative justice experiments from around the world – a process known as meta-analysis – researchers concluded that it does cut crime.2 In the two years afterwards, offenders who went through the restorative justice process were significantly less likely to commit another crime. For society, the benefits more than covered the costs. In the London experiment, the gains from crime reduction were worth fourteen times more than the cost of running the restorative justice process. And in a result that surprised some theorists, restorative justice seems to work particularly well for violent crimes.
Results from the Canberra experiment showed that restorative justice also helped victims. Compared with those cases that were randomised to go to court, victims of violence were less likely to fear that the offender would hurt them again if their case went through restorative justice. Under restorative justice, victims were five times more likely to get a sincere apology. Victims of violence were also asked if they would harm the offender if they got the chance. When cases went to court, nearly half the victims said afterwards that they still wanted to take revenge – compared with less than one in ten cases that went through restorative justice. Since many crimes are motivated by revenge, this suggests that restorative justice may help avoid tit-for-tat cycles of violence.3
In criminal justice the instinctive solution is not always the one that produces the best results. In the United States, the violent crime rate has halved since the early 1990s.4 Meanwhile, the incarceration rate has almost doubled, with nearly 1 per cent of American adults behind bars.5 Black men who do not finish high school have a two in three chance of going to jail at some point in their lives.6 As Senator Cory Booker noted in 2015, ‘Right now, we have more African-Americans under criminal supervision than all of the slaves in 1850.’
Could randomised trials help reduce both crime and incarceration? In this chapter, I discuss four kinds of criminal justice experiments: prevention, policing, punishment and prison. Creating a society with less crime and less punishment requires getting each of these steps right. Can randomised trials help us do it?
*
The exercise is called ‘The Fist’. The young men are split into pairs. One is given a golf ball. The other is told he has thirty seconds to get the ball.
Immediately, students start grabbing, hitting and wrestling.
After the time is up, the teacher asks why no one simply asked for the ball. ‘He wouldn’t have given it,’ says one. ‘He would have thought I was a punk,’ replies another.
Then the teacher turns to those with the ball, and asks how they would have responded to a polite request. ‘I would have given it; it’s just a stupid ball,’ one replies.
The young men – from rough inner-city neighbourhoods – are participating in a crime prevention program called ‘Becoming a Man’. The goal is to shift teenagers from acting automatically to thinking deliberately, recognising that the right strategy on the street might be the wrong approach in the classroom. For example, a young man in a high-crime neighbourhood who complies with requests like ‘give me your phone’ may be seen as a soft target for future crimes. By contrast, if the same young man fails to comply with a request by his teacher to sit down in class, he may be suspended from school.
‘Becoming a Man’ doesn’t tell youths never to fight. Unlike children in affluent suburbs, teenagers growing up in high-poverty neighbourhoods may need to act tough just to stay safe. So the program’s role-play exercises encourage teenagers to choose the right response for the situation. Making eye contact could be fatal when walking past a rival gang member, but is essential in a job interview. Based on cognitive behavioural therapy, ‘Becoming a Man’ aims to get youths to slow down, judge the situation and deliberately choose whether to comply, argue or fight back.
Does it work? From 2009 to 2014, researchers in Chicago carried out two randomised trials, in which teenagers were randomly assigned into ‘Becoming a Man’ programs or after-school sports.7 ‘Becoming a Man’ cut arrests by a large amount: between one-third and one-half. Some researchers now think that reducing ‘automaticity’ – the tendency of young men to instinctively lash out – may do more to improve the lives of young men than standard academic remediation and job training programs. Edinson, a Year 12 student at Amundsen High School, summed up the program’s philosophy: ‘A boy has problems. A man finds solutions to his problems.’8
Remarkably, the effects of cognitive behavioural therapy seem to show up in other contexts too. In war-torn Liberia, researchers recruited nearly a thousand of the most violent men in the nation’s capital, and then randomly offered some of them the chance to complete a short course designed to reduce automaticity and improve self-awareness.9 A year later, men who had gone through cognitive behavioural therapy were less likely to deal drugs, steal or carry a weapon. The effect was particularly strong for those who had received a second randomised treatment – a US$200 cash grant. Together, the impact of therapy and a one-time payment halved the number of crimes from sixty-six to thirty per year.
One of the things I like about the Liberian experiment is that Chris Blattman and his co-authors admit they were surprised by their own results. Because the cash grant was a one-off payment, they hadn’t expected it would have any impact on long-term behaviour. The research team are now exploring the mechanism through which a temporary payment can change lives.
There’s a lot we don’t know about the world, so being surprised by experimental findings is perfectly healthy. Indeed, we should be deeply suspicious of anyone who claims they know what works based only on theory or small-scale observation. As the economist John Maynard Keynes once put it, ‘When the facts change, I change my mind. What do you do, sir?’10
Programs that divert troubled young people away from offending can have a huge impact on their life trajectories. In Chicago, African-American high schoolers randomly assigned to get an eight-week part-time summer job were 40 per cent less likely to commit a violent crime. The effect persisted for more than a year after the job ended.11 When prevention programs work, they not only save young people from having to go through life with a criminal record, they also save victims from the pain and cost of crimes that are averted, and save taxpayers the cost of paying for more jails.
*
In the early 1970s, a Michigan bowling alley ran an advertisement that said: ‘Have Some Fun. Beat Your Wife Tonight.’12 It reflected the fact that family violence was considered normal at that time. Abused women often found themselves being asked, ‘What did you do to provoke him?’ Social workers urged women to stay with abusive men for the sake of their children. Hospitals looked the other way as they patched up injuries. Safe shelters were rare. When responding to a family violence call, officers sometimes laughed at the victim. They seldom removed the man from his home. Police typically took a ‘speak softly’ approach, aiming to defuse the tension. One state’s police manual advised officers, ‘Don’t be too harsh or critical.’13
By the late 1970s, police attitudes had begun to shift. The Battered Women’s Movement emerged to challenge the idea that violence in the home should be treated more leniently than violence on the street.14 A study of spouse killings found that most had been preceded by at least five police call-outs in the year before the murder.15 Some argued that lives could have been saved if the police had been tougher, while others thought that arrests would make little difference. Many victims were unwilling to sign complaint statements, fearing that the stigma of arrest could lead the man to take revenge later.
Patrick Murphy, president of the US Police Foundation, argued that police responses were based on little more than ‘hunch, supposition, tradition’. So in 1981 the Foundation engaged in an unusual experiment with the city of Minneapolis, seeking ‘through scientific inquiry . . . to supplant
tradition with fact in resolving the question: How can the police deter future domestic violence?’16
In the Minneapolis Domestic Violence Experiment, officers were given a special pad of report forms which listed three kinds of responses: arrest the perpetrator, send the perpetrator away from the home for eight hours, or provide advice to the couple. The three responses – arrest, send or advise – appeared in random order throughout the report pad. In cases of family violence where the victim was not seriously injured, officers were to follow the action indicated on the top page of the pad.
The results were unequivocal. Measured either by police reports or victim surveys, households where the perpetrator was arrested saw about half as much violence in the following six months as those in which police officers sent the perpetrator away or provided advice to the couple.17 Reading reports of the study, the New York police commissioner immediately told his officers to make an arrest if a victim of family violence wanted to press charges.18 Within months, Dallas, Houston and Minneapolis had changed their policies too.19 A year after the study came out, the national rate of arrest in family violence cases had risen from 10 per cent to 31 per cent.20 Two years later, it was 46 per cent.
In subsequent years, replication studies produced a more nuanced finding, suggesting that the effect of arrest on family violence was stronger when perpetrators were employed than when they were jobless.21 But the shift in policing practice for family violence would almost certainly not have been as sharp without a randomised trial. Understanding the best way for police to respond to family violence remains critical. Globally, half of all female homicide victims are killed by their partners or family members.22 As a United Nations report notes: ‘With bitter irony, women run the highest risk of being killed by those who are expected to care for and even protect them.’23
