Randomistas

by Andrew Leigh

  58Jeremy S. Furyk, Carl J. O’Kane, Peter J. Aitken, et al., ‘Fast versus slow bandaid removal: A randomised trial’, Medical Journal of Australia, vol. 191, 2009, pp. 682–3.

  59T. Bakuradze, R. Lang, T. Hofmann, et al., ‘Consumption of a dark roast coffee decreases the level of spontaneous DNA strand breaks: a randomized controlled trial’, European Journal of Nutrition, vol. 54, no. 1, 2015, pp. 149–56.

  60Ateev Mehrotra & Allan Prochazka, ‘Improving value in health care—Against the annual physical’, New England Journal of Medicine, vol. 373, no. 16, 2015, pp. 1485–7.

  61Specifically, the study found that twenty-eight randomised trials funded between 1977 and 1999 were responsible for an additional 470,000 quality-adjusted life years over a ten-year period: S.C. Johnston, J.D. Rootenberg, S. Katrak, et al., ‘Effect of a US National Institutes of Health programme of clinical trials on public health and costs’, The Lancet, vol. 367, no. 9519, 2006, pp. 1319–27.

  62For other examples of medical procedures discarded when the evidence didn’t suggest they worked, see Thomas B. Freeman, Dorothy E. Vawter, Paul E. Leaverton, et al., ‘Use of placebo surgery in controlled trials of a cellular-based therapy for Parkinson’s disease’, New England Journal of Medicine, vol. 341, no. 13, 1999, pp. 988–92.

  63Some medical subfields are virtually untouched by the randomised trials revolution. For example, randomised trials currently account for only a tiny fraction of the published literature in plastic surgery: Colleen M. McCarthy, E. Dale Collins & Andrea L. Pusic, ‘Where do we find the best evidence?’ Plastic and Reconstructive Surgery, vol. 122, no. 6, 2008, pp. 1942–7.

  64Harris, Surgery, pp. 1285–1326.

  65Harris, Surgery, p. 82.

  66Atul Gawande, ‘Overkill’, New Yorker, 11 May 2015.

  67Aaron L. Schwartz, Bruce E. Landon, Adam G. Elshadug, et al., ‘Measuring low-value care in Medicare’, JAMA Internal Medicine, vol. 174, no. 7, 2014, pp. 1067–76.

  68Adam G. Elshaug, Amber M. Watt, Linda Mundy & Cameron D. Willis, ‘Over 150 potentially low-value health care practices: an Australian study’, Medical Journal of Australia, vol. 197, no. 10, 2012, pp. 556–60.

  3 DECREASING DISADVANTAGE, ONE COIN TOSS AT A TIME

  1Daniel’s story is told in Scott Hannaford, ‘Violence, lack of housing and family breakdown leaving young Canberrans homeless’, Canberra Times, 26 September 2015, p. 1.

  2Author’s interview with Yi-Ping Tseng and Sacred Heart Mission social workers.

  3Author’s telephone interview with Guy Johnson.

  4Guy Johnson, Sue Grigg & Yi-Ping Tseng, ‘The J2SI pilot: Using a randomised trial to evaluate an innovative homelessness intervention’, in Gemma Carey, Kathy Landvogt & Jo Barraket (eds), Creating and Implementing Public Policy: Cross-Sectoral Debates, New York: Routledge, 2016, pp. 113–26.

  5Guy Johnson, Daniel Kuehnle, Sharon Parkinson, et al., Sustaining Exits from Long-Term Homelessness: A Randomised Controlled Trial Examining the 48-Month Social Outcomes from the Journey to Social Inclusion Pilot Program, St Kilda, Vic.: Sacred Heart Mission, 2014.

  6The three-year outcome was provided to me via email by Yi-Ping Tseng. At the four-year mark (one year after the program ended), the number of people with a job was zero in the treatment group and one in the control group.

  7Quoted in Larry Gordon, ‘A social experiment in pulling up stakes’, Los Angeles Times, 23 September 1997.

  8For a chronological summary of the results, see Jonathan Rothwell, ‘Sociology’s revenge: Moving to Opportunity (MTO) revisited’, Social Mobility Memos, Washington, DC: Brookings Institution, 6 May 2015.

  9Quoted in S.J. Popkin, L.E. Harris & M.K. Cunningham, Families in Transition: A Qualitative Analysis of the MTO Experience, final report submitted to the US Department of Housing and Urban Development, Office of Policy Development and Research, 2002, p. 49.

  10Quoted in Popkin, Harris & Cunningham, Families in Transition, p. 50.

  11Raj Chetty, Nathaniel Hendren & Lawrence F. Katz, ‘The effects of exposure to better neighborhoods on children: New evidence from the moving to opportunity experiment’, American Economic Review, vol. 106, no. 4, 2016, pp. 855–902.

  12Philip K. Robins, ‘A comparison of the labor supply findings from the four negative income tax experiments’, Journal of Human Resources, vol. 20, no. 4, 1985, pp. 567–82; Robert A. Moffitt, ‘The negative income tax: Would it discourage work?’ Monthly Labor Review, 1981, pp. 23–7.

  13Robert A. Moffitt, ‘The negative income tax and the evolution of U.S. welfare policy’, Journal of Economic Perspectives, vol. 17, no. 3, 2003, pp. 119–40; Robert A. Levine, Harold Watts, Robinson Hollister, Walter Williams, et al., ‘A retrospective on the negative income tax experiments: Looking back at the most innovate field studies in social policy’ in Karl Widerquist, Michael Anthony Lewis & Steven Pressman (eds), The Ethics and Economics of the Basic Income, Aldershot: Ashgate Publishing, 2005, pp. 95–108.

  14This is based on the finding that the earned income tax credit reduces the national poverty rate by 1.5 percentage points: Raj Chetty, John N. Friedman & Emmanuel Saez, ‘Using differences in knowledge across neighborhoods to uncover the impacts of the EITC on earnings’, American Economic Review, vol. 103, no. 7, 2013, pp. 2683–721.

  15Willard G. Manning, Joseph P. Newhouse & Naihua Duan, ‘Health insurance and the demand for medical care: Evidence from a randomized experiment’, American Economic Review, vol. 77, no. 3, 1987, pp. 251–77; Kathleen N. Lohr, Robert H. Brook, Caren J. Kamberg, et al., ‘Use of medical care in the RAND Health Insurance Experiment: Diagnosis-and service-specific analyses in a randomized controlled trial’, Medical Care, vol. 24, no. 9, 1986, S1–S87.

  16Joseph P. Newhouse, ‘Consumer-directed health plans and the RAND Health Insurance Experiment’, Health Affairs, vol. 23, no. 6, 2004, pp. 107–13; Robert H. Brook, Emmett B. Keeler, Kathleen N. Lohr, et al., The Health Insurance Experiment: A Classic RAND Study Speaks to the Current Health Care Reform Debate, Santa Monica, CA: RAND Corporation, RB-9174-HHS, 2006.

  17Amy Finkelstein, Sarah Taubman, Bill Wright, et al., ‘The Oregon Health Insurance Experiment: Evidence from the first year’, Quarterly Journal of Economics, vol. 127, no. 3, 2012, pp. 1057–1106.

  18The US Vietnam draft had taken place in earlier years, but 1969 was the first year that the birthdate lottery was used. Due to inadequate mixing of the balls, December birthdates tended to be chosen earlier, a problem that was corrected in subsequent lottery drawings.

  19Quoted in Wesley Abney, ‘Live from Washington, It’s Lottery Night 1969!’, HistoryNet, 25 November 2009.

  20Kerry Pardue, ‘When were you in the war’, Veteran Stories, National Vietnam Veterans Museum, undated.

  21Joshua D. Angrist, ‘Lifetime earnings and the Vietnam era draft lottery: Evidence from social security administrative records’, American Economic Review, 1990, vol. 80, no. 3, pp. 313–36; Joshua D. Angrist, Stacey H. Chen & Jae Song, ‘Long-term consequences of Vietnaam-era conscription: New estimates using social security data’, American Economic Review, vol. 101, no. 2, 2011, pp. 334–8.

  22Joshua D. Angrist & Stacey H. Chen, ‘Schooling and the Vietnam-era GI Bill: Evidence from the draft lottery’, American Economic Journal: Applied Economics, vol. 3, no. 2, 2011, pp. 96–118.

  23Angrist & Chen, ‘Schooling’; D. Conley & J. Heerwig, 2012, ‘The long-term effects of military conscription on mortality: Estimates from the Vietnam-era draft lottery’, Demography, vol. 49, pp. 84155.

  24Jason M. Lindo & Charles Stoecker, ‘Drawn into violence: Evidence on “what makes a criminal” from the Vietnam draft lotteries’, Economic Inquiry, vol. 52, no. 1, 2014, pp. 239–58. See also Chris Rohlfs, ‘Does combat exposure make you a more violent or criminal person? Evidence from the Vietnam draft’, Journal of Human Resources, vol. 45, no. 2, 2010, pp. 271–300.

  25Robert S. Erikson & Laura Stoker, ‘Caught in the draft: The effects of Vietnam draft lottery status on political attitudes’, American
Political Science Review, vol. 105, no. 2, 2011, pp. 221–37.

  26Peter Siminski, Simon Ville & Alexander Paull, ‘Does the military turn men into criminals? New evidence from Australia’s conscription lotteries’, Journal of Population Economics, vol. 29, no. 1, 2014, pp. 1–22; David W. Johnston, Michael A. Shields & Peter Siminski, ‘Long-term health effects of Vietnam-era military service: A quasi-experiment using Australian conscription lotteries’, Journal of Health Economics, vol. 45, no 1, 2016, pp. 12–26. After the war ended, there is no evidence of excess mortality among veterans, relative to non-veterans: Peter Siminski & Simon Ville, ‘Long-run mortality effects of Vietnam-era army service: Evidence from Australia’s conscription lotteries’, American Economic Review, vol. 101, no. 3, 2011, pp. 345–9.

  27Peter Siminski, ‘Employment effects of army service and veterans’ compensation: Evidence from the Australian Vietnam-era conscription lotteries’, Review of Economics and Statistics, vol. 95, no. 1, 2013, pp. 87–97.

  28For a useful summary of the experimental job training studies, see Heckman, LaLonde & Smith, ‘The economics and econometrics’. Evidence suggests that in developing countries, job training may be useful for youths: Attanasio, Kugler & Meghir, ‘Subsidizing vocational training’.

  29As one study sums it up: ‘This finding should not be surprising, because most of these programs cost only a few thousand dollars or less per participant . . . To expect such programs to raise participants’ subsequent annual earnings by several thousand dollars would imply that these social investments consistently have an extraordinary rate of return.’ Heckman, LaLonde & Smith, ‘The economics and econometrics’.

  30Ron Haskins, ‘Social programs that work’, New York Times, 31 December 2014.

  31J.M. Pedersen, M. Rosholm & M. Svarer, ‘Experimental evidence on the effects of early meetings and activation’, IZA Discussion Paper 6970, Bonn: IZA, 2012; J. Kluve, ‘The effectiveness of European active labor market programs’, Labour Economics, vol. 17, no. 6, 2010, pp. 904–18; B. Meyer, ‘Lessons from the U.S. unemployment insurance experiments’, Journal of Economic Literature, vol. 33, 1995, pp. 91–131. An Australian randomised trial of caseworker meetings found that the intervention group worked fewer hours, but because the meeting was also accompanied by a review of the unemployed person’s eligibility for benefits, this may simply reflect a desire among recipients to keep their hours low enough that their payments did not get cut off. Robert Breunig, Deborah A. Cobb-Clark, Yvonne Dunlop & Marion Terrill, ‘Assisting the long-term unemployed: Results from a randomised trial’, Economic Record, vol. 79, no. 244, 2003, pp. 84–102.

  32Michael Rosholm, ‘Do case workers help the unemployed? Evidence for making a cheap and effective twist to labor market policies for unemployed workers’, IZA World of Labor, Bonn: IZA, 2014, p. 72.

  33Steffen Altmann, Armin Falk, Simon Jäger & Florian Zimmermann, ‘Learning about job search: A field experiment with job seekers in Germany’, CESifo Working Paper No. 5355, Munich: CESifo.

  34Bruno Crépon, Esther Duflo, Marc Gurgand, et al., ‘Do labor market policies have displacement effects? Evidence from a clustered randomized experiment’, Quarterly Journal of Economics, vol. 128, no. 2, 2013, pp. 531–80.

  35‘Finland tests a new form of welfare’, The Economist, 24 June 2017.

  36Kristen Underhill, Paul Montgomery & Don Operario, ‘Sexual abstinence only programmes to prevent HIV infection in high income countries: Systematic review’, British Medical Journal, vol. 335, no. 7613, 2007, pp. 248–52.

  37Janet Elise Rosenbaum, ‘Patient teenagers? A comparison of the sexual behavior of virginity pledgers and matched nonpledgers’, Pediatrics, vol. 123, no. 1, 2009, e110-e120.

  38Alba DiCenso, Gordon Guyatt, Andrew Willan & Lauren Griffith, ‘Interventions to reduce unintended pregnancies among adolescents: Systematic review of randomised controlled trials’, British Medical Journal, vol. 324, no. 7351, 2002, p. 1426; Kristen Underhill, Paul Montgomery & Don Operario, ‘Sexual abstinence only programmes to prevent HIV infection in high income countries: systematic review’, British Medical Journal, vol. 335, no. 7613, 2007, pp. 248–52; Heather D. Boonstra, ‘Advancing sexuality education in developing countries: Evidence and implications’, Guttmacher Policy Review, vol. 14, no. 3, 2011, pp. 17–23. One rare example of a successful program to reduce adolescent pregnancy is the randomised evaluation of ‘Teen Options to Prevent Pregnancy’ in Ohio, a program that reduced repeat pregnancies among young mothers by supporting them to devise an appropriate birth control plan: Jack Stevens, Robyn Lutz, Ngozi Osuagwu, et al., ‘A randomized trial of motivational interviewing and facilitated contraceptive access to prevent rapid repeat pregnancy among adolescent mothers’, American Journal of Obstetrics and Gynecology, vol. 217, no. 4, 2017, pp. 423.e1–423.e9.

  39The literature on tobacco taxes is reviewed in Michelle Scollo & Margaret Winstanley, Tobacco in Australia: Facts and issues, Melbourne: Cancer Council Victoria, 2015.

  40Kevin G. Volpp, Andrea B. Troxel, Mark V. Pauly, et al., ‘A randomized, controlled trial of financial incentives for smoking cessation’, New England Journal of Medicine, vol. 360, no. 7, 2009, pp. 699–709.

  41Center for Disease Control and Prevention (CDC), ‘Annual smoking-attributable mortality, years of potential life lost, and economic costs – United States, 1995–1999’, Morbidity and Mortality Weekly Report, vol. 51, no. 14, 2002, p. 300. The pattern is similar when smokers quit: Michael T. Halpern, Richard Shikiar, Anne M. Rentz & Zeba M. Khan, ‘Impact of smoking status on workplace absenteeism and productivity’, Tobacco control, vol. 10, no. 3, 2001, pp. 233–8.

  42The productivity estimate from the CDC’s 2002 study was $1760 a year in 1999 dollars, which translates to over $2500 in today’s money. Note that this does not include productivity losses due to absenteeism.

  43Xavier Giné, Dean Karlan & Jonathan Zinman, ‘Put your money where your butt is: a commitment contract for smoking cessation’, American Economic Journal: Applied Economics, vol. 2, 2010, pp. 213–35.

  44Financial incentives don’t always help people give up bad habits. One recent study found that offering people a 10 per cent discount on healthy foods did not significantly change consumption patterns: John Cawley, Andrew S. Hanks, David R. Just & Brian Wansink, ‘Incentivizing nutritious diets: A field experiment of relative price changes and how they are framed’, NBER Working Paper No. 21929, Cambridge, MA: NBER, 2016.

  45Daniel B. Herman, Sarah Conover, Prakash Gorroochurn, et al., ‘Randomized trial of critical time intervention to prevent homelessness after hospital discharge’, Psychiatric Services, vol. 62, no. 7, 2011, pp. 713–19.

  4 THE PIONEERS OF RANDOMISATION

  1Joseph Brent, Charles Sanders Peirce: A Life, Bloomington: Indiana University Press, 1998, p. 40.

  2Charles Sanders Peirce & Joseph Jastrow, ‘On small differences in sensation’, Memoirs of the National Academy of Sciences, Vol. 3, 1885, pp. 73–83

  3Stephen Stigler, Statistics on the Table: The History of Statistical Concepts and Methods, Cambridge: Harvard University Press, 1999, p. 195.

  4Brent, Charles Sanders Peirce, p. 53.

  5For a history and survey of field of experimental psychology, see Raymond Nickerson & Richard Pew, ‘Psychological experimentation addressing practical concerns’ in Alice F. Healy & Robert W. Proctor (eds) Handbook of Psychology, Vol. 4, Hoboken: John Wiley, 2003, pp. 649–76.

  6This account is drawn from Joan Fisher Box, R.A. Fisher, The Life of a Scientist, New York: Wiley, 1978. See also Deborah Nolan & Terry P. Speed, Stat Labs: Mathematical Statistics Through Applications, New York: Springer, 2000, p. 101; David Salsburg, The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, New York: Henry Holt, 2001.

  7Geoffrey Miller, The Mating Mind: How Sexual Choice Shaped the Evolution of Human Nature, London: Heineman, 2000, p. 54

  8World Health Organization, ‘Tuberculosis Fact sheet’, Geneva: World Health Organization, 2017.

 
9This section draws primarily on Lise Wilkinson, ‘Sir Austin Bradford Hill: Medical statistics and the quantitative approach to prevention of disease’, Addiction, vol. 92, nol. 6, 1997, pp. 657–66.

  10Austin Bradford Hill, ‘Principles of medical statistics’, The Lancet, 2 January 1937, pp. 41–3.

  11The co-discoverers of streptomycin were Selman Waksman (who would go on to win the Nobel Prize in 1952) and his postgraduate student, Albert Schatz (who rarely receives his share of the credit).

  12This is the 1946 figure, sourced from Public Health England, ‘Tuberculosis mortality and mortality rate, 1913 to 2013’, 1 January 2013.

  13Veronique Mistiaen, ‘Time, and the great healer’, Guardian, 3 November 2002.

  14Judith Gueron, ‘The politics of random assignment: Implementing studies and impacting policy’, Journal of Children’s Services, vol. 3, nol. 1, 2008, pp. 14–26.

  15Gueron, ‘The politics of random assignment’, p.14.

  16Judith Gueron & Howard Rolston, Fighting for Reliable Evidence, New York: Russell Sage, 2013, pp. xvi–ii.

  17Judith Gueron, ‘Remarks on accepting the Peter H. Rossi Award’, Association for Public Policy Analysis and Management Conference, Los Angeles, 7 November 2008.

  18Gueron & Rolston, Fighting for Reliable Evidence, p. 33.

  19Gueron & Rolston, Fighting for Reliable Evidence, p. 39.

  20Gueron & Rolston, Fighting for Reliable Evidence, p. 71.

  21The woman Reagan referred to was Linda Taylor. Recent investigation suggests that welfare fraud may well have been one of her more minor crimes: Josh Levin, ‘The welfare queen’, Slate, 19 December 2013.

  22William Stevens, ‘The welfare consensus’, New York Times, 22 June 1988; William Stevens, ‘Some preliminary results in the rush from welfare to work’, New York Times, 21 August 1988.

  23Judith Gueron, ‘The politics of random assignment’.

  24Gueron, ‘Remarks’.

  25Judith Gueron, ‘Fostering research excellence and impacting policy and practice: The welfare reform story’, Journal of Policy Analysis and Management, vol. 22, nol. 2, 2003, pp. 163–74.