Randomistas

by Andrew Leigh

  21The video is available at https://youtu.be/zh1uoxH9q5g.

  22Francisco Campos, Michael Frese, Markus Goldstein, et al., ‘Teaching personal initiative beats traditional training in boosting small business in West Africa’, Science, vol. 357, no. 6357, 2017, pp. 1287–90.

  23Klaus Schwab (ed.), The Global Competitiveness Report 2016–2017, Geneva: World Economic Forum, 2016.

  24Freedom House, Freedom in the World 2017, New York: Freedom House, 2017.

  25‘State fragility index and matrix 2015’, updated figures for Monty G. Marshall & Benjamin R. Cole, Global Report 2014: Conflict, Governance and State Fragility, Vienna: Center for Systemic Peace, 2014.

  26Jidong Chen, Jennifer Pan & Yiqing Xu, ‘Sources of authoritarian responsiveness: A field experiment in China’, American Journal of Political Science, vol. 60, no. 2, 2016, pp. 383–400.

  27Marianne Bertrand, Simeon Djankov, Rema Hanna & Sendhil Mullainathan, ‘Obtaining a driver’s license in India: An experimental approach to studying corruption’ Quarterly Journal of Economics, vol. 122, no. 4, 2007, pp. 1639–76.

  28Marco Gonzalez-Navarro & Climent Quintana-Domeque, ‘Paving streets for the poor: Experimental analysis of infrastructure effects’, Review of Economics and Statistics, vol. 98, no. 2, 2016, pp. 254–67.

  29Kenneth Lee, Edward Miguel & Catherine Wolfram, ‘Experimental evidence on the demand for and costs of rural electrification’, NBER Working Paper 22292, Cambridge, MA: National Bureau of Economic Research, 2016.

  30Joppe de Ree, Karthik Muralidharan, Menno Pradhan, Halsey Rogers, ‘Double for nothing? Experimental evidence on an unconditional teacher salary increase in Indonesia’, Quarterly Journal of Economics, forthcoming.

  31Karthik Muralidharan, Paul Niehaus & Sandip Sukhtankar, ‘Building state capacity: Evidence from biometric smartcards in India’, American Economic Review, vol, 106, no. 10, 2016, pp. 2895–929.

  32Even for those who survive, malaria can be a debilitating disease. A person who contracts malaria as a child earns one-third less as an adult: Abhijit Banerjee & Esther Duflo, Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty, New York: Public Affairs, 2011, p. 44.

  33World Health Organization, ‘Fact sheet: World malaria report 2015’, 9 December 2015. The statistic refers to children under five.

  34William Easterly, The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good, New York: Penguin, 2006, p. 12.

  35Jeffrey Sachs, ‘Good news on malaria control’, Scientific American, 1 August 2009.

  36Jessica Cohen & Pascaline Dupas, ‘Free distribution or cost-sharing? Evidence from a randomized malaria prevention experiment’, Quarterly Journal of Economics, vol. 125, no. 1, 2010, pp. 1-45; Pascaline Dupas, ‘What matters (and what does not) in households’ decision to invest in malaria prevention?’ American Economic Review: Papers & Proceedings, vol. 99, no. 2, 2009, pp. 224–30; Pascaline Dupas, ‘Short-run subsidies and long-run adoption of new health products: Evidence from a field experiment.” Econometrica, vol. 82, no. 1, January 2014, pp. 197–228.

  37Jeffrey Sachs, ‘The case for aid’, Foreign Policy, 21 January 2014.

  38Michael Kremer & Edward Miguel, ‘The illusion of sustainability’, Quarterly Journal of Economics, vol. 122, no. 3, 2007, pp. 1007–65; Michael Kremer, E. Miguel & S. Mullainathan, ‘Source dispensers and home delivery of chlorine in Kenya’, Innovations for Poverty Action, Working Paper, 2014. For a useful literature summary, see Abdul Latif Jameel Poverty Action Lab, ‘Pricing preventive Health Products’, undated, available at www.povertyactionlab.org.

  39Abhijit Vinayak Banerjee, Esther Duflo, Rachel Glennerster & Dhruva Kothari, ‘Improving immunisation coverage in rural India: Clustered randomised controlled evaluation of immunisation campaigns with and without incentives’, British Medical Journal, vol. 340, 2010, c2220.

  40Blake Mycoskie, Start Something That Matters, New York: Spiegel and Grau, 2012, p. 5.

  41‘Free two shoes’, The Economist, 5 November 2016.

  42The one-for-one companies that sell these products are Warby Parker, One World Play Project, Sir Richard’s, Smile Squared, One Million Lights and FIGS.

  43Bruce Wydick, Elizabeth Katz, Flor Calvo, et al., ‘Shoeing the children: The impact of the TOMS shoe donation program in rural El Salvador’, World Bank Economic Review, 2017.

  44Bruce Wydick, ‘The impact of TOMS shoes’, Across Two Worlds blog, 16 March 2015.

  45See the ‘Death on the Roads’ page, at www.who.int.

  46James Habyarimana & William Jack, ‘Heckle and chide: Results of a randomized road safety intervention in Kenya’, Journal of Public Economics, vol. 95, no. 11, 2011, pp. 1438–46.

  47Paul Gertler, Manisha Shah, Maria Laura Alzua, Lisa Cameron, et al., ‘How does health promotion work? Evidence from the dirty business of eliminating open defecation’, NBER Working Paper 20997, Cambridge, MA: National Bureau of Economic Research, 2015.

  48On the role of Santiago Levy in creating the Progresa experiment, see Banerjee & Duflo, Poor Economics, p. 78.

  49Susan W. Parker & Graciela M. Teruel, ‘Randomization and social program evaluation: The case of Progresa’, The Annals of the American Academy of Political and Social Science, vol. 599, no. 1, 2005, pp. 199–219.

  50Susan Parker & Petra Todd, ‘Conditional cash transfers: The case of Progresa/Oportunidades’, Journal of Economic Literature, vol. 55, no. 3, 2017, pp. 866–915.

  51Dana Burde & Leigh L. Linden, ‘Bringing education to Afghan girls: A randomized controlled trial of village-based schools’, American Economic Journal: Applied Economics, vol. 5, no. 3, 2013, pp. 27–40.

  52The student gains were approximately half a standard deviation (larger for girls, and smaller for boys). I assume that the typical student improves at a rate of half a standard deviation each school year. See, for example, Andrew Leigh, ‘Estimating teacher effectiveness from two-year changes in students’ test score’, Economics of Education Review, vol. 29, no. 3, 2010, pp. 480–8.

  53These figures are for 2009, and are drawn from WHO/UNAIDS, ‘Fast facts on HIV’, 2010, available at www.who.int.

  54Quoted in Esther Duflo, ‘AIDS prevention: Abstinence vs. risk reduction’, VoxEU blog, 20 April 2009.

  55Samuel Ponce de Leon, Maria Eugenia Jimenez-Corona, Ana Maria Velasco & Antonio Lazcano, ‘The Pope, condoms, and the evolution of HIV’, The Lancet Infectious Diseases, vol. 9, no. 8, 2009, pp. 461–2.

  56Duflo, Esther, Pascaline Dupas & Michael Kremer, ‘Education, HIV, and early fertility: Experimental evidence from Kenya’, American Economic Review, vol. 105, no. 9, 2015, pp. 2757–97.

  57Pascaline Dupas, ‘Do teenagers respond to HIV risk information? Evidence from a field experiment in Kenya’, American Economic Journal: Applied Economics, vol. 3, no. 1, 2011, pp. 1–34.

  58These were reported to school students to be the infection rates from a nearby city (Kisumu), as distinct from the national rates.

  59John Gapper, ‘Lunch with the FT: Esther Duflo’, Financial Times, 17 March 2012.

  60Gapper, ‘Lunch with the FT’.

  61Asimina Caminis, ‘Putting economic policy to the test’, Finance and Development, September 2003, pp. 4–7.

  62Ian Parker, ‘The poverty lab’, New Yorker, 17 May 2010.

  63Abdul Latif Jameel Poverty Action Lab, ‘Increasing test score performance’, undated, available at www.povertyactionlab.org.

  64Innovations for Poverty Action, ‘Financial inclusion program brief’, 15 June 2016, available at www.poverty-action.org.

  65One analysis found that if you took any two randomised trials in development, their confidence intervals would overlap approximately 83 per cent of the time: Eva Vivalt, ‘Heterogeneous treatment effects in impact evaluation,’ American Economic Review, vol. 105, no. 5, 2015, pp. 467–70.

  66Paul Glewwe & Karthik Muralidharan, ‘Improving education outcomes in developing countries – evidence, knowledge gaps, and policy im
plications’ in Eric Hanushek, Stephen Machin & Ludger Woessman (eds), Handbook of the Economics of Education, Vol. 5, Amsterdam: North Holland, 2016, pp. 653–744.

  67Shwetlena Sabarwal, David K. Evans & Anastasia Marshak, ‘The permanent input hypothesis: The case of textbooks and (no) student learning in Sierra Leone’, World Bank Policy Research Working Paper, vol. 7021, 2014.

  68Jishnu Das, Stefan Dercon, James Habyarimana, et al., ‘School inputs, household substitution, and test scores’ American Economic Journal: Applied Economics, vol. 5, no. 2, 2013, pp. 29–57.

  69Isaac Mbiti & Karthik Muralidharan, ‘Inputs, incentives, and complementarities in primary education: Experimental evidence from Tanzania’, unpublished working paper, 2015.

  70Paul Glewwe, Michael Kremer & Sylvie Moulin. ‘Many children left behind? Textbooks and test scores in Kenya’, American Economic Journal: Applied Economics, vol. 1, no. 1, 2009, pp. 112–35.

  71Angus S. Deaton, ‘Instruments, randomization, and learning about development’, Journal of Economic Literature, vol. 48, no. 2, 2010, pp. 424–55.

  72Gueron & Rolston, Fighting for Reliable Evidence, p. 427.

  73Quoted in Adam Gopnik, ‘The double man: Why Auden is an indispensable poet of our time’, New Yorker, 23 September 2002. Gopnik observes: ‘Auden shared Popper’s sense that open societies were built on skeptical faith rather than on fatuous confidence’.

  8 FARMS, FIRMS AND FACEBOOK

  1This description relates to the experiment as it commenced in 1843. For details, see ‘Broadbalk Winter Wheat Experiment’ at e-RA: The Electronic Rothamsted Archive, www.era.rothamsted.ac.uk.

  2Quoted in Jonathan Silvertown, Paul Poulton, Edward Johnston, et al., ‘The Park Grass Experiment 1856–2006: Its contribution to ecology’, Journal of Ecology, vol. 94, no. 4, 2006, pp. 801–14.

  3Late-nineteenth century estimate from Vaclav Smil, Enriching the Earth: Fritz Haber, Carl Bosch, and the Transformation of World Food, Cambridge, MA: MIT Press, 2001, p. 245. Current estimate from Food and Agriculture Organisation of the United Nations, Current World Fertilizer Trends and Outlook to 2015, Rome: FAO, 2011.

  4C.W. Wrigley, ‘Farrer, William James (1845–1906)’, Australian Dictionary of Biography, Volume 8, National Centre of Biography, Australian National University, 1981, available at http://adb.anu.edu.au.

  5Author’s interview with Greg Rebetzke. These models choose from among many possible random allocations of treatments across a grid, with the aim of minimising the number of treatments that recur on the same horizontal or vertical axis.

  6Quoted in Leslie Brokaw, ‘In experiments we trust: From Intuit to Harrah’s casinos’, MIT Sloan Management Review, 3 March 2011.

  7‘From Harvard economist to casino CEO’, Planet Money, 15 November 2011.

  8Others disagree. See, for example, Ira Glass, ‘Blackjack’, This American Life, Episode 466, 8 June 2016.

  9Quoted in Leslie Brokaw, ‘In experiments we trust: From Intuit to Harrah’s Casinos’, MIT Sloan Management Review, 3 March 2011.

  10Quoted in Jeffrey Pfeffer and Victoria Chang, ‘Gary Loveman and Harrah’s Entertainment’, Stanford Business School Case No. OB45, Stanford, CA, 2003.

  11For an excellent overview of some of the studies in this field, see Omar Al-Ubaydli & John List, ‘Field experiments in markets’, in Abhijit Banerjee & Esther Duflo (eds), Handbook of Field Experiments, Amsterdam: Elsevier, vol. 1, 2017, pp. 271–307.

  12Jim Manzi, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society, New York: Basic Books, 2012, p. 144.

  13Eric T. Anderson & Duncan Simester, ‘A step-by-step guide to smart business experiments’, Harvard Business Review, March 2011.

  14Quoted in Bharat N. Anand, Michael G. Rukstad & Christopher Paige, ‘Capital One financial corporation’, Harvard Business School Case 700-124, April 2000.

  15Stefan Thomke & Jim Manzi, ‘The discipline of business experimentation’, Harvard Business Review, December 2014.

  16Christian Rudder, ‘We experiment on human beings!’, OkTrends blog, 28 July 2014.

  17Rudder, ‘We experiment on human beings!’

  18Christian Rudder, Dataclysm: Love, Sex, Race, and Identity–What Our Online Lives Tell Us about Our Offline Selves, New York: Broadway Books, 2015, p. 17.

  19Rudder, ‘We experiment on human beings!’

  20One possible explanation of this is that the match quality algorithm was not very good.

  21Uri Gneezy & John List, The Why Axis: Hidden Motives and the Undiscovered Economics of Everyday Life, New York: Public Affairs, 2013, pp. 237–8.

  22Interview by Russ Roberts with Quora CEO Adam D’Angelo, EconTalk, 8 August 2016, available at www.econtalk.org. Before starting an experiment, employees must either specify the hypothesis they are trying to test, or declare that they are simply running a ‘learning experiment’., If a new feature performs well in a hypothesis experiment, it will typically be deployed to all Quora users. But if something comes up as statistically significant in a learning experiment, the employee needs to run a second experiment to prove that it wasn’t just a fluke the first time.

  23Brian Christian, ‘The A/B test: Inside the technology that’s changing the rules of business’, Wired, 25 April 2012.

  24‘Little things that mean a lot’, Economist, 19 July 2014.

  25Jim Manzi, head of Applied Predictive Technologies, boasts that his firm is now running randomised trials for ‘30 to 40 per cent of the largest retailers, hotel chains, restaurant chains and retail banks in America’: Manzi, Uncontrolled, p. 147.

  26Quoted in ‘Test of dynamic pricing angers Amazon customers’, Washington Post, 7 October 2000.

  27Quoted in Troy Wolverton, ‘Now showing: random DVD prices on Amazon’, C|Net, 5 September 2000.

  28‘Amazon.com issues statement regarding random price testing’, Amazon.com, 27 September 2000.

  29For an excellent discussion of randomised experiments in the field of marketing, see Duncan Simester, ‘Field experiments in Marketing’, in Banerjee & Duflo (eds), Handbook of Field Experiments, pp. 465–97.

  30Eric T. Anderson & Duncan I. Simester, ‘Effects of $9 price endings on retail sales: Evidence from field experiments,” Quantitative Marketing and Economics, vol. 1, no. 1, 2003, pp. 93–110.

  31Tanjim Hossain & John Morgan, ‘… plus shipping and handling: Revenue (non) equivalence in field experiments on ebay’, Advances in Economic Analysis and Policy, vol. 5, no. 2, 2006.

  32The share of gold card customers who accepted the upgrade was 21 per cent for the platinum card offer, but 14 per cent for a gold card that offered the same benefits as the platinum card: Leonardo Bursztyn, Bruno Ferman, Stefano Fiorin, et al., ‘Status goods: Experimental evidence from platinum credit cards’, NBER Working Paper No. 23414, Cambridge, MA: NBER, 2017.

  33Haipeng Chen, Howard Marmorstein, Michael Tsiros & Akshay R. Rao, ‘When more is less: The impact of base value neglect on consumer preferences for bonus packs over price discounts’, Journal of Marketing, vol. 76, no. 4, 2012, pp. 64–77.

  34Brian Wansink, Robert J. Kent & Stephen J. Hoch, 1998, ‘An anchoring and adjustment model of purchase quantity decisions’, Journal of Marketing Research, vol. 35, no. 1, pp. 71–81

  35Kusum L. Ailawadi, Bari A. Harlam, Jacques César & David Trounce, ‘Quantifying and improving promotion effectiveness at CVS’, Marketing Science, vol. 26, no. 4, 2007, pp. 566–75.

  36Ju-Young Kim, Martin Natter & Martni Spann, ‘Pay what you want: A new participative pricing mechanism’, Journal of Marketing, vol. 73, 2009, pp. 44–58.

  37Greer K. Gosnell, John A. List & Robert Metcalfe, ‘A new approach to an age-old problem: Solving externalities by incenting workers directly’, NBER Working Paper No. 22316, Cambridge, MA: NBER, 2016.

  38Bruce S. Shearer, ‘Piece rates, fixed wages and incentives: Evidence from a field experiment’, Review of Economic Studies, vol. 71, no. 2, 2004, pp. 513–34.

  39Lan Shi, ‘Incentive e
ffect of piece-rate contracts: Evidence from two small field experiments’, B.E. Journal of Economic Analysis & Policy, vol. 10, no. 1 (Topics), Article 61, 2010.

  40For a summary of the strawberry studies, see Oriana Bandiera, Iwan Barankay & Imran Rasul, ‘Field experiments with firms’, Journal of Economic Perspectives, vol. 25, no. 3, 2011, pp. 63–82. The authors do not disclose which kind of soft fruit their subjects pick – this information is contained in Tim Harford, ‘The fruits of their labors’, Slate, 23 August 2008.

  41An even more nefarious way of delivering bonuses is to ‘provisionally’ pay workers a bonus, but then to say that it will be withdrawn if performance targets are not met. This exploitation of employee ‘loss aversion’ did indeed raise productivity in a randomised experiment in a Chinese factory: Tanjim Hossain & John A. List. ‘The behavioralist visits the factory: Increasing productivity using simple framing manipulations’, Management Science, vol. 58, no. 12, 2012, pp. 2151–67. Similarly, ridesharing company Lyft found that new drivers were more likely to shift from a quiet time of the week to a busy time of the week if the difference was expressed as a loss than a gain (the company ultimately chose not to implement the results of the study): Noam Scheiber, ‘How Uber uses psychological tricks to push its drivers’ buttons’, New York Times, 2 April 2017.

  42Alexandre Mas & Enrico Moretti, ‘Peers at work’, American Economic Review, vol. 99, no. 1, 2009, pp. 112–45; Oriana Bandiera, Iwan Barankay and Imran Rasul, ‘Social incentives in the workplace’, Review of Economic Studies, vol. 77, no. 2, 2010, pp. 417–58; Lamar Pierce and Jason Snyder, ‘Ethical spillovers in firms: Evidence from vehicle emissions testing,’ Management Science, vol. 54, no. 11, 2008, pp. 1891–1903. Note that only the Bandiera et al. study uses true random assignment; in the other two studies the authors argue that the assignment of employees to teams is effectively random – in other words, unrelated to their co-workers’ productivity.

  43Nava Ashraf, Oriana Bandiera & B. Kelsey Jack, ‘No margin, no mission? A field experiment on incentives for public service delivery’, Journal of Public Economics, vol. 120, 2014, pp. 1–17.