by Andrew Leigh
44Iwan Barankay, ‘Rankings and social tournaments: Evidence from a crowd-sourcing experiment’, Working Paper, Wharton School of Business, University of Pennsylvania, 2011.
45Steven D. Levitt & John A. List, ‘Was there really a Hawthorne effect at the Hawthorne plant? An analysis of the original illumination experiments’, American Economic Journal: Applied Economics, vol. 3, no.1, 2011, pp. 224–38.
46Matthew Stewart, The Management Myth: Why the Experts Keep Getting It Wrong, New York: Norton, 2009.
47Jill Lepore, ‘Not so fast’, New Yorker, 12 October 2009.
48Nicholas Bloom, Benn Eifert, Aprajit Mahajan, et al., ‘Does management matter? Evidence from India’, Quarterly Journal of Economics, vol. 128, no. 1, 2013, pp. 1–51.
49The fact that Accenture were the consulting firm involved in the study is revealed in Tim Harford, ‘A case for consultants?’, Financial Times, 13 November 2010.
50Similarly, a study that randomly offered export opportunities to Egyptian rug manufacturers found that exposure to export markets boosted their subsequent profits by up to one-quarter. However, such a large effect might not be realised in high-income countries. See David Atkin, Amit K. Khandelwal & Adam Osman. ‘Exporting and firm performance: Evidence from a randomized experiment’, Quarterly Journal of Economics, vol. 132, no. 2, 2017, pp. 551–615.
51Chicago University’s John List is now pushing the power of statistics still further, running an experiment with United Airlines’ loyalty program across just four US cities – two in the treatment group and two in the control group. It will be interesting to see whether, after accounting for cluster effects, the results are statistically significant.
52Leonard M. Lodish, Magid Abraham, Stuart Kalmenson, et al., ‘How TV advertising works: A meta-analysis of 389 real world split cable TV advertising experiments’ Journal of Marketing Research, vol. 32, no. 2, 1995, pp. 125–39. The study is an update of the pathbreaking work of Margaret Blair, ‘An empirical investigation of advertising wearin and wearout” Journal of Advertising Research, vol. 27, no. 6, 1987, pp. 45–50.
53Randall A. Lewis & Justin M. Rao, ‘The unfavorable economics of measuring the returns to advertising,’ The Quarterly Journal of Economics, vol. 130, no. 4, 2015, pp. 1941–73. More precisely, their Super Bowl impossibility theorem finds that ‘it is nearly impossible for a firm to be large enough to afford the ad, but small enough to reliably detect meaningful differences in ROI’.
54Randall A. Lewis, Justin M. Rao & David H. Reiley, ‘Here, there, and everywhere: Correlated online behaviors can lead to overestimates of the effects of advertising’ in Proceedings of the 20th International Conference on World Wide Web, ACM, 2011, pp. 157–66.
55Brian Christian, ‘The A/B test: Inside the technology that’s changing the rules of business’, Wired, 25 April 2012.
56Ben Gomes, ‘Search experiments, large and small’, Google Official Blog, 26 August 2008
57Dan Cobley, quoted in Matthew Syed, Black Box Thinking: Why Most People Never Learn from Their Mistakes – But Some Do, Portfolio, New York, 2015, pp. 184–5.
58‘In 2010 alone, the company investigated more than 13,000 proposed changes, of which around 8200 were tested in side-by-side comparisons that were evaluated by raters. Of those, 2800 were further evaluated by a tiny fraction of the live traffic in a “sandbox” area of the website. Analysts prepared an independent report of those results, which were then evaluated by a committee. That process led to 516 improvements that were made to the search algorithm from the initial 13,000 proposals.’: Stefan Thomke, ‘Unlocking innovation through business experimentation’, European Business Review, 10 March 2013.
59Quoted in Thomke, ‘Unlocking innovation’. See also Manzi, Uncontrolled, pp. 128, 142.
60A classic example of where sample size doesn’t help was Literary Digest’s attempt to predict the 1936 presidential election by surveying nearly 2 million of its readers. Failing to recognise that its readers were more affluent than the electorate at large, the magazine forecast that Republican Alf Landon would beat incumbent Democratic President Franklin D. Roosevelt. Landon ended up with just 8 of the 531 electoral college votes.
61Huizhi Xie & Juliette Aurisset, ‘Improving the sensitivity of online controlled experiments: Case studies at Netflix.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 645–54. ACM, 2016.
62Carlos A. Gomez-Uribe & Neil Hunt, ‘The Netflix recommender system: Algorithms, business value, and innovation’, ACM Transactions on Management Information Systems (TMIS), vol. 6, no. 4, 2016, p. 13.
63Gomez-Uribe & Hunt, ‘The Netflix recommender system’, p. 13.
64Adam D.I. Kramer, Jamie E. Guillory & Jeffrey T. Hancock, ‘Experimental evidence of massive-scale emotional contagion through social networks’, Proceedings of the National Academy of Sciences, vol. 3, no. 24, 2014, pp. 8788–90.
65Because 22.4 per cent of Facebook posts contained negative words, and 46.8 per cent contained negative words, the study also had two control groups: one of which randomly omitted 2.24 per cent of all posts, and another that randomly omitted 4.68 per cent of all posts.
66Oddly, some commentators seem unaware of the finding, continuing to make claims like ‘Facebook makes us feel inadequate, so we try to compete, putting a positive spin and a pretty filter on an ordinary moment – prompting someone else to do the same . . . when you sign up to Facebook you put yourself under pressure to appear popular, fun and loved, regardless of your reality’: Daisy Buchanan, ‘Facebook bragging’s route to divorce’, Australian Financial Review, 27 August 2016
67Kate Bullen & John Oates, ‘Facebook’s ‘experiment’ was socially irresponsible’, Guardian, 2 July 2014.
68Quoted in David Goldman, ‘Facebook still won’t say “sorry” for mind games experiment’, CNNMoney, 2 July 2014.
9 TESTING THEORIES IN POLITICS AND PHILANTHROPY
1Julian Jamison & Dean Karlan, ‘Candy elasticity: Halloween experiments on public political statements’, Economic Inquiry, vol. 54, no. 1, 2016, pp. 543–7.
2This experiment is outlined in detail in Dan Siroker, ‘How Obama raised $60 million by running a simple experiment’, Optimizely blog, 29 November 2010.
3Quoted in Brian Christian, ‘The A/B test: Inside the technology that’s changing the rules of business’, Wired, 25 April 2012.
4Alan S. Gerber & Donald P. Green, ‘Field experiments on voter mobilization: An overview of a burgeoning literature’ in Banerjee & Duflo (eds), Handbook of Field Experiments, pp. 395–438.
5Harold F. Gosnell, Getting-out-the-vote: An Experiment in the Stimulation of Voting, Chicago: University of Chicago Press, 1927. As Gerber and Green point out, Gosnell’s experiments used matched pairs of streets, but it is not clear from his write-up how he chose which street to be in the treatment and control groups. See Gerber and Green, ‘Field experiments on voter mobilization’.
6Donald P. Green & Alan S. Gerber, Get Out the Vote: How to Increase Voter Turnout, 2nd edition, Washington DC: Brookings Institution Press, 2008, p. 14.
7For an example of a widely cited academic work which takes essentially this approach, see Steven Rosenstone & John Hansen, Mobilization, Participation, and Democracy in America, New York: MacMillan, 1993.
8The experiment was carried out at the beginning of 2006, when Governor Perry faced the prospect of a challenge in the Republican primary from Carole Keeton Strayhorn (she ultimately decided to run as an independent). For more details, see Alan S. Gerber, James G. Gimpel, Donald Green & Daron Shaw, ‘How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized field experiment’, American Political Science Review, vol. 105, no. 1, 2011, pp. 135–50.
9One reason that it is easier to detect the impact of political advertisements than commercial advertisements is that voters constitute a larger share of the population than do customers of most products (though falling voting rates and rising market c
oncentration mean that this may not always be true).
10Gerber & Green, ‘Field experiments on voter mobilization’, Table 4.
11Alan S. Gerber, Donald P. Green & Christopher W. Larimer, ‘Social pressure and voter turnout: Evidence from a large-scale field experiment’, American Political Science Review, vol. 102, no. 1, 2008, pp. 33–48.
12Gregg R. Murray & Richard E. Matland, ‘Mobilization effects using mail social pressure, descriptive norms, and timing’, Political Research Quarterly, vol. 67, no. 2, 2014, pp. 304–19.
13The honour roll experiment produced a 2 percentage point increase in turnout: Costas Panagopoulos, ‘Positive social pressure and prosocial motivation: Evidence from a large-scale field experiment on voter mobilization’, Political Psychology, vol. 34, no. 2, 2013, pp. 265–75. The gratitude experiments (run in Georgia, New Jersey and New York) produced a 2.4 to 2.5 percentage point increase in turnout: Costas Panagopoulos, ‘Thank you for voting: Gratitude expression and voter mobilization’, Journal of Politics, vol. 73, no. 3, 2011, pp. 707–17.
14Gerber & Green, ‘Field experiments on voter mobilization’, Table 4.
15Green & Gerber, Get Out the Vote, p. 69.
16Green & Gerber, Get Out the Vote, p. 92.
17Gerber & Green, ‘Field experiments on voter mobilization’.
18Green & Gerber, Get Out the Vote, p. 92.
19Lisa Garcia Bedolla & Melissa R. Michelson, Mobilizing inclusion: Transforming the electorate through get-out-the-vote campaigns, New Haven, CT: Yale University Press, 2012.
20Gerber & Green, ‘Field experiments on voter mobilization’; Vincent Pons, ‘Does door-to-door canvassing affect vote shares? Evidence from a countrywide field experiment in France’, Working Paper, Harvard Business School, 2014; Guillaume Liégey, Arthur Muller & Vincent Pons, Porte à porte: Reconquérir la démocratie sur le terrain, Calmann-Lévy, 2013; Peter John & Tessa Brannan, ‘How different are telephoning and canvassing? Results from a “get out the vote” field experiment in the British 2005 general election’, British Journal of Political Science, vol. 38, no. 3, 2008, pp. 565–74.
21Green & Gerber, Get Out the Vote, p. 37.
22David Broockman & Joshua Kalla, ‘Experiments show this is the best way to win campaigns. But is anyone actually doing it?’, Vox, 13 November 2014.
23David W. Nickerson, ‘Does email boost turnout?’ Quarterly Journal of Political Science, vol. 2, no. 4, 2008, pp. 369–79.
24Alissa F. Stollwerk, ‘Does e-mail affect voter turnout? An experimental study of the New York City 2005 election’, unpublished manuscript, Institution for Social and Policy Studies, Yale University, 2006; Alissa F. Stollwerk, ‘Does partisan e-mail affect voter turnout? An examination of two field experiments in New York City’, unpublished manuscript, Department of Political Science, Columbia University, 2016.
25The study that looked at email from friends and acquaintances is Tiffany C. Davenport, ‘Unsubscribe: The effects of peer-to-peer email on voter turnout – results from a field experiment in the June 6, 2006, California primary election’, unpublished manuscript, Yale University, 2012, quoted in Donald P. Green, Mary C. McGrath & Peter M. Aronow, ‘Field experiments and the study of voter turnout’, Journal of Elections, Public Opinion & Parties, vol. 23, no. 1, 2013, pp. 27–48. The study that looked at email from the voting registrar is Neil Malhotra, Melissa R. Michelson & Ali Adam Valenzuela, ‘Emails from official sources can increase turnout’, Quarterly Journal of Political Science, vol. 7, no. 3, 2012, pp. 321–32.
26Allison Dale & Aaron Strauss, ‘Don’t forget to vote: text message reminders as a mobilization tool’, American Journal of Political Science, vol. 53, 2009, pp. 787–804; Neil Malhotra, Melissa R. Michelson, Todd Rogers & Ali Adam Valenzuela, ‘Text messages as mobilization tools: the conditional effect of habitual voting and election salience’, American Politics Research, vol. 39, 2011, pp. 664–81.
27Quoted in David E. Broockman & Donald P. Green, ‘Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments’, Political Behavior, vol. 36, no. 2, 2014, pp. 263–89. More recently, Brad Parscale, Donald Trump’s digital director, claimed that ‘Facebook and Twitter were the reason we won this thing’: Issie Lapowsky, ‘Here’s how Facebook actually won Trump the presidency’, Wired, 15 November 2016. In a similar vein, see Sue Halpern, ‘How he used Facebook to win’, New York Review of Books, 8 June 2017.
28Quoted in Lapowsky, ‘Here’s how Facebook actually won Trump the presidency’.
29Broockman and Green, ‘Do online advertisements increase political candidates’ name recognition or favorability?’ pp. 263–89.
30Kevin Collins, Laura Keane & Josh Kalla, ‘Youth voter mobilization through online advertising: Evidence from two GOTV field experiments’, paper presented at the Annual Meeting of the American Political Science Association, Washington, DC, 2014.
31Robert M. Bond, Christopher J. Fariss, Jason J. Jones, et al., ‘A 61-million-person experiment in social influence and political mobilization’, Nature, vol. 489, no. 7415, 2012, pp. 295–8, cited in ‘A new kind of weather’, Economist, 26 March 2016.
32Craig E. Landry, Andreas Lange, John A. List, et al., ‘Toward an understanding of the economics of charity: Evidence from a field experiment’, Quarterly Journal of Economics, vol. 121, no. 2, 2006, pp. 747–82.
33In a South African study, adding a photo of an attractive woman onto a loan flyer had the same impact on take-up among male respondents as a 40 per cent reduction in the interest rate: Karlan & Appel, More Than Good Intentions, p. 47.
34The experiment raised money for two charities – a Chicago children’s hospital and a North Carolina research institution: Stefano DellaVigna, John A. List & Ulrike Malmendier, ‘Testing for altruism and social pressure in charitable giving’, Quarterly Journal of Economics, vol. 127, no. 1, 2012, pp. 1–56. The results were confirmed in a replication experiment: Cynthia R. Jasper & Anya Savikhin Samek, ‘Increasing charitable giving in the developed world’, Oxford Review of Economic Policy, vol. 30, no. 4, 2014, pp. 680–96.
35The classic economics study on warm glow giving is James Andreoni, ‘Impure altruism and donations to public goods: A theory of warm-glow giving’, Economic Journal, vol. 100, no. 401, 1990, pp. 464–77.
36James Andreoni, Justin M. Rao & Hannah Trachtman, ‘Avoiding the ask: A field experiment on altruism, empathy, and charitable giving’, Journal of Political Economy, vol. 125, no. 3, 2017, pp. 625–53. Results are taken from Table 2. A few shoppers exited the supermarket through a third door, but the authors carefully explain why theirs can be thought of as a two-exit study.
37John A. List & David Lucking-Reiley, ‘The effects of seed money and refunds on charitable giving: Experimental evidence from a university capital campaign’, Journal of Political Economy, vol. 110, no. 1, 2002, pp. 215–33; Steffen Huck, Imran Rasul & Andrew Shephard, ‘Comparing charitable fundraising schemes: Evidence from a natural field experiment and a structural model’, American Economic Journal: Economic Policy, vol. 7, no. 2, 2015, pp. 326–69.
38Dean Karlan & John A. List, ‘Does price matter in charitable giving? Evidence from a large-scale natural field experiment’, American Economic Review, vol. 97, no. 5, 2007, pp. 1774–93.
39Kent E. Dove, Conducting a Successful Capital Campaign, 2nd edition, San Francisco: Jossey Bass, 2000, p. 15, quoted in Dean Karlan & John A. List, ‘Does price matter in charitable giving? Evidence from a large-scale natural field experiment’, American Economic Review, vol. 97, no. 5, 2007, pp. 1774–93.
40Gneezy & List, The Why Axis, pp. 204–5.
41Armin Falk, ‘Gift exchange in the field’, Econometrica, vol. 75, no. 5, 2007, pp. 1501–11.
42Tova Levin, Steven Levitt & John List, ‘A glimpse into the world of high capacity givers: Experimental evidence from a university capital campaign’, NBER Working Paper 22099, Cambridge, MA: NBER, 2016.
43Richard Martin and John Randal, ‘How i
s donation behaviour affected by the donations of others?’ Journal of Economic Behavior & Organization, vol. 67, no. 1, 2008, pp. 228–38.
44James T. Edwards & John A. List, ‘Toward an understanding of why suggestions work in charitable fundraising: Theory and evidence from a natural field experiment’, Journal of Public Economics, vol. 114, 2014, pp. 1–13; Jen Shang & Rachel Croson, ‘A field experiment in charitable contribution: The impact of social information on the voluntary provision of public goods’, Economic Journal, vol. 119, no. 540, 2009, pp. 1422–39; David Reiley & Anya Savikhin Samek, ‘How do suggested donations affect charitable gifts? Evidence from a field experiment in public broadcasting’, CESR-Schaeffer Working Paper 2015-031, 2015.
45John List interview, published online on 11 March 2013, available at https://youtu.be/LwF7MEuspU0?t=63.
46Joanne M. Miller & Jon A. Krosnick. ‘Threat as a motivator of political activism: A field experiment’, Political Psychology, vol. 25, no. 4, 2004, pp. 507–23.
47‘Politics by numbers’, The Economist, 26 March 2016
48Leonard Wantchekon, ‘Clientelism and voting behavior: Evidence from a field experiment in Benin’, World Politics, vol 55, no 3, 2003, pp. 399–422.
49Kelly Bidwell, Katherine Casey & Rachel Glennerster, ‘Debates: Voting and expenditure responses to political communication’, Working Paper, Stanford University, 2016.
50Quoted in Tina Rosenberg, ‘Smart African politics: Candidates debating under a tree’, New York Times, 10 November 2015.
51Daniel M. Butler & David E. Broockman, ‘Do politicians racially discriminate against constituents? A field experiment on state legislators’, American Journal of Political Science, vol. 55, 2011, pp. 463–77.
52This is the result among those applications that did not include a partisan signal. The results are more mixed if all applications are included in the analysis.
53Gwyneth McClendon, ‘Race responsiveness, and electoral strategy: A field experiment with South African politicians’, Manuscript, Harvard University, 2013.
