Statistical Inference as Severe Testing

by Deborah G Mayo

  Good , I. J. (1971 b). ‘ Commentary on D. J. Bartholomew’ in Godambe , V. and Sprott , D. (eds.), p. 431.

  Good , I. J. (1976 ). ‘ The Bayesian Influence, or How to Sweep Subjectivism Under the Carpet’ , in Harper , W. and Hooker , C. (eds.), pp. 25 – 174 .

  Good , I. J. (1983 ). Good Thinking: The Foundations of Probability and Its Applications . Minneapolis, MN : University of Minnesota Press .

  Goodman , S. (1992 ). ‘ A Comment on Replication, P-values and Evidence ’ , Statistics in Medicine 11 (7 ), 875– 9 .

  Goodman , S. (1993 ). ‘ P-values, Hypothesis Tests, and Likelihood: Implications for Epidemiology of a Neglected Historical Debate ’ , American Journal of Epidemiology 137 (5 ), 485– 96 .

  Goodman , S. (1999 ). ‘ Toward Evidence-Based Medical Statistics. 2: The Bayes Factor ’ , Annals of Internal Medicine , 130 (12 ), 1005– 13 .

  Goodman , S. and Greenland S. (2007 ). ‘ Assessing the Unreliability of the Medical Literature: A Response to “ Why Most Published Research Findings Are False”’ , Johns Hopkins University, Department of Biostatistics Working Papers. Working Paper 135, pp. 1 – 25 .

  Gopnik , A. (2009 ). August 11 Interview in The Edge , https://edge.org/conversation/amazing-babies .

  Gorroochurn , P. (2016 ). Classic Topics on the History of Modern Mathematical Statistics: From Laplace to More Recent Times , Hoboken, NJ : Wiley.

  Greenland , S. (2012 ). ‘ Nonsignificance Plus High Power Does Not Imply Support for the Null Over the Alternative ’ , Annals of Epidemiology 22 , 364– 8 .

  Greenland , S. and Poole , C. (2013 ). ‘ Living with P Values: Resurrecting a Bayesian Perspective on Frequentist Statistics ’ and ‘ Rejoinder: Living with Statistics in Observational Research ’ , Epidemiology 24 (1 ), 62– 8 ; 73– 8 .

  Greenland , S. , Senn , S. , Rothman , K. , et al. (2016 ). ‘ Statistical Tests, P values, Confidence Intervals, and Power: A Guide to Misinterpretations ’ , European Journal of Epidemiology 31 (4 ), 337– 50 .

  Gurney , J. , Mueller , B. , Davis S. , et al. (1996 ). ‘ Childhood Brain Tumor Occurrence in Relation to Residential Power Line Configurations, Electric Heating Sources, and Electric Appliance Use ’ , American Journal of Epidemiology 143 , 120– 8 .

  Hacking , I. (1965 ). Logic of Statistical Inference . Cambridge : Cambridge University Press .

  Hacking , I. (1972 ). ‘ Review: Likelihood ’ , British Journal for the Philosophy of Science 23 (2 ), 132– 7 .

  Hacking , I. (1980 ). ‘ The Theory of Probable Inference: Neyman, Peirce and Braithwaite ’ , in Mellor , D. (ed.), Science, Belief and Behavior: Essays in Honour of R. B. Braithwaite , Cambridge : Cambridge University Press , pp. 141– 60 .

  Haidt , J. and Iyer , R. (2016 ). ‘ How to Get Beyond Our Tribal Politics’ , Wall Street Journal (11/10/2016).

  Haig , B. (2016 ). ‘ Tests of Statistical Significance Made Sound ’ , Educational and Psychological Measurement 77 (3 ) 489 – 506 .

  Hand , D. (2014 ). The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day , 1st edn. New York : Scientific American/ Farrar, Straus and Giroux .

  Hannig , J. (2009 ). ‘ On Generalized Fiducial Inference ’ , Statistica Sinica 19 , 491 – 544 .

  Harlow , H. (1958 ). ‘ The Nature of Love ’ , American Psychologist 13 , pp. 673– 85 .

  Harper , W. and Hooker , C. (eds.) (1976 ). Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science , Volume II . Boston, MA : D. Reidel .

  Hawthorne , J. and Fitelson , B. (2004 ). ‘ Re-Solving Irrelevant Conjunction with Probabilistic Independence ’ , Philosophy of Science 71 , 505– 14 .

  Hempel , C. G. (1945 ). ‘ Studies in the Logic of Confirmation (I.) ’ , Mind 54 (213 ), 1 – 26 .

  Hendry , D. (2011 ). ‘ Empirical Economic Model Discovery and Theory Evaluation ’ , Rationality, Markets and Morals (RMM) 2 , 115– 45 .

  Hitchcock , C. and Sober , E. (2004 ). ‘ Prediction versus accommodation and the risk of overfitting ’ , The British Journal for the Philosophy of Science 55 (1 ), 1 – 34 .

  Hoenig , J. and Heisey , D. (2001 ). ‘ The Abuse of Power: The Pervasive Fallacy of Power Calculations in Data Analysis ’ , The American Statistician 55 (1 ), 1 – 6 .

  Howson , C. (1997 a). ‘ A Logic of Induction ’ , Philosophy of Science 64 (2 ), 268– 90 .

  Howson , C. (1997 b). ‘ Error Probabilities in Error ’ , Philosophy of Science 64 , Supplemental Issue PSA 1996: Symposia Papers. Edited by L. Darden (1996 ). S185 – S194 .

  Howson , C. (2017 ). ‘ Putting on the Garber Style? Better Not ’ , Philosophy of Science 84 (4 ), 659– 76 .

  Howson , C. and Urbach , P. (1993 ). Scientific Reasoning: The Bayesian Approach . La Salle, IL : Open Court .

  Hubbard , R. , and Bayarri , M. J. (2003 ). ‘ Confusion Over Measures of Evidence versus Errors ’ and ‘ Rejoinder ’ , The American Statistician 57 (3 ), 171– 8 ; 181– 2 .

  Huber , P. J. (2011 ). Data Analysis: What Can Be Learned from the Past 50 Years? , New York : Wiley .

  Huff , D. (1954 ). How to Lie with Statistics , 1st edn. New York : W. W. Norton & Company.

  Hume , D. (1739 ). A Treatise of Human Nature . BiblioBazaar .

  Hurlbert , S. and Lombardi , C. (2009 ). ‘ Final Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the NeoFisherian ’ , Annales Zoologici Fennici 46 , 311– 49 .

  Ioannidis , J. (2005 ). ‘ Why Most Published Research Findings are False ’ , PLoS Medicine 2 (8 ), 0696 – 0701 .

  Ioannidis , J. (2016 ). ‘ The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses ’ , Milbank Quarterly 94 (3 ), 485 – 514 .

  Irony , T. and Singpurwalla , N. (1997 ). ‘ Non-informative Priors Do not Exist: A Dialogue with José M. Bernardo ’ , Journal of Statistical Planning and Inference 65 (1 ), 159– 77 .

  Jefferys , W. and Berger , J. (1992 ). ‘ Ockham’ s Razor and Bayesian Analysis ’ , American Scientist 80 , 64 – 72 .

  Jeffreys , H. (1919 ). ‘ Contribution to Discussion on the Theory of Relativity’ , and ‘ On the Crucial Test of Einstein’ s Theory of Gravitation ’ , Monthly Notices of the Royal Astronomical Society 80 , 96 – 118 ; 138– 54 .

  Jeffreys , H. ([1939 ]/ 1961 ). Theory of Probability . Oxford : Oxford University Press .

  Jeffreys , H. (1955 ). ‘ The Present Position in Probability Theory ’ , The British Journal for the Philosophy of Science 5 , 275– 89 .

  Johnson , V. (2013 a). ‘ Revised Standards of Statistical Evidence ’ , Proceedings of the National Academy of Sciences (PNAS) 110 (48 ), 19313– 17 .

  Johnson , V. (2013 b). ‘ Uniformly Most Powerful Bayesian Tests ’ , The Annals of Statistics 41 (4 ), 1716– 41 .

  Kadane , J. (2006 ). ‘ Is “ Objective Bayesian Analysis” Objective, Bayesian, or Wise? (Comment on Articles by Berger and by Goldstein) ’ , Bayesian Analysis 1 (3 ), 433– 6 .

  Kadane , J. (2008 ). ‘ Comment on Article by Gelman ’ , Bayesian Analysis 3 (3 ), 455– 8 .

  Kadane , J. (2011 ). Principles of Uncertainty . Boca Raton, FL : Chapman and Hall/CRC .

  Kadane , J. (2016 ). ‘ Beyond Hypothesis Testing ’ , Entropy 18 (5 ), article 199, 1 – 5 .

  Kahneman , D. (2012 ). ‘ A proposal to deal with questions about priming effects’ email. Link to letter in Bartlett 2012a.

  Kahneman , D. (2014 ). ‘ A New Etiquette for Replication ’ , Social Psychology 45 (4 ), 299 – 311 .

  Kahneman , D. , Slovic , P. , and Tversky , A. (1982 ). Judgment under Uncertainty: Heuristics and Biases . New York : Cambridge University Press .

  Kaku , M. (2005 ). Einstein’ s Cosmos: How Albert Einstein’ s Vision Transformed Our Understanding of Space and Time (Great Discoveries) . New York : W. W. Norton & Company .

  Kalbfleisch , J. and Sprott , D. (1976 ). ‘ On Tests of Significance’ , in Harper , W. and Hooker , C. , pp. 259– 72 .

  Kass , R. (1998 ). �
� [R. A. Fisher in the 21st Century]: Comment .’ Statistical Science 13 (2 ), 115– 16 .

  Kass , R. (2011 ). ‘ Statistical Inference: The Big Picture (with discussion and rejoinder) ’ , Statistical Science 26 (1 ), 1 – 20 .

  Kass , R. and Wasserman , L. (1996 ). ‘ The Selection of Prior Distributions by Formal Rules ’ , Journal of the American Statistical Association 91 , 1343– 70 .

  Kaye , D. and Freedman , D. (2011 ). ‘ Reference Guide on Statistics’ , in Reference Manual on Scientific Evidence , 3rd edn. pp. 83 – 178 .

  Kempthorne , O. (1976 ). ‘ Statistics and the Philosophers’ , in Harper , W . and Hooker , C. (eds.), 273 – 314 .

  Kennefick , D. (2009 ). ‘ Testing Relativity from the 1919 Eclipse: A Question of Bias ’ , Physics Today 62 (3 ), 37 – 42 .

  Kerridge , D. (1963 ). ‘ Bounds for the Frequency of Misleading Bayes Inferences ’ , The Annals of Mathematical Statistics 34 (3 ), 1109– 10 .

  Keynes , J. (1921 ). A Treatise on Probability . London : MacMillan and Co .

  Kheifets , L. , Sussman , S. , and Preston-Martin , S. (1999 ). ‘ Childhood Brain Tumors and Residential Electromagnetic Fields (EMF) ’ , Reviews of Environmental Contamination and Toxicology 159 , 111– 29 .

  Kish , L. (1970 ). ‘ Some Statistical Problems in Research Design ’ , in Morrison , D. and Henkel , R. (eds.), pp. 127– 41 . (First published 1959, American Sociological Review 24 (3 ), 328 ).

  Kruschke , J. K. and Liddell , T. M. (2017 ). ‘ The Bayesian New Statistics: Hypothesis Testing, Estimation, Meta-analysis, and Power Analysis from a Bayesian Perspective’ , Psychonomic Bulletin & Review , 1 – 29 .

  Kuhn , T. (1970 ). ‘ Logic of Discovery or Psychology of Research?’ , in Lakatos , I. and Musgrave , A. (eds.), pp. 1 – 23 .

  Kyburg , H. (1992 ). ‘ The Scope of Bayesian Reasoning’ , PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association 1992 , 139– 52 .

  Kyburg , H. (2003 ). ‘ Probability as a Guide to Life’ , in Kyburg , H. E. and Thalos , M. (eds.), pp. 135– 52 .

  Kyburg , H. and Thalos , M. (eds.) (2003 ). Probability Is the Very Guide of Life: The Philosophical Uses of Chance . Chicago, IL : Open Court .

  Lad . F. (2006 ). ‘ Objective Bayesian Statistics … Do You Buy It? Should We Sell It? (Comment on Articles by Berger and by Goldstein) ’ , Bayesian Analysis 1 (3 ), 441– 4 .

  Lakatos , I. (1970 ). ‘ Falsification and the Methodology of Scientific Research Programmes’ , in Lakatos , I. and Musgrave , A. (eds.), pp. 91 – 138 .

  Lakatos , I. (1978 ). The Methodology of Scientific Research Programmes . Cambridge : Cambridge University Press .

  Lakatos , I. and Musgrave , A. (eds.) (1970 ). Criticism and the Growth of Knowledge . Cambridge : Cambridge University Press .

  Lakens , D. (2017 ). ‘ Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-analyses ’ , Social Psychological & Personality Science 8 (4 ), 355– 62 .

  Lakens , D. , et al. (2018 ). ‘ Justify your Alpha’ , Nature Human Behavior 2 , 168– 71 .

  Lambert , C. (2010 ). ‘ Stop Ignoring Experimental Design (or My Head Will Explode)’ , Blogpost on GoldenHelix.com (9/29/2010).

  Lambert , C. and Black , L. (2012 ). ‘ Learning From Our GWAS Mistakes: From Experimental Design to Scientific Method ’ , Biostatistics 13 (2 ), 195 – 203 .

  Lambert , P. , Sutton , A. , Burton , P. , Abrams , K. , and Jones , D. (2005 ). ‘ How Vague is Vague? A Simulation Study of the Impact of the Use of Vague Prior Distributions in MCMC Using WinBUGS ’ , Statistics in Medicine 24 , 2401– 28 .

  Laudan , L. (1978 ). Progress and Its Problems . Berkeley, CA : University of California Press .

  Laudan , L. (1983 ). ‘ The Demise of the Demarcation Problem ’ , in R. S. Cohen and L. Laudan (eds.), Physics, Philosophy and Psychoanalysis . Dordrecht, The Netherlands : D. Reidel , pp. 111– 27 .

  Laudan , L. (1996 ). Beyond Positivism and Relativism: Theory, Method, and Evidence . Boulder, CL : Westview Press .

  Laudan , L. (1997 ). ‘ How About Bust? Factoring Explanatory Power Back into Theory Evaluation ’ , Philosophy of Science 64 , 303– 16 .

  Leek , J. (2016 ). ‘ Statistical Vitriol’ , Blogpost on SimplyStatistics.com (09/29/16).

  Lehmann , E. (1981 ). ‘ An Interpretation of Completeness and Basu’ s Theorem ’ , Journal of the American Statistical Association 76 (374 ), 335– 40 .

  Lehmann , E. (1986 ). Testing Statistical Hypotheses , 2nd edn. New York : Wiley .

  Lehmann , E. (1988 ). ‘ Jerzy Neyman, 1894– 1981’ , Technical Report No. 155, May 1988.

  Lehmann , E. (1990 ). ‘ Model Specification: The Views of Fisher and Neyman, and Later Developments ’ , Statistical Science 5 (2 ), 160 – 168 .

  Lehmann , E. (1993 a). ‘ The Bertrand-Borel Debate and the Origins of the Neyman-Pearson Theory ’ , in Ghosh , J. , Mitra , S. , Parthasarathy , K. and Prak Ma Rao , L. (eds.), Statistics and Probability: A Raghu Raj Bahadur Festschrift , New Delhi : Wiley Eastern , 371– 80 . Reprinted in Lehmann 2012 , pp. 965– 74 .

  Lehmann , E. (1993 b). ‘ The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two? ’ , Journal of the American Statistical Association 88 (424 ), 1242– 9 .

  Lehmann , E. (2011 ). Fisher, Neyman, and the Creation of Classical Statistics , 1st edn. New York: Springer .

  Lehmann , E. (2012 ). Selected Works of E. L. Lehmann , Rojo , J. (ed.). New York : Springer .

  Lehmann , E. and Romano , J. (2005 ). Testing Statistical Hypotheses , 3rd edn. New York : Springer .

  Letzter , R. (2016 ). ‘ Scientists Are Furious after a Famous Psychologist Accused Her Peers of “ Methodological Terrorism”’ , Business Insider (9/22/2016), businessinsider.com/susan-fiske-methodological-terrorism-2016– 9 .

  Levelt Committee, Noort Committee, Drenth Committee (2012 ). ‘ Flawed Science: The Fraudulent Research Practices of Social Psychologist Diederik Stapel’ , Stapel Investigation: Joint Tilburg/Groningen/Amsterdam investigation of the publications by Mr. Stapel (www.commissielevelt.nl/ ).

  Levi , I. (1980 ). The Enterprise of Knowledge: An Essay on Knowledge, Credal Probability, and Change . Cambridge, MA : MIT Press .

  Lindemann , F. (1919 ). ‘ Contribution to “ Discussion on the Theory of Relativity” ’ , Monthly Notices of the Royal Astronomical Society 80 , 114 .

  Lindley , D. (1957 ). ‘ A Statistical Paradox ’ , Biometrika 44 , 187– 92 .

  Lindley , D. (1969 ). ‘ Discussion of Compound Decisions and Empirical Bayes , J. B. Copas ’ , Journal of the Royal Statistical Society: Series B 31 , 397 – 425 .

  Lindley , D. (1971 ). ‘ The Estimation of Many Parameters’ , in Godambe , V. and Sprott , D. (eds.), pp. 435– 55 .

  Lindley , D. (1976 ). ‘ Bayesian Statistics’ , in Harper , W. and Hooker , C. (eds.), pp. 353– 62 .

  Lindley , D. (1982 ). ‘ The Role of Randomization in Inference ’ , PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association , 1982(2 ), 431– 46 .

  Lindley , D. (2000 ). ‘ The Philosophy of Statistics ’ (with Discussion), Journal of the Royal Statistical Society: Series D 49 (3 ), 293 – 337 .

  Lindley , D. and Novick , M. (1981 ). ‘ The Role of Exchangeability in Inference ’ , Annals of Statistics 9 (1 ), 45 – 58 .

  Little , R. (2006 ). ‘ Calibrated Bayes: A Bayes/Frequentist Roadmap ’ , The American Statistician 60 (3 ), 213– 23 .

  Lodge , O. (1919 ). ‘ Contribution to “ Discussion on the Theory of Relativity” ’ , Monthly Notices of the Royal Astronomical Society 80 , 106– 9 .

  Logan , B. (2012 ). ‘ Jackie Mason Review’ , The Guardian (2/21/2012).

  Longino , H. (2002 ). The Fate of Knowledge . Princeton, NJ : Princeton University Press .

  Madigan , D. and Raftery , A. (1994 ). ‘ Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam’ s Window ’ , Journal of the American Statistical Association 89 (428 ), 1535– 46 .
/>   Maher , P. (2004 ). ‘ Bayesianism and Irrelevant Conjunction ’ , Philosophy of Science 71 , 515– 20 .

  Marcus , G. (2018 ). ‘ Deep Learning: A Critical Appraisal’ , arXiv: 1801.00631 preprint, 1– 27.

  Martin , R. and Liu , C. (2013 ). ‘ Inferential Models: A Framework for Prior-free Posterior Probabilistic Inference ’ , Journal of the American Statistical Association 108 , 301– 13 .

  Mayo , D. (1980 ). ‘ The Philosophical Relevance of Statistics’ , PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1980 , 97 – 109 .

  Mayo , D. (1983 ). ‘ An Objective Theory of Statistical Testing ’ , Synthese 57 (3 ), 297 – 340 .

  Mayo , D. (1988 ). ‘ Toward a More Objective Understanding of the Evidence of Carcinogenic Risk ’ , PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association 1988 , 2 , 489 – 503 .

  Mayo , D. (1991 ). ‘ Novel Evidence and Severe Tests ’ , Philosophy of Science 58 (4 ), 523– 52 .

  Mayo , D. (1996 ). Error and the Growth of Experimental Knowledge . Chicago : University of Chicago Press .

  Mayo , D. (1997 a). ‘ Duhem’ s Problem, the Bayesian Way, and Error Statistics, or “ What’ s Belief Got to Do with It?”’ and ‘ Response to Howson and Laudan’ , Philosophy of Science 64 (2 ), 222– 44 , 323– 33 .

  Mayo , D. (1997 b). ‘ Severe Tests, Arguing From Error, and Methodological Underdetermination ’ , Philosophical Studies 86 (3 ), 243– 66 .

  Mayo , D. (2003 a). ‘ Severe Testing as a Guide for Inductive Learning’ , in Kyburg , H. E. and Thalos , M. (eds.), pp. 89 – 117 .

  Mayo , D. (2003 b). ‘ Could Fisher, Jeffreys and Neyman Have Agreed on Testing? Commentary on J. Berger’ s Fisher Address ’ , Statistical Science 18 , 19 – 24 .

  Mayo , D. (2004 ). ‘ An Error-statistical Philosophy of Evidence’ and ‘ Rejoinder’ , in Taper , M. and Lele , S. (eds.), pp. 79 – 97 , 101– 18 .

  Mayo , D. (2005 a). ‘ Peircean Induction and the Error-Correcting Thesis ’ , Transactions of the Charles S. Peirce Society: A Quarterly Journal in American Philosophy 41 (2 ), 299 – 319 .

  Mayo , D. (2005 b). ‘ Evidence as Passing Severe Tests: Highly Probable versus Highly Probed Hypotheses’ , in Achinstein, P. (ed.), Scientific Evidence , Johns Hopkins, pp. 95– 127.