Statistical Inference as Severe Testing

by Deborah G Mayo

  Burnham , K. and Anderson , D. (2014 ). ‘ P values Are Only an Index to Evidence: 20th- vs. 21st-century Statistical Science ’ , Ecology 95 (3 ), 627– 30 .

  Byrd , J. (2018 ). ‘ Models and Methods for Osteometric Sorting ’ , in Adams , B. and Byrd , J. (eds.), Recovery, Analysis, and Identification of Commingled Human Remains , Totowa, NJ: Humana Press , pp. 199 – 220 .

  Carlin , B. and Louis , T. (2008 ). Bayesian Methods for Data Analysis , 3rd edn. Boca Raton, FL : Chapman and Hall/CRC .

  Carnap , R. (1962 ). Logical Foundations of Probability , 2nd edn. Chicago, IL : University of Chicago Press .

  Cartlidge , E. (2016 ). ‘ Theorizing About the LHC’ s 750 GeV Bump’ , Physicsworld.com (4/19/2016).

  Cartwright , N. (2012 ). ‘ RCTs, Evidence, and Predicting Policy Effectiveness ’ , in Kincaid , H. , The Oxford Handbook of Philosophy of Social Science , Oxford : Oxford University Press .

  Cartwright , N. and Hardie , J. (2012 ). Evidence-Based Policy: A Practical Guide to Doing It Better . Oxford : Oxford University Press .

  Casella , G. and Berger , R. (1987 a). ‘ Reconciling Bayesian and Frequentist Evidence in the One-sided Testing Problem ’ , Journal of the American Statistical Association 82 (397 ), 106– 11 .

  Casella , G. and Berger , R. (1987 b). ‘ Comment on Testing Precise Hypotheses by J. O. Berger and M. Delampady ’ , Statistical Science 2 (3 ), 344– 7 .

  Casella , G. and Berger , R. (2002 ). Statistical Inference , 2nd edn. Belmont, CA : Duxbury Press .

  Castelvecchi , D. and Gibney , E. (2016 ). ‘ Hints of New LHC Particle Get Slightly Stronger’ , Nature online (3/18/2016).

  Center for Open Science, University of Virginia (2015 ). Reproducibility Project : cos.io/pr/2015– 09-29.

  Chalmers , A. (2010 ). ‘ Can Scientific Theories Be Warranted?’ , in Mayo , D. and Spanos , A. (eds.), pp. 58 – 72 .

  Cherkassky , V. (2012 ). ‘ Vladimir Cherkassky Responds on Foundations of Simplicity’ , Guest Blogpost on Errorstatistics.com (7/6/2012).

  CMS Experiment (2012). ‘ Observation of a New Particle with a Mass of 125 GeV’ , Press statement, Compact Muon Solenoid Experiment at CERN [blog], July 4, 2012 . http://cms.web.cern.ch/news/observation-new-particle-mass-125-gev .

  The Cochrane Collaboration : cochrane.org .

  Cohen , J. (1962 ). ‘ The Statistical Power of Abnormal-Social Psychological Research: A Review ’ , Journal of Abnormal and Social Psychology 65 , 145– 53 .

  Cohen , J. (1988 ). Statistical Power Analysis for the Behavioral Sciences , 2nd edn. Hillsdale, NJ : Erlbaum .

  Cohen , J. (1990 ). ‘ Things I Have Learned (So Far) ’ , American Psychologist 45 (12 ), 1304– 12 .

  Colquhoun , D. (2011 ). ‘ In Praise of Randomisation: The Importance of Causality in Medicine and Its Subversion by Philosophers of Science ’ , Proceedings of the British Academy 171 , 321– 40 .

  Colquhoun , D. (2014 ). ‘ An Investigation of the False Discovery Rate and the Misinterpretation of P-values ’ , Royal Society Open Science 1 (3 ), 140216 (16 pages).

  Committee on Science, Technology, Law Policy and Global Affairs, Committee on the Development of the Third Edition of the Reference Manual on Scientific Evidence , and Federal Judicial Center (2011 ). Reference Manual on Scientific Evidence , 3rd edn. Washington, D.C. : National Academy Press .

  COMpare Team (2015 ). ‘ Letter to NEJM (Dan Longo, Deputy Editor)’ on 11/ 25/2015 (compare-trials.org ).

  Coombes , K. , Wang J. , and Baggerly , K. (2007 ). ‘ Microarrays: Retracing Steps ’ , Nature Medicine 13 (11 ), 1276– 7 .

  Cousins , R. (2017 ). ‘ The Jeffreys-Lindley Paradox and Discovery Criteria in High Energy Physics ’ , Synthese 194 , 395 – 432 .

  Cox , D. (1958 ). ‘ Some Problems Connected with Statistical Inference ’ , Annals of Mathematical Statistics 29 (2 ), 357– 72 .

  Cox , D. (1977 ). ‘ The Role of Significance Tests ’ (with Discussion), Scandinavian Journal of Statistics 4 , 49 – 70 .

  Cox , D. (1978 ). ‘ Foundations of Statistical Inference: The Case for Eclecticism ’ , Australian Journal of Statistics 20 (1 ), 43 – 59 .

  Cox , D. (1982 ). ‘ Statistical Significance Tests ’ , British Journal of Clinical Pharmacology 14 , 325 – 331 .

  Cox , D. (1995 ). ‘ Comment on “ Model Uncertainty, Data Mining and Statistical Inference by C. Chatfield” ’ , Journal of the Royal Statistical Society: Series A 158 , 419– 66 .

  Cox , D. (2000 ). ‘ Comments on the Paper by Lindley ’ , Journal of the Royal Statistical Society: Series D 49 (3 ), 321– 4 .

  Cox , D. (2006 a). Principles of Statistical Inference . Cambridge : Cambridge University Press .

  Cox , D. (2006 b). ‘ Frequentist and Bayesian Statistics: A Critique’ (keynote address), in Statistical Problems in Particle Physics, Astrophysics and Cosmology , Lyons , L. and Müge Karagöz Ünel (eds.), London : Imperial College Press , pp. 3– 6.

  Cox , D. and Hinkley , D. (1974 ). Theoretical Statistics . London : Chapman and Hall .

  Cox , D. and Mayo , D. (2010 ). ‘ Objectivity and Conditionality in Frequentist Inference’ , in Mayo , D and Spanos , A. (eds.), pp. 276 – 304 .

  Cox , D. and Mayo , D. (2011 ). ‘ A Statistical Scientist Meets a Philosopher of Science: A Conversation between Sir David Cox and Deborah Mayo ’ , in Rationality, Markets and Morals (RMM) 2 , 103– 14 .

  Crupi , V. and Tentori , K. (2010 ). ‘ Irrelevant Conjunction: Statement and Solution of a New Paradox ’ , Philosophy of Science 77 , 1 – 13 .

  Cumming , G. (2012 ). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis . New York : Routledge .

  Dawid , A. (1976 ). ‘ Properties of Diagnostic Data Distributions ’ , Biometrics 32 (3 ), 647– 58 .

  Dawid , A. (2000 ). ‘ Comment on a Paper by Lindley ’ , Journal of the Royal Statistical Society: Series D 49 (3 ), 325– 6 .

  Dawid , A. (2015 ). ‘ Comment’ on ‘ Can You Change Your Bayesian Prior?’ , Blogpost on Errorstatistics.com (6/18/2015).

  Dawid , A. , Stone , M. , and Zidek , J. (1973 ). ‘ Marginalization Paradoxes in Bayesian and Structural Inference ’ , Journal of the Royal Statistical Society: Series B 35 (2 ), 189 – 233 .

  Deaton , A. (2010 ). ‘ Instruments, Randomization, and Learning about Development ’ , Journal of Economic Literature 48 (2 ), 424– 55 .

  de Finetti , B. (1972 ). Probability, Induction and Statistics: The Art of Guessing . New York : John Wiley & Sons .

  de Finetti , B. (1974 ). Theory of Probability: A Critical Introductory Treatment , 2 volumes. New York : John Wiley & Sons .

  Diaconis , P. (1978 ). ‘ Statistical Problems in ESP Research ’ , Science 201 (4351 ), 131– 6 .

  Dienes , Z. (2008 ). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference . Basingstoke, UK: Palgrave Macmillan .

  Dominus , S. (2017 ). ‘ When the Revolution Came for Amy Cuddy ’ , The New York Times Magazine , New York Times Company (October 18, 2017).

  Doudna , J. and Charpentier , E. (2014 ). ‘ The New Frontier of Genome Engineering with CRISPR-Cas9 ’ , Science 346 (6213 ), 1077 (1– 9).

  Draper , D. and Madigan , D. (1997 ). ‘ The Scientific Value of Bayesian Statistical Methods ’ , part of ‘ Banter on Bayes: Debating the Usefulness of Bayesian Approaches to Solving Practical Problems,’ hosted by Hearst, M., IEEE Intelligent Systems and Their Applications 12 (6 ), 18 – 20 .

  Duhem , P. (1954 ). The Aim and Structure of Physical Theory . New Jersey : Princeton University Press .

  Dupré , J. (1993 ). The Disorder of Things . Cambridge, MA : Harvard University Press .

  Durante , K. , Rae , A. , and Griskevicius , V. (2013 ). ‘ The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle ’ , Psychological Science 24 (6 ), 1007– 16 .

  Dyson , E. , Eddington , A. , and Davidson , C. (1920 ). ‘ A Determination of the Deflection of Light by the Sun’ s Gravitational Field, from Observa
tions Made at the Total Eclipse of May 29, 1919 ’ , Philosophical Transactions of the Royal Society of London , A220 , 291 – 333 .

  Earman , J. (1992 ). Bayes or Bust?: A Critical Examination of Bayesian Confirmation Theory . Cambridge, MA : MIT Press .

  Earman , J. and Glymour , C. (1980 ). ‘ Relativity and Eclipses: The British Eclipse Expeditions of 1919 and Their Predecessors ’ , Historical Studies in the Physical Sciences 11 (1 ), 49 – 85 .

  Eddington , A. ([1920 ]1987 ). Space, Time and Gravitation: An Outline of the General Relativity Theory , Cambridge Science Classics Series. Cambridge : Cambridge University Press .

  Edwards , A. F. (1972 ). Likelihood . Cambridge : Cambridge University Press .

  Edwards , W. , Lindman , H. , and Savage , L. (E, L, & S) (1963 ). ‘ Bayesian Statistical Inference for Psychological Research ’ , Psychological Review 70 (3 ), 193 – 242 .

  Efron , B. (1979 ). ‘ Bootstrap Methods: Another Look at the Jackknife ’ , The Annals of Statistics 7 (1 ), 1 – 26 .

  Efron , B. (1986 ). ‘ Why Isn’ t Everyone a Bayesian? ’ , The American Statistician 40 (1 ), 1 – 5 .

  Efron , B. (1998 ). ‘ R. A. Fisher in the 21st Century ’ and ‘ Rejoinder’ , Statistical Science 13 (2 ), 95 – 114 ; 121– 2 .

  Efron , B. (2000 ). ‘ Comments on the Paper by Lindley ’ , Journal of the Royal Statistical Society: Series D 49 (3 ), 330– 1 .

  Efron , B. (2013 ). ‘ A 250-Year Argument: Belief, Behavior, and the Bootstrap ’ , Bulletin of the American Mathematical Society 50 (1 ), 126– 46 .

  Ellis , P. (2010 ). Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results . Cambridge : Cambridge University Press .

  Feynman , R. (1974 ). ‘ Cargo Cult Science ,’ Caltech Commencement Speech. Reprinted in Surely You’ re Joking Mr. Feynman!: Adventures of a Curious Character (1985), New York : Norton & Company , pp. 382– 91 .

  Fisher , R. A. (1926 ). ‘ The Arrangement of Field Experiments ’ , Journal of Ministry of Agriculture 33 , 503– 13 . Reprinted in R. A. Fisher: Contributions to Mathematical Statistics (1950 ), Paper 17, New York : John Wiley & Sons .

  Fisher , R. A. (1930 ). ‘ Inverse Probability ’ , Mathematical Proceedings of the Cambridge Philosophical Society 26 (4 ), 528– 35 .

  Fisher , R. A. (1934 a). Statistical Methods for Research Workers . Edinburgh : Oliver and Boyd . (First published in 1925.) Reprinted in Fisher 1990.

  Fisher , R. A. (1934 b). ‘ Two New Properties of Mathematical Likelihood ’ , Proceedings of the Royal Society of London. Series A 144 (852 ), 285 – 307 .

  Fisher . R. A. (1934 c). ‘ Discussion and Commentary’ pp . 137– 8 in Neyman 1934.

  Fisher , R. A. (1935 a). The Design of Experiments , 1st edn., Edinburgh : Oliver and Boyd. Reprinted in Fisher 1990 .

  Fisher , R. A. (1935 b). ‘ The Logic of Inductive Inference ’ , Journal of the Royal Statistical Society 98 (1 ), 39 – 82 .

  Fisher , R. A. (1936 ), ‘ Uncertain Inference ’ , Proceedings of the American Academy of Arts and Sciences 71 , 248– 58 .

  Fisher , R. A. (1938 ). ‘ Presidential Address ’ , Sankhyā : The Indian Journal of Statistics 4 (1 ), 14 – 17 .

  Fisher , R. A. (1939 ). ‘ The Comparison of Samples with Possibly Unequal Variances’ , Annals of Eugenics IX (pt II), 174– 80. Reprinted in Fisher, R. (1950) (‘ Author’ s Note’ 35.173a-35.173b); 35.174-35.180.

  Fisher , R. A. (1950 ). Contributions to Mathematical Statistics . New York : John Wiley & Sons .

  Fisher , R. A. (1955 ). ‘ Statistical Methods and Scientific Induction’ , Journal of the Royal Statistical Society: Series B 17 (1 ), 69 – 78 .

  Fisher , R. A. (1956 ). Statistical Methods and Scientific Inference . Edinburgh : Oliver and Boyd. Reprinted in Fisher 1990 .

  Fisher , R. A. (1990 ). Statistical Methods, Experimental Design, and Scientific Inference , (ed.), Bennett, J. H. Oxford : Oxford University Press .

  Fitelson , B. (2002 ). ‘ Putting the Irrelevance Back into the Problem of Irrelevant Conjunction ’ , Philosophy of Science 69 (4 ), 611– 22 .

  Folks , J. (1995 ). ‘ A Conversation with Oscar Kempthorne ’ , Statistical Science 10 (4 ), 321– 36 .

  Forster , M. and Sober , E. (1994 ). ‘ How to Tell When Simpler, More Unified, or Less Ad Hoc Theories Will Provide More Accurate Predictions ’ , The British Journal for the Philosophy of Science 45 (1 ), 1 – 35 .

  Fraser , D. (1961 ). ‘ On Fiducial Inference ’ , Annals of Mathematical Statistics 32 (3 ), 661 – 676 .

  Fraser , D. (1998 ). ‘ [R. A. Fisher in the 21st Century]: Comment ’ , Statistical Science 13 (2 ), 118 – 120 .

  Fraser , D. (2011 ). ‘ Is Bayes Posterior Just Quick and Dirty Confidence?’ and ‘ Rejoinder’ , Statistical Science 26 (3 ), 299 – 316 ; 329– 31 .

  Fraser , D. (2014 ). ‘ Why Does Statistics Have Two Theories? ’ , in Lin , X. , Banks , D. , Genest , C. , Molenberghs , G. , Scott , D. and Wang , J.-L. (eds.), Past, Present and Future of Statistical Science , 237 – 252 . CRC Press , Boca Raton, FL .

  Fraser , D. and Reid , N. (2002 ). ‘ Strong Matching of Frequentist and Bayesian Parametric Inference ’ , Journal of Statistical Planning and Inference 103 , 263– 85 .

  Freedman , D. (2009 ). Statistical Models: Theory and Practice , 2nd edn. Cambridge : Cambridge University Press .

  Freiman , J. A. , Chalmers , T. C. , Smith , H. , and Kuebler , R. R. (1978 ). ‘ The Importance of Beta, the Type II Error and Sample Size in the Design and Interpretation of the Randomized Control Trial: Survey of 71 Negative Trials ’ , The New England Journal of Medicine 299 (13 ), 690– 4 .

  Gaifman , H. and Vasudevan , A. (2012 ). ‘ Deceptive Updating and Minimal Information Methods ’ , Synthese 187 (1 ), 147– 78 .

  Gelman , A. (2011 ). ‘ Induction and Deduction in Bayesian Data Analysis ’ , Rationality, Markets and Morals (RMM) 2 , 67 – 78 .

  Gelman , A. (2012 ). ‘ Ethics and the Statistical Use of Prior Information ’ , Chance Magazine 25 (4 ), 52– 4 .

  Gelman , A. (2013 ). ‘ P Values and Statistical Practice ’ , Epidemiology 24 (1 ), 69 – 72 .

  Gelman , A. (2015 ). ‘ Comment on “ Can You Change Your Bayesian Prior (ii)”’ . Blogpost on Errorstatistics.com (6/18/2015).

  Gelman , A. and Carlin , J. (2014 ). ‘ Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors ’ , Perspectives on Psychological Science 9 , 641– 51 .

  Gelman , A. , Carlin , J. , Stern , H. , et al. (2004 ). Bayesian Data Analysis , 2nd edn. Boca Raton, FL : Chapman & Hall/CRC.

  Gelman , A. and Hennig , C. (2017 ). ‘ Beyond Subjective and Objective in Statistics ’ , Journal of the Royal Statistical Society: Series A 180 (4 ), 967 – 1033 .

  Gelman , A. and Loken , E. (2014 ). ‘ The Statistical Crisis in Science ’ , American Scientist 2 , 460– 5 .

  Gelman , A. , Meng , X-L. , and Stern , H. (1996 ). ‘ Posterior Predictive Assessment of Model Fitness Via Realized Discrepancies ’ (with Discussion), Statistica Sinica 6 , 733 – 807 .

  Gelman , A. and Robert , C. (2013 ). ‘“ Not Only Defended but Also Applied” : The Perceived Absurdity of Bayesian Inference’ and ‘ Rejoinder’ , The American Statistician 67 (1 ), 1 – 5 ; 15– 6 .

  Gelman , A. and Shalizi , C. (2013 ). ‘ Philosophy and the Practice of Bayesian Statistics’ and ‘ Rejoinder’ , British Journal of Mathematical and Statistical Psychology 66 (1 ), 8 – 38 ; 76 – 80 .

  Gibbons , J. D. and Pratt , J. W. (1975 ). ‘ P-values: Interpretation and Methodology ’ , The American Statistician 29 (1 ), 20– 5 .

  Giere , R. N. (1969 ). ‘ Bayesian Statistics and Biased Procedures ’ , Synthese 20 (3 ), 371– 87 .

  Giere , R. N. (1988 ). Explaining Science. Chicago : University of Chicago Press .

  Gigerenzer , G. (1991 ). ‘ How to Make Cognitive Illusions Disappear: Beyond “ Heuristics and Biases” ’ , European Review of Social Psychology 2 , 83 – 115 .

&nbs
p; Gigerenzer , G. (2002 ). Adaptive Thinking: Rationality in the Real World . Oxford : Oxford University Press .

  Gigerenzer , G. (2004 ). ‘ Mindless Statistics ’ , Journal of Socio-Economics 33 (5 ), 587 – 606 .

  Gigerenzer , G. and Marewski , J. (2015 ). ‘ Surrogate Science: The Idol of a Universal Method for Scientific Inference ’ , Journal of Management 41 (2 ), 421– 40 .

  Gigerenzer , G. , Swijtink , Z. , Porter , T. , et al. (1989 ). Empire of Chance: How Probability Changed Science and Everyday Life . Cambridge : Cambridge University Press .

  Gill , R. (2014 ). ‘ Who Ya Gonna Call for Statistical Fraudbusting? R. A. Fisher, P-values, and Error Statistics (Again)’ , Guest Blogpost on Errorstatistics.com (5/10/2014).

  Gillies , D. (2000 ). Philosophical Theories of Probability. New York : Routledge .

  Gillies , D. (2001 ). ‘ Bayesianism and the Fixity of the Theoretical Framework ’ , in Corfield , D. and Williamson , J. (eds.), Foundations of Bayesianism 24 , The Netherlands : Springer , pp. 363– 79 .

  Ghosh , J. , Delampady , M. , and Samanta , T. (2010 ). An Introduction to Bayesian Analysis: Theory and Methods . New York : Springer .

  Glymour , C. (1980 ). Theory and Evidence . Princeton : Princeton University Press .

  Glymour , C. (2010 ). ‘ Explanation and Truth’ , in Mayo , D. and Spanos , A. (eds.), pp. 331– 50 .

  Godambe , V. and Sprott , D. (eds.) (1971 ). Foundations of Statistical Inference . Toronto : Holt, Rinehart and Winston of Canada .

  Goldacre , B. (2008 ). Bad Science . London : HarperCollins Publishers .

  Goldacre , B. (2012 ). Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients . London : Fourth Estate.

  Goldacre , B. (2016 ). ‘ Make Journals Report Clinical Trials Properly ’ , Nature 530 (7588 ), 7 .

  Goldman , A. (1986 ). Epistemology and Cognition . Cambridge, MA : Harvard University Press .

  Goldman , A. (1999 ). Knowledge in a Social World . Oxford : Oxford University Press .

  Good , I. J. (1971 a). ‘ The Probabilistic Explication of Information, Evidence, Surprise, Causality, Explanation, and Utility’ and ‘ Reply’ , in Godambe , V. and Sprott , D. (eds.), pp. 108– 22 , 131– 41 .