A Statistical Model to Investigate the Reproducibility Rate Based on Replication Experiments. (13th June 2018)

• Record Type: Journal Article
• Authors: Pauli, Francesco
• Is Part Of: International statistical review. Volume 87:Number 1(2019)
• Page Start: 68
• Page End: 79
• Publication Date: 2018-06-13
• Subjects: p‐value -- false discovery rate -- reproducibility crisis -- mixture model; Statistics -- Periodicals; Statistics -- Bibliography -- Periodicals; Statistics -- Bibliography; Statistics -- Periodicals; Statistique; Statistique -- Périodiques; Statistique -- Bibliographie -- Périodiques; Statistique -- Étude et enseignement -- Périodiques; Statistique -- Étude et enseignement -- Bibliographie -- Périodiques; Ressource Internet (Descripteur de forme); Périodique électronique (Descripteur de forme); 519.2
• DOI: 10.1111/insr.12273
• Languages: English
• ISSNs: 0306-7734

Abstract

Summary: The reproducibility crisis, that is, the fact that many scientific results are difficult to replicate, pointing to their unreliability or falsehood, is a hot topic in the recent scientific literature, and statistical methodologies, testing procedures and p ‐values, in particular, are at the centre of the debate. Assessment of the extent of the problem–the reproducibility rate or the false discovery rate–and the role of contributing factors are still an open problem. Replication experiments, that is, systematic replications of existing results, may offer relevant information on these issues. We propose a statistical model to deal with such information, in particular to estimate the reproducibility rate and the effect of some study characteristics on its reliability. We analyse data from a recent replication experiment in psychology finding a reproducibility rate broadly coherent with other assessments from the same experiment. Our results also confirm the expected role of some contributing factor (unexpectedness of the result and room for bias) while they suggest that the similarity between original study and the replica is not so relevant, thus mitigating some criticism directed to replication experiments.