Bayesian prediction intervals for assessing P-value variability in prospective replication studies. Issue 12 (December 2017)

• Record Type: Journal Article
• Authors: Vsevolozhskaya, Olga; Ruiz, Gabriel; Zaykin, Dmitri
• Is Part Of: Translational psychiatry. Volume 7:Issue 12(2017)
• Page Start: 1
• Page End: 15
• Publication Date: 2017-12
• Subjects: Psychiatry -- Research -- Periodicals; Neurosciences -- Research -- Periodicals; Psychiatry -- Periodicals; Neurosciences -- Periodicals; Translational Research -- Periodicals; Health Policy -- Periodicals; Public Health -- Periodicals; 616.89
• DOI: 10.1038/s41398-017-0024-3
• Languages: English
• ISSNs: 2158-3188

Abstract

Abstract Increased availability of data and accessibility of computational tools in recent years have created an unprecedented upsurge of scientific studies driven by statistical analysis. Limitations inherent to statistics impose constraints on the reliability of conclusions drawn from data, so misuse of statistical methods is a growing concern. Hypothesis and significance testing, and the accompanyingP -values are being scrutinized as representing the most widely applied and abused practices. One line of critique is thatP -values are inherently unfit to fulfill their ostensible role as measures of credibility for scientific hypotheses. It has also been suggested that whileP -values may have their role as summary measures of effect, researchers underappreciate the degree of randomness in theP -value. High variability ofP -values would suggest that having obtained a smallP -value in one study, one is, ne vertheless, still likely to obtain a much largerP -value in a similarly powered replication study. Thus, "replicability ofP -value" is in itself questionable. To characterizeP -value variability, one can use prediction intervals whose endpoints reflect the likely spread ofP -values that could have been obtained by a replication study. Unfortunately, the intervals currently in use, the frequentistP -intervals, are based on unrealistic implicit assumptions. Namely, P -intervals are constructed with the assumptions that imply substantial chances of encountering large values of effect size in an observational study, which leads to bias. The long-run frequentist probability provided byP -intervals is similar in interpretation to that of the classical confidence intervals, but the endpoints of any particular interval lack interpretation as probabilistic bounds for the possible spread of futureP -values that may have been obtained in replication studies. Along with classical frequentist intervals, there exists a Bayesian viewpoint toward interval construction in which the endpoints of an interval have a meaningful probabilistic interpretation. We propose Bayesian intervals for prediction ofP -value variability in prospective replication studies. Contingent upon approximate prior knowledge of the effect size distribution, our proposed Bayesian intervals have endpoints that are directly interpretable as probabilistic bounds for replicationP -values, and they are resistant to selection bias. We showcase our approach by its application toP -values reported for five psychiatric disorders by the Psychiatric Genomics Consortium group.