Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study. (April 2023)

• Record Type: Journal Article
• Authors: Chuang, Zachary; Martin, Janet; Shapiro, Jordan; Nguyen, Derek; Neocleous, Penelope; Jones, Philip M.
• Is Part Of: British journal of anaesthesia. Volume 130:Number 4(2023)
• Page Start: 412
• Page End: 420
• Publication Date: 2023-04
• Subjects: Bayes factor -- evidence-based medicine -- false-positive risk -- P-value -- reproducibility -- statistics; Anesthesiology -- Periodicals; Anesthesia -- Periodicals; 617.9605
• DOI: 10.1016/j.bja.2022.11.001
• Languages: English
• ISSNs: 0007-0912

Abstract

Abstract: Background: Reproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P -value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals. Methods: We identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P -values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR50 ). The primary outcomes were the percentage of RCTs maintaining statistical significance at P <0.005 and minFPR50 . Results: We included 318 RCTs. P -values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P <0.005 threshold. The mean (standard deviation) minFPR50 was 22% (20). At P =0.005, the minFPR50 was approximately 5%. Conclusions: These proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P -value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.