Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study. (April 2023) - Chuang, Zachary; Martin, Janet; Shapiro, Jordan; Nguyen, Derek; Neocleous, Penelope; Jones, Philip M.
Abstract: Background: Reproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P -value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals. Methods: We identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P -values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR50 ). The primary outcomes were the percentage of RCTs maintaining statistical significance at P <0.005 and minFPR50 . Results: We included 318 RCTs. P -values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P <0.005 threshold. The mean (standard deviation) minFPR50 was 22% (20). At P =0.005, the minFPR50 was approximately 5%. Conclusions: These proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P -value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.