Effects of a proposal to alter the statistical significance threshold on previously published orthopaedic trauma randomized controlled trials. Issue 11 (November 2019) - Johnson, Austin L.; Evans, Sheridan; Checketts, Jake X.; Scott, Jared T.; Wayant, Cole; Johnson, Mark; Norris, Brent; Vassar, Matt
Highlights: It has been suggested that we adopt a new significance level by lowering the threshold from .05 to .005. This proposal is meant to serve as a temporizing measure while long-term strategies are developed, tested, and implemented. A common misconception in clinical research is that statistical significance equates to clinical significance. Abstract: Introduction: A recent proposal suggests changing the threshold for statistical significance from a P value of .05 to .005 to minimize bias and increase reproducibility of future studies. P values less than .05 but greater than .005 would be reclassified as "suggestive", whereas P values less than .005 would be considered significant. The present study explores how lowering the P value threshold would affect the interpretation of previously published orthopaedic trauma randomized controlled trials (RCTs) and whether outcomes from these trials would maintain statistical significance under the proposed P value threshold. Methods: All RCTs published between January 01, 2016 and January 31, 2018 in the Journal of Orthopaedic Trauma, Injury, and Archives of Orthopaedic and Trauma Surgery were screened by at least 2 authors. Data from included trials were extracted in blinded and duplicate fashion. All P values for primary endpoints were included from each study. Results: We identified 124 primary endpoints from 48 trials: 39.5% (49/124) of endpoints had a P value less than .05 and 60.5% (75/124) had a P value greater than .05. Overall, 51.0% (25/49) of statistically significant primary endpoints were less than .005, while 49.0% (24/49) would be reclassified as suggestive. Conclusion: Based on our results, adopting a lower threshold of significance would heavily alter the significance of orthopaedic trauma RCTs and should be further evaluated and cautiously considered when viewing the effect such a proposal on orthopaedic practice.