Design, analysis and reporting of multi-arm trials and strategies to address multiple testing. (16th March 2020)

• Record Type: Journal Article
• Authors: Odutayo, Ayodele; Gryaznov, Dmitry; Copsey, Bethan; Monk, Paul; Speich, Benjamin; Roberts, Corran; Vadher, Karan; Dutton, Peter; Briel, Matthias; Hopewell, Sally; Altman, Douglas G
• Is Part Of: International journal of epidemiology. Volume 49:Number 3(2020)
• Page Start: 968
• Page End: 978
• Publication Date: 2020-03-16
• Subjects: Multi-arm trials -- multiple testing -- type I error -- type II error; Epidemiology -- Periodicals; 614.4
• DOI: 10.1093/ije/dyaa026
• Languages: English
• ISSNs: 0300-5771

Abstract

Abstract: Background: It is unclear how multiple treatment comparisons are managed in the analysis of multi-arm trials, particularly related to reducing type I (false positive) and type II errors (false negative). Methods: We conducted a cohort study of clinical-trial protocols that were approved by research ethics committees in the UK, Switzerland, Germany and Canada in 2012. We examined the use of multiple-testing procedures to control the overall type I error rate. We created a decision tool to determine the need for multiple-testing procedures. We compared the result of the decision tool to the analysis plan in the protocol. We also compared the pre-specified analysis plans in trial protocols to their publications. Results: Sixty-four protocols for multi-arm trials were identified, of which 50 involved multiple testing. Nine of 50 trials (18%) used a single-step multiple-testing procedures such as a Bonferroni correction and 17 (38%) used an ordered sequence of primary comparisons to control the overall type I error. Based on our decision tool, 45 of 50 protocols (90%) required use of a multiple-testing procedure but only 28 of the 45 (62%) accounted for multiplicity in their analysis or provided a rationale if no multiple-testing procedure was used. We identified 32 protocol–publication pairs, of which 8 planned a global-comparison test and 20 planned a multiple-testing procedure in their trial protocol. However, four of these eight trials (50%) did not use the global-comparison test. Likewise, 3 of the 20 trials (15%) did not perform the multiple-testing procedure in the publication. The sample size of our study was small and we did not have access to statistical-analysis plans for the included trials in our study. Conclusions: Strategies to reduce type I and type II errors are inconsistently employed in multi-arm trials. Important analytical differences exist between planned analyses in clinical-trial protocols and subsequent publications, which may suggest selective reporting of analyses.