Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site. (30th August 2021)

• Record Type: Journal Article
• Authors: Solanes, Aleix; Palau, Pol; Fortea, Lydia; Salvador, Raymond; González-Navarro, Laura; Llach, Cristian Daniel; Valentí, Marc; Vieta, Eduard; Radua, Joaquim
• Is Part Of: Psychiatry research. Volume 314(2021)
• Publication Date: 2021-08-30
• Subjects: Bias -- Effects of the site -- Machine learning -- Magnetic resonance imaging; Psychiatry -- Periodicals; Brain -- Imaging -- Periodicals; Psychiatry -- Periodicals; Diagnostic Imaging -- Periodicals; Psychiatrie -- Périodiques; Cerveau -- Imagerie pour le diagnostic -- Périodiques; 616.890754
• DOI: 10.1016/j.pscychresns.2021.111313
• Languages: English
• ISSNs: 0925-4927

Abstract

Highlights: The effects of the site (EoS) may bias multisite machine learning MRI studies. Removing the EoS when training the model may be insufficient. We provide examples of these biases and statistical methods to avoid them. Researchers must control the EoS when estimating the accuracy. We also provide an R package ("multisite.accuracy") to ease this task. Abstract: Brain MRI researchers conducting multisite studies, such as within the ENIGMA Consortium, are very aware of the importance of controlling the effects of the site (EoS) in the statistical analysis. Conversely, authors of the novel machine-learning MRI studies may remove the EoS when training the machine-learning models but not control them when estimating the models' accuracy, potentially leading to severely biased estimates. We show examples from a toy simulation study and real MRI data in which we remove the EoS from both the "training set" and the "test set" during the training and application of the model. However, the accuracy is still inflated (or occasionally shrunk) unless we further control the EoS during the estimation of the accuracy. We also provide several methods for controlling the EoS during the estimation of the accuracy, and a simple R package ("multisite.accuracy") that smoothly does this task for several accuracy estimates (e.g., sensitivity/specificity, area under the curve, correlation, hazard ratio, etc.).