Applied forecasting for delayed cerebral ischemia prediction post subarachnoid hemorrhage: Methodological fallacies. (2022)

• Record Type: Journal Article
• Authors: Alexopoulos, Georgios; Zhang, Justin; Karampelas, Ioannis; Khan, Maheen; Quadri, Nabiha; Patel, Mayur; Patel, Niel; Almajali, Mohammad; Mattei, Tobias A.; Kemp, Joanna; Coppens, Jeroen; Mercier, Philippe
• Is Part Of: Informatics in medicine unlocked. Volume 28(2022)
• Publication Date: 2022
• Subjects: Delayed cerebral ischemia -- Clinical vasospasm -- Subarachnoid hemorrhage -- Applied predictive modeling -- Applied forecasting -- Imbalanced binary classification; ANN artificial neural networks -- aSAH aneurysmal subarachnoid hemorrhage -- AUC area under the curve -- DCI delayed cerebral ischemia -- EGB extreme gradient boosting -- EVD external ventricular drain -- FS Fisher scale -- LASSO least absolute shrinkage and selection operator -- LR logistic regression -- mFS modified Fisher scale -- ML machine learning -- MICE multivariate imputation by chained equations -- PPV positive predictive value -- PR precision and recall -- RF random forests -- RELU rectified linear unit activation function (ReLU) -- ROC receiver operating characteristic -- ROSE random over-sampling examples -- TCD transcranial doppler; Medical informatics -- Periodicals; 610.285
• DOI: 10.1016/j.imu.2021.100817
• Languages: English
• ISSNs: 2352-9148

Abstract

Abstract: Introduction: Delayed Cerebral Ischemia (DCI) is an important cause of morbidity and mortality after aneurysmal Subarachnoid Hemorrhage (aSAH). Researchers have utilized various methods for predicting patients at risk for DCI progression. Methods: An eight-year retrospective review of aSAH patients who presented to St Louis University Hospital. The records were screened for demographic, clinical, and radiographic parameters. DCI was the primary outcome. We identified 16 features to fit various forecasting models and selected the best binary classifier through comprehensive machine learning (ML) workflows. Regression and ensemble tree-based algorithms were utilized, based on their performance on tabular data. We investigated whether a single model could outperform in our dataset. Due to the expected outcome class imbalance (DCI), we selected precision, recall, and F-score as threshold metrics. Precision-recall curves were used for model performance ranking. Results: Of the 213 aSAH patients analyzed, 42 progressed to DCI (19.7%). The mean age was 55.7 years. The outcome variable (DCI) was imbalanced with a class ratio of 1:4. Bivariate analysis revealed two significant associations: The "Hunt-and-Hess scale" (p-value = 0.016), and "Posthemorrhagic hydrocephalus" (p-value < 0.001). The all-relevant important factors during feature selection were: "Fisher scale, " "Modified Fisher scale, " "Hunt-and-Hess scale, " and "Posthemorrhagic hydrocephalus". "Treatment type" was tentative. The random forests model achieved a pooled accuracy of 71.1% (95%CI: 60.4, 83.4) with an F1-score of 0.484. The best binary classifier utilized extreme gradient boosting while trained on the all-relevant predictors plus "Aneurysm type." Extreme gradient boosting achieved a predictive accuracy of 84.3% (95%CI: 75.9, 93.4) with an F1-score of 0.684. We describe the challenges that arise during training of a binary classifier on imbalanced datasets, and, while going through an extensive comparison review of similar published studies, we not only demonstrate the model's performance but also identify multiple forecasting methodological fallacies in neurological research. Conclusion: By implementing baseline patient characteristics combined with radiographic grading scales, we built a simple yet robust, highly accurate—but, most importantly—useful binary classifier for DCI prediction. The model is available online, and it can be utilized clinically as an effective forecasting tool (https://georgiosalexopoulos.shinyapps.io/download/ ). Highlights: We propose the best binary classifier for early prediction of delayed cerebral ischemia in patients with ruptured intracranial aneurysms. A comprehensive machine learning workflow for delayed cerebral ischemia forecasting. Highlighting the challenges of binary classifier training on imbalanced data and identify methodological AI fallacies in neurological research.