AUC Undermines AD Biomarker Discovery. (20th December 2022)

• Record Type: Journal Article
• Authors: Smith, Kenneth; Climer, Sharlee
• Is Part Of: Alzheimer's & dementia. Volume 18(2022)Supplement 5
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-12-20
• Subjects: Alzheimer's disease -- Periodicals; Alzheimer Disease -- Periodicals; Dementia -- Periodicals; Démence; Maladie d'Alzheimer; Périodique électronique (Descripteur de forme); Ressource Internet (Descripteur de forme); 616.83
• DOI: 10.1002/alz.067698
• Languages: English
• ISSNs: 1552-5260

Abstract

Abstract: Background: Efforts are underway to accelerate precision medicine for late‐onset Alzheimer disease (AD). The realization of this endeavor can only be attained by identifying AD subtypes and developing methods to diagnose and treat each subtype. Liquid biopsies offer great potential for mounting this challenge. Identification of promising biomarkers requires evaluation of its ability to diagnose AD, and the area under the receiver operating characteristic curve (AUC) is commonly used for this purpose. This metric is equally weighted on the true positive rate (TPR) and false positive rate (FPR). When a given biomarker represents a single subtype, we expect the TPR to be relatively low as only a subset of AD cases are associated. Consequently, we hypothesized that screening based on AUC may discard valuable subtype biomarkers. Method: An in silico experiment was conducted in which an 'ideal' biomarker for a subtype was modeled and the corresponding AUC was computed. A pseudo biomarker was designed to perfectly identify 20% of the cases and all other individuals were assigned levels near zero. Biomarker levels were generated to create a normal distribution centered on 0, with a standard deviation of 0.01, to represent 1000 controls and 800 cases, and 200 additional cases were represented by a normal distribution centered on 1, with a standard deviation of 0.2, to represent the subtype cases (Figs. 1 and 2 ). The experiment was conducted 1000 times. Result: The mean AUC across the 1000 trials was 0.6001, with a maximum of 0.6292 and minimum of 0.5624. The standard deviation for the AUC values was 0.0109. The AUC plot for one of the trials is shown in Fig. 3 . Conclusion: The pseudo biomarker perfectly identified 20% of the cases while having FPR = 0, yet it would likely be discarded due to the low AUC value, indicating that AUC is not a suitable metric for evaluating biomarkers for AD subtypes. Alternative evaluation metrics should be considered while traversing the path towards successful precision medicine for AD.