P105 Validation of a machine learning model in a prospective cohort: the RISQ study. (19th June 2022)

• Record Type: Journal Article
• Authors: Alexander Ho, Kai Man; Rosenfeld, Avi; Hogan, Áine; McBain, Hazel; Duku, Margaret; Wolfson, Paul; Hennelly, Laura; Macabodbod, Lester; Lovat, Laurence
• Is Part Of: Gut. Volume 71(2022)Supplement 1
• Page Start: A92
• Page End: A92
• Publication Date: 2022-06-19
• Subjects: Gastroenterology -- Periodicals; 616.33
• DOI: 10.1136/gutjnl-2022-BSG.162
• Languages: English
• ISSNs: 0017-5749

Abstract

Abstract : Introduction: Previously our group had identified 20 features which were associated with the development of upper gastrointestinal (UGI) cancers using a machine learning approach.[1] We sought to refine this model and to validate this in an independent dataset to assess its generalisability in an interim analysis. Methods: We selected patients who were recruited for the multicentre Saliva to Predict rIsk of disease using Transcriptomics and epigenetics (SPIT) study to develop our model. Patients were recruited from 2-week wait suspected UGI pathways and additionally enriched with patients with confirmed oesophageal adenocarcinoma admitted as inpatients. We used regularised logistic regression (glmnet) from the caret package in R software to create the model. 60% of the data with 10-fold cross validation was used for training, with the remaining 40% for testing. For validation, we used data from the predicting RIsk of disease uSing detailed Questionnaires (RISQ) study, an ongoing prospective multicentre study using the questionnaire based on the our previous work. 1 We evaluated the model using area under the receiver operating characteristic curve (AUC). Results: We included 93 cancer and 715 non-cancer patients for training and testing and 21 cancer and 203 non-cancer patients for validation. We further reduced the model to 18 features without significant detriment to model performance. In the training and testing data AUC was 0.86 (95%CI: 0.81-0.91) and 0.75 (95%CI: 0.67-0.83) respectively. We set a threshold of 0.03 as a cut off based on a cost function where false negatives had a 50-time greater impact than false positive cases ( figure 1 ). For the validation cohort we achieved an AUC of 0.95 (95%CI: 0.90-1.00). This equated to a sensitivity 0.952 and a specificity of 0.897 for detecting cancer. Conclusions: Initial results from our model compare favourably with the Edinburgh Dysphagia Scale, which has a sensitivity and specificity of 0.984 and 0.093 respectively. 2 It also appears to have a high specificity, potentially helping to reduce unnecessary endoscopies. We aim to further increase the size of the validation cohort to ensure its robustness and generalisability. Our model could be applied to triaging and prioritising endoscopic referral backlogs as a result of COVID-19. 3 References: Ho KMA, et al. Gut 2021;70 :A136. Murray IA, et al. World J Gastroenterol 2012;18 :4357–62. Ho KMA, et al. Lancet Gastroenterol Hepatol 2021;6 :381–90.