PWE-107 Direct observation of procedural skills (dops) assessment in gastroscopy: validity and competency development during training. Issue 2 (June 2019)

• Record Type: Journal Article
• Authors: Siau, Keith; Crossley, Prof James; Dunckley, Paul; Johnson, Gavin; Feeney, Mark; Hawkes, Neil; Beales, Ian
• Is Part Of: Gut. Volume 68:Issue 2(2019)
• Page Start: A251
• Page End: A252
• Publication Date: 2019-06
• Subjects: Gastroenterology -- Periodicals; 616.33
• DOI: 10.1136/gutjnl-2019-BSGAbstracts.478
• Languages: English
• ISSNs: 0017-5749

Abstract

Abstract : Introduction: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of a novel formative direct observation of procedural skills (DOPS) assessment tool in gastroscopy, and study competency development using DOPS. Methods: This was a prospective multicentre analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. Results: 10086 DOPS were submitted for 987 trainees from 275 UK training centres. Exploratory factor analysis identified three distinct item groupings: representing 'pre-procedure', 'technical' and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included: trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of 3 assessments from 3 assessors was sufficient to achieve the acceptable reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of: 'pre-procedure' (10–24 procedures), 'technical' (15–74 procedures), 'post-procedure non-technical' skills (20–24 procedures), and overall competency (22–49 procedures) [Abstract PWE-107 Figure 1]. Higher lifetime procedure count (P<0.001), DOPS count (P<0.001), trainee specialty (P=0.028) and assessor specialty (P=0.002), trainee seniority (P=0.011) and lower case difficulty (P<0.001) were multivariable predictors of DOPS competence. Conclusion: This study establishes milestones for competency acquisition during gastroscopy training and provides novel validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.