OC-021 Changes in scoring of direct observation of procedural skills (dops) forms in endoscopy training and their impact on competence assessment. (17th June 2017)

• Record Type: Journal Article
• Authors: Siau, K; Dunckley, P; Anderson, J; Beales, I; Broughton, R; Feeney, M; Hawkes, N; McKaig, B; Thomas-Gibson, S; Valori, R; Wells, C; Johnson, G; Haycock, A
• Is Part Of: Gut. Volume 66(2017)Supplement 2
• Page Start: A11
• Page End: A11
• Publication Date: 2017-06-17
• Subjects: Assessment -- certification -- DOPS -- endoscopy training; Gastroenterology -- Periodicals; 616.33
• DOI: 10.1136/gutjnl-2017-314472.21
• Languages: English
• ISSNs: 0017-5749

Abstract

Abstract : Introduction: DOPS are validated tools for assessing competence in endoscopy. Previously, DOPS were scored on a 4-point competence-based scale, with scores of 3 and 4 signifying competence. In July 2016, the DOPS rating scale changed to a supervision-based scale that has been shown to be more reliable, [1] with 4 ratings from maximal supervision, up to competent without supervision. We aimed to assess whether changes to the rating scale have affected distribution of scores and hence demonstrate validity. Method: Scores were collected from DOPS for gastroscopy (1934), sigmoidoscopy (517), colonoscopy (2296) and polypectomy (370) in the 6 months before July 2016 (old DOPS) and after (new DOPS), for trainees at early stages of training (total procedures<100). To allow analysis, the new DOPS rating scales was aligned to a 4-point scale, hence a score of 4 on new DOPS = Scores 3 or 4 on old DOPS, and scores on the new and old DOPS compared using the Mann-Whitney U-test. Results: 5117 DOPS (77.7% new and 22.3% old) were included for analysis. Overall, there were variations in distributions of scores (p<0.001) between forms ( Graph ). Compared to new DOPS, scores of 1 were underutilised on old DOPS (0.6% vs 3.0%, p<0.001). Frequencies of low scores (pooled scores of 1 and 2) were similar for gastroscopy (p=0.53) and sigmoidoscopy (p=0.34), but not for colonoscopy (new 11.9% vs. old 13.9%, p<0.001) and polypectomy (new 6.8% vs.19.9%, p<0.001). Trainees on old DOPS were more likely to be rated as competent (score 3 or 4) compared to new DOPS (86.4% vs. 55.8%, p<0.001). On subgroup analysis, this was evident for gastroscopy (86.3% vs. 49.1%, p<0.001), colonoscopy (86.1% vs. 58.2%, p<0.001), sigmoidoscopy (90.6% vs. 62.0%, p<0.001), but not polypectomy (80.1% vs. 67.9%, p=0.12). Conclusion: Endoscopy assessors are applying a greater range of scores using a new DOPS rating scale based on degree of supervision, in two cohorts of trainees matched for experience. This indicates better construct validity with the new rating scale. Further work is underway to determine the reliability of the new DOPS to inform summative assessment and certification for UK endoscopy trainees. Reference: . Crossley Jet al. Med Educ2011Jun;45(6):560–9. Disclosure of Interest: None Declared