How Consistent Is Competent? Examining Variance in Psychomotor Skills Assessment. (May 2020)

• Record Type: Journal Article
• Authors: Labbé, Mathilde; Young, Meredith; Mascarella, Marco; Husein, Murad; Doyle, Philip C.; Nguyen, Lily H.P.
• Is Part Of: Academic medicine. Volume 95:Number 5(2020)
• Publication Date: 2020-05
• Subjects: Medical education -- Periodicals; Medical policy -- Periodicals; Medical personnel -- Periodicals; Periodicals; 610.711
• DOI: 10.1097/ACM.0000000000002985
• Languages: English
• ISSNs: 1040-2446

Abstract

Abstract : Purpose: Direct assessment of trainee performance across time is a core tenet of competency-based medical education. Unlike variability of psychomotor skills across levels of expertise, performance variability exhibited by a particular trainee across time remains unexplored. The goal of this study was to document the consistency of individual surgeons' technical skill performance. Method: A secondary analysis of assessment data (collected in 2010–2012, originally published in 2015) generated by a prospective cohort of participants at Montreal Children's Hospital with differing levels of expertise was conducted in 2017. Trained raters scored blinded recordings of a myringotomy and tube insertion performed 4 times by junior and senior residents and attending surgeons over a 6-month period using a previously reported assessment tool. Descriptive exploratory analyses and univariate comparison of standard deviations (SDs) were conducted to document variability within individuals across time and across training levels. Results: Thirty-six assessments from 9 participants were analyzed. The SD of scores for junior residents was highly variable (5.8 out of a scale of 30 compared with 1.8 for both senior residents and attendings [ F (2, 19) = 5.68, P < 0.05]). For a given individual, the range of scores was twice as large for junior residents than for senior residents and attendings. Conclusions: Surgical residents may display highly variable performances across time, and individual variability appears to decrease with increasing expertise. Operative skill variability could be underrepresented in direct observation assessment; emphasis on an adequate amount of repetitive evaluations for junior residents may be needed to support judgments of competence or entrustment.