Examining the Effects of Narrative Commentary on Evaluators' Summative Assessments of Resident Performance. Issue 3 (September 2020)

• Record Type: Journal Article
• Authors: Lefebvre, Cedric; Hiestand, Brian; Glass, Casey; Masneri, David; Hosmer, Kathleen; Hunt, Meagan; Hartman, Nicholas
• Is Part Of: Evaluation & the health professions. Volume 43:Issue 3(2020)
• Page Start: 159
• Page End: 161
• Publication Date: 2020-09
• Subjects: evaluation -- narrative -- comments -- milestones -- performance assessment; Medical care -- Evaluation -- Periodicals; Medicine -- Ability testing -- Periodicals; 362.105
• DOI: 10.1177/0163278718820415
• Languages: English
• ISSNs: 0163-2787

Abstract

Anchor-based, end-of-shift ratings are commonly used to conduct performance assessments of resident physicians. These performance evaluations often include narrative assessments, such as solicited or "free-text" commentary. Although narrative commentary can help to create a more detailed and specific assessment of performance, there are limited data describing the effects of narrative commentary on the global assessment process. This single-group, observational study examined the effect of narrative comments on global performance assessments. A subgroup of the clinical competency committee, blinded to resident identity, assigned a single, consensus-based performance score (1–6) to each resident based solely on end-of-shift milestone scores. De-identified narrative comments from end-of-shift evaluations were then included and the process was repeated. We compared milestone-only scores to milestone plus narrative commentary scores using a nonparametric sign test. During the study period, 953 end-of-shift evaluations were submitted on 41 residents. Of these, 535 evaluations included free-text narrative comments. In 17 of the 41 observations, performance scores changed after the addition of narrative comments. In two cases, scores decreased with the addition of free-text commentary. In 15 cases, scores increased. The frequency of net positive change was significant ( p = .0023). The addition of narrative commentary to anchor-based ratings significantly influenced the global performance assessment of Emergency Medicine residents by a committee of educators. Descriptive commentary collected at the end of shift may inform more meaningful appraisal of a resident's progress in a milestone-based paradigm. The authors recommend clinical training programs collect unstructured narrative impressions of residents' performance from supervising faculty.