Reliability of assessment of medical students' non-technical skills using a behavioural marker system: does clinical experience matter?. (29th September 2020)

• Record Type: Journal Article
• Authors: Clarke, Benjamin; Smith, Samantha E; Phillips, Emma Claire; Hamilton, Ailsa; Kerins, Joanne; Tallentire, Victoria R
• Is Part Of: BMJ simulation & technology enhanced learning. Volume 7:Number 5(2021)
• Page Start: 285
• Page End: 292
• Publication Date: 2020-09-29
• Subjects: Non-Technical Skills -- Medical Student -- Simulation Based Education -- Reliability; Medicine -- Simulation methods -- Periodicals; Medical innovations -- Periodicals; 610.113
• DOI: 10.1136/bmjstel-2020-000705
• Languages: English
• ISSNs: 2056-6697

Abstract

Abstract : Introduction: Non-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students' Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students' non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience. Methods: Forty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters' levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC). Results: Eleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65. Conclusions: Using 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.