Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments. Issue 7 (3rd September 2012)

• Record Type: Journal Article
• Authors: Gallagher, Anthony G.; O'Sullivan, Gerald C.; Leonard, Gerald; Bunting, Brendan P.; McGlade, Kieran J.
• Is Part Of: ANZ journal of surgery. Volume 84:Issue 7/8(2014)
• Page Start: 568
• Page End: 573
• Publication Date: 2012-09-03
• Subjects: Surgery -- Periodicals; 617.005
• DOI: 10.1111/j.1445-2197.2012.06236.x
• Languages: English
• ISSNs: 1445-1433

Abstract

<abstract abstract-type="main"> <title>Abstract</title> <sec id="ans6236-sec-0001" sec-type="section"> <title>Background</title> <p>The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand.</p> </sec> <sec id="ans6236-sec-0002" sec-type="section"> <title>Methods</title> <p>We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure‐specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons.</p> </sec> <sec id="ans6236-sec-0003" sec-type="section"> <title>Results</title> <p>The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (<italic>P</italic> < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments.</p> </sec> <sec id="ans6236-sec-0004" sec-type="section"> <title>Discussion</title> <p>We propose binary procedure‐based assessment checklists as more reliable assessment instruments with more robust reproducibility.</p> </sec> </abstract>