Examining the Reliability of Scores From a Performance Assessment of Practice-Based Competencies. (December 2019)

• Record Type: Journal Article
• Authors: Roduta Roberts, Mary; Alves, Cecilia Brito; Werther, Karin; Bahry, Louise M.
• Is Part Of: Journal of psychoeducational assessment. Volume 37:Number 8(2019)
• Page Start: 973
• Page End: 988
• Publication Date: 2019-12
• Subjects: rater reliability -- rater effects -- generalizability analysis -- analytic scores -- holistic scores; Ability -- Testing -- Periodicals; Educational tests and measurements -- Periodicals; 371.713
• DOI: 10.1177/0734282918816849
• Languages: English
• ISSNs: 0734-2829

Abstract

The purpose of this study was to examine the reliability and sources of score variation from a performance assessment of practice competencies within an occupational therapy program. Data from 99 students who participated in a practical exam were examined. A generalizability analysis of analytic, total, and overall holistic scores was completed accounting for the nested design. The results demonstrated that rater pairs produced highly reliable (i.e., G > 0.93, ϕ > 0.90) overall holistic, total, and analytic scores. In all scoring scenarios, student variation accounted for the largest percentage of total variance with less variation attributable to raters. However, the interaction-error term accounted for the second most variation within the system. Findings from this study suggest possible differences in rater behavior raising questions for future research around the processes raters use to generate holistic versus analytic scores.