Interassessor agreement of portfolio-based competency assessment for orthotists/prosthetists in Australia: a mixed method study. Issue 3 (17th June 2021)

• Record Type: Journal Article
• Authors: Dillon, Michael P.; Puli, Louise; Ridgewell, Emily; Anderson, Sarah P; Chiavaroli, Neville; Clarke, Leigh
• Is Part Of: Prosthetics and orthotics international. Volume 45:Issue 3(2021)
• Page Start: 276
• Page End: 288
• Publication Date: 2021-06-17
• Subjects: competency -- portfolio -- standards -- orthotist -- prosthetist -- reliability -- agreement -- professional regulation; Orthopedic apparatus -- Periodicals; Prosthesis -- Periodicals; Implants, Artificial -- Periodicals; 617.905
• DOI: 10.1097/PXR.0000000000000022
• Languages: English
• ISSNs: 0309-3646

Abstract

Abstract : Background: Internationally qualified orthotists/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve the processes. Objectives: To quantify interassessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association's Entry Level Competency Standards and where there was significant disagreement between assessors, to explore the reasons why. Study design: Mixed methods: explanatory sequential. Method: Fifteen portfolios were assigned to independent assessors. Assessors determined whether the evidence presented met the requirements of each performance indicator. Interassessor agreement was calculated using Gwet's Agreement Coefficient 1 (AC1), and these data informed semistructured interviews to explore the reasons for disagreement. Results: Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71), which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). Interviews with assessors suggested that disagreement could be attributed to the complexity of some performance indicators, unconscious bias, and the appropriateness of the evidence presented. Conclusions: Although most performance indicators in Australian Orthotic Prosthetic Association's Entry Level Competency Standard were associated with moderate to substantial interassessor agreement, there are opportunities to improve agreement by simplifying the wording of some performance indicators and revising guidelines to help applicants curate the most appropriate evidence for each performance indicator.