Examining rater accuracy and consistency with a special education observation protocol. (March 2020)

• Record Type: Journal Article
• Authors: Johnson, Evelyn S.; Zheng, Yuzhu; Crawford, Angela R.; Moylan, Laura A.
• Is Part Of: Studies in educational evaluation. Volume 64(2020)
• Publication Date: 2020-03
• Subjects: Rater accuracy -- Teacher observation -- Rater consistency -- Feedback -- Special education; Curriculum evaluation -- Periodicals; Educational tests and measurements -- Periodicals; Programmes d'études -- Evaluation -- Périodiques; Tests et mesures en éducation -- Périodiques; 375.006
• DOI: 10.1016/j.stueduc.2019.100827
• Languages: English
• ISSNs: 0191-491X

Abstract

Highlights: Special education teachers do not receive training in math content pedagogy. Observation tools aligned with EBPs offer one way to improve practice. Rater feedback must be accurate and consistent to improve teacher practice. Abstract: Research indicates that instructional aspects of teacher performance are the most difficult to reach consensus on, significantly limiting teacher observation as a way to systematically improve instructional practice. Understanding the rationales that raters provide as they evaluate teacher performance with an observation protocol offers one way to better understand the training efforts required to improve rater accuracy. The purpose of this study was to examine the accuracy of raters evaluating special education teachers' implementation of evidence-based math instruction. A mixed-methods approach was used to investigate: 1) the consistency of the raters' application of the scoring criteria to evaluate teachers' lessons, 2) raters' accuracy on two lessons with those given by expert-raters, and 3) the raters' understanding and application of the scoring criteria through a think-aloud process. The results show that raters had difficulty understanding some of the high inference items in the rubric and applying them accurately and consistently across the lessons. Implications for rater training are discussed.