Agreement of Teachers on Evaluating Assessments of Learning Progressions in English Language Arts and Mathematics. Issue 1 (12th April 2018)

• Record Type: Journal Article
• Authors: van Rijn, Peter; Graf, Edith Aurora; Arieli‐Attali, Meirav; Song, Yi
• Is Part Of: ETS research report series. Issue 1(2018)
• Page Start: 1
• Page End: 25
• Publication Date: 2018-04-12
• Subjects: Learning progressions -- summative assessment -- mathematics -- ELA -- rater agreement -- teachers; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12199
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract: In this study, we explored the extent to which teachers agree on the ordering and separation of levels of two different learning progressions (LPs) in English language arts (ELA) and mathematics. In a panel meeting akin to a standard‐setting procedure, we asked teachers to link the items and responses of summative educational assessments to LP levels. We evaluated four types of agreement among teachers, with test developers, and with trained raters. Although the results were quite mixed, agreement on the LP levels of mathematics items was generally better than that for ELA tasks. The agreement on the LP levels of ELA sample responses was generally better than that for mathematics. Implications of the results are discussed. Abstract : Report Number: ETS RR–18‐11