Developing a Machine‐Supported Coding System for Constructed‐Response Items in PISA. Issue 1 (12th October 2017)

• Record Type: Journal Article
• Authors: Yamamoto, Kentaro; He, Qiwei; Shin, Hyo Jeong; von Davier, Matthias
• Is Part Of: ETS research report series. Issue 1(2017)
• Page Start: 1
• Page End: 15
• Publication Date: 2017-10-12
• Subjects: Machine‐supported coding -- constructed‐response items -- human coding -- large‐scale assessments -- PISA; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12169
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract : Approximately a third of the Programme for International Student Assessment (PISA) items in the core domains (math, reading, and science) are constructed‐response items and require human coding (scoring). This process is time‐consuming, expensive, and prone to error as often (a) humans code inconsistently, and (b) coding reliability in international large‐scale assessments varies across items and countries. The shift in PISA 2015 from paper‐ to computer‐based assessment digitized all responses and associated coding, thus providing opportunities to introduce technology and analytical methods to improve data processing and analyses in future cycles. In this computer‐based setting, this research explains the framework and approach for improving the accuracy and efficiency of the coding process in constructed‐response items for future PISA cycles. In the pilot study associated with this research, we demonstrate how the proposed machine‐supported coding system can be utilized using sample reading items from the PISA 2015 cycle. Abstract : Report Number: ETS RR‐17–47