Validity Arguments for AI‐Based Automated Scores: Essay Scoring as an Illustration. Issue 3 (8th June 2022)

• Record Type: Journal Article
• Authors: Ferrara, Steve; Qunbar, Saed
• Editors: Dorsey, David; Michaels, Hillary
• Is Part Of: Journal of educational measurement. Volume 59:Issue 3(2022)
• Page Start: 288
• Page End: 313
• Publication Date: 2022-06-08
• Subjects: Educational tests and measurements -- Periodicals; 371.2605
• DOI: 10.1111/jedm.12333
• Languages: English
• ISSNs: 0022-0655

Abstract

Abstract: In this article, we argue that automated scoring engines should be transparent and construct relevant—that is, as much as is currently feasible. Many current automated scoring engines cannot achieve high degrees of scoring accuracy without allowing in some features that may not be easily explained and understood and may not be obviously and directly relevant to the target assessment construct. We address the current limitations on evidence and validity arguments for scores from automated scoring engines from the points of view of the Standards for Educational and Psychological Testing (i.e., construct relevance, construct representation, and fairness) and emerging principles in Artificial Intelligence (e.g., explainable AI, an examinee's right to explanations, and principled AI). We illustrate these concepts and arguments for automated essay scores.