Model meets reality: Validating a new behavioral measure for test-taking effort. Issue 2 (3rd April 2021)

• Record Type: Journal Article
• Authors: Ulitzsch, Esther; Penk, Christiane; von Davier, Matthias; Pohl, Steffi
• Is Part Of: Educational assessment. Volume 26:Issue 2(2021)
• Page Start: 104
• Page End: 124
• Publication Date: 2021-04-03
• Subjects: Educational tests and measurements -- United States -- Periodicals; 371.26097305
• DOI: 10.1080/10627197.2020.1858786
• Languages: English
• ISSNs: 1062-7197

Abstract

ABSTRACT: Identifying and considering test-taking effort is of utmost importance for drawing valid inferences on examinee competency in low-stakes tests. Different approaches exist for doing so. The speed-accuracy+engagement model aims at identifying non-effortful test-taking behavior in terms of nonresponse and rapid guessing based on responses and response times. The model allows for identifying rapid-guessing behavior on the item-by-examinee level whilst jointly modeling the processes underlying rapid guessing and effortful responding. To assess whether the model indeed provides a valid measure of test-taking effort, we investigate (1) convergent validity with previously developed behavioral as well as self-report measures on guessing behavior and effort, (2) fit within the nomological network of test-taking motivation derived from expectancy-value theory, and (3) ability to detect differences between groups that can be assumed to differ in test-taking effort. Results suggest that the model captures central aspects of non-effortful test-taking behavior. While it does not cover the whole spectrum of non-effortful test-taking behavior, it provides a measure for some aspects of it, in a manner that is less subjective than self-reports. The article concludes with a discussion of implications for the development of behavioral measures of non-effortful test-taking behavior.