Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not?. Issue 1 (2nd January 2017)

• Record Type: Journal Article
• Authors: Rios, Joseph A.; Guo, Hongwen; Mao, Liyang; Liu, Ou Lydia
• Is Part Of: International journal of testing. Volume 17:Issue 1(2017)
• Page Start: 74
• Page End: 104
• Publication Date: 2017-01-02
• Subjects: test taking -- motivation -- careless responding -- rapid guessing -- low-stakes testing -- validity; Psychological tests -- Periodicals; Educational tests and measurements -- Periodicals; 150.28705
• DOI: 10.1080/15305058.2016.1231193
• Languages: English
• ISSNs: 1530-5058

Abstract

Abstract : When examinees' test-taking motivation is questionable, practitioners must determine whether careless responding is of practical concern and if so, decide on the best approach to filter such responses. As there has been insufficient research on these topics, the objectives of this study were to: a) evaluate the degree of underestimation in the true mean when careless responses are present, and b) compare the effectiveness of two filtering procedures in purifying biased aggregated-scores. Results demonstrated that: a) the true mean was underestimated by around 0.20 SD s if the total amount of careless responses exceeded 6.25%, 12.5%, and 12.5% for easy, moderately difficult, and difficult tests, respectively, and b) listwise deleting data from unmotivated examinees artificially inflated the true mean by as much as .42 SD s when ability was related to careless responding. Findings from this study have implications for when and how practitioners should handle careless responses for group-based low-stakes assessments.