Examining validity evidence for multidimensional forced choice measures with different scoring approaches. (1st March 2018)

• Record Type: Journal Article
• Authors: Lee, Philseok; Lee, Sunhee; Stark, Stephen
• Is Part Of: Personality and individual differences. Volume 123(2018)
• Page Start: 229
• Page End: 235
• Publication Date: 2018-03-01
• Subjects: Noncognitive construct -- Multidimensional forced choice format -- Response bias -- Scoring methods -- Item response theory; Personality -- Periodicals; Individuality -- Periodicals; Individuality -- Periodicals; Personality Development -- Periodicals; Personnalité -- Périodiques; Individualité -- Périodiques; 155.205
• DOI: 10.1016/j.paid.2017.11.031
• Languages: English
• ISSNs: 0191-8869

Abstract

Abstract: This study aims to investigate whether the Thurstonian Item Response Theory (TIRT), a recently developed normative scoring method, yields better validity evidence than the traditional partially ipsative scoring methods for multidimensional forced choice measures. For this purpose, we compared the construct- and criterion-related validity evidence of three different scoring methods for MFC measures, including 1) a partially ipsative method based on classical test theory (PI-CTT), 2) an analogous partially ipsative method using the graded item response theory (PI-IRT), and 3) the TIRT method. We also included in our analyses the validity evidence for a single-statement (SS) Big Five personality measures. Overall, the validity evidence for the three types of MFC scoring methods was comparable to the validity evidence for the traditional summative scoring (SS-CTT) method. Interestingly enough, the PI-CTT scoring method, the simplest method for MFC scoring, was about as effective as the more complex TIRT method. We discuss practical implications of the results and offer suggestions for future research. Highlights: All three partially ipsative (PI) scoring methods yielded good validity evidence. Thurstonian Item Response Theory (TIRT) also provided good validity evidence. The simplest PI scoring method was about as effective as the complex TIRT method. TIRT methods are more desirable than the PI method for testing applications.