A Note on Construct Validity of the Anchoring Method in PISA 2012. (October 2018)

• Record Type: Journal Article
• Authors: Stankov, Lazar; Lee, Jihyun; von Davier, Matthias
• Is Part Of: Journal of psychoeducational assessment. Volume 36:Number 7(2018)
• Page Start: 709
• Page End: 724
• Publication Date: 2018-10
• Subjects: anchoring vignettes -- correcting correlations -- cross-cultural difference -- response bias; Ability -- Testing -- Periodicals; Educational tests and measurements -- Periodicals; 371.713
• DOI: 10.1177/0734282917702270
• Languages: English
• ISSNs: 0734-2829

Abstract

We examine construct validity of the anchoring method used with 12 noncognitive scales from the Programme for International Student Assessment (PISA) 2012 project. This method combines individuals' responses to vignettes and self-rated scores based on Likert-type items. It has been reported that the use of anchoring vignettes can reverse country-level correlations between academic achievement scores and noncognitive measures from negative to positive, and therefore align them with the typically reported individual-level correlations. Using the PISA 2012 data, we show that construct validity of this approach may be open to question because the anchored scales produce a different set of latent dimensions in comparison with nonanchored scales, even though both scales were created from the same set of individual responses. We also demonstrate that only one of three vignettes may be responsible for the resolution of the "paradox" highlighting that the choice of vignettes may be more important than what was previously reported.