Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale. (April 2021)

• Record Type: Journal Article
• Authors: Reardon, Sean F.; Kalogrides, Demetra; Ho, Andrew D.
• Is Part Of: Journal of educational and behavioral statistics. Volume 46:Number 2(2021)
• Page Start: 138
• Page End: 167
• Publication Date: 2021-04
• Subjects: linking -- scaling -- multilevel modeling -- achievement testing -- NAEP; Educational statistics -- Periodicals; Social sciences -- Statistical methods -- Periodicals; 370.2
• DOI: 10.3102/1076998619874089
• Languages: English
• ISSNs: 1076-9986

Abstract

Linking score scales across different tests is considered speculative and fraught, even at the aggregate level. We introduce and illustrate validation methods for aggregate linkages, using the challenge of linking U.S. school district average test scores across states as a motivating example. We show that aggregate linkages can be validated both directly and indirectly under certain conditions such as when the scores for at least some target units (districts) are available on a common test (e.g., the National Assessment of Educational Progress). We introduce precision-adjusted random effects models to estimate linking error, for populations and for subpopulations, for averages and for progress over time. These models allow us to distinguish linking error from sampling variability and illustrate how linking error plays a larger role in aggregates with smaller sample sizes. Assuming that target districts generalize to the full population of districts, we can show that standard errors for district means are generally less than .2 standard deviation units, leading to reliabilities above .7 for roughly 90% of districts. We also show how sources of imprecision and linking error contribute to both within- and between-state district comparisons within versus between states. This approach is applicable whenever the essential counterfactual question—"what would means/variance/progress for the aggregate units be, had students taken the other test?"—can be answered directly for at least some of the units.