Assessing the assessments: Taking stock of learning outcomes data in India. (July 2021)

• Record Type: Journal Article
• Authors: Johnson, Doug; Parrado, Andres
• Is Part Of: International journal of educational development. Volume 84(2021)
• Publication Date: 2021-07
• Subjects: Learning outcome data -- ASER -- NAS -- Variance decomposition; Education -- Periodicals; Education -- Social aspects -- Periodicals; Education -- Economic aspects -- Periodicals; Education -- Periodicals; Éducation -- Périodiques; Education; Periodicals; 370.11
• DOI: 10.1016/j.ijedudev.2021.102409
• Languages: English
• ISSNs: 0738-0593

Abstract

Highlights: We assess the reliability of India's two main sources of representative data on learning outcomes: the ASER and NAS tests. We find that NAS state averages are unrealistically high and contain little information about relative state performance. ASER data is reliable for comparing state averages but less so for looking at changes in state averages or district averages. Analysts should refrain from using NAS to make comparisons across districts/states or assess improvements over time. Abstract: Data on learning outcomes is essential for tracking progress in achieving education goals, understanding what education policies work (and don't work), and holding public officials accountable. We assess the accuracy and reliability of India's two nationally representative surveys on learning outcomes, ASER and NAS, so that users of these datasets may better understand when, and for what purposes, these two datasets can reasonably be used. After restricting our sample to maximize comparability between the two datasets, we find that NAS state averages are significantly higher than ASER state averages and averages from an independently conducted and nationally representative survey (IHDS). In addition, state rankings based on NAS data display almost no correlation with state rankings based on ASER, IHDS, or net state domestic product per capita. We conclude that NAS state averages are likely artificially high and contain little information about states' relative performance. The presence of severe bias in the NAS data suggests that this data should be used carefully or not at all for comparisons between states, constructing learning profiles, or any other purpose. We then analyze the internal reliability of ASER data using variance decomposition methods. We find that while ASER data is mostly reliable for comparing state averages, it is less reliable for looking at district averages, or changes in district and state averages over time. We conclude that analysts may use ASER data with confidence for comparisons between states in a single year, constructing learning profiles, and assessing learning inequality but should exercise caution when comparing changes in state scores and avoid using ASER district-level data.