Multilevel Mixture Modeling with Propensity Score Weights for Quasi-Experimental Evaluation of Virtual Learning Environments. Issue 6 (2nd November 2021)

• Record Type: Journal Article
• Authors: Leite, Walter L.; Jing, Zeyuan; Kuang, Huan; Kim, Dongho; Huggins-Manley, A. Corinne
• Is Part Of: Structural equation modeling. Volume 28:Issue 6(2021)
• Page Start: 964
• Page End: 982
• Publication Date: 2021-11-02
• Subjects: Causal inference -- program evaluation -- latent class analysis -- multilevel mixture modeling -- virtual learning environments; Multivariate analysis -- Periodicals; Social sciences -- Statistical methods -- Periodicals; 519.535
• DOI: 10.1080/10705511.2021.1919895
• Languages: English
• ISSNs: 1070-5511

Abstract

ABSTRACT: With the growing use of virtual learning environments (VLE), innovative methods to evaluate their performance are increasingly needed. A key difficulty in evaluating VLE using system logs is the large heterogeneity of usage patterns. The current study demonstrates an approach to classify complex patterns of student-level and classroom-level usage with latent class analysis, then estimate average treatment effects (ATEs) of membership in student or classroom classes, as well as joint effects. The approach accounts for uncertainty of latent classes with a three-step method, and nonrandom selection into classes using inverse probability weighting. We demonstrate the approach with an analysis of usage of an Algebra VLE and estimate causal effects of latent class membership on a high-stakes Algebra I standardized assessment. Challenges of using system logs for evaluation of VLE are discussed with respect to measurement error, construct validity, latent classes enumeration, and comparison of classes with respect to distal outcomes.