Random Permutation Testing Applied to Measurement Invariance Testing with Ordered-Categorical Indicators. Issue 4 (4th July 2018) - Kite, Benjamin A.; Jorgensen, Terrence D.; Chen, Po-Yi
Abstract : We describe and evaluate a random permutation test of measurement invariance with ordered-categorical data. To calculate a p -value for the observed (∆) χ 2, an empirical reference distribution is built by repeatedly shuffling the grouping variable, then saving the χ 2 from a configural model, or the ∆ χ 2 between configural and scalar-invariance models, fitted to each permuted dataset. The current gold standard in this context is a robust mean- and variance-adjusted ∆ χ 2 test proposed by Satorra (2000), which yields inflated Type I errors, particularly when thresholds are asymmetric, unless samples sizes are quite large (Bandalos, 2014; Sass et al., 2014). In a Monte Carlo simulation, we compare permutation to three implementations of Satorra's robust χ 2 across a variety of conditions evaluating configural and scalar invariance. Results suggest permutation can better control Type I error rates while providing comparable power under conditions that the standard robust test yields inflated errors.