Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach: A Simulation Study. (October 2013)

• Record Type: Journal Article
• Authors: López-López, José Antonio; Botella, Juan; Sánchez-Meca, Julio; Marín-Martínez, Fulgencio
• Is Part Of: Journal of educational and behavioral statistics. Volume 38:Number 5(2013)
• Page Start: 443
• Page End: 469
• Publication Date: 2013-10
• Subjects: reliability -- reliability generalization -- meta-analysis -- coefficient alpha -- mixed-effects meta-regression; Educational statistics -- Periodicals; Social sciences -- Statistical methods -- Periodicals; 370.2
• DOI: 10.3102/1076998612466142
• Languages: English
• ISSNs: 1076-9986

Abstract

Since heterogeneity between reliability coefficients is usually found in reliability generalization studies, moderator analyses constitute a crucial step for that meta-analytic approach. In this study, different procedures for conducting mixed-effects meta-regression analyses were compared. Specifically, four transformation methods for the reliability coefficients, two estimators of the residual between-studies variance, and two methods for testing regression coefficients significance were combined in a Monte Carlo simulation study. The different methods were compared in terms of bias and mean square error (MSE) of the slope estimates, and Type I error and statistical power rates for the slope statistical tests. The results of the simulation study did not vary as a function of the residual variance estimator. All transformation methods provided negatively biased estimates, but both bias and MSE were reasonably small in all cases. In contrast, important differences were found regarding statistical tests, with the method proposed by Knapp and Hartung showing a better adjustment to the nominal significance level and higher power rates than the standard method.