Estimating Standardized Effect Sizes for Two- and Three-Level Partially Nested Data. (1st November 2016)

• Record Type: Journal Article
• Authors: Lai, Mark H. C.; Kwok, Oi-man
• Is Part Of: Multivariate behavioral research. Volume 51:Number 6(2016)
• Page Start: 740
• Page End: 756
• Publication Date: 2016-11-01
• Subjects: Effect size -- partially nested -- partial clustering -- multilevel; Psychometrics -- Periodicals; Psychology, Experimental -- Periodicals; Psychology, Experimental; Psychometrics; Periodicals; 150.15195
• DOI: 10.1080/00273171.2016.1231606
• Languages: English
• ISSNs: 0027-3171

Abstract

ABSTRACT: Although previous research has discussed an effect size estimator for partially nested cluster randomized designs, the existing estimator (a) is not efficient when used with primary data, (b) can be biased when the homogeneity of variance assumption is violated, and (c) has not yet been empirically evaluated for its finite sample properties. The present paper addresses these limitations by proposing an alternative maximum likelihood estimator for obtaining standardized mean difference effect size and the corresponding sampling variance for partially nested data, as well as the variants that do not make an assumption of homogeneity of variance. The typical estimator, denoted as d ( d W with pooled SD and d C with control arm SD ), requires input of summary statistics such as observed means, variances, and the intraclass correlation, and is useful for meta-analyses and secondary data analyses; the newly proposed estimator δ ^ ( δ ^ W and δ ^ C ) takes parameter estimates from a correctly specified multilevel model as input and is mainly of interest to researchers doing primary research. The simulation results showed that the two methods ( d and δ ^ ) produced unbiased point and variance estimates for effect size. As expected, in general, δ ^ was more efficient than d with unequal cluster sizes, especially with large average cluster size and large intraclass correlation. Furthermore, under heterogeneous variances, δ ^ demonstrated a greater relative efficiency with small sample size for the unclustered control arm. Real data examples, one from a youth preventive program and one from an eating disorder intervention, were used to demonstrate the methods presented. In addition, we extend the discussion to a scenario with a three-level treatment arm and an unclustered control arm, and illustrate the procedures for effect size estimation using a hypothetical example of multiple therapy groups of clients clustered within therapists.