Pooling Interactions Into Error Terms in Multisite Experiments. (December 2022)

• Record Type: Journal Article
• Authors: Chan, Wendy; Hedges, Larry Vernon
• Is Part Of: Journal of educational and behavioral statistics. Volume 47:Number 6(2022)
• Page Start: 639
• Page End: 665
• Publication Date: 2022-12
• Subjects: multisite experiments -- randomized block designs -- pooling -- ANOVA -- interactions; Educational statistics -- Periodicals; Social sciences -- Statistical methods -- Periodicals; 370.2
• DOI: 10.3102/10769986221104800
• Languages: English
• ISSNs: 1076-9986

Abstract

Multisite field experiments using the (generalized) randomized block design that assign treatments to individuals within sites are common in education and the social sciences. Under this design, there are two possible estimands of interest and they differ based on whether sites or blocks have fixed or random effects. When the average treatment effect is assumed to be identical across sites, it is common to omit site by treatment interactions and "pool" them into the error term in classical experimental design. However, prior work has not addressed the consequences of pooling when site by treatment interactions are not zero. This study assesses the impact of pooling on inference in the presence of nonzero site by treatment interactions. We derive the small sample distributions of the test statistics for treatment effects under pooling and illustrate the impacts on rejection rates when interactions are not zero. We use the results to offer recommendations to researchers conducting studies based on the multisite design.