A CALL FOR CAUTIOUS INTERPRETATION OF META-ANALYTIC REVIEWS. Issue 1 (27th March 2021)

• Record Type: Journal Article
• Authors: Boers, Frank; Bryfonski, Lara; Faez, Farahnaz; McKay, Todd
• Is Part Of: Studies in second language acquisition. Volume 43:Issue 1(2021)
• Page Start: 2
• Page End: 24
• Publication Date: 2021-03-27
• Subjects: Second language acquisition -- Periodicals; Language and languages -- Study and teaching -- Periodicals; 401.9
• DOI: 10.1017/S0272263120000327
• Languages: English
• ISSNs: 0272-2631

Abstract

Abstract: Meta-analytic reviews collect available empirical studies on a specified domain and calculate the average effect of a factor. Educators as well as researchers exploring a new domain of inquiry may rely on the conclusions from meta-analytic reviews rather than reading multiple primary studies. This article calls for caution in this regard because the outcome of a meta-analysis is determined by how effect sizes are calculated, how factors are defined, and how studies are selected for inclusion. Three recently published meta-analyses are reexamined to illustrate these issues. The first illustrates the risk of conflating effect sizes from studies with different design features; the second illustrates problems with delineating the variable of interest, with implications for cause-effect relations; and the third illustrates the challenge of determining the eligibility of candidate studies. Replication attempts yield outcomes that differ from the three original meta-analyses, suggesting also that conclusions drawn from meta-analyses need to be interpreted cautiously.