A generalized‐weights solution to sample overlap in meta‐analysis. (18th September 2020)

• Record Type: Journal Article
• Authors: Bom, Pedro R. D.; Rachinger, Heiko
• Is Part Of: Research synthesis methods. Volume 11:Number 6(2020)
• Page Start: 812
• Page End: 832
• Publication Date: 2020-09-18
• Subjects: generalized weights -- inference -- meta‐analysis -- sample overlap; Research -- Methodology -- Periodicals; Research -- Statistical methods -- Periodicals; 507.2
• DOI: 10.1002/jrsm.1441
• Languages: English
• ISSNs: 1759-2879

Abstract

Abstract : Meta‐studies are often conducted on empirical findings obtained from overlapping samples. Sample overlap is common in research fields that strongly rely on aggregated observational data (eg, economics and finance), where the same set of data may be used in several studies. More generally, sample overlap tends to occur whenever multiple estimates are sampled from the same study. We show analytically how failing to account for sample overlap causes high rates of false positives, especially for large meta‐sample sizes. We propose a generalized‐weights (GW) meta‐estimator, which solves the sample overlap problem by explicitly modeling the variance‐covariance matrix that describes the structure of dependence among estimates. We show how this matrix can be constructed from information that is usually available from basic sample descriptions in the primary studies (ie, sample sizes and number of overlapping observations). The GW meta‐estimator amounts to weighting each empirical outcome according to its share of independent sampling information. We use Monte Carlo simulations to (a) demonstrate how the GW meta‐estimator brings the rate of false positives to its nominal level, and (b) quantify the efficiency gains of the GW meta‐estimator relative to standard meta‐estimators. The GW meta‐estimator is fairly straightforward to implement and can solve any case of sample overlap, within or between studies. Highlights: Meta‐analyses are often conducted on empirical outcomes based on samples containing common observations. Sample overlap induces a correlation structure among empirical outcomes that harms the statistical properties of meta‐analysis methods. We derive the analytic conditions under which sample overlap causes conventional meta‐estimators to exhibit high rates of false positives. We propose a generalized‐weights (GW) solution to sample overlap, which involves approximating the variance‐covariance matrix that describes the correlation structure between outcomes; we show how to construct this matrix from information typically reported in the primary studies. We conduct Monte Carlo simulations to quantify the efficiency gains of the proposed GW estimator and show how it brings the rate of false positives near its nominal level. Although we focus on meta‐analyses of regression coefficients, our approach can, in principle, be modified and extended to effect sizes more commonly used in other research fields, such as Cohen's d or odds ratios.