For example, in this paper Eva Vivalt, using a sample of impact evaluations, regresses effect size on variables like number of studies and sample size. As one would expect, the larger the sample size, the smaller the estimated effect size. I always wondered if you could just use the regression coefficients she presents to estimate how much an effect size would be expected to shrink if one conducted a larger study.
This is an interesting idea, but a note of caution here is that effect sizes could shrink in larger studies for 2 reasons: 1. the “good” reasons of less publication bias and more power, etc, 2. the “bad” (bias) reasons that larger studies may be more likely to be implemented more loosely (maybe by government rather than a motivated NGO, for example). The latter issue isn’t statistical, it’s that a genuinely different treatment is being applied.
Whether or not this matters depends on exactly the question you’re asking, but there is some risk in blurring the two sources of shrinkage in effect sizes over the size of the study.
Yep, totally agree that this would be tricky! There’d be a lot of details to think through. I would note that Vivalt does run regressions where, e.g., the kind of organization implementing the program (government vs NGO) is included as a covariate, and the coefficient on sample size doesn’t change much (-0.011 vs −0.013 in the single linear regression; see table 7, p. 31).
This is an interesting idea, but a note of caution here is that effect sizes could shrink in larger studies for 2 reasons: 1. the “good” reasons of less publication bias and more power, etc, 2. the “bad” (bias) reasons that larger studies may be more likely to be implemented more loosely (maybe by government rather than a motivated NGO, for example). The latter issue isn’t statistical, it’s that a genuinely different treatment is being applied.
Whether or not this matters depends on exactly the question you’re asking, but there is some risk in blurring the two sources of shrinkage in effect sizes over the size of the study.
Yep, totally agree that this would be tricky! There’d be a lot of details to think through. I would note that Vivalt does run regressions where, e.g., the kind of organization implementing the program (government vs NGO) is included as a covariate, and the coefficient on sample size doesn’t change much (-0.011 vs −0.013 in the single linear regression; see table 7, p. 31).