On non-specific discount factors: one approach which I was interested in when doing a lot of this work was to use estimates we have of how much effect sizes shrink when more and/or larger studies are conducted.
For example, in this paper Eva Vivalt, using a sample of impact evaluations, regresses effect size on variables like number of studies and sample size. As one would expect, the larger the sample size, the smaller the estimated effect size. I always wondered if you could just use the regression coefficients she presents to estimate how much an effect size would be expected to shrink if one conducted a larger study.
I don’t think this strikes at exactly what you or HLI are trying to get at. But I do think it’s valuable to ponder how we might get at “principled” discount rates, given all we know about the validity problems these studies exhibit.
For example, in this paper Eva Vivalt, using a sample of impact evaluations, regresses effect size on variables like number of studies and sample size. As one would expect, the larger the sample size, the smaller the estimated effect size. I always wondered if you could just use the regression coefficients she presents to estimate how much an effect size would be expected to shrink if one conducted a larger study.
This is an interesting idea, but a note of caution here is that effect sizes could shrink in larger studies for 2 reasons: 1. the “good” reasons of less publication bias and more power, etc, 2. the “bad” (bias) reasons that larger studies may be more likely to be implemented more loosely (maybe by government rather than a motivated NGO, for example). The latter issue isn’t statistical, it’s that a genuinely different treatment is being applied.
Whether or not this matters depends on exactly the question you’re asking, but there is some risk in blurring the two sources of shrinkage in effect sizes over the size of the study.
Yep, totally agree that this would be tricky! There’d be a lot of details to think through. I would note that Vivalt does run regressions where, e.g., the kind of organization implementing the program (government vs NGO) is included as a covariate, and the coefficient on sample size doesn’t change much (-0.011 vs −0.013 in the single linear regression; see table 7, p. 31).
I am not sure if I’m a fan of this, because the true effect of an intervention will vary across place in ways that will affect the results of future studies, but shouldn’t affect our assessment of deworming in this context. So future studies might extend deworming to countries where worm burdens are lower and thus find lower effects of deworming. But it would be a mistake to conclude that deworming in Kenya is less effective based on those studies.
You might say we can control for the country being studied, but that is only the case if there are many studies in one country, which is rarely the case.
On non-specific discount factors: one approach which I was interested in when doing a lot of this work was to use estimates we have of how much effect sizes shrink when more and/or larger studies are conducted.
For example, in this paper Eva Vivalt, using a sample of impact evaluations, regresses effect size on variables like number of studies and sample size. As one would expect, the larger the sample size, the smaller the estimated effect size. I always wondered if you could just use the regression coefficients she presents to estimate how much an effect size would be expected to shrink if one conducted a larger study.
I don’t think this strikes at exactly what you or HLI are trying to get at. But I do think it’s valuable to ponder how we might get at “principled” discount rates, given all we know about the validity problems these studies exhibit.
This is an interesting idea, but a note of caution here is that effect sizes could shrink in larger studies for 2 reasons: 1. the “good” reasons of less publication bias and more power, etc, 2. the “bad” (bias) reasons that larger studies may be more likely to be implemented more loosely (maybe by government rather than a motivated NGO, for example). The latter issue isn’t statistical, it’s that a genuinely different treatment is being applied.
Whether or not this matters depends on exactly the question you’re asking, but there is some risk in blurring the two sources of shrinkage in effect sizes over the size of the study.
Yep, totally agree that this would be tricky! There’d be a lot of details to think through. I would note that Vivalt does run regressions where, e.g., the kind of organization implementing the program (government vs NGO) is included as a covariate, and the coefficient on sample size doesn’t change much (-0.011 vs −0.013 in the single linear regression; see table 7, p. 31).
I am not sure if I’m a fan of this, because the true effect of an intervention will vary across place in ways that will affect the results of future studies, but shouldn’t affect our assessment of deworming in this context. So future studies might extend deworming to countries where worm burdens are lower and thus find lower effects of deworming. But it would be a mistake to conclude that deworming in Kenya is less effective based on those studies.
You might say we can control for the country being studied, but that is only the case if there are many studies in one country, which is rarely the case.