To add to this problem, I think that more uncertain causes are more likely to have overestimated point-estimates even errors non-withstanding. This is for a few reasons, one because we are by nature more optimistic than reality about small probabilities, and that lower quality studies such as cross sectional studies and cohort tend to overestimate effects. When better studies like RCTs or even larger cohorts are done effects often become smaller or disappear.
Unfortunately for many interventions the best data we have is low quality data, and we anchor on those likely overestimated values.
Nick, just wanted to share that I personally strongly held the view that RCTs usually produce smaller effects than observational trials until reading a 2024 Cochrane study examining this specific question—in short, I think it supports a prior view that well-conducted observational studies (at least in healthcare) are not much more likely to overestimate effects than RCTs. To your credit you explicitly include “larger cohorts” among your example of “better studies”, so perhaps you already know this! But it was news to me. (Of course I also agree that small and/or low-quality studies, regardless of methodology, should be taken with caution).
Hey there yes that’s a great review. I’m not sure how relevant to this development stuff it is though, because
It only accepts really high quality observational studies
it’s focuses on human health. We’re reasonably good at controlling for confounders with humans, but we have very little clue how to do that with development interventions.
I would love a similar review for development studies but I doubt there would be enough good quality research to do a similar comparison
I have a paper that can help answer this, which uses JPAL and IPA studies! However, you might think observational study overestimates come from selection bias during the publication process—our result doesn’t say anything about that.
“First, we find that there is little bias on average. Using our
best-performing observational method (DDML), there is a statistically insignificant and modest
negative mean bias of −0.025 standard deviations. This implies that observational studies do not
systematically over- or underestimate the welfare impact of the programs they evaluate.”
That’s right, but it should be possible to model that in a very similar hierarchical manner and adjust accordingly, too, if you buy into the original framework laid out in the post.
(I haven’t fully thought it thru but it does strike me as fundamentally possible with the same caveats of not knowing parameters, not that I’d suggest using the toy model style maths in practice).
Wow this is brilliant thank you!
To add to this problem, I think that more uncertain causes are more likely to have overestimated point-estimates even errors non-withstanding. This is for a few reasons, one because we are by nature more optimistic than reality about small probabilities, and that lower quality studies such as cross sectional studies and cohort tend to overestimate effects. When better studies like RCTs or even larger cohorts are done effects often become smaller or disappear.
Unfortunately for many interventions the best data we have is low quality data, and we anchor on those likely overestimated values.
Nick, just wanted to share that I personally strongly held the view that RCTs usually produce smaller effects than observational trials until reading a 2024 Cochrane study examining this specific question—in short, I think it supports a prior view that well-conducted observational studies (at least in healthcare) are not much more likely to overestimate effects than RCTs. To your credit you explicitly include “larger cohorts” among your example of “better studies”, so perhaps you already know this! But it was news to me. (Of course I also agree that small and/or low-quality studies, regardless of methodology, should be taken with caution).
Hey there yes that’s a great review. I’m not sure how relevant to this development stuff it is though, because
It only accepts really high quality observational studies
it’s focuses on human health. We’re reasonably good at controlling for confounders with humans, but we have very little clue how to do that with development interventions.
I would love a similar review for development studies but I doubt there would be enough good quality research to do a similar comparison
I have a paper that can help answer this, which uses JPAL and IPA studies! However, you might think observational study overestimates come from selection bias during the publication process—our result doesn’t say anything about that.
https://www.jondequidt.com/pdfs/Lalonde30.pdf
“First, we find that there is little bias on average. Using our best-performing observational method (DDML), there is a statistically insignificant and modest negative mean bias of −0.025 standard deviations. This implies that observational studies do not systematically over- or underestimate the welfare impact of the programs they evaluate.”
That’s right, but it should be possible to model that in a very similar hierarchical manner and adjust accordingly, too, if you buy into the original framework laid out in the post.
(I haven’t fully thought it thru but it does strike me as fundamentally possible with the same caveats of not knowing parameters, not that I’d suggest using the toy model style maths in practice).