I didn’t explicitly say it, but I think GiveWell did a fantastic job of re-analysing and adjusting here, your range and final estimate for estimating the mortality of the intervention are pretty similar to what my intuition would have moved to, and I agree with your methods of getting there—especially doing your own meta-analysis only using the RCTs.
A couple of other comments
1. I agree with you on excluding the Haushofer/Kremer study, but not for the reasons you state. I dont’ really understand why exclude a study just because “we believe the effect size it reports is implausibly large, and it has a substantial impact on the pooled estimate” . This seems unnecessarily subjective to meI’m not sure why 30% might be plausible but 60%ish is not? I know 60% does seem subjectively implausible but I’m not sure that’s enough of a reason to exclude a study. If a study is methodologically sound, then why not include it.
BUT I think its reasonable to exclude the study because it was done retrospectively, so was not an RCT at all—You say on your website that is is an RCT, but it is not. They retrospectively gather data on mortality post-hoc well after the original RCT was done. The study itself doesn’t claim to be an RCT. As you only want to consider RCTs, this would exclude it automatically as it is not one. Its a small thing but maybe you could consider correcting this on the website?
2. (This is very minor) I like your plausibility “cap”, but feel like it might be a little low given that 5 meta-analysed RCTs did show a 25-30% mortality reduction. Would it not perhaps be more logical to use the research figure as the cap? Its hard to “reason” our way to plausible percentages here, precisely because we have little idea why the mortality reduction is happening.
The question I’m very interested in of course is why, and I really hope some of the grants you are making will go towards , rather than purely investigating the magnitude of mortality question—which is obviously still the primary purpose of the studies. The most important way to make progress on the answer might be to ascertain the causes of death in those who die in the treatment vs. control groups, so I hope they are planning to collect that data as best they can at the very least. Your study page doesn’t mention that any of the RCT work is geared towards the why—but it’s not too late ;).
Again amazing job on analysing this, and funding more research on the topic. I don’t think it would have happened without you!
Thanks so much for engaging Alex.
I didn’t explicitly say it, but I think GiveWell did a fantastic job of re-analysing and adjusting here, your range and final estimate for estimating the mortality of the intervention are pretty similar to what my intuition would have moved to, and I agree with your methods of getting there—especially doing your own meta-analysis only using the RCTs.
A couple of other comments
1. I agree with you on excluding the Haushofer/Kremer study, but not for the reasons you state. I dont’ really understand why exclude a study just because “we believe the effect size it reports is implausibly large, and it has a substantial impact on the pooled estimate” . This seems unnecessarily subjective to me I’m not sure why 30% might be plausible but 60%ish is not? I know 60% does seem subjectively implausible but I’m not sure that’s enough of a reason to exclude a study. If a study is methodologically sound, then why not include it.
BUT I think its reasonable to exclude the study because it was done retrospectively, so was not an RCT at all—You say on your website that is is an RCT, but it is not. They retrospectively gather data on mortality post-hoc well after the original RCT was done. The study itself doesn’t claim to be an RCT. As you only want to consider RCTs, this would exclude it automatically as it is not one. Its a small thing but maybe you could consider correcting this on the website?
2. (This is very minor) I like your plausibility “cap”, but feel like it might be a little low given that 5 meta-analysed RCTs did show a 25-30% mortality reduction. Would it not perhaps be more logical to use the research figure as the cap? Its hard to “reason” our way to plausible percentages here, precisely because we have little idea why the mortality reduction is happening.
The question I’m very interested in of course is why, and I really hope some of the grants you are making will go towards , rather than purely investigating the magnitude of mortality question—which is obviously still the primary purpose of the studies. The most important way to make progress on the answer might be to ascertain the causes of death in those who die in the treatment vs. control groups, so I hope they are planning to collect that data as best they can at the very least. Your study page doesn’t mention that any of the RCT work is geared towards the why—but it’s not too late ;).
Again amazing job on analysing this, and funding more research on the topic. I don’t think it would have happened without you!