I feel like as president of 1Day Sooner I should probably chime in—first, I wanted to say this type of work—critiquing advocacy campaigns and analyses from EA or EA-aligned groups—is very valuable and should be encouraged. I’m appreciative of SoGIve for publishing this and think they should be commended for spending the time to conduct this analysis. I think creating a healthy ecosystem for disagreement and the right incentives to encourage criticism and full-throated debate is important.
On the object-level question, I’m obviously biased but I think most of the difference in cost-effectiveness in the SoGive analysis goes away if you adjust for the fact that vaccines are only given to children under 5 but only ~15-20% of bednets cover children under 5. Because 75% of malaria mortality is in children under 5, bednets are cheaper per person protected but the vaccines are much more targeted to people whose protection is most valuable. (The development benefit effects of reducing morbidity in children are also age-skewed in vaccines’ favor though that’s less dramatic).
Insecticide resistance (probably reduces bednets’ effectiveness to about 80% of what they’d otherwise be) and durability (GW estimates each bednet purchased provides about 1.7 year of coverage) are probably also relevant. The AMF tab of the GiveWell spreadsheet is a useful resource in thinking through these questions.
For more of my thinking, here’s my side of the email correspondence with Sanjay at SoGive. (I didn’t include text from other people on the thread because I haven’t asked their permission to share). An interesting meta-question is what should be the norm about making these sort of red-teaming or adversarial post-review correspondences public. My guess is it’s probably a good thing to default to because it incentivizes people to be on their best behavior (and the benefits to confidentiality of being able to speak frankly don’t seem that strong in these cases). But I don’t think it’s obvious either way and would be curious what other people think.
Overall, I’m eager to see more analysis done digging into the Imperial/Oxford modeling of cost benefit of the R21 vaccine (which comes to about 630 lives saved per 100K vaccinated, see Table 2) and what’s publicly available about the WHO estimate of 13% all-cause mortality reduction from RTS,S. (Here’s an older preprint that finds a smaller benefit—more recent data that was publicly reported is apparently higher). So I think generally the follow-on research plan Sanjay discusses makes sense from my perspective, and I’d be personally supportive of anyone who wants to contribute to that work.
Thank you Josh. I’ve found 1Day Sooner’s collaborative spirit to be exemplary here—both being welcoming of the challenge and adding useful thoughts.
It seems intuitive to me that the following package of considerations may lead to vaccines and nets/SMC having roughly the same cost-effectiveness:
vaccines are 10x (ish) more expensive (bad for vaccines)
vaccines are more targeted at the most vulnerable ages (good for vaccines)
misc other considerations, like insecticide resistance (this is a bit hand-wavey at the moment, but I guess probably nets out to being good for vaccines)
A hint at the possibility that this might end up with similar cost-effectiveness is the Imperial/Oxford study that Josh mentioned.
Other considerations include:
Replicability: when we create these sorts of models, we don’t normally give 100% credit to the efficacy figures for vaccines. Rather we recognise there isn’t yet a large evidence base, and typically the efficacy is more likely to go down than up with more evidence; estimating how much it would go down by as we got more and more evidence is sometimes called an internal validity adjustment, or replicability adjustment.
For bednets this is negligible (adjust by 5%), and with good reason—nets are a very well-established intervention with plenty of evidence
I expect that such an adjustment for vaccines might be significantly less favourable, but I’d need to look at the evidence properly to say anything more precise than that
My best guess, having not researched it properly, is that the cost-effectiveness of the R21 vaccine probably will still be clearly behind that of existing interventions (bednets/SMC), but it may well be above the cost-effectiveness thresholds of the bodies you’re advocating to (e.g. because they are large aid agencies with large budgets), meaning that the advocacy still makes sense. I wouldn’t be surprised if either of these guesses were incorrect
I feel like as president of 1Day Sooner I should probably chime in—first, I wanted to say this type of work—critiquing advocacy campaigns and analyses from EA or EA-aligned groups—is very valuable and should be encouraged. I’m appreciative of SoGIve for publishing this and think they should be commended for spending the time to conduct this analysis. I think creating a healthy ecosystem for disagreement and the right incentives to encourage criticism and full-throated debate is important.
On the object-level question, I’m obviously biased but I think most of the difference in cost-effectiveness in the SoGive analysis goes away if you adjust for the fact that vaccines are only given to children under 5 but only ~15-20% of bednets cover children under 5. Because 75% of malaria mortality is in children under 5, bednets are cheaper per person protected but the vaccines are much more targeted to people whose protection is most valuable. (The development benefit effects of reducing morbidity in children are also age-skewed in vaccines’ favor though that’s less dramatic).
Insecticide resistance (probably reduces bednets’ effectiveness to about 80% of what they’d otherwise be) and durability (GW estimates each bednet purchased provides about 1.7 year of coverage) are probably also relevant. The AMF tab of the GiveWell spreadsheet is a useful resource in thinking through these questions.
For more of my thinking, here’s my side of the email correspondence with Sanjay at SoGive. (I didn’t include text from other people on the thread because I haven’t asked their permission to share). An interesting meta-question is what should be the norm about making these sort of red-teaming or adversarial post-review correspondences public. My guess is it’s probably a good thing to default to because it incentivizes people to be on their best behavior (and the benefits to confidentiality of being able to speak frankly don’t seem that strong in these cases). But I don’t think it’s obvious either way and would be curious what other people think.
Overall, I’m eager to see more analysis done digging into the Imperial/Oxford modeling of cost benefit of the R21 vaccine (which comes to about 630 lives saved per 100K vaccinated, see Table 2) and what’s publicly available about the WHO estimate of 13% all-cause mortality reduction from RTS,S. (Here’s an older preprint that finds a smaller benefit—more recent data that was publicly reported is apparently higher). So I think generally the follow-on research plan Sanjay discusses makes sense from my perspective, and I’d be personally supportive of anyone who wants to contribute to that work.
Thank you Josh. I’ve found 1Day Sooner’s collaborative spirit to be exemplary here—both being welcoming of the challenge and adding useful thoughts.
It seems intuitive to me that the following package of considerations may lead to vaccines and nets/SMC having roughly the same cost-effectiveness:
vaccines are 10x (ish) more expensive (bad for vaccines)
vaccines are more targeted at the most vulnerable ages (good for vaccines)
misc other considerations, like insecticide resistance (this is a bit hand-wavey at the moment, but I guess probably nets out to being good for vaccines)
A hint at the possibility that this might end up with similar cost-effectiveness is the Imperial/Oxford study that Josh mentioned.
Other considerations include:
Replicability: when we create these sorts of models, we don’t normally give 100% credit to the efficacy figures for vaccines. Rather we recognise there isn’t yet a large evidence base, and typically the efficacy is more likely to go down than up with more evidence; estimating how much it would go down by as we got more and more evidence is sometimes called an internal validity adjustment, or replicability adjustment.
For bednets this is negligible (adjust by 5%), and with good reason—nets are a very well-established intervention with plenty of evidence
I expect that such an adjustment for vaccines might be significantly less favourable, but I’d need to look at the evidence properly to say anything more precise than that
My best guess, having not researched it properly, is that the cost-effectiveness of the R21 vaccine probably will still be clearly behind that of existing interventions (bednets/SMC), but it may well be above the cost-effectiveness thresholds of the bodies you’re advocating to (e.g. because they are large aid agencies with large budgets), meaning that the advocacy still makes sense. I wouldn’t be surprised if either of these guesses were incorrect