(I only skimmed your post, and it has been some time since I’ve read either the GiveWell intervention reports or the studies they draw from)
I appreciate attempts to criticize/red-team existing EA organizations and EA evaluations of interventions. That said, this argument mostly falls flat for me.
My understanding is that the structure of the GiveWell recommendation for the Against Malaria Foundation (AMF) is really quite simple:
At the intervention level, there are strong, RCT-backed evidence that long-lasting insecticidal bednets are very good at preventing mosquito borne illnesses and overall decreasing child mortality.
At the charity level, AMF is unusually good at distributing such bednets at scale.
These arguments are not iron-clad. For example, for #1, maybe you think insecticidal bednets are so a priori implausible as an anti-malaria intervention that you would not trust any level of RCT evidence? But this just falls flat to me, as “bednets that prevent/kill mosquitoes makes it harder for malarial mosquitoes to sting kids at night” passes some very simple sanity checks, at least for me. (Or perhaps you think drawing GiveWell’s conclusion from the RCTs is statistically wrong, because of reasons? If so, it’d be good to list the reasons!)
Another reason you might doubt #2 is relevant is if you’re suspicious that AMF can confer similar results as would be implied by the RCTs. For example, if you think the places AMF works in is so “out-of-distribution” relative to the RCTs, because of lower malarial load[1]. But my understanding is that a) the GiveWell analysis accounts for this and b) the malarial loads aren’t that different.
There are a number of other reasons that I would not go into that engages with the argument structure.
However, your critique does not engage with the structure of the argument, and instead[2] argues that because there’s no direct empirical evidence of AMF’s specific distribution of bednets saving lives, we cannot assume that AMF’s bednets save lives.
I currently think your post is an overly myopic treatment of the evidence. For a better extension by my lights, I’d be interested to see more engagement from you on whether the structure of the original argument is wrong, or alternatively, why you think your alternative formulation/framework of the problem ought to be the preferred one. I would also be interested in a very different critique of AMF that takes GiveWell’s structure as a given but argues that by those lights, AMF is not a good donation target (eg because the intervention research is actually shoddy, or because AMF is actually bad at delivering bednets).
[1] My understanding is that, in contrast, substantially lower worm load is a serious reason to be skeptical of the present-day impact of deworming interventions.
[2] You also argue that there’s observational data against AMF’s effectiveness because the countries they work in don’t have obviously lower malarial loads. However I think causality is just pretty hard to determine from observational data, for reasons Charles mentions here.
To clarify, my point is not just there’s no direct empirical evidence of AMF’s specific distributions saving lives. My point is that there is no direct evidence of any non-RCT/”real world” distributions saving lives.
Further, this is not because nobody is looking for such evidence. GiveWell’s macro review of the evidence suggests every time somebody has looked for evidence of non-RCT/”real world” distributions saving lives they’ve come up with nothing.
I agree with your summary of the GiveWell argument (strong RCT evidence + AMF as competent distributor). However, in order to turn these two facts into a prediction about future we need to add the assumption that the RCT evidence applies to future distributions. This is the weak link in the chain. As you say, differences in malarial load could distort things. Differences in the underlying health of the population, differences in net usage and increasing insecticide resistance are other contenders, along with many more I’m sure. If we can’t see any evidence of impact after distributing hundreds of millions of bednets then it seems reasonable to question if this key assumption is leading us astray.
(I only skimmed your post, and it has been some time since I’ve read either the GiveWell intervention reports or the studies they draw from)
I appreciate attempts to criticize/red-team existing EA organizations and EA evaluations of interventions. That said, this argument mostly falls flat for me.
My understanding is that the structure of the GiveWell recommendation for the Against Malaria Foundation (AMF) is really quite simple:
At the intervention level, there are strong, RCT-backed evidence that long-lasting insecticidal bednets are very good at preventing mosquito borne illnesses and overall decreasing child mortality.
At the charity level, AMF is unusually good at distributing such bednets at scale.
These arguments are not iron-clad. For example, for #1, maybe you think insecticidal bednets are so a priori implausible as an anti-malaria intervention that you would not trust any level of RCT evidence? But this just falls flat to me, as “bednets that prevent/kill mosquitoes makes it harder for malarial mosquitoes to sting kids at night” passes some very simple sanity checks, at least for me. (Or perhaps you think drawing GiveWell’s conclusion from the RCTs is statistically wrong, because of reasons? If so, it’d be good to list the reasons!)
Another reason you might doubt #2 is relevant is if you’re suspicious that AMF can confer similar results as would be implied by the RCTs. For example, if you think the places AMF works in is so “out-of-distribution” relative to the RCTs, because of lower malarial load[1]. But my understanding is that a) the GiveWell analysis accounts for this and b) the malarial loads aren’t that different.
There are a number of other reasons that I would not go into that engages with the argument structure.
However, your critique does not engage with the structure of the argument, and instead[2] argues that because there’s no direct empirical evidence of AMF’s specific distribution of bednets saving lives, we cannot assume that AMF’s bednets save lives.
I currently think your post is an overly myopic treatment of the evidence. For a better extension by my lights, I’d be interested to see more engagement from you on whether the structure of the original argument is wrong, or alternatively, why you think your alternative formulation/framework of the problem ought to be the preferred one. I would also be interested in a very different critique of AMF that takes GiveWell’s structure as a given but argues that by those lights, AMF is not a good donation target (eg because the intervention research is actually shoddy, or because AMF is actually bad at delivering bednets).
[1] My understanding is that, in contrast, substantially lower worm load is a serious reason to be skeptical of the present-day impact of deworming interventions.
[2] You also argue that there’s observational data against AMF’s effectiveness because the countries they work in don’t have obviously lower malarial loads. However I think causality is just pretty hard to determine from observational data, for reasons Charles mentions here.
Thanks Linch, interesting thoughts.
To clarify, my point is not just there’s no direct empirical evidence of AMF’s specific distributions saving lives. My point is that there is no direct evidence of any non-RCT/”real world” distributions saving lives.
Further, this is not because nobody is looking for such evidence. GiveWell’s macro review of the evidence suggests every time somebody has looked for evidence of non-RCT/”real world” distributions saving lives they’ve come up with nothing.
I agree with your summary of the GiveWell argument (strong RCT evidence + AMF as competent distributor). However, in order to turn these two facts into a prediction about future we need to add the assumption that the RCT evidence applies to future distributions. This is the weak link in the chain. As you say, differences in malarial load could distort things. Differences in the underlying health of the population, differences in net usage and increasing insecticide resistance are other contenders, along with many more I’m sure. If we can’t see any evidence of impact after distributing hundreds of millions of bednets then it seems reasonable to question if this key assumption is leading us astray.