This seems very ungenerous to the global health space (...)
Fair, they’re not just monitoring inputs, but also often proper use and implementation (on top of other factors that may affect cost-effectiveness, which they adjust for). However, that could still leave many potential holes along the way to the outcomes that actually matter in themselves.
In the case of AMF, compared to the RCTs, there can be changes to or differences in
the mosquitoes (insecticide resistance, other characteristics),
the malaria organisms (Plasmodium),
net characteristics/quality,
net use,
recipient reports, e.g. their honesty or interpretations,
immune system responses to malaria and how resilient humans are to malaria or death by malaria in other ways, e.g. due to nutrition, exercise, exposure to other harmful things, other environmental factors, population (epi)genetics,
other things GiveWell or I haven’t thought of.
I think GiveWell (or the charities) tracks and/or makes some adjustments for some of these, too, of course, and we might expect the others not to matter much at all.
On the other hand, counting cataract surgeries seems pretty close to tracking an actual outcome that matters in itself or that is itself close to welfare, improved vision.[1]
That is how RCTs work. You can’t have a separate RCT for every situation unfortunately.
But we could have ongoing RCTs of GiveWell recommendations to check that the charities are still having important effects, although that may raise ethical issues at this point. Instead, M&E, observational research, fact checking and other investigation could be used to provide more independent evidence for outcomes closer to the ones we actually care about and our causal effects on them, like has been done for corporate chicken welfare work.
They could verify the severity of cataracts cases being treated and that the cataracts are actually cured by the surgeries (in a representative or random sample of treated individuals). (Maybe they do all this; I didn’t check.) Cataracts don’t go away on their own, so we only need to make assumptions about how many would have otherwise gotten (successful) treatment anyway (and when) to estimate the causal effects on cataracts.
Still, the quality of life impacts of the cataract surgeries could also be different compared to studies. There could be differences in social support. There could be differences in vision unrelated to cataracts, like people being more nearsighted, reducing the impact of cataract surgery without correction for nearsightedness, although I don’t expect differences in nearsightedness to matter much.
Fair, they’re not just monitoring inputs, but also often proper use and implementation (on top of other factors that may affect cost-effectiveness, which they adjust for). However, that could still leave many potential holes along the way to the outcomes that actually matter in themselves.
In the case of AMF, compared to the RCTs, there can be changes to or differences in
the mosquitoes (insecticide resistance, other characteristics),
the malaria organisms (Plasmodium),
net characteristics/quality,
net use,
recipient reports, e.g. their honesty or interpretations,
immune system responses to malaria and how resilient humans are to malaria or death by malaria in other ways, e.g. due to nutrition, exercise, exposure to other harmful things, other environmental factors, population (epi)genetics,
other things GiveWell or I haven’t thought of.
I think GiveWell (or the charities) tracks and/or makes some adjustments for some of these, too, of course, and we might expect the others not to matter much at all.
On the other hand, counting cataract surgeries seems pretty close to tracking an actual outcome that matters in itself or that is itself close to welfare, improved vision.[1]
But we could have ongoing RCTs of GiveWell recommendations to check that the charities are still having important effects, although that may raise ethical issues at this point. Instead, M&E, observational research, fact checking and other investigation could be used to provide more independent evidence for outcomes closer to the ones we actually care about and our causal effects on them, like has been done for corporate chicken welfare work.
They could verify the severity of cataracts cases being treated and that the cataracts are actually cured by the surgeries (in a representative or random sample of treated individuals). (Maybe they do all this; I didn’t check.) Cataracts don’t go away on their own, so we only need to make assumptions about how many would have otherwise gotten (successful) treatment anyway (and when) to estimate the causal effects on cataracts.
Still, the quality of life impacts of the cataract surgeries could also be different compared to studies. There could be differences in social support. There could be differences in vision unrelated to cataracts, like people being more nearsighted, reducing the impact of cataract surgery without correction for nearsightedness, although I don’t expect differences in nearsightedness to matter much.