I think his fundamental objections are philosophical (i.e. he’s more annoyed by the rhetoric of GiveWell et al than the inevitable limitations to their research). Most of the details he picks out are weaknesses GiveWell themselves highlighted and others are general foreign aid critiques, some of which apply less to small orgs distributing nets and pills than other types of aid programme and organization.
The wider idea that GiveWell and similar RCT-oriented analysis usually misses second order effects especially when undertaken by people with little experience of the developing world is valid but not novel: a more nuanced critique would note that many of these second order effects absent from GiveWell figures are positive and most of them are comparatively small. Criticising a charity evaluator for not estimating how many future deaths are likely from bandit attacks on charities’ offices is more dramatic than questioning what proportion of nets would otherwise have been sold to families by local shops anyway, but it’s not a more useful illustration of their methodological limitations.
I actually broadly agree with his general argument that EA overestimates the importance of being smart and analytical and underestimates local/sector knowledge, but I’m not sure there’s much in there that’s actionable insight (and you could probably apply half the fully general aid criticisms he applies to EA to the surfer-helping-his-friends-in-Indonesia example which he actually likes too)
Yeah, I’m not really bothered by objections at the abstract level of “but you can never account for every side effect”, since as you say there’s no real reason to think they are net negative. (I get the feeling that he thinks “that’s not the point you evil, ends-justifies-the-means utilitarians, if your doing harm you should stop, whether or not it leads to some greater good!” But I think that’s confusing the view that you shouldn’t deliberately do bad things for the greater good, with the implausible claim that no one should ever doing anything that was a necessary step in a bad thing happening. The latter would just recommend never doing anything significant, or probably even insignificant, ever.)
What does bother me is if:
GiveWell is not properly accounting for or being honest about negative side effects that it actually does know about.
GiveWell is overstating the evidence for the reliability/magnitude of the primary effectd the interventions they recommend are designed to bring about.
I got the impression he was also endorsing 1 and 2, though it’s not exactly giving a detailed defence of them.
I think his argument is mainly “aid is waaayyyy more unpredictable and difficult to measure than your neat little tables crediting yourselves with how efficient you are at saving lives suggest”, with GiveWell ironically getting the biggest bashing because of how explicit they are about highlighting limitations in their small print. Virtually all the negative side effects and recommendation retractions he’s highlighted come straight from their presentations of their evidence on their website. He’s also insistent they need to balance the positives of lifesaving against harm from nets being redeployed for fishing, but ironically the only people I’ve seen agree with him on that point are EAs.
It’s less an argument they’re not properly accounting for stuff and more that the summaries with the donation button below sound a lot more confident about impact than the summaries with the details for people that actually want to read them. I’m reminded of Holden’s outspoken criticism of big NGOs simplifying their message to “do x for $y per month” being “donor illusion” back in the day…
I guess I feel “what are they supposed to do, not put their bottom-line best estimate in the summary?”. Maybe he’d be satisfied if all the summaries said “our best guess is probably off by quite a lot, but sadly this is unavoidable, we still think your donations will on average do more good if you listen to us than if you try to find the best choice yourself”?
I think his fundamental objections are philosophical (i.e. he’s more annoyed by the rhetoric of GiveWell et al than the inevitable limitations to their research). Most of the details he picks out are weaknesses GiveWell themselves highlighted and others are general foreign aid critiques, some of which apply less to small orgs distributing nets and pills than other types of aid programme and organization.
The wider idea that GiveWell and similar RCT-oriented analysis usually misses second order effects especially when undertaken by people with little experience of the developing world is valid but not novel: a more nuanced critique would note that many of these second order effects absent from GiveWell figures are positive and most of them are comparatively small. Criticising a charity evaluator for not estimating how many future deaths are likely from bandit attacks on charities’ offices is more dramatic than questioning what proportion of nets would otherwise have been sold to families by local shops anyway, but it’s not a more useful illustration of their methodological limitations.
I actually broadly agree with his general argument that EA overestimates the importance of being smart and analytical and underestimates local/sector knowledge, but I’m not sure there’s much in there that’s actionable insight (and you could probably apply half the fully general aid criticisms he applies to EA to the surfer-helping-his-friends-in-Indonesia example which he actually likes too)
Yeah, I’m not really bothered by objections at the abstract level of “but you can never account for every side effect”, since as you say there’s no real reason to think they are net negative. (I get the feeling that he thinks “that’s not the point you evil, ends-justifies-the-means utilitarians, if your doing harm you should stop, whether or not it leads to some greater good!” But I think that’s confusing the view that you shouldn’t deliberately do bad things for the greater good, with the implausible claim that no one should ever doing anything that was a necessary step in a bad thing happening. The latter would just recommend never doing anything significant, or probably even insignificant, ever.)
What does bother me is if:
GiveWell is not properly accounting for or being honest about negative side effects that it actually does know about.
GiveWell is overstating the evidence for the reliability/magnitude of the primary effectd the interventions they recommend are designed to bring about.
I got the impression he was also endorsing 1 and 2, though it’s not exactly giving a detailed defence of them.
I think his argument is mainly “aid is waaayyyy more unpredictable and difficult to measure than your neat little tables crediting yourselves with how efficient you are at saving lives suggest”, with GiveWell ironically getting the biggest bashing because of how explicit they are about highlighting limitations in their small print. Virtually all the negative side effects and recommendation retractions he’s highlighted come straight from their presentations of their evidence on their website. He’s also insistent they need to balance the positives of lifesaving against harm from nets being redeployed for fishing, but ironically the only people I’ve seen agree with him on that point are EAs.
It’s less an argument they’re not properly accounting for stuff and more that the summaries with the donation button below sound a lot more confident about impact than the summaries with the details for people that actually want to read them. I’m reminded of Holden’s outspoken criticism of big NGOs simplifying their message to “do x for $y per month” being “donor illusion” back in the day…
I guess I feel “what are they supposed to do, not put their bottom-line best estimate in the summary?”. Maybe he’d be satisfied if all the summaries said “our best guess is probably off by quite a lot, but sadly this is unavoidable, we still think your donations will on average do more good if you listen to us than if you try to find the best choice yourself”?