I think we agree on somewhat more than it seems at first glance. I don’t think the current GiveWell top charities are the pinnacle of cost-effectiveness, support further cause exploration and incubating the most promising ideas into charities, and think it’s quite possible for EA funders to miss important stuff.
The crux is that I don’t think it’s warranted to directly compare cost-effectiveness analyses conducted on a few weeks of desktop research, expert interviews and commissioning of surveys and quantitative modelling to evaluations of specific charities at scale and in action, and I think your original post did that with allusions to scamming, GiveWell charities as $1000 liters of milk, and being a sucker.
Although CEARCH is too young to us retrospectively compare its analyses to the cost-effectiveness of launched charities, I think something like drug development is a good analogy. Lots of stuff looks great on paper, in situ, or even in animal models, only to fall apart entirely in the multi-phase human clinical trial process on the way to full approval. Comparing how a drug does in limited animal models to how another drug does in Phase III trials is comparing apples to oranges. Moreover, “risk the model/drug will fall apart in later phases” is distinct from “risk after Phase III trials that the model/drug will not work in a specific country/patient.”
To be very clear, this is not a criticism of CEARCH—as I see it, its job is to screen candidate interventions, not to bring them up to the level of mature, shovel-ready interventions. The next step would be either incubation or a deep dive on a specific target charity already doing this work. I would expect to see a ton of false positives, just as I would expect that from the earliest phases of drug development. It’s worth it to find the next ten-figure drug / blockbuster EA intervention.
And these causes pretty easy to find. CEARCH has been started in 2022 and has already found 4 causes 10x GiveWell under my aforementioned pessimistic assumptions. CE and RP have found more. There are big funding gaps, because there are many causes like this. There are many big world governments to do lobbying to. We should aim to close the funding gaps as soon as possible, because that would help more people.
I think this should make you question your assumptions to some extent. GiveWell has evaluated tons of interventions for a number of years, and made significant grants for a number of them. If CEARCH has come up with 4 causes that are 10X top charities in ~ a year with 2 FTEs, while GiveWell hasn’t come up with anything better than 1x in many years with lots more FTEs, what conclusion do we draw from that? I think it more likely that CEARCH is applying more generous assumptions than that GiveWell is badly screwing up its analysis of intervention after intervention. (And no one else, e.g., Founders’ Pledge, has been able to come up with clearly better interventions either, at least based on neartermist global health priorities.)
More generous assumptions come with the territory of early-stage CEAs, so I am not suggesting that is problematic given CEARCH’s mission. But I think its analysis supports a conclusion of “we should incubate a charity pursuing this intervention,” not “we should conclude that our GiveWell donations were very poor value and immediately divert tens of millions of dollars into sodium-reduction policy.” In my view, your original post was relatively closer to the later than your reply comment.
As for CE, it estimates that “starting a high-impact charity has the same impact as donating $200,000 to the most effective NGOs every year.” That doesn’t suggest a belief that a lot of its incubated charities are 10x+ GiveWell and able to absorb significant funding.
GiveWell has shown a willingness to fund policy work out of its All Grants Fund where it thinks the cost-effectiveness is there (cf. $7MM to the Centre for Pesticide Suicide Prevention for general support in January 2021, also for work on alcohol policy). So a general antipathy toward policy/lobbying work doesn’t seem to explain what is going on here. Rather, I think there’s a fundamental, difficult-to-resolve disagreement about the EV of lobbying/policy work. It’s certainly possible that I—and it seems, most EA funders—are simply wrong in our estimation on that point. But I don’t think referring to the criterion standard non-policy interventions as $1000 liters of milk acknowledges that disagreement and the reasons for it.
If that was true, then all EAs seeking to maximize expected value would roughly agree on where to donate their money. Rather, we see the community being split into 4 main parts (global H&P, animals, existential risk, meta). Some people in EA simply don’t and won’t donate to some of these parts. This shows that at least a part of the community might donate to worse charities.
I think this is predominately about the donor’s values and ethical framework (e.g., the relative value of human vs. animal welfare, the extent to which future lives matter), although there are some strategic elements as well. I’m not aware of any reason to think the people who donate to global health are hostile to lobbying efforts if that is the most effective approach.
I might’ve used too strong of a language with my original post, such as the talk about being a sucker. For me it’s useful to think about donations as a product I’m buying, but I probably took it too far. And I don’t think I’ve properly emphasized my main message, which was (as I’ve added later) - the explore-exploit tradeoff for causes is really hard if you don’t know how far exploration could take you. Honestly, I’m most interested in your take on that.I initially only used GiveWell and CEARCH to demonstrate that argument and show I how got to it.
The drug analogy is interesting, although I prefer the start-up analogy. Drug development is more binary—some drugs can just flat-out fail in humans, while start-ups are more of a spectrum (the ROI might be smaller than thought etc.). I don’t see a reason to think of CEARCH recommended programs or for most other exploratory stuff as binary. Of course lobbying could flat-out fail, but it’s unlikely we’ll have to update our beliefs that this charity would NEVER work, as might happen in drug development. And obviously with start-ups, there’s also a lot of difference between the initial market research and the later stages (as you said).
GiveWell has a lot of flaws for cause exploration. They really focus on charity research, not cause research. It’s by design really biased towards existing causes and charities. The charities must be interested and cooperate with GiveWell. They look for the track record, i.e. charities operating in high-risk, low tractibillity areas such as policy have a harder time. In most cases it makes sense, sometimes it can miss great opportunities.
Yes, they’ve funded some policy focused charities, but they might’ve funded much more if they were more EV maximizing instead of risk-averse. Seeing the huge leverage such options provide, it’s entirely possible.
Also, they aren’t always efficient—look at GiveDirectly. Their bar for top charities was 10x GiveDirectly for years, yet they kept GiveDirectly as a top charity until last year??? This is not some small, hard to notice inefficiency. It literally is their consistent criteria for their flagship charities. Can you imagine a for-profit company telling their investors “well, we believe these other channels have a ROI of at least 10x, but please also consider investing in this channel with x ROI”, for multiple years? I can’t. Let alone putting that less efficient channel as one of the best investments…
That’s exactly what I mean when I say altruism, even EA, can have gross inefficiency in allocations. It’s not special to GiveWell, I’m just exemplifying.
If GiveWell can make such gross mistakes, then probably others can. Another example was their relative lack of research on family planning, which I’ve written about. They’re doing A LOT of great things too. But I must say I am a bit skeptical of their decision making sometimes.
Keep in mind, CEARCH would have to be EXTREMELY optimistic in order for us to say that it hasn’t found a couple of causes 10x GiveWell. We are talking about 40x optimistic. That might be the case, but IMO it’s a strong enough assertion to require proof. Do you have examples of something close to 40x optimism in cost-effectiveness?
I agree that a lot of the difference in EAs donations can come from differing perspectives, probably most. But I think even some utilitarian, EV maximizing, 0-future discount, animal equalists EAs donate to different causes (or any other set of shared beliefs). It’s definitely not impossible.
As for other examples of 10x GiveWell cost-effectiveness in global health:
An Israeli non-profit, which produced an estimate of 4.3$ QALYs per dollar, in cooperation with EA Israel. A volunteer said to me he believed they were about 3x too optimistic, but that’s still around 10x GiveWell.
Also here is an example of 4x disagreement between GiveWell and Founders Pledge, and an even bigger disagreement with RP, on a mass media campaign for family planning. Even the best in the business can disagree.
I think we agree on somewhat more than it seems at first glance. I don’t think the current GiveWell top charities are the pinnacle of cost-effectiveness, support further cause exploration and incubating the most promising ideas into charities, and think it’s quite possible for EA funders to miss important stuff.
The crux is that I don’t think it’s warranted to directly compare cost-effectiveness analyses conducted on a few weeks of desktop research, expert interviews and commissioning of surveys and quantitative modelling to evaluations of specific charities at scale and in action, and I think your original post did that with allusions to scamming, GiveWell charities as $1000 liters of milk, and being a sucker.
Although CEARCH is too young to us retrospectively compare its analyses to the cost-effectiveness of launched charities, I think something like drug development is a good analogy. Lots of stuff looks great on paper, in situ, or even in animal models, only to fall apart entirely in the multi-phase human clinical trial process on the way to full approval. Comparing how a drug does in limited animal models to how another drug does in Phase III trials is comparing apples to oranges. Moreover, “risk the model/drug will fall apart in later phases” is distinct from “risk after Phase III trials that the model/drug will not work in a specific country/patient.”
To be very clear, this is not a criticism of CEARCH—as I see it, its job is to screen candidate interventions, not to bring them up to the level of mature, shovel-ready interventions. The next step would be either incubation or a deep dive on a specific target charity already doing this work. I would expect to see a ton of false positives, just as I would expect that from the earliest phases of drug development. It’s worth it to find the next ten-figure drug / blockbuster EA intervention.
I think this should make you question your assumptions to some extent. GiveWell has evaluated tons of interventions for a number of years, and made significant grants for a number of them. If CEARCH has come up with 4 causes that are 10X top charities in ~ a year with 2 FTEs, while GiveWell hasn’t come up with anything better than 1x in many years with lots more FTEs, what conclusion do we draw from that? I think it more likely that CEARCH is applying more generous assumptions than that GiveWell is badly screwing up its analysis of intervention after intervention. (And no one else, e.g., Founders’ Pledge, has been able to come up with clearly better interventions either, at least based on neartermist global health priorities.)
More generous assumptions come with the territory of early-stage CEAs, so I am not suggesting that is problematic given CEARCH’s mission. But I think its analysis supports a conclusion of “we should incubate a charity pursuing this intervention,” not “we should conclude that our GiveWell donations were very poor value and immediately divert tens of millions of dollars into sodium-reduction policy.” In my view, your original post was relatively closer to the later than your reply comment.
As for CE, it estimates that “starting a high-impact charity has the same impact as donating $200,000 to the most effective NGOs every year.” That doesn’t suggest a belief that a lot of its incubated charities are 10x+ GiveWell and able to absorb significant funding.
GiveWell has shown a willingness to fund policy work out of its All Grants Fund where it thinks the cost-effectiveness is there (cf. $7MM to the Centre for Pesticide Suicide Prevention for general support in January 2021, also for work on alcohol policy). So a general antipathy toward policy/lobbying work doesn’t seem to explain what is going on here. Rather, I think there’s a fundamental, difficult-to-resolve disagreement about the EV of lobbying/policy work. It’s certainly possible that I—and it seems, most EA funders—are simply wrong in our estimation on that point. But I don’t think referring to the criterion standard non-policy interventions as $1000 liters of milk acknowledges that disagreement and the reasons for it.
I think this is predominately about the donor’s values and ethical framework (e.g., the relative value of human vs. animal welfare, the extent to which future lives matter), although there are some strategic elements as well. I’m not aware of any reason to think the people who donate to global health are hostile to lobbying efforts if that is the most effective approach.
I might’ve used too strong of a language with my original post, such as the talk about being a sucker. For me it’s useful to think about donations as a product I’m buying, but I probably took it too far. And I don’t think I’ve properly emphasized my main message, which was (as I’ve added later) - the explore-exploit tradeoff for causes is really hard if you don’t know how far exploration could take you. Honestly, I’m most interested in your take on that. I initially only used GiveWell and CEARCH to demonstrate that argument and show I how got to it.
The drug analogy is interesting, although I prefer the start-up analogy. Drug development is more binary—some drugs can just flat-out fail in humans, while start-ups are more of a spectrum (the ROI might be smaller than thought etc.). I don’t see a reason to think of CEARCH recommended programs or for most other exploratory stuff as binary. Of course lobbying could flat-out fail, but it’s unlikely we’ll have to update our beliefs that this charity would NEVER work, as might happen in drug development. And obviously with start-ups, there’s also a lot of difference between the initial market research and the later stages (as you said).
GiveWell has a lot of flaws for cause exploration. They really focus on charity research, not cause research. It’s by design really biased towards existing causes and charities. The charities must be interested and cooperate with GiveWell. They look for the track record, i.e. charities operating in high-risk, low tractibillity areas such as policy have a harder time. In most cases it makes sense, sometimes it can miss great opportunities.
Yes, they’ve funded some policy focused charities, but they might’ve funded much more if they were more EV maximizing instead of risk-averse. Seeing the huge leverage such options provide, it’s entirely possible.
Also, they aren’t always efficient—look at GiveDirectly. Their bar for top charities was 10x GiveDirectly for years, yet they kept GiveDirectly as a top charity until last year??? This is not some small, hard to notice inefficiency. It literally is their consistent criteria for their flagship charities. Can you imagine a for-profit company telling their investors “well, we believe these other channels have a ROI of at least 10x, but please also consider investing in this channel with x ROI”, for multiple years? I can’t. Let alone putting that less efficient channel as one of the best investments…
That’s exactly what I mean when I say altruism, even EA, can have gross inefficiency in allocations. It’s not special to GiveWell, I’m just exemplifying.
If GiveWell can make such gross mistakes, then probably others can. Another example was their relative lack of research on family planning, which I’ve written about. They’re doing A LOT of great things too. But I must say I am a bit skeptical of their decision making sometimes.
Keep in mind, CEARCH would have to be EXTREMELY optimistic in order for us to say that it hasn’t found a couple of causes 10x GiveWell. We are talking about 40x optimistic. That might be the case, but IMO it’s a strong enough assertion to require proof. Do you have examples of something close to 40x optimism in cost-effectiveness?
I agree that a lot of the difference in EAs donations can come from differing perspectives, probably most. But I think even some utilitarian, EV maximizing, 0-future discount, animal equalists EAs donate to different causes (or any other set of shared beliefs). It’s definitely not impossible.
As for other examples of 10x GiveWell cost-effectiveness in global health:
CE has estimated another charity yields $5.62 per DALY.
An Israeli non-profit, which produced an estimate of 4.3$ QALYs per dollar, in cooperation with EA Israel. A volunteer said to me he believed they were about 3x too optimistic, but that’s still around 10x GiveWell.
Also here is an example of 4x disagreement between GiveWell and Founders Pledge, and an even bigger disagreement with RP, on a mass media campaign for family planning. Even the best in the business can disagree.
Sorry for this being a bit of a rave