After chatting with Alex Gordon-Brown, I updated significantly towards his position, which I’ve attempted to summarise below. Many thanks to him for taking the time to talk; I’ve done my best to accurately represent the conversation, but there may be mistakes. All of the following are conditional on focusing on near-term, human-centric charities.
Three key things I changed my mind on:
I had mentally characterised EA as starting with Givewell-style reasoning, and then moving on to less quantifiable things. Whereas Alex (who was around at the time) pointed out that there were originally significant disagreements between EAs and Givewell, in particular with EAs arguing for less quantifiable approaches. EA and Givewell then ended up converging more over time, both as EAs found that it was surprisingly hard to beat Givewell charities even allowing for less rigorous analysis, and also as people at Givewell (e.g. the ones now running OpenPhil) became more convinced in less-quantifiable EA methodologies.
Insofar as the wider world has the impression of EA as synonymous with Givewell-style reasoning, a lot of that comes from media reports focusing on it in ways we weren’t responsible for.
Alex claims that Doing Good Better, which also leans in this direction, wasn’t fully representative of the beliefs of core EAs at the time it was published.
Alex says that OpenPhil has found Givewell charities surprisingly hard to beat, and that this (along with other EA knowledge and arguments, such as the 100x multiplier) is sufficient to make a “compelling case” for them.
Alex acknowledges that not many people who recommend Givewell are doing so because of this evidence; in some sense, it’s a “happy coincidence” that the thing people were already recommending has been vindicated. But he thinks that there are enough careful EAs who pay attention to OpenPhil’s reports that, if their conclusions had been the opposite, I would have heard people publicly making this case.
Alex argues that I’m overly steelmanning the criticism that EA has received. EA spent a lot of time responding to criticisms that it’s impossible to know that any charities are doing a lot of good (e.g. because of potential corruption, and so on), and criticisms that we should care more about people near us, and so on. Even when it came to “systemic change” critiques, these usually weren’t principled critiques about the importance of systemic change in general, but rather just “you should focus on my personal cause”, in particular highly politicised causes.
Alex also notes that the Givewell headline claim “We search for the charities that save or improve lives the most per dollar” is relatively new (here’s an earlier version) and has already received criticism.
Things I haven’t changed my mind about:
I still think that most individual EAs should be much more careful in recommending Givewell charities. OpenPhil’s conclusions are based primarily off (in their words) “back-of-the-envelope calculations”, the details of which we don’t know. I think that, even if this is enough to satisfy people who trust OpenPhil’s researchers and their methodologies, it’s far less legible and rigorous than most people who hear about EA endorsement of Givewell charities would expect. Indeed, they still conclude that (in expectation) their hits-based portfolio will moderately outperform Givewell.
OpenPhil’s claims are personally not enough to satisfy me. I think by default I won’t endorse Givewell charities. Instead I’ll abstain from having an opinion on what the best near-term human-centric charities are, and push for something more longtermist like pandemic prevention as a “default” outreach cause area instead. But I also don’t think it’s unreasonable for other people to endorse Givewell charities under the EA name.
I still think that the 100x multiplier argument is (roughly) cancelled out by the multiplier going the other way, of wealthy countries having at least 100x more influence over the world. So, while it’s still a good argument for trying to help the poorest people, it doesn’t seem like a compelling argument for trying to help the poorest people via direct interventions in poor countries.
Overall lessons: I overestimated the extent to which my bubble was representative of EA, and also the extent to which I understood the history of EA accurately.
Alex and I finished by briefly discussing AI safety, where I’m quite concerned about a lack of justification for many of the claims EAs make. I’m hoping to address this more elsewhere.
Thanks for the write-up. A few quick additional thoughts on my end:
You note that OpenPhil still expect their hits-based portfolio to moderately outperform Givewell in expectation. This is my understanding also, but one slight difference of interpretation is that it leaves me very baseline skeptical that most ‘systemic change’ charities people suggest would also outperform, given the amount of time Open Phil has put into this question relative to the average donor.
I think it’s possible-to-likely I’m mirroring your ‘overestimating how representative my bubble was’ mistake, despite having explicitly flagged this type of error before because it’s so common. In particular, many (most?) EAs first encounter the community at university, whereas my first encounter was after university, and it wouldn’t shock me if student groups were making more strident/overconfident claims than I remember in my own circles. On reflection I now have anecdotal evidence of this from 3 different groups.
Abstaning on the ‘what is the best near-term human-centric charity’ question, and focusing on talking about the things that actually appear to you to be among the best options, is a response I strongly support. I really wish more longtermists took this approach, and I also wish EAs in general would use ‘we’ less and ‘I’ more when talking about what they think about optimal opportunities to do good.
it leaves me very baseline skeptical that most ‘systemic change’ charities people suggest would also outperform, given the amount of time Open Phil has put into this question relative to the average donor.
I have now read OpenPhil’s sample of the back-of-the-envelope calculations on which their conclusion that it’s hard to beat GiveWell was based. They were much rougher than I expected. Most of them are literally just an estimate of the direct benefits and costs, with no accounting for second-order benefits or harms, movement-building effects, political effects, etc. For example, the harm of a year of jail time is calculated as 0.5 QALYs plus the financial cost to the government—nothing about long-term effects of spending time in jail, or effects on subsequent crime rates, or community effects. I’m not saying that OpenPhil should have included these effects, they are clear that these are only intended as very rough estimates, but it means that I now don’t think it’s justified to treat this blog post as strong evidence in favour of GiveWell.
Here’s just a basic (low-confidence) case for the cost-efficacy of political advocacy: governmental policies can have enormous effects, even when they attract little mainstream attention (e.g. PEPFAR). But actually campaigning for a specific policy is often only the last step in the long chain of getting the cause into the Overton Window, building a movement, nurturing relationships with politicians, identifying tractable targets, and so on, all of which are very hard to measure, and which wouldn’t show up at all in these calculations by OpenPhil. Given this, what evidence is there that funding these steps wouldn’t outperform GiveWell for many policies?
Your other points make sense, although I’m now worried that abstaining about near-term human-centric charities will count as implicit endorsement. I don’t know very much about quantitatively analysing interventions though, so it’s plausible that my claims in this comment are wrong.
You seem to be implicitly focusing on the question ‘how certain are we these will turn out to be best’. I’m focusing on the question ‘Denise and I are likely to make a donation to near-term human-centric causes in the next few months; is there something I should be donating to above Givewell charities’.
Listing unaccounted-for second order effects is relevant for the first, but not decision-relevant until the effects are predictable-in-direction and large; it needs to actually impact my EV meaningfully. Currently, I’m not seeing a clear argument for that. ‘Might have wildly large impacts’, ‘very rough estimates’, ‘policy can have enormous effects’...these are all phrases that increase uncertainty rather than concretely change EVs and so are decision-irrelevant. (That’s not quite true; we should penalise rough things’ calculated EV more in high-uncertainty environments due to winners’ curse effects, but that’s secondary to my main point here).
Another way of putting it is that this is the difference between one’s confidence level that what you currently think is best will still be what you think is best 20 years from now, versus trying to identify the best all-things-considered donation opportunity right now with one’s limited information.
So concretely, I think it’s very likely that in 20 years I’ll think one of the >20 alternatives I’ve briefly considered will look like it was a better use of my money that Givewell charities, due to the uncertainty you’re highlighting. But I don’t know which one, and I don’t expect it to outperform 20x, so picking one essentially at random still looks pretty bad.
A non-random way to pick would be if Open Phil, or someone else I respect, shifted their equivalent donation bucket to some alternative. AFAIK, this hasn’t happened. That’s the relevance of those decisions to me, rather than any belief that they’ve done a secret Uber-Analysis.
Hmm, I agree that we’re talking past each other. I don’t intend to focus on ex post evaluations over ex ante evaluations. What I intend to focus on is the question: “when an EA make the claim that GiveWell charities are the charities with the strongest case for impact in near-term human-centric terms, how justified are they?” Or, relatedly, “How likely is it that somebody who is motivated to find the best near-term human-centric charities possible, but takes a very different approach than EA does (in particular by focusing much more on hard-to-measure political effects) will do better than EA?”
In my previous comment, I used a lot of phrases which you took to indicate the high uncertainty of political interventions. My main point was that it’s plausible that a bunch of them exist which will wildly outperform GiveWell charities. I agree I don’t know which one, and you don’t know which one, and GiveWell doesn’t know which one. But for the purposes of my questions above, that’s not the relevant factor; the relevant factor is: does someone know, and have they made those arguments publicly, in a way that we could learn from if we were more open to less quantitative analysis? (Alternatively, could someone know if they tried? But let’s go with the former for now.)
In other words, consider two possible worlds. In one world GiveWell charities are in fact the most cost-effective, and all the people doing political advocacy are less cost-effective than GiveWell ex ante (given publicly available information). In the other world there’s a bunch of people doing political advocacy work which EA hasn’t supported even though they have strong, well-justified arguments that their work is very impactful (more impactful than GiveWell’s top charities), because that impact is hard to quantitatively estimate. What evidence do we have that we’re not in the second world? In both worlds GiveWell would be saying roughly the same thing (because they have a high bar for rigour). Would OpenPhil be saying different things in different worlds? Insofar as their arguments in favour of GiveWell are based on back-of-the-envelope calculations like the ones I just saw, then they’d be saying the same thing in both worlds, because those calculations seem insufficient to capture most of the value of the most cost-effective political advocacy. Insofar as their belief that it’s hard to beat GiveWell is based on other evidence which might distinguish between these two worlds, they don’t explain this in their blog post—which means I don’t think the post is strong evidence in favour of GiveWell top charities for people who don’t already trust OpenPhil a lot.
But for the purposes of my questions above, that’s not the relevant factor; the relevant factor is: does someone know, and have they made those arguments [that specific intervention X will wildly outperform] publicly, in a way that we could learn from if we were more open to less quantitative analysis?
I agree with this. I think the best way to settle this question is to link to actual examples of someone making such arguments. Personally, my observation from engaging with non-EA advocates of political advocacy is that they don’t actually make a case; when I cash out people’s claims it usually turns out they are asserting 10x − 100x multipliers, not 100x − 1000x multipliers, let alone higher than that. It appears the divergence in our bottom lines is coming from my cosmopolitan values and low tolerance for act/omission distinctions, and hopefully we at least agree that if even the entrenched advocate doesn’t actually think their cause is best under my values, I should just move on.
As an aside, I know you wrote recently that you think more work is being done by EA’s empirical claims than moral claims. I think this is credible for longtermism but mostly false for Global Health/Poverty. People appear to agree they can save lives in the deveoping world incredibly cheaply, in fact usually giving lower numbers than I think are possible. We aren’t actually that far apart on the empirical state of affairs. They just don’t want to. They aren’t refusing to because they have even better things to do, because most people do very little. Or as Rob put it:
Many people donate a small fraction of their income, despite claiming to believe that lives can be saved for remarkably small amounts. This suggests they don’t believe they have a duty to give even if lives can be saved very cheaply – or that they are not very motivated by such a duty.
I think that last observation would also be my answer to ‘what evidence do we have that we aren’t in the second world?’ Empirically, most people don’t care, and most people who do care are not trying to optimise for the thing I am optimising for (in many cases it’s debateable whether they are trying to optimise at all). So it would be surprising if they hit the target anyway, in much the same way it would be surprising if AMF were the best way to improve animal welfare.
Thanks for writing this up! I’ve found this thread super interesting to follow, and it’s shifted my view on a few important points.
One lingering thing that seems super important is longtermism vs prioritising currently existing people. It still seems to me that GiveWell charities aren’t great from a longtermist perspective, but that the vast majority of people are not longtermists. Which creates a weird tension when doing outreach, since I rarely want to begin by trying to pitch longtermism, but it seems disingenuous to pitch GiveWell charities.
Given that many EAs are not longtermist though, this seems overall fine for the “is the movement massively misleading people” question
I don’t think that the moral differences between longtermists and most people in similar circles (e.g. WEIRD) are that relevant, actually. You don’t need to be a longtermist to care about massive technological change happening over the next century. So I think it’s straightforward to say things like “We should try to have a large-scale moral impact. One very relevant large-scale harm is humans going extinct; so we should work on things which prevent it”.
This is what I plan to use as a default pitch for EA from now on.
After chatting with Alex Gordon-Brown, I updated significantly towards his position, which I’ve attempted to summarise below. Many thanks to him for taking the time to talk; I’ve done my best to accurately represent the conversation, but there may be mistakes. All of the following are conditional on focusing on near-term, human-centric charities.
Three key things I changed my mind on:
I had mentally characterised EA as starting with Givewell-style reasoning, and then moving on to less quantifiable things. Whereas Alex (who was around at the time) pointed out that there were originally significant disagreements between EAs and Givewell, in particular with EAs arguing for less quantifiable approaches. EA and Givewell then ended up converging more over time, both as EAs found that it was surprisingly hard to beat Givewell charities even allowing for less rigorous analysis, and also as people at Givewell (e.g. the ones now running OpenPhil) became more convinced in less-quantifiable EA methodologies.
Insofar as the wider world has the impression of EA as synonymous with Givewell-style reasoning, a lot of that comes from media reports focusing on it in ways we weren’t responsible for.
Alex claims that Doing Good Better, which also leans in this direction, wasn’t fully representative of the beliefs of core EAs at the time it was published.
Alex says that OpenPhil has found Givewell charities surprisingly hard to beat, and that this (along with other EA knowledge and arguments, such as the 100x multiplier) is sufficient to make a “compelling case” for them.
Alex acknowledges that not many people who recommend Givewell are doing so because of this evidence; in some sense, it’s a “happy coincidence” that the thing people were already recommending has been vindicated. But he thinks that there are enough careful EAs who pay attention to OpenPhil’s reports that, if their conclusions had been the opposite, I would have heard people publicly making this case.
Alex argues that I’m overly steelmanning the criticism that EA has received. EA spent a lot of time responding to criticisms that it’s impossible to know that any charities are doing a lot of good (e.g. because of potential corruption, and so on), and criticisms that we should care more about people near us, and so on. Even when it came to “systemic change” critiques, these usually weren’t principled critiques about the importance of systemic change in general, but rather just “you should focus on my personal cause”, in particular highly politicised causes.
Alex also notes that the Givewell headline claim “We search for the charities that save or improve lives the most per dollar” is relatively new (here’s an earlier version) and has already received criticism.
Things I haven’t changed my mind about:
I still think that most individual EAs should be much more careful in recommending Givewell charities. OpenPhil’s conclusions are based primarily off (in their words) “back-of-the-envelope calculations”, the details of which we don’t know. I think that, even if this is enough to satisfy people who trust OpenPhil’s researchers and their methodologies, it’s far less legible and rigorous than most people who hear about EA endorsement of Givewell charities would expect. Indeed, they still conclude that (in expectation) their hits-based portfolio will moderately outperform Givewell.
OpenPhil’s claims are personally not enough to satisfy me. I think by default I won’t endorse Givewell charities. Instead I’ll abstain from having an opinion on what the best near-term human-centric charities are, and push for something more longtermist like pandemic prevention as a “default” outreach cause area instead. But I also don’t think it’s unreasonable for other people to endorse Givewell charities under the EA name.
I still think that the 100x multiplier argument is (roughly) cancelled out by the multiplier going the other way, of wealthy countries having at least 100x more influence over the world. So, while it’s still a good argument for trying to help the poorest people, it doesn’t seem like a compelling argument for trying to help the poorest people via direct interventions in poor countries.
Overall lessons: I overestimated the extent to which my bubble was representative of EA, and also the extent to which I understood the history of EA accurately.
Alex and I finished by briefly discussing AI safety, where I’m quite concerned about a lack of justification for many of the claims EAs make. I’m hoping to address this more elsewhere.
Thanks for the write-up. A few quick additional thoughts on my end:
You note that OpenPhil still expect their hits-based portfolio to moderately outperform Givewell in expectation. This is my understanding also, but one slight difference of interpretation is that it leaves me very baseline skeptical that most ‘systemic change’ charities people suggest would also outperform, given the amount of time Open Phil has put into this question relative to the average donor.
I think it’s possible-to-likely I’m mirroring your ‘overestimating how representative my bubble was’ mistake, despite having explicitly flagged this type of error before because it’s so common. In particular, many (most?) EAs first encounter the community at university, whereas my first encounter was after university, and it wouldn’t shock me if student groups were making more strident/overconfident claims than I remember in my own circles. On reflection I now have anecdotal evidence of this from 3 different groups.
Abstaning on the ‘what is the best near-term human-centric charity’ question, and focusing on talking about the things that actually appear to you to be among the best options, is a response I strongly support. I really wish more longtermists took this approach, and I also wish EAs in general would use ‘we’ less and ‘I’ more when talking about what they think about optimal opportunities to do good.
I have now read OpenPhil’s sample of the back-of-the-envelope calculations on which their conclusion that it’s hard to beat GiveWell was based. They were much rougher than I expected. Most of them are literally just an estimate of the direct benefits and costs, with no accounting for second-order benefits or harms, movement-building effects, political effects, etc. For example, the harm of a year of jail time is calculated as 0.5 QALYs plus the financial cost to the government—nothing about long-term effects of spending time in jail, or effects on subsequent crime rates, or community effects. I’m not saying that OpenPhil should have included these effects, they are clear that these are only intended as very rough estimates, but it means that I now don’t think it’s justified to treat this blog post as strong evidence in favour of GiveWell.
Here’s just a basic (low-confidence) case for the cost-efficacy of political advocacy: governmental policies can have enormous effects, even when they attract little mainstream attention (e.g. PEPFAR). But actually campaigning for a specific policy is often only the last step in the long chain of getting the cause into the Overton Window, building a movement, nurturing relationships with politicians, identifying tractable targets, and so on, all of which are very hard to measure, and which wouldn’t show up at all in these calculations by OpenPhil. Given this, what evidence is there that funding these steps wouldn’t outperform GiveWell for many policies?
(See also Scott Alexander ’s rough calculations on the effects of FDA regulations, which I’m not very confident in, but which have always stuck in my head as an argument that how dull-sounding policies might have wildly large impacts.)
Your other points make sense, although I’m now worried that abstaining about near-term human-centric charities will count as implicit endorsement. I don’t know very much about quantitatively analysing interventions though, so it’s plausible that my claims in this comment are wrong.
I think we’re still talking past each other here.
You seem to be implicitly focusing on the question ‘how certain are we these will turn out to be best’. I’m focusing on the question ‘Denise and I are likely to make a donation to near-term human-centric causes in the next few months; is there something I should be donating to above Givewell charities’.
Listing unaccounted-for second order effects is relevant for the first, but not decision-relevant until the effects are predictable-in-direction and large; it needs to actually impact my EV meaningfully. Currently, I’m not seeing a clear argument for that. ‘Might have wildly large impacts’, ‘very rough estimates’, ‘policy can have enormous effects’...these are all phrases that increase uncertainty rather than concretely change EVs and so are decision-irrelevant. (That’s not quite true; we should penalise rough things’ calculated EV more in high-uncertainty environments due to winners’ curse effects, but that’s secondary to my main point here).
Another way of putting it is that this is the difference between one’s confidence level that what you currently think is best will still be what you think is best 20 years from now, versus trying to identify the best all-things-considered donation opportunity right now with one’s limited information.
So concretely, I think it’s very likely that in 20 years I’ll think one of the >20 alternatives I’ve briefly considered will look like it was a better use of my money that Givewell charities, due to the uncertainty you’re highlighting. But I don’t know which one, and I don’t expect it to outperform 20x, so picking one essentially at random still looks pretty bad.
A non-random way to pick would be if Open Phil, or someone else I respect, shifted their equivalent donation bucket to some alternative. AFAIK, this hasn’t happened. That’s the relevance of those decisions to me, rather than any belief that they’ve done a secret Uber-Analysis.
Hmm, I agree that we’re talking past each other. I don’t intend to focus on ex post evaluations over ex ante evaluations. What I intend to focus on is the question: “when an EA make the claim that GiveWell charities are the charities with the strongest case for impact in near-term human-centric terms, how justified are they?” Or, relatedly, “How likely is it that somebody who is motivated to find the best near-term human-centric charities possible, but takes a very different approach than EA does (in particular by focusing much more on hard-to-measure political effects) will do better than EA?”
In my previous comment, I used a lot of phrases which you took to indicate the high uncertainty of political interventions. My main point was that it’s plausible that a bunch of them exist which will wildly outperform GiveWell charities. I agree I don’t know which one, and you don’t know which one, and GiveWell doesn’t know which one. But for the purposes of my questions above, that’s not the relevant factor; the relevant factor is: does someone know, and have they made those arguments publicly, in a way that we could learn from if we were more open to less quantitative analysis? (Alternatively, could someone know if they tried? But let’s go with the former for now.)
In other words, consider two possible worlds. In one world GiveWell charities are in fact the most cost-effective, and all the people doing political advocacy are less cost-effective than GiveWell ex ante (given publicly available information). In the other world there’s a bunch of people doing political advocacy work which EA hasn’t supported even though they have strong, well-justified arguments that their work is very impactful (more impactful than GiveWell’s top charities), because that impact is hard to quantitatively estimate. What evidence do we have that we’re not in the second world? In both worlds GiveWell would be saying roughly the same thing (because they have a high bar for rigour). Would OpenPhil be saying different things in different worlds? Insofar as their arguments in favour of GiveWell are based on back-of-the-envelope calculations like the ones I just saw, then they’d be saying the same thing in both worlds, because those calculations seem insufficient to capture most of the value of the most cost-effective political advocacy. Insofar as their belief that it’s hard to beat GiveWell is based on other evidence which might distinguish between these two worlds, they don’t explain this in their blog post—which means I don’t think the post is strong evidence in favour of GiveWell top charities for people who don’t already trust OpenPhil a lot.
I agree with this. I think the best way to settle this question is to link to actual examples of someone making such arguments. Personally, my observation from engaging with non-EA advocates of political advocacy is that they don’t actually make a case; when I cash out people’s claims it usually turns out they are asserting 10x − 100x multipliers, not 100x − 1000x multipliers, let alone higher than that. It appears the divergence in our bottom lines is coming from my cosmopolitan values and low tolerance for act/omission distinctions, and hopefully we at least agree that if even the entrenched advocate doesn’t actually think their cause is best under my values, I should just move on.
As an aside, I know you wrote recently that you think more work is being done by EA’s empirical claims than moral claims. I think this is credible for longtermism but mostly false for Global Health/Poverty. People appear to agree they can save lives in the deveoping world incredibly cheaply, in fact usually giving lower numbers than I think are possible. We aren’t actually that far apart on the empirical state of affairs. They just don’t want to. They aren’t refusing to because they have even better things to do, because most people do very little. Or as Rob put it:
I think that last observation would also be my answer to ‘what evidence do we have that we aren’t in the second world?’ Empirically, most people don’t care, and most people who do care are not trying to optimise for the thing I am optimising for (in many cases it’s debateable whether they are trying to optimise at all). So it would be surprising if they hit the target anyway, in much the same way it would be surprising if AMF were the best way to improve animal welfare.
Thanks for writing this up! I’ve found this thread super interesting to follow, and it’s shifted my view on a few important points.
One lingering thing that seems super important is longtermism vs prioritising currently existing people. It still seems to me that GiveWell charities aren’t great from a longtermist perspective, but that the vast majority of people are not longtermists. Which creates a weird tension when doing outreach, since I rarely want to begin by trying to pitch longtermism, but it seems disingenuous to pitch GiveWell charities.
Given that many EAs are not longtermist though, this seems overall fine for the “is the movement massively misleading people” question
I don’t think that the moral differences between longtermists and most people in similar circles (e.g. WEIRD) are that relevant, actually. You don’t need to be a longtermist to care about massive technological change happening over the next century. So I think it’s straightforward to say things like “We should try to have a large-scale moral impact. One very relevant large-scale harm is humans going extinct; so we should work on things which prevent it”.
This is what I plan to use as a default pitch for EA from now on.