Thank Jamie, I think cause prioritisation is super important as you say, but I don’t think its as neglected as you think, at least not within the scope of global health and wellbeing. I agree that the substance of your the 3 part list being important, but I wouldn’t consider the list the best measure of how much hard cause prioritisation work has been done. It seems a bit strawman-ish as I think there are good reasons (see below) why those “exact” things aren’t being done.
First I think precise ranking of “cause areas”is nearly impossible as its hard to meaningfully calculate the “cost-effectiveness” of a cause, you can only accurately calculate the cost-effectiveness of an intervention which specifically targets that cause. So if you did want a meaningful rank, you at least need to have an intervention which has probably already been tried and researched to some degree at least.
Secondly I think having public specific rankings has potential to be both meaningless and reputationally dangerous. I think clustering the best interventions we know of and sharing the estimated cost effectiveness is fantastic (like Givewell, 80,000 hours, CEARCH and Copenhagen do), but I don’t think adding ranked specificity is very helpful because....
Uncertainty is so high and confidence intervals so wide in these calculations that specific rankings can be fairly meaningless. When all confidence intervals for interventions overlap, I think providing a specific ranking can be almost dishonest
Specific public rankings for causes/interventions has the potential downside of being inflammatory and unhelpful for the effective altruism movement. We’ve already seen some obvious backlash and downside of the big plug for working towards AI safety being put forward as something like “the most important” intervention. Imagine if orgs were publicly pushing seemingly concrete rankings? Much of the public and intellectual world is likely to misunderstand the purpose of it and criticise, or even understand well and criticizse...
I think that 80,000 hours, Open Phil and CEARCH do the substance what you are looking for pretty well and put a lot of money and hours into it—I don’t think hard work in this area is “surprisingly rare” I’m not sure if adding a whole lot more organisations here would achieve much, but there might be more room for efforts there!
Also I personally think that GiveWell might do the most work which achieves the substance of what you are looking for within global health and wellbeing. They are devoted to finding the most cost effective interventions in the world that exist right now. Their “top charities” page is in some ways a handful of what they think are the “no 1″ ranked interventions. Yes they only consider interventions with a lot of evidence behind them and are fairly conservative but I think it achieves much of the substance of your 3 steps.
I’d be interested to hear what you think might be the upsides of “ranking” specifically vs clustering our best estimates at effective cause areas/interventions.
I’d be interested to get comment from @Joel Tan here as he and the CEARCH team have probably considered this question more than most of us
Very helpful comment, thank you for taking the time to write out this reply and sharing useful reflections and resources!
First I think precise ranking of “cause areas”is nearly impossible as its hard to meaningfully calculate the “cost-effectiveness” of a cause, you can only accurately calculate the cost-effectiveness of an intervention which specifically targets that cause. So if you did want a meaningful rank, you at least need to have an intervention which has probably already been tried and researched to some degree at least.
There’s a lot going on here. I suspect I’m more optimistic than you that sharing uncertain but specific rankings is helpful for clarifying views and making progress? I agree in principle that what we want to do is evaluate specific actions (“interventions”), but I still think you can rank expected cost-effectiveness at a slightly more zoomed-out level, as long as you are comparing across roughly similar levels of abstraction. (Implicitly, you’re evaluating the average intervention in that category, rather than a single intervention.) Given these things, I don’t think I endorse the view that “you at least need to have an intervention which has probably already been tried and researched to some degree at least.”
Secondly I think having public specific rankings has potential to be both meaningless and reputationally dangerous.
I agree with the reputational risks and the potential for people to misunderstand your claim or think that it’s more confident than it is, etc. I somewhat suspect that this will be mitigated by there just being more such rankings though, as well as having clear disclaimers. E.g. at the moment, people might look at 80k and Open Phil rankings and conclude that there must be strong evidence behind the ratings. But if they see that there are 5 different ranked lists with only some amount of overlap, it’s implicitly pretty clear that there’s a lot of subjectivity and difficult decision-making going into this. (I don’t agree with it being “meaningless” or “dishonest”—I think that relates to the points above.)
Also I personally think that GiveWell might do the most work which achieves the substance of what you are looking for within global health and wellbeing. Also like you mentioned the Copenhagen Consensus also does a pretty good job of outlining what they think might be the 12 best interventions (best things first) with much reasoning and calculation behind each one.
Thanks a lot for these pointers! I will look into them more carefully. This is exactly the sort of thing I was hoping to receive in response to this quick take, so thanks a lot for your help. Best Things First sounds great and I’ve added it to my Audible wishlist. Is this what you have in mind for GiveWell? (Context: I’m not very familiar with global health.)
I’d be interested to hear what you think might be the upsides of “ranking” specifically vs clustering our best estimates at effective cause areas/interventions.
Oh this might have just been me using unintentionally specific language. I would have included “tiered” lists as part of “ranked”. Indeed the Open Phil list is tiered rather than numerically ranked. Thank you for highlighting this though, I’ve edited the original post to add the word “tiered”. (Is that what you meant by “clustering our best estimates at effective cause areas/interventions? Lmk if you meant something else.)
I think Nick is fundamentally correct that because uncertainty is so high, sorting isn’t particularly useful. Most grantmaking organizations, to my understanding, prefer to use a cost-effectiveness threshold/funding bar, to decide whether or not to recommend/support a particular cause/intervention/charity.
For ourselves, we use 10x GiveWell for GHD, as (a) most of the money we move is EA and the counterfactual is GiveWell (so to have impact we the ideas we redirect funding/talent to be more cost-effective than GiveWell in expectation, and (b) we have such an aggressive bar because GiveWell is very robust in their discounting relative to us (which takes a lot of time and effort). An aggressive bar helps ensure that even if your estimated cost-effectiveness estimate is too optimistic relative to GiveWell, it can eat a lot of implicit discounts while still ensuring that the true cost-effectiveness is >GiveWell. (so when we say something is >=10x GiveWell it’s not literally so, more of a reasonably high confidence claim that it’s probably more cost-effective (in expectation).
I understand the reasons for ranking relative to a given cost-effectiveness bar (or by a given cost-effectiveness metric). That provides more information than constraining the ranking to a numerical list so I appreciate that.
Btw, if you had 5-10 mins spare I think it’d be really helpful to add explanation notes to the cells in the top row of the spreadsheet. E.g. I don’t know what “MEV” stands for, or what the “cost-effectiveness” or “cause no.” columns are referring to. (Currently these things mean that I probably won’t share the spreadsheet with people because I’d need to do a lot of explaining or caveating to them, whereas I’d be more likely to share it if it was more self-explanatory.)
Hi Jaime, I’ve updated to clarify that the “MEV” column is just “DALYs per USD 100,000″. Have hidden some of the other columns (they’re just for internal administrative/labelling purposes).
Thank Jamie, I think cause prioritisation is super important as you say, but I don’t think its as neglected as you think, at least not within the scope of global health and wellbeing. I agree that the substance of your the 3 part list being important, but I wouldn’t consider the list the best measure of how much hard cause prioritisation work has been done. It seems a bit strawman-ish as I think there are good reasons (see below) why those “exact” things aren’t being done.
First I think precise ranking of “cause areas”is nearly impossible as its hard to meaningfully calculate the “cost-effectiveness” of a cause, you can only accurately calculate the cost-effectiveness of an intervention which specifically targets that cause. So if you did want a meaningful rank, you at least need to have an intervention which has probably already been tried and researched to some degree at least.
Secondly I think having public specific rankings has potential to be both meaningless and reputationally dangerous. I think clustering the best interventions we know of and sharing the estimated cost effectiveness is fantastic (like Givewell, 80,000 hours, CEARCH and Copenhagen do), but I don’t think adding ranked specificity is very helpful because....
Uncertainty is so high and confidence intervals so wide in these calculations that specific rankings can be fairly meaningless. When all confidence intervals for interventions overlap, I think providing a specific ranking can be almost dishonest
Specific public rankings for causes/interventions has the potential downside of being inflammatory and unhelpful for the effective altruism movement. We’ve already seen some obvious backlash and downside of the big plug for working towards AI safety being put forward as something like “the most important” intervention. Imagine if orgs were publicly pushing seemingly concrete rankings? Much of the public and intellectual world is likely to misunderstand the purpose of it and criticise, or even understand well and criticizse...
I think that 80,000 hours, Open Phil and CEARCH do the substance what you are looking for pretty well and put a lot of money and hours into it—I don’t think hard work in this area is “surprisingly rare” I’m not sure if adding a whole lot more organisations here would achieve much, but there might be more room for efforts there!
Also I personally think that GiveWell might do the most work which achieves the substance of what you are looking for within global health and wellbeing. They are devoted to finding the most cost effective interventions in the world that exist right now. Their “top charities” page is in some ways a handful of what they think are the “no 1″ ranked interventions. Yes they only consider interventions with a lot of evidence behind them and are fairly conservative but I think it achieves much of the substance of your 3 steps.
Also like you mentioned the Copenhagen Consensus also does a pretty good job of outlining what they think might be the 12 best interventions (best things first) with much reasoning and calculation behind each one. This is not ft off a straight rank
I’d be interested to hear what you think might be the upsides of “ranking” specifically vs clustering our best estimates at effective cause areas/interventions.
I’d be interested to get comment from @Joel Tan here as he and the CEARCH team have probably considered this question more than most of us
Interesting point thanks!
Very helpful comment, thank you for taking the time to write out this reply and sharing useful reflections and resources!
There’s a lot going on here. I suspect I’m more optimistic than you that sharing uncertain but specific rankings is helpful for clarifying views and making progress? I agree in principle that what we want to do is evaluate specific actions (“interventions”), but I still think you can rank expected cost-effectiveness at a slightly more zoomed-out level, as long as you are comparing across roughly similar levels of abstraction. (Implicitly, you’re evaluating the average intervention in that category, rather than a single intervention.) Given these things, I don’t think I endorse the view that “you at least need to have an intervention which has probably already been tried and researched to some degree at least.”
I agree with the reputational risks and the potential for people to misunderstand your claim or think that it’s more confident than it is, etc. I somewhat suspect that this will be mitigated by there just being more such rankings though, as well as having clear disclaimers. E.g. at the moment, people might look at 80k and Open Phil rankings and conclude that there must be strong evidence behind the ratings. But if they see that there are 5 different ranked lists with only some amount of overlap, it’s implicitly pretty clear that there’s a lot of subjectivity and difficult decision-making going into this. (I don’t agree with it being “meaningless” or “dishonest”—I think that relates to the points above.)
Thanks a lot for these pointers! I will look into them more carefully. This is exactly the sort of thing I was hoping to receive in response to this quick take, so thanks a lot for your help. Best Things First sounds great and I’ve added it to my Audible wishlist. Is this what you have in mind for GiveWell? (Context: I’m not very familiar with global health.)
Oh this might have just been me using unintentionally specific language. I would have included “tiered” lists as part of “ranked”. Indeed the Open Phil list is tiered rather than numerically ranked. Thank you for highlighting this though, I’ve edited the original post to add the word “tiered”. (Is that what you meant by “clustering our best estimates at effective cause areas/interventions? Lmk if you meant something else.)
Thanks again!
Thanks for the thoughts, Jaime and Nick!
For what it’s worth, CEARCH’s list of evaluated causes (or more specifically, top interventions in various causes) and their estimated cost-effectiveness is here: https://docs.google.com/spreadsheets/d/14y9IGAyS6s4kbDLGQCI6_qOhqnbn2jhCfF1o2GfyjQg/edit#gid=0
I think Nick is fundamentally correct that because uncertainty is so high, sorting isn’t particularly useful. Most grantmaking organizations, to my understanding, prefer to use a cost-effectiveness threshold/funding bar, to decide whether or not to recommend/support a particular cause/intervention/charity.
For ourselves, we use 10x GiveWell for GHD, as (a) most of the money we move is EA and the counterfactual is GiveWell (so to have impact we the ideas we redirect funding/talent to be more cost-effective than GiveWell in expectation, and (b) we have such an aggressive bar because GiveWell is very robust in their discounting relative to us (which takes a lot of time and effort). An aggressive bar helps ensure that even if your estimated cost-effectiveness estimate is too optimistic relative to GiveWell, it can eat a lot of implicit discounts while still ensuring that the true cost-effectiveness is >GiveWell. (so when we say something is >=10x GiveWell it’s not literally so, more of a reasonably high confidence claim that it’s probably more cost-effective (in expectation).
Thank you!
I understand the reasons for ranking relative to a given cost-effectiveness bar (or by a given cost-effectiveness metric). That provides more information than constraining the ranking to a numerical list so I appreciate that.
Btw, if you had 5-10 mins spare I think it’d be really helpful to add explanation notes to the cells in the top row of the spreadsheet. E.g. I don’t know what “MEV” stands for, or what the “cost-effectiveness” or “cause no.” columns are referring to. (Currently these things mean that I probably won’t share the spreadsheet with people because I’d need to do a lot of explaining or caveating to them, whereas I’d be more likely to share it if it was more self-explanatory.)
Hi Jaime, I’ve updated to clarify that the “MEV” column is just “DALYs per USD 100,000″. Have hidden some of the other columns (they’re just for internal administrative/labelling purposes).