Great job writing this up Michael. I’d like to see many more people explaining their reasoning like this.
I was a bit surprised, however, to see 80,000 Hours as listed as “unclear has positive effective” when the charity you conclude is best, REG, likely wouldn’t exist if it weren’t for 80,000 Hours.
Similarly, your other finalist, ACE, is a spin-off of 80,000 Hours.
A number of other organisations on your shortlist have also been boosted by us, including: CSER (received seed funding from someone etg in part due to us), Charity Science (Joey and Xio are 80k plan changes, they also received seed funding from etg donors, they recently hired someone who decided to work in EA orgs due in part to 80k), GWWC (recently hired someone who switched to EA orgs in part due to us), FHI (now managed by Niel Bowerman, an 80k plan change)...
This sounds worryingly close to claiming credit for all “etg donors”, all EAs’ careers and all EA organisations that have had some contact with EA organizations. Of course people like Jonas Vollmer are going to say nice things about 80,000 Hours when asked, and it would be impolitic for any organisation to challenge this, so I’ll say it: I don’t think all of GBS Switzerland’s activities can be classed as counterfactually dependent on 80,000 Hours getting funding. Likewise the volunteers who founded Effective Animal Activism (the predecessor of ACE) or CSER or Effective Fundraising (the predecessor of Charity Science) might have done so at some point anyway, for all I know, and it’s hard to buy their saying otherwise as unbiased.
This isn’t to single out 80,000 Hours as the only organisation with these murky counterfactuals, it’s only jumping off your comment. I’ve likewise heard people say that people were running fundraisers before Charity Science started recruiting people to do so and that people were giving (or, if students, planning to) before signing up to Giving What We Can’s list, and that neither organisation can claim credit for everything these people then go on to do.
I agree the counterfactuals are murky, so I’d never say it was 100% due to us. Nevertheless, I think we played a significant role.
We also certainly don’t claim credit for all etg donors, only those who say they were influenced by us and made a significant plan change (something like 25-50% of the total).
Thanks for writing this Ben! I would like to see more representatives from orgs giving cases like this one where you go beyond saying “we’re high impact” and explain why you believe you’re the most high impact.
Here’s the main reasons why I didn’t consider 80K further:
Based on my prior knowledge of 80K and the brief time I spent investigating it, I didn’t see good evidence that 80K played an important causal role in pushing people toward substantially better careers. Similarly, I don’t see much reason to believe that those organizations you listed wouldn’t have happened without 80K.
Some of 80K’s recommendations confuse me and seem wrong. I agree with Peter Hurford’s recent post about the importance of earning to give. I’m concerned because 80K’s current stance on etg looks fairly obviously wrong, and everyone I’ve talked to about this whose opinion I highly respect has agreed that it looks fairly obviously wrong. More generally, 80K’s public info on career recommendations focus more on individual fit and don’t say much about how much good different careers do or how to do maximal good through those careers.
It seems dubious that 80K could continue to have as large an impact as you claim it has had in the past.
80K is funded by YC and does not have clear room for more funding. I don’t know what 80K could do if they had more money from me that it can’t do now, and my donations may displace the donations of other donors.
I am open to considering donating to 80K more seriously if you can address these concerns and also give good reason to believe that the way 80K directs people’s careers is likely to have a larger positive impact on the long term future than e.g. reducing AI risk. It’s not obvious to me that 80K has a multiplicative effect in the same way REG does.
On organisations in particular you say “I don’t see much reason to believe that those organisations wouldn’t have happened without 80k”. The founders of those organisations say they likely wouldn’t have existed otherwise, why do you think the founders are wrong?
With ACE in particular: we came up with the idea, it was started by an intern working at 80k and initially housed within 80k, we introduced them to their first seed donors, and an 80k team member continues to play a role on the board.
2) I’m not sure Peter Hurford and 80k actually disagree on the proportion of ppl who should do etg. We say 15-25% in the long-run. He says 50% or perhaps higher, but then in the comments he clarifies (in the reply to AGB) that he means 50% of those choosing between etg and direct work, and not counting those going into academic, policy, grant making etc. If you suppose 50% of people will do that, then Peter thinks 25% of people should etg all considered, in line with our estimate.
There’s also a few other differences in how we each frame the question and define etg which could easily explain remaining differences (see Will’s comments). I also listed a bunch of problems with Peter’s arguments on the thread which he didn’t yet address.
On 3), that’s a very big claim. Why? I expect the vast majority of 80k’s impact lies in the future. There’s the potential to develop a GiveWell-analogue but for career choice for all socially motivated graduates.
On 4), YC only provides $100,000 of funding once, so being YC-funded doesn’t mean we never need to fundraise again.
However, it’s true we haven’t publicly said we have room for more funding, so you have no way to know. I think we do have a large room for more funding though.
I think we have a multiplicative effect exactly like REG does, except we direct people to better careers rather than directing money.
Why don’t you think we’re moving people towards higher impact careers?
That’s not exactly what I said. What I said is that I don’t think there’s strong evidence that 80K is moving people toward higher impact careers.
80K’s impact page lists a bunch of career changes that people made after talking to 80K. But it’s not clear how many of these changes would have happened anyway or how much value 80K provided in the process. You also have to consider things that aren’t happening. If 80K claims credit for money donated by people who are now earning to give, then it should also subtract money not donated by people who now aren’t earning to give. The value of a career change isn’t from the value of the person’s career but from the marginal difference between their current career and their counterfactual career.
80K has moved people in lots of different directions and there’s no clear pattern I can see from public data. I’d expect that some careers are considerably more important and neglected than others, and 80K should be pushing people toward these careers in general, but I don’t see this happening.
On 3), if you believe most of 80K’s impact comes from helping start new effective charities (which you sort of imply I should believe when you claim that ACE and REG would not exist without 80K), then we should expect this effect to get a lot weaker in the future. I don’t think 80K played as big a role in creating ACE and REG as you say it did (there was a lot of demand for something like ACE when it came about so something similar probably would have arisen soon), but even if it did, creating new effective charities has rapidly diminishing marginal returns. The space of possible highly-effective charities (i.e. ones that are much more effective than top global poverty charities) is not that big.
On 4), there’s a huge gulf between “We don’t yet have all the money we could ever use” and “Giving us more funding would let us continue to be as effective as we have been with current funds.” You really only claim the former, but you have to establish the latter for 80K to be the best place to donate.
More people setting up or working in effective altruist charities
More people building career capital
More people working on xrisk
These are things people very unusually do otherwise, so it’s very unlikely they would happen without effective altruism or 80,000 Hours. Of course, it’s hard to untangle 80k’s impact on career choices from the rest of the EA movement, but it seems likely that 80k gets a substantial fraction of the impact. First, we’re the main group doing career stuff within the movement. Second, we’ve done a huge amount to boost the EA movement (e.g. we were the first org to use the term publicly), so if the EA movement has a lot of impact, then a significant fraction is due to us.
Note that a similar objection applies to the other charities you propose:
e.g. if REG / Charity Science / GWWC / GiveWell didn’t exist, much of the impact would happen anyway eventually because the other groups would eventually step in to compensate. But that doesn’t mean none of them are having much impact.
If 80K claims credit for money donated by people who are now earning to give, then it should also subtract money not donated by people who now aren’t earning to give. The value of a career change isn’t from the value of the person’s career but from the marginal difference between their current career and their counterfactual career.
Of course, we address this in the evaluation.
In short, I think in many of the cases the effectiveness boosts are very large, so when you subtract the impact they would have had anyway, it’s less than 10%. It depends on your view of how good “standard career choice” is.
On 3), if you believe most of 80K’s impact comes from helping start new effective charities (which you sort of imply I should believe when you claim that ACE and REG would not exist without 80K), then we should expect this effect to get a lot weaker in the future.
I’d say our impact comes from giving people better information about how to have a social impact in their career, and so redirecting them into higher impact career paths.
You can try to quantify a component of that by looking at additional donations due to our members, number of new organisations founded, or other measures.
So, new organisations founded is just one component of our impact that’s relatively tractable to analyse. More often, people assess us in terms of extra donations for charity raised from more people pursuing earning to give. Our estimate is that those earning to give will donate an extra $10m+ of counterfactually-adjusted funds to high-impact charities within the next 3 years because of us. I think either of these methods mean we’ve been very cost-effective in the past (historical financial costs are under $500k), and that’s ignoring over half the plan changes.
even if it did, creating new effective charities has rapidly diminishing marginal returns. The space of possible highly-effective charities (i.e. ones that are much more effective than top global poverty charities) is not that big.
It seems really unclear to me how close we are to that margin. Bear in mind explicitly effective altruist funding is under 0.04% of total US philanthropy. It seems like we could expand the number of organisations a great deal before hitting substantially diminishing returns. In particular when you consider how little research has been done, relatively speaking, it’s unlikely we discovered the best things already.
If we did run out of ideas for new organisations, 80k could move its focus to scaling up existing orgs. (Many people who’ve changed plans due to 80k have gone to work at existing EA orgs rather than found new ones). Or we could just encourage everyone to earn to give and donate to top global poverty charities.
Also, why you do you expect entrepreneurial-talent in EA to hit diminishing returns faster than donations? If anything, I expect we’ll hit diminishing returns to additional donations faster than with talent, because funding gaps are so much easier to resolve than talent-gaps (e.g. one billionaire could flood EA with money tomorrow). And that means REG also doesn’t have as much upside as it looks because in the future they won’t be able to direct the money as effectively as well.
There’s a huge gulf between “We don’t yet have all the money we could ever use” and “Giving us more funding would let us continue to be as effective as we have been with current funds.”
Of course. I actually think we’re going to be more effective with future funds because we’re getting better and better at changing plans, so our cost per plan change is falling. This is because our main focus in the past was learning and research, which is only just starting to pay off. There’s a lot more to say here though!
It feels like we’re getting off track here. You originally claimed that 80K played a large role in creating REG and ACE (the implication presumably being that I should donate to 80K). Now we’re talking about the strength of evidence on how 80K has changed people’s career paths.
Although its evidence is weaker than I’d like, I’m still fairly confident that 80K has a positive impact, and I’m glad it exists. I just don’t see that it’s the best place to donate. Are you trying to convince me that 80K’s activities are valuable, or that I should donate to it? If it’s the former, I already believe that. If the latter, you need to:
show why 80K has a higher impact than anything else
do more to make the strength of evidence supporting 80K more robust
demonstrate that 80K can effectively use marginal funds
Now that’s a pretty high bar, but I’m donating a lot of money and I want to make sure I direct it well.
In the original document you say next to 80k “unclear whether it has a positive effect”. So I was starting there.
REG and ACE are relevant because they’re examples of the value of the plan changes we cause. If you think 80k is changing plans such that more high impact organisations are created, then it’s likely 80k is also effective. (Though may of course still not be worth funding due to a lack of RFMF, but that’s not what you said initially).
Of course there’s a lot more to talk about. I’m always happy to answer more questions or share details about how marginal funds would be used via email.
Since it’s so sensitive to what “from a good university” or “altruistically motivated” mean, it would make more sense to argue over a few hypothetical marginal case studies.
Great job writing this up Michael. I’d like to see many more people explaining their reasoning like this.
I was a bit surprised, however, to see 80,000 Hours as listed as “unclear has positive effective” when the charity you conclude is best, REG, likely wouldn’t exist if it weren’t for 80,000 Hours.
Similarly, your other finalist, ACE, is a spin-off of 80,000 Hours.
https://80000hours.org/about/impact/new-organisations/
A number of other organisations on your shortlist have also been boosted by us, including: CSER (received seed funding from someone etg in part due to us), Charity Science (Joey and Xio are 80k plan changes, they also received seed funding from etg donors, they recently hired someone who decided to work in EA orgs due in part to 80k), GWWC (recently hired someone who switched to EA orgs in part due to us), FHI (now managed by Niel Bowerman, an 80k plan change)...
This sounds worryingly close to claiming credit for all “etg donors”, all EAs’ careers and all EA organisations that have had some contact with EA organizations. Of course people like Jonas Vollmer are going to say nice things about 80,000 Hours when asked, and it would be impolitic for any organisation to challenge this, so I’ll say it: I don’t think all of GBS Switzerland’s activities can be classed as counterfactually dependent on 80,000 Hours getting funding. Likewise the volunteers who founded Effective Animal Activism (the predecessor of ACE) or CSER or Effective Fundraising (the predecessor of Charity Science) might have done so at some point anyway, for all I know, and it’s hard to buy their saying otherwise as unbiased.
This isn’t to single out 80,000 Hours as the only organisation with these murky counterfactuals, it’s only jumping off your comment. I’ve likewise heard people say that people were running fundraisers before Charity Science started recruiting people to do so and that people were giving (or, if students, planning to) before signing up to Giving What We Can’s list, and that neither organisation can claim credit for everything these people then go on to do.
I agree the counterfactuals are murky, so I’d never say it was 100% due to us. Nevertheless, I think we played a significant role.
We also certainly don’t claim credit for all etg donors, only those who say they were influenced by us and made a significant plan change (something like 25-50% of the total).
Thanks for writing this Ben! I would like to see more representatives from orgs giving cases like this one where you go beyond saying “we’re high impact” and explain why you believe you’re the most high impact.
Here’s the main reasons why I didn’t consider 80K further:
Based on my prior knowledge of 80K and the brief time I spent investigating it, I didn’t see good evidence that 80K played an important causal role in pushing people toward substantially better careers. Similarly, I don’t see much reason to believe that those organizations you listed wouldn’t have happened without 80K.
Some of 80K’s recommendations confuse me and seem wrong. I agree with Peter Hurford’s recent post about the importance of earning to give. I’m concerned because 80K’s current stance on etg looks fairly obviously wrong, and everyone I’ve talked to about this whose opinion I highly respect has agreed that it looks fairly obviously wrong. More generally, 80K’s public info on career recommendations focus more on individual fit and don’t say much about how much good different careers do or how to do maximal good through those careers.
It seems dubious that 80K could continue to have as large an impact as you claim it has had in the past.
80K is funded by YC and does not have clear room for more funding. I don’t know what 80K could do if they had more money from me that it can’t do now, and my donations may displace the donations of other donors.
I am open to considering donating to 80K more seriously if you can address these concerns and also give good reason to believe that the way 80K directs people’s careers is likely to have a larger positive impact on the long term future than e.g. reducing AI risk. It’s not obvious to me that 80K has a multiplicative effect in the same way REG does.
Hi Michael,
On 1) have you seen our evaluation documents? https://80000hours.org/about/impact/ Why don’t you think we’re moving people towards higher impact careers?
On organisations in particular you say “I don’t see much reason to believe that those organisations wouldn’t have happened without 80k”. The founders of those organisations say they likely wouldn’t have existed otherwise, why do you think the founders are wrong?
With ACE in particular: we came up with the idea, it was started by an intern working at 80k and initially housed within 80k, we introduced them to their first seed donors, and an 80k team member continues to play a role on the board.
2) I’m not sure Peter Hurford and 80k actually disagree on the proportion of ppl who should do etg. We say 15-25% in the long-run. He says 50% or perhaps higher, but then in the comments he clarifies (in the reply to AGB) that he means 50% of those choosing between etg and direct work, and not counting those going into academic, policy, grant making etc. If you suppose 50% of people will do that, then Peter thinks 25% of people should etg all considered, in line with our estimate.
There’s also a few other differences in how we each frame the question and define etg which could easily explain remaining differences (see Will’s comments). I also listed a bunch of problems with Peter’s arguments on the thread which he didn’t yet address.
Our career research in general is highly focused on which careers do the most good (in the past we’ve mainly received criticism that we focus on personal fit too little—it’s quite hard to say what’s best to focus on if you want to maximise long-run impact https://80000hours.org/2014/10/interview-holden-karnofsky-on-the-importance-of-personal-fit/). We only list personal fit as one factor in our key principles: https://80000hours.org/career-guide/basics/ Our career reviews discuss impact just as much as personal fit: https://80000hours.org/career-guide/profiles/
On 3), that’s a very big claim. Why? I expect the vast majority of 80k’s impact lies in the future. There’s the potential to develop a GiveWell-analogue but for career choice for all socially motivated graduates.
On 4), YC only provides $100,000 of funding once, so being YC-funded doesn’t mean we never need to fundraise again.
However, it’s true we haven’t publicly said we have room for more funding, so you have no way to know. I think we do have a large room for more funding though.
I think we have a multiplicative effect exactly like REG does, except we direct people to better careers rather than directing money.
On 1)
That’s not exactly what I said. What I said is that I don’t think there’s strong evidence that 80K is moving people toward higher impact careers.
80K’s impact page lists a bunch of career changes that people made after talking to 80K. But it’s not clear how many of these changes would have happened anyway or how much value 80K provided in the process. You also have to consider things that aren’t happening. If 80K claims credit for money donated by people who are now earning to give, then it should also subtract money not donated by people who now aren’t earning to give. The value of a career change isn’t from the value of the person’s career but from the marginal difference between their current career and their counterfactual career.
80K has moved people in lots of different directions and there’s no clear pattern I can see from public data. I’d expect that some careers are considerably more important and neglected than others, and 80K should be pushing people toward these careers in general, but I don’t see this happening.
On 3), if you believe most of 80K’s impact comes from helping start new effective charities (which you sort of imply I should believe when you claim that ACE and REG would not exist without 80K), then we should expect this effect to get a lot weaker in the future. I don’t think 80K played as big a role in creating ACE and REG as you say it did (there was a lot of demand for something like ACE when it came about so something similar probably would have arisen soon), but even if it did, creating new effective charities has rapidly diminishing marginal returns. The space of possible highly-effective charities (i.e. ones that are much more effective than top global poverty charities) is not that big.
On 4), there’s a huge gulf between “We don’t yet have all the money we could ever use” and “Giving us more funding would let us continue to be as effective as we have been with current funds.” You really only claim the former, but you have to establish the latter for 80K to be the best place to donate.
Hey Michael,
It’s better to look at the evaluations rather than the list of studies if you want to get a systematic picture of career changes.
e.g. here: https://80000hours.org/2014/05/plan-change-analysis-and-cost-effectiveness/#what-were-the-changes
The most common changes are:
More people earning to give
More people setting up or working in effective altruist charities
More people building career capital
More people working on xrisk
These are things people very unusually do otherwise, so it’s very unlikely they would happen without effective altruism or 80,000 Hours. Of course, it’s hard to untangle 80k’s impact on career choices from the rest of the EA movement, but it seems likely that 80k gets a substantial fraction of the impact. First, we’re the main group doing career stuff within the movement. Second, we’ve done a huge amount to boost the EA movement (e.g. we were the first org to use the term publicly), so if the EA movement has a lot of impact, then a significant fraction is due to us.
Note that a similar objection applies to the other charities you propose: e.g. if REG / Charity Science / GWWC / GiveWell didn’t exist, much of the impact would happen anyway eventually because the other groups would eventually step in to compensate. But that doesn’t mean none of them are having much impact.
Of course, we address this in the evaluation.
In short, I think in many of the cases the effectiveness boosts are very large, so when you subtract the impact they would have had anyway, it’s less than 10%. It depends on your view of how good “standard career choice” is.
I’d say our impact comes from giving people better information about how to have a social impact in their career, and so redirecting them into higher impact career paths.
You can try to quantify a component of that by looking at additional donations due to our members, number of new organisations founded, or other measures.
So, new organisations founded is just one component of our impact that’s relatively tractable to analyse. More often, people assess us in terms of extra donations for charity raised from more people pursuing earning to give. Our estimate is that those earning to give will donate an extra $10m+ of counterfactually-adjusted funds to high-impact charities within the next 3 years because of us. I think either of these methods mean we’ve been very cost-effective in the past (historical financial costs are under $500k), and that’s ignoring over half the plan changes.
https://80000hours.org/2015/07/update-how-many-extra-donations-have-we-caused/
It seems really unclear to me how close we are to that margin. Bear in mind explicitly effective altruist funding is under 0.04% of total US philanthropy. It seems like we could expand the number of organisations a great deal before hitting substantially diminishing returns. In particular when you consider how little research has been done, relatively speaking, it’s unlikely we discovered the best things already.
If we did run out of ideas for new organisations, 80k could move its focus to scaling up existing orgs. (Many people who’ve changed plans due to 80k have gone to work at existing EA orgs rather than found new ones). Or we could just encourage everyone to earn to give and donate to top global poverty charities.
Also, why you do you expect entrepreneurial-talent in EA to hit diminishing returns faster than donations? If anything, I expect we’ll hit diminishing returns to additional donations faster than with talent, because funding gaps are so much easier to resolve than talent-gaps (e.g. one billionaire could flood EA with money tomorrow). And that means REG also doesn’t have as much upside as it looks because in the future they won’t be able to direct the money as effectively as well.
Of course. I actually think we’re going to be more effective with future funds because we’re getting better and better at changing plans, so our cost per plan change is falling. This is because our main focus in the past was learning and research, which is only just starting to pay off. There’s a lot more to say here though!
“More people building career capital” … “These are things people very unusually do otherwise”
Why do you think it’s unusual for people to build career capital?
True, that one’s an exception. The other 3⁄4 are unusual otherwise though.
It feels like we’re getting off track here. You originally claimed that 80K played a large role in creating REG and ACE (the implication presumably being that I should donate to 80K). Now we’re talking about the strength of evidence on how 80K has changed people’s career paths.
Although its evidence is weaker than I’d like, I’m still fairly confident that 80K has a positive impact, and I’m glad it exists. I just don’t see that it’s the best place to donate. Are you trying to convince me that 80K’s activities are valuable, or that I should donate to it? If it’s the former, I already believe that. If the latter, you need to:
show why 80K has a higher impact than anything else
do more to make the strength of evidence supporting 80K more robust
demonstrate that 80K can effectively use marginal funds
Now that’s a pretty high bar, but I’m donating a lot of money and I want to make sure I direct it well.
Nitpick: robust evidence doesn’t seem necessary—weak evidence of high upside potential should also count.
Hi Michael,
In the original document you say next to 80k “unclear whether it has a positive effect”. So I was starting there.
REG and ACE are relevant because they’re examples of the value of the plan changes we cause. If you think 80k is changing plans such that more high impact organisations are created, then it’s likely 80k is also effective. (Though may of course still not be worth funding due to a lack of RFMF, but that’s not what you said initially).
The closest we’ve got recently to publicly arguing the case for 80k is here: https://80000hours.org/2015/08/plans-for-the-coming-year-may-2015/
Of course there’s a lot more to talk about. I’m always happy to answer more questions or share details about how marginal funds would be used via email.
We have to debate back and forth and figure out this EtG thing properly.
I think Hurford’s points about EtG are obviously really wrong. I find it baffling so many people are convinced.
See my comment here: http://effective-altruism.com/ea/mk/peter_hurford_thinks_that_a_large_proportion_of/515.
That smart people who have thought about this can have such different views is worrying.
Since it’s so sensitive to what “from a good university” or “altruistically motivated” mean, it would make more sense to argue over a few hypothetical marginal case studies.
I’m not too worried about this; it’s just a hard problem. That means we should perhaps invest more into solving it.