I’m glad that you write this sort of thing. 80K is one of the few organizations that I see writing “why you should donate to us” articles. I believe more organizations should do this because they generally know more about their own accomplishments than anyone else. I wouldn’t take an organization’s arguments as seriously as a third party’s because they’re necessarily biased toward themselves, but they can still provide a useful service to potential donors by presenting the strongest arguments in favor of donating to them.
I have written before about why I’m not convinced that I should donate to 80K (see the comments on the linked comment thread). I have essentially the same concerns that I did then. Since you’re giving more elaborate arguments than before, I can respond in more detail about why I’m still not convinced.
My fundamental concern with 80K is that the evidence it its favor is very weak. My favorite meta-charity is REG because it has a straightforward causal chain of impact, and it raises a lot of money for charities that I believe do much more good in expectation than GiveWell top charities. 80K can claim the latter to some extent but cannot claim the former.
Below I give a few of the concerns I have with 80K, and what could convince me to donate.
Highly indirect impact. A lot of 80K’s claims to impact rely on long chains such that your actual effect is pretty indirect. For example, the claim that an IASPC is worth £7500 via getting people to sign the GWWC pledge relies on assuming:
These people would not have signed the pledge without 80K.
These people would not have done something similarly or more valuable otherwise.
The GWWC pledge is as valuable as GWWC claims it is.
I haven’t seen compelling evidence that any of these is true, and they all have to be true for 80K to have the impact here that it claims to have.
Problems with counterfactuals.
When someone switches from (e.g.) earning to give to direct work, 80K adds this to its impact stats. When someone else switches from direct work to earning to give, 80K also adds this to its impact stats. The only way these can both be good is if 80K is moving people toward their comparative advantages, which is a much harder claim to justify. I would like to see more effort on 80K’s part to figure out whether its plan changes are actually causing people to do more good.
Questionable marketing tactics.
This is somewhat less of a concern, but I might as well bring it up here. 80K uses very aggressive marketing tactics (invasive browser popups, repeated asks to sign up for things, frequent emails) that I find abrasive. 80K justifies these by claiming that it increases sign-ups, and I’m sure it does, but these metrics don’t account for the cost of turning people off.
By comparison, GiveWell does essentially no marketing but has still attracted more attention than any other EA organization, and it has among the best reputations of any EA org. It attracts donors by producing great content rather than by cajoling people to subscribe to its newsletter. For most orgs I don’t believe this would work because most orgs just aren’t capable of producing valuable content, but like GiveWell, 80K produces plenty of good content.
Perhaps 80K’s current marketing tactics are a good idea on balance, but we have no way of knowing. 80K’s metrics can only observe the value its marketing produces and not the value it destroys. It may be possible to get better evidence on this; I haven’t really thought about it.
Past vs. future impact.
80K has made a bunch of claims about its historical impact. I’m skeptical that the impact has been as big as 80K claims, but I’m also skeptical that the impact will continue to be as big. For example, 80K claims substantial credit for about a half dozen new organizations. Do we have any reason to believe that 80K will cause more organizations to be created, and that they will be as effective as the ones it contributed to in the past? 80K’s writeup claims that it will but doesn’t give much justification. Similarly, 80K claims that a lot of benefit comes from its articles, but writing new articles has diminishing utility as you start to cover the most important ideas.
In summary, to persuade me to donate to 80K, you need to convince me that it has sufficiently high leverage that it does more good than the single best direct-work org, and it has higher leverage than any other meta org. More importantly, you need to find strong evidence that 80K actually has the impact it claims to have, or better demonstrate that the existing evidence is sufficient.
Our last chain got very long, and it didn’t feel like we made much progress, so I’m going to limit myself to one reply.
My fundamental concern with 80K is that the evidence it its favor is very weak. My favorite meta-charity is REG because it has a straightforward causal chain of impact, and it raises a lot of money for charities that I believe do much more good in expectation than GiveWell top charities. 80K can claim the latter to some extent but cannot claim the former.
I agree that the counterfactuals + chain of impact are clearer with REG, however, strength of evidence is only one criteria I’d use when assessing a charity. To start with, I’m fairly risk-neutral, so I’m open to accepting weak evidence of high upside. But there’s a lot of other considerations to use when comparing charities, including some of these:
80k’s main purpose is to solve talent gaps rather than funding gaps, and about 70% of the plan changes don’t donate. So probably most of the benefits of 80k aren’t captured by the donation multiplier. We’ve also argued that talent gaps are more pressing than funding gaps. The nature of solving talent gaps means that your evidence will always be weak, so if donors only accept strong evidence, the EA community would never systematically deal with talent gaps.
Better growth prospects—REG hasn’t grown the last few years, whereas we’ve grown 20-fold; the remaining upside is also much better with 80k (due to the reasons in the post).
Better movement building benefits. 80k is getting hundreds of people into the EA movement, who are taking key positions at EA orgs, have valuable expertise etc. REG has only got a handful of poker players into EA.
80k produces valuable research for the movement, REG doesn’t.
Donations to REG might funge with the rest of EAS, which mostly works on stuff with similar or even weaker evidence than 80k (EA movement building in German-speaking countries, Foundational Research Institute, animal-focused policy advocacy).
Larger room for more funding. REG has been struggling to find someone to lead the project, so has limited room for funding. I’m keen to see that REG gets enough to cover their existing staff and pay for an Executive Director, but beyond that, 80k is better able to use additional funds.
On your specific criticisms of the donation multiplier calculation via getting people to pledge:
These people would not have signed the pledge without 80K.
These people would not have done something similarly or more valuable otherwise.
It seems like people take the GWWC pledge independently from their choice of job, so there isn’t an opportunity cost problem here.
The GWWC pledge is as valuable as GWWC claims it is.
I’m persuaded by GWWC’s estimates: https://www.givingwhatwecan.org/impact/
In fact, I think they undercount the value, because they ignore the chance of GWWC landing a super-large donor, which seems likely to be in the next 10 years. GWWC/CEA has already advised multiple billionaires.
On the opportunity cost point:
When someone switches from (e.g.) earning to give to direct work, 80K adds this to its impact stats. When someone else switches from direct work to earning to give, 80K also adds this to its impact stats.
Most of the plan changes are people moving from “regular careers” to “effective altruist style careers”, rather than people who are already EAs switching between great options.
If someone switched from a good option to another good option, we may not count it as a plan change in the first place. Second, we wouldn’t give it an equally high “impact rating”.
I would like to see more effort on 80K’s part to figure out whether its plan changes are actually causing people to do more good.
In terms of whether people are going into the right options, our core purpose is research into which careers are highest impact, so that’s where most of our effort already goes.
We’re highly concerned by this, and think about it a lot. We haven’t seen convincing evidence that we’ve been turning lots of people off (very unclear counterfactuals!). Our bounce rates etc. are if anything better than standard websites and I expect most people leave simply because they’re not interested. Note as well that our intro advice isn’t EA branded, which makes it much less damaging for the rest of the movement if we do make a bad impression.
We’ve also thought a lot about the analogy with GiveWell, and even discussed it with them. I think there’s some important differences. Our growth has also been faster than GiveWell the last few years.
Do we have any reason to believe that 80K will cause more organizations to be created, and that they will be as effective as the ones it contributed to in the past?
4 new organisations seems like a pretty good track record—it’s about one per year.
I expect more because we actively encourage people to set up new EA organisations (or work for the best existing ones) as part of our career advice.
Also note that our impact doesn’t rest on this—we have lots of other pathways to impact.
but writing new articles has diminishing utility as you start to cover the most important ideas.
You can also get increasing returns. The more articles you have, the more impressive the site seems, so the more convincing it is. More articles can also bring the domain more traffic, which benefits all the other articles. More articles mean you cover more situations, which makes a larger fraction of users happy, increasing your viral coefficient.
You need to convince me that it has sufficiently high leverage that it does more good than the single best direct-work org, and it has higher leverage than any other meta org.
Yes. Though where it gets tricky is making the assessment at the margin.
Yes. Though where it gets tricky is making the assessment at the margin.
I was wondering about this too. Is your calculation of the marginal cost per plan change just the costs for 2016 divided by the plan changes in 2016? That doesn’t seem to be an assessment at the margin.
However, like I say in the post, I’d caution against focusing too much on the marginal multiplier since I think more of our impact over 2017 will come from long-term growth rather the 2017 plan changes.
One margin is just running more workshops and that seems to have roughly the same marginal cost as our annual average. We just aren’t close to running out of promising people interested in coming along.
To second this I was very surprised how little the attendance to the careers workshops we ran at Cambridge dropped off. I think we ended up doing five 200 person careers workshops before they stopped selling out.
If there is similar (or even only half as much) demand at other universities then there is a lot of opportunity to scale.
We actually were running out on other campuses until we figured out how to get online advertising to convert into workshop attendance—now feels we can do several times as many workshops as we’re doing now without running out.
When we “impact rate” the plan changes, we also try to make an initial assessment of how much is counterfactually due to us (as well as how much extra impact results non-counterfactually adjusted).
We then to more in-depth analysis of the counterfactuals in crucial cases. Because we think the impact of plan changes it fat tailed, if we can understand the top 5% of them, we get a reasonable overall picture.
We do this analysis in documents like this: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/
Each individual case is debateable, but I think there’s a large enough volume of cases now to justify that we’re having a substantial impact.
I’m glad that you write this sort of thing. 80K is one of the few organizations that I see writing “why you should donate to us” articles. I believe more organizations should do this because they generally know more about their own accomplishments than anyone else. I wouldn’t take an organization’s arguments as seriously as a third party’s because they’re necessarily biased toward themselves, but they can still provide a useful service to potential donors by presenting the strongest arguments in favor of donating to them.
I have written before about why I’m not convinced that I should donate to 80K (see the comments on the linked comment thread). I have essentially the same concerns that I did then. Since you’re giving more elaborate arguments than before, I can respond in more detail about why I’m still not convinced.
My fundamental concern with 80K is that the evidence it its favor is very weak. My favorite meta-charity is REG because it has a straightforward causal chain of impact, and it raises a lot of money for charities that I believe do much more good in expectation than GiveWell top charities. 80K can claim the latter to some extent but cannot claim the former.
Below I give a few of the concerns I have with 80K, and what could convince me to donate.
Highly indirect impact. A lot of 80K’s claims to impact rely on long chains such that your actual effect is pretty indirect. For example, the claim that an IASPC is worth £7500 via getting people to sign the GWWC pledge relies on assuming:
These people would not have signed the pledge without 80K.
These people would not have done something similarly or more valuable otherwise.
The GWWC pledge is as valuable as GWWC claims it is.
I haven’t seen compelling evidence that any of these is true, and they all have to be true for 80K to have the impact here that it claims to have.
Problems with counterfactuals.
When someone switches from (e.g.) earning to give to direct work, 80K adds this to its impact stats. When someone else switches from direct work to earning to give, 80K also adds this to its impact stats. The only way these can both be good is if 80K is moving people toward their comparative advantages, which is a much harder claim to justify. I would like to see more effort on 80K’s part to figure out whether its plan changes are actually causing people to do more good.
Questionable marketing tactics.
This is somewhat less of a concern, but I might as well bring it up here. 80K uses very aggressive marketing tactics (invasive browser popups, repeated asks to sign up for things, frequent emails) that I find abrasive. 80K justifies these by claiming that it increases sign-ups, and I’m sure it does, but these metrics don’t account for the cost of turning people off.
By comparison, GiveWell does essentially no marketing but has still attracted more attention than any other EA organization, and it has among the best reputations of any EA org. It attracts donors by producing great content rather than by cajoling people to subscribe to its newsletter. For most orgs I don’t believe this would work because most orgs just aren’t capable of producing valuable content, but like GiveWell, 80K produces plenty of good content.
Perhaps 80K’s current marketing tactics are a good idea on balance, but we have no way of knowing. 80K’s metrics can only observe the value its marketing produces and not the value it destroys. It may be possible to get better evidence on this; I haven’t really thought about it.
Past vs. future impact.
80K has made a bunch of claims about its historical impact. I’m skeptical that the impact has been as big as 80K claims, but I’m also skeptical that the impact will continue to be as big. For example, 80K claims substantial credit for about a half dozen new organizations. Do we have any reason to believe that 80K will cause more organizations to be created, and that they will be as effective as the ones it contributed to in the past? 80K’s writeup claims that it will but doesn’t give much justification. Similarly, 80K claims that a lot of benefit comes from its articles, but writing new articles has diminishing utility as you start to cover the most important ideas.
In summary, to persuade me to donate to 80K, you need to convince me that it has sufficiently high leverage that it does more good than the single best direct-work org, and it has higher leverage than any other meta org. More importantly, you need to find strong evidence that 80K actually has the impact it claims to have, or better demonstrate that the existing evidence is sufficient.
Hi Michael,
Thanks for the comments.
Our last chain got very long, and it didn’t feel like we made much progress, so I’m going to limit myself to one reply.
I agree that the counterfactuals + chain of impact are clearer with REG, however, strength of evidence is only one criteria I’d use when assessing a charity. To start with, I’m fairly risk-neutral, so I’m open to accepting weak evidence of high upside. But there’s a lot of other considerations to use when comparing charities, including some of these:
A larger multiplier—REG’s multiplier is about 4x once you include opportunity costs (see figures https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#raising-for-effective-giving); whereas I’ve argued our multiplier at the margin is at least 15x.
80k’s main purpose is to solve talent gaps rather than funding gaps, and about 70% of the plan changes don’t donate. So probably most of the benefits of 80k aren’t captured by the donation multiplier. We’ve also argued that talent gaps are more pressing than funding gaps. The nature of solving talent gaps means that your evidence will always be weak, so if donors only accept strong evidence, the EA community would never systematically deal with talent gaps.
Better growth prospects—REG hasn’t grown the last few years, whereas we’ve grown 20-fold; the remaining upside is also much better with 80k (due to the reasons in the post).
Better movement building benefits. 80k is getting hundreds of people into the EA movement, who are taking key positions at EA orgs, have valuable expertise etc. REG has only got a handful of poker players into EA.
80k produces valuable research for the movement, REG doesn’t.
Donations to REG might funge with the rest of EAS, which mostly works on stuff with similar or even weaker evidence than 80k (EA movement building in German-speaking countries, Foundational Research Institute, animal-focused policy advocacy).
Larger room for more funding. REG has been struggling to find someone to lead the project, so has limited room for funding. I’m keen to see that REG gets enough to cover their existing staff and pay for an Executive Director, but beyond that, 80k is better able to use additional funds.
On your specific criticisms of the donation multiplier calculation via getting people to pledge:
We ask them whether they would have taken it otherwise, and don’t count people who said yes. More discussion here: http://effective-altruism.com/ea/154/thoughts_on_the_meta_trap/9hk And here: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#giving-what-we-can-pledges
It seems like people take the GWWC pledge independently from their choice of job, so there isn’t an opportunity cost problem here.
I’m persuaded by GWWC’s estimates: https://www.givingwhatwecan.org/impact/ In fact, I think they undercount the value, because they ignore the chance of GWWC landing a super-large donor, which seems likely to be in the next 10 years. GWWC/CEA has already advised multiple billionaires.
On the opportunity cost point:
Most of the plan changes are people moving from “regular careers” to “effective altruist style careers”, rather than people who are already EAs switching between great options.
If someone switched from a good option to another good option, we may not count it as a plan change in the first place. Second, we wouldn’t give it an equally high “impact rating”.
In specific cases, we also say what the counterfactuals were e.g. see the end of the section on earning to give here: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#the-earning-to-give-community
In terms of whether people are going into the right options, our core purpose is research into which careers are highest impact, so that’s where most of our effort already goes.
In terms of the specific cases, there’s more detail here: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/
We’re highly concerned by this, and think about it a lot. We haven’t seen convincing evidence that we’ve been turning lots of people off (very unclear counterfactuals!). Our bounce rates etc. are if anything better than standard websites and I expect most people leave simply because they’re not interested. Note as well that our intro advice isn’t EA branded, which makes it much less damaging for the rest of the movement if we do make a bad impression.
We’ve also thought a lot about the analogy with GiveWell, and even discussed it with them. I think there’s some important differences. Our growth has also been faster than GiveWell the last few years.
4 new organisations seems like a pretty good track record—it’s about one per year.
I expect more because we actively encourage people to set up new EA organisations (or work for the best existing ones) as part of our career advice.
Also note that our impact doesn’t rest on this—we have lots of other pathways to impact.
You can also get increasing returns. The more articles you have, the more impressive the site seems, so the more convincing it is. More articles can also bring the domain more traffic, which benefits all the other articles. More articles mean you cover more situations, which makes a larger fraction of users happy, increasing your viral coefficient.
Yes. Though where it gets tricky is making the assessment at the margin.
I was wondering about this too. Is your calculation of the marginal cost per plan change just the costs for 2016 divided by the plan changes in 2016? That doesn’t seem to be an assessment at the margin.
Hi Rohin,
For the multipliers, I was trying to make a marginal estimate.
For the costs, there’s a lot more detail here (it’s not simply 2016 costs / 2016 #): https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#whats-the-marginal-cost-per-plan-change
However, like I say in the post, I’d caution against focusing too much on the marginal multiplier since I think more of our impact over 2017 will come from long-term growth rather the 2017 plan changes.
One margin is just running more workshops and that seems to have roughly the same marginal cost as our annual average. We just aren’t close to running out of promising people interested in coming along.
To second this I was very surprised how little the attendance to the careers workshops we ran at Cambridge dropped off. I think we ended up doing five 200 person careers workshops before they stopped selling out.
If there is similar (or even only half as much) demand at other universities then there is a lot of opportunity to scale.
We actually were running out on other campuses until we figured out how to get online advertising to convert into workshop attendance—now feels we can do several times as many workshops as we’re doing now without running out.
Do you have any info on how reliable self-reports are wrt counterfactuals about career changes and EWWC pledging?
I can imagine that people would not be very good at predicting that accurately.
One would expect some social acceptability bias that might require substantial creativity and work to measure.
Hi there,
It’s definitely hard for people to estimate.
When we “impact rate” the plan changes, we also try to make an initial assessment of how much is counterfactually due to us (as well as how much extra impact results non-counterfactually adjusted).
We then to more in-depth analysis of the counterfactuals in crucial cases. Because we think the impact of plan changes it fat tailed, if we can understand the top 5% of them, we get a reasonable overall picture. We do this analysis in documents like this: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/
Each individual case is debateable, but I think there’s a large enough volume of cases now to justify that we’re having a substantial impact.