Our last chain got very long, and it didn’t feel like we made much progress, so I’m going to limit myself to one reply.
My fundamental concern with 80K is that the evidence it its favor is very weak. My favorite meta-charity is REG because it has a straightforward causal chain of impact, and it raises a lot of money for charities that I believe do much more good in expectation than GiveWell top charities. 80K can claim the latter to some extent but cannot claim the former.
I agree that the counterfactuals + chain of impact are clearer with REG, however, strength of evidence is only one criteria I’d use when assessing a charity. To start with, I’m fairly risk-neutral, so I’m open to accepting weak evidence of high upside. But there’s a lot of other considerations to use when comparing charities, including some of these:
80k’s main purpose is to solve talent gaps rather than funding gaps, and about 70% of the plan changes don’t donate. So probably most of the benefits of 80k aren’t captured by the donation multiplier. We’ve also argued that talent gaps are more pressing than funding gaps. The nature of solving talent gaps means that your evidence will always be weak, so if donors only accept strong evidence, the EA community would never systematically deal with talent gaps.
Better growth prospects—REG hasn’t grown the last few years, whereas we’ve grown 20-fold; the remaining upside is also much better with 80k (due to the reasons in the post).
Better movement building benefits. 80k is getting hundreds of people into the EA movement, who are taking key positions at EA orgs, have valuable expertise etc. REG has only got a handful of poker players into EA.
80k produces valuable research for the movement, REG doesn’t.
Donations to REG might funge with the rest of EAS, which mostly works on stuff with similar or even weaker evidence than 80k (EA movement building in German-speaking countries, Foundational Research Institute, animal-focused policy advocacy).
Larger room for more funding. REG has been struggling to find someone to lead the project, so has limited room for funding. I’m keen to see that REG gets enough to cover their existing staff and pay for an Executive Director, but beyond that, 80k is better able to use additional funds.
On your specific criticisms of the donation multiplier calculation via getting people to pledge:
These people would not have signed the pledge without 80K.
These people would not have done something similarly or more valuable otherwise.
It seems like people take the GWWC pledge independently from their choice of job, so there isn’t an opportunity cost problem here.
The GWWC pledge is as valuable as GWWC claims it is.
I’m persuaded by GWWC’s estimates: https://www.givingwhatwecan.org/impact/
In fact, I think they undercount the value, because they ignore the chance of GWWC landing a super-large donor, which seems likely to be in the next 10 years. GWWC/CEA has already advised multiple billionaires.
On the opportunity cost point:
When someone switches from (e.g.) earning to give to direct work, 80K adds this to its impact stats. When someone else switches from direct work to earning to give, 80K also adds this to its impact stats.
Most of the plan changes are people moving from “regular careers” to “effective altruist style careers”, rather than people who are already EAs switching between great options.
If someone switched from a good option to another good option, we may not count it as a plan change in the first place. Second, we wouldn’t give it an equally high “impact rating”.
I would like to see more effort on 80K’s part to figure out whether its plan changes are actually causing people to do more good.
In terms of whether people are going into the right options, our core purpose is research into which careers are highest impact, so that’s where most of our effort already goes.
We’re highly concerned by this, and think about it a lot. We haven’t seen convincing evidence that we’ve been turning lots of people off (very unclear counterfactuals!). Our bounce rates etc. are if anything better than standard websites and I expect most people leave simply because they’re not interested. Note as well that our intro advice isn’t EA branded, which makes it much less damaging for the rest of the movement if we do make a bad impression.
We’ve also thought a lot about the analogy with GiveWell, and even discussed it with them. I think there’s some important differences. Our growth has also been faster than GiveWell the last few years.
Do we have any reason to believe that 80K will cause more organizations to be created, and that they will be as effective as the ones it contributed to in the past?
4 new organisations seems like a pretty good track record—it’s about one per year.
I expect more because we actively encourage people to set up new EA organisations (or work for the best existing ones) as part of our career advice.
Also note that our impact doesn’t rest on this—we have lots of other pathways to impact.
but writing new articles has diminishing utility as you start to cover the most important ideas.
You can also get increasing returns. The more articles you have, the more impressive the site seems, so the more convincing it is. More articles can also bring the domain more traffic, which benefits all the other articles. More articles mean you cover more situations, which makes a larger fraction of users happy, increasing your viral coefficient.
You need to convince me that it has sufficiently high leverage that it does more good than the single best direct-work org, and it has higher leverage than any other meta org.
Yes. Though where it gets tricky is making the assessment at the margin.
Yes. Though where it gets tricky is making the assessment at the margin.
I was wondering about this too. Is your calculation of the marginal cost per plan change just the costs for 2016 divided by the plan changes in 2016? That doesn’t seem to be an assessment at the margin.
However, like I say in the post, I’d caution against focusing too much on the marginal multiplier since I think more of our impact over 2017 will come from long-term growth rather the 2017 plan changes.
One margin is just running more workshops and that seems to have roughly the same marginal cost as our annual average. We just aren’t close to running out of promising people interested in coming along.
To second this I was very surprised how little the attendance to the careers workshops we ran at Cambridge dropped off. I think we ended up doing five 200 person careers workshops before they stopped selling out.
If there is similar (or even only half as much) demand at other universities then there is a lot of opportunity to scale.
We actually were running out on other campuses until we figured out how to get online advertising to convert into workshop attendance—now feels we can do several times as many workshops as we’re doing now without running out.
Hi Michael,
Thanks for the comments.
Our last chain got very long, and it didn’t feel like we made much progress, so I’m going to limit myself to one reply.
I agree that the counterfactuals + chain of impact are clearer with REG, however, strength of evidence is only one criteria I’d use when assessing a charity. To start with, I’m fairly risk-neutral, so I’m open to accepting weak evidence of high upside. But there’s a lot of other considerations to use when comparing charities, including some of these:
A larger multiplier—REG’s multiplier is about 4x once you include opportunity costs (see figures https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#raising-for-effective-giving); whereas I’ve argued our multiplier at the margin is at least 15x.
80k’s main purpose is to solve talent gaps rather than funding gaps, and about 70% of the plan changes don’t donate. So probably most of the benefits of 80k aren’t captured by the donation multiplier. We’ve also argued that talent gaps are more pressing than funding gaps. The nature of solving talent gaps means that your evidence will always be weak, so if donors only accept strong evidence, the EA community would never systematically deal with talent gaps.
Better growth prospects—REG hasn’t grown the last few years, whereas we’ve grown 20-fold; the remaining upside is also much better with 80k (due to the reasons in the post).
Better movement building benefits. 80k is getting hundreds of people into the EA movement, who are taking key positions at EA orgs, have valuable expertise etc. REG has only got a handful of poker players into EA.
80k produces valuable research for the movement, REG doesn’t.
Donations to REG might funge with the rest of EAS, which mostly works on stuff with similar or even weaker evidence than 80k (EA movement building in German-speaking countries, Foundational Research Institute, animal-focused policy advocacy).
Larger room for more funding. REG has been struggling to find someone to lead the project, so has limited room for funding. I’m keen to see that REG gets enough to cover their existing staff and pay for an Executive Director, but beyond that, 80k is better able to use additional funds.
On your specific criticisms of the donation multiplier calculation via getting people to pledge:
We ask them whether they would have taken it otherwise, and don’t count people who said yes. More discussion here: http://effective-altruism.com/ea/154/thoughts_on_the_meta_trap/9hk And here: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#giving-what-we-can-pledges
It seems like people take the GWWC pledge independently from their choice of job, so there isn’t an opportunity cost problem here.
I’m persuaded by GWWC’s estimates: https://www.givingwhatwecan.org/impact/ In fact, I think they undercount the value, because they ignore the chance of GWWC landing a super-large donor, which seems likely to be in the next 10 years. GWWC/CEA has already advised multiple billionaires.
On the opportunity cost point:
Most of the plan changes are people moving from “regular careers” to “effective altruist style careers”, rather than people who are already EAs switching between great options.
If someone switched from a good option to another good option, we may not count it as a plan change in the first place. Second, we wouldn’t give it an equally high “impact rating”.
In specific cases, we also say what the counterfactuals were e.g. see the end of the section on earning to give here: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#the-earning-to-give-community
In terms of whether people are going into the right options, our core purpose is research into which careers are highest impact, so that’s where most of our effort already goes.
In terms of the specific cases, there’s more detail here: https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/
We’re highly concerned by this, and think about it a lot. We haven’t seen convincing evidence that we’ve been turning lots of people off (very unclear counterfactuals!). Our bounce rates etc. are if anything better than standard websites and I expect most people leave simply because they’re not interested. Note as well that our intro advice isn’t EA branded, which makes it much less damaging for the rest of the movement if we do make a bad impression.
We’ve also thought a lot about the analogy with GiveWell, and even discussed it with them. I think there’s some important differences. Our growth has also been faster than GiveWell the last few years.
4 new organisations seems like a pretty good track record—it’s about one per year.
I expect more because we actively encourage people to set up new EA organisations (or work for the best existing ones) as part of our career advice.
Also note that our impact doesn’t rest on this—we have lots of other pathways to impact.
You can also get increasing returns. The more articles you have, the more impressive the site seems, so the more convincing it is. More articles can also bring the domain more traffic, which benefits all the other articles. More articles mean you cover more situations, which makes a larger fraction of users happy, increasing your viral coefficient.
Yes. Though where it gets tricky is making the assessment at the margin.
I was wondering about this too. Is your calculation of the marginal cost per plan change just the costs for 2016 divided by the plan changes in 2016? That doesn’t seem to be an assessment at the margin.
Hi Rohin,
For the multipliers, I was trying to make a marginal estimate.
For the costs, there’s a lot more detail here (it’s not simply 2016 costs / 2016 #): https://80000hours.org/2016/12/has-80000-hours-justified-its-costs/#whats-the-marginal-cost-per-plan-change
However, like I say in the post, I’d caution against focusing too much on the marginal multiplier since I think more of our impact over 2017 will come from long-term growth rather the 2017 plan changes.
One margin is just running more workshops and that seems to have roughly the same marginal cost as our annual average. We just aren’t close to running out of promising people interested in coming along.
To second this I was very surprised how little the attendance to the careers workshops we ran at Cambridge dropped off. I think we ended up doing five 200 person careers workshops before they stopped selling out.
If there is similar (or even only half as much) demand at other universities then there is a lot of opportunity to scale.
We actually were running out on other campuses until we figured out how to get online advertising to convert into workshop attendance—now feels we can do several times as many workshops as we’re doing now without running out.