I tried to sum up the key messages in plain language in a Twitter thread, in case that helps clarify.
Benjamin_Todd
Karma: 6,114
I think that’s a good summary, but it’s not only winner-takes-all effects that generate heavy-tailed outcomes.
You can get heavy tailed outcomes if performance is the product of two normally distributed factors (e.g. intelligence x effort).
It can also arise from the other factors that Max lists in another comment (e.g. scalable outputs, complex production).
Luck can also produce heavy tailed outcomes if it amplifies outcomes or is itself heavy-tailed.
This is cool.
One theoretical point in favour of complexity is that complex production often looks like an ‘o-ring’ process, which will create heavy-tailed outcomes.
On your main point, this was the kind of thing we were trying to make clearer, so it’s disappointing that hasn’t come through.
Just on the particular VC example:
I’m suspicious you can do a good job of predicting ex ante outcomes. After all, that’s what VCs would want to do and they have enormous resources. Their strategy is basically to pick as many plausible winners as they can fund.
Most VCs only pick from the top 1-5% of startups. E.g. YC’s acceptance rate is 1%, and very few startups they reject make it to series A. More data on VC acceptance rates here: https://80000hours.org/2014/06/the-payoff-and-probability-of-obtaining-venture-capital/
So, I think that while it’s mostly luck once you get down to the top 1-5%, I think there’s a lot of predictors before that.
Also see more on predictors of startup performance here: https://80000hours.org/2012/02/entrepreneurship-a-game-of-poker-not-roulette/
How much does performance differ between people?
Thank you for this summary!
One thought that struck me is that most of the objections seem most likely to come up in response to ‘GiveWell style EA’.
I expect the objections that would be raised to a longtermist-first EA would be pretty different, though with some overlap. I’d be interested in any thoughts on what they would be.
I also (speculatively) wonder if a longtermist-first EA might ultimately do better with this audience. You can do a presentation that starts with climate change, and then point out that the lack of political representation for future generations is a much more general problem.
In addition, longtermist EAs favour hits based giving, and that makes it clear that policy change is among the best interventions, while acknowledging it’s very hard to measure effects, which seems more palatable than an approach highly focused on measurement of narrow metrics.
I agree—but my impression is that they consider track record when making the forward-looking estimates, and they also update their recommendations over time, in part drawing on track record. I think “doesn’t consider track record” is a straw man, though there could be an interesting argument about whether more weight should be put on track record as opposed to other factors (e.g. intervention selection, cause selection, team quality).
Impact = money moved * average charity effectiveness. FP tracks money to their recommended charities, and this is their published research on the effectiveness of those charities, and why they recommended them.
We make the impact evaluation I note above available to donors (and our donors also do their own version of it). We also publish top line results publicly in our annual reviews (e.g. number of impact-adjusted plan changes) , but don’t publish the case studies since they involve a ton of sensitive personal information.
Just a quick comment that I don’t think the above is a good characterisation of how 80k assesses its impact. Describing our whole impact evaluation would take a while, but some key elements are:
-
We think impact is heavy tailed, so we try to identify the most high-impact ‘top plan changes’. We do case studies of what impact they had and how we helped. This often involves interviewing the person, and also people who can assess their work. (Last year these interviews were done by a third party to reduce desirability bias). We then do a rough fermi estimate of the impact.
-
We also track the number of a wider class of ‘criteria-based plan changes’, but then take a random sample and make fermi estimates of impact so we can compare their value to the top plan changes.
If we had to choose a single metric, it would be something closer to impact-adjusted years of extra labour added to top causes, rather than the sheer number of plan changes.
We also look at other indicators like:
-
There have been other surveys of the highest-impact people who entered EA in recent years, evaluating which fraction came from 80k, which let’s us make an estimate of the percentage of the EA workforce from 80k.
-
We look at the EA survey results, which let’s us track things like how many people are working at EA orgs and entered via 80k.
We use number of calls as a lead metric, not an impact metric. Technically it’s the number of calls with people who made an application above a quality bar, rather than the raw number. We’ve checked and it seems to be a proxy for the number of impact-adjusted plan changes that result from advising.
This is not to deny that assessing our impact is extremely difficult, and ultimately involves a lot of judgement calls—we were explicit about that in the last review—but we’ve put a lot more work into it than the above implies – probably around 5-10% of team time in recent years.
I think similar comments could be made by several of the other examples e.g. GWWC also tracks dollars donated each year to effective charities (now via the EA Funds) and total dollars pledged. They track the number of pledges as well since that’s a better proxy for the community building benefits.
-
Tldr: I agree the ‘top of the funnel’ seems to not be growing (i.e. how many people are reached each year). This was at least in part due to a deliberate shift in strategy. I think the ‘bottom’ of the funnel (e.g. money and people focused on EA) is still growing. Eventually we’ll need to get the top of the funnel growing again, and people are starting to focus on this more.
Around 2015, DGB and The Most Good You Can Do were both launched, which both involved significant media attention that aimed to reach lots of people (e.g. two TED talks). 80k was also focused on reaching more people.
After that, the sense was that the greater bottleneck was taking all these newly interested people (and the money from Open Phil), and making sure that results in actually useful things happening, rather than reaching even more people.
(There was also some sense of wanting to shore up the intellectual foundations, and make sure EA is conveyed accurately, rather than as “earn to give for malaria nets”, which seems vital for its long-term potential. There was also a shift towards niche outreach, rather than mass media—since mass media seems better for raising donations to global health, but less useful for something like reducing GCBRs, and although good at reaching lots of people, wasn’t as effective as the niche stuff.)
E.g. in 2018, 80k switched to focusing on our key ideas page and podcast, which are more about making sure already interested people understand our ideas than reaching new people; Will focused on research and niche outreach, and is now writing a book on longtermism. GWWC was scaled down, and Toby wrote a book about existential risk.
This wasn’t obviously a mistake since I think that if you track ‘total money committed to EA’ and ‘total number of people willing to change career (or take other significant steps)’, it’s still growing reasonably (perhaps ~20% per year?), and these metrics are closer to what ultimately matter. (Unfortunately I don’t have a good source for this claim and it relies on judgement calls, though Open Phil’s resources have increased due to the increase in Dustin Moskovitz’s net worth; several other donors have made a lot of money; the EA Forum is growing healthily; 80k is getting ~200 plan changes per year; the student groups keep recruiting people each year etc.)
One problem is that if the top of the funnel isn’t growing, then eventually we’ll ‘use up’ the pool of interested people who might become more dedicated, so it’ll turn into a bottleneck at some point.
And all else equal, more top of funnel growth would be better, so it’s a shame we haven’t done more.
My impression is that people are starting to focus more on growing the top of the funnel again. However, I still think the focus is more on niche outreach, so you’d need to track a metric more like ‘total number of engaged people’ to evaluate it.
I agree it’s important to do both, and people often neglect to understand the day-to-day reality, and to zoom in on very concrete opportunities rather than think about broad paths.
You could see it as a form of near vs. far mode bias.
That’s true, but we considered switching at several points later on (and the local groups did in fact switch).
This makes sense—seems like a good thing to think about.
One other point you didn’t mention is that by picking another name, it reduces brand risks for EA, and that means you can act a bit more independently. (E.g. once a local group ran an event that got picked up by national newspapers.)
You can also make your group name about something more specific. Another problem of effective altruism is that it’s broad and abstract which makes it hard to explain.
For instance, if you want your group to be practical and aimed at students, picking something about careers / finding a job can be very appealing. (Though this wouldn’t be a good choice if you wanted to be more niche and intellectual.) This was one of the reasons why we picked GWWC and 80k as names in the early days, rather than leading with effective altruism.
The direct translations of Effective Altruism can sound a bit forced. The dutch “Effectief Altruïsme” is a case in point. This can make you seem a bit “out of touch”. We checked with German and Spanish EA’s and they confirm that the direct translations sound rather awkward in their native tongue (a Polish EA we talked to have said that the name actually sounds really positive in Polish, so this point may not be relevant in all languages).
Agree! I write a bit more about this topic here, and talk about how the first Chinese translation wasn’t ideal.
My personal impression is that it’s a bit easier to find operations staff than before, but still difficult.
We wrote this update at the top of the original article:
Note: Though we edited this article lightly in 2020, it was written in 2018 and is now somewhat out of date.
Due to this post and other efforts, people in the effective altruist community have become more interested in working in operations, and it has also come to seem easier to fill these roles with people not already active in the community. As a result, several recent hiring rounds for these roles were successful, and there are now fewer open positions, and when positions open up they’ve become more competitive.
This means that the need for more people to pursue this path is somewhat less than when we wrote the post. However, many of the major points in this post still apply to some degree, there is still a need for more people working in operations management, and we expect this need to persist as the community grows. For these reasons, we still think this is a promising path to consider. We also think the information on how to enter operations roles and how to assess your fit for them will still be valuable to people pursuing this path.
Yes—I was thinking of the writing on economic growth research—that’s still helping to advance a hits based approach to helping the global poor.
Great to see an attempt to take a hits based approach to climate change and global health that’s also open to small donors.
Thanks!
Hi there, I agree it’s an interesting opening argument. I was wondering if you had seen this article before, which takes a similar approach: https://80000hours.org/career-guide/making-a-difference/
One concern about this approach is that I think it can seem obviously “low leverage” to the types of people focused on entrepreneurship, policy change and research (some of the people we most want to appeal to), and can give them the impression that EA is mainly about ‘high confidence’ giving rather than ‘high expected value’ giving, which is already a one of the most common misconceptions out there.