Cody’s answer below and mine above give better ‘overall’ answers to your question, but—if you’d like to see something concrete and incomplete you could look at this appendix of job board placements we’re aware of.
brentonmayer
Karma: 262
Basically: these just take a really long time!
Lumping 2021 and 2022 progress together into a single public report meant that we saved hundreds of hours of staff time.
A few other things that might be worth mentioning:
I’m not sure whether we’ll use 1 or 2 year cycles for public annual reviews in future.
This review (14 pages + appendices) was much less in-depth and so much less expensive to produce than 2020 (42 pages + appendices) or 2019 (109 pages + appendices). If we end up thinking that our public reviews should be more like this going forward then the annual approach would be much less costly.
In 2021, we only did a ‘mini-annual review’ internally, in which we attempted to keep the time cost of the review relatively low and not open up major strategic questions.
We didn’t fundraise in 2021.
I regret not publishing a blog post at the time stating this decision.
Hi Vaidehi—I’m answering here as I was responsible for 80k’s impact evaluation until late last year.
My understanding is that plan changes (previously IASPC’s then DIPY’s) were a core metric 80K used in previous years to evaluate impact. It seems that there has been a shift to a new metric—CBPC’s (see below).
This understanding is a little off. Instead, it’s that in 2019 we decided to switch from IASPCs to DIPYs and CBPCs.
The best place to read about the transition is the mistakes page here, and I think the best places to read detail on how these metrics work is the 2019 review for DIPYs and the 2020 review for CBPCs. (There’s a 2015 blog post on IASPCs.)~~~
Some more general comments on how I think about this:
A natural way to think about 80k’s impact is as a funnel which culminates in a single metric which we can relate to as a for profit does to revenue.
I haven’t been able to create a metric which is overall strong enough to make me want to rely on it like that.
The closest I’ve come is the DIPY, but it’s got major problems:Lags by years.
Takes hundreds of hours to put together.
Requires a bunch of judgement calls—these are hard for people without context to assess and have fairly low inter-rater reliability (between people, but also the same people over time).
Most (not all) of them come from case studies where people are asked questions directly by 80,000 Hours staff. That introduces some sources of error, including from social-desirability bias.
The case studies it’s based on can’t be shared publicly.
Captures a small fraction of our impact.
Doesn’t capture externalities.
(There’s a bit more discussion on impact eval complexities in the 2019 annual review.)
So, rather than thinking in terms of a single metric to optimise, when I think about 80k’s impact and strategy I consider several sources of information and attempt to weigh each of them appropriately given their strengths and weaknesses.
The major ones are listed in the full 2022 annual review, which I’ll copy out here:
The 80,000 Hours user survey. A summary of the 2022 user survey is linked in the appendix.
Our in-depth case study analyses, which produce our top plan changes (last analysed in 2020). EDIT: this process produces the DIPYs as well. I’ve made a note of this in the public annual review—apologies, doing this earlier might have prevented you getting the impression that we retired them.
Our own data about how users interact with our services (e.g. our historical metrics linked in the appendix).
Our and others’ impressions of the quality of our visible output.
~~~
On your specific questions:
I understand that we didn’t make predictions about CBPCs in 2021.
Otherwise, I think the above is probably the best general answer to give to most of these—but lmk if you have follow ups :)
Ah nice, understood!
I don’t think you’ll find anything from us which is directly focused on most of these questions. (It’s also not especially obvious that this is our comparative advantage within the community.)
But we do have some relevant public content. Much of it is in our annual review, including its appendices.
You also might find these results of the OP EA/LT survey interesting.
Hi—thanks for taking the time to think through these, write them out and share them! We really appreciate getting feedback from people who use our services and who have a sense of how others do.
I work on 80k’s internal systems, including our impact evaluation (which seems relevant to your ideas).
I’ve made sure that the four points will be seen by the relevant people at 80k for each of these.
Re. #1, I’m confused about whether you’re more referring to ‘message testing’ (i.e. what ideas/framings make our ideas appealing to which audiences) or ‘long term follow up with users to see how their careers/lives have change’. (I can imagine various combinations of these.)
Could you elaborate?
I was interested in seeing a breakdown of the endpoints, before they’d been compressed into the scales AAC uses above.
Jamie kindly pulled this spreadsheet together for me, which I’m sharing (with permission), as I thought it might be helpful to other readers too.
Through overpopulation and excessive consumption, humanity is depleting its natural resources, polluting its habitat, and causing the extinction of other species. Continuing like this will lead to the collapse of civilisation and likely our own extinction.
This one seems very common to me, and sadly people often feel fatalistic about it.
Two things that feeling might come from:
People rarely talking about aspects of it which are on a positive trajectory (e.g. the population of whales, acid rain, CFC emissions, UN population projections).
The sense that there are so related things to solve—such that even if we managed to fix (say) climate change then we’d still see (say) our fisheries cause the collapse of the ocean’s ecosystem.
It’s really cool to see these laid out next to another like this! Thanks for posting Katja :)
Makes sense! FWIW, I really enjoyed reading your post. There’s definitely something nice about how listing specific vacancies forces us to get down to get really concrete about what all this theorising actually means, even though doing so has been a bit challenging sometimes!
Thanks for the post Henry! I work at 80,000 Hours and have thought a little bit (along with Maria) about some of the indirect effects of the job board recently—especially about the degree to which it’ll be seen as representing our all-considered views of the best jobs. So it’s good to have some discussion of it!
Like you, I’m really excited about people using the job board to expand their ideas of what EA/long termist roles can look like, especially to types of roles which don’t have (something like) “effective altruism” somewhere in the name. Rob wrote a bit more about this here.
That being said, I do share many of Habryka, Aidan and Ben’s concerns about people thinking of it as representative of good opportunities in EA. It’s missing roles which orgs don’t advertise, lots of opportunities at early stage orgs, roles you design yourself and doesn’t foreground graduate school enough (yet!).
You can read more about In the user guide/FAQ about how we hope for people to think about the roles we list. In particular, I’m keen for people to keep this in mind:
“there is a good chance that your best option is actually a role that is not featured on the board. If you find a role that seems promising but is not listed on our board, you should not infer that it is less promising than the roles that we do feature.
Hi Aidan,
I’m Brenton from 80,000 Hours—thanks for writing this up! It seems really important that people don’t think of us as “tell[ing] them how to have an impactful career”. It sounds absolutely right to me that having a high impact career requires “a lot of independent thought and planning”—career advice can’t be universally applied.
I did have a few thoughts, which you could consider incorporating if you end up making a top level post. The most substantive two are:
Many of the priority paths are broader than you might be thinking.
A significant amount of our advice is designed to help people think through how to approach their careers, and will be useful regardless of whether they’re aiming for a priority path.
Many of the priority paths are broader than you might be thinking:
Most people won’t be able to step into an especially high impact role directly out of undergrad, so unsurprisingly, many of the priority paths require people to build up career capital before they can get into high impact positions. We’d think of people who are building up career capital focused on (say) AI policy as being ‘on a priority path’. We also think of people who aren’t in the most competitive positions as being within the path
For instance, let’s consider AI policy. We think that path includes graduate school, all the options outlined in our writeup on US AI policy and the 161 roles currently on the job board under the relevant filter. It’s also worth remembering that the job board has still left most of the relevant roles out: none of them are congressional staffers for example, which we’d also think of as under this priority path.
A significant amount of our advice is designed to help people think through how to approach their careers, and will be useful regardless of whether they’re aiming for a priority path.
In our primary articles on how to plan your career, we spend a lot of time talking about general career strategy and ways to generate options. The articles encourage people to go through a process which should generate high impact options, of which only some will be in the priority paths:
The career strategy and planning and decision making sections of key ideas
Unfortunately, there’s something in the concreteness of a list of top options which draws people in particularly strongly. This is a communication challenge that we’ve worked on a bit, but don’t think we have a great answer to yet. We discussed this in our ‘Advice on how to read our advice’. In the future we’ll add some more ‘niche’ paths, which may help somewhat.
A few more minor points:
Your point about Bill Gates was really well put. It reminded me of my colleague Michelle’s post on ‘Keeping absolutes in mind’, which you might enjoy reading.
We don’t think that the priority paths are the only route through which people can affect the long term future.
I found the tone of this comment generally great, and two of my colleagues commented the same. I appreciate that going through this shift you’ve gone through would have been hard and it’s really impressive that you’ve come out of it with such a balanced view, including being able to acknowledge the tradeoffs that we face in what we work on. Thank you for that.
If you make a top level post (which I’d encourage you to do), feel free to quote any part of this comment.
Cheers, Brenton
Thanks for this post—I agree with your main point that there are many ways to contribute without working at organisations that explicitly identify with the effective altruism community, as would the rest of 80,000 Hours (where I work). In fact, I might go further in emphasising this.
The overwhelming majority of high impact roles in the world lie outside those organisations – with governments, foundations, intergovernmental agencies, large companies and, as you point out, academia. The majority of people interested in effective altruism should be taking roles in places like these, not EA orgs. Unfortunately, when we highlight specific roles there’s a bias towards opportunities we know about due to our involvement in the community, but where we’ve managed to correct for that (such as in the AI strategy and governance problem area of our job board) it’s clear that there are lots of valuable roles focusing on our top problems at a wide range of organisations.
______
I agree that when considering their future career path, people should think about what skills and expertise they already have(link, link.)That might mean—if you’re enjoying and succeeding in your current path—staying there and using that position to influence your field / company in a positive direction. Though it also might also mean thinking about how your skills might translate to other effective careers. For example, governments tend to be keen to hire people with science PhDs or tech skills, as shown by things like the AAAS fellowship and Tech Congress in the US. These don’t tend to feel like a natural step from a PhD, but being a scientific adviser in government seems plausibly pretty high leverage.
Since you mentioned academia, I thought readers might be interested in a few resources that might be useful for them if they’re looking to influence their academic field. There’s a Facebook group for EA academics to share what they’re working on and help each other. Luke Muehlhauser wrote an excellent report on cases where people successfully and unsuccessfully tried to deliberately build new fields. One case study that’s particularly interestingly is that of neoliberal economics (written up compellingly by Kerry Vaughan), which is often held up as a great example of what can be achieved through careful work both within academia and with the people who disseminate ideas – journalists, authors, think tanks etc. Finally, there’s our career review.
A few nice examples I’ve seen along these lines:
ACE’s graphs on how relatively neglected farm animal welfare is.
Wait But Why on putting time in perspective.
A bunch of art on space, of which this clip of the virgo supercluster is an example.
And my favourite - ‘If the Moon Were Only 1 Pixel—a tediously accurate scale model of the solar system’.
Thanks Catherine. I’m going to quote the relevant part of my conclusion here, as I think the overall results of high school outreach are one of the most remarkable things to have come out of this review, but they haven’t seen any discussion here so far.
I’ve been very surprised at how little measured success high school EA outreach efforts have yielded. This post has compiled evidence from many competent people trying out multiple different methods, which in total have had over 5 years of full time equivalent work go into them. This has resulted in:
Three students becoming counterfactually interested in EA enough that they became involved in university groups or made a career change. I would guess that this work (mostly Catherine’s) accounts for the majority (>75%) of the measured success.
10-20 students becoming counterfactually interested in EA enough to reduce their meat consumption or start fundraisers for ACE or GiveWell recommended charities.
<$20,000 USD raised for ACE or GiveWell recommended charities.
“I don’t think I’m assuming that.”
That’s fair—my bad.
I think that it felt worthwhile making this point because an obvious response to your conclusion that “demand for jobs at professional EA organizations will continue to be very high” is to not worry if demand for these jobs drops. Or one could go further, and think that it would be good if demand dropped, given that there are costs to being an unsuccessful applicant. I appreciate that you’re agnostic on whether people should have that response, but I personally think it would be bad—in part due to the reasoning in my previous comment.
[I work at 80,000 Hours]
It seems like you’re assuming that it would be better if EA organisations could make their jobs less desirable, in order to put off applicants so that the jobs would be less competitive. That doesn’t seem right to me.
Making the jobs less desirable is likely to either put off applicants at random, or even disproportionately put off the most experienced applicants who are most picky about jobs. That would seem reasonable to do if EA orgs were getting plenty of applicants above the bar to hire, and didn’t think there would be much difference in job performance amongst them. But that doesn’t seem to be the situation these organisations are reporting. Given that, we’d expect that reducing applicants by making the jobs less desirable would harm the beneficiaries of the organisations.
This is a good thought! I actually went through a month or two of being pretty excited about doing something like this early last year. Unfortunately I think there are quite a few issues around how well the data we have from advising represents what paths EAs in general are aiming for, such that we (80,000 Hours) are not the natural home for this project. We discussed including a question on this in the EA survey with Rethink last year, though I understand they ran out of time/space for it.
I think there’s an argument that we should start collecting/publicising whatever (de-identified) data we can get anyway, because any additional info on this is useful and it’s not that hard for 80,000 Hours to get. I think the reason that doing this feels less compelling to me is that this information would only answer a small part of the question we’re ultimately interested in.
We want to know the expected impact of a marginal person going to work in a given area.
To answer that, we’d need something like:
The number of EAs aiming at a given area, weighted by dedication, seniority and likelihood of success.
The same data for people who are not EAs but are aiming to make progress on the same problem. In some of our priority paths, EAs are a small portion of the relevant people.
An estimate of the extent to which different paths have diminishing returns and complementarity. (That linked post might be worth reading for more of our thoughts on coordinating as a community.)
We then probably want something around time—how close to making an impact are the people currently aiming at this path, how long does it take someone who doesn’t have any experience to make an impact, how much do we want talent there now vs later etc.
I think without doing that extra analysis, I wouldn’t really know how to interpret the results and we’ve found that releasing substandard data can get people on the wrong track. I think that doing this analysis well would be pretty great, but it’s also a big project with a lot of tricky judgement calls, so it doesn’t seem at the top of our priority list.
What should be done in the meantime? I think this piece is currently the best guide we have on how to systematically work through your career decisions. Many of the factors you mentioned are considered (although not precisely quantified) when we recommend priority paths because we try to consider neglectedness (both now and our guess at the next few years). For example, we think AI policy and AI technical safety could both absorb a lot more people before hitting large diminishing returns so we’re happy to recommend that people invest in the relevant career capital. Even if lots of people do so, we expect this investment to still pay off.
Thanks for the thought!
You might be interested in the analysis we did in 2020. To pull out the phrase that I think most closely captures what you’re after:
~~~
We did a scrappy internal update to our above 2020 analysis, but haven’t prioritised cleaning it up / coming to agreements internally and presenting it externally. (We think that cost effectiveness per FTE has reduced, as we say in the review, but are not sure how much.)
The basic reasoning for that is:
We’ve found that these analyses aren’t especially valuable for people who are making decisions about whether to invest their resources in 80k (most importantly—donors and staff (or potential staff)). These groups tend to either a) give these analyses little weight in their decision making, or b) prefer to engage with the source material directly and run their own analysis, so that they’re able to understand and trust the conclusions more.
The updates since 2020 don’t seem dramatic to me: we haven’t repeated the case studies/DIPY analysis, Open Phil hasn’t repeated their survey, the user survey figures didn’t update me substantially, and the results of the EA survey have seemed similar to previous years. (I’m bracketing out marketing here, which I do think seems significantly less cost effective per dollar, though not necessarily per FTE.)
Coming to figures that we’re happy to stand behind here takes a long time.
~~~
This appendix of the 2022 review might also be worth looking at—it shows FTEs and a sample of lead metrics for each programme.