A naive analysis on if EA is Talent constrained

agent1823 Mar 2020 23:09 UTC
30 points
17 comments15 min readEA link

I am deeply grateful to Aaron Gertler who reviewed this article. His comments were very thorough (he didn’t leave any hyperlink unclicked). He managed to question several of my claims. I have updated parts of this post based on his comments. I also want to thank Carrick Flynn, Peter Hurford, EA Applicant, Jon Behar, Ben West, 80,000 hours for helping me directly or indirectly. They provided valuable info in their posts/​comments/​email which I have used in this article. That said, again, they should not be viewed as endorsing anything in this. All mistakes are mine. All views are mine.

Introduction

I have been using 80,000 Hours (80k) since 2017 and have read almost all their posts, spent weeks after weeks reading them to figure out what I should be doing in life. They seem to have done a ton of research and put out many many posts, for us readers to benefit from.

In the process they have made a lot of claims which are hard and time-consuming to verify as we don’t have the insights, contacts or the data that 80k is exposed to. For example they estimate “an additional person working on the most effective issues will have over 100 times as much impact as an additional person working on a typical issue”. To verify this with one example, I would need estimates from say Open Phil on the impact of an employee. I tried, but they are unable to put effort into it at the moment.

Maybe 80k can be asked for clarification directly? Unfortunately, 80k doesn’t seem approachable other than through coaching[1] (which is only for the stellar). Comments sections seem to be deserted to ask for help, and at the time, I didn’t know of any other sources doing this sort of research and coaching for people[2]. Based on reading 80k for years I formed the impression as shared by fellow EA applicant:

Hey you! You know, all these ideas that you had about making the world a better place, like working for Doctors without Borders? They probably aren’t that great. The long-term future is what matters. And that is not funding constrained, so earning to give is kind of off the table as well. But the good news is, we really, really need people working on these things. We are so talent constrained...--- EA applicant in the EA Forum

And looking at the 277 karma this post got (the highest of any post on the Forum), it might appear that a “lot of people” share(d) this sentiment that EA orgs could potentially be seriously Talent Constrained (TC).

A few weeks back I stumbled upon some articles in the EA forum and to my surprise it appeared that some EA orgs were suggesting that they were not TC. Until this point I don’t think it occurred to me that 80k’ claims (“EA is TC”) could be wrong or lost in translation or that I should test it. Nevertheless, having seen orgs say otherwise, it felt like a good idea to dig into it at least now.

The following article is my naive investigation on if EA is TC. Before we start going deep into whether EA is TC or not, we must first state the definition clearly.

Definitions

We are going to primarily deal with the term “Talent Constrained” (TC). 80k defines TC in “Why you should work on Talent gaps” (Nov 2015) as,

For some causes, additional money can buy substantial progress. In others, the key bottleneck is finding people with a specific skill set. This second set of causes are more “talent constrained” than “funding constrained”; we say they have a “talent gap”.

So, a cause is TC if finding people with a specific skill set, proves to be difficult. The difficulty I assume is in the lack of those skilled people, and not some process/​management constraint[3]. “EA Concepts”, clears this confusion up with a better worded “example”:

Organization A: Has annual funding of $5m, so can fund more staff, and has been actively hiring for a year, but has been unable to find anyone suitable… Organization A is more talent constrained than funding constrained...

In this post, discussions are focused on Orgs that are TC and not Causes that are TC. When I read that AI strategy is TC with the lack of “Disentanglement Research” (DR), I don’t know what to do about it. But if I know FHI and many other orgs are TC in DR, then I could potentially upskill in DR, and close the talent gap. So looking at causes for me, is less helpful, less concrete and is not what I have set out to understand.

Why I think EA is TC

EA has been and is talent constrained, according to surveys based on input from several EA orgs[4]: 2017 survey, 2018 survey, 2019 survey. These surveys were conducted by 80k and CEA. In all the surveys EAs on average claim to be more Talent Constrained than Funding Constrained. For example, in 2019 EA orgs reported feeling more Talent Constrained (3 out of 5 rating) and less Funding Constrained (1 out of 5 rating)[5].

80k doesn’t seem to have changed it’s position on this matter since a while. In 2015, 80k suggested that we should focus on providing talent to the community rather than ETG, in “Why you should focus on talent gaps and not funding gaps”. One of the examples they give is about AI Safety where there are people who are ready to donate even more funds, but think there isn’t enough “talent pool”. More posts such as “Working at EA orgs (June 2017), “The world desperately needs AI strategists (June 2017), “Why operations management is the biggest bottleneck in EA” (March 2018), and High-Impact-Careers (Aug 2018), continue to make the case for EA orgs being TC. Even in their recent post, “Key Ideas” (October 2019)--which is mostly recycled from the 2018 article on High-Impact-Careers—they continue to say that the bottleneck to GPR for example, is researchers and operations people[6].

In Nov 2018, they wrote a post to clarify any misconceptions regarding the understanding of the term TC: “Think twice before talking about Talent gaps”. 80k informs us that EA orgs are not TC in general but are TC by specific skills. Some examples (according to them) being, people capable of Disentanglement Research in Strategy and Policy (FHI, OpenAI, Deepmind), dedicated people in influential government positions etc… This is great, the claim is becoming narrower: EA is TC in specifically X. So what is this X?

Where is the EA specifically TC

There seem to be a list of posts from 80k from which we can gather where EA is specifically TC. They are:

The surveys from 2017 to 2019 that informed us that the EA Orgs are TC, provide information on “what sort of talent the EA orgs and EA as a whole would need more of, in the next 5 years?”. This question sounds like a proxy to “Where is EA specifically TC?”. 80k seems to agree with this proxy-approximation of the question as evidenced here[7] and here[8]. The top 7 results (out of 20 or so) are below:

2017 2017 (EA) 2018 2018 (EA) 2019 2019 (EA)
1 GR G&P Oper. G&P GR G&P
2 Good Calib. Good Calib. Mngment Oper. Oper. Mngment
3 Mngment Mngment GR ML/​AI Mngment GPR
4 Off. mngers ML/​AI ML/​AI Mngment ML Founding
5 Oper. Movt. build GPR GPR Econ/​math Soc. Skill
6 Math GR Founder GR HighEA* ML/​AI
7 ML/​AI Oper. Soc. skill Founding GPR Movt. Build

* High level overview of EA
*** Government and Policy

For the talents that are unclear[9], I am unable to do anything with them at the moment. For the ones that I have clear examples for, I proceed further.

Another way to arrive at or to supplement this list, is to look at the top problem profiles and check what the bottlenecks are. For example, in the profile on shaping AI (March 2017), we see that 80k calls for people to help in AI Technical research, AI Strategy and Policy, Complimentary roles and, Advocacy and Capacity building. So basically EVERYTHING IN AI except ETG, is TC (it appears). In the problem profile on GPR (July 2018), 80k suggests that they mainly need researchers trained in math, econ, phil etc… Also needed are academic managers and operations staff. A very similar story for working at EA orgs as well.

Is it just me or is EA TC in “general”? Like when researchers, operations people and managers are in shortage at GPR orgs, AI orgs and other EA orgs, then who else is left?

In the post on High Impact Careers (August 2018), 80k suggests the following priority career paths and what they are constrained by:

In brief, we think our list of top problems (AI safety, biorisk, EA, GPR, nuclear security, institutional decision-making) are mainly constrained by research insights, either those that directly solve the issue or insights about policy solutions. --- High Impact Careers

In focused bottleneck posts for Operations and AI Strategy just the title already informs how TC the situation is:

In conclusion, the surveys say GRs, ML/​AI people, GPR people and movement building are TC (2019). The problem profiles seem to suggest that GPR and AI are completely TC except for ETG (2017,2018). Whereas the High-impact-careers post says that research insights (good researchers) and policy solutions (good policy people) are the most constrained (2018). It appears that there is some discrepancy between different articles—every article doesn’t seem to say the same thing—but we move on with the key message that all these things listed could be potentially TC. But are they really TC though?

The Evidence

GR in GPR

Researchers in GPR are claimed to be constrained. GR’s also stand on top of the survey lists shown before, for 2019. Yet, Open Phil seems to paint a very different picture. For the recent hiring round by Open Phil (started in Feb 2018 and ended in December 2018) they wanted to hire 5 GRs. They report that more than a 100 strong resumes with missions aligned to that of Open Phil were received. 59 of them were selected after remote work tests and went into an interview. Of this, 17 of them were offered a 3 month trial and 5 selected in the end. “Multiple people” they met in this round are claimed to have potential to excel in roles at Open Phil in the future. Open Phil does not seem to feel that there is a lack of skilled people. It appears that they had plenty to choose from and that they have found suitable candidates.

A similar case is observed with EAF. In EAF’s November 2018 hiring round they wanted to hire 1 GR (for grant evaluation) and 1 operations person. Within just 2 weeks, 66 people applied to this EA org which was in a non-hub[10]. These 66 trickled down to 10 interviews after work tests, 4 were offered trials and 2 were selected in the end. No TC in GR here either.

Would Open Phil like to hire more GRs? For sure, but they don’t have the capability to deploy such a pool of available talent, they say. They seem to be constrained by something else, something not “talent”.

AI Strategy and Policy

Researchers in AI Strategy and Policy are also claimed to be constrained. The surveys echo the same as well. But Carrick from FHI (Sep 2017) suggests that AI Policy implementation and research work is essentially on hold until Disentanglement Research progresses. And that even “extremely talented people” will not be able to contribute directly until then. Similar to Open Phil, institutional capacity to absorb and utilize more researchers in Strategy is constrained, according to Carrick. It must be noted that this is just one persons view on the matter and that a stronger version of evidence for this would be if several AI orgs agreed with Carrick’s view.

Except for the TC in Disentanglement research (DR)---where there seems to be large demand and if you meet the bar, you will get a job—there seems to be no sign of TC in Strategy and Policy, at the moment.

Once DR progresses, there would be a need for “a lot of AI researchers”, Carrick expects. It’s been 2.5 years since the post by Carrick, and as late as Nov 2018, 80k continues to cite Carrick’s article. This seems to suggest that not much might have changed. I have tried requesting Carrick to write a reboot of his initial post and hopefully he can further clarify the TC or lack there of.

Researchers and Management staff in other EA orgs

The co-founder and board member of Rethink Charity seems to suggest that both senior and junior staff for Rethink Charity and Charity Science were not hard to find, aka not TC.

I’ve certainly had no problem finding junior staff for Rethink Priorities, Rethink Charity, or Charity Science (Note: Rethink Priorities is part of Rethink Charity but both are entirely separate from Charity Science)… and so far we’ve been lucky enough to have enough strong senior staff applications that we’re still finding ourselves turning down really strong applicants we would otherwise really love to hire.---Peter Hurford says in the 2019 survey

The Life You Can Save’s Jon Behar, agrees with Peter. He adds that it’s not the lack of talent but the lack of money to add new staff which is the bottleneck for TLYCS.

Charity Entrepreneurship’s incubation program has grown from 140 applications to ~2000 applications for 15-20 positions since last year. It’s plausibly not TC this year atleast.

Conclusion

An org is TC in Talent X, if it is not able to find “skilled” people despite “hiring actively”. So far we have seen that Open Phil, EAF, Rethink Charity, Charity Science, TLYCS and FHI, are able to find the skilled people they need—except for one concrete example of Disentanglement Research in FHI (and possibly similar institutes). Contrary to the claims from 80k, it appears that several orgs are not TC.

I am really upset with 80k. First it was focusing too much on career capital (CC) and positions like management consulting, and now TC. Getting into the TC debate only opened a Pandora’s box of more issues. Recently, I discovered that their discussion on replaceability is plausibly wrong. They have gone back and forth[11] on it in the past and currently have suggested that it depends. They ended up inflating impact associated with people working in EA orgs and have now taken it back. They severely downplayed how competitive it is to get jobs in EA orgs[12]. And there are so many cases[13] of people who feel the same way, not without reason. I traveled with 80k on the CC hype and spent months on identifying positions of “maximum CC”[14]. Then I did a 1 year course of Data Science at Coursera. After that I jumped onto the work-at-an-EA-org-because-TC hype and was just about to upskill in statistics and apply for GR positions because they need me.

So many crucial mistakes that cost people like me and others[13:1] a lot of time, and the world a “lot of” dollars. And when someone requests one of the members of 80k to not just serve the elite and that perhaps maybe invest in a small conversation with the non-elite EAs to save them years of wasting time, there is no reply.

Thus, I find it very hard to trust the claims listed in 80k. And there are so many of those claims in every post and it’s just impractical to verify each one of them. Rather than relying on the interpretation of English[15] and generalization of advice for everyone, I find EA forums a much easier place to get information from, challenge claims and get responses for (quickly). I found most of the evidence against TC including the Pandora’s box of issues, there. A lot of the successful people from the EA world seem approachable there with chats, comments and AMAs. Recently I was able to chat with Ben West, Aaron Gertler, Peter Hurford, Jon Behar, Jeff Kaufman and Stefan Torges. A bigger celebrity list of people can be seen commenting in posts, such as Carrick Flynn and 80k’s very own Rob Wiblin.

Final message

Caution: Just because an org is not TC, it doesn’t mean that you should reject that org.

Why is this debate so important?

Whether an org is TC or not, has implications on the impact made. The true impact you make when a job is TC at an EA org is (much) higher, than when the job is not TC. A junior GR at GiveWell is expected to move 2.4m$ if the job was TC. The same GR is expected to move only 244k in the case that the hired GR is better than the next-best-candidate by 10% (Not TC). Such is the distinction between being TC and not.

The above example assumes no spillover effects. But is that correct? Why is there no spillover? Should I work in EA or not? How much value do people really get out of working at an EA org? What is best path for my aspiring EA career?

Stay tuned...

Footnotes

  1. ↩︎

    I approched with questions on DS and they informed me that they don’t give advice over email. I applied for coaching and didn’t make the cut. What I asked?

    • Do I gain sufficient skills to migrate to Direct work (say analyst in GiveWell) having worked in Management consulting (M.C) for 5 years?

    This is in case things don’t seem to work out towards becoming a partner.

    • Are there examples of high impact direct workers who came from M.C?

    I would like to scan their profile to get a feel of what is possible.

  2. ↩︎

    EAF seems to offer career coaching here.

    EAF’s operations analyst is also doing coaching here.

  3. ↩︎

    If 80k on the other hand suggested that TC included everything that made it hard (such as hiring bottleneck) to find people with specific skillsets then TC is such a misnomer. Joel from EA forum puts it well:

    I could be mistaken, but it would seem odd to say you’re “funding constrained” but can’t use more funding at the moment. Whereas we are saying orgs are “talent constrained” but can’t make use of available talent… I feel a “talent bottleneck” implies an insufficient supply of talent/​applicants, which doesn’t seem to be the case. I guess it’s more that there’s insufficient talent actually working on the problems, but it’s not a matter of supply, so it’s more of a “hiring bottleneck” or an “organizational capacity bottleneck”.---Joel EA Forum

  4. ↩︎

    2018 survey includes:

    80,000 Hours (3), AI Impacts (1), Animal Charity Evaluators (2), Center for Applied Rationality (2), Centre for Effective Altruism (2), Centre for the Study of Existential Risk (1), Berkeley Center for Human-Compatible AI (1), Charity Science: Health (1), DeepMind (1), Foundational Research Institute (2), Future of Humanity Institute (2), GiveWell (1), Global Priorities Institute (2), LessWrong (1), Machine Intelligence Research Institute (1), Open Philanthropy Project (4), OpenAI (1), Rethink Charity (2), Sentience Institute (1), SparkWave (1), and Other (5)

  5. ↩︎

    Funding Constrained

    1 = how much things cost is never a practical limiting factor for you; 5 = you are considering shrinking to avoid running out of money

    Talent constrained

    1 = you could hire many outstanding candidates who want to work at your org if you chose that approach, or had the capacity to absorb them, or had the money; 5 = you can’t get any of the people you need to grow, or you are losing the good people you have

  6. ↩︎

    80k about GPR: “To make this happen, perhaps the biggest need right now is to find more researchers able to make progress on the key questions of the field. There is already enough funding available to hire more people if they could demonstrate potential in the area (though there’s a greater need for funding than with AI safety)”

    “Another bottleneck to progress on global priorities research might be operations staff, as discussed earlier, so that’s another option to consider if you want to work on this issue.”

  7. ↩︎

    These positions are both our own assessment and backed up by results of our surveys of community leaders about talent constraints, skill needs and key bottlenecks.”

  8. ↩︎

    What skills are the organizations most short of?

  9. ↩︎

    There are several talents listed in the surveys which I don’t understand. I don’t have any examples for what they could mean. For example, “Communications other than marketing and movement building”, “high level knowledge and enthusiasm about effective altruism” and “broad general knowledge about many relevant topics”. Some of the other “talents” mentioned seem too generalized. When I think of “one-on-one social skills”, it could be referring to anything like policy people talking to politicians, or Career Counselors convincing people to change their career path, or even people in the frontline of fundraising. If the surveyors wanted to inform the community that frontline fundraisers are required with “good social skills” (whatever that means), then exactly that in the survey seems much more beneficial than what they have currently done. Contrast this to talents such as GR or Operations. It is clear what these mean. For GR I can think of researchers at Open Phil or GiveWell. For operations I think of Tara from FHI.

  10. ↩︎

    … following list for SF area: 80khours (SF, Oxford), GiveWell (San Francisco), Open Philanthropy project (San Franscisco), 80khours (Oxford and SF), OpenAI (SF), MIRI (Berkeley), Center for Applied Rationality (Berkeley), AI Impact (Berkeley), Animal Charity Evaluator (Berkeley without any office space), Animal Equality (US UK),

    The following for UK area: Center for Study of Existential Risk (Oxford), Future of Humanity Institute (Oxford, UK), Global Priorities Institute (Oxford), Sentience Institute (London), Giving What We Can (Oxford), Founders Pledge (London), Centre for Effective Altruism (Oxford). Against Malaria Foundation (St. Albans UK), Sightsavers (U.K), Founders Pledge (London).

    The following for other areas: Evidence Action (Washington DC), Helen Keller International (Washington D.C), Give Directly (NYC), Poverty Action Lab (Cambridge, MA, US), Good Food Institute (Washington D.C, US), Center for Global Development (Washington D.C, US).

    So for the EA community it looks like the clustering does happen in UK (Oxford, London) and San Francisco area. These regions are the only regions that host the annual EA conferences, not surprisingly.

  11. ↩︎

    2012: They seem to be suggesting here while talking about doctors, aid workers, campaigners, “That’s because careers that are normally thought to be ethical tend to be extremely competitive. That means that if you don’t take the job, someone else will take your place.”

    2014: They went on to suggest that replaceability might not be as important as you might think. In 2017 they seem to continue to promote that idea in “Working at effective altruist organizations”.

    In 2019 article on “how replaceable are top candidates in large hiring rounds”, they seem to suggest that it depends on the type of distribution of the candidates (log normal or normal).

  12. ↩︎

    Claim: “If you get involved in the community, and prove your interest and general competence, there’s a decent chance you’ll be able to find a role regardless of your qualifications and experience.”

    Example: EA applicant from EA forum.

    He applied to 20 jobs. He didn’t get a single job, neither did his friends—with the characteristics as above—get jobs. His profile seems to match the one in the claim.

    Note: The claim says “decent chance” and not “for sure” though. I give them that. Although many people seem interpret it differently.

  13. ↩︎↩︎

    Links of posts where people were completely misinformed about how competitive the EA world is (look in the comments as well):

    1. https://​​forum.effectivealtruism.org/​​posts/​​jmbP9rwXncfa32seH/​​after-one-year-of-applying-for-ea-jobs-it-is-really-really

    2. https://​​www.facebook.com/​​groups/​​473795076132698/​​permalink/​​1077231712455695/​​

    3. https://​​physticuffs.tumblr.com/​​post/​​183108805284/​​slatestarscratchpad-this-post-is-venting-it

  14. ↩︎

    I am not a big fan of these broad terminologies as they don’t allow ME to act on them. For example, “Best ways to gain Career Capital (CC) are: Work at a growing organisation that has a reputation for high performance; Getting a graduate degree; Working in Tech sector; Taking a data science job; Working in think tanks; Making “good connections”, Having runway etc… ” Literally everything under the sun.

    I am unable to act on it. I could in theory pursue everything. I don’t know how to compare which has higher CC and lower CC. The definition says: “CC puts you in a better position to make a difference in the future, including skills, connections, credentials and runway.” When I work in Data Science in a FAANG job do I have higher CC compared to when I work on a computer science degree? I don’t know.

    Economists routinely measure the impact of high-school dropout vs high-school diploma vs some years of college but dropout vs undergrad degree vs grad degree, in different fields, using the variable “median weekly earnings” or “lifetime earnings”. So when someone says, “you need a degree to get ahead in life”, I can imagine what they mean $470 weekly wage increase. Whereas when someone says, “Computer Science PhD is good CC”, I am lost. Contrast that to saying “Best ways to gain CC is by looking at earnings”. Then I could look at median earnings for Data Science Faang job vs Phd in computer science in say top 20 university (based on my capability) and get ahead in life.

  15. ↩︎

    “If you get involved in the community, and prove your interest and general competence, there’s a decent chance you’ll be able to find a role regardless of your qualifications and experience.” --- 80k

    This seems to imply to me that people like EA applicant should have gotten a job. But he didn’t. I think examples would be much better to understand what they mean. What does decent chance mean?

agent1823 Mar 2020 23:09 UTC
30 points
17 comments15 min readEA link