[Question] How did the AI Safety talent pipeline come to work so well?

[AlejoAcelas🔸]

I think the EA/​AIS community has been remarkably successful at recruiting and deploying talent to address AI misalignment and misuse risks. I’d like to reverse engineer their approach to see if we can apply a similar recipe to other “grand challenges” emerging from rapid AI progress.

What’s missing from my naive explanation for the recent success of the AIS community?

• Conceptual Foundations (pre-2016): Early conceptual work established the importance of AI Safety and clarified the most valuable lines of action

  • Examples: Nick Bostrom’s Superintelligence, The Sequences and Lesswrong, Amodei et al.’s Concrete Problems in AI Safety

• Skill and career capital development (~2016-2021): Those producing conceptual work, along with their most engaged supporters, began doing direct work in the area. While much pre-LLM work may not have been directly impactful, it built crucial skills and career capital through organizations, connections, experience running successful programs, etc.

  • Examples: Open Philanthropy’s AI risks program, EA community consolidation (EAGs, university groups), AGI Safety Fundamentals course, CSET, CHAI, SERI MATS

• Talent conversion (2022 onwards): When AI gained broader attention and impact opportunities became clearer, there was already a stock of arguments for AI Safety’s importance and a community ready to address talent pipeline bottlenecks and enroll more people in useful work.

  • Examples: Open Philanthropy’s RFPs, MATS, BlueDot, Anthropic and DeepMind AIS teams, ARENA, AI-risk-focused events within EA

In this simplified story, if we want to draw more talent to newly formulated AI challenges, the most important things are:

• Broadcasting the case for working in the area and identifying concrete lines of action

• Developing a base of early practitioners with the skills and career capital needed to pave the way for future work

• Creating a large influx of people with initial inclination to work in the area (e.g., from the EA community)

[Anton Zheltoukhov]

that’s just false. alignment community even to this day is significantly less then 1000 people worldwide. i would not call that a success. i’m not blaming the people involved. they’ve done what they could. lets just not deceive ourselves here

[AlejoAcelas🔸]

Well, AI sentience probably has <100 people and I’d be joyful if that increased to 1k. I’m still interested even if the question is “How can we fail less misserably at creating a talent pipeline?”