Not to karma-farm, but added a few more thoughts here
Sudhanshu Kasewa
Karma: 286
Following on from this post:
A few more things I often say that obliquely relate to networking:
Build in public and seek feedback. People like engaging with stuff online. Do stuff in public—writing, github projects, YouTube shorts, anything—and ask people for feedback. Repeatedly doing interesting things increases the quality of feedback and engagement in a positive cycle, and opens doors for deeper collaboration. Consider being pseudonymous if it is stressful to do it with own name.
Offer your services, volunteer, collaborate. Stolen from Laura’s piece on How to have an impact when the job market is not cooperating
Host events to attract the kind of people you want to engage with. IIUC this is how Mox SF started.
Consider doing things in-person. Lots of relationships are forged in post-event social settings.
Great post! Was just thinking about an intuition pump of my own re: EV earlier today, and it has a similar backdrop, of vaccine development. Also, you gave me a line with which to lead into it:
The work I do doesn’t end up helping other researchers get closer to coming up with a cure.
Oh but it could have helped! It probably does (but there are exceptions like if your work is heavily misguided to the degree that nobody would have worked on it, or is gated).
By doing the work and showing it doesn’t lead to a cure, you’re freeing someone else who would have done that work to do some other work instead. Assuming they would still be searching for a cure, you’ve increased the probability that the remaining researchers do in fact find a cure.
I encounter “in 99.9% of worlds, I end up making no progress” a lot in my work, and I offer in its place that it is important and valuable to chase down many different bets to their conclusions, that the vaccine is not developed by a single party alone in isolation from all the knowledge being generated around them, but through the collected efforts of thousands of failed attempts from as many groups. The victor can claim only the lion’s share of the credit, not all of it; every (plausible) failed attempted gets some part of the value generated from the endeavour as a whole, even ex post.
“anyone” is a high bar! Maybe worth looking at what notable orgs might want to fund, as a way of spotting “useful safety work not covered by enough people”?
I notice you’re already thinking about this in some useful ways, nice. I’d love to see a clean picture of threat models overlaid with plans/orgs that aim to address them.
I think the field is changing too fast for any specific claim here to stay true in 6-12m.
Signal boost: Check out the “Stars” and “Follows” on my github account for ideas of where to get stuck into AI safety.
A lot of people want to understand AI safety by playing around with code and closing some issues, but don’t know where to find such projects. So I’ve recently starting scanning github for AI safety relevant projects and repositories. I’ve starred some, and followed some orgs/coders there as well, to make it easy for you to find these and get involved.
Excited to get more suggestions too! Feel to comment here, or send them to me at sk@80000hours.org
Thanks. I sort of don’t buy that that’s what the Mechanize piece says, and in any case “no matter what you do” sounds a bit fatalistic, similar to death. Sure, we all die, but does that really mean we shouldn’t try and live healthier for longer?
Not directly relating to your claim, but:
The Mechanize piece claims “Full automation is desirable”, which I don’t think I agree with both a priori and after reading their substantiation. It does not contend with the possibilities of catastrophic risks from fully automating, say, bioweapon research and development; it might be inevitable, but on desirability I think it’s clear that it’s only desirable once—at the bare minimum—substantial risks have been planned for and/or suitably mitigated. It’s totally reasonable to delay the inevitable!
Thanks Matt. Good read.
A stronger technological determinism tempers this optimism by saying that the kinds of minds you get will be whichever are easiest to build or maintain, and that those quite-specific minds will dominate no matter what you do.
Is there a thing you would point to that substantiates or richly argues for this claim? It seems non-obvious to me.
I try to maintain this public doc of AI safety cheap tests and resources, although it’s due a deep overhaul.
Suggestions and feedback welcome!
Scrappy note on the AI safety landscape. Very incomplete, but probably a good way to get oriented to (a) some of the orgs in the space, and (b) how the space is carved up more generally.
(A) Technical
(i) A lot of the safety work happens in the scaling-based AGI companies (OpenAI, GDM, Anthropic, and possibly Meta, xAI, Mistral, and some Chinese players). Some of it is directly useful, some of it is indirectly useful (e.g. negative results, datasets, open-source models, position pieces etc.), and some is not useful and/or a distraction. It’s worth developing good assessment mechanisms/instincts about these.
(ii) A lot of safety work happens in collaboration with the AGI companies, but by individuals/organisations with some amount of independence and/or different incentives. Some examples: METR, Redwood, UK AISI, Epoch, Apollo. It’s worth understanding what they’re doing with AGI cos and what their theories of change are.
(iii) Orgs that don’t seem to work directly with AGI cos but are deeply technically engaging with frontier models and their relationship to catastrophic risk: places like Palisade, FAR AI, CAIS. These orgs maintain even more independence, and are able to do/say things which maybe the previous tier might not be able to. A recent cool thing was CAIS finding that models don’t do well on remote work tasks—only 2.5% of tasks—in contrast to OpenAI’s findings in GDPval suggests models have an almost 50% win-rate against industry professionals on a suite of “economically valuable, real-world tasks” tasks.
(iv) Orgs that are pursuing other* technical AI safety bets, different from the AGI cos: FAR AI, ARC, Timaeus, Simplex AI, AE Studio, LawZero, many independents, some academics at e.g. CHAI/Berkeley, MIT, Stanford, MILA, Vector Institute, Oxford, Cambridge, UCL and elsewhere. It’s worth understanding why they want to make these bets, including whether it’s their comparative advantage, an alignment with their incentives/grants, or whether they’re seeing things that others haven’t been able to see yet. (*Some of the above might be pursuing similar bets to AGI cos but with fewer resources or with increased independence etc.)
(v) Orgs pursuing non-software technical bets: e.g. FlexHEG, TamperSec
(B) Non-technical or less technical, but still aimed (or could be aimed) at directly** working the problem
(i) Orgs that do more policy-focussed/outreach/advocacy/other-non-technical things: e.g. MIRI, CAIS, RAND, CivAI, FLI, Safe AI forum, SaferAI, EU AI office, CLTR, GovAI, LawAI, CSET, CSER
(ii) AGI cos policy and governance teams, e.g. the RSP teams, the government engagement teams, and maybe even some influence and interaction with product teams and legal departments.
** “directly” here means something like “make a strong case to delay the development of AGI giving us more time to technically solve the problem”, a first-order effect, rather than something like “fund someone who can make a case to delay...”, which is a higher order effect
(C) Field-building/Talent development/Physical infrastructure
(i) Direct talent development: Constellation, Kairos, BlueDot, ARENA, MATS, LASR, Apart Research, Tarbell, etc. These orgs aim to increase the number of people going into above categories or speed them up. They don’t usually (aim to) work directly on the problem, but sometimes incidentally do (e.g. via high quality outputs from MATS). There can be a multiplier effect for working in such orgs.
(ii) Infra: Constellation, FAR AI, Mox, LISA
(iii) Incubators: e.g. Seldon Labs, Constellation, Catalyze, EF, Fifty-Fifty
(D) Moving money
(i) Non-profit/philanthropic donors: e.g. OpenPhil, SFF, EA Funds, LongView, Schmidt Futures
(ii) VCs: e.g. Halcyon, Fifty-Fifty
For added coverage,
(E) Others
(i) Multipolar scenarios: CLR, ACS Prague, FOCAL (CMU), CAIF
(ii) Digital consciousness type-things: CLR, Eleos, NYU Center for Mind, Ethics, and Policy
(iii) Post-AGI futures: Forethought, MIT FutureTech
(F) For-profits trying to translate AI safety work into some kind of business model to validate research and possibly be well situated should more regulation mandate evals, audit, certifications etc.: e.g. Goodfire, Lakera, GraySwan, possibly dozens more startups + big professional services firms would be itching to get in on this when the regulations happen.
It is very worth investigating whether to work on any of these: The field is wide open and there are many approaches to pursue. “Defence in depth” (1, 2, 3) implies that there is work to be done across a lot of different attack surfaces, and so it’s maybe not so central to identify a singular best thing to work on; it’s enough to find something that has a plausible theory of change, that seems to be neglected and/or is patching some hole in a huge array of defences—we need lots of people/orgs/resources to help with finding and patching the countless holes!
PSA: If you’re doing evals things, every now and then you should look back at OpenPhil’s page on capabilities evals to check against their desiderata and questions in sections 2.1-2.2, 3.1-3.4, 4.1-4.3 as a way to critically appraise the work you’re doing.
I was reminded of this post (Purchase Fuzzies and Utilons Separately), and it’s something I do myself: work in some speculative EV-maximising space, but donate to “definitely doing good” things.
Thanks for doing this, Ben!
Readers: Here’s a spreadsheet with the above Taxonomy, and some columns which I’m hoping we can collectively populate with some useful pointers for each topic:
Does [academic] work in this topic help with reducing GCRs/X-risks from AI?
What’s the theory of change[1] for this topic?
What skills does this build, that are useful for AI existential safety?
What are some Foundational Papers in this topic?
What are some Survey Papers in this topic?
Which academic labs are doing meaningful work on this topic?
What are the best academic venues/workshops/conferences/journals for this topic?
What other projects are working on this topic?
Any guidance on how to get involved, who to speak with etc. about this topic?
For security reasons, I have not made it ‘editable’, but please comment on the sheet and I’ll come by in a few days and update the cells.
[1] softly categorised as Plausible, Hope, Grand Hope
Hi! Thanks for sharing your story. Some quick thoughts:
Could you quit your part-time job and instead use that time better? Could you take on a different part-time position that lets you build useful skills and networks?
I think it’s probably okay to finish your masters (since you’re already halfway through it); if you can find a job that’s going to get you hands-on experience on, say, LLMs, and/or build skills that could be robustly useful in a variety of roles, it might tip the scales in favour of dropping out (but by then you’ll be even closer to finishing, so maybe it makes sense to just complete it).
With or without your masters, with or without your job, I think it’s useful to get situationally aware about what’s happening in AI, and get comfortable (proficient, even) with using AI tools to enhance your own productivity and growth.
Benjamin Todd wrote about how not to lose your job to AI. I might not fully endorse this piece, but I think it’s directionally correct and has some good ideas to think about.
All the best!
Hi Marc, thanks for the question.
Lots has been said about the value of PhDs:
Lewis Hammond gives advice here about doing PhDs
Rohin Shah talks a little about PhDs here in relation to AI safety work
Adam Gleave makes a positive case here for AI safety
[Caveat: having dropped out of a PhD myself, I might be biased against doing one.] I think our piece on doing PhDs mostly holds up, but I’d make a few updates away from doing one:
AI things might happen soon, and in many worlds it would be better to do good soon rather than build a lot of skill first. (This obviously doesn’t apply if you are able to do good via your PhD as well.)
For many of us, we’re likely to really succeed at a PhD only if we are obsessed about it: an An advisee once said to me “Only do a PhD, if it’s something that you would do for free, in your free time, after work, on nights and weekends.”
A softer version of this: I think many of us are not calibrated about what attitudes and behaviours help to do a PhD—I certainly wasn’t! Before committing, try to come to grips with what you’re signing up for.
Finally, I subscribe to a PhD not being an end-in-itself, and instead a way to get some role/job/opportunity, which otherwise might be very unlikely without it. In impact-focussed spaces, I don’t think there are that many opportunities gate-kept by a PhD credential: academia/professorship, the “research scientist” title (but not necessarily the work), and maybe some policy positions. Orgs and managers who care about impact, care more about “can do you the work?” rather than “do you have the credential?”; could you get the legible skills to “do the work” in a paid job with better hours in fewer years, or do you have to do a PhD instead?
Hope this helps! All the best.
Hi Adam, so exciting that you want to use your skills for doing good. I’d go even further and say that “doing good” is its own goal to shoot for, and I want more folks thinking about “What are the best opportunities to do the most good?” first, and only then filtering by some subset of their relevant skills that might make them a good fit.
This is described more in Part 4 our career guide, where we outline a framework—Scale, Tractability, Neglectedness—to identify global issues with some of the highest opportunities for positive impact. We’ve applied such an analysis in our work for many years, and have in-depth articles on what we believe to be some of the world’s most pressing problems.
We also host a job board to aggregate opportunities aimed at helping with these. If you’re looking for more “goal-directed” next steps, perhaps scanning through those jobs can give you a sense of what the world needs right now, and how you can help. All the best!
Hi alphaplus, thank you for the questions. I’m glad to hear your health is improving.
I want to start by saying: your (written) English seems fine! Even if you’re concerned about your speaking skills, you can always lean in to your written ability to connect, exchange ideas, and grow.
Without knowing more, it’s hard to give very tailored advice, so here are some messages I think more folks should take seriously:
AI could be a big deal, soon. It could create huge dangers.
In light of this, lots of stuff needs doing, e.g. technical research, governance, cybersecurity, international cooperation.
even if the most egregious risks of AI don’t materialise soon (or at all), I’ll claim (albeit without justification) that having an understanding of how these technologies are transforming the world puts one is a good place to help out in many future scenarios.
As a result, I advocate more people develop ‘situational awareness’, and make their plans keeping (the possibility of) rapid AI progress in mind.
To your main question of “What would you do if you were in my position?”, there are several ways to progress. One procedure is articulated here:
Make some best guesses (hypotheses) about which options seem best.
Identify your key uncertainties about those hypotheses.
Go and investigate those uncertainties.
The key point is to try things, get feedback, and update your beliefs, and try again. Once you have more clarity, you’ll be able to aim for and commit to specific paths.
Finally, there are no real barriers to entry to engaging with Effective Altruism! If you think you’ll find value in connecting with folks in the community, you absolutely should. In addition to this Forum, there are plenty of other spaces, e.g. EAG(x) events you can attend, or slack channels you can join.
Hi Stephanie,
Thanks for writing in. It’s great that you’re thinking of using your career to help AI go well, and have been building skills and applying for roles to that end.
I’m sorry to hear you’ve been struggling with landing a role. Here are some ideas:
Are you getting enough feedback on your application materials? It’s good to solicit input from trusted sources, mentors, others on similar journeys as you, so that you’re sure that you’re not missing some key pieces in your applications.
Are you talking/connecting to people in roles/orgs you’re interested in? As we get more experience, it’s more likely that we’ll find roles through our network and collaborators; so it’s useful to invest in those relationships, not just by being a jobseeker, but by aiming to help out and add value wherever you can (maybe by giving feedback/critiques, exchanging your expertise for something you want to learn, or volunteering in some way).
Are you continuing to make your skills and experience legible? As my colleague @Matt Beard puts it “you should obsessively improve at an in-demand skill set in a legible way”. Those skills could be within writing, speaking, research, analysis, code, hardware, interpersonal collaboration, project management, organisation building, strategic thinking, and so on. The idea here is analogous to “build it and they will come”, and countless folks have translated their visible expertise into high-impact roles. Check out our skills pages for more details.
A related thing is to use such public productivity and output as a way to increase your feedback surface area, to pick up on where you can grow. Aim to post your work in places where folks are happy to engage in good faith and offer constructive input. The EA Forum and LessWrong are great places for this!
Are you maybe the right person to start a new org working on something because nobody else is doing it, and it’s incredibly important? I often say in advising calls “You can just do things”, because it’s true and sometimes we forget that. Yes, it can be daunting, and it’s worth considering your own personal circumstances, but all things considered, I want more people to be willing to take those kinds of risks.
Similarly, you might even want to implement someone else’s idea, or replicate or improve on an existing project—there are plenty of excellent ideas out there that need more people executing them.
Some more resources:
Hope this helps! All the best.
Quick thoughts:
Great that you have work like the arxiv paper! You could even explicitly ask for feedback on that work
Make it easy for people to understand your work: Try and answer questions like “Why did I do this? What did I learn and/or what update did I make? What is my theory of change?”, and so on...
Make it easy for people to engage with your work: Display it prominently, tweet about it, write a blogpost on lesswrong about it. Polish and publish the code base (see an example here), and so on...
Everyone has their own style of building relationships. I think a powerful way to do so is to try and add value to others: can you summarise/discuss their work in public, or give them feedback, or extend it in an interesting way? Are there volunteer or part-time opportunities that you can help out with? Can you identify issues in their codebases and improve them?
Got sent a set of questions from ARBOx to handle async; thought I’d post my answers publicly:
Can you explain more about mundane utility? How do you find these opportunities?
Lots of projects need people and help! E.g. Can you contribute to EleutherAI, or close issues in Neuronpedia? Some more ideas:
Contribute to the projects within SK’s github follows and stars
Make some contributions within Big list of lists of AI safety project ideas 2025
Reach out to projects that you think are doing cool work and ask if you can help!
BlueDot ideas for SWEs
I’m an experienced software engineer. How can I contribute to AI safety?
The software engineer’s guide to making your first AI safety contribution in <1 week
From a non-coding perspective, you could e.g.
Facilitate BlueDot courses
Give people feedback on their research proposals, drafts, etc.
Be accountability partners
Offer to talk to people and share what you know with those who know less than you
Check out these pieces from my colleagues:
How to have an impact when the job market is not cooperating by Laura G Salmeron
Your Goal Isn’t Really to Get a Job by Matt Beard
What is your theory of change?
As an 80k advisor, my ToC is “Try and help someone to do something more impactful than if they had not spoken to me.”
Mainly, this is helping get people more familiar with/excited about/doing things related to AI safety. It’s also about helping them with resources and sometimes warm introductions to people who can help them even more.
Are there any particular pipelines / recommended programs for control research?
Just the things you probably already know about – MATS, Astra are likely your best bets, but look through these papers to see if there are any low hanging fruit as future work
What are the most neglected areas of work in the AIS space?
Hard question, with many opinions! I’m particularly concerned that “making illegible problems legible” is neglected. See Wei Dai’s writing about this
Legible vs. Illegible AI Safety Problems
Problems I’ve Tried to Legibilize
More groundedly, I’m concerned we’re not doing enough work on Gradual Disempowerment and more broadly questions of {how to have a flourishing future/what is a flourishing future} even if we avoid catastrophic risks
In general, AI safety work needs to contend with a collection of subproblems. See davidad’s opinion – A list of core AI safety problems
There are many other such opinions, and it’s good to scan through them to work out how they’re all connected, so that you can see the forest for the trees; and also to work out which problems you’re drawn to/compelled by, and seek out what’s neglected within those 🙂
Some questions about ops-roles:
What metrics should I use to evaluate my performance in ops/fieldbuilding roles? I find ops to be really scattered and messy, and so it’s hard to point to consistent metrics.
Hard to talk about this in concrete terms, because ops is so varied; every task can have its own set of metrics. Instead, think through this strategically:
Be clear on the theory(ies) of change, and your roles/activities/tasks in it(them). Once you can articulate those things, the metrics worth measuring become a lot clearer
Sometimes we’re not tracking impact because impact evaluation is notoriously difficult. Look for proxies. Red-team them with people you admire
Fieldbuilding metrics can be easier to generate, but I don’t claim to be an expert here – ask folks at BlueDot, or the fellowships for better input.
How many people completed the readings?
How many people did I get to sign up for the bluedot course?
How many of those finished the bluedot course?
How many people did I get into an Apart Hackathon?
Did any of my people win?
And so on…
Likewise, I have a hard time discerning what “ops” really means. What are the best tangible “ops” skills I should go out of my way to skill up on if I want to work in the field building/programmes space? Are there “hard” ops skills I should become really good at (like, familiarity with certain software programmes, etc)
Ops is usually a “get stuff done” bucket of work. Yes, it can help to have functional experience in an ops domain like “Finance” or “IT/office tech infra/website” (and especially “Legal”), but a LOT of ops can be learned on the job/on your own; AI safety is stacked full of folks who didn’t let “I don’t know anything about ops” stop them from figuring it out and getting it done
Under what circumstances should a “technical person” consider switching their career to fieldbuilding?
First things first:
Fieldbuilding is not a consolation prize. Do fieldbuilding if you’re really passionate about helping AI go well, and fieldbuilding is your comparative advantage.
And doubling down on that:
It really really really helps if fieldbuilders are very competent. A fieldbuilder who doesn’t know their shit about AI risk and AI safety can propagate bad ideas among the people they’re inducting into the field.
This can have incredibly high costs
Pollutes the commons
Wastes time downstream where all this would need to be corrected
Bounces people who might be able to quickly get up to speed, because their initial contact with these fieldbuilders is of poor quality, poor argumentation, poor epistemics
Conversely a great fieldbuilder is one who knows how to tend their flock, what they need to prosper and grow to become competent at thinking about AI safety properly, and being able to do AI safety things
How would you recommend going about doing independent project work for upskilling in-place of doing something like SPAR or MATS?
Why not both? In general, I want people to ask themselves this question when making decisions. You can do a lot more than you give yourself credit for.
At the current margins SPAR, MATS etc. are probably better than independent work
Some of these fellowships have pretty high signal to employers (based on evidence that has been generated over time)
There is a lot that these fellowships offer that are sometimes hard to get without them
Research support, mentorship, community engagement, well-scoped projects with deliverables and accountability
Also softer things like physical space , some money
But if you’re great at doing stuff independently, go for it! Neel Nanda didn’t need a fellowship.
A key idea is to keep your eye on the ball – be productive!
The point is generate outputs
That make you learn
That show that you have learned
That are related to AI safety
That get feedback
That show that you update based on (relevant/good/high-quality) feedback
lfg!