Ajeya

Karma: 2,037

Ajeya 6 Oct 2025 17:28 UTC
4 points
2 ∶ 1
in reply to: elifland’s comment on: Ajeya’s Quick takes

Is the 1-3% x-risk from bio including bio catastrophes mediated by AI (via misuse and/or misalignment? Is it taking into account ASI timelines?

I’m largely deferring to ASB on these numbers, so he can potentially speak in more detail, but my guess is this includes AI-mediated misuse and accident (people using LLMs or bio design tools to invent nastier bioweapons and then either deliberately or accidentally releasing them), but excludes misaligned AIs using bioweapons as a tactic in an AI takeover attempt. Since the biodefenses work could also help with the latter, the importance ratio here is probably somewhat stacking the deck in favor of AI (though I don’t think it’s a giant skew, because bioweapons are just one path to AI takeover).

ASB has pretty short ASI timelines that are broadly similar to mine and these numbers take that into account.

Also, just comparing % x-risk seems to miss out on the value of shaping AI upside / better futures, s-risks + acausal stuff, etc. (also are you counting ai-enabled coups / concentration of power?). And relatedly the general heuristic of working on the thing that will be the dominant determinant of the future once developed (and which might be developed soon).

If you feel moved by these things and are a good fit to work on them, that’s a much stronger reason to work on AI over bio than most people have. But the vast bulk of generalist EAs working on AI are working on AI takeover and more mundane misuse stuff that feels like it’s a pretty apples-to-apples comparison to bio.

Ajeya 2 Oct 2025 18:20 UTC
71 points
8 ∶ 11
on: Ajeya’s Quick takes
I bet a number of generalist EAs (people who are good at operations, conceptual research / analysis, writing, generally getting shit done) should probably switch from working on AI safety and policy to working on biosecurity on the current margin.

While AI risk is a lot more important overall (on my views there’s ~20-30% x-risk from AI vs ~1-3% from bio), it seems like bio is a lot more neglected right now and there’s a lot of pretty straightforward object-level work to do that could take a big bite out of the problem (something that’s much harder to come by in AI, especially outside of technical safety).

If you’re a generalist working on AI because it’s the most important thing, I’d seriously consider making the switch. A good place to start could be applying to work with my colleague ASB to help our bio team seed and scale organizations working on stuff like pathogen detection, PPE stockpiling, and sterilization tech. IMO switching should be especially appealing if:
- You find yourself unsatisfied by how murky the theories of change are in AI world and how hard it is to feel good about whether your work is actually important and net positive
- You have a hard sciences or engineering background, especially mechanical engineering, materials science, physics, etc (or of course a background in biology, though that’s less necessary/relevant than you may assume!)
- You want a vibe of solving technical problems with strong feedback loops rather than a vibe of doing communications and politics, but you’re not a good fit for ML research
To be clear, bio is definitely not my lane and I don’t have super deep thinking on this topic beyond what I’m sharing in this quick take (and I’m partly deferring to others on the overall size of bio risk). But from my zoomed-out view, the problem seems both very real and refreshingly tractable.
What links here?

Ajeya 27 Oct 2023 0:10 UTC
3 points
0 ∶ 0
in reply to: Mishaal’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Thanks Mishaal!
1. I think previous experience taking on operationally challenging projects is definitely the most important thing here, though it may not necessarily be traditional job experience (running a student group or local group can also provide good experience here). Beyond that, demonstrating pragmatism and worldliness in interviews (for example, when discussing real or hypothetical operational or time management challenges) is useful.
2. I think an important quality in a role like this is steadiness — not getting easily overwhelmed by juggling a lot of competing tasks, having the ability to get the easy stuff done quickly and make smart calls about prioritizing between the harder more nebulous tasks. And across all our roles, being comfortable with upward feedback and disagreement is key.

Ajeya 24 Oct 2023 22:50 UTC
17 points
1 ∶ 0
in reply to: Max Nadeau’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
For me personally, research and then grantmaking at Open Phil has been excellent for my career development, and it’s pretty implausible that grad school in ML or CS, or an ML engineering role at an AI company, or any other path I can easily think of, would have been comparably useful.
If I had pursued an academic path, then assuming I was successful on that path, I would be in my first or maybe second year as an assistant professor right about now (or maybe I’d just be starting to apply for such a role). Instead, at Open Phil, I wrote less-academic reports and posts about less established topics in a more home-grown style, gave talks in a variety of venues, talked to podcasters and journalists, and built lots of relationships in industry, academia, and the policy world in the course of funding and advising people. I am likely more noteworthy among AI companies, policymakers, and even academic researchers than I would have been if I had spent that time doing technical research in a grad school and then went for a faculty role — and I additionally get to direct funding, an option which wouldn’t have been easily available to me on that alternative path.
The obvious con of OP relative to a path like that is that you have to “roll your own” career path to a much greater degree. If you go to grad school, you will definitely write papers, and then be evaluated based on how many good papers you’ve written; there isn’t something analogous you will definitely be made to do and evaluated on at OP (at least not something clearly publicly visible). But I think there are a lot of pros:
- The flipside of the social awkwardness and stress that Linch highlighted in one of his questions is that a grantmaking role teaches you how to navigate delicate power dynamics, say no, give tough feedback, and make non-obvious decisions that have tangible consequences on reasonably short timeframes. I think I’ve developed more social maturity and operational effectiveness than I would have in a research role; this is a pretty important and transferrable skillset.
- There is more space than there would be in a grad school or AI lab setting to think about weird questions that sit at the intersection of different fields and have no obvious academic home, such as the trajectory of AI development and timelines to very powerful AI. While independent research or other small-scale nonprofit research groups could offer a similar degree of space to think about “weird stuff,” OP is unusual in combining that kind of latitude with the ability to direct funding (and thus the ability to help make big material projects happen in the world).

Ajeya 24 Oct 2023 22:25 UTC
9 points
0 ∶ 0
in reply to: Max Nadeau’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
I’m very interested in these paths. In fact, I currently think that well over half the value created by the projects we have funded or will fund in 2023 will go through “providing evidence for dangerous capabilities” and “demonstrating emergent misalignment;” I wouldn’t be surprised if that continues being the case.

Ajeya 24 Oct 2023 22:16 UTC
2 points
1 ∶ 0
in reply to: Max Nadeau’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
The way I approach the role, it involves thinking deeply about what technical research we want to see in the world and why, and trying to articulate that to potential grantees (in one-on-one conversations, posts like this one, RFPs, talks at conferences, etc) so that they can form a fine-grained understanding of how we’re thinking about the core problems and where their research interests overlap with Open Phil’s philanthropic goals in the space. To do this well, it’s really valuable to have a good grip on the existing work in the relevant area(s).

Ajeya 24 Oct 2023 5:05 UTC
3 points
1 ∶ 0
in reply to: Max Nadeau’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
I think this is definitely a real dynamic, but a lot of EAs seem to exaggerate it a lot in their minds and inappropriately round the impact of external research down to 0. Here are a few scattered points on this topic:
- Third party researchers can influence the research that happens at labs through the normal diffusion process by which all research influences all other research. There’s definitely some barrier to research insight diffusing from academia to companies (and e.g. it’s unfortunately common for an academic project to have no impact on company practice because it just wasn’t developed with the right practical constraints in mind), but it still happens all the time (and some types of research, e.g. benchmarks, are especially easy to port over). If third party research can influence lab practice to a substantial degree, then funding third party research just straightforwardly increases the total amount of useful research happening, since labs can’t hire everyone who could do useful work.
- It will increasingly be possible to do good (non-interpretability) research on large models through APIs provided by labs, and Open Phil could help facilitate that and increase the rate at which it happens. We can also help facilitate greater compute budgets and engineering support.
- The work of the lab-external safety research community can also impact policy and public opinion; the safety teams at scaling labs are not their only audience. For example, capability evaluations and model organisms work both have the potential to have at least as big an impact on policy as they do on technical safety work happening inside labs.
- We can fund nonprofits and companies which directly interface with AI companies in a consulting-like manner (e.g. red-teaming consultants); I expect an increasing fraction of our opportunities to look like this.
- Academics and other external safety researchers we fund now can end up joining scaling labs later (as e.g. Ethan Perez and Collin Burns did), to implement ideas that they developed on the outside; I think this is likely to happen more and more.
- Some research directions benefit less than others from access to cutting edge models. For example, it seems like there’s a lot of interpretability work that can be done on very small models, whereas scalable oversight work seems harder to do without quite smart models.

Ajeya 24 Oct 2023 4:54 UTC
3 points
0 ∶ 0
in reply to: Max Nadeau’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Professors typically have their own salaries covered, but need to secure funding for each new student they take on, so providing funding to an academic lab allows them to take on more students and grow (it’s not always the case that everyone is taking on as many students as they can manage). Additionally, it’s often hard for professors to get funding for non-student expenses (compute, engineering help, data labeling contractors, etc) through NSF grants and similar, which are often restricted to students.

Ajeya 24 Oct 2023 1:30 UTC
11 points
2 ∶ 0
in reply to: Linch’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Yeah, I feel a lot of this stress as well, though FWIW for me personally research was more stressful. I don’t think there’s any crisp institutional advice or formula for dealing with this kind of thing unfortunately. One disposition that I think makes it hard to be a grantmaker at OP (in addition to your list, which I think is largely overlapping) is being overly attached to perfection and satisfyingly clean, beautifully-justifiable answers and decisions.

Ajeya 24 Oct 2023 1:25 UTC
6 points
0 ∶ 0
in reply to: Jonas Hallgren 🔸’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
It’s hard to project forward of course, but currently there are ~50 applicants to the TAIS team and ~100 to the AI governance team (although I think a number of people are likely to apply close to the deadline).

Ajeya 24 Oct 2023 1:16 UTC
2 points
0 ∶ 0
in reply to: C Sakura Keen’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
There is certainly no defined age cutoff, and we are usually extra excited when we can hire candidates who bring many years of career experience to the table in addition to other qualifications!

Ajeya 24 Oct 2023 1:14 UTC
2 points
0 ∶ 0
in reply to: quinn’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
I’ll just add that in a lot of cases, I fund technical research that I think is likely to help with policy goals (for example, work in the space of model organisms of misalignment can feed into policy goals).

Ajeya 24 Oct 2023 1:11 UTC
1 point
1 ∶ 1
in reply to: Tom Barnes🔸’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
We decided to change the name to reflect the fact that we don’t think you need to take a long-termist philosophical stance to work on AI risks and biorisks; the specific new name was chosen from among a few contenders after a survey process. The name change doesn’t reflect any sharp break from how we have operated in practice for the last while, so I don’t think there are any specific strategy changes it implies.

Ajeya 24 Oct 2023 1:09 UTC
2 points
0 ∶ 0
in reply to: Vaipan’s comment on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
On technical AI safety, fundamentally having more grantmaking and research capacity (junior or senior) will help us make more grants to great projects that we wouldn’t have been able to otherwise; I wrote about that team’s hiring needs in this separate post. In terms of AI safety more broadly (outside of just my team), I’d say there is a more severe constraint on people who can mentor junior researchers, but the field could use more strong researchers at all levels of seniority.

Ajeya 21 Oct 2023 18:37 UTC
19 points
0 ∶ 0
in reply to: Joseph Miller’s comment on: New roles on my team: come build Open Phil’s technical AI safety program with me!
I only got into grantmaking less than a year ago (in November 2022), and shortly after I unburied myself from FTXFF-collapse-related grants around January, I started hiring in a private round which led to Max joining (a private round is generally much less of a logistical lift than a big public round). I’m now joining this big public round along with other OP GCR teams because combining hiring rounds makes it easier on the back-end. See Luke’s AMA answers here and here for more detail on the “Why are you hiring now rather than previously?” question, and my comment here for more color on my personal working situation over the last ten months or so.

Ajeya 19 Oct 2023 16:56 UTC
7 points
0 ∶ 0
on: Open Philanthropy is hiring for multiple roles across our Global Catastrophic Risks teams
A quick note that I’ve written a post announcing new roles on my team (Technical AI Safety), and discussing the team’s situation as a whole, here!

Ajeya 12 Oct 2023 18:33 UTC
4 points
0 ∶ 0
in reply to: EvanMcVail’s comment on: There should be more AI safety orgs
And a quick note that we’ve also added an executive assistant / operations role since Evan wrote this comment!

Ajeya 26 Sep 2023 2:54 UTC
59 points
0 ∶ 0
on: There should be more AI safety orgs
(Cross-posted to LessWrong.)
I’m a Senior Program Officer at Open Phil, focused on technical AI safety funding. I’m hearing a lot of discussion suggesting funding is very tight right now for AI safety, so I wanted to give my take on the situation.
At a high level: AI safety is a top priority for Open Phil, and we are aiming to grow how much we spend in that area. There are many potential projects we’d be excited to fund, including some potential new AI safety orgs as well as renewals to existing grantees, academic research projects, upskilling grants, and more.
At the same time, it is also not the case that someone who reads this post and tries to start an AI safety org would necessarily have an easy time raising funding from us. This is because:
- All of our teams whose work touches on AI (Luke Muehlhauser’s team on AI governance, Claire Zabel’s team on capacity building, and me on technical AI safety) are quite understaffed at the moment. We’ve hired several people recently, but across the board we still don’t have the capacity to evaluate all the plausible AI-related grants, and hiring remains a top priority for us.
  - And we are extra-understaffed for evaluating technical AI safety proposals in particular. I am the only person who is primarily focused on funding technical research projects (sometimes Claire’s team funds AI safety related grants, primarily upskilling, but a large technical AI safety grant like a new research org would fall to me). I currently have no team members; I expect to have one person joining in October and am aiming to launch a wider hiring round soon, but I think it’ll take me several months to build my team’s capacity up substantially.
  - I began making grants in November 2022, and spent the first few months full-time evaluating applicants affected by FTX (largely academic PIs as opposed to independent organizations started by members of the EA community). Since then, a large chunk of my time has gone into maintaining and renewing existing grant commitments and evaluating grant opportunities referred to us by existing advisors. I am aiming to reserve remaining bandwidth for thinking through strategic priorities, articulating what research directions seem highest-priority and encouraging researchers to work on them (through conversations and hopefully soon through more public communication), and hiring for my team or otherwise helping Open Phil build evaluation capacity in AI safety (including separately from my team).
  - As a result, I have deliberately held off on launching open calls for grant applications similar to the ones run by Claire’s team (e.g. this one); before onboarding more people (and developing or strengthening internal processes), I would not have the bandwidth to keep up with the applications.
- On top of this, in our experience, providing seed funding to new organizations (particularly organizations started by younger and less experienced founders) often leads to complications that aren’t present in funding academic research or career transition grants. We prefer to think carefully about seeding new organizations, and have a different and higher bar for funding someone to start an org than for funding that same person for other purposes (e.g. career development and transition funding, or PhD and postdoc funding).
  - I’m very uncertain about how to think about seeding new research organizations and many related program strategy questions. I could certainly imagine developing a different picture upon further reflection — but having low capacity combines poorly with the fact that this is a complex type of grant we are uncertain about on a lot of dimensions. We haven’t had the senior staff bandwidth to develop a clear stance on the strategic or process level about this genre of grant, and that means that we are more hesitant to take on such grant investigations — and if / when we do, it takes up more scarce capacity to think through the considerations in a bespoke way rather than having a clear policy to fall back on.
What links here?

Ajeya 5 Apr 2023 22:27 UTC
54 points
3 ∶ 1
in reply to: richard_ngo’s comment on: Critiques of prominent AI safety labs: Redwood Research
I work at Open Philanthropy, and in the last few months I took on much of our technical AI safety grantmaking.
In November and December, Jacob sent me a list of academics he felt that someone at Open Phil should reach out to and solicit proposals from. I was interested in these opportunities, but at the time, I was full-time on processing grant proposals that came in through Open Philanthropy’s form for grantees affected by the FTX crash and wasn’t able to take them on.
This work tailed off in January, and since then I’ve focused on a few bigger grants, some writing projects, and thinking through how I should approach further grantmaking. I think I should have reached out to at least a few of the people Jacob suggested earlier (e.g. in February). I didn’t make any explicit decision to reject someone that Jacob thought was a slam dunk because I disagreed with his assessment — rather, I was slower to reach out to talk to people he thought I should fund than I could have been.
I plan to talk to several of the leads Jacob sent my way in Q2, and (while I would plan to think through the case for these grants myself to the extent I can) I expect to end up agreeing a lot with Jacob’s assessments.
With that said, Jacob and I do have more nebulous higher-level disagreements about things like how truth-tracking academic culture tends to be and how much academic research has contributed to AI alignment so far, and in some indirect way these disagreements probably contributed to me prioritizing these reach outs less highly than someone else might have.

Ajeya 29 Mar 2023 0:13 UTC
4 points
0 ∶ 0
in reply to: OscarD🔸’s comment on: New blog: Planned Obsolescence
Here’s the link to the various podcasting services it’s on! https://www.buzzsprout.com/2160905/share