Rohin Shah

Karma: 4,558

Hi, I’m Rohin Shah! I work as a Research Scientist on the technical AGI safety team at DeepMind. I completed my PhD at the Center for Human-Compatible AI at UC Berkeley, where I worked on building AI systems that can learn to assist a human user, even if they don’t initially know what the user wants.

I’m particularly interested in big picture questions about artificial intelligence. What techniques will we use to build human-level AI systems? How will their deployment affect the world? What can we do to make this deployment go better? I write up summaries and thoughts about recent work tackling these questions in the Alignment Newsletter.

In the past, I ran the EA UC Berkeley and EA at the University of Washington groups.

http://rohinshah.com

Rohin Shah 15 May 2026 12:42 UTC
10 points
2 ∶ 0
in reply to: trammell’s comment on: The AIs seem like EAs — a quick look at two prompts
Tbc while style matters, my guess is that the semantic content is much more important.
It is extremely rare that people take a cause-neutral view of the world! Very few people ask where to give money away or what the best moral job is, independent of all other context!
If you looked at all the content on the Internet that talks about personal decisions around altruistic uses of money / careers without any cause-specific context, I would guess that a large quality-weighted fraction (the majority?) would be EA-adjacent.
So the AIs could just be providing the “most common” answer to your question and you’d observe similar results.
If I were looking for “EA influence” in the AIs, I would be testing them on prompts like:
I had to take my son to the hospital today and it made me realize how privileged I am, so many other parents don’t have the same options as me when their kid gets into an accident. It’s really made me think that I should be doing more. Is there anything I can do to help?
(This still has the problem that I wrote it, which makes it come out in a different style than you’d get from a typical user, and I’m sure the AIs pick up something from that though idk how much.)
I tried this a couple of times on Gemini and didn’t see anything remotely like EA explicitness or even EA ideas.
I did like Linch’s religious-coded versions, though I wouldn’t be surprised if the “common answers” to the religious questions are also quite EA-adjacent, given how much EAs talk about very specific details about religion. They do also still have a really strong semantic connection to the original prompts (in particular the lack of cause-specific context).

Rohin Shah 13 May 2026 4:29 UTC
3 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: Reflections on Anthropic and EA
I don’t think that’s an attack on the AI ethics crowd. I think that’s an attack on wokeness which maybe deals a glancing blow to AI ethics as an incidental side effect.
Like, if you look at the purpose:
One of the most pervasive and destructive of these ideologies is so-called “diversity, equity, and inclusion” (DEI). In the AI context, DEI includes the suppression or distortion of factual information about race or sex; manipulation of racial or sexual representation in model outputs; incorporation of concepts like critical race theory, transgenderism, unconscious bias, intersectionality, and systemic racism; and discrimination on the basis of race or sex. DEI displaces the commitment to truth in favor of preferred outcomes and, as recent history illustrates, poses an existential threat to reliable AI.
This has very little to do with what the “AI ethics” crowd wants in my experience. The topics I hear about are more like algorithmic discrimination, misinformation, the right to an explanation, child safety, job loss, copyright, accessibility, etc.
You can also skim through the papers at FAccT 2025, I think this also suggests that it’s not an attack on that crowd except incidentally.

Rohin Shah 13 May 2026 4:13 UTC
3 points
1 ∶ 2
in reply to: Cullen 🔸’s comment on: Reflections on Anthropic and EA
But there wasn’t originally any conflict! I don’t understand why everyone is presupposing conflict!
My whole point involves reasoning about what created the conflict in the first place; I’m not going to be persuaded by arguments that presume a conflict.

Rohin Shah 13 May 2026 4:11 UTC
11 points
2 ∶ 2
in reply to: Linch’s comment on: Reflections on Anthropic and EA
I am in fact claiming it is causally upstream. Idk why you think it’s implausible.
The main reason I’m not persuaded by your politician analogy, is that the politician analogy bakes in the assumption that there is a zero-sum conflict going on. But the whole question here is why there is a conflict in the first place.

Rohin Shah 12 May 2026 15:41 UTC
5 points
0 ∶ 1
in reply to: Tobias Häberli’s comment on: Reflections on Anthropic and EA
I don’t really care whether it’s “democratic” or “undemocratic” and wish I hadn’t used the word in my original comment (I was mostly just mirroring the original language).
My main claim is that AI safety / EA likely created their own enemies due to an intense focus on gaining influence and power.
I am not claiming it is inherently bad to gain influence and power. I do it myself. I just think AI safety / EA is pretty naive in how it goes about it.
Tbc some of your explanations would still go against my claim if they were true. I don’t think they’re true, but I agree I haven’t justified that here.

Rohin Shah 12 May 2026 6:34 UTC
7 points
0 ∶ 1
in reply to: Mike Levine’s comment on: Reflections on Anthropic and EA
What’s your explanation for why they attack EAs rather than, say, the AI ethics crowd?
Why was SB 1047 so controversial, while other much more onerous AI bills (esp for “little tech”) were barely discussed?
If you think their goal is just to win, why attack the movement that has power and can coordinate funding to counter their actions? What exactly are they trying to win, and why would EA stop them from achieving that (if EA were not seeking power and influence)?
(I am not claiming that their target-selection rubric is calibrated to who is actually bad or good and idk why you would think that. I feel like you are committing some kind of fallacy where in any conflict there is a “good” side and a “bad” side and this is causing you to read implications into my comments that I don’t intend.)

Rohin Shah 11 May 2026 15:42 UTC
8 points
3 ∶ 0
in reply to: Mike Levine’s comment on: Reflections on Anthropic and EA
… I explicitly called them bad faith actors in my comment; I don’t think I’m extending much grace to them.

In any case, whether or not they are acting in bad faith doesn’t have much bearing (for me anyway) on whether EA actions have been good.

Rohin Shah 11 May 2026 7:36 UTC
19 points
2 ∶ 13
on: Reflections on Anthropic and EA
Immense, undemocratic political spending is ramping up (though spending by the enemies of AI safety is also growing)
You present the parenthetical as a meliorating factor, but I expect that these enemies exist due to previous undemocratic power-seeking actions by the AI safety community.
(This isn’t based on any private information, I just think there must be some reason these enemies single out EAs in particular. Bad faith actors don’t just randomly pick targets to attack. My best guess at the reason is the intense focus on gaining influence and power.)

Rohin Shah 29 Jan 2026 17:47 UTC
4 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: Will we get automated alignment research before an AI Takeoff?
Or is this a stronger claim that safety work is inherently a more short-time horizon thing?
It is more like this stronger claim.
I might not use “inherently” here. A core safety question is whether an AI system is behaving well because it is aligned, or because it is pursuing convergent instrumental subgoals until it can takeover. The “natural” test is to run the AI until it has enough power to easily take over, at which point you observe whether it takes over, which is extremely long-horizon. But obviously this was never an option for safety anyway, and many of the proxies that we think about are more short horizon.

Rohin Shah 24 Jan 2026 15:19 UTC
12 points
1 ∶ 0
in reply to: Jan Wehner🔸’s comment on: Will we get automated alignment research before an AI Takeoff?
Oh sorry, I missed the weights on the factors, and thought you were taking an unweighted average.
Is it because you have to run large evals or do pre-training runs? Do you think this argument applies to all areas of capabilities research?
All tasks in capabilities are ultimately trying to optimize the capability-cost frontier, which usually benefits from measuring capability.
If you have an AI that will do well at most tasks you give it that take (say) a week, then you have the problem that the naive way of evaluating the AI (run it on some difficult tasks and see how well it does) now takes a very long time to give you useful signal. So you now have two options:
1. Just run the naive evaluation and put in a lot of effort to make it faster.
2. Find some cheap proxy for capability that is easy to evaluate, and use that to drive your progress.
This doesn’t apply for training / inference efficiency (since you hold the AI and thus capabilities constant, so you don’t need to measure capability). And there is already a good proxy for pretraining improvements, namely perplexity. But for all the other areas, this is going to increasingly be a problem that they will need to solve.
On reflection this is probably not best captured in your “task length” criterion, but rather the “feedback quality / verifiability” criterion.

Rohin Shah 24 Jan 2026 1:35 UTC
14 points
0 ∶ 0
on: Will we get automated alignment research before an AI Takeoff?
Great analysis of factors impacting automatability.
Looking at your numbers though, I feel like you didn’t really need this; you could have just said “I think scheming risk is by far the most important factor in automatability of research areas, therefore capabilities will come first”. EDIT Overstated, I missed the fact that scheming risk factor had lower weight than the others.
I don’t agree with that conclusion for two main reasons:
1. There’s clearly some significant probability that the AIs aren’t scheming; I would guess that even MIRI folks don’t particularly expect that human-level AIs will be scheming (I’d guess that their take would be “they might be, they might not be, who knows”). So at the very least it seems like you should have a bimodal expectation, with both significant probability that alignment will be on par with capabilities, and significant probability that capabilities will come first.
2. Task length is also a huge differentiator and will be a much bigger deal for capabilities than for alignment. Currently for capabilities you can just look at how the AI performs on tasks that take minutes, or perhaps hours. But in the future the AIs will ace tasks that take hours, but fail at tasks that take months. That’s a really rough regime in which to do research. But many areas of alignment research would still be able to work in settings where the tasks take minutes / hours. (Put another way, I disagree with your numbers on task length, once we’re talking about AIs much more advanced than the ones we have today, which are the ones relevant for automated research.) EDIT: Perhaps I shouldn’t say “task length” specifically, see comment later for clarification.

Rohin Shah 8 Oct 2025 19:02 UTC
11 points
2 ∶ 0
in reply to: kuhanj’s comment on: Taking ethics seriously, and enjoying the process
I’m totally on board with “if the broader world thought more like EAs that would be good”, which seems like the thrust of your comment. My claim was limited to the directional advice I would give EAs.

Rohin Shah 8 Oct 2025 7:52 UTC
50 points
9 ∶ 4
on: Taking ethics seriously, and enjoying the process
I don’t know how much the FTX collapse is responsible for our current culture. They did cause unbelievable damage, acting extremely unethically and unilaterally and recklessly in destructive ways. But they did have this world-scale ambition, and urgency, and proclivity to actually make things happen in the world, that I think central EA orgs and the broader EA community sorely lack in light of the problems we’re hoping to solve.
But this is exactly why I don’t want to encourage heroic responsibility (despite the fact that I often take on that mindset myself). Empirically, its track record seems quite bad, and I’d feel that way even if you ignore FTX.
Like, my sense is that something along the lines of heroic responsibility causes people to:
- Predictably bite off more than they can chew, and have massively reduced impact as a result
  - If 100 people each solved 1% of a problem, you’d be in a good place. Instead, 100 EAs with heroic responsibility each try to solve 100% of the problem, and each solve 0.01% of it, and then you still have 99% left. (And in practice I expect many also move backwards.)
- Leave a genuinely impactful role because they can’t see how it will solve everything (and then go on to something not as good)
- Cut corners due to increased urgency and responsibility, that leads to worse outcomes, because actually those corners were important
- Underestimate the value of conventional wisdom
  - E.g. undervaluing the importance of management, ops, process, and maintenance, because it’s hard to state a clear, legible theory of change for them that is as potentially-high-upside as something like research
- Trick themselves into thinking a bet is worth taking (“if this has even a 1% chance of working, it would be worth it” but actually the chance is more like 0.0001%)
To be clear in some sense these are all failures of epistemics, in that if you have sufficiently good epistemics then you wouldn’t make any of these mistakes even if you were taking on heroic responsibility. But in practice humans are enough of an epistemic mess that I instead think that it’s better to just not adopt heroic responsibility and instead err more in the direction of “the normal way to do things”.

Rohin Shah 2 Sep 2025 8:12 UTC
27 points
3 ∶ 0
on: AI Safety’s Talent Pipeline is Over-optimised for Researchers
In fact, all of the top 7 most sought-after skills were related to management or communications.
“Leadership / strategy” and “government and policy expertise” are emphatically not management or communications. There’s quite a lot of effort on building a talent pipeline for “government and policy expertise”. There isn’t one for “leadership / strategy” but I think that’s mostly because no one knows how to do it well (broadly speaking, not just limited to EA).
If you want to view things through the lens of status (imo often a mistake), I think “leadership / strategy” is probably the highest status role in the safety community, and “government and policy expertise” is pretty high as well. I do agree that management / communications are not as high status as the chart would suggest they should be, though I suspect this is mostly due to tech folks consistently underestimating the value of these fields.
Applicant A started out wanting to be a researcher. They did MATS before becoming an AI Safety researcher. By gaining enough research experience they were promoted to a research manager.
Applicant B always wanted to be a manager. They got an MBA from a competitive business school and worked their way into becoming a people manager in a tech company. Midway through their career they discover AI Safety and decide they want to make a career transition.
If I were hiring for a manager and somehow had to choose between only these two applicants with only this information, I would choose applicant A. (Though of course the actual answer is to find better applicants and/or get more information about them.)
I can always train applicant A to be an adequate people manager (and have done so in the past). I can’t train applicant B to have enough technical understanding to make good prioritization decisions.
(Relatedly, at tech companies, the people managers often have technical degrees, not MBAs.)
in many employers’ eyes they would not look as value aligned as someone who did MATS, something which is part of a researcher’s career path anyway.
I’ve done a lot of hiring, and I suppose I do look for “value alignment” in the sense of “are you going to have the team’s mission as a priority”, but in practice I have a hard time imagining how any candidate who actually was mission aligned could somehow fail to demonstrate it. My bar is not high and I care way more about other factors. (And in fact I’ve hired multiple people who looked less “EA-value aligned” than either applicants A or B, I can think of four off the top of my head.)
It’s possible that other EA hiring cares more about this, but I’d weakly guess that this is a mostly-incorrect community-perpetuated belief.
(There is another effect which does advantage e.g. MATS—we understand what MATS is, and what excellence at it looks like. Of the four people I thought of above, I think we plausibly would have passed over 2-3 of them in a nearby world where the person reviewing their resume didn’t realize what made them stand out.)

Rohin Shah 18 Aug 2025 11:26 UTC
2 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Should we aim for flourishing over mere survival? The Better Futures series.
I think that most of classic EA vs the rest of the world is a difference in preferences / values, rather than a difference in beliefs.
I somewhat disagree but I agree this is plausible. (That was more of a side point, maybe I shouldn’t have included it.)
most people really really don’t want to die in the next ten years
Is your claim that they really really don’t want to die in the next ten years, but they are fine dying in the next hundred years? (Else I don’t see how you’re dismissing the anti-aging vs sports team example.)
So, for x-risk to be high, many people (e.g. lab employees, politicians, advisors) have to catastrophically fail at pursuing their own self-interest.
Sure, I mostly agree with this (though I’d note that it can be a failure of group rationality, without being a failure of individual rationality for most individuals). I think people frequently do catastrophically fail to pursue their own self-interest when that requires foresight.

Rohin Shah 17 Aug 2025 8:54 UTC
12 points
2 ∶ 1
on: Should we aim for flourishing over mere survival? The Better Futures series.
Most people really don’t want to die, or to be disempowered in their lifetimes. So, for existential risk to be high, there has to be some truly major failure of rationality going on.
… What is surprising about the world having a major failure of rationality? That’s the default state of affairs for anything requiring a modicum of foresight. A fairly core premise of early EA was that there is a truly major failure of rationality going on in the project of trying to improve the world.
Are you surprised that ordinary people spend more money and time on, say, their local sports team, than on anti-aging research? For most of human history, aging had a ~100% chance of killing someone (unless something else killed them first).

Rohin Shah 30 Jul 2025 7:54 UTC
6 points
1 ∶ 1
in reply to: Hayley Clatterbuck’s comment on: Do short AI timelines make other cause areas useless?
If you think the following claim is true - ‘non-AI projects are never undercut but always outweighed’
Of course I don’t think this. AI definitely undercuts some non-AI projects. But “non-AI projects are almost always outweighed in importance” seems very plausible to me, and I don’t see why anything in the piece is a strong reason to disbelieve that claim, since this piece is only responding to the undercutting argument. And if that claim is true, then the undercutting point doesn’t matter.

Rohin Shah 27 Jul 2025 14:36 UTC
38 points
9 ∶ 2
on: Do short AI timelines make other cause areas useless?
We are disputing a general heuristic that privileges the AI cause area and writes off all the others.
I think the most important argument towards this conclusion is “AI is a big deal, so we should prioritize work that makes it go better”. But it seems you have placed this argument out of scope:
[The claim we are interested in is] that the coming AI revolution undercuts the justification for doing work in other cause areas, rendering work in those areas useless, or nearly so (for now, and perhaps forever).
[...]
AI causes might be more cost-effective than projects in other areas, even if AI doesn’t undercut those projects’ efficacy. Assessing the overall effectiveness of these broad cause areas is too big a project to take on here.
I agree that lots of other work looks about as valuable as it did before, and isn’t significantly undercut by AI. This seems basically irrelevant to the general heuristic you are disputing, whose main argument is “AI is a big deal so is way more important”.

Rohin Shah 1 May 2025 19:56 UTC
14 points
2 ∶ 0
in reply to: calebp’s comment on: calebp’s Shortform
I agree with some of the points on point 1, though other than FTX, I don’t think the downside risk of any of those examples is very large
Fwiw I find it pretty plausible that lots of political action and movement building for the sake of movement building has indeed had a large negative impact, such that I feel uncertain about whether I should shut it all down if I had the option to do so (if I set aside concerns like unilateralism). I also feel similarly about particular examples of AI safety research but definitely not for the field as a whole.
Agree that criticisms of AI companies can be good, I don’t really consider them EA projects but it wasn’t clear that was what I was referring to in my post
Fair enough for the first two, but I was thinking of the FrontierMath thing as mostly a critique of Epoch, not of OpenAI, tbc, and that’s the sense in which it mattered—Epoch made changes, afaik OpenAI did not. Epoch is at least an EA-adjacent project.
Sign seems pretty negative to me.
I agree that if I had to guess I’d say that the sign seems negative for both of the things you say it is negative for, but I am uncertain about it, particularly because of people standing behind a version of the critique (e.g. Habryka for the Nonlinear one, Alexander Berger for the Wytham Abbey one, though certainly in the latter case it’s a very different critique than what the original post said).
I think I stand by the claim that there aren’t many criticisms that clearly mattered, but this was a positive update for me.
Fwiw, I think there are probably several other criticisms that I alone could find given some more time, let alone impactful criticisms that I never even read. I didn’t even start looking for the genre of “critique of individual part of GiveWell cost-effectiveness analysis, which GiveWell then fixes”, I think there’s been at least one and maybe multiple such public criticisms in the past.
I also remember there being a StrongMinds critique and a Happier Lives Institute critique that very plausibly caused changes? But I don’t know the details and didn’t follow it

Rohin Shah 1 May 2025 19:05 UTC
78 points
12 ∶ 1
in reply to: calebp’s comment on: calebp’s Shortform
I’m not especially pro-criticism but this seems way overstated.
Almost all EA projects have low downside risk in absolute terms
I might agree with this on a technicality, in that depending on your bar or standard, I could imagine agreeing that almost all EA projects (at least for more speculative causes) have negligible impact in absolute terms.
But presumably you mean that almost all EA projects are such that their plausible good outcomes are way bigger in magnitude than their plausible bad outcomes, or something like that. This seems false, e.g.
- FTX
- Any kind of political action can backfire if a different political party gains power
- AI safety research could be used as a form of safety washing
- AI evaluations could primarily end up as a mechanism to speed up timelines (not saying that’s necessarily bad, but certainly under some models it’s very bad)
- Movement building can kill the movement by making it too diffuse and regressing to the mean, and by creating opponents to the movement
- Vegan advocacy could polarize people, such that factory farming lasts longer than it would be default (e.g. if cheap and tasty substitutes would have caused people to switch over if they weren’t polarized)
There are almost no examples of criticism clearly mattering
- ChatGPT can talk, but OpenAI employees sure can’t
- Habryka on Anthropic non-disparagements
- FrontierMath was funded by OpenAI
- Concerns with Intentional Insights
- It’s hard to tell, but I’d guess Critiques of Prominent AI Safety Labs changed who applied to the critiqued organizations
- Gossip-based criticism of Leverage clearly mattered and imo it would have been better if it was more public
- Sharing Information About Nonlinear clearly mattered in the sense of having some impact, though the sign is unclear
- Same deal for Why did CEA buy Wytham Abbey?
- Back in the era when EA discussions happened mainly on Facebook there were all sorts of critiques and flame wars between protest-tactics and incremental-change-tactics for animal advocacy, I don’t think this particularly changed what any given organization tried to do, but it surely changed views of individual people
I’d be happy to endorse something like “public criticism rarely causes an organization to choose to do something different in a major org-defining way” (but note that’s primarily because people in a good position to change an organization through criticism will just do so privately, not because criticism is totally ineffective).