Rohin Shah

Karma: 4,341

Hi, I’m Rohin Shah! I work as a Research Scientist on the technical AGI safety team at DeepMind. I completed my PhD at the Center for Human-Compatible AI at UC Berkeley, where I worked on building AI systems that can learn to assist a human user, even if they don’t initially know what the user wants.

I’m particularly interested in big picture questions about artificial intelligence. What techniques will we use to build human-level AI systems? How will their deployment affect the world? What can we do to make this deployment go better? I write up summaries and thoughts about recent work tackling these questions in the Alignment Newsletter.

In the past, I ran the EA UC Berkeley and EA at the University of Washington groups.

http://rohinshah.com

Rohin Shah Feb 17, 2024, 11:42 AM
12 points
0 ∶ 0
on: Who’s hiring? (Feb-May 2024)
Research Scientist and Research Engineer roles in AI Safety and Alignment at Google DeepMind.
Location: Hybrid (3 days/week in the office) in San Francisco / Mountain View / London.
Application deadline: We don’t have a final deadline yet, but will keep the roles open for at least another two weeks (i.e. until March 1, 2024), and likely longer.
For further details, see the roles linked above. You may also find my FAQ useful.

Rohin Shah Feb 2, 2024, 11:05 AM
25 points
9 ∶ 2
in reply to: Elizabeth’s comment on: EV investigation into Owen and Community Health
(Fyi, I probably won’t engage more here, due to not wanting to spend too much time on this)
Jonas’s comment is a high level assessment that is only useful insofar as you trust his judgment.
This is true, but I trust basically any random commenter a non-zero amount (unless their comment itself gives me reasons not to trust them). I agree you can get more trust if you know the person better. But even the amount of trust for “literally a random person I’ve never heard of” would be enough for the evidence to matter to me.
I’m only saying that I think large updates based on Jonas’s statement are a mistake for people who already know Owen was an EA leader in good standing for many years and had many highly placed friends.
SBF was an EA leader in good standing for many years and had many highly placed friends. It’s pretty notable to me that there weren’t many comments like Jonas’s for SBF, while there are for Owen.
In contrast, lyra’s comment contains a lot of details I can use to inform my own reasoning.
It seems so noisy to compare karma counts on two different counts. There are all sorts of things we could be failing to miss about why people voted the way they did. Maybe people are voting Jonas’s comment up more because they liked that it went more out of its way to acknowledge that the past behavior was bad and that a temporary ban is good.
It seems like a mistake to treat karma as “the community’s estimate of the evidence that the comment would provide to a new reader who knows that Owen was a leader in good standing but otherwise doesn’t know anything about what’s going on”. I agree you’ll find all sorts of ways that karma counts don’t reflect that.

Rohin Shah Jan 31, 2024, 9:50 AM
59 points
20 ∶ 3
in reply to: Elizabeth’s comment on: EV investigation into Owen and Community Health
The evidence Jonas provides is equally consistent with “Owen has a flaw he has healed” and “Owen is a skilled manipulator who charms men, and harasses women”.
Surely there are a lot of other hypotheses as well, and Jonas’s evidence is relevant to updating on those?
More broadly, I don’t think there’s any obvious systemic error going on here. Someone who knows the person reasonably well, giving a model for what the causes of the behavior were, that makes predictions about future instances, clearly seems like evidence one should take into account.
(I do agree the comment would be more compelling with more object-level details, but I don’t think that makes it a systemic error to be happy with the comment that exists.)

Rohin Shah Sep 22, 2023, 6:23 AM
12 points
2 ∶ 0
in reply to: DanielFilan’s comment on: AI Pause Will Likely Backfire
Yeah, I don’t think it’s accurate to say that I see assistance games as mostly irrelevant to modern deep learning, and I especially don’t think that it makes sense to cite my review of Human Compatible to support that claim.
The one quote that Daniel mentions about shifting the entire way we do AI is a paraphrase of something Stuart says, and is responding to the paradigm of writing down fixed, programmatic reward functions. And in fact, we have now changed that dramatically through the use of RLHF, for which a lot of early work was done at CHAI, so I think this reflects positively on Stuart.
I’ll also note that in addition to the “Learning to Interactively Learn and Assist” paper that does CIRL with deep RL which Daniel cited above, I also wrote a paper with several CHAI colleagues that applied deep RL to solve assistance games.
My position is that you can roughly decompose the overall problem into two subproblems: (1) in theory, what should an AI system do? (2) Given a desire for what the AI system should do, how do we make it do that?
The formalization of assistance games is more about (1), saying that AI systems should behave more like assistants than like autonomous agents (basically the point of my paper linked above). These are mostly independent. Since deep learning is an answer to (2) while assistance games are an answer to (1), you can use deep learning to solve assistance games.
I’d also say that the current form factor of ChatGPT, Claude, Bard etc is very assistance-flavored, which seems like a clear success of prediction at least. On the other hand, it seems unlikely that CHAI’s work on CIRL had much causal impact on this, so in hindsight it looks less useful to have done this research.
All this being said, I view (2) as the more pressing problem for alignment, and so I spend most of my time on that, which implies not working on assistance games as much any more. So I think it’s overall reasonable to take me as mildly against work on assistance games (but not to take me as saying that it is irrelevant to modern deep learning).

Rohin Shah Jun 26, 2023, 8:46 PM
14 points
2 ∶ 0
in reply to: Angelina Li’s comment on: Downsides of Small Organizations in EA
Fyi, the list you linked doesn’t contain most of what I would consider the “small” orgs in AI, e.g. off the top of my head I’d name ARC, Redwood Research, Conjecture, Ought, FAR AI, Aligned AI, Apart, Apollo, Epoch, Center for AI Safety, Bluedot, Ashgro, AI Safety Support and Orthogonal. (Some of these aren’t even that small.) Those are the ones I’d be thinking about if I were to talk about merging orgs.
Maybe the non-AI parts of that list are more comprehensive, but my guess is that it’s just missing most of the tiny orgs that OP is talking about (e.g. OP’s own org, QURI, isn’t on the list).

(EDIT: Tbc I’m really keen on actually doing the exercise of naming concrete examples—great suggestion!)

Rohin Shah Jun 13, 2023, 10:52 PM
6 points
0 ∶ 0
in reply to: mariushobbhahn’s comment on: Critiques of prominent AI safety labs: Conjecture
:) I’m glad we got to agreement!
(Or at least significantly closer, I’m sure there are still some minor differences.)

Rohin Shah Jun 13, 2023, 3:48 PM
32 points
8 ∶ 0
in reply to: mariushobbhahn’s comment on: Critiques of prominent AI safety labs: Conjecture
On hits-based research: I certainly agree there are other factors to consider in making a funding decision. I’m just saying that you should talk about those directly instead of criticizing the OP for looking at whether their research was good or not.
(In your response to OP you talk about a positive case for the work on simulators, SVD, and sparse coding—that’s the sort of thing that I would want to see, so I’m glad to see that discussion starting.)
On VCs: Your position seems reasonable to me (though so does the OP’s position).
On recommendations: Fwiw I also make unconditional recommendations in private. I don’t think this is unusual, e.g. I think many people make unconditional recommendations not to go into academia (though I don’t).
I don’t really buy that the burden of proof should be much higher in public. Reversing the position, do you think the burden of proof should be very high for anyone to publicly recommend working at lab X? If not, what’s the difference between a recommendation to work at org X vs an anti-recommendation (i.e. recommendation not to work at org X)? I think the three main considerations I’d point to are:
1. (Pro-recommendations) It’s rare for people to do things (relative to not doing things), so we differentially want recommendations vs anti-recommendations, so that it is easier for orgs to start up and do things.
2. (Anti-recommendations) There are strong incentives to recommend working at org X (obviously org X itself will do this), but no incentives to make the opposite recommendation (and in fact usually anti-incentives). Similarly I expect that inaccuracies in the case for the not-working recommendation will be pointed out (by org X), whereas inaccuracies in the case for working will not be pointed out. So we differentially want to encourage the opposite recommendations in order to get both sides of the story by lowering our “burden of proof”.
3. (Pro-recommendations) Recommendations have a nice effect of getting people excited and positive about the work done by the community, which can make people more motivated, whereas the same is not true of anti-recommendations.
Overall I think point 2 feels most important, and so I end up thinking that the burden of proof on critiques / anti-recommendations should be lower than the burden of proof on recommendations—and the burden of proof on recommendations is approximately zero. (E.g. if someone wrote a public post recommending Conjecture without any concrete details of why—just something along the lines of “it’s a great place doing great work”—I don’t think anyone would say that they were using their power irresponsibly.)
I would actually prefer a higher burden of proof on recommendations, but given the status quo if I’m only allowed to affect the burden of proof on anti-recommendations I’d probably want it to go down to ~zero. Certainly I’d want it to be well below the level that this post meets.
What links here?
- Will Aldred's comment on Future of Humanity Institute 2005-2024: Final Report by Pablo (Apr 22, 2024, 2:22 AM; 6 points)

Rohin Shah Jun 12, 2023, 10:34 PM
139 points
34 ∶ 7
in reply to: mariushobbhahn’s comment on: Critiques of prominent AI safety labs: Conjecture
I’m not very compelled by this response.
It seems to me you have two points on the content of this critique. The first point:
I think it’s bad to criticize labs that do hits-based research approaches for their early output (I also think this applies to your critique of Redwood) because the entire point is that you don’t find a lot until you hit.
I’m pretty confused here. How exactly do you propose that funding decisions get made? If some random person says they are pursuing a hits-based approach to research, should EA funders be obligated to fund them?
Presumably you would want to say “the team will be good at hits-based research such that we can expect a future hit, for X, Y and Z reasons”. I think you should actually say those X, Y and Z reasons so that the authors of the critique can engage with them; I assume that the authors are implicitly endorsing a claim like “there aren’t any particularly strong reasons to expect Conjecture to do more impactful work in the future”.
The second point:
Your statements about the VCs seem unjustified to me. How do you know they are not aligned? [...] I haven’t talked to the VCs either, but I’ve at least asked people who work(ed) at Conjecture.
Hmm, it seems extremely reasonable to me to take as a baseline prior that the VCs are profit-motivated, and the authors explicitly say
We have heard credible complaints of this from their interactions with funders. One experienced technical AI safety researcher recalled Connor saying that he will tell investors that they are very interested in making products, whereas the predominant focus of the company is on AI safety.
The fact that people who work(ed) at Conjecture say otherwise means that (probably) someone is wrong, but I don’t see a strong reason to believe that it’s the OP who is wrong.
At the meta level you say:
I do not understand where the confidence with which you write the post (or at least how I read it) comes from.
And in your next comment:
I think we should really make sure that we say true things when we criticize people, quantify our uncertainty, differentiate between facts and feelings and do not throw our epistemics out of the window in the process.
But afaict, the only point where you actually disagree with a claim made in the OP (excluding recommendations) is in your assessment of VCs? (And in that case I feel very uncompelled by your argument.)
In what way has the OP failed to say true things? Where should they have had more uncertainty? What things did they present as facts which were actually feelings? What claim have they been confident about that they shouldn’t have been confident about?
(Perhaps you mean to say that the recommendations are overconfident. There I think I just disagree with you about the bar for evidence for making recommendations, including ones as strong as “alignment researchers shouldn’t work at organization X”. I’ve given recommendations like this to individual people who asked me for a recommendation in the past, on less evidence than collected in this post.)

Rohin Shah May 22, 2023, 7:36 AM
2 points
0 ∶ 0
in reply to: Nathan Young’s comment on: Linch’s Shortform
Wait, you think the reason we can’t do brain improvement is because we can’t change the weights of individual neurons?

That seems wrong to me. I think it’s because we don’t know how the neurons work.
Did you read the link to Cold Takes above? If so, where do you disagree with it?
(I agree that we’d be able to do even better if we knew how the neurons work.)
Similarly I’d be surprised if you thought that beings as intelligent as humans could recursively improve NNs. Cos currently we can’t do that, right?
Humans can improve NNs? That’s what AI capabilities research is?
(It’s not “recursive” improvement but I assume you don’t care about the “recursive” part here.)

Rohin Shah May 16, 2023, 8:22 PM
2 points
0 ∶ 0
in reply to: Nathan Young’s comment on: Linch’s Shortform
I think it’s within the power of beings equally as intelligent as us (similarly as mentioned above I think recursive improvement in humans would accelerate if we had similar abilities).

Rohin Shah Apr 24, 2023, 10:32 PM
6 points
1 ∶ 0
in reply to: Ben_West🔸’s comment on: A freshman year during the AI midgame: my approach to the next year
I thought yes, but I’m a bit unhappy about that assumption (I forgot it was there). If you go by the intended spirit of the assumption (see the footnote) I’m probably on board, but it seems ripe for misinterpretation (“well if you had just deployed GPT-5 it really could have run an automated company, even though in practice we didn’t do that because we were worried about safety and/or legal liability and/or we didn’t know how to prompt it etc”).

Rohin Shah Apr 21, 2023, 6:47 PM
14 points
5 ∶ 3
in reply to: Greg_Colbourn ⏸️ ’s comment on: A freshman year during the AI midgame: my approach to the next year
You could look at these older conversations. There’s also Where I agree and disagree with Eliezer (see also my comment) though I suspect that won’t be what you’re looking for.
Mostly though I think you aren’t going to get what you’re looking for because it’s a complicated question that doesn’t have a simple answer.
(I think this regardless of whether you frame the question as “do we die?” or “do we live?”, if you think the case for doom is straightforward I think you are mistaken. All the doom arguments I know of seem to me like they establish plausibility, not near-certainty, though I’m not going to defend that here.)
Would you be willing to put this in numerical form (% chance) as a rough expectation?
Idk, I don’t really want to make claims about GPT-5 / GPT-6, since that depends on OpenAI’s naming decisions. But I’m at < 5% (probably < 1%, but I’d want to think about it) on “the world will be transformed” (in the TAI sense) within the next 3 years.

Rohin Shah Apr 20, 2023, 7:50 AM
14 points
3 ∶ 2
in reply to: Greg_Colbourn ⏸️ ’s comment on: A freshman year during the AI midgame: my approach to the next year
First off, let me say that I’m not accusing you specifically of “hype”, except inasmuch as I’m saying that for any AI-risk-worrier who has ever argued for shorter timelines (a class which includes me), if you know nothing else about that person, there’s a decent chance their claims are partly “hype”. Let me also say that I don’t believe you are deliberately benefiting yourself at others’ expense.
That being said, accusations of “hype” usually mean an expectation that the claims are overstated due to bias. I don’t really see why it matters if the bias is survival motivated vs finance motivated vs status motivated. The point is that there is bias and so as an observer you should discount the claims somewhat (which is exactly how it was used in the original comment).
what do you make of Connor Leahy’s take that LLMs are basically “general cognition engines” and will scale to full AGI in a generation or two (and with the addition of various plugins etc to aid “System 2″ type thinking, which are freely being offered by the AutoGPT crowd)?
Could happen, probably won’t, though it depends what is meant by “a generation or two”, and what is meant by “full AGI” (I’m thinking of a bar like transformative AI).
(I haven’t listened to the podcast but have thought about this idea before. I do agree it’s good to think of LLMs as general cognition engines, and that plugins / other similar approaches will be a big deal.)

Rohin Shah Apr 20, 2023, 7:26 AM
4 points
0 ∶ 0
in reply to: Magnus Vinding’s comment on: Two contrasting models of “intelligence” and future growth
I don’t yet understand why you believe that hardware scaling would come to grow at much higher rates than it has in the past.
If we assume innovations decline, then it is primarily because future AI and robots will be able to automate far more tasks than current AI and robots (and we will get them quickly, not slowly).
Imagine that currently technology A that automates area X gains capabilities at a rate of 5% per year, which ends up leading to a growth rate of 10% per year.
Imagine technology B that also aims to automate area X gains capabilities at a rate of 20% per year, but is currently behind technology A.
Generally, at the point when B exceeds A, I’d expect growth rates of X-automating technologies to grow from 10% to >20% (though not necessarily immediately, it can take time to build the capacity for that growth).
For AI, the area X is “cognitive labor”, technology A is “the current suite of productivity tools”, and technology B is “AI”.
For robots, the area X is “physical labor”, technology A is “classical robotics”, and technology B is “robotics based on foundation models”.
That was just assuming hardware scaling, and it justifies a growth in some particular growth rates, but not a growth explosion. If you add in the software efficiency, then I think you are just straightforwardly generating lots of innovations (what else is leading to the improved software efficiency?) and that’s how you get the growth explosion, at least until you run out of software efficiency improvements to make.

Rohin Shah Apr 19, 2023, 2:56 PM
6 points
0 ∶ 0
in reply to: Magnus Vinding’s comment on: Two contrasting models of “intelligence” and future growth
I don’t disagree with any of the above (which is why I emphasized that I don’t think the scaling argument is sufficient to justify a growth explosion). I’m confused why you think the rate of growth of robots is at all relevant, when (general-purpose) robotics seem mostly like a research technology right now. It feels kind of like looking at the current rate of growth of fusion plants as a prediction of the rate of growth of fusion plants after the point where fusion is cheaper than other sources of energy.
(If you were talking about the rate of growth of machines in general I’d find that more relevant.)

Rohin Shah Apr 19, 2023, 8:23 AM
9 points
0 ∶ 0
in reply to: Magnus Vinding’s comment on: Two contrasting models of “intelligence” and future growth
I am confused by your argument against scaling.
My understanding of the scale-up argument is:
1. Currently humans are state-of-the-art at various tasks relevant to growth.
2. We are bottlenecked on scaling up humans by a variety of things (e.g. it takes ~20 years to train up a new human, you can’t invest money into the creation of new humans with the hope of getting a return on it, humans only work ~8 hours a day)
3. At some point AI / robots will be able to match human performance at these tasks.
4. AI / robots will not be bottlenecked on those things.
In some sense I agree with you that you have to see efficiency improvements, but the efficiency improvements are things like “you can create new skilled robots in days, compared to the previous SOTA of 20 years”. So I think if you accept (3) then I think you are already accepting massive efficiency improvements.
I don’t see why current robot growth rates are relevant. When you have two different technologies A and B where A works better now, but B is getting better faster than A, then there will predictably be a big jump in the use of B once it exceeds A, and extrapolating the growth rates of B before it exceeds A is going to predictably mislead you.
(For example, I’d guess that in 1975, you would have done better thinking about how / when the personal computer would overtake other office productivity technologies, perhaps based on Moore’s law, rather than trying to extrapolate the growth rate of personal computers. Indeed, according a random website I just found, it looks like the growth rate accelerated till the EDIT: 1980s, though it’s hard to tell from the graph.)
(To be clear, this argument doesn’t necessarily get you to “transformative impact on growth comparable to the industrial revolution”, I’d guess you do need to talk about innovations to get that conclusion. But I’m just not seeing why you don’t expect a ton of scaling even if innovations are rarer, unless you deny (3), but it mostly seems like you don’t deny (3).)

Rohin Shah Apr 19, 2023, 6:58 AM
25 points
10 ∶ 7
in reply to: Steven Byrnes’s comment on: A freshman year during the AI midgame: my approach to the next year
“Hype” typically means Person X is promoting a product, that they benefit from the success of that product, and that they are probably exaggerating the impressiveness of that product in bad faith (or at least, with a self-serving bias).
All of this seems to apply to AI-risk-worriers?
- AI-risk-worriers are promoting a narrative that powerful AI will come soon
- AI-risk-worriers are taken more seriously, have more job opportunities, get more status, get more of their policy proposals, etc, to the extent that this narrative is successful
- My experience is that AI products are less impressive than the impression I would get from listening to AI-risk-worriers, and self-serving bias seems like an obvious explanation for this.
I generally agree that as a discourse norm you don’t want to go around accusing people of bad faith, but as a matter of truth-seeking my best guess is that a substantial fraction of short-timelines amongst AI-risk-worriers is in fact “hype”, as you’ve defined it.

Rohin Shah Apr 15, 2023, 6:45 PM
10 points
0 ∶ 0
in reply to: Magnus Vinding’s comment on: Two contrasting models of “intelligence” and future growth
Thanks for this, it’s helpful. I do agree that declining growth rates is significant evidence for your view.
I disagree with your other arguments:
For one, an AI-driven explosion of this kind would most likely involve a corresponding explosion in hardware (e.g. for reasons gestured at here and here), and there are both theoretical and empirical reasons to doubt that we will see such an explosion.
I don’t have a strong take on whether we’ll see an explosion in hardware efficiency; it’s plausible to me that there won’t be much change there (and also plausible that there will be significant advances, e.g. getting 3D chips to work—I just don’t know much about this area).
But the central thing I imagine in an AI-driven explosion is an explosion in the amount of hardware (i.e. way more factories producing chips, way more mining operations getting the raw materials, etc), and an explosion in software efficiency (see e.g. here and here). So it just doesn’t seem to matter that much if we’re at the limits of hardware efficiency.
it is worth noting that the decline in growth rates that we have seen since the 1960s is not only due to decreasing population growth, as there are also other factors that have contributed, such as certain growth potentials that have been exhausted, and the consequent decline in innovations per capita.
I realize that I said the opposite in a previous comment thread, but right now it feels consistent with explosive growth to say that innovations per capita are going to decline; indeed I agree with Scott Alexander that it’s hard to imagine it being any other way. The feedback loop for explosive growth is output → people / AI → ideas → output, a core part of that feedback loop is about increasing the “population”.
(Though the fact that I said the opposite in a previous comment thread suggests I should really delve into the math to check my understanding.)
(The rest of this is nitpicky details.)
Incidentally, the graphs you show for the decline in innovations per capita start dropping around 1900 (I think, I am guessing for the one that has “% of people” as its x-axis), which is pretty different from the 1960s.
Also, I’m a bit skeptical of the graph showing a 5x drop. It’s based off of an analysis of a book written by Asimov in 1993 presenting a history of science and discovery. I’m pretty worried that this will tend to disadvantage the latest years, because (1) there may have been discoveries that weren’t recognized as “meeting the bar”, since their importance hadn’t yet been understood, (2) Asimov might not have been aware of the most recent discoveries (though this point could also go the other way), and (3) as time goes on, discoveries become more and more diverse (there are way more fields) and hard to understand without expertise, and so Asimov might not have noticed them.
What links here?
- Magnus Vinding's comment on Two contrasting models of “intelligence” and future growth by Magnus Vinding (Apr 19, 2023, 4:56 PM; 5 points)

Rohin Shah Feb 23, 2023, 3:29 PM
3 points
1 ∶ 0
in reply to: EJT’s comment on: There are no coherence theorems
I think it does [change the conclusion].
Upon rereading I realize I didn’t state this explicitly, but my conclusion was the following:
If an agent has complete preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
Transitivity depending on completeness doesn’t invalidate that conclusion.

Rohin Shah Feb 22, 2023, 3:34 PM
5 points
3 ∶ 1
in reply to: EJT’s comment on: There are no coherence theorems
Okay, it seems like we agree on the object-level facts, and what’s left is a disagreement about whether people have been making a major error. I’m less interested in that disagreement so probably won’t get into a detailed discussion, but I’ll briefly outline my position here.
The error is claiming that
- There exist theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
I haven’t seen anyone point out that that claim is false.
The main way in which this claim is false (on your way of using words) is that it fails to note some of the antecedents in the theorem (completeness, maybe transitivity).
But I don’t think this is a reasonable way to use words, and I don’t think it’s reasonable to read the quotes in your appendix as claiming what you say they claim.
Converting math into English is a tricky business. Often a lot of the important “assumptions” in a theorem are baked into things like the type signature of a particular variable or the definitions of some key terms; in my toy theorem above I give two examples (completeness and lack of time-dependence). You are going to lose some information about what the theorem says when you convert it from math to English; an author’s job is to communicate the “important” parts of the theorem (e.g. the conclusion, any antecedents that the reader may not agree with, implications of the type signature that limit the applicability of the conclusion), which will depend on the audience.
As a result when you read an English description of a theorem, you should not expect it to state every antecedent. So it seems unreasonable to me to critique a claim in English about a theorem existing purely because it didn’t list all the antecedents.
I think it is reasonable to critique a claim in English about a theorem on the basis that it didn’t highlight an important antecedent that limits its applicability. If you said “AI alignment researchers should make sure to highlight the Completeness axiom when discussing coherence theorems” I’d be much more sympathetic (though personally my advice would be “AI alignment researchers should make sure to either argue for or highlight as an assumption the point that the AI is goal-directed / has preferences”).
Gustafsson thinks you need Completeness to get a money-pump for this version of transitivity working
Yup, good point, I think it doesn’t change the conclusion.