Rohin Shah

Karma: 4,475

Hi, I’m Rohin Shah! I work as a Research Scientist on the technical AGI safety team at DeepMind. I completed my PhD at the Center for Human-Compatible AI at UC Berkeley, where I worked on building AI systems that can learn to assist a human user, even if they don’t initially know what the user wants.

I’m particularly interested in big picture questions about artificial intelligence. What techniques will we use to build human-level AI systems? How will their deployment affect the world? What can we do to make this deployment go better? I write up summaries and thoughts about recent work tackling these questions in the Alignment Newsletter.

In the past, I ran the EA UC Berkeley and EA at the University of Washington groups.

http://rohinshah.com

Rohin Shah 8 Oct 2025 19:02 UTC
11 points
2 ∶ 0
in reply to: kuhanj’s comment on: Taking ethics seriously, and enjoying the process
I’m totally on board with “if the broader world thought more like EAs that would be good”, which seems like the thrust of your comment. My claim was limited to the directional advice I would give EAs.

Rohin Shah 8 Oct 2025 7:52 UTC
49 points
9 ∶ 4
on: Taking ethics seriously, and enjoying the process
I don’t know how much the FTX collapse is responsible for our current culture. They did cause unbelievable damage, acting extremely unethically and unilaterally and recklessly in destructive ways. But they did have this world-scale ambition, and urgency, and proclivity to actually make things happen in the world, that I think central EA orgs and the broader EA community sorely lack in light of the problems we’re hoping to solve.
But this is exactly why I don’t want to encourage heroic responsibility (despite the fact that I often take on that mindset myself). Empirically, its track record seems quite bad, and I’d feel that way even if you ignore FTX.
Like, my sense is that something along the lines of heroic responsibility causes people to:
- Predictably bite off more than they can chew, and have massively reduced impact as a result
  - If 100 people each solved 1% of a problem, you’d be in a good place. Instead, 100 EAs with heroic responsibility each try to solve 100% of the problem, and each solve 0.01% of it, and then you still have 99% left. (And in practice I expect many also move backwards.)
- Leave a genuinely impactful role because they can’t see how it will solve everything (and then go on to something not as good)
- Cut corners due to increased urgency and responsibility, that leads to worse outcomes, because actually those corners were important
- Underestimate the value of conventional wisdom
  - E.g. undervaluing the importance of management, ops, process, and maintenance, because it’s hard to state a clear, legible theory of change for them that is as potentially-high-upside as something like research
- Trick themselves into thinking a bet is worth taking (“if this has even a 1% chance of working, it would be worth it” but actually the chance is more like 0.0001%)
To be clear in some sense these are all failures of epistemics, in that if you have sufficiently good epistemics then you wouldn’t make any of these mistakes even if you were taking on heroic responsibility. But in practice humans are enough of an epistemic mess that I instead think that it’s better to just not adopt heroic responsibility and instead err more in the direction of “the normal way to do things”.

Rohin Shah 2 Sep 2025 8:12 UTC
27 points
3 ∶ 0
on: AI Safety’s Talent Pipeline is Over-optimised for Researchers
In fact, all of the top 7 most sought-after skills were related to management or communications.
“Leadership / strategy” and “government and policy expertise” are emphatically not management or communications. There’s quite a lot of effort on building a talent pipeline for “government and policy expertise”. There isn’t one for “leadership / strategy” but I think that’s mostly because no one knows how to do it well (broadly speaking, not just limited to EA).
If you want to view things through the lens of status (imo often a mistake), I think “leadership / strategy” is probably the highest status role in the safety community, and “government and policy expertise” is pretty high as well. I do agree that management / communications are not as high status as the chart would suggest they should be, though I suspect this is mostly due to tech folks consistently underestimating the value of these fields.
Applicant A started out wanting to be a researcher. They did MATS before becoming an AI Safety researcher. By gaining enough research experience they were promoted to a research manager.
Applicant B always wanted to be a manager. They got an MBA from a competitive business school and worked their way into becoming a people manager in a tech company. Midway through their career they discover AI Safety and decide they want to make a career transition.
If I were hiring for a manager and somehow had to choose between only these two applicants with only this information, I would choose applicant A. (Though of course the actual answer is to find better applicants and/or get more information about them.)
I can always train applicant A to be an adequate people manager (and have done so in the past). I can’t train applicant B to have enough technical understanding to make good prioritization decisions.
(Relatedly, at tech companies, the people managers often have technical degrees, not MBAs.)
in many employers’ eyes they would not look as value aligned as someone who did MATS, something which is part of a researcher’s career path anyway.
I’ve done a lot of hiring, and I suppose I do look for “value alignment” in the sense of “are you going to have the team’s mission as a priority”, but in practice I have a hard time imagining how any candidate who actually was mission aligned could somehow fail to demonstrate it. My bar is not high and I care way more about other factors. (And in fact I’ve hired multiple people who looked less “EA-value aligned” than either applicants A or B, I can think of four off the top of my head.)
It’s possible that other EA hiring cares more about this, but I’d weakly guess that this is a mostly-incorrect community-perpetuated belief.
(There is another effect which does advantage e.g. MATS—we understand what MATS is, and what excellence at it looks like. Of the four people I thought of above, I think we plausibly would have passed over 2-3 of them in a nearby world where the person reviewing their resume didn’t realize what made them stand out.)

Rohin Shah 18 Aug 2025 11:26 UTC
2 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Should we aim for flourishing over mere survival? The Better Futures series.
I think that most of classic EA vs the rest of the world is a difference in preferences / values, rather than a difference in beliefs.
I somewhat disagree but I agree this is plausible. (That was more of a side point, maybe I shouldn’t have included it.)
most people really really don’t want to die in the next ten years
Is your claim that they really really don’t want to die in the next ten years, but they are fine dying in the next hundred years? (Else I don’t see how you’re dismissing the anti-aging vs sports team example.)
So, for x-risk to be high, many people (e.g. lab employees, politicians, advisors) have to catastrophically fail at pursuing their own self-interest.
Sure, I mostly agree with this (though I’d note that it can be a failure of group rationality, without being a failure of individual rationality for most individuals). I think people frequently do catastrophically fail to pursue their own self-interest when that requires foresight.

Rohin Shah 17 Aug 2025 8:54 UTC
10 points
2 ∶ 1
on: Should we aim for flourishing over mere survival? The Better Futures series.
Most people really don’t want to die, or to be disempowered in their lifetimes. So, for existential risk to be high, there has to be some truly major failure of rationality going on.
… What is surprising about the world having a major failure of rationality? That’s the default state of affairs for anything requiring a modicum of foresight. A fairly core premise of early EA was that there is a truly major failure of rationality going on in the project of trying to improve the world.
Are you surprised that ordinary people spend more money and time on, say, their local sports team, than on anti-aging research? For most of human history, aging had a ~100% chance of killing someone (unless something else killed them first).

Rohin Shah 30 Jul 2025 7:54 UTC
6 points
1 ∶ 1
in reply to: Hayley Clatterbuck’s comment on: Do short AI timelines make other cause areas useless?
If you think the following claim is true - ‘non-AI projects are never undercut but always outweighed’
Of course I don’t think this. AI definitely undercuts some non-AI projects. But “non-AI projects are almost always outweighed in importance” seems very plausible to me, and I don’t see why anything in the piece is a strong reason to disbelieve that claim, since this piece is only responding to the undercutting argument. And if that claim is true, then the undercutting point doesn’t matter.

Rohin Shah 27 Jul 2025 14:36 UTC
38 points
9 ∶ 2
on: Do short AI timelines make other cause areas useless?
We are disputing a general heuristic that privileges the AI cause area and writes off all the others.
I think the most important argument towards this conclusion is “AI is a big deal, so we should prioritize work that makes it go better”. But it seems you have placed this argument out of scope:
[The claim we are interested in is] that the coming AI revolution undercuts the justification for doing work in other cause areas, rendering work in those areas useless, or nearly so (for now, and perhaps forever).
[...]
AI causes might be more cost-effective than projects in other areas, even if AI doesn’t undercut those projects’ efficacy. Assessing the overall effectiveness of these broad cause areas is too big a project to take on here.
I agree that lots of other work looks about as valuable as it did before, and isn’t significantly undercut by AI. This seems basically irrelevant to the general heuristic you are disputing, whose main argument is “AI is a big deal so is way more important”.

Rohin Shah 1 May 2025 19:56 UTC
14 points
2 ∶ 0
in reply to: calebp’s comment on: calebp’s Shortform
I agree with some of the points on point 1, though other than FTX, I don’t think the downside risk of any of those examples is very large
Fwiw I find it pretty plausible that lots of political action and movement building for the sake of movement building has indeed had a large negative impact, such that I feel uncertain about whether I should shut it all down if I had the option to do so (if I set aside concerns like unilateralism). I also feel similarly about particular examples of AI safety research but definitely not for the field as a whole.
Agree that criticisms of AI companies can be good, I don’t really consider them EA projects but it wasn’t clear that was what I was referring to in my post
Fair enough for the first two, but I was thinking of the FrontierMath thing as mostly a critique of Epoch, not of OpenAI, tbc, and that’s the sense in which it mattered—Epoch made changes, afaik OpenAI did not. Epoch is at least an EA-adjacent project.
Sign seems pretty negative to me.
I agree that if I had to guess I’d say that the sign seems negative for both of the things you say it is negative for, but I am uncertain about it, particularly because of people standing behind a version of the critique (e.g. Habryka for the Nonlinear one, Alexander Berger for the Wytham Abbey one, though certainly in the latter case it’s a very different critique than what the original post said).
I think I stand by the claim that there aren’t many criticisms that clearly mattered, but this was a positive update for me.
Fwiw, I think there are probably several other criticisms that I alone could find given some more time, let alone impactful criticisms that I never even read. I didn’t even start looking for the genre of “critique of individual part of GiveWell cost-effectiveness analysis, which GiveWell then fixes”, I think there’s been at least one and maybe multiple such public criticisms in the past.
I also remember there being a StrongMinds critique and a Happier Lives Institute critique that very plausibly caused changes? But I don’t know the details and didn’t follow it

Rohin Shah 1 May 2025 19:05 UTC
77 points
12 ∶ 1
in reply to: calebp’s comment on: calebp’s Shortform
I’m not especially pro-criticism but this seems way overstated.
Almost all EA projects have low downside risk in absolute terms
I might agree with this on a technicality, in that depending on your bar or standard, I could imagine agreeing that almost all EA projects (at least for more speculative causes) have negligible impact in absolute terms.
But presumably you mean that almost all EA projects are such that their plausible good outcomes are way bigger in magnitude than their plausible bad outcomes, or something like that. This seems false, e.g.
- FTX
- Any kind of political action can backfire if a different political party gains power
- AI safety research could be used as a form of safety washing
- AI evaluations could primarily end up as a mechanism to speed up timelines (not saying that’s necessarily bad, but certainly under some models it’s very bad)
- Movement building can kill the movement by making it too diffuse and regressing to the mean, and by creating opponents to the movement
- Vegan advocacy could polarize people, such that factory farming lasts longer than it would be default (e.g. if cheap and tasty substitutes would have caused people to switch over if they weren’t polarized)
There are almost no examples of criticism clearly mattering
- ChatGPT can talk, but OpenAI employees sure can’t
- Habryka on Anthropic non-disparagements
- FrontierMath was funded by OpenAI
- Concerns with Intentional Insights
- It’s hard to tell, but I’d guess Critiques of Prominent AI Safety Labs changed who applied to the critiqued organizations
- Gossip-based criticism of Leverage clearly mattered and imo it would have been better if it was more public
- Sharing Information About Nonlinear clearly mattered in the sense of having some impact, though the sign is unclear
- Same deal for Why did CEA buy Wytham Abbey?
- Back in the era when EA discussions happened mainly on Facebook there were all sorts of critiques and flame wars between protest-tactics and incremental-change-tactics for animal advocacy, I don’t think this particularly changed what any given organization tried to do, but it surely changed views of individual people
I’d be happy to endorse something like “public criticism rarely causes an organization to choose to do something different in a major org-defining way” (but note that’s primarily because people in a good position to change an organization through criticism will just do so privately, not because criticism is totally ineffective).

Rohin Shah 4 Mar 2025 10:27 UTC
18 points
1 ∶ 0
in reply to: Sarah Cheng 🔸’s comment on: Habryka’s Quick takes
Of course, it’s true that they could ignore serious criticism is they wanted to, but my sense is that people actually quite often feel unable to ignore criticism.
As someone sympathetic to many of Habryka’s positions, while also disagreeing with many of Habryka’s positions, my immediate reaction to this was “well that seems like a bad thing”, c.f.
shallow criticism often gets valorized
I’d feel differently if you had said “people feel obliged to take criticism seriously if it points at a real problem” or something like that, but I agree with you that the mechanism is more like “people are unable to ignore criticism irrespective of its quality” (the popularity of the criticism matters, but sadly that is only weakly correlated with quality).
What links here?
- Rohin Shah's comment on calebp’s Quick takes by calebp (1 May 2025 19:05 UTC; 77 points)

Rohin Shah 24 Jan 2025 11:38 UTC
2 points
0 ∶ 0
in reply to: trammell’s comment on: Notes on risk compensation
Tbc if the preferences are written in words like “expected value of the lightcone” I agree it would be relatively easy to tell which was which, mainly by identifying community shibboleths. My claim is that if you just have the input/output mapping of (safety level of AI, capabilities level of AI) --> utility, then it would be challenging. Even longtermists should be willing to accept some risk, just because AI can help with other existential risks (and of course many safety researchers—probably the majority at this point—are not longtermists).

Rohin Shah 23 Jan 2025 9:39 UTC
4 points
0 ∶ 0
on: Notes on risk compensation
What you call the “lab’s” utility function isn’t really specific to the lab; it could just as well apply to safety researchers. One might assume that the parameters would be set in such a way as to make the lab more C-seeking (e.g. it takes less C to produce 1 util for the lab than for everyone else).
But at least in the case of AI safety, I don’t think this is the case. I doubt I could easily distinguish a lab capabilities researcher (or lab leadership, or some “aggregate lab utility function”) from an external safety researcher if you just gave me their utility functions over C and S. (AI safety has significant overlap with transhumanism; relative to the rest of humanity they are way more likely to think there are huge benefits to development of safe AGI.) In practice it seems like the issue is more like epistemic disagreement.
You could still recover many of the conclusions in this post by positing that an increase to S leads to a proportional decrease in probability of non-survival, and the proportion is the same between the lab and everyone else, but the absolute numbers aren’t. I’d still feel like this was a poor model of the real situation though.

Rohin Shah 20 Oct 2024 21:51 UTC
6 points
0 ∶ 0
in reply to: Linch’s comment on: EA “Worldviews” Need Rethinking
I agree reductions in infant mortality likely have better long-run effects on capacity growth than equivalent levels of population growth while keeping infant mortality rates constant, which could mean that you still want to focus on infant mortality while not prioritizing increasing fertility.
I would just be surprised if the decision from the global capacity growth perspective ended up being “continue putting tons of resources into reducing infant mortality, but not much into increasing fertility” (which I understand to be the status quo for GHD), because:
- Probably the dominant consideration for importance is how good / bad it is to grow the population, and it is unlikely that the differential effects from reducing infant mortality vs increasing fertility end up changing the decision
- Probably it is easier / cheaper to increase fertility than to reduce infant mortality, because very little effort has been put into increasing fertility (to my knowledge)
That said, it’s been many years since I closely followed the GHD space, and I could easily be wrong about a lot of this.

Rohin Shah 6 Jul 2024 8:29 UTC
9 points
5 ∶ 3
in reply to: peterbarnett’s comment on: 80,000 hours should remove OpenAI from the Job Board (and similar EA orgs should do similarly)
?? It’s the second bullet point in the cons list, and reemphasized in the third bullet?
If you’re saying “obviously this is the key determinant of whether you should work at a leading AI company so there shouldn’t even be a pros / cons table”, then obviously 80K disagrees given they recommend some such roles (and many other people (e.g. me) also disagree so this isn’t 80K ignoring expert consensus). In that case I think you should try to convince 80K on the object level rather than applying political pressure.

Rohin Shah 6 Jul 2024 8:23 UTC
12 points
1 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: 80,000 hours should remove OpenAI from the Job Board (and similar EA orgs should do similarly)
… That paragraph doesn’t distinguish at all between OpenAI and, say, Anthropic. Surely you want to include some details specific to the OpenAI situation? (Or do your object-level views really not distinguish between them?)

Rohin Shah 9 May 2024 8:34 UTC
7 points
2 ∶ 1
on: Updates on the EA catastrophic risk landscape
There’s currently very little work going into issues that arise even if AI is aligned, including the deployment problem
The deployment problem (as described in that link) is a non-problem if you know that AI is aligned.

Rohin Shah 2 May 2024 9:02 UTC
2 points
0 ∶ 0
in reply to: Matthew_Barnett’s comment on: Analyzing the moral value of unaligned AIs
In contrast, I think the fact that these AIs will be trained on human-generated data and deliberately shaped by humans to fulfill human-like functions and to be human-compatible should be given substantial weight.
… This seems to be saying that because we are aligning AI, they will be more utilitarian. But I thought we were discussing unaligned AI?
I agree that the fact we are aligning AI should make one more optimistic. Could you define what you mean by “unaligned AI”? It seems quite plausible that I will agree with your position, and think it amounts to something like “we were pretty successful with alignment”.
The lack of robust evidence for your position seems sufficient, in my opinion, for the main thesis of my original post to hold.
I agree with theses like “it tentatively appears that the normative value of alignment work is very uncertain, and plausibly approximately neutral, from a total utilitarian perspective”, and would go further and say that alignment work is plausibly negative from a total utilitarian perspective.
I disagree with the implied theses in statements like “I’m not very sympathetic to pausing or slowing down AI as a policy proposal.”
If you wrote a post that just said “look, we’re super uncertain about things, here’s your reminder that there are worlds in which alignment work is negative”, I’d be on board with it. But it feels like a motte-and-bailey to write a post that is clearly trying to cause the reader to feel a particular way about some policy, and then retreat to “well my main thesis was very weak and unobjectionable”.
Some more minor comments:
You label this argument as strong but I think it is weak
Well, I can believe it’s weak in some absolute sense. My claim is that it’s much stronger than all of the arguments you make put together.
There is a strong selection pressure for AIs to display outward behaviors that are not overly alien-like. Indeed, the pressure seems to be for AIs to be inhumanly altruistic and kind in their actions.
This is a pretty good example of something I’d call different! You even use the adjective “inhumanly”!
To the extent your argument is that this is strong evidence that the AIs will continue to be altruistic and kind, I think I disagree, though I’ve now learned that you are imagining lots of alignment work happening when making the unaligned AIs, so maybe I’d agree depending on the specific scenario you’re imagining.
I disagree with the characterization that my argument relies primarily on the notion that “you can’t rule out” the possibility of AIs being even more utilitarian than humans.
Sorry, I was being sloppy there. My actual claim is that your arguments either:
- Don’t seem to bear on the question of whether AIs are more utilitarian than humans, OR
- Don’t seem more compelling than the reversed versions of those arguments.
I pointed out that AIs could potentially have a higher density of moral value per unit of matter, and I believe there are straightforward reasons to expect this to be the case, as AIs could be optimized very efficiently in terms of physical space. This is not merely a “you can’t rule it out” type of argument, in my view.
I agree that there’s a positive reason to expect AIs to have a higher density of moral value per unit of matter. I don’t see how this has any (predictable) bearing on whether AIs will be more utilitarian than humans.
Similarly, in the post, I pointed out that humans have many anti-utilitarian intuitions and it seems very plausible that AIs would not share (or share fewer of) these intuitions.
Applying the reversal test:
Humans have utilitarian intuitions too, and it seems very plausible that AIs would not share (or share fewer of) these intuitions.
I don’t especially see why one of these is stronger than the other.
(And if the AI doesn’t share any of the utilitarian intuitions, it doesn’t matter at all if it also doesn’t share the anti-utilitarian intuitions; either way it still won’t be a utilitarian.)
To give another example [...] AIs might care more about reproduction than humans (who by comparison, seem to want to have small population sizes with high per-capita incomes, rather than large population sizes with low per capita incomes as utilitarianism would recommend)
Applying the reversal test:
AIs might care less about reproduction than humans (a large majority of whom will reproduce at least once in their life).
Personally I find the reversed version more compelling.
I think you may be giving humans too much credit for being slightly utilitarian. [...] people (often those in the EA community) attribute a wide range of positive qualities to humans [...]
Fwiw my reasoning here mostly doesn’t depend on facts about humans other than binary questions like “do humans ever display property X”, since by and large my argument is “there is quite a strong chance that unaligned AIs do not have property X at all”.
Though again this might change depending on what exactly you mean by “unaligned AI”.
(I don’t necessarily disagree with your hypotheses as applied to the broader world—they sound plausible, though it feels somewhat in conflict with the fact that EAs care about AI consciousness a decent bit—I just disagree with them as applied to me in this particular comment thread.)
I think the real world data points should epistemically count for more than the intuitions?
I don’t buy it. The “real world data points” procedure here seems to be: take two high-level concepts (e.g. affective empathy, proclivity towards utilitarianism), draw a line between them, extrapolate way way out of distribution. I think this procedure would have a terrible track record when applied without the benefit of hindsight.
I expect my arguments based on intuitions would also have a pretty bad track record, but I do think they’d outperform the procedure above.
More generally, I think it is hard to argue about what might be true if real world data counts for less than intuitions.
Yup, this is an unfortunate fact about domains where you don’t get useful real world data. That doesn’t mean you should start using useless real world data.
Isn’t this the effect you alluded to, when you named reasons why some humans are utilitarians?
Yes, but I think the relevance is mostly whether or not the being feels pleasure or pain at all, rather than the magnitude with which it feels it. (Probably the magnitude matters somewhat, but not very much.)
Among humans I would weakly predict the opposite effect, that people with less pleasure-pain salience are more likely to be utilitarian (mostly due to a predicted anticorrelation with logical thinking / decoupling / systemizing nature).

Rohin Shah 1 May 2024 8:04 UTC
6 points
2 ∶ 0
in reply to: Matthew_Barnett’s comment on: Analyzing the moral value of unaligned AIs
This suggests affective empathy may not be strongly predictive of utilitarian motivations.
I can believe that if the population you are trying to predict for is just humans, almost all of whom have at least some affective empathy. But I’d feel pretty surprised if this were true in whatever distribution over unaligned AIs we’re imagining. In particular, I think if there’s no particular reason to expect affective empathy in unaligned AIs, then your prior on it being present should be near-zero (simply because there are lots of specific claims about unaligned AIs about that complicated most of which will be false). And I’d be surprised if “zero vs non-zero affective empathy” was not predictive of utilitarian motivations.
I definitely agree that AIs might feel pleasure and pain, though I’m less confident in it than you seem to be. It just seems like AI cognition could be very different from human cognition. For example, I would guess that pain/pleasure are important for learning in humans, but it seems like this is probably not true for AI systems in the current paradigm. (For gradient descent, the learning and the cognition happen separately—the AI cognition doesn’t even get the loss/reward equivalent as an input so cannot “experience” it. For in-context learning, it seems very unclear what the pain/pleasure equivalent would be.)
this line of argument would overlook the real possibility that unaligned AIs could [...] have an even stronger tendency towards utilitarian motivations.
I agree this is possible. But ultimately I’m not seeing any particularly strong reasons to expect this (and I feel like your arguments are mostly saying “nothing rules it out”). Whereas I do think there’s a strong reason to expect weaker tendencies: AIs will be different, and on average different implies fewer properties that humans have. So aggregating these I end up concluding that unaligned AIs will be less utilitarian in expectation.
(You make a bunch of arguments for why AIs might not be as different as we expect. I agree that if you haven’t thought about those arguments before you should probably reduce your expectation of how different AIs will be. But I still think they will be quite different.)
this line of argument would overlook the real possibility that unaligned AIs could be more conscious than humans,
I don’t see why it matters if AIs are more conscious than humans? I thought the relevant question we’re debating is whether they are more likely to be utilitarians. Maybe the argument is that if they are more conscious-in-the-sense-of-feeling-pleasure-and-pain they are more likely to be utilitarians? If so I might buy that but feel like it’s a weak effect.
As a consequence it’s hard for me to see a strong reason for preferring humans over AIs if you cared about pyramid-maximization.
Sure, but a big difference is that no human cares about pyramid-maximization, whereas some humans are utilitarians?
(Maybe some humans do care about pyramid-maximization? I’d need to learn more about those humans before I could have any guess about whether to prefer humans over AIs.)
Consciousness seems to be a fairly convergent function of intelligence
I would say “fairly convergent function of biologically evolved intelligence”. Evolution faced lots of constraints we don’t have in AI design. For example, cognition and learning had to be colocated in space and time (i.e. done in a single brain), whereas for AIs these can be (and are) separated. Seems very plausible that consciousness-in-the-sense-of-feeling-pleasure-and-pain is a solution needed under the former constraint but not the latter. (Maybe I’m at 20% chance that something in this vicinity is right, though that is a very made-up number.)

Rohin Shah 30 Apr 2024 21:20 UTC
4 points
0 ∶ 0
in reply to: Matthew_Barnett’s comment on: Analyzing the moral value of unaligned AIs
Given my new understanding of the meaning of “contingent” here, I’d say my claims are:
1. I’m unsure about how contingent the development of utilitarianism in humans was. It seems quite plausible that it was not very historically contingent. I agree my toy model does not accurately capture my views on the contingency of total utilitarianism.
2. I’m also unsure how contingent it is for unaligned AI, but aggregating over my uncertainty suggests more contingent.
One way to think about this is to ask: why are any humans utilitarians? To the extent it’s for reasons that don’t apply to unaligned AI systems, I think you should feel like it is less likely for unaligned AI systems to be utilitarians. So e.g. if I thought humans were utilitarians primarily because it is simple to express in concepts that humans and AIs share, then I would agree with you. But in fact I feel like it is pretty important that humans feel pleasure and pain, and have empathy, to explain why some humans are utilitarians. (Mostly I think the “true explanation” will have to appeal to more than simplicity, and the additional features this “true explanation” will appeal to are very likely to differ between humans and AIs.)
Compare to the hypothetical argument: humans aren’t very obsessed with building pyramids --> our current level of obsession with pyramid building is probably not unusual, in the sense that you might easily expect aliens/AIs to be similarly obsessed with building pyramids, or perhaps even more obsessed.
Indeed I feel like AIs probably build fewer pyramids in expectation, for basically the same reason. (The concrete hypothesis I generated for why humans build pyramids was “maybe pyramids were especially easy to build historically”.)
General note: I want to note that my focus on AI alignment is not necessarily coming from a utilitarian perspective. I work on AI alignment because in expectation I think a world with aligned AI will better reflect “my values” (which I’m uncertain about and may not reflect utilitarianism) than a world with unaligned AI. But I’m happy to continue talking about the implications for utilitarians.

Rohin Shah 30 Apr 2024 20:02 UTC
2 points
0 ∶ 0
in reply to: Matthew_Barnett’s comment on: Analyzing the moral value of unaligned AIs
I agree it’s clear that you claim that unaligned AIs are plausibly comparably utilitarian as humans, maybe more.
What I didn’t find was discussion of how contingent utilitarianism is in humans.
Though actually rereading your comment (which I should have done in addition to reading the post) I realize I completely misunderstood what you meant by “contingent”, which explains why I didn’t find it in the post (I thought of it as meaning “historically contingent”). Sorry for the misunderstanding.
Let me backtrack like 5 comments and retry again.