I donât think impartiality to the extent of not caring more about the people one loves is a core value for very many EAs? Yes, itâs pretty central to EA that most people are excessively partial, but I donât recall ever seeing someone advocate full impartiality.
Some of the reaction here may be based on Jaime acting in a professional, rather than a personal, capacity when working in AI.
There are a number of jobs and roles that expect your actions in a professional capacity to be impartial in the sense of not favoring your loved ones over others. For instance, a politician should not give any more weight to the effects of proposed legislation on their own mother than the effect on any other constituent. Government service in general has this expectation. One could argue that (like serving as a politician), working in AI involves handing out significant risks and harms to non-consenting othersâand that should trigger a duty of impartiality.
Government workers and politicians are free to favor their own mother in their personal life, of course.
It seems like the view expressed reduces to an existing-person-effecting view. Is their any plausible mechanism by which an action by Epoch is supposed to impact Sevillaâs friends/ârelatives specifically? I seriously doubt it. The only plausible mechanism would be that AI goes well instead of poorly, which would benefit all existing people. This makes the politician comparison, as stated, dis-analogousness. Would you say that if a politician said their motivation to become a politician was to make a better world for their children, for example, that would somehow violate their duties? Seems like a lot of politicians might have issue if that were the case.
I think this suggests a risk that the real infraction here is honestly stating the consideration about friends and family. Is it really the case that no-one leading AI safety orgs that are aiming for deceleration are motivated, at least partly, by the desire to protect their own friends and family from the consequences of AI going poorly? I will confess that is a big part of my own reasons from being interested in this topic. I would be very surprised if the standard being suggested here was really as ubiquitous as these comments suggest.
Iâd agree that a lot of people who care about AI safety do so because they want to leave the world a better place for their children (which encompasses their childrenâs wellbeing related to being parents themselves and having to worry about their own childrenâs future). But thereâs no trade off between personal and impartial preferences there. That seems to me to be quite different from saying youâre prioritising eg your parents and grandparents getting to have extended lifespans over other peopleâs childrenâs wellbeing.
The discussion also isnât about the effects of Epochâs specific work, so Iâm a bit confused by your argument relying on that.
From Jaime:
âBut I want to be clear that even if you convinced me somehow that the risk that AI is ultimately bad for the world goes from 15% to 1% if we wait 100 years I would not personally take that deal. If it reduced the chances by a factor of 100 I would consider it seriously. But 100 years has a huge personal cost to me, as all else equal it would likely imply everyone I know [italics mine] being dead.
To be clear I donât think this is the choice we are facing or we are likely to face.â
But thereâs no trade off between personal and impartial preferences there. That seems to me to be quite different from saying youâre prioritising eg your parents and grandparents getting to have extended lifespans over other peopleâs childrenâs wellbeing.
I can see why you would interpret it this way given the context, but I read the statement differently. Based on my read of the thread, the comment was in response to a question about benefiting people sooner rather than later. This is why I say it reduces to an existing-person-effecting view (which, at least as far as I am aware, is not an unacceptable position to hold in EA). The question is functionally about current vs future people, not literally Sevillaâs friends and family specifically. I think this matches the âmaking the world better for your childrenâ idea. You can channel a love of friends and family into an altruistic impulse, so long as there isnât some specific conflict-of-interest where youâre benefiting them specifically. I think the statement in question is consistent with that.
The discussion also isnât about the effects of Epochâs specific work, so Iâm a bit confused by your argument relying on that.
Iâm bringing this up because I think its implausible that anything that is being discussed here has some specific relevance to Sevillaâs friends and family as individuals (in support of my point above). In other words, due to the nature of the actions being taken
thereâs no trade off between personal and impartial preferences there
In what way are any concrete actions that are relevant here prioritizing Sevillaâs family over other peopleâs children? Although I can see how it might initially seem that way I donât think thatâs what the statement was intended to communicate.
Have you read the whole Twitter thread including Jaimeâs responses to comments? He repeatedly emphasises that itâs about his literal friends, family and self, and hypothetical moderate but difficult trade offs with the welfare of others.
When I click the link I see three posts that go Sevilla, Lifland, Sevilla. I based my comments above on those. I havenât read through all the other replies by others or posts responding to them. If there is context in those or else where that is relevant Iâm open to changing my mind based on that.
He repeatedly emphasises that itâs about his literal friends, family and self, and hypothetical moderate but difficult trade offs with the welfare of others.
Can you say what statements lead you to this conclusion? For example, you quote him saying something I havenât seen, perhaps part of the thread I didnât read.
âBut I want to be clear that even if you convinced me somehow that the risk that AI is ultimately bad for the world goes from 15% to 1% if we wait 100 years I would not personally take that deal. If it reduced the chances by a factor of 100 I would consider it seriously. But 100 years has a huge personal cost to me, as all else equal it would likely imply everyone I know [italics mine] being dead. To be clear I donât think this is the choice we are facing or we are likely to face.â
To me, this seems to confirm what I said above:
Based on my read of the thread, the comment was in response to a question about benefiting people sooner rather than later. This is why I say it reduces to an existing-person-effecting view (which, at least as far as I am aware, is not an unacceptable position to hold in EA). The question is functionally about current vs future people, not literally Sevillaâs friends and family specifically.
Yes, Sevilla is motivated specifically by considerations about those he loves, and yes, there is a trade-off, but that trade-off is really about current vs future people. People who arenât longtermists for example would also implicate this same trade-off. I donât think Sevilla would be getting the same reaction here if he just said he isnât a longtermist. Because of the nature of the available actions, the interests of Sevillaâs loved-ones is aligned with those of current people (but not necessarily future people). The reason why âeveryone [he] know[s]â will be dead is because everyone will be dead, in that scenario.
You might think that having loved-ones as a core motivation above other people is inherently a problem. I think this is answered above by Jeff Kaufman:
I donât think impartiality to the extent of not caring more about the people one loves is a core value for very many EAs? Yes, itâs pretty central to EA that most people are excessively partial, but I donât recall ever seeing someone advocate full impartiality.
I agree with this statement. Therefore my view is that simply stating that youâre more motivated by consequences to your loved-ones is not, in and of itself, a violation of a core EA idea.
Jason offers a refinement of this view. Perhaps what Kaufman says is true, but what if there is a more specific objection?
There are a number of jobs and roles that expect your actions in a professional capacity to be impartial in the sense of not favoring your loved ones over others. For instance, a politician should not give any more weight to the effects of proposed legislation on their own mother than the effect on any other constituent.
Perhaps the issue is not necessarily that Sevilla has the motivation itself, but that his role comes with a specific conflict-of-interest-like duty, which the statement suggests he is violating. My response was addressing this argument. I claim that the duty isnât as broad as Jason seems to imply:
It seems like the view expressed reduces to an existing-person-effecting view. Is their any plausible mechanism by which an action by Epoch is supposed to impact Sevillaâs friends/ârelatives specifically? I seriously doubt it. The only plausible mechanism would be that AI goes well instead of poorly, which would benefit all existing people. This makes the politician comparison, as stated, dis-analogousness. Would you say that if a politician said their motivation to become a politician was to make a better world for their children, for example, that would somehow violate their duties? Seems like a lot of politicians might have issue if that were the case.
Does a politician who votes for a bill and states they are doing so to âmake a better world for their childrenâ, violate a conflict-of-interest duty? Jasonâs argument seems to suggest they would. Letâs assume they are being genuine, they really are significantly motivated by care for their children, more than for a random citizen. They apply more weight to the impact of the legislation on their children then to others, violating Jasonâs proposed criteria.
Yet I donât think we would view such statements as disqualifying for a politician. The reason is that the mechanism by which they benefit their children really only operates by also helping everyone else. Most legislation wonât have any different impact on their children compared to any other person. So while the statement nominally suggests a conflict-of-interest, in practice the politicians incentives are aligned, the only way that voting for this legislation helps their children is that it helps everyone, and that includes their children. If the legislation plausibly did have a specific impact on their child (for example impacting an industry their child works in), then that really could be a conflict-of-interest. My claim is there needs to be some greater specificity for a conflict to exist. Sevillaâs case is more like the first case than the second, or at least that is my claim:
Is their any plausible mechanism by which an action by Epoch is supposed to impact Sevillaâs friends/ârelatives specifically? I seriously doubt it. The only plausible mechanism would be that AI goes well instead of poorly, which would benefit all existing people.
So, what has Sevilla done wrong? My analysis is this. It isnât simply that he is more motivated to help his loved-ones (Kaufman argument). Nor is it something like a conflict-of-interest (my argument). In another comment on this thread I said this:
People can do a bad thing because they are just wrong in their analysis of a situation or their decision-making.
I think, at bottom, the problem is that Sevilla makes mistake in his analysis and/âor decision-making about AI. His statements arenât norm-violating, they are just incorrect (at least some of them are, in my opinion). I think its worth having clarity about what the actual âproblemâ is.
The reason why âeveryone [he] know[s]â will be dead is because everyone will be dead, in that scenario.
We are already increasing maximum human lifespan, so I wouldnât be surprised if many people who are babies now are still alive in 100 years. And even if they arenât, thereâs still the element of their wellbeing while they are alive being affected by concerns about the world they will be leaving their own children to.
Although I havenât thought deeply about the issue you raise you could definitely be correct, and I think they are reasonable things to discuss. But I donât see their relevance to my arguments above. The quote you reference is itself discussing a quote from Sevilla that analyzes a specific hypothetical. I donât necessarily think Sevilla had the issues you raise in mind when we was addressing that hypothetical. I donât think his point was that based on forecasts of life extension technology he had determined that acceleration was the optimal approach in light of his weighing of 1 year-olds vs 50 year-olds. I think his point is more similar to what I mention above about current vs future people. I took a look at more of the X discussion, including the part where that quote comes from, and I think it is pretty consistent with this view (although of course others may disagree). Maybe he should factor in the things you mention, but to the extent his quote is being used to determine his views, I donât think the issues you raise are relevant unless he was considering them when he made the statement. On the other hand, I think discussing those things could be useful in other, more object level discussions. Thatâs kind of what I was getting at here:
I think, at bottom, the problem is that Sevilla makes mistake in his analysis and/âor decision-making about AI. His statements arenât norm-violating, they are just incorrect (at least some of them are, in my opinion). I think its worth having clarity about what the actual âproblemâ is.
I know Iâve been commenting here a lot, and I understand my style may seem confrontational and abrasive in some cases. I also donât want to ruin peopleâs day with my self-important rants, so, having said my piece, Iâll drop the discussion for now and let you get on with other things.
(although it you would like to response you are of course welcome, I just mean to say I wonât continue the back-and-forth after, so as not to create a pressure to keep responding.)
I donât think youâre being confrontational, I just think youâre over-complicating someone saying they support things that might bring AGI forward to 2035 instead of 2045 because otherwise it will be too late for their older relatives. And itâs not that motivating to debate things that feel like over-complications.
I agree that there are no plausible circumstances in which anyoneâs relatives will benefit in a way not shared with a larger class of people. However, I do think groups of people differ in ways that are relevant to how important fast AI development vs. more risk-averse AI development is to their interests. Giving undue weight to the interests of a group of people because oneâs friends or family are in that group would still raise the concern I expressed above.
One group thatâif they were considering their own interests onlyâmight be rationally expected to accept somewhat more risk than the population as a whole are those who are ~50-55+. As Jaime wrote:
For some of my older relatives, it might make a big difference to their health and wellbeing whether AI-fueled explosive growth happens in 10 vs 20 years.
A similar outcome could also happen if (e.g.) the prior generation of my family has passed on, I had young children, and as a result of prioritizing their interests I didnât give enough weight to older individualsâ desire to have powerful AI soon enough to improve and/âor extend their lives.
the prior generation of my family has passed on, I had young children
This seems to suggest that you think the politicians âmaking the world better for my childrenâ statement would then also be problematic. Do you agree with that?
Iâll be honest, this argument seems a bit too clever. Is the underlying problem with the statement really that it implies a set of motivations that might slightly up-weight a certain age group? One of the comments speaks of âcore valuesâ for EA. Is that really a core value? Iâm pretty sure I recall reading an argument by McAskill about how actually we should more heavily weight young people in various ways (I think it was voting), for example. I serious doubt most EAs could claim that they literally are distributionally exact in weighting all morally relevant entities in every decision they make. I think the âcore valueâ that exists probably isnât really this demanding, although I could be wrong.
Prioritising young people often makes sense from an impartial welfare standpoint, because young people have more years left, so there is more welfare to be affected. With voting in particular, itâs the younger people who have to deal with the longer term consequences of any electoral outcome. You see this in climate change related critiques of the Baby Boomer generation.
See eg
âEffective altruism can be defined by four key values: âŠ
2. Impartial altruism: all people count equally â effective altruism aims to give everyoneâs interests equal weight, no matter where or when they live. When combined with prioritisation, this often results in focusing on neglected groupsâŠâ
Prioritising young people often makes sense from an impartial welfare standpoint
Sure, I think you can make a reasonable argument for that, but if someone disagreed with that, would you say they lack impartiality? To me it seems like something that is up for debate, within the âmargin-of-errorâ of what is meant by impartiality. Two EAs could come down on different sides of that issue and still be in good standing in the community, and wouldnât be considered to not believe in the general principle of impartiality. Likewise, I think we can interpret Jeff Kaufmanâs argument above as expressing a similar view about an individualâs loved-ones. It is within the âmargin-of-errorâ of impartiality to still have a higher degree of concern for loved-ones, even if that might not be living up to the platonic ideal of impartiality.
My point in bringing this up is, the exact reason why the statement in question is bad seems to be shifting a bit over the conversation. Is the core reason that Sevillaâs statement is objectionable really that it might up-weight people in a certain age group?
TFD, I think your analysis is correct and incisive. Iâm grateful to you for writing these comments on this post.
It seems clear that if Jaime had different views about the risk-reward of hypothetical 21st century AGI, nobody would be complaining about him loving his family.
Accusing Jaime of âselfishnessâ, even though he used that term himself in (what I interpret to be) a self-deprecating way, seems really unfair and unreasonable, and just excessively mean. As you and Jeff Kaufman pointed out, many people who are accepted into the EA movement have the same or similar views as Jaime on who to prioritize and so on. These criticisms would not be levied against Jaime if he were not an AI risk skeptic.
The social norms of EA or at least the EA Forum are different today than they were ten years ago. Ten years ago, if you said you only care about people who are either alive today or who will be born in the next 100 years, and you donât think much about AGI because global poverty seems a lot more important, then you would be fully qualified to be the president of a university EA group, get a job at a meta-EA organization, or represent the views of the EA movement to a public audience.
Today, it seems like there are a lot more people who self-identify as EAs who see focusing on global poverty as more or less a waste of time relative to the only thing that matters, which is that the Singularity is coming in about 2-5 years (unless we take drastic action), and all our efforts should be focused on making sure the Singularity goes good and not bad â including trying to delay it if that helps. People who disagree with this view have not yet been fully excluded from EA but it seems like some people are pretty mean to people who disagree. (I am one of the people who disagrees.)
As a side note, itâs also strange to me that people are treating the founding of Mechanize as if it has a realistic chance to accelerate AGI progress more than a negligible amount â enough of a chance of enough of an acceleration to be genuinely concerning. AI startups are created all the time. Some of them state wildly ambitious goals, like Mechanize. They typically fail to achieve these goals. The startup Vicarious comes to mind.
There are many startups trying to automate various kinds of physical and non-physical labour. Some larger companies like Tesla and Alphabet are also working on this. Why would Mechanize be particularly concerning or be particularly likely to succeed?
The social norms of EA or at least the EA Forum are different today than they were ten years ago. Ten years ago, if you said you only care about people who are either alive today or who will be born in the next 100 years, and you donât think much about AGI because global poverty seems a lot more important, then you would be fully qualified to be the president of a university EA group, get a job at a meta-EA organization, or represent the views of the EA movement to a public audience.
This isnât just a social thing, itâs also response to a lot of changes in AI timelines over the past ten years. Back then a lot of us had views like âmost experts think powerful AI is far off, Iâm not going to sink a bunch of time into how it might affect my various options for doing goodâ, but as expert views have shifted that makes less sense. While âdonât think much about AGI because global poverty seems a lot more importantâ is still a reasonable position to hold (ex: people who think we canât productively influence how AI goes and so we should focus on doing as much good as we can in areas we can affect), I think it requires a good bit more reasoning and thought than it did ten years ago.
(On the other hand, I see âonly care about people who are either alive today or who will be born in the next 100 yearsâ as still within the range of common EA views (ex).)
I see it primarily as a social phenomenon because I think the evidence we have today that AGI will arrive by 2030 is less compelling than the evidence we had in 2015 that AGI would arrive by 2030. In 2015, it was a little more plausible that AGI could arrive by 2030 because that was 15 years away and who knows what can happen in 15 years.
Now that 2030 is a little less than 5 years away, AGI by 2030 is a less plausible prediction than it was in 2015 because thereâs less time left and itâs more clear it wonât happen.
I donât think the reasons people believe AGI will arrive by 2030 are primarily based on evidence but are primarily a sociological phenomenon. People were ready to believe this regardless of the evidence going back to Ray Kurzweilâs The Age of Spiritual Machines in 1999 and Eliezer Yudkowskyâs âEnd-of-the-World Betâ in 2017. People donât really pay attention to whether the evidence is good or bad, they ignore obvious evidence and arguments against near-term AGI, and they mostly make a choice to ignore or attack people who express disagreement and instead tune into the relentless drumbeat of people agreeing with them. This is sociology, not epistemology.
Donât believe me? Talk to me again in 5 years and send me a fruit basket. (Or just kick the can down the road and say AGI is coming in 2035...)
Expert opinion has changed? First, expert opinion is not itself evidence, itâs peopleâs opinions about evidence. What evidence are the experts basing their beliefs on? That seems way more important than someone just saying a number based on an intuition.
Second, expert opinion does not clearly support the idea of near-term AGI.
As of 2023, the expert opinion on AGI was⊠well, first of all, really confusing. The AI Impacts survey found that the experts believed there is a 50% chance by 2047 that âunaided machines can accomplish every task better and more cheaply than human workers.â And also that thereâs a 50% chance that by 2116 âmachines could be built to carry out the task better and more cheaply than human workers.â I donât know why these predictions are 69 years apart.
Regardless, 2047 is sufficiently far away that it might as well be 2057 or 2067 or 2117. This is just people generating a number using a gut feeling. We donât know how to build AGI and we have no idea how long it will take to figure out how to. No amount of thinking of numbers or saying numbers can escape this fundamental truth.
We actually wonât have to wait long to see that some of the most attention-catching near-term AI predictions are false. Dario Amodei, the CEO of Anthropic (a company that is said to be âliterally creating Godâ), has predicted that by some point between June 2025 and September 2025, 90% of all code will written by AI rather than humans. In late 2025 and early 2026, when itâs clear Dario was wrong about this (when, not if), maybe some people will start to be more skeptical of attention-grabbing expert predictions. But maybe not.
There are already strong signs of AGI discourse being irrational and absurd. On April 16, 2025, Tyler Cowen claimed that OpenAIâs o3 model is AGI and asked, âis April 16th AGI day?â. In a follow-up post on April 17, seemingly in response to criticism, he said, âI donât mind if you donât want to call it AGIâ, but seemed to affirm he still thinks o3 is AGI.
On one hand, I hope that in 5 years the people who promoted the idea of AGI by 2030 will lose a lot of credibility and maybe will do some soul-searching to figure out how they could be so wrong. On the other hand, there is nothing preventing people from being irrational indefinitely, such as:
Defining whatever exists in 2030 as AGI (Tyler Cowen already did it in 2025, and Ray Kurzweil innovated the technique years ago).
Kicking the can down the road a few years, and repeat as necessary (similar to how Elon Musk has predicted that the Tesla fleet will achieve Level 4â5 autonomy in a year every year from 2015 to 2025 and has not given up the game despite his losing streak).
Telling a story in which AGI didnât happen only because effective altruists or other good actors successfully delayed AGI development.
I think part of the sociological problem is that people are just way too polite about how crazy this all is and how awful the intellectual practices of effective altruists have been on this topic. (Sorry!) So, Iâm being blunt about this to try to change that a little.
I see it primarily as a social phenomenon because I think the evidence we have today that AGI will arrive by 2030 is less compelling than the evidence we had in 2015 that AGI would arrive by 2030.
The evidence we have today that there will be AGI by 2030 is clearly dramatically stronger than the evidence we had in 2015 that there would be AGI by 2020, and that is surely the relevant comparison. This is not EA specificâwe have been ahead of the curve in thinking AI would be a big deal, but the whole world has updated in this direction, and it would be strange if we hadnât as well.
My personal take is that there are pretty reasonable arguments that what we have seen in AI/âML since 2015 suggests AI will be a big deal. I like the way I have seen Yoshua Bengio talk about it âover the next few years, or a few decadesâ. I share the view that either of those possibilities are reasonable. People who are highly confident that something like AGI is going to arrive over the next few years are more confident in this than I am, but I think that view is within the bounds of reasonable interpretation of the evidence. I think it is also with-in the bounds of reasonable to have the opposite view, that something like AGI is most likely further than a few years away.
Donât believe me? Talk to me again in 5 years and send me a fruit basket. (Or just kick the can down the road and say AGI is coming in 2035...)
I think this is a healthy attitude and that I think is worth appreciating. We may get answers to these questions over the next few years. That seems pretty positive to me. We will be able to resolve some of these disagreements productively by observing what happens. I hope people who have different views now keep this in mind and that the environment is still in a good place for people who disagree now to work together in the future if some of these disagreements get resolved.
I will offer the ea forum internet-points equivalent of a fruit basket to anyone who would like one in the future if we disagree now and in the future they are proven right and I am proven wrong.
I think part of the sociological problem is that people are just way too polite about how crazy this all is and how awful the intellectual practices of effective altruists have been on this topic.
Can you saw what view it is you think is crazy? It seems quite reasonable to me to think that AI is going to be a massive deal and therefore that it would be highly useful to influence how it goes. On other other hand, I think people often over-estimate the robustness of the arguments for any given strategy for how to actually do that influencing. In other words, its reasonable to prioritize AI, but peopleâs AI takes are often very over-confident.
For what itâs worth, I basically agree with the view that Mechanize is unlikely to be successful at itâs goals:
As a side note, itâs also strange to me that people are treating the founding of Mechanize as if it has a realistic chance to accelerate AGI progress more than a negligible amount â enough of a chance of enough of an acceleration to be genuinely concerning. AI startups are created all the time. Some of them state wildly ambitious goals, like Mechanize. They typically fail to achieve these goals. The startup Vicarious comes to mind.
There are many startups trying to automate various kinds of physical and non-physical labour. Some larger companies like Tesla and Alphabet are also working on this. Why would Mechanize be particularly concerning or be particularly likely to succeed?
It seems clear that if Jaime had different views about the risk-reward of hypothetical 21st century AGI, nobody would be complaining about him loving his family.
I do think this is substantially correct, but I also want to acknowledge that these can be difficult subjects to navigate. I think anyone has done anything wrong, Iâm sure I myself have done something similar to this many times. But I do think its worth trying to understand where the central points of disagreement lie, and I think this really is the central disagreement.
On the question of changing EA attitudes towards AI over the years, although I personally think AI will be a big deal, could be dangerous, and those issues are worth of significant attention, I also can certainly see reasons why people might disagree and why those people would have reasonable grievances with decisions by certain EA people and organizations.
An idea that I have pondered for a while about EA is a theory about which âboundariesâ a community emphasizes. Although Iâve only ever interacted with EA by reading related content online, my perception is that EA really emphasizes the boundary around the EA community itself, while de-emphasizing the boundaries around individual people or organizations. The issues around Epoch I think demonstrate this. The feeling of betrayal comes from viewing âthe communityâ as central. I think a lot of other cultures that place more emphasize on those other boundaries might react differently. For example, at most companies I have worked at, although certainly they would never be happy to see an employee leave, they wouldnât view moving to another job as a betrayal, even if an employee went to work for a direct competitor. I personally think placing more emphasis on orgs/âindividuals rather than the community as a whole could have some benefits, such as with the issue you raise about how to navigate changing views on AI.
Although emphasizing âthe communityâ might seem like its ideal for cooperation, I think it can actually harm cooperation in the presence of substantial disagreements, because it generates dynamics like what is going on here. People feel like they canât cooperate with people across the disagreement. We will probably see some of these disagreements resolved over the next few years as AI progresses. I for one hope that even if I am wrong I can take any necessary corrections on-board and still work with people who I disagreed with to make positive contributions. Likewise, I hope that if I am right, people who I disagreed with still feel like they can work with me despite that.
As a side note, itâs also strange to me that people are treating the founding of Mechanize as if it has a realistic chance to accelerate AGI progress more than a negligible amount â enough of a chance of enough of an acceleration to be genuinely concerning. AI startups are created all the time. Some of them state wildly ambitious goals, like Mechanize. They typically fail to achieve these goals. The startup Vicarious comes to mind.
I admit I had a similar thought, but I am of two minds about it. On the one hand, I think intentions do matter. I think it is reasonable to point out if you think someone is making a mistake, even if you think ultimately that mistake is unlikely to have a substantial impact because the person is unlikely to succeed in what they are trying to do.
On the other hand, I do think the degree of the reaction and the way that people are generalizing seems like people are almost pricing in the idea that the actions in question have already had a huge impact. So I do wonder if people are kind of over-updating on this specific case for similar reasons to what you mention.
I donât think impartiality to the extent of not caring more about the people one loves is a core value for very many EAs? Yes, itâs pretty central to EA that most people are excessively partial, but I donât recall ever seeing someone advocate full impartiality.
Some of the reaction here may be based on Jaime acting in a professional, rather than a personal, capacity when working in AI.
There are a number of jobs and roles that expect your actions in a professional capacity to be impartial in the sense of not favoring your loved ones over others. For instance, a politician should not give any more weight to the effects of proposed legislation on their own mother than the effect on any other constituent. Government service in general has this expectation. One could argue that (like serving as a politician), working in AI involves handing out significant risks and harms to non-consenting othersâand that should trigger a duty of impartiality.
Government workers and politicians are free to favor their own mother in their personal life, of course.
It seems like the view expressed reduces to an existing-person-effecting view. Is their any plausible mechanism by which an action by Epoch is supposed to impact Sevillaâs friends/ârelatives specifically? I seriously doubt it. The only plausible mechanism would be that AI goes well instead of poorly, which would benefit all existing people. This makes the politician comparison, as stated, dis-analogousness. Would you say that if a politician said their motivation to become a politician was to make a better world for their children, for example, that would somehow violate their duties? Seems like a lot of politicians might have issue if that were the case.
I think this suggests a risk that the real infraction here is honestly stating the consideration about friends and family. Is it really the case that no-one leading AI safety orgs that are aiming for deceleration are motivated, at least partly, by the desire to protect their own friends and family from the consequences of AI going poorly? I will confess that is a big part of my own reasons from being interested in this topic. I would be very surprised if the standard being suggested here was really as ubiquitous as these comments suggest.
Iâd agree that a lot of people who care about AI safety do so because they want to leave the world a better place for their children (which encompasses their childrenâs wellbeing related to being parents themselves and having to worry about their own childrenâs future). But thereâs no trade off between personal and impartial preferences there. That seems to me to be quite different from saying youâre prioritising eg your parents and grandparents getting to have extended lifespans over other peopleâs childrenâs wellbeing.
The discussion also isnât about the effects of Epochâs specific work, so Iâm a bit confused by your argument relying on that.
From Jaime:
âBut I want to be clear that even if you convinced me somehow that the risk that AI is ultimately bad for the world goes from 15% to 1% if we wait 100 years I would not personally take that deal. If it reduced the chances by a factor of 100 I would consider it seriously. But 100 years has a huge personal cost to me, as all else equal it would likely imply everyone I know [italics mine] being dead. To be clear I donât think this is the choice we are facing or we are likely to face.â
I can see why you would interpret it this way given the context, but I read the statement differently. Based on my read of the thread, the comment was in response to a question about benefiting people sooner rather than later. This is why I say it reduces to an existing-person-effecting view (which, at least as far as I am aware, is not an unacceptable position to hold in EA). The question is functionally about current vs future people, not literally Sevillaâs friends and family specifically. I think this matches the âmaking the world better for your childrenâ idea. You can channel a love of friends and family into an altruistic impulse, so long as there isnât some specific conflict-of-interest where youâre benefiting them specifically. I think the statement in question is consistent with that.
Iâm bringing this up because I think its implausible that anything that is being discussed here has some specific relevance to Sevillaâs friends and family as individuals (in support of my point above). In other words, due to the nature of the actions being taken
In what way are any concrete actions that are relevant here prioritizing Sevillaâs family over other peopleâs children? Although I can see how it might initially seem that way I donât think thatâs what the statement was intended to communicate.
Have you read the whole Twitter thread including Jaimeâs responses to comments? He repeatedly emphasises that itâs about his literal friends, family and self, and hypothetical moderate but difficult trade offs with the welfare of others.
When I click the link I see three posts that go Sevilla, Lifland, Sevilla. I based my comments above on those. I havenât read through all the other replies by others or posts responding to them. If there is context in those or else where that is relevant Iâm open to changing my mind based on that.
Can you say what statements lead you to this conclusion? For example, you quote him saying something I havenât seen, perhaps part of the thread I didnât read.
To me, this seems to confirm what I said above:
Yes, Sevilla is motivated specifically by considerations about those he loves, and yes, there is a trade-off, but that trade-off is really about current vs future people. People who arenât longtermists for example would also implicate this same trade-off. I donât think Sevilla would be getting the same reaction here if he just said he isnât a longtermist. Because of the nature of the available actions, the interests of Sevillaâs loved-ones is aligned with those of current people (but not necessarily future people). The reason why âeveryone [he] know[s]â will be dead is because everyone will be dead, in that scenario.
You might think that having loved-ones as a core motivation above other people is inherently a problem. I think this is answered above by Jeff Kaufman:
I agree with this statement. Therefore my view is that simply stating that youâre more motivated by consequences to your loved-ones is not, in and of itself, a violation of a core EA idea.
Jason offers a refinement of this view. Perhaps what Kaufman says is true, but what if there is a more specific objection?
Perhaps the issue is not necessarily that Sevilla has the motivation itself, but that his role comes with a specific conflict-of-interest-like duty, which the statement suggests he is violating. My response was addressing this argument. I claim that the duty isnât as broad as Jason seems to imply:
Does a politician who votes for a bill and states they are doing so to âmake a better world for their childrenâ, violate a conflict-of-interest duty? Jasonâs argument seems to suggest they would. Letâs assume they are being genuine, they really are significantly motivated by care for their children, more than for a random citizen. They apply more weight to the impact of the legislation on their children then to others, violating Jasonâs proposed criteria.
Yet I donât think we would view such statements as disqualifying for a politician. The reason is that the mechanism by which they benefit their children really only operates by also helping everyone else. Most legislation wonât have any different impact on their children compared to any other person. So while the statement nominally suggests a conflict-of-interest, in practice the politicians incentives are aligned, the only way that voting for this legislation helps their children is that it helps everyone, and that includes their children. If the legislation plausibly did have a specific impact on their child (for example impacting an industry their child works in), then that really could be a conflict-of-interest. My claim is there needs to be some greater specificity for a conflict to exist. Sevillaâs case is more like the first case than the second, or at least that is my claim:
So, what has Sevilla done wrong? My analysis is this. It isnât simply that he is more motivated to help his loved-ones (Kaufman argument). Nor is it something like a conflict-of-interest (my argument). In another comment on this thread I said this:
I think, at bottom, the problem is that Sevilla makes mistake in his analysis and/âor decision-making about AI. His statements arenât norm-violating, they are just incorrect (at least some of them are, in my opinion). I think its worth having clarity about what the actual âproblemâ is.
We are already increasing maximum human lifespan, so I wouldnât be surprised if many people who are babies now are still alive in 100 years. And even if they arenât, thereâs still the element of their wellbeing while they are alive being affected by concerns about the world they will be leaving their own children to.
Although I havenât thought deeply about the issue you raise you could definitely be correct, and I think they are reasonable things to discuss. But I donât see their relevance to my arguments above. The quote you reference is itself discussing a quote from Sevilla that analyzes a specific hypothetical. I donât necessarily think Sevilla had the issues you raise in mind when we was addressing that hypothetical. I donât think his point was that based on forecasts of life extension technology he had determined that acceleration was the optimal approach in light of his weighing of 1 year-olds vs 50 year-olds. I think his point is more similar to what I mention above about current vs future people. I took a look at more of the X discussion, including the part where that quote comes from, and I think it is pretty consistent with this view (although of course others may disagree). Maybe he should factor in the things you mention, but to the extent his quote is being used to determine his views, I donât think the issues you raise are relevant unless he was considering them when he made the statement. On the other hand, I think discussing those things could be useful in other, more object level discussions. Thatâs kind of what I was getting at here:
I know Iâve been commenting here a lot, and I understand my style may seem confrontational and abrasive in some cases. I also donât want to ruin peopleâs day with my self-important rants, so, having said my piece, Iâll drop the discussion for now and let you get on with other things.
(although it you would like to response you are of course welcome, I just mean to say I wonât continue the back-and-forth after, so as not to create a pressure to keep responding.)
I donât think youâre being confrontational, I just think youâre over-complicating someone saying they support things that might bring AGI forward to 2035 instead of 2045 because otherwise it will be too late for their older relatives. And itâs not that motivating to debate things that feel like over-complications.
I agree that there are no plausible circumstances in which anyoneâs relatives will benefit in a way not shared with a larger class of people. However, I do think groups of people differ in ways that are relevant to how important fast AI development vs. more risk-averse AI development is to their interests. Giving undue weight to the interests of a group of people because oneâs friends or family are in that group would still raise the concern I expressed above.
One group thatâif they were considering their own interests onlyâmight be rationally expected to accept somewhat more risk than the population as a whole are those who are ~50-55+. As Jaime wrote:
A similar outcome could also happen if (e.g.) the prior generation of my family has passed on, I had young children, and as a result of prioritizing their interests I didnât give enough weight to older individualsâ desire to have powerful AI soon enough to improve and/âor extend their lives.
This seems to suggest that you think the politicians âmaking the world better for my childrenâ statement would then also be problematic. Do you agree with that?
Iâll be honest, this argument seems a bit too clever. Is the underlying problem with the statement really that it implies a set of motivations that might slightly up-weight a certain age group? One of the comments speaks of âcore valuesâ for EA. Is that really a core value? Iâm pretty sure I recall reading an argument by McAskill about how actually we should more heavily weight young people in various ways (I think it was voting), for example. I serious doubt most EAs could claim that they literally are distributionally exact in weighting all morally relevant entities in every decision they make. I think the âcore valueâ that exists probably isnât really this demanding, although I could be wrong.
Prioritising young people often makes sense from an impartial welfare standpoint, because young people have more years left, so there is more welfare to be affected. With voting in particular, itâs the younger people who have to deal with the longer term consequences of any electoral outcome. You see this in climate change related critiques of the Baby Boomer generation.
See eg
âEffective altruism can be defined by four key values: âŠ
2. Impartial altruism: all people count equally â effective altruism aims to give everyoneâs interests equal weight, no matter where or when they live. When combined with prioritisation, this often results in focusing on neglected groupsâŠâ
https://ââ80000hours.org/ââ2020/ââ08/ââmisconceptions-effective-altruism/ââ
Sure, I think you can make a reasonable argument for that, but if someone disagreed with that, would you say they lack impartiality? To me it seems like something that is up for debate, within the âmargin-of-errorâ of what is meant by impartiality. Two EAs could come down on different sides of that issue and still be in good standing in the community, and wouldnât be considered to not believe in the general principle of impartiality. Likewise, I think we can interpret Jeff Kaufmanâs argument above as expressing a similar view about an individualâs loved-ones. It is within the âmargin-of-errorâ of impartiality to still have a higher degree of concern for loved-ones, even if that might not be living up to the platonic ideal of impartiality.
My point in bringing this up is, the exact reason why the statement in question is bad seems to be shifting a bit over the conversation. Is the core reason that Sevillaâs statement is objectionable really that it might up-weight people in a certain age group?
TFD, I think your analysis is correct and incisive. Iâm grateful to you for writing these comments on this post.
It seems clear that if Jaime had different views about the risk-reward of hypothetical 21st century AGI, nobody would be complaining about him loving his family.
Accusing Jaime of âselfishnessâ, even though he used that term himself in (what I interpret to be) a self-deprecating way, seems really unfair and unreasonable, and just excessively mean. As you and Jeff Kaufman pointed out, many people who are accepted into the EA movement have the same or similar views as Jaime on who to prioritize and so on. These criticisms would not be levied against Jaime if he were not an AI risk skeptic.
The social norms of EA or at least the EA Forum are different today than they were ten years ago. Ten years ago, if you said you only care about people who are either alive today or who will be born in the next 100 years, and you donât think much about AGI because global poverty seems a lot more important, then you would be fully qualified to be the president of a university EA group, get a job at a meta-EA organization, or represent the views of the EA movement to a public audience.
Today, it seems like there are a lot more people who self-identify as EAs who see focusing on global poverty as more or less a waste of time relative to the only thing that matters, which is that the Singularity is coming in about 2-5 years (unless we take drastic action), and all our efforts should be focused on making sure the Singularity goes good and not bad â including trying to delay it if that helps. People who disagree with this view have not yet been fully excluded from EA but it seems like some people are pretty mean to people who disagree. (I am one of the people who disagrees.)
As a side note, itâs also strange to me that people are treating the founding of Mechanize as if it has a realistic chance to accelerate AGI progress more than a negligible amount â enough of a chance of enough of an acceleration to be genuinely concerning. AI startups are created all the time. Some of them state wildly ambitious goals, like Mechanize. They typically fail to achieve these goals. The startup Vicarious comes to mind.
There are many startups trying to automate various kinds of physical and non-physical labour. Some larger companies like Tesla and Alphabet are also working on this. Why would Mechanize be particularly concerning or be particularly likely to succeed?
This isnât just a social thing, itâs also response to a lot of changes in AI timelines over the past ten years. Back then a lot of us had views like âmost experts think powerful AI is far off, Iâm not going to sink a bunch of time into how it might affect my various options for doing goodâ, but as expert views have shifted that makes less sense. While âdonât think much about AGI because global poverty seems a lot more importantâ is still a reasonable position to hold (ex: people who think we canât productively influence how AI goes and so we should focus on doing as much good as we can in areas we can affect), I think it requires a good bit more reasoning and thought than it did ten years ago.
(On the other hand, I see âonly care about people who are either alive today or who will be born in the next 100 yearsâ as still within the range of common EA views (ex).)
I see it primarily as a social phenomenon because I think the evidence we have today that AGI will arrive by 2030 is less compelling than the evidence we had in 2015 that AGI would arrive by 2030. In 2015, it was a little more plausible that AGI could arrive by 2030 because that was 15 years away and who knows what can happen in 15 years.
Now that 2030 is a little less than 5 years away, AGI by 2030 is a less plausible prediction than it was in 2015 because thereâs less time left and itâs more clear it wonât happen.
I donât think the reasons people believe AGI will arrive by 2030 are primarily based on evidence but are primarily a sociological phenomenon. People were ready to believe this regardless of the evidence going back to Ray Kurzweilâs The Age of Spiritual Machines in 1999 and Eliezer Yudkowskyâs âEnd-of-the-World Betâ in 2017. People donât really pay attention to whether the evidence is good or bad, they ignore obvious evidence and arguments against near-term AGI, and they mostly make a choice to ignore or attack people who express disagreement and instead tune into the relentless drumbeat of people agreeing with them. This is sociology, not epistemology.
Donât believe me? Talk to me again in 5 years and send me a fruit basket. (Or just kick the can down the road and say AGI is coming in 2035...)
Expert opinion has changed? First, expert opinion is not itself evidence, itâs peopleâs opinions about evidence. What evidence are the experts basing their beliefs on? That seems way more important than someone just saying a number based on an intuition.
Second, expert opinion does not clearly support the idea of near-term AGI.
As of 2023, the expert opinion on AGI was⊠well, first of all, really confusing. The AI Impacts survey found that the experts believed there is a 50% chance by 2047 that âunaided machines can accomplish every task better and more cheaply than human workers.â And also that thereâs a 50% chance that by 2116 âmachines could be built to carry out the task better and more cheaply than human workers.â I donât know why these predictions are 69 years apart.
Regardless, 2047 is sufficiently far away that it might as well be 2057 or 2067 or 2117. This is just people generating a number using a gut feeling. We donât know how to build AGI and we have no idea how long it will take to figure out how to. No amount of thinking of numbers or saying numbers can escape this fundamental truth.
We actually wonât have to wait long to see that some of the most attention-catching near-term AI predictions are false. Dario Amodei, the CEO of Anthropic (a company that is said to be âliterally creating Godâ), has predicted that by some point between June 2025 and September 2025, 90% of all code will written by AI rather than humans. In late 2025 and early 2026, when itâs clear Dario was wrong about this (when, not if), maybe some people will start to be more skeptical of attention-grabbing expert predictions. But maybe not.
There are already strong signs of AGI discourse being irrational and absurd. On April 16, 2025, Tyler Cowen claimed that OpenAIâs o3 model is AGI and asked, âis April 16th AGI day?â. In a follow-up post on April 17, seemingly in response to criticism, he said, âI donât mind if you donât want to call it AGIâ, but seemed to affirm he still thinks o3 is AGI.
On one hand, I hope that in 5 years the people who promoted the idea of AGI by 2030 will lose a lot of credibility and maybe will do some soul-searching to figure out how they could be so wrong. On the other hand, there is nothing preventing people from being irrational indefinitely, such as:
Defining whatever exists in 2030 as AGI (Tyler Cowen already did it in 2025, and Ray Kurzweil innovated the technique years ago).
Kicking the can down the road a few years, and repeat as necessary (similar to how Elon Musk has predicted that the Tesla fleet will achieve Level 4â5 autonomy in a year every year from 2015 to 2025 and has not given up the game despite his losing streak).
Telling a story in which AGI didnât happen only because effective altruists or other good actors successfully delayed AGI development.
I think part of the sociological problem is that people are just way too polite about how crazy this all is and how awful the intellectual practices of effective altruists have been on this topic. (Sorry!) So, Iâm being blunt about this to try to change that a little.
The evidence we have today that there will be AGI by 2030 is clearly dramatically stronger than the evidence we had in 2015 that there would be AGI by 2020, and that is surely the relevant comparison. This is not EA specificâwe have been ahead of the curve in thinking AI would be a big deal, but the whole world has updated in this direction, and it would be strange if we hadnât as well.
My personal take is that there are pretty reasonable arguments that what we have seen in AI/âML since 2015 suggests AI will be a big deal. I like the way I have seen Yoshua Bengio talk about it âover the next few years, or a few decadesâ. I share the view that either of those possibilities are reasonable. People who are highly confident that something like AGI is going to arrive over the next few years are more confident in this than I am, but I think that view is within the bounds of reasonable interpretation of the evidence. I think it is also with-in the bounds of reasonable to have the opposite view, that something like AGI is most likely further than a few years away.
I think this is a healthy attitude and that I think is worth appreciating. We may get answers to these questions over the next few years. That seems pretty positive to me. We will be able to resolve some of these disagreements productively by observing what happens. I hope people who have different views now keep this in mind and that the environment is still in a good place for people who disagree now to work together in the future if some of these disagreements get resolved.
I will offer the ea forum internet-points equivalent of a fruit basket to anyone who would like one in the future if we disagree now and in the future they are proven right and I am proven wrong.
Can you saw what view it is you think is crazy? It seems quite reasonable to me to think that AI is going to be a massive deal and therefore that it would be highly useful to influence how it goes. On other other hand, I think people often over-estimate the robustness of the arguments for any given strategy for how to actually do that influencing. In other words, its reasonable to prioritize AI, but peopleâs AI takes are often very over-confident.
For what itâs worth, I basically agree with the view that Mechanize is unlikely to be successful at itâs goals:
I appreciate your comment.
I do think this is substantially correct, but I also want to acknowledge that these can be difficult subjects to navigate. I think anyone has done anything wrong, Iâm sure I myself have done something similar to this many times. But I do think its worth trying to understand where the central points of disagreement lie, and I think this really is the central disagreement.
On the question of changing EA attitudes towards AI over the years, although I personally think AI will be a big deal, could be dangerous, and those issues are worth of significant attention, I also can certainly see reasons why people might disagree and why those people would have reasonable grievances with decisions by certain EA people and organizations.
An idea that I have pondered for a while about EA is a theory about which âboundariesâ a community emphasizes. Although Iâve only ever interacted with EA by reading related content online, my perception is that EA really emphasizes the boundary around the EA community itself, while de-emphasizing the boundaries around individual people or organizations. The issues around Epoch I think demonstrate this. The feeling of betrayal comes from viewing âthe communityâ as central. I think a lot of other cultures that place more emphasize on those other boundaries might react differently. For example, at most companies I have worked at, although certainly they would never be happy to see an employee leave, they wouldnât view moving to another job as a betrayal, even if an employee went to work for a direct competitor. I personally think placing more emphasis on orgs/âindividuals rather than the community as a whole could have some benefits, such as with the issue you raise about how to navigate changing views on AI.
Although emphasizing âthe communityâ might seem like its ideal for cooperation, I think it can actually harm cooperation in the presence of substantial disagreements, because it generates dynamics like what is going on here. People feel like they canât cooperate with people across the disagreement. We will probably see some of these disagreements resolved over the next few years as AI progresses. I for one hope that even if I am wrong I can take any necessary corrections on-board and still work with people who I disagreed with to make positive contributions. Likewise, I hope that if I am right, people who I disagreed with still feel like they can work with me despite that.
I admit I had a similar thought, but I am of two minds about it. On the one hand, I think intentions do matter. I think it is reasonable to point out if you think someone is making a mistake, even if you think ultimately that mistake is unlikely to have a substantial impact because the person is unlikely to succeed in what they are trying to do.
On the other hand, I do think the degree of the reaction and the way that people are generalizing seems like people are almost pricing in the idea that the actions in question have already had a huge impact. So I do wonder if people are kind of over-updating on this specific case for similar reasons to what you mention.
Yeah that sounds right to me as a gloss