I’m not sure I feel as concerned about this as others. tl;dr—They have different beliefs from Safety-concerned EAs, and their actions are a reflection of those beliefs.
It seems broadly bad that the alumni from a safety-focused AI org
Was Epoch ever a ‘safety-focused’ org? I thought they were trying to understand what’s happening with AI, not taking a position on Safety per se.
…have left to form a company which accelerates AI timelines
I think Matthew and Tamay think this is positive, since they think AI is positive. As they say, they think explosive growth can be translated into abundance. They don’t think that the case for AI risk is strong, or significant, especially given the opportunity cost they see from leaving abundance on the table.
As I learned more and the situation unfolded I have become more skeptical of AI Risk.
The same thing seems to be happening with me, for what it’s worth.
People seem to think that there is an ‘EA Orthodoxy’ on this stuff, but there either isn’t as much as people think, or people who disagree with it are no longer EAs. I really don’t think it makes sense to clamp down on ‘doing anything to progress AI’ as being a hill for EA to die on.
(1) The epistemic community around EA, rationality, and AI safety, should stay open to criticism of key empirical assumptions (like the level of risks from AI, risks of misalignments, etc.) in a healthy way.
(2) We should still condemn people who adopt contrarian takes with unreasonable-seeming levels of confidence and then take actions based on them that we think are likely doing damage.
In addition, there’s possibly also a question of “how much do people who benefit from AI safety funding and AI safety association have an obligation to not take unilateral actions that most of the informed people in the community consider negative.” (FWIW I don’t think the obligation here would be absolute even if Epoch had been branded as centrally ‘AI safety,’ and I acknowledge that the branding issue seems contested; also, it wasn’t Jamie [edit: Jaime] the founder who left in this way, and of the people who went off to found this new org, Matthew Barnett, for instance, has been really open about his contrarian takes, so insofar as Epoch’s funders had concerns about the alignment of employees at Epoch, it was also—to some degree, at least—on them to ask for more information or demand some kind of security guarantee if they felt worried. And maybe this did happen—I’m just flagging that I don’t feel like we onlookers necessarily have the info, and so it’s not clear whether anyone has violated norms of social cooperation here or whether we’re just dealing with people getting close to the boundaries of unilateral action in a way that is still defensible because they’ve never claimed to be more aligned than they were, never accepted funding that came with specific explicit assumptions, etc.)
or whether we’re just dealing with people getting close to the boundaries of unilateral action in a way that is still defensible because they’ve never claimed to be more aligned than they were, never accepted funding that came with specific explicit assumptions, etc.)
Caveats up front: I note the complexity of figuring out what Epoch’s own views are, as opposed to Jaime’s [corrected spelling] view or the views of the departing employees. I also do not know what representations were made. Therefore, I am not asserting that Epoch did something or needs to do something, merely that the concern described below should be evaluated.
People and organizations change their opinions all the time. One thing I’m unclear on is whether there was a change in position here should that created an obligation to offer to return and/or redistribute unused donor funds.
I note that, in February 2023, Epoch was fundraising through September 2025. I don’t know its cash flows, but I cite that to show it is plausible they were operating on safety-focused money obtained before a material change to less safety-focused views. In other words, the representations to donors may have been appropriate when the money was raised but outdated by the time it was spent.
I think it’s fair to ask whether a donor would have funded a longish runway if it had known the organization’s views would change by the time the monies were spent. If the answer is “no,” that raises the possibility that the organization may be ethically obliged to refund or regrant the unspent grant monies.
I can imagine circumstances in which the answers are no and yes: for instance, suppose the organization was a progressive political advocacy organization that decided to go moderate left instead. It generally will not be appropriate for that org to use progressives’ money to further its new stance. On the other hand, any shift here was less pronounced, and there’s a stronger argument that the donors got the forecasting/information outputs they paid for.
Anyway, for me all this ties into post-FTX discussions about giving organizations a healthy financial runway. People in those discussions did a good job flagging the downsides of short-term grants without confidence in renewal, as well as the high degree of power funders hold in the ecosystem. But AI is moving fast; this isn’t something more stable like anti-malarial work. So the chance of organizational drift seems considerably higher here.
How do we deal with the possibility that honest organizational changes will create a inconsistency with the implicit donor-recipient understanding at the time of grant? I don’t claim to have the answer, or how to apply it here.
By the way, the name is ‘Jaime’, not ‘Jamie’. The latter doesn’t exist in Spanish and the two are pronounced completely differently (they share one phoneme out of five, when aligned phoneme by phoneme).
(I thought I should mention it since the two names often look indistinguishable in written form to people who are not aware that they differ.)
How common is it for such repayments to occur, and what do you think would be the standard for the level of clarity of the commitment, and who does that commitment would have to be to? For example, is there a case that 80k hours should refund payments in light of their pivot to focus on AI? I know there are differences, their funder could support the move etc., but in the spirit of the thing, where is the line here?
Editing to add: One of my interests in this topic is that EA/rationalists seem to have some standards/views that diverge somewhat from what I would characterize as more “mainstream” approaches to these kinds of things. Re-reading the OP, I noticed a detail I initially missed:
I think Matthew and Tamay think this is positive, since they think AI is positive.
I don’t see how this alleviates concern. Sure they’re acting consistently with their beliefs*, but that doesn’t change the fact that what they’re doing is bad.
I don’t mean to suggest that any one of these possibilities is particularly likely, or they they are all plausible. I haven’t followed this incident closely. FWIW, my vague sense is that the Mechanize founders had all expressed skepticism about the standard AI safety arguments for a while, in a way that seems hard to reconcile with (1) or (2).
it suggests the concern is an object level one, not a meta one. the underlying “vibe” I am getting from a lot of these discussions is that the people in question have somehow betrayed EA/the community/something else. That is a meta concern, one of norms. You could “betray” the community even if you are on the AI deceleration side things. If the people in question or Epoch made a specific commitment that they violated, that would be a “meta” issue, and would be one regardless of their “side” on the deceleration question. Perhaps they did do such a thing, but I haven’t seen convincing information suggesting this. I think that really the main explanatory variable here is in fact what “side” this suggests they are on. If that is the case, I think it is worth having clarity about it. People can do a bad thing because they are just wrong in their analysis of a situation or their decision-making. That doesn’t mean their actions constitute a betrayal.
Note—this was written kinda quickly, so might be a bit less tactful than I would write if I had more time.
Making a quick reply here after binge listening to three Epoch-related podcasts in the last week, and I basically think my original perspective was vindicated. It was kinda interesting to see which points were repeated or phrased a different way—would recommend if your interested in the topic.
The initial podcast with Jaime, Ege, and Tamay. This clearly positions the Epoch brain trust as between traditional academia and the AI Safety community (AISC). tl;dr—academia has good models but doesn’t take ai seriously, and AISC the opposite (from Epoch’s PoV)
The ‘debate’ between Matthew and Ege. This should have clued people in, because while full of good content, by the last hour/hour and half it almost seemed to turn into ‘openly mocking and laughing’ at AISC, or at least the traditional arguments. I also don’t buy those arguments, but I feel like the reaction Matthew/Ege have shows that they just don’t buy the root AISC claims.
The recent podcast Dwarkesh with Ege & Tamay. This is the best of the 3, but probably also best listened too after the first too, since Dwarkesh actually pushes back on quite a few claims, which means Ege & Tamay flush out their views more—personal highlight was what the reference class for AI Takeover actually means.
Basically, the Mechanize cofounders don’t agree at all with ‘AI Safety Classic’, I am very confident that they don’t buy the arguments at all, that they don’t identify with the community, and somewhat confident that they don’t respect the community or its intellectual output that much.
Given that their views are: a) AI will be a big deal soon (~a few decades), b) returns to AI will be very large, c) Alignment concerns/AI risks are overrated, and d) Other people/institutions aren’t on the ball, then starting an AI Start-up seems to make sense.
What is interesting to note, and one I might look into in the future, is just how much these differences in expectation of AI depend on differences in worldview, rather than differences in technical understanding of ML or understanding of how the systems work on a technical level.
So why are people upset?
Maybe they thought the Epoch people were more part of the AISC than they actually were? Seems like the fault of the people believe this, not Epoch or the Mechanize founders.
Maybe people are upset that Epoch was funded by OpenPhil, and this seems to have lead to ‘AI acceleration’? I think that’s plausible, but Epoch has still produced high-quality reports and information, which OP presumably wanted them to do. But I don’t think equating EA == OP, or anyone funded by OP, is a useful concept to me.
Maybe people are upset at any progress in AI capabilities. But that assumes that Mechanize will be successful in its aims, not guaranteed. It also seems to reify the concept of ‘capabilities’ as one big thing which i don’t think makes sense. Making a better Stockfish, or a better AI for FromSoft bosses does not increase x-risk, for instance.
Maybe people think that the AI Safety Classic arguments are just correct and therefore people taking actions other than it. But then many actions seem bad by this criteria all the time, so odd this would provoke such a reaction. I also don’t think EA should hang its hat on ‘AI Safety Classic’ arguments being correct anyway.
Probably some mix of it. I personally remain not that upset because a) I didn’t really class Epoch as ‘part of the community’, b) I’m not really sure I’m ‘part of the community’ either and c) my views are at least somewhat similar to the Epoch set above, though maybe not as far in their direction, so I’m not as concerned object-level either.
Even assuming OP funding != EA, one still might consider OP funding to count as funding from the AI Safety Club (TM), and for the Mechanize critics to be speaking in their capacity as members of the AISC rather than of EA. Being upset that AISC money supported development of people who are now working to accelerate AI seems understandable to me.
Epoch fundraised on the Forum in early 2023 and solicited applications for employment on the Forum as recently as December 2024. Although I don’t see any specific references to the AISC in those posts, it wouldn’t be unreasonable to assume some degree of alignment from its posting of fundraising and recruitment asks on the Forum without any disclaimer. (However, I haven’t heard a good reason to impute Epoch’s actions to the Mechanize trio specifically.)
My own take on AI Safety Classic arguments is I’ve become convinced by o3/Sonnet 3.7 that the alignment is very easy hypothesis is looking a lot shakier than it used to be, and I suspect future capabilities progress is likely to be at best neutral, and probably worse for alignment being very easy.
I do think you can still remain optimistic based on other cases, but a pretty core crux is I think alignment does need to be solved if AIs are able to automate the economy, and this is pretty robust to variations on what happens with AI.
The big reason for this is that once your labor is valueless, but your land/capital isn’t, you have fundamentally knocked out a load-bearing pillar of the argument that expropriation is less useful than trade.
This is to a first approximation why we do not trade with most non-human species, rather than enslaving/killing them.
(For farm animals, their labor is useful, but the stuff lots of humans want from animals fundamentally requires expropriation/violating farm animal property rights)
A good scenario for what happens if we fail is at minimum the intelligence curse scenario elaborated on by Rudolf Lane and Luke Drago below:
I’m not sure I feel as concerned about this as others. tl;dr—They have different beliefs from Safety-concerned EAs, and their actions are a reflection of those beliefs.
Was Epoch ever a ‘safety-focused’ org? I thought they were trying to understand what’s happening with AI, not taking a position on Safety per se.
I think Matthew and Tamay think this is positive, since they think AI is positive. As they say, they think explosive growth can be translated into abundance. They don’t think that the case for AI risk is strong, or significant, especially given the opportunity cost they see from leaving abundance on the table.
Also important to note is what Epoch boss Jaime says in this very comment thread.
The same thing seems to be happening with me, for what it’s worth.
People seem to think that there is an ‘EA Orthodoxy’ on this stuff, but there either isn’t as much as people think, or people who disagree with it are no longer EAs. I really don’t think it makes sense to clamp down on ‘doing anything to progress AI’ as being a hill for EA to die on.
I think there are two competing failure modes:
(1) The epistemic community around EA, rationality, and AI safety, should stay open to criticism of key empirical assumptions (like the level of risks from AI, risks of misalignments, etc.) in a healthy way.
(2) We should still condemn people who adopt contrarian takes with unreasonable-seeming levels of confidence and then take actions based on them that we think are likely doing damage.
In addition, there’s possibly also a question of “how much do people who benefit from AI safety funding and AI safety association have an obligation to not take unilateral actions that most of the informed people in the community consider negative.” (FWIW I don’t think the obligation here would be absolute even if Epoch had been branded as centrally ‘AI safety,’ and I acknowledge that the branding issue seems contested; also, it wasn’t Jamie [edit: Jaime] the founder who left in this way, and of the people who went off to found this new org, Matthew Barnett, for instance, has been really open about his contrarian takes, so insofar as Epoch’s funders had concerns about the alignment of employees at Epoch, it was also—to some degree, at least—on them to ask for more information or demand some kind of security guarantee if they felt worried. And maybe this did happen—I’m just flagging that I don’t feel like we onlookers necessarily have the info, and so it’s not clear whether anyone has violated norms of social cooperation here or whether we’re just dealing with people getting close to the boundaries of unilateral action in a way that is still defensible because they’ve never claimed to be more aligned than they were, never accepted funding that came with specific explicit assumptions, etc.)
Caveats up front: I note the complexity of figuring out what Epoch’s own views are, as opposed to Jaime’s [corrected spelling] view or the views of the departing employees. I also do not know what representations were made. Therefore, I am not asserting that Epoch did something or needs to do something, merely that the concern described below should be evaluated.
People and organizations change their opinions all the time. One thing I’m unclear on is whether there was a change in position here should that created an obligation to offer to return and/or redistribute unused donor funds.
I note that, in February 2023, Epoch was fundraising through September 2025. I don’t know its cash flows, but I cite that to show it is plausible they were operating on safety-focused money obtained before a material change to less safety-focused views. In other words, the representations to donors may have been appropriate when the money was raised but outdated by the time it was spent.
I think it’s fair to ask whether a donor would have funded a longish runway if it had known the organization’s views would change by the time the monies were spent. If the answer is “no,” that raises the possibility that the organization may be ethically obliged to refund or regrant the unspent grant monies.
I can imagine circumstances in which the answers are no and yes: for instance, suppose the organization was a progressive political advocacy organization that decided to go moderate left instead. It generally will not be appropriate for that org to use progressives’ money to further its new stance. On the other hand, any shift here was less pronounced, and there’s a stronger argument that the donors got the forecasting/information outputs they paid for.
Anyway, for me all this ties into post-FTX discussions about giving organizations a healthy financial runway. People in those discussions did a good job flagging the downsides of short-term grants without confidence in renewal, as well as the high degree of power funders hold in the ecosystem. But AI is moving fast; this isn’t something more stable like anti-malarial work. So the chance of organizational drift seems considerably higher here.
How do we deal with the possibility that honest organizational changes will create a inconsistency with the implicit donor-recipient understanding at the time of grant? I don’t claim to have the answer, or how to apply it here.
By the way, the name is ‘Jaime’, not ‘Jamie’. The latter doesn’t exist in Spanish and the two are pronounced completely differently (they share one phoneme out of five, when aligned phoneme by phoneme).
(I thought I should mention it since the two names often look indistinguishable in written form to people who are not aware that they differ.)
Thank you Pablo for defending the integrity of my name—literally 😆
How common is it for such repayments to occur, and what do you think would be the standard for the level of clarity of the commitment, and who does that commitment would have to be to? For example, is there a case that 80k hours should refund payments in light of their pivot to focus on AI? I know there are differences, their funder could support the move etc., but in the spirit of the thing, where is the line here?
Editing to add: One of my interests in this topic is that EA/rationalists seem to have some standards/views that diverge somewhat from what I would characterize as more “mainstream” approaches to these kinds of things. Re-reading the OP, I noticed a detail I initially missed:
to me this does seem like it implicates a more mainstream view of a potential conflict-of-interest.
I don’t see how this alleviates concern. Sure they’re acting consistently with their beliefs*, but that doesn’t change the fact that what they’re doing is bad.
*I assume, I don’t really know
Intuitively, it seems we should respond differently depending on which of these three possibilities is true:
They think that what they are doing is negative for the world, but do it anyway, because it is good for themselves personally.
They do not think that what they are doing is negative for the world, but they believe this due to motivated cognition.
They do not think that what they are doing is negative for the world, and this belief was not formed in a way that seems suspect.
From an act consequentialist perspective, these differences do not matter intrinsically, but they are still instrumentally relevant.[1]
I don’t mean to suggest that any one of these possibilities is particularly likely, or they they are all plausible. I haven’t followed this incident closely. FWIW, my vague sense is that the Mechanize founders had all expressed skepticism about the standard AI safety arguments for a while, in a way that seems hard to reconcile with (1) or (2).
it suggests the concern is an object level one, not a meta one. the underlying “vibe” I am getting from a lot of these discussions is that the people in question have somehow betrayed EA/the community/something else. That is a meta concern, one of norms. You could “betray” the community even if you are on the AI deceleration side things. If the people in question or Epoch made a specific commitment that they violated, that would be a “meta” issue, and would be one regardless of their “side” on the deceleration question. Perhaps they did do such a thing, but I haven’t seen convincing information suggesting this. I think that really the main explanatory variable here is in fact what “side” this suggests they are on. If that is the case, I think it is worth having clarity about it. People can do a bad thing because they are just wrong in their analysis of a situation or their decision-making. That doesn’t mean their actions constitute a betrayal.
Note—this was written kinda quickly, so might be a bit less tactful than I would write if I had more time.
Making a quick reply here after binge listening to three Epoch-related podcasts in the last week, and I basically think my original perspective was vindicated. It was kinda interesting to see which points were repeated or phrased a different way—would recommend if your interested in the topic.
The initial podcast with Jaime, Ege, and Tamay. This clearly positions the Epoch brain trust as between traditional academia and the AI Safety community (AISC). tl;dr—academia has good models but doesn’t take ai seriously, and AISC the opposite (from Epoch’s PoV)
The ‘debate’ between Matthew and Ege. This should have clued people in, because while full of good content, by the last hour/hour and half it almost seemed to turn into ‘openly mocking and laughing’ at AISC, or at least the traditional arguments. I also don’t buy those arguments, but I feel like the reaction Matthew/Ege have shows that they just don’t buy the root AISC claims.
The recent podcast Dwarkesh with Ege & Tamay. This is the best of the 3, but probably also best listened too after the first too, since Dwarkesh actually pushes back on quite a few claims, which means Ege & Tamay flush out their views more—personal highlight was what the reference class for AI Takeover actually means.
Basically, the Mechanize cofounders don’t agree at all with ‘AI Safety Classic’, I am very confident that they don’t buy the arguments at all, that they don’t identify with the community, and somewhat confident that they don’t respect the community or its intellectual output that much.
Given that their views are: a) AI will be a big deal soon (~a few decades), b) returns to AI will be very large, c) Alignment concerns/AI risks are overrated, and d) Other people/institutions aren’t on the ball, then starting an AI Start-up seems to make sense.
What is interesting to note, and one I might look into in the future, is just how much these differences in expectation of AI depend on differences in worldview, rather than differences in technical understanding of ML or understanding of how the systems work on a technical level.
So why are people upset?
Maybe they thought the Epoch people were more part of the AISC than they actually were? Seems like the fault of the people believe this, not Epoch or the Mechanize founders.
Maybe people are upset that Epoch was funded by OpenPhil, and this seems to have lead to ‘AI acceleration’? I think that’s plausible, but Epoch has still produced high-quality reports and information, which OP presumably wanted them to do. But I don’t think equating EA == OP, or anyone funded by OP, is a useful concept to me.
Maybe people are upset at any progress in AI capabilities. But that assumes that Mechanize will be successful in its aims, not guaranteed. It also seems to reify the concept of ‘capabilities’ as one big thing which i don’t think makes sense. Making a better Stockfish, or a better AI for FromSoft bosses does not increase x-risk, for instance.
Maybe people think that the AI Safety Classic arguments are just correct and therefore people taking actions other than it. But then many actions seem bad by this criteria all the time, so odd this would provoke such a reaction. I also don’t think EA should hang its hat on ‘AI Safety Classic’ arguments being correct anyway.
Probably some mix of it. I personally remain not that upset because a) I didn’t really class Epoch as ‘part of the community’, b) I’m not really sure I’m ‘part of the community’ either and c) my views are at least somewhat similar to the Epoch set above, though maybe not as far in their direction, so I’m not as concerned object-level either.
To steelman this:
Even assuming OP funding != EA, one still might consider OP funding to count as funding from the AI Safety Club (TM), and for the Mechanize critics to be speaking in their capacity as members of the AISC rather than of EA. Being upset that AISC money supported development of people who are now working to accelerate AI seems understandable to me.
Epoch fundraised on the Forum in early 2023 and solicited applications for employment on the Forum as recently as December 2024. Although I don’t see any specific references to the AISC in those posts, it wouldn’t be unreasonable to assume some degree of alignment from its posting of fundraising and recruitment asks on the Forum without any disclaimer. (However, I haven’t heard a good reason to impute Epoch’s actions to the Mechanize trio specifically.)
My own take on AI Safety Classic arguments is I’ve become convinced by o3/Sonnet 3.7 that the alignment is very easy hypothesis is looking a lot shakier than it used to be, and I suspect future capabilities progress is likely to be at best neutral, and probably worse for alignment being very easy.
I do think you can still remain optimistic based on other cases, but a pretty core crux is I think alignment does need to be solved if AIs are able to automate the economy, and this is pretty robust to variations on what happens with AI.
The big reason for this is that once your labor is valueless, but your land/capital isn’t, you have fundamentally knocked out a load-bearing pillar of the argument that expropriation is less useful than trade.
This is to a first approximation why we do not trade with most non-human species, rather than enslaving/killing them.
(For farm animals, their labor is useful, but the stuff lots of humans want from animals fundamentally requires expropriation/violating farm animal property rights)
A good scenario for what happens if we fail is at minimum the intelligence curse scenario elaborated on by Rudolf Lane and Luke Drago below:
https://intelligence-curse.ai/defining/
Well, there kind of is. Maybe you think it’s incorrect, but that’s a separate matter.