(1) The epistemic community around EA, rationality, and AI safety, should stay open to criticism of key empirical assumptions (like the level of risks from AI, risks of misalignments, etc.) in a healthy way.
(2) We should still condemn people who adopt contrarian takes with unreasonable-seeming levels of confidence and then take actions based on them that we think are likely doing damage.
In addition, there’s possibly also a question of “how much do people who benefit from AI safety funding and AI safety association have an obligation to not take unilateral actions that most of the informed people in the community consider negative.” (FWIW I don’t think the obligation here would be absolute even if Epoch had been branded as centrally ‘AI safety,’ and I acknowledge that the branding issue seems contested; also, it wasn’t Jamie [edit: Jaime] the founder who left in this way, and of the people who went off to found this new org, Matthew Barnett, for instance, has been really open about his contrarian takes, so insofar as Epoch’s funders had concerns about the alignment of employees at Epoch, it was also—to some degree, at least—on them to ask for more information or demand some kind of security guarantee if they felt worried. And maybe this did happen—I’m just flagging that I don’t feel like we onlookers necessarily have the info, and so it’s not clear whether anyone has violated norms of social cooperation here or whether we’re just dealing with people getting close to the boundaries of unilateral action in a way that is still defensible because they’ve never claimed to be more aligned than they were, never accepted funding that came with specific explicit assumptions, etc.)
or whether we’re just dealing with people getting close to the boundaries of unilateral action in a way that is still defensible because they’ve never claimed to be more aligned than they were, never accepted funding that came with specific explicit assumptions, etc.)
Caveats up front: I note the complexity of figuring out what Epoch’s own views are, as opposed to Jaime’s [corrected spelling] view or the views of the departing employees. I also do not know what representations were made. Therefore, I am not asserting that Epoch did something or needs to do something, merely that the concern described below should be evaluated.
People and organizations change their opinions all the time. One thing I’m unclear on is whether there was a change in position here should that created an obligation to offer to return and/or redistribute unused donor funds.
I note that, in February 2023, Epoch was fundraising through September 2025. I don’t know its cash flows, but I cite that to show it is plausible they were operating on safety-focused money obtained before a material change to less safety-focused views. In other words, the representations to donors may have been appropriate when the money was raised but outdated by the time it was spent.
I think it’s fair to ask whether a donor would have funded a longish runway if it had known the organization’s views would change by the time the monies were spent. If the answer is “no,” that raises the possibility that the organization may be ethically obliged to refund or regrant the unspent grant monies.
I can imagine circumstances in which the answers are no and yes: for instance, suppose the organization was a progressive political advocacy organization that decided to go moderate left instead. It generally will not be appropriate for that org to use progressives’ money to further its new stance. On the other hand, any shift here was less pronounced, and there’s a stronger argument that the donors got the forecasting/information outputs they paid for.
Anyway, for me all this ties into post-FTX discussions about giving organizations a healthy financial runway. People in those discussions did a good job flagging the downsides of short-term grants without confidence in renewal, as well as the high degree of power funders hold in the ecosystem. But AI is moving fast; this isn’t something more stable like anti-malarial work. So the chance of organizational drift seems considerably higher here.
How do we deal with the possibility that honest organizational changes will create a inconsistency with the implicit donor-recipient understanding at the time of grant? I don’t claim to have the answer, or how to apply it here.
By the way, the name is ‘Jaime’, not ‘Jamie’. The latter doesn’t exist in Spanish and the two are pronounced completely differently (they share one phoneme out of five, when aligned phoneme by phoneme).
(I thought I should mention it since the two names often look indistinguishable in written form to people who are not aware that they differ.)
How common is it for such repayments to occur, and what do you think would be the standard for the level of clarity of the commitment, and who does that commitment would have to be to? For example, is there a case that 80k hours should refund payments in light of their pivot to focus on AI? I know there are differences, their funder could support the move etc., but in the spirit of the thing, where is the line here?
Editing to add: One of my interests in this topic is that EA/rationalists seem to have some standards/views that diverge somewhat from what I would characterize as more “mainstream” approaches to these kinds of things. Re-reading the OP, I noticed a detail I initially missed:
I think there are two competing failure modes:
(1) The epistemic community around EA, rationality, and AI safety, should stay open to criticism of key empirical assumptions (like the level of risks from AI, risks of misalignments, etc.) in a healthy way.
(2) We should still condemn people who adopt contrarian takes with unreasonable-seeming levels of confidence and then take actions based on them that we think are likely doing damage.
In addition, there’s possibly also a question of “how much do people who benefit from AI safety funding and AI safety association have an obligation to not take unilateral actions that most of the informed people in the community consider negative.” (FWIW I don’t think the obligation here would be absolute even if Epoch had been branded as centrally ‘AI safety,’ and I acknowledge that the branding issue seems contested; also, it wasn’t Jamie [edit: Jaime] the founder who left in this way, and of the people who went off to found this new org, Matthew Barnett, for instance, has been really open about his contrarian takes, so insofar as Epoch’s funders had concerns about the alignment of employees at Epoch, it was also—to some degree, at least—on them to ask for more information or demand some kind of security guarantee if they felt worried. And maybe this did happen—I’m just flagging that I don’t feel like we onlookers necessarily have the info, and so it’s not clear whether anyone has violated norms of social cooperation here or whether we’re just dealing with people getting close to the boundaries of unilateral action in a way that is still defensible because they’ve never claimed to be more aligned than they were, never accepted funding that came with specific explicit assumptions, etc.)
Caveats up front: I note the complexity of figuring out what Epoch’s own views are, as opposed to Jaime’s [corrected spelling] view or the views of the departing employees. I also do not know what representations were made. Therefore, I am not asserting that Epoch did something or needs to do something, merely that the concern described below should be evaluated.
People and organizations change their opinions all the time. One thing I’m unclear on is whether there was a change in position here should that created an obligation to offer to return and/or redistribute unused donor funds.
I note that, in February 2023, Epoch was fundraising through September 2025. I don’t know its cash flows, but I cite that to show it is plausible they were operating on safety-focused money obtained before a material change to less safety-focused views. In other words, the representations to donors may have been appropriate when the money was raised but outdated by the time it was spent.
I think it’s fair to ask whether a donor would have funded a longish runway if it had known the organization’s views would change by the time the monies were spent. If the answer is “no,” that raises the possibility that the organization may be ethically obliged to refund or regrant the unspent grant monies.
I can imagine circumstances in which the answers are no and yes: for instance, suppose the organization was a progressive political advocacy organization that decided to go moderate left instead. It generally will not be appropriate for that org to use progressives’ money to further its new stance. On the other hand, any shift here was less pronounced, and there’s a stronger argument that the donors got the forecasting/information outputs they paid for.
Anyway, for me all this ties into post-FTX discussions about giving organizations a healthy financial runway. People in those discussions did a good job flagging the downsides of short-term grants without confidence in renewal, as well as the high degree of power funders hold in the ecosystem. But AI is moving fast; this isn’t something more stable like anti-malarial work. So the chance of organizational drift seems considerably higher here.
How do we deal with the possibility that honest organizational changes will create a inconsistency with the implicit donor-recipient understanding at the time of grant? I don’t claim to have the answer, or how to apply it here.
By the way, the name is ‘Jaime’, not ‘Jamie’. The latter doesn’t exist in Spanish and the two are pronounced completely differently (they share one phoneme out of five, when aligned phoneme by phoneme).
(I thought I should mention it since the two names often look indistinguishable in written form to people who are not aware that they differ.)
Thank you Pablo for defending the integrity of my name—literally 😆
How common is it for such repayments to occur, and what do you think would be the standard for the level of clarity of the commitment, and who does that commitment would have to be to? For example, is there a case that 80k hours should refund payments in light of their pivot to focus on AI? I know there are differences, their funder could support the move etc., but in the spirit of the thing, where is the line here?
Editing to add: One of my interests in this topic is that EA/rationalists seem to have some standards/views that diverge somewhat from what I would characterize as more “mainstream” approaches to these kinds of things. Re-reading the OP, I noticed a detail I initially missed:
to me this does seem like it implicates a more mainstream view of a potential conflict-of-interest.