Some very quick thoughts from EY’s TIME piece from the perspective of someone ~outside of the AI safety work. I have no technical background and don’t follow the field closely, so likely to be missing some context and nuance; happy to hear pushback!
Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
My immediate reaction when reading this was something like “wow, is this representative of AI safety folks? Are they willing to go to any lengths to stop AI development?”. I’ve heard anecdotes of people outside of all this stuff saying this piece reads like a terrorist organisation, for example, which I think is a stronger term than I’d describe, but I think suggestions like this does unfortunately play into potential comparisons to ecofascists.
As someone seen publicly to be a thought leader and widely regarded as a founder of the field, there are some risks to this kind of messaging. It’s hard to evaluate how this trades off, but I definitely know communities and groups that would be pretty put off by this, and it’s unclear how much value the sentences around willingness to escalate nuclear war are actually adding.
It’s an empirical Q about how to tradeoff between risks from nuclear war and risks from AI, but the claim of “preventing AI extinction is a priority above a nuclear exchange” is ~trivially true; the reverse is also true: “preventing extinction from nuclear war is a priority above preventing AI training runs”. Given the difficulty of illustrating and defending a position that the risks of AI training runs is substantially higher than that of a nuclear exchange to the general public, I would have erred on the side of caution when saying things that are as politically charged as advocating for nuclear escalation (or at least something can be interpreted as such).
I wonder which superpower EY trusts to properly identify a hypothetical “rogue datacentre” that’s worthy of a military strike for the good of humanity, or whether this will just end up with parallels to other failed excursions abroad ‘for the greater good’ or to advance individual national interests.
If nuclear weapons are a reasonable comparison, we might expect limitations to end up with a few competing global powers to have access to AI developments, and countries that do not. It seems plausible that criticism around these treaties being used to maintain the status quo in the nuclear nonproliferation / disarmament debate may be applicable here too.
Unlike nuclear weapons (though nuclear power may weaken this somewhat), developments in AI has the potential to help immensely with development and economic growth.
Thus the conversation may eventually bump something that looks like:
Richer countries / first movers that have obtained significant benefits of AI take steps to prevent other countries from catching up.[1]
Rich countries using the excuse of preventing AI extinction as a guise to further national interests
Development opportunities from AI for LMICs are similarly hindered, or only allowed in a way that is approved by the first movers in AI.
Given the above, and that conversations around and tangential to AI risk already receive some pushback from the Global South community for distracting and taking resources away from existing commitments to UN Development Goals, my sense is that folks working in AI governance / policy would likely strongly benefit from scoping out how these developments are affecting Global South stakeholders, and how to get their buy-in for such measures.
(disclaimer: one thing this gestures at is something like—“global health / development efforts can be instrumentally useful towards achieving longtermist goals”[2], which is something I’m clearly interested in as someone working in global health. While it seems rather unlikely that doing so is the best way of achieving longtermist goals on the margin[3], it doesn’t exclude some aspect of this in being part of a necessary condition for important wins like an international treaty, if that’s what is currently being advocated for. It is also worth mentioning because I think this is likely to be a gap / weakness in existing EA approaches).
this is applicable to a weaker extent even in the event that international agreements on indefinite moratorium on new large training runs passes, if you see AI as a potential equalising force or if you think first movers might be worried about this
Some very quick thoughts from EY’s TIME piece from the perspective of someone ~outside of the AI safety work. I have no technical background and don’t follow the field closely, so likely to be missing some context and nuance; happy to hear pushback!
My immediate reaction when reading this was something like “wow, is this representative of AI safety folks? Are they willing to go to any lengths to stop AI development?”. I’ve heard anecdotes of people outside of all this stuff saying this piece reads like a terrorist organisation, for example, which I think is a stronger term than I’d describe, but I think suggestions like this does unfortunately play into potential comparisons to ecofascists.
As someone seen publicly to be a thought leader and widely regarded as a founder of the field, there are some risks to this kind of messaging. It’s hard to evaluate how this trades off, but I definitely know communities and groups that would be pretty put off by this, and it’s unclear how much value the sentences around willingness to escalate nuclear war are actually adding.
It’s an empirical Q about how to tradeoff between risks from nuclear war and risks from AI, but the claim of “preventing AI extinction is a priority above a nuclear exchange” is ~trivially true; the reverse is also true: “preventing extinction from nuclear war is a priority above preventing AI training runs”. Given the difficulty of illustrating and defending a position that the risks of AI training runs is substantially higher than that of a nuclear exchange to the general public, I would have erred on the side of caution when saying things that are as politically charged as advocating for nuclear escalation (or at least something can be interpreted as such).
I wonder which superpower EY trusts to properly identify a hypothetical “rogue datacentre” that’s worthy of a military strike for the good of humanity, or whether this will just end up with parallels to other failed excursions abroad ‘for the greater good’ or to advance individual national interests.
If nuclear weapons are a reasonable comparison, we might expect limitations to end up with a few competing global powers to have access to AI developments, and countries that do not. It seems plausible that criticism around these treaties being used to maintain the status quo in the nuclear nonproliferation / disarmament debate may be applicable here too.
Unlike nuclear weapons (though nuclear power may weaken this somewhat), developments in AI has the potential to help immensely with development and economic growth.
Thus the conversation may eventually bump something that looks like:
Richer countries / first movers that have obtained significant benefits of AI take steps to prevent other countries from catching up.[1]
Rich countries using the excuse of preventing AI extinction as a guise to further national interests
Development opportunities from AI for LMICs are similarly hindered, or only allowed in a way that is approved by the first movers in AI.
Given the above, and that conversations around and tangential to AI risk already receive some pushback from the Global South community for distracting and taking resources away from existing commitments to UN Development Goals, my sense is that folks working in AI governance / policy would likely strongly benefit from scoping out how these developments are affecting Global South stakeholders, and how to get their buy-in for such measures.
(disclaimer: one thing this gestures at is something like—“global health / development efforts can be instrumentally useful towards achieving longtermist goals”[2], which is something I’m clearly interested in as someone working in global health. While it seems rather unlikely that doing so is the best way of achieving longtermist goals on the margin[3], it doesn’t exclude some aspect of this in being part of a necessary condition for important wins like an international treaty, if that’s what is currently being advocated for. It is also worth mentioning because I think this is likely to be a gap / weakness in existing EA approaches).
this is applicable to a weaker extent even in the event that international agreements on indefinite moratorium on new large training runs passes, if you see AI as a potential equalising force or if you think first movers might be worried about this
Egoistic Altruism
Beware surprising and suspicious convergence