Scott, thanks so much for this post. It’s been years coming in my opinion. FWIW, my reason for making ARCHES (AI Research Considerations for Human Existential Safety) explicitly about existential risk, and not about “AI safety” or some other glomarization, is that I think x-risk and x-safety are not long-term/far-off concerns that can be procrastinated away.
https://forum.effectivealtruism.org/posts/aYg2ceChLMRbwqkyQ/ai-research-considerations-for-human-existential-safety (with David Krueger)
Ideally, we need to engage as many researchers as possible, thinking about as many aspects of a functioning civilization as possible, to assess how A(G)I can creep into those corners of civilization and pose an x-risk, with cybersecurity / internet infrastructure and social media being extremely vulnerable fronts that are easily salient today.
As I say this, I worry that other EAs will get worried that talking to folks working on cybersecurity or recommender systems necessarily means abandoning existential risk as a priority, because those fields have not historically taken x-risk seriously.
However, for better or for worse, it’s becoming increasingly easy for everyone to imagine cybersecurity and/or propaganda disasters involving very powerful AI systems, such that x-risk is increasingly not-a-stretch-for-the-imagination. So, I’d encourage anyone who feels like “there is no hope to convince [group x] to care” to start re-evaluating that position (e.g., rather than aiming/advocating for drastic interventions like invasive pivotal acts). I can’t tell whether or not you-specifically are in the “there is no point in trying” camp, but others might be, and in any case I thought it might be good to bring up
In summary: as tech gets scarier, we should have some faith that people will be more amenable to arguments that it is in fact dangerous, and re-examine whether this-group or that-group is worth engaging on the topic of existential safety as a near-term priority.
Thanks for calling this out, whoever you are. This is another reason why, for the last ~5 years or so, I’ve been pushing for the terminology “AI x-risk” and “AI x-safety” to replace “(long-term) AI risk” and “(long-term) AI safety”. For audiences not familiar with the “x” or “existential” terminology, one can say “large scale risk” to point the large stakes, rather than always saying “long term”.
(Also, the fact that I don’t know who you are is actually fairly heartening :)