Sophie—thanks for an excellent, important, and timely article.
My main concern here is that a huge influx of ‘technical AI safety research’ money from beneficiaries of an Anthropic IPO would be very biased in favor of pro-AI corporate safety-washing, rather than anti-AI activism.
Funders dispersing post-IPO Anthropic founder/employee funds would be under heavy pressure to buy into what we might call the ‘Anthropic Model of AI safety’, which seems to have a few assumptions that are often implicit, but that are IMHO very misguided and very dangerous, e.g. the assumptions that
ASI (Artificial Superintelligence) alignment is a solvable problem, at least in principle—and indeed, is likely so be solved within a few years, if we can just throw enough talent and money at the problem;
ASI alignment is more likely to be solved the more we advanced AI capabilities, because that gives us more insight into how advanced AI systems work (rather than, by example, funding more research on the game theory issues around ASI vs human goals, incentives, and payoffs, or more research on characterizing the nature and diversity of human morals, values, and goals that ASI would be trying to align with);
extinction risks (e.g. as indexed by p(doom)) can be reduced most effectively by accelerating ‘technical AI safety research’, rather than by anti-AI activism, by severe gov’t regulations and global treaters, and by moral stigmatization of AI founders, researchers, and devs.
If the vast majority of AI safety work and advocacy comes from post-IPO Anthropic money, the likely result is that the entire EA focus on AI safety will get warped into chasing ‘technical AI safety’ jobs and money, rather than fighting the AI industry at the grassroots political level, or policy level, or public moral level. The people like me who see Anthropic as just as reckless and evil as OpenAI, Google, xAI, or DeepSeek, will get squeezed out. People who think that anyone participating in the ‘AI arms race’ is basically endangering our kids for their own greed and hubris will get silenced.
This post-IPO Anthropic money would not get spent on promoting a bipartisan political consensus within the US to shut down all further ASI development. It would not get spent on raising public awareness of AI risks in China, among CCP leaders, influencers, and citizens. Rather, it would get spent on hiring armies of ‘technical AI safety’ grant-recipients to work within or alongside AI companies.
Those ‘AI safety researchers’ will not rock the boat of the AI industry. They will not advocate for the kinds of pauses or shutdowns advocated by Pause AI, Stop AI, or Control AI. They will be easily co-opted by trillion-dollar AI companies into being their safety-washing minions, their PR representatives, and their reassurances to the public that ‘the AI industry is taking safety very seriously indeed’, while it races ahead towards ASI.
That’s my concern. If the ‘Anthropic Model of AI safety’ is simply wrong in major ways—e.g. if ‘ASI alignment’ is not a solvable problem, if ASI alignment isn’t best solved by advancing AI capabilities, and if ‘technical AI safety research’ actually increases p(doom) (e.g. as safety-washing that gives misleading comfort to legislators, citizens, and other nations), then this Anthropic IPO, and resulting distortions of AI safety efforts, could be a disastrous development.
Thanks for your comment! I broadly agree that Anthropic money is probably going to go overwhelmingly towards technical AI safety work-shaped things, which is not great given how neglected political advocacy work is already.
I don’t agree with every premise (I think ASI alignment research is worth pursuing even if we’re uncertain it’s solvable), but, similarly to how we should not let CG’s priorities define the field, I agree we should not let one company’s theory of change become the entire field’s theory of change.
The best mitigation, I think, is to fight for every marginal dollar that doesn’t go to technical safety research. Right now, the c3 research ecosystem is flush with money while c4 advocacy work is still highly funding-constrained. I’ve written a follow-up making the case that individual donors should redirect what would have been c3 donations toward c4 lobbying and advocacy organizations instead: stop donating to AI safety research*
Sophie—thanks for an excellent, important, and timely article.
My main concern here is that a huge influx of ‘technical AI safety research’ money from beneficiaries of an Anthropic IPO would be very biased in favor of pro-AI corporate safety-washing, rather than anti-AI activism.
Funders dispersing post-IPO Anthropic founder/employee funds would be under heavy pressure to buy into what we might call the ‘Anthropic Model of AI safety’, which seems to have a few assumptions that are often implicit, but that are IMHO very misguided and very dangerous, e.g. the assumptions that
ASI (Artificial Superintelligence) alignment is a solvable problem, at least in principle—and indeed, is likely so be solved within a few years, if we can just throw enough talent and money at the problem;
ASI alignment is more likely to be solved the more we advanced AI capabilities, because that gives us more insight into how advanced AI systems work (rather than, by example, funding more research on the game theory issues around ASI vs human goals, incentives, and payoffs, or more research on characterizing the nature and diversity of human morals, values, and goals that ASI would be trying to align with);
extinction risks (e.g. as indexed by p(doom)) can be reduced most effectively by accelerating ‘technical AI safety research’, rather than by anti-AI activism, by severe gov’t regulations and global treaters, and by moral stigmatization of AI founders, researchers, and devs.
If the vast majority of AI safety work and advocacy comes from post-IPO Anthropic money, the likely result is that the entire EA focus on AI safety will get warped into chasing ‘technical AI safety’ jobs and money, rather than fighting the AI industry at the grassroots political level, or policy level, or public moral level. The people like me who see Anthropic as just as reckless and evil as OpenAI, Google, xAI, or DeepSeek, will get squeezed out. People who think that anyone participating in the ‘AI arms race’ is basically endangering our kids for their own greed and hubris will get silenced.
This post-IPO Anthropic money would not get spent on promoting a bipartisan political consensus within the US to shut down all further ASI development. It would not get spent on raising public awareness of AI risks in China, among CCP leaders, influencers, and citizens. Rather, it would get spent on hiring armies of ‘technical AI safety’ grant-recipients to work within or alongside AI companies.
Those ‘AI safety researchers’ will not rock the boat of the AI industry. They will not advocate for the kinds of pauses or shutdowns advocated by Pause AI, Stop AI, or Control AI. They will be easily co-opted by trillion-dollar AI companies into being their safety-washing minions, their PR representatives, and their reassurances to the public that ‘the AI industry is taking safety very seriously indeed’, while it races ahead towards ASI.
That’s my concern. If the ‘Anthropic Model of AI safety’ is simply wrong in major ways—e.g. if ‘ASI alignment’ is not a solvable problem, if ASI alignment isn’t best solved by advancing AI capabilities, and if ‘technical AI safety research’ actually increases p(doom) (e.g. as safety-washing that gives misleading comfort to legislators, citizens, and other nations), then this Anthropic IPO, and resulting distortions of AI safety efforts, could be a disastrous development.
Hi Geoffrey,
Thanks for your comment! I broadly agree that Anthropic money is probably going to go overwhelmingly towards technical AI safety work-shaped things, which is not great given how neglected political advocacy work is already.
I don’t agree with every premise (I think ASI alignment research is worth pursuing even if we’re uncertain it’s solvable), but, similarly to how we should not let CG’s priorities define the field, I agree we should not let one company’s theory of change become the entire field’s theory of change.
The best mitigation, I think, is to fight for every marginal dollar that doesn’t go to technical safety research. Right now, the c3 research ecosystem is flush with money while c4 advocacy work is still highly funding-constrained. I’ve written a follow-up making the case that individual donors should redirect what would have been c3 donations toward c4 lobbying and advocacy organizations instead: stop donating to AI safety research*
Would be curious for your takes on the follow-up!
Sophie—thanks for your reply. I agree. And I strongly upvoted your more recent post that you mentioned, which is excellent.