My own take on AI Safety Classic arguments is I’ve become convinced by o3/Sonnet 3.7 that the alignment is very easy hypothesis is looking a lot shakier than it used to be, and I suspect future capabilities progress is likely to be at best neutral, and probably worse for alignment being very easy.
I do think you can still remain optimistic based on other cases, but a pretty core crux is I think alignment does need to be solved if AIs are able to automate the economy, and this is pretty robust to variations on what happens with AI.
The big reason for this is that once your labor is valueless, but your land/capital isn’t, you have fundamentally knocked out a load-bearing pillar of the argument that expropriation is less useful than trade.
This is to a first approximation why we do not trade with most non-human species, rather than enslaving/killing them.
(For farm animals, their labor is useful, but the stuff lots of humans want from animals fundamentally requires expropriation/violating farm animal property rights)
A good scenario for what happens if we fail is at minimum the intelligence curse scenario elaborated on by Rudolf Lane and Luke Drago below:
My own take on AI Safety Classic arguments is I’ve become convinced by o3/Sonnet 3.7 that the alignment is very easy hypothesis is looking a lot shakier than it used to be, and I suspect future capabilities progress is likely to be at best neutral, and probably worse for alignment being very easy.
I do think you can still remain optimistic based on other cases, but a pretty core crux is I think alignment does need to be solved if AIs are able to automate the economy, and this is pretty robust to variations on what happens with AI.
The big reason for this is that once your labor is valueless, but your land/capital isn’t, you have fundamentally knocked out a load-bearing pillar of the argument that expropriation is less useful than trade.
This is to a first approximation why we do not trade with most non-human species, rather than enslaving/killing them.
(For farm animals, their labor is useful, but the stuff lots of humans want from animals fundamentally requires expropriation/violating farm animal property rights)
A good scenario for what happens if we fail is at minimum the intelligence curse scenario elaborated on by Rudolf Lane and Luke Drago below:
https://intelligence-curse.ai/defining/