Well-understood goals in agents that gain power and take over the lightcone is exactly the thing we’d be addressing with AI alignment, so this seems like an argument for investing in AI alignment—which I think most people would see as far closer to preventing existential risk.
That said, without a lot more progress, powerful agents with simple goals is actually just a fancy way of guaranteeing of a really bad outcome, almost certainly including human extinction.
Well-understood goals in agents that gain power and take over the lightcone is exactly the thing we’d be addressing with AI alignment, so this seems like an argument for investing in AI alignment—which I think most people would see as far closer to preventing existential risk.
That said, without a lot more progress, powerful agents with simple goals is actually just a fancy way of guaranteeing of a really bad outcome, almost certainly including human extinction.