For P1, I assumed that AGI going well for humans was basically even ASI going well for humans (just: āit happens weāre in a good scenario for the fleshy humansā). I donāt know if ASI is much less likely to go right than AGIāsomething as capable as AGI could already very easily be misaligned and Iām not sure that scales with increase in capabilities.
For P2, Iām scared that we donāt necessarily need the top of the distribution of ASI to do this. I could imagine non-AGI worlds where human-brain-driven technological progress gets us there, though it seems very complicated resource-wise at this stage.
I agree that these two arguments together could undermine my vague āone order of magnitude differenceā claim, but Iām not sure how much I believe them. I do come down to believing that most of my considerations will face counter-considerations which I am currently unaware of.
Fair pushback!
For P1, I assumed that AGI going well for humans was basically even ASI going well for humans (just: āit happens weāre in a good scenario for the fleshy humansā). I donāt know if ASI is much less likely to go right than AGIāsomething as capable as AGI could already very easily be misaligned and Iām not sure that scales with increase in capabilities.
For P2, Iām scared that we donāt necessarily need the top of the distribution of ASI to do this. I could imagine non-AGI worlds where human-brain-driven technological progress gets us there, though it seems very complicated resource-wise at this stage.
I agree that these two arguments together could undermine my vague āone order of magnitude differenceā claim, but Iām not sure how much I believe them. I do come down to believing that most of my considerations will face counter-considerations which I am currently unaware of.