AGI, whether rogue or human-aligned, may not decide to keep other planets free of biological animals (though it seems like a bigger risk for human-aligned AGI)
This is a really interesting point that I hadnāt thought of before.
Very lightly held counterargument to your conclusion:
P1: The more capable an AGI system is, the harder it is to align.
P2: Terraforming other planets requires AGI at the very top of the capability distribution.
P3: The pool of systems capable of terraforming is therefore drawn disproportionately from the capability range where misalignment is most likely.
Conclusion: Most worlds containing planet-terraforming AGI are probably rogue-AGI worlds. So the āspreading wild animal suffering to new planetsā scenario may be more associated with alignment failure than alignment success.
Corollary: If you agree you should be mildly agree-voting.
For P1, I assumed that AGI going well for humans was basically even ASI going well for humans (just: āit happens weāre in a good scenario for the fleshy humansā). I donāt know if ASI is much less likely to go right than AGIāsomething as capable as AGI could already very easily be misaligned and Iām not sure that scales with increase in capabilities.
For P2, Iām scared that we donāt necessarily need the top of the distribution of ASI to do this. I could imagine non-AGI worlds where human-brain-driven technological progress gets us there, though it seems very complicated resource-wise at this stage.
I agree that these two arguments together could undermine my vague āone order of magnitude differenceā claim, but Iām not sure how much I believe them. I do come down to believing that most of my considerations will face counter-considerations which I am currently unaware of.
This is a really interesting point that I hadnāt thought of before.
Very lightly held counterargument to your conclusion:
P1: The more capable an AGI system is, the harder it is to align.
P2: Terraforming other planets requires AGI at the very top of the capability distribution.
P3: The pool of systems capable of terraforming is therefore drawn disproportionately from the capability range where misalignment is most likely.
Conclusion: Most worlds containing planet-terraforming AGI are probably rogue-AGI worlds. So the āspreading wild animal suffering to new planetsā scenario may be more associated with alignment failure than alignment success.
Corollary: If you agree you should be mildly agree-voting.
Fair pushback!
For P1, I assumed that AGI going well for humans was basically even ASI going well for humans (just: āit happens weāre in a good scenario for the fleshy humansā). I donāt know if ASI is much less likely to go right than AGIāsomething as capable as AGI could already very easily be misaligned and Iām not sure that scales with increase in capabilities.
For P2, Iām scared that we donāt necessarily need the top of the distribution of ASI to do this. I could imagine non-AGI worlds where human-brain-driven technological progress gets us there, though it seems very complicated resource-wise at this stage.
I agree that these two arguments together could undermine my vague āone order of magnitude differenceā claim, but Iām not sure how much I believe them. I do come down to believing that most of my considerations will face counter-considerations which I am currently unaware of.