I think this is a very important question that should probably get its own post.
I’m currently very uncertain about it but I imagine the most realistic scenario is a mix of a lot of different approaches that never feels fully stable. I guess it might be similar to nuclear weapons today but on steroids, i.e. different actors have control over the technology, there are some norms and rules that most actors abide by, there are some organizations that care about non-proliferation, etc. But overall, a small perturbation could still blow up the system.
A really stable scenario probably requires either some very tough governance, e.g. preventing all but one actor from getting to AGI, or high-trust cooperation between actors, e.g. by working on the same AGI jointly.
Overall, I currently don’t see a realistic scenario that feels more stable than nuclear weapons seem today which is not very reassuring.
I’d argue it’s even less stable than nukes, but one reassuring point: There will ultimately be a very weird future with thousands, Millions or billions of AIs, post humans and genetically engineered beings, and the borders are very porous and dissolvable and that ultimately is important to keep in mind. Also we don’t need arbitrarily long alignment, just aligning it for 50-100 years is enough. Ultimately nothing needs to be in the long term stable, just short term chaos and stability.
Because it’s possible that even in unstable, diverse futures, catastrophe can be avoided. As to the long-term future after the Singularity, that’s a question we will deal with it when we get there
I don’t think “dealing with it when we get there” is a good approach to AI safety. I agree that bad outcomes could be averted in unstable futures but I’d prefer to reduce the risk as much as possible nonetheless.
I think this is a very important question that should probably get its own post.
I’m currently very uncertain about it but I imagine the most realistic scenario is a mix of a lot of different approaches that never feels fully stable. I guess it might be similar to nuclear weapons today but on steroids, i.e. different actors have control over the technology, there are some norms and rules that most actors abide by, there are some organizations that care about non-proliferation, etc. But overall, a small perturbation could still blow up the system.
A really stable scenario probably requires either some very tough governance, e.g. preventing all but one actor from getting to AGI, or high-trust cooperation between actors, e.g. by working on the same AGI jointly.
Overall, I currently don’t see a realistic scenario that feels more stable than nuclear weapons seem today which is not very reassuring.
I’d argue it’s even less stable than nukes, but one reassuring point: There will ultimately be a very weird future with thousands, Millions or billions of AIs, post humans and genetically engineered beings, and the borders are very porous and dissolvable and that ultimately is important to keep in mind. Also we don’t need arbitrarily long alignment, just aligning it for 50-100 years is enough. Ultimately nothing needs to be in the long term stable, just short term chaos and stability.
I’m not sure why this should be reassuring. It doesn’t sound clearly good to me. In fact, it sounds pretty controversial.
Because it’s possible that even in unstable, diverse futures, catastrophe can be avoided. As to the long-term future after the Singularity, that’s a question we will deal with it when we get there
I don’t think “dealing with it when we get there” is a good approach to AI safety. I agree that bad outcomes could be averted in unstable futures but I’d prefer to reduce the risk as much as possible nonetheless.