I think this is an interesting question. I donât have a solid answer, but here are some related thoughts:
How likely we are to land in this scenario in the first place, and what shape it might take, seems related to:
Questions around how âhardâ, âfastâ, and/âor âdiscontinuousâ AI takeoff will be
Questions like âWill we know when transformative AI is coming soon? How far in advance? How confidently?â
Questions like âWould there be clearer evidence of AI risk in future, if itâs indeed quite risky? Will that lead to better behaviours regarding AI safety and governance?â
Your question as a whole seems similar to the last of the questions listed above.
And you seem to highlight the interesting idea that clearer evidence of AI risk in future (via a âsub-existentialâ catastrophe) could lead to worse behaviours regarding AI safety and governance.
And you also seem to highlight that we can/âshould think now about how to influence what behaviours might occur at that point (rather than merely trying to predict behaviours).
It seems like if suppressing AI were easy /â safe, that would have been the first choice of AI safety people, at least until such time as alignment is thoroughly solved
My tentative impression is that this is true of many AI safety people, but Iâm not sure itâs true of all of them. That is, itâs plausible to me that a decent number of people concerned about AI risk might not want to âsuppress AIâ even if this was tractable and wouldnât pose risks of e.g. making mainstream AI researchers angry at longtermists.
Hereâs one argument for that position: There are also other existential risks, and AI might help us with many of them. If you combine that point with certain empirical beliefs, it might suggest that slowing down (non-safety-focused) AI research could actually increase existential risk. (See Differential technological development: Some early thinking.)
(Iâm not saying that that conclusion is correct; I donât know what the best estimates of the empirical details would reveal.)
One obvious fear being that a government powerful enough to suppress AI could itself cause a persistent dystopia.
I think itâs slightly worse than this; I think permanent and complete suppression of AI would probably itself be an existential catastrophe, as it seems it would likely result in humanity falling far short of fulfilling its potential.
This seems related to Bostromâs notion of âplateauingâ. (I also review some related ideas here.)
This is distinct from (and in addition to) your point that permanent and complete suppression of AI is evidence that a government is powerful enough to cause other bad outcomes.
(This isnât a strong argument against temporary or partial suppression of AI, though.)
I think this is an interesting question. I donât have a solid answer, but here are some related thoughts:
How likely we are to land in this scenario in the first place, and what shape it might take, seems related to:
Questions around how âhardâ, âfastâ, and/âor âdiscontinuousâ AI takeoff will be
Questions like âWill we know when transformative AI is coming soon? How far in advance? How confidently?â
Questions like âWould there be clearer evidence of AI risk in future, if itâs indeed quite risky? Will that lead to better behaviours regarding AI safety and governance?â
(For notes and sources on those questions, see Crucial questions for longtermists, and particularly this doc.)
Your question as a whole seems similar to the last of the questions listed above.
And you seem to highlight the interesting idea that clearer evidence of AI risk in future (via a âsub-existentialâ catastrophe) could lead to worse behaviours regarding AI safety and governance.
And you also seem to highlight that we can/âshould think now about how to influence what behaviours might occur at that point (rather than merely trying to predict behaviours).
My tentative impression is that this is true of many AI safety people, but Iâm not sure itâs true of all of them. That is, itâs plausible to me that a decent number of people concerned about AI risk might not want to âsuppress AIâ even if this was tractable and wouldnât pose risks of e.g. making mainstream AI researchers angry at longtermists.
Hereâs one argument for that position: There are also other existential risks, and AI might help us with many of them. If you combine that point with certain empirical beliefs, it might suggest that slowing down (non-safety-focused) AI research could actually increase existential risk. (See Differential technological development: Some early thinking.)
(Iâm not saying that that conclusion is correct; I donât know what the best estimates of the empirical details would reveal.)
I think itâs slightly worse than this; I think permanent and complete suppression of AI would probably itself be an existential catastrophe, as it seems it would likely result in humanity falling far short of fulfilling its potential.
This seems related to Bostromâs notion of âplateauingâ. (I also review some related ideas here.)
This is distinct from (and in addition to) your point that permanent and complete suppression of AI is evidence that a government is powerful enough to cause other bad outcomes.
(This isnât a strong argument against temporary or partial suppression of AI, though.)