While it is my belief that the there is some wider context missing from certain aspects of this post (e.g., what sorts of AI progress are we talking about, perhaps strong AI or transformative AI?- this makes a difference), the analogy does a fair job at illustrating that the intention to use advanced AI to engender progress (beneficial outcomes for humanity) might have unintended and antithetical effects instead. This seems to actually encapsulate the core idea of AI safety / alignment, roughly that a system which is capable of engendering vasts amounts of scientific and economic gains for humanity need not be (and is unlikely to be) aligned with human values and flourishing by default and thus is capable of taking actions that may cause great harm to humanity. The Future of Life’s page on the Benefits and Risk of AI comments on this:
The AI is programmed to do something beneficial, but it develops a destructive method for achieving its goal: This can happen whenever we fail to fully align the AI’s goals with ours, which is strikingly difficult.
Needless to comment on this further, but the worst outcomes possible from unaligned AI seem extremely likely to exceed unemployment in severity.
I don’t think it’s just about “methods”. There was nothing wrong about the Cotton Gin’s “methods”, or its end goal. The problem was that other actors adapted their behavior to its presence, and, in fact, it’s not hard to see that this was very likely in retrospect (not that they could have predicted it, but if we “rerolled” the universe, it would probably happen again).
While it is my belief that the there is some wider context missing from certain aspects of this post (e.g., what sorts of AI progress are we talking about, perhaps strong AI or transformative AI?- this makes a difference), the analogy does a fair job at illustrating that the intention to use advanced AI to engender progress (beneficial outcomes for humanity) might have unintended and antithetical effects instead. This seems to actually encapsulate the core idea of AI safety / alignment, roughly that a system which is capable of engendering vasts amounts of scientific and economic gains for humanity need not be (and is unlikely to be) aligned with human values and flourishing by default and thus is capable of taking actions that may cause great harm to humanity. The Future of Life’s page on the Benefits and Risk of AI comments on this:
Needless to comment on this further, but the worst outcomes possible from unaligned AI seem extremely likely to exceed unemployment in severity.
I don’t think it’s just about “methods”. There was nothing wrong about the Cotton Gin’s “methods”, or its end goal. The problem was that other actors adapted their behavior to its presence, and, in fact, it’s not hard to see that this was very likely in retrospect (not that they could have predicted it, but if we “rerolled” the universe, it would probably happen again).