A similar analogy with the fossil fuel industry is mentioned by Stuart Russell (crediting Danny Hillis) here:
let’s say the fossil fuel industry as if it were an AI system. I think this is an interesting line of thought, because what he’s saying basically and — other people have said similar things — is that you should think of a corporation as if it’s an algorithm and it’s maximizing a poorly designed objective, which you might say is some discounted stream of quarterly profits or whatever. And it really is doing it in a way that’s oblivious to lots of other concerns of the human race. And it has outwitted the rest of the human race.
It also seems that “things go really badly if you optimise straightforwardly for one goal” bears similarities to criticisms of central planning or utopianism in general though.
A similar analogy with the fossil fuel industry is mentioned by Stuart Russell (crediting Danny Hillis) here:
It also seems that “things go really badly if you optimise straightforwardly for one goal” bears similarities to criticisms of central planning or utopianism in general though.