When I consider one part of AI risk as ‘things go really badly if you optimise straightforwardly for one goal’ I occasionally think about the similarity to criticisms of market economies (aka critiques of ‘capitalism’).
I am a bit confused why this does not come up explicitly, but possibly I have just missed it, or am conceptually confused.
Some critiques of market economies think this is exactly what the problem with market economies is: they should maximize for what people want, but instead they maximize for profit instead, and these two goals are not as aligned as one might hope. You could just call it the market economy alignment problem.
A paperclip maximizer might create all the paperclips, no matter what it costs and no matter what the programmers’ intentions were. The Netflix recommender system recommends movies to people which glue them to Netflix, whether they endorse this or not, to maximize profit for Netflix. Some random company invents a product and uses marketing that makes having the product socially desirable, even though people would not actually have wanted it on reflection.
These problems seem very alike to me. I am not sure where I am going with this, it does kind of feel to me like there is something interesting hiding here, but I don’t know what.
EA feels culturally opposed to ‘capitalism critiques’ to me, but they at least share this one line of argument. Maybe we are even missing out on a group of recruits.
Some ‘latestage capitalism’ memes seem very similar to Paul’s What Failure looks like to me.
Edit: Actually, I might be using the terms market economy and capitalism wrongly here and drawing the differences in the wrong place, but it’s probably not important.
A similar analogy with the fossil fuel industry is mentioned by Stuart Russell (crediting Danny Hillis) here:
let’s say the fossil fuel industry as if it were an AI system. I think this is an interesting line of thought, because what he’s saying basically and — other people have said similar things — is that you should think of a corporation as if it’s an algorithm and it’s maximizing a poorly designed objective, which you might say is some discounted stream of quarterly profits or whatever. And it really is doing it in a way that’s oblivious to lots of other concerns of the human race. And it has outwitted the rest of the human race.
It also seems that “things go really badly if you optimise straightforwardly for one goal” bears similarities to criticisms of central planning or utopianism in general though.
People do bring this up a fair bit—see for example some previous related discussion on Slatestarcodex here and the EA forum here.
I think most AI alignment people would be relatively satisfied with an outcome where our controls over AI outcomes were as strong as our current control over corporations: optimisation for a criteria that requires continual human input from a broad range of people, while keeping humans in-the-loop of decision making inside the optimisation process, and with the ability to impose additional external constrains at run-time (regulations).
Thank you so much for the links! Possibly I was just being a bit blind.
I was pretty excited about the Aligning Recommender systems article as I had also been thinking about that, but only now managed to read it in full. I somehow had missed Scott’s post.
I’m not sure whether they quite get to the bottom of the issue though (though I am not sure whether there is a bottom of the issue, we are back to ‘I feel like there is something more important here but I don’t know what’).
The Aligning recommender systems article discusses the direct relevance to more powerful AI alignment a fair bit which I was very keen to see. I am slightly surprised that there is little discussion on the double layer of misaligned goals—first Netflix does not recommend what users would truly want, second it does that because it is trying to maximize profit. Although it is up to debate whether aligning ‘recommender systems’ to peoples’ reflected preferences would actually bring in more money than just getting them addicted to the systems, which I doubt a bit.
Your second paragraph feels like something interesting in the capitalism critiques—we already have plenty of experience with misalignment in market economies between profit maximization and what people truly want, are there important lessons we can learn from this?
For capitalism more generally, GPI also has “Alternatives to GDP” in their research agenda, presumably because the GDP measure is what the whole world is pretty much optimizing for, and creating a new measure might be really high value.
[epistemic status: musing]
When I consider one part of AI risk as ‘things go really badly if you optimise straightforwardly for one goal’ I occasionally think about the similarity to criticisms of market economies (aka critiques of ‘capitalism’).
I am a bit confused why this does not come up explicitly, but possibly I have just missed it, or am conceptually confused.
Some critiques of market economies think this is exactly what the problem with market economies is: they should maximize for what people want, but instead they maximize for profit instead, and these two goals are not as aligned as one might hope. You could just call it the market economy alignment problem.
A paperclip maximizer might create all the paperclips, no matter what it costs and no matter what the programmers’ intentions were. The Netflix recommender system recommends movies to people which glue them to Netflix, whether they endorse this or not, to maximize profit for Netflix. Some random company invents a product and uses marketing that makes having the product socially desirable, even though people would not actually have wanted it on reflection.
These problems seem very alike to me. I am not sure where I am going with this, it does kind of feel to me like there is something interesting hiding here, but I don’t know what. EA feels culturally opposed to ‘capitalism critiques’ to me, but they at least share this one line of argument. Maybe we are even missing out on a group of recruits.
Some ‘latestage capitalism’ memes seem very similar to Paul’s What Failure looks like to me.
Edit: Actually, I might be using the terms market economy and capitalism wrongly here and drawing the differences in the wrong place, but it’s probably not important.
A similar analogy with the fossil fuel industry is mentioned by Stuart Russell (crediting Danny Hillis) here:
It also seems that “things go really badly if you optimise straightforwardly for one goal” bears similarities to criticisms of central planning or utopianism in general though.
People do bring this up a fair bit—see for example some previous related discussion on Slatestarcodex here and the EA forum here.
I think most AI alignment people would be relatively satisfied with an outcome where our controls over AI outcomes were as strong as our current control over corporations: optimisation for a criteria that requires continual human input from a broad range of people, while keeping humans in-the-loop of decision making inside the optimisation process, and with the ability to impose additional external constrains at run-time (regulations).
Thank you so much for the links! Possibly I was just being a bit blind. I was pretty excited about the Aligning Recommender systems article as I had also been thinking about that, but only now managed to read it in full. I somehow had missed Scott’s post.
I’m not sure whether they quite get to the bottom of the issue though (though I am not sure whether there is a bottom of the issue, we are back to ‘I feel like there is something more important here but I don’t know what’).
The Aligning recommender systems article discusses the direct relevance to more powerful AI alignment a fair bit which I was very keen to see. I am slightly surprised that there is little discussion on the double layer of misaligned goals—first Netflix does not recommend what users would truly want, second it does that because it is trying to maximize profit. Although it is up to debate whether aligning ‘recommender systems’ to peoples’ reflected preferences would actually bring in more money than just getting them addicted to the systems, which I doubt a bit.
Your second paragraph feels like something interesting in the capitalism critiques—we already have plenty of experience with misalignment in market economies between profit maximization and what people truly want, are there important lessons we can learn from this?
I mused about something similar here—about corporations as dangerous optimization demons which will cause GCRs if left unchecked :
https://forum.effectivealtruism.org/posts/vy2QCTXfWhdiaGWTu/corporate-global-catastrophic-risks-c-gcrs-1
Not sure how fruitful it was.
For capitalism more generally, GPI also has “Alternatives to GDP” in their research agenda, presumably because the GDP measure is what the whole world is pretty much optimizing for, and creating a new measure might be really high value.