I suspect that principal–agent problems are the biggest single obstacle to alignment. That leads me to suspect it’s less tractable than you indicate.
I’m interested in what happened with Netflix. Ten years ago their recommendation system seemed focused almost exclusively on maximizing user ratings of movies. That dramatically improved my ability to find good movies.
Yet I didn’t notice many people paying attention to those benefits. Netflix has since then shifted toward less aligned metrics. I’m less satisfied with Netflix now, but I’m unclear what other users think of the changes.
I suspect that principal–agent problems are the biggest single obstacle to alignment. That leads me to suspect it’s less tractable than you indicate.
I’m interested in what happened with Netflix. Ten years ago their recommendation system seemed focused almost exclusively on maximizing user ratings of movies. That dramatically improved my ability to find good movies.
Yet I didn’t notice many people paying attention to those benefits. Netflix has since then shifted toward less aligned metrics. I’m less satisfied with Netflix now, but I’m unclear what other users think of the changes.