One of the key questions here that’s been brought up is how difficult it would be to actually express one’s utility function.
I think this is the main problem. There are several reasons why it’s difficult:
It’s just hard to express
People would be trying to game the algorithm
I think in practice though:
There are at least some domains where this is fairly easy. For example, if I call fire fighters, I expect my unique preferences to matter a lot less than “just do the job well”.
Things will get better over time (better data, better estimation abilities)
There are other benefits that we get that are worth some cost here. (You get to encode a utility function that’s more preferable to your normal one; lower discount rate, and you get to be much more pro-social or coordinate much easier than you typically would)
One of the key questions here that’s been brought up is how difficult it would be to actually express one’s utility function.
I think this is the main problem. There are several reasons why it’s difficult:
It’s just hard to express
People would be trying to game the algorithm
I think in practice though:
There are at least some domains where this is fairly easy. For example, if I call fire fighters, I expect my unique preferences to matter a lot less than “just do the job well”.
Things will get better over time (better data, better estimation abilities)
There are other benefits that we get that are worth some cost here. (You get to encode a utility function that’s more preferable to your normal one; lower discount rate, and you get to be much more pro-social or coordinate much easier than you typically would)