I do briefly discuss bounded utility functions as an objection to the argument for fanatical Superintelligences. I generally take the view that imposing bounded utility functions is difficult to do in a way that doesn’t seem arbitrary—in practice this might be less of an issue as one might be able observe the agent and impose bounded functions when necessary (I think this may raise other questions, but it does seem very possible in practice).
I don’t think bounded utility functions are bad intrinsically, but I do think the problems created by denying fanaticism (a denial which can result form overly imposing bounded utility functions) are potentially worse than fanaticism. By these problems I’m referring back to those provided in Wilkinson’s paper.
More importantly though, if we are trying to predict how superintelligent AIs will behave, we can’t assume that they’ll share our intuitions about the unpalatability of unbounded utility functions!
I think this is a very good point, and agree that we could end up with Superintelligences either imposing a bounded utility function or being fanatical. I think I would be somewhat intuitively inclined to think they would be fanatical more often than not in this case, but that isn’t really a substantiated view on my part. Either way we still end up with fanatical verdicts being reached and the concerns that entails.
Thanks for the comment!
I do briefly discuss bounded utility functions as an objection to the argument for fanatical Superintelligences. I generally take the view that imposing bounded utility functions is difficult to do in a way that doesn’t seem arbitrary—in practice this might be less of an issue as one might be able observe the agent and impose bounded functions when necessary (I think this may raise other questions, but it does seem very possible in practice).
I don’t think bounded utility functions are bad intrinsically, but I do think the problems created by denying fanaticism (a denial which can result form overly imposing bounded utility functions) are potentially worse than fanaticism. By these problems I’m referring back to those provided in Wilkinson’s paper.
I think this is a very good point, and agree that we could end up with Superintelligences either imposing a bounded utility function or being fanatical. I think I would be somewhat intuitively inclined to think they would be fanatical more often than not in this case, but that isn’t really a substantiated view on my part. Either way we still end up with fanatical verdicts being reached and the concerns that entails.