Thanks again for your comment. Two quick related points:
People at the places you mention are definitely already doing interesting work relevant to ideal theory, e.g., regarding institutional design. Something distinctive about AI ideal governance research that I do think is less common is consideration of the normative components of AI governance issues.
On reflection, your comments and examples have convinced me that in the original post I didn’t take the ‘weirdness problem’ seriously enough. Although I’d guess we might still have a slight disagreement about the scope (and possibly the implications) of the problem, I certainly see that it is particularly salient for longtermists at the moment given the debates around the publication of Will MacAskill’s new book. As an example of ideal governance research that considers longer-term issues (including some ‘weird’ ones) in an analytical manner, Nick Bostrom, Allan Dafoe and Carrick Flynn’s paper on ‘Public Policy and Superintelligent AI’ may be of interest.
Okay. Thanks for clarifying that for me—I think we agree more than I expected, because I’m pretty in favour of their institutional design work.
I think you’re right that we have a disagreement w/r/t scope and implications, but it’s not clear to me to what extent this is also just a difference in ‘vibe’ which might dissolve if we discussed specific implications. In any case, I’ll take a look at that paper.
Thanks again for your comment. Two quick related points:
People at the places you mention are definitely already doing interesting work relevant to ideal theory, e.g., regarding institutional design. Something distinctive about AI ideal governance research that I do think is less common is consideration of the normative components of AI governance issues.
On reflection, your comments and examples have convinced me that in the original post I didn’t take the ‘weirdness problem’ seriously enough. Although I’d guess we might still have a slight disagreement about the scope (and possibly the implications) of the problem, I certainly see that it is particularly salient for longtermists at the moment given the debates around the publication of Will MacAskill’s new book. As an example of ideal governance research that considers longer-term issues (including some ‘weird’ ones) in an analytical manner, Nick Bostrom, Allan Dafoe and Carrick Flynn’s paper on ‘Public Policy and Superintelligent AI’ may be of interest.
Taking each of your points in turn:
Okay. Thanks for clarifying that for me—I think we agree more than I expected, because I’m pretty in favour of their institutional design work.
I think you’re right that we have a disagreement w/r/t scope and implications, but it’s not clear to me to what extent this is also just a difference in ‘vibe’ which might dissolve if we discussed specific implications. In any case, I’ll take a look at that paper.