First—if you’re talking about nearer-term questions, like ‘What’s the right governance structure for a contemporary AI developer to ensure its board acts in the common interest?’ or ‘How can we help workers reskill after being displaced from the transport industry’ then I agree that doesn’t seem too strange. However, I don’t see how this would differ from the work that folks at places LPP and Gov.AI are doing already.
Second—if you’re talking about longer-term ideal governance questions, I reckon even relatively mundane topics are likely to seem pretty weird when studied in a longtermist context, because the bottom line for researchers will be how contemporary governance affects future generations.
To use your example of the future of work, an important question in that topic might be whether and when we should attribute legal personhood to digital labourers, with the bottom line concerning the effect of any such policy on the moral expansiveness of future societies. The very act of supposing that digital workers as smart as humans will one day exist is relatively weird, let alone considering their legal status, let further alone discussing the potential ethics of a digital civilisation.
This is of course a single, cherry-picked example, but I think that most papers justifying specific positive visions of the future will need to consider the impact of these intermediate positive worlds on the longterm future, which will appear weird and uncomfortably utopian. Meanwhile, I suspect that work with a negative focus (‘How can we prevent an arms race with China?’) or a more limited scope (‘How can we use data protection regulations to prevent bad actors from accessing sensitive datasets?’) doesn’t require this sort of abstract speculation, suggesting that research into ideal AI governance carries reputational hazards that others forms of safety/governance work do not. I’m particularly concerned that this will open up AI governance to more hit-pieces of this variety, turning off potential collaborators whose first interaction with longtermism is bad faith critique.
Thanks again for your comment. Two quick related points:
People at the places you mention are definitely already doing interesting work relevant to ideal theory, e.g., regarding institutional design. Something distinctive about AI ideal governance research that I do think is less common is consideration of the normative components of AI governance issues.
On reflection, your comments and examples have convinced me that in the original post I didn’t take the ‘weirdness problem’ seriously enough. Although I’d guess we might still have a slight disagreement about the scope (and possibly the implications) of the problem, I certainly see that it is particularly salient for longtermists at the moment given the debates around the publication of Will MacAskill’s new book. As an example of ideal governance research that considers longer-term issues (including some ‘weird’ ones) in an analytical manner, Nick Bostrom, Allan Dafoe and Carrick Flynn’s paper on ‘Public Policy and Superintelligent AI’ may be of interest.
Okay. Thanks for clarifying that for me—I think we agree more than I expected, because I’m pretty in favour of their institutional design work.
I think you’re right that we have a disagreement w/r/t scope and implications, but it’s not clear to me to what extent this is also just a difference in ‘vibe’ which might dissolve if we discussed specific implications. In any case, I’ll take a look at that paper.
I have a couple thoughts on this.
First—if you’re talking about nearer-term questions, like ‘What’s the right governance structure for a contemporary AI developer to ensure its board acts in the common interest?’ or ‘How can we help workers reskill after being displaced from the transport industry’ then I agree that doesn’t seem too strange. However, I don’t see how this would differ from the work that folks at places LPP and Gov.AI are doing already.
Second—if you’re talking about longer-term ideal governance questions, I reckon even relatively mundane topics are likely to seem pretty weird when studied in a longtermist context, because the bottom line for researchers will be how contemporary governance affects future generations.
To use your example of the future of work, an important question in that topic might be whether and when we should attribute legal personhood to digital labourers, with the bottom line concerning the effect of any such policy on the moral expansiveness of future societies. The very act of supposing that digital workers as smart as humans will one day exist is relatively weird, let alone considering their legal status, let further alone discussing the potential ethics of a digital civilisation.
This is of course a single, cherry-picked example, but I think that most papers justifying specific positive visions of the future will need to consider the impact of these intermediate positive worlds on the longterm future, which will appear weird and uncomfortably utopian. Meanwhile, I suspect that work with a negative focus (‘How can we prevent an arms race with China?’) or a more limited scope (‘How can we use data protection regulations to prevent bad actors from accessing sensitive datasets?’) doesn’t require this sort of abstract speculation, suggesting that research into ideal AI governance carries reputational hazards that others forms of safety/governance work do not. I’m particularly concerned that this will open up AI governance to more hit-pieces of this variety, turning off potential collaborators whose first interaction with longtermism is bad faith critique.
Thanks again for your comment. Two quick related points:
People at the places you mention are definitely already doing interesting work relevant to ideal theory, e.g., regarding institutional design. Something distinctive about AI ideal governance research that I do think is less common is consideration of the normative components of AI governance issues.
On reflection, your comments and examples have convinced me that in the original post I didn’t take the ‘weirdness problem’ seriously enough. Although I’d guess we might still have a slight disagreement about the scope (and possibly the implications) of the problem, I certainly see that it is particularly salient for longtermists at the moment given the debates around the publication of Will MacAskill’s new book. As an example of ideal governance research that considers longer-term issues (including some ‘weird’ ones) in an analytical manner, Nick Bostrom, Allan Dafoe and Carrick Flynn’s paper on ‘Public Policy and Superintelligent AI’ may be of interest.
Taking each of your points in turn:
Okay. Thanks for clarifying that for me—I think we agree more than I expected, because I’m pretty in favour of their institutional design work.
I think you’re right that we have a disagreement w/r/t scope and implications, but it’s not clear to me to what extent this is also just a difference in ‘vibe’ which might dissolve if we discussed specific implications. In any case, I’ll take a look at that paper.