Miranda_Zhang comments on Some governance research ideas to prevent malevolent control over AGI and why this might matter a hell of a lot

[Miranda_Zhang]

Thanks a lot for your work on this neglected topic!

You mention,

Those counter-considerations seem potentially as strong as the motivations I list in favor of a focus on malevolent actors.

Could you give more detail on which of the counter-considerations (and motivations) you consider strongest?

[Jim Buhler]

Thanks Miranda! :)

I personally think the strongest argument for reducing malevolence is its relevance for s-risks (see section Robustness: Highly beneficial even if we fail at alignment), since I believe s-risks are much more neglected than they should be.

And the strongest counter-considerations for me would be

• Uncertainty regarding the value of the future. I’m generally much more excited about making the future go better rather than “bigger” (reducing X-risk does the latter), so the more reducing malevolence does the latter more than the former, the less certain I am it should be a priority. (Again, this applies to any kind of work that reduces X-risks, though.)

• Info /​ attention hazards. Perhaps the best way to avoid these malevolence scenarios is to ignore them and avoid making them more salient.

Interesting question you asked, thanks! I added a link to this comment in a footnote.