Secondly, prioritizing competence. Ultimately, humanity is mostly in the same boat: we’re the incumbents who face displacement by AGI. Right now, many people are making predictable mistakes because they don’t yet take AGI very seriously. We should expect this effect to decrease over time, as AGI capabilities and risks become less speculative. This consideration makes it less important that decision-makers are currently concerned about AI risk, and more important that they’re broadly competent, and capable of responding sensibly to confusing and stressful situations, which will become increasingly common as the AI revolution speeds up.
I think this is a good point.
At the same time, I think you can infer that people who don’t take AI risk seriously are somewhat likely to lack important forms of competence. This is inference is only probabilistic, but it’s IMO pretty strong already (it’s a lot stronger now than it used to be four years ago) and it’ll get stronger still.
It also depends how much a specific person has been interacting with the technology; meaning, it probably applies a lot less to DC policy people, but applies more to ML scientists or people at AI labs.
This seems cool!
I could imagine that many people will gravitate towards moral parliament approaches even when all the moral considerations are known. If moral anti-realism is true, there may not come a point in moral reflection under idealized circumstances where it suddenly feels like “ah, now the answer is obvious.” So, we can also think of moral parliament approaches as a possible answer to undecidedness when all the considerations are laid open.
I feel like only seeing it as an approach to moral uncertainty (so that, if we knew more about moral considerations, we’d just pick one of the first-order normative theories) is underselling the potential scope of applications of this approach.