Thanks for taking the time to share this, Hayden. It was very useful.
To what extent do behavioural science and systems thinking/change matter for AI governance?
To give you my view: I think that nearly all outcomes that EA cares about are mediated by individual and group behaviours and decisions: Who thinks what and does what (e.g., WRT. careers, donations, and advocacy) etc. All of this occurs in a broader context of social norms and laws etc.
Based on all this, I think that it is important to understand what people think and do, why they think and do what they do, and how to change that. Also, to understand how various contextual factors such as social norms and laws affect what people think and do and can be changed.
I notice related work on areas such as climate change, and I project that similar will be needed in AI governance. However, I don’t know the extent to which people working on AI governance share that view or what work, if anything, that has been done. I’d be interested to hear any thoughts that you have time to share.
Also, I’d really appreciate if you can suggest any good literature or people to engage with.
I’m not Hayden but I think behavioural science is useful area for thinking about AI governance, in particular about the design of human-computer interfaces. One example with current widely deployed AI systems is recommender engines (this is not a HCI eg). I’m trying to understand the tendencies of recommenders towards biases like concentration, or contamination problems, and how they impact user behaviour and choice. Additionally, how what they optimise for does/does not capture their values, whether that’s because of a misalignment of values between the user and the company or because it’s just really hard to learn human preferences because they’re complex. In doing this, it’s really tricky to actually distinguish in the wild between the choice architecture (behavioural parts) vs the algorithm when it comes to attributing to users’ actions.
Recommender systems are a great example of a broader concern. Another is lethal autonomous weapons, where a big focus is “meaningful human control”. Automation bias is an issue even up to the nuclear level—the concern is that people will more blindly trust ML systems, and won’t disbelieve them as people did in several Cold War close calls (eg Petrov not believing his computer warning of an attack). See Autonomy and machine learning at the interface of nuclear weapons, computers and people.
Thanks for taking the time to share this, Hayden. It was very useful.
To what extent do behavioural science and systems thinking/change matter for AI governance?
To give you my view: I think that nearly all outcomes that EA cares about are mediated by individual and group behaviours and decisions: Who thinks what and does what (e.g., WRT. careers, donations, and advocacy) etc. All of this occurs in a broader context of social norms and laws etc.
Based on all this, I think that it is important to understand what people think and do, why they think and do what they do, and how to change that. Also, to understand how various contextual factors such as social norms and laws affect what people think and do and can be changed.
I notice related work on areas such as climate change, and I project that similar will be needed in AI governance. However, I don’t know the extent to which people working on AI governance share that view or what work, if anything, that has been done. I’d be interested to hear any thoughts that you have time to share.
Also, I’d really appreciate if you can suggest any good literature or people to engage with.
I’m not Hayden but I think behavioural science is useful area for thinking about AI governance, in particular about the design of human-computer interfaces. One example with current widely deployed AI systems is recommender engines (this is not a HCI eg). I’m trying to understand the tendencies of recommenders towards biases like concentration, or contamination problems, and how they impact user behaviour and choice. Additionally, how what they optimise for does/does not capture their values, whether that’s because of a misalignment of values between the user and the company or because it’s just really hard to learn human preferences because they’re complex. In doing this, it’s really tricky to actually distinguish in the wild between the choice architecture (behavioural parts) vs the algorithm when it comes to attributing to users’ actions.
Hi both,
Yes behavioural science isn’t a topic I’m super familiar with, but it seems very important!
I think most of the focus so far has been on shifting norms/behaviour at top AI labs, for example nudging Publication and Release Norms for Responsible AI.
Recommender systems are a great example of a broader concern. Another is lethal autonomous weapons, where a big focus is “meaningful human control”. Automation bias is an issue even up to the nuclear level—the concern is that people will more blindly trust ML systems, and won’t disbelieve them as people did in several Cold War close calls (eg Petrov not believing his computer warning of an attack). See Autonomy and machine learning at the interface of nuclear weapons, computers and people.
Jess Whittlestone’s PhD was in Behavioural Science, now she’s Head of AI Policy at the Centre for Long-Term Resilience.