Haydn has been a Research Associate and Academic Project Manager at the University of Cambridge’s Centre for the Study of Existential Risk since Jan 2017.
HaydnBelfield
Karma: 3,751
Hi Stefan,
Thanks for this response.
You’re quite right that if this post were arguing that there is an overall pattern, it would quite clearly be inadequate. It doesn’t define the universe of cases or make clear how representative these cases are of that universe, the two main studies could be criticised for selecting on the dependent variable, and its based primarily on quotes from two books.
However, I didn’t set out to answer something like the research question “which is more common in 20th century history, mistakenly sprinting or mistakenly failing to sprint?”—though I think that’s a very interesting question, and would like someone to look into it!
My intention for this blog post was for it to be fairly clear and memorable, aimed at a general audience—especially perhaps a machine learning researcher who doesn’t know much about history. The main takeaway I wanted wasn’t for people to think “this is the most common/likely outcome” but rather to add a historic example to their repertoire that they can refer to—“this was an outcome”. It was supposed to be a cautionary tale, a prompt to people to think not “all sprints are wrong” but rather “wait am I in an Ellsberg situation?”—and if so to have some general, sensible recommendations and questions to ask.
My aim was to express a worry (“be careful about mistaken sprints”) and illustrate that with two clear, memorable stories. There’s a reasonable scenario in the next few decades that we’re in a situation where we feel we need to back a sprint, prompted by concern about another group/country’s sprint. If we do, and I’m not around to say “hey lets be careful about this and check we’re actually in a race” then I hope these two case studies may stick in someone’s mind and lead them to say “OK but lets just check, don’t want to make the same mistake as Szilard and Ellsberg...”
Congratulations! So glad this is out.
This is hugely useful, thanks for putting it together!
If these experts regularly have a large impact on these decisions, that’s an argument for transparency about them. This is a factor that could of course be outweighed by other considerations (ability to give frank advice, confidentiality, etc). Perhaps might be worth asking them how they’d feel about being named (with no pressure attached, obviously).
Also, can one volunteer as an expert? I would—and I imagine others (just on this post, perhaps Ian and Sam?) would too.
Yep great news!
Just a quick one: this is great and groundbreaking work, thanks for doing it!
Thank you, and I agree on both counts.
The authors’ takeaway is:
“The implication of these historical outcomes is that in order to reliably affect decision-making, you must yourself be the decision-maker. Prestige, access to decision-makers, relevant expertise, and cogent reasoning are not sufficient; even with all these you are liable to be ignored. By understanding the complex workings of decision-making at the highest levels, you can improve your chances of influencing outcomes in the way you desire, but even if you understand how the game is played, you are ultimately subject to the judgment of those who wield power, and this judgment can be frustratingly capricious. Without even such an understanding, you stand little or no chance whatsoever. ”
I’m sympathetic to this view, and think they’re right about this case study (eg see my Are you really in a race? The Cautionary Tales of Szilárd and Ellsberg).
Nevertheless, I think this claim is overconfident and unfounded. We can’t just generalise from one case to the entire universe of cases! A more accurate assessment needs to reckon with the success of the nuclear and biological weapons arms control epistemic community in the early 1970s (such as Kissinger and Meselson) - as well as the many other examples of scientific advisers being influential.
This looks like absolutely fascinating, much-needed work. I particularly appreciate the variety of methodological approaches. Looking forward to reading!
Merci
I assume Carl is thinking of something along the lines of “try and buy most new high-end chips”. See eg Sam interviewed by Rob.
Probably “environment-shaping”, but I imagine future posts will discuss each perspective in more detail.
It’s really important that there is public, good-faith, well-reasoned critique of this important chapter in a central book in the field. You raise some excellent points that I’d love to see Ord (and/or others) respond to. Congratulations on your work, and thank you!
More than Philip Tetlock (author of Superforecasting)?
Does that particular quote from Yudkowsky not strike you as slightly arrogant?
There’s a whole AI ethics and safety field that would have been much smaller and less influential.
From my paper Activism by the AI Community: Analysing Recent Achievements and Future Prospects.
“2.2 Ethics and safety
There has been sustained activism from the AI community to emphasise that AI should be developed and deployed in a safe and beneficial manner. This has involved Open Letters, AI principles, the establishment of new centres, and influencing governments.
The Puerto Rico Conference in January 2015 was a landmark event to promote the beneficial and safe development of AI. It led to an Open Letter signed by over 8,000 people calling for the safe and beneficial development of AI, and a research agenda to that end [21]. The Asilomar Conference in January 2017 led to the Asilomar AI Principles, signed by several thousand AI researchers [23]. Over a dozen sets of principles from a range of groups followed [61].
The AI community has established several research groups to understand and shape the societal impact of AI. AI conferences have also expanded their work to consider the impact of AI. New groups include:
OpenAI (December 2015)
Centre for Human-Compatible AI (August 2016)
Leverhulme Centre for the Future of Intelligence (October 2016)3
DeepMind Ethics and Society (October 2017)
UK Government’s Centre for Data Ethics and Innovation (November 2017)”
Great post! Mass extinctions and historical societal collapses are important data sources—I would also suggest ecological regime shifts. My main takeaway is actually about multicausality: several ‘external’ shocks typically occur in a similar period. ‘Internal’ factors matter too—very similar shocks can affect societies very differently depending on their internal structure and leadership. When complex adaptive systems shift equilibria, several causes are normally at play.
Myself, Luke Kemp and Anders Sandberg (and many others!) have three seperate chapters touching on these topics in a forthcoming book on ‘Historical Systemic Collapse’ edited by Princeton’s Miguel Centeno et al . Hopefully coming out this year.
Thanks for this. I’m more counselling “be careful about secrecy” rather than “don’t be secret”. Especially be careful about secret sprints, being told you’re in a race but can’t see the secret information why, and careful about “you have to take part in this secret project”.
On the capability side, the shift in AI/ML publication and release norms towards staged release (not releasing full model immediately but carefully checking for misuse potential first), structured access (through APIs) and so on has been positive, I think.
On the risks/analysis side, MIRI have their own “nondisclosed-by-default” policy on publication. CSER and other academic research groups tend towards more of a “disclosed-by-default” policy.
Hi both,
Yes behavioural science isn’t a topic I’m super familiar with, but it seems very important!
I think most of the focus so far has been on shifting norms/behaviour at top AI labs, for example nudging Publication and Release Norms for Responsible AI.
Recommender systems are a great example of a broader concern. Another is lethal autonomous weapons, where a big focus is “meaningful human control”. Automation bias is an issue even up to the nuclear level—the concern is that people will more blindly trust ML systems, and won’t disbelieve them as people did in several Cold War close calls (eg Petrov not believing his computer warning of an attack). See Autonomy and machine learning at the interface of nuclear weapons, computers and people.
Jess Whittlestone’s PhD was in Behavioural Science, now she’s Head of AI Policy at the Centre for Long-Term Resilience.
I found this presentation of a deployment problem really concrete and useful, thanks.