There’s been quite a bit written on the “pro” side:
Also ARCHES, Concrete Problems in AI safety, etc
But not so much on the “con” side—people have generally just thought about opportunity cost. Your point that it might speed up harmful (due to safety, misuse or structural risks) applications is a really useful and important one! Would be hard to weigh things up—getting into tricky differential technological development territory. Would love for there to be more thinking on this topic.
On the other hand, this isn’t as much of a constraint in opposition. Political Advisors are like senior senior parliamentary researchers—everyone’s part of one (tiny!) team.
This is a great overview, thanks for writing it up—more people should work for MPs!
Some other useful resoures from 80,000 Hours on this topic: https://80000hours.org/career-reviews/party-politics-uk/ https://80000hours.org/2014/02/an-estimate-of-the-expected-influence-of-becoming-a-politician/ https://80000hours.org/2012/02/how-hard-is-it-to-become-prime-minister-of-the-united-kingdom/
4. Cotra aims to predict when it will be possible for “a single computer program [to] perform a large enough diversity of intellectual labor at a high enough level of performance that it alone can drive a transition similar to the Industrial Revolution.”—that is a “growth rate [of the world economy of] 20%-30% per year if used everywhere it would be profitable to use”
Your scenario is premise 4 “Some deployed APS systems will be exposed to inputs where they seek power in unintended and high-impact ways (say, collectively causing >$1 trillion dollars of damage), because of problems with their objectives” (italics added).
Your bar is (much?) lower, so we should expect your scenario to come (much?) earlier.
Great report, really fascinating stuff. Draws together lots of different writing on the subject, and I really like how you identify concerns that speak to different perspectives (eg to Drexler’s CAIS and classic Bostrom superintelligence).
Three quick bits of feedback:
I feel like some of Jess Whittlestone and collaborators’ recent research would be helpful in your initial framing, eg
Prunkl, C. and Whittlestone, J. (2020). Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society. - on capability vs impact
Gruetzemacher, R. and Whittlestone, J. (2019). The Transformative Potential of Artificial Intelligence. - on different scales of impact
Cremer, C. Z., & Whittlestone, J. (2021). Artificial Canaries: Early Warning Signs for Anticipatory and Democratic Governance of AI. - on milestones and limitations
I don’t feel like you do quite enough to argue for premise 5 “Some of this power-seeking will scale (in aggregate) to the point of permanently disempowering ~all of humanity | (1)-(4).”Which is, unfortunately, a pretty key premise and the one I have the most questions about! My impression is that section 6.3 is where that argumentation is intended to occur, but I didn’t leave it with a sense of how you thought this would scale, disempower everyone, and be permanent. Would love for you to say more on this.
On a related, but distinct point, one thing I kept thinking is “does it matter that much if its an AI system that takes over the world and disempowers most people?”. Eg you set out in 6.3.1 a number of mechanisms by which an AI system could gain power—but 10 out of the 11 you give (all except Destructive capacity) seem relevant to a small group of humans in control of advanced capabilities too.Presumably we should also be worried about a small group doing this as well? For example, consider a scenario in which a powerhungry small group, or several competing groups, use aligned AI systems with advanced capabilities (perhaps APS, perhaps not) to the point of permanently disempowering ~all of humanity.If I went through and find-replaced all the “PS-misaligned AI system” with “power-hungry small group”, would it read that differently? To borrow Tegmark’s terms, does it matter if its Omega Team or Prometheus?I’d be interested in seeing some more from you about whether you’re also concerned about that scenario, whether you’re more/less concerned, and how you think its different from the AI system scenario.
Again, really loved the report, it is truly excellent work.
Indeed. Seems supported by a quantum suicide argument—no matter how unlikely the observer, there always has to be a feeling of what-its-like-to-be that observer.
It’s worth adding that both Stephen Bush and Jeremy Cliffe at the New Statesman both do prediction posts and review them at the end of each year. The meme is spreading! They’re also two of the best journalists to follow about UK Labour politics (Bush) and EU politics (Cliffe) - if you’re interested in those topics, as I am.
I think the closest things we’ve got that’s similar to this are:
Luke Muehlhauser’s work on ‘amateur macrohistory’ https://lukemuehlhauser.com/industrial-revolution/
The (more academic) Peter Turchin’s Seshat database: http://seshatdatabank.info/
I would say more optimistic. I think there’s a pretty big difference between emergence (a shift from authoritarianism to democracy) - and democratic backsliding, that is autocratisation (a shift from democracy to authoritarianism). Once that shift has consolidated, there’s lots of changes that makes it self-reinforcing/path-dependent: norms and identities shift, economic and political power shifts, political institutions shift, the role of the military shifts. Some factors are the same for emergence and persistence, like wealth/growth, but some aren’t (which I would say are pretty key) like getting authoritarian elites to accept democratisation.
Two books on emergence that I’ve found particularly interesting are
The international dimensions of democratization: Europe and the Americas; edited by Laurence Whitehead 2001 (on underplayed international factors)
Conservative parties and the birth of democracy; Daniel Ziblatt 2017 (on buying off elites to accept this permanent change)
However as I said, the impact of AI systems does raise uncertainty, and is super fascinating.
Something I’m very concerned about, which I don’t believe you touched, is the fate of democracies after a civilizational collapse. I’ve got a book chapter coming out on this later this year, that I hope I may be able to share a preprint of.
Interesting post! If you wanted to read into the comparative political science literature a little more, you might be interested in diving into the subfield of democratic backsliding (as opposed to emergence):
A third wave of autocratization is here: what is new about it? Lührmann & Lindberg 2019
How Democracies Die. Steven Levitsky and Daniel Ziblatt 2018
On Democratic Backsliding Bermeo, Nancy 2016
Two Modes of Democratic Breakdown: A Competing Risks Analysis of Democratic Durability; Maeda, K. 201
Authoritarian Reversals and Democratic Consolidation in American Political Science Review; Milan Svolik; 2008
Institutional Design and Democratic Consolidation in the Third World Timothy J. Power; Mark J. Gasiorowski; 04/1997
What Makes Democracies Endure? Jose Antonio Cheibub; Adam Przeworski; Fernando Papaterra Limongi Neto; Michael M. Alvarez 1996
The breakdown of democratic regimes: crisis, breakdown, and reequilibration Book by Juan J. Linz 1978
One of the common threads in this subfield is that once a democracy has ‘consolidated’, it seems to be fairly resilient to coups and perhaps incumbent takeover.
I certainly agree that how this interacts with new AI systems: automation, surveillance and targeting/profiling, and autonomous weapons systems is absolutely fascinating. For one early stab, you might be interested in my colleagues’:
Tackling threats to informed decision-making in democratic societies: Promoting epistemic security in a technologically-advanced world (see here for a news article).
That’s right, I think they should be higher priorities. As you show in your very useful post, Ord has nuclear and climate change at 1/1000 and AI at 1⁄10. I’ve got a draft book chapter on this, which I hope to be able to share a preprint of soon.
I’m really sorry to hear that from both of you, I agree it’s a serious accusation.
For longtermism as a whole, as I argued in the post, I don’t understand describing it as white supremacy—like e.g. antiracism or feminism, longtermism is opposed to an unjust power structure.
Sorry its taking a while to get back to you!
In the meantime, you might be interested in this from our Catherine Richards: https://www.cser.ac.uk/resources/reframing-threat-global-warming/
Thanks for the comment and these very useful links—will check with our food expert colleague and get back to you, especially on the probability question.
Just personally, however, let me note that we say that those four factors you mention are current ‘sources of significant stress’ for systems for the production and allocation of food—and we note that while ‘global food productivity and production has increased dramatically’ we are concerned about the ‘vulnerability of our global food supply to rapid and global disruptions’ and shocks. The three ways we describe climate change further reducing food security are growing conditions, agricultural pests and diseases, and the occurrence of extreme weather events.
Note also that the global catastrophe is the shock (hazard) plus how it cascades through interconnected systems with feedback. We’re explicitly suggesting that the field move beyond ‘is x a catastrophe?’ to ‘how does x effect critical systems, which can feed into one another, and may act more on our vulnerability and exposure than as a direct, single hazard’.
Interesting! I would feel I had been quasirandomly selected to allocate our shared pool of donations—and would definitely feel some obligation/responsibility.
As evidence that other people feel the same way, I would point to the extensive research and write-ups that previously selected allocators have done. A key explanation for why they’ve done that is a sense of obligation/responsibility for the group.
As others have said, great piece! Well argued and evidenced and on an important and neglected topic. I broadly agree with your point estimates for the three cases.
I think it might be worth saying a bit more (perhaps in a seperate section near the top) about why your estimates of survival are not higher. What explains the remaining 0.01-0.3 uncertainty? How could it lead ‘directly’ to extinction? In different sections you talk about WMD, food availability etc, but I would have found it useful to have all that together. That would allow you to address general reasons for uncertainty too. The most compelling single reason for me, for example, is the unprecedented nature of a global, post-industrial collapse.
On your suggestions for other research directions:
I’d be super interested in someone going through the old Cold War RAND reports from the 1950s+1960s looking at collapse/recovery after nuclear war, and the wider literature on civil defence. Did the Soviets produce anything similar? I don’t know! Going through the ‘prepper’ literature might also maybe be useful? Perhaps as useful as scifi.
“For example, I think I’ve heard somewhere that places with higher levels of social trust have lower levels of looting, hoarding, and other antisocial disaster behavior.” You’re thinking of Aldrich, D. P. (2012). Building Resilience. University of Chicago Press. The wider field is disaster risk reduction (DRR).
Your policy seems reasonable. Although I wonder if the analogy with a regular lottery might risk confusing people. When one thinks of “entering a regular lottery for charitable giving”, one might think of additional money—money that counterfactually wouldn’t have gone to charity. But that’s not true of donor lotteries—there is no additional money.
On your second point: “making requests to pool money in a way that rich donors expect to lose control” describes the EA Funds, which I don’t think are a scam. In fact, the EA funds pool money in such a way that donors are certain to lose control.
Hey thanks for the comment!
As mentioned, I’m offering a bunch of alternatives—not all of which I support—to help us examine our current system. ‘Reverse-donation-weighted’ in particular is more of a prompt to “why do we think donation-weighting is normal or unproblematic—what might we be missing out on or reinforcing with donation-weighting?”
Note that the current ‘donor lottery’ is a form of random donor pooling—but with donation-weighting. I see donation weighting as a weird halfway house between EA Funds and (threshold) Random Pooling. With donation-weighting you don’t get the hiring process or expertise of EA Funds, and you get way fewer of the benefits of randomisation than (threshold) Random Pooling.
The alternative I’m most sympathetic to (threshold random donor pooling in a cause-area) isn’t affected by your second and third points. The allocator wouldn’t be some rural-museums-obsessive, it would be a “typical well-informed EA”—and because its within a cause area we could be even more sure it won’t be spent on e.g. a rural museum. Threshold random donor pooling in a cause-area would expand the search space within global health, or within animal rights, etc. And finally, the threshold would prevent raids.