Crossposting a comment: As co-author of one of the mentioned pieces, I’d say it’s really great to see the AGI xrisk message mainstreaming. It doesn’t nearly go fast enough, though. Some (Hawking, Bostrom, Musk) have already spoken out about the topic for close to a decade. So far, that hasn’t been enough to change common understanding. Those, such as myself, who hope that some form of coordination could save us, should give all they have to make this go faster. Additionally, those who think regulation could work should work on robust regulation proposals which are currently lacking. And those who can should work on international coordination, which is currently also lacking.
A lot of work to be done. But the good news is that the window of opportunity is opening, and a lot of people could work on this which currently aren’t. This could be a path to victory.
Otto
Paper Summary: The Effectiveness of AI Existential Risk Communication to the American and Dutch Public
Organizing a debate with experts and MPs to raise AI xrisk awareness: a possible blueprint
[Crosspost] Why Uncontrollable AI Looks More Likely Than Ever
AI X-risk in the News: How Effective are Recent Media Items and How is Awareness Changing? Our New Survey Results.
Introducing the Existential Risk Observatory
[Crosspost] An AI Pause Is Humanity’s Best Bet For Preventing Extinction (TIME)
Unveiling the American Public Opinion on AI Moratorium and Government Intervention: The Impact of Media Exposure
Existential Risk Observatory: results and 2022 targets
[Crosspost] AI Regulation May Be More Important Than AI Alignment For Existential Safety
I hope that this article sends the signals that pausing the development of the largest AI-models is good, informing society about AGI xrisk is good, and that we should find a coordination method (regulation) to make sure we can effectively stop training models that are too capable.
What I think we should do now is:
1) Write good hardware regulation policy proposals that could reliably pause the development towards AGI.
2) Campaign publicly to get the best proposal implemented, first in the US and then internationally.
This could be a path to victory.
I don’t know if everyone should drop everything else right now, but I do agree that raising awareness about AI xrisks should be a major cause area. That’s why I quit my work on the energy transition about two years ago to found the Existential Risk Observatory, and this is what we’ve been doing since (resulting in about ten articles in leading Dutch newspapers, this one in TIME, perhaps the first comms research, a sold out debate, and a passed parliamentary motion in the Netherlands).
I miss two significant things on the list of what people can do to help:1) Please, technical people, work on AI Pause regulation proposals! There is basically one paper now, possibly because everyone else thought a pause was too far outside the Overton window. Now we’re discussing a pause anyway and I personally think it might be implemented at some point, but we don’t have proper AI Pause regulation proposals, which is a really bad situation. Researchers (both policy and technical), please fix that, fix it publicly, and fix it soon!
2) You can start institutes or projects that aim to inform the societal debate about AI existential risk. We’ve done that and I would say it worked pretty well so far. Others could do the same thing. Funders should be able to choose from a range of AI xrisk communication projects to spend their money most effectively. This is currently really not the case.
I agree that this strategy is underexplored. I would prioritize the following work in this direction as follows:
What kind of regulation would be sufficiently robust to slow down, or even pause, all AGI capabilities actors? This should include research/software regulation, hardware regulation, and data regulation. I think a main reason why many people think this strategy is unlikely to work is that they don’t believe any practical regulation would be sufficiently robust. But to my knowledge, that key assumption has never been properly investigated. It’s time we do so.
How could we practically implement sufficiently robust regulation? What would be required to do so?
How can we inform sufficiently large portions of society about AI xrisk to get robust regulation implemented? We are planning to do more research on this topic at the Existential Risk Observatory this year (we already have some first findings).
If you want to spend money quickly on reducing carbon dioxide emissions, you can buy emission rights and destroy them. In schemes such as the EU ETS, destroyed emission rights should lead to direct emission reduction. This has technically been implemented already. Even cheaper is probably to buy and destroy rights in similar schemes in other regions.
Great work, thanks a lot for doing this research! As you say, this is still very neglected. Also happy to see you’re citing our previous work on the topic. And interesting finding that fear is such a driver! A few questions:
- Could you share which three articles you’ve used? Perhaps this is in the dissertation, but I didn’t have the time to read that in full.
- Since it’s only one article per emotion (fear, hope, mixed), perhaps some other article property (other than emotion) could also have led to the difference you find?
- What follow-up research would you recommend?
- Is there anything orgs like ours (Existential Risk Observatory) (or, these days, MIRI, that also focuses on comms) should do differently?
As a side note, we’re conducting research right now on where awareness has gone after our first two measurements (that were 7% and 12% in early/mid ’23, respectively). We might also look into the existence and dynamics of a tipping point.
Again, great work, hope you’ll keep working in the field in the future!
Announcing #AISummitTalks featuring Professor Stuart Russell and many others
Great idea, congrats on the founding and looking forward to working with you!
Enough happened to write a small update about the Existential Risk Observatory.
First, we made progress in our core business: informing the public debate. We have published two more op-eds (in Dutch, one with a co-author from FLI) in a reputable, large newspaper. Our pieces warn against existential risk, especially from AGI, and propose low-hanging fruit type of measures the Dutch government could take to reduce risk (e.g. extra AI safety research).
A change w.r.t. the previous update, is that we see serious, leading journalists become interested in the topic. One leading columnist has already written a column about AI existential risk in a leading newspaper. Another journalist is planning to write a major article about it. This same person proposed having a debate about AI xrisk at the leading debate center, which would be well-positioned to influence yet others, and he proposed to use his network for the purpose. This is definitely not yet a fully-fledged informed societal debate yet, but it does update our expectations in relevant ways:
The idea of op-eds translating into broader media attention is realistic.
That attention is generally constructive, and not derogatory.
Most of the informing takes place in a social, personal context.
From our experience, the process is really to inform leaders of the societal debate, who then inform others. We have for example organized an existential risk drink, where thought leaders, EAs, and journalists could talk to each other, which worked very well. Key figures should hear accurate existential risk information from different sides. Social proof is key. Being honest, sincere, coherent, and trying to receive as well as send, goes a long way, too.
Another update is that we will receive funding from the SFF and are in serious discussions with two other funds. We are really happy that this proves that our approach, reducing existential risk by informing the public debate, has backing in the existential risk community. We are still resource-constrained, but also massively manpower- and management-constrained. Our vision is a world where everyone is informed about existential risk. We cannot achieve this vision alone, but would like other institutes (new and existing) to join us in the communication effort. That we have received funding for informing the societal debate is evidence that others can, too. We are happy to share information about what we are doing and how others could do the same at talks, for example for local EA groups or at events.
Our targets for this year remain the same:
Publish at least three articles about existential risk in leading media in the Netherlands.
Publish at least three articles about existential risk in leading media in the US.
Receive funding for stability and future upscaling.
We will start working on next year’s targets in Q4.
Anyway I posted this here because I think it somewhat resembles the policy of buying and closing coal mines. You’re deliberately creating scarcity. Since there are losers when you do that, policymakers might respond. I think creating scarcity in carbon rights is more efficient and much more easy to implement than creating scarcity in coal, but indeed suffers from some of the same drawbacks.
It’s definitely good to think about whether a pause is a good idea. Together with Joep from PauseAI, I wrote down my thoughts on the topic here.
Since then, I have been thinking a bit on the pause and comparing it to a more frequently mentioned option, namely to apply model evaluations (evals) to see how dangerous a model is after training.
I think the difference between the supposedly more reasonable approach of evals and the supposedly more radical approach of a pause is actually smaller than it seems. Evals aim to detect dangerous capabilities. What will need to happen when those evals find that, indeed, a model has developed such capabilities? Then we’ll need to implement a pause. Evals or a pause is mostly a choice about timing, not a fundamentally different approach.
With evals, however, we’ll move precisely to the brink, look straight into the abyss, and then we plan to halt at the last possible moment. Unfortunately, though, we’re in thick mist and we can’t see the abyss (this is true even when we apply evals, since we don’t know which capabilities will prove existentially dangerous, and since an existential event may already occur before running the evals).
And even if we would know where to halt: we’ll need to make sure that the leading labs will practically succeed in pausing themselves (there may be thousands of people working there), that the models aren’t getting leaked, that we’ll implement the policy that’s needed, that we’ll sign international agreements, and that we gain support from the general public. This is all difficult work that will realistically take time.
Pausing isn’t as simple as pressing a button, it’s a social process. No one knowns how long that process of getting everyone on the same page will take, but it could be quite a while. Is it wise to start that process at the last possible moment, namely when the evals turn red? I don’t think so. The sooner we start, the higher our chance of survival.
Also, there’s a separate point that I think is not sufficiently addressed yet: we don’t know how to implement a pause beyond a few years duration. If hardware and algorithms improve, frontier models could democratize. While I believe this problem can be solved by international (peaceful) regulation, I also think this will be hard and we will need good plans (hardware or data regulation proposals) for how to do this in advance. We currently don’t have these, so I think working on them should be a much higher priority.