Global moratorium on AGI, now (Twitter). Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn ⏸️
Karma: 5,647
those doing so should caveat that they are designed to mitigate the possibility (and not certainty) of catastrophic outcomes. This should be obvious, but given that people will be waiting in the wings to weaponise anything that could be called a regulatory overreaction, I think it’s worth doing.
I think to a lot of people, it matters just how much of a possibility there is. From what I’ve seen, many people are just (irrationally imo!) willing to bite the bullet on yolo-ing ASI if there is “only a 10%” chance of extinction. For this reason I counter with my actual assessment: doom is the default outcome of AGI/ASI (~90% likely). Only very few people are willing to bite that bullet! (And much more common is for people to fall back on dismissing the risk as “low”—e.g. experts saying “only” 1-25%).
Just going to remark that Kevin Roose’s comment (shown in the image) of “it’s not Dyson Spheres by 2040” is.. well, how can he possibly know that? We’re 15 years away from 2040 and exponential progress in AI is still ongoing.
Beyond capacity building; it’s not completely clear to me that there are robustly good interventions in AI safety, and I think more work is needed to prioritize interventions.
I think it’s pretty clear[1] that stopping further AI development (or Pausing) is a robustly good intervention in AI Safety (reducing AI x-risk).
However, what happens if these tendencies resurface when “shit hits the fan”?
I don’t think this could be pinned on PauseAI, when at no point has PauseAI advocated or condoned violence. Many (basically all?) political campaigns attract radical fringes. Non-violent moderates aren’t responsible for them.
To be clear, given the situation we now find ourselves in, I also think it’s perfectly reasonable to work on AI x-risk reduction for selfish reasons. Or just wanting your family and friends to live.
You mention S-risk. I tend to try not to think too much about this, but it needs to be considered in any EV estimate of working on AI Safety. I think appropriately factoring it in could be overwhelming in terms of concluding that preventing ASI from being built is the number 1 priority. Preventing space colonisation x-risk could be achieved by letting ASI extinct everything. But how likely is ASI-induced extinction over ASI-induced S-risk (ASI simulating, or physically creating, astronomical amounts of unimaginable suffering, on a scale far larger than human space colonisation could ever achieve)?
I’m concerned that given the nearness of AI-x-risk and high likelihood that we all die in the near future, to some extent you are trying to seek comfort in complex cluelessness and moral uncertainty. If we go extinct (along with all the rest of known life in the universe), maybe it would be for the best? I don’t know, I think I would rather live to find out, and help steer the future toward more positive paths (we can end factory farming before space colonisation happens in earnest). I also kind of think “what’s the point in doing all these other EA interventions if the world just ends in a few years?” Sure, there is some near term benefit to those helped here and now, but everyone still all ends up dead.
Going to push back a bit on the idea that preventing AI x-risk might be net negative as a reason for not trying to do this. Applying the reversal test: given the magnitude of expected values either way, positive or negative, from x-risk reduction, it would be an extreme coincidence if doing nothing was the best action. Why not work to increase x-risk? I appreciate there are other second order effects that may be overwhelming—the bad optics of being viewed as a “supervillain” for one! But this perhaps is a useful intuition pump for why in fact decreasing x-risk is likely to actually be +EV.
I think it’s very hard to get urgent political action if all communication about the issue is hedged and emphasises uncertainty—i.e. the kind of language that AI Safety, EA and LW people here are used to, rather than the kind of language that in used in everyday politics, let alone the kind of language that is typically used to emphasise the need for urgent evasive action.
I think the risk of lost credibility from signalling too much confidence is only really credibility in the eyes of technical AI people, not the general public or government policymakers / regulators—which are the people that matter now.To be clear, I’m not saying that all nuance should be lost—as with anything, detailed nuanced information and opinion will always there for people to read should they wish to dive deeper. But it’s fine to signal confidence in short public-facing comms, given the stakes (likely short timelines and high p(doom)).
Is it possible for us all to, as humanity, not die from rogue ASI without anyone ever being accused of crying wolf again?
Will there be a clear Fire Alarm that is pulled by a consensus of AI Safety researchers before we’re past the point of no return?
Will sufficient political action happen in time to avert doom if so, without any prior tabloid (sensationalist) reporting on the issue before hand?
Or will the necessary strong public support for such action just be the result of everyone in the world waking up and reading sober, nuanced, well-reasoned, contextful Less Wrong posts warning of our imminent doom at the appropriate time (not before; when the wolf is clearly visible)?
AI’s are already getting money with crypto memecoins. Wondering if there might be some kind of unholy mix of AI generated memecoins, crypto ransomware and self-replicating AI viruses unleashed in the near future.
Re “role-playing”, that is moot when it’s the end result that matters—what actions the AI takes in the world. See also: Frontier AI systems have surpassed the self-replicating red line.
Shutdown Avoidance (“do self-replication before being killed”), combined with the recent Apollo o1 research on propensity to attempt self-exfiltration, pretty much closes the loop on misaligned AIs escaping when given sufficient scaffolding to do so.
One can hope that the damage is limited and that it serves as an appropriate wake-up call to governments. I guess we’ll see..
Surely it’s just a matter of time—now that the method has been published—before AI models are spreading like viruses?
Frontier AI systems have surpassed the self-replicating red line
This is brilliant. I agree with almost all of it[1] - it’s a good articulation of how my own thinking on this has evolved over the last couple of years[2]. My timelines might be shorter, and my p(doom) higher, but it’s good to see an exposition for how one need not have such short timelines or high p(doom) to still draw the same conclusions. I recently donated significant amounts to PauseAI Global and PauseAI US. Your $30k to PauseAI US will get them to 5⁄6 of their current fundraising target—thank you!
- ^
Some points of disagreement, additional information, and emphasis in other comments I made as I read through.
- ^
Actually to be fair, it’s more detailed!
- ^
You should talk to David Pearce. His view of physicalism (phenomenal binding) precludes consciousness in digital minds[1].
I know of at least one potential counterexample: OpenAI’s RLHF was developed by AI safety people who joined OpenAI to promote safety. But it’s not clear that RLHF helps with x-risk.
I’d go further and say that it’s not actually a counterexample. RLHF allowed OpenAI to be hugely profitable—without it they wouldn’t’ve been able to publicly release their models and get their massive userbase.
Of course, if we do somehow survive all this, people will accuse me and others like me of crying wolf. But 1⁄10 outcomes aren’t that uncommon! I’m willing to take the reputation hit though, whether justified or not.
I think in general a big problem with AI x-risk discourse is that there are a lot of innumerate people around, who just don’t understand what probability means (or at least act like they don’t, and count everything as a confident statement even if appropriately hedged).