Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
Thanks for sharing the post Zed :) Like titotal says, I hope you consider staying around. I think AI-risk (AIXR) sceptic posts should be welcomed on the Forum. Iâm someone whoâd probably count as AIXR sceptic for the EA community (but not the wider world/âpublic). Itâs clearly an area you think EA as a whole is making a mistake, so Iâve read the post and recent comments and have some thoughts that I hope you might find useful:
I think there are some good points you made:
I really appreciate posts that push against the âEA Orthodoxyâ on the Forum that start off useful discussions. I think âred-teamingâ ideas is a great example of necessary error-correction, so regardless of how much I agree or not, I want to give you plaudits for that.
On humility in long-term forecastsâI completely agree here. Iâm sure youâve come across it but Tetlockâs recent forecasting tournament deals with this question and does indeed find Forecasters place lower AIXR than subject-matter experts.[1] But Iâd still say that a risk of extinction roughly ~1% is worth considering as an important risk worth consideration and more investigation, wouldnât you?
I think your scepticism on very short timelines is directionally very valid. I hope that those who have made very, very short timeline predictions on Metaculus are willing to update if those dates[2] come and go without AGI. I think one way out of the poor state of the AGI debate is for more people to make concrete falsifiable predictions.
While I disagree with your reasoning about what the EA position on AIXR is (see below), I think itâs clear that many people think that is the position, so Iâd really like to here how youâve come to this impression and what EA or the AIXR community could do to present a more accurate picture of itself. I think reducing this gap would be useful for all sides.
Some parts that I didnât find convincing:
You view Hansonâs response as a knock-down argument. But he only addresses the âfoomâ cases and only does so heuristically, not from any technical arguments. I think more credible counterarguments are being presented by experts such as Belrose & Pope, who you might find convincing (though I think they have non-trivial subjective estimates of AIXR too fwiw).
I really donât like the move to psychoanalyse people in terms of bias. Is bias at play? Of course, itâs at play for all humans, but therefore just as likely for those who are super optimistic as those pessimistic a-priori. I think once something breaks through enough to be deemed âworthy of considerationâ then we ought to do most of our evaluation on the merits of the arguments given. You even say this at the end of the âfooling oneselfâ section! I guess I think the questions of âare AIXR concerns valid?â and âif not, why are they so prominent?â are probably worth two separate posts imo. Similarly to this, I think you sometimes conflate the questions of âare AIXR concerns valid?â and âif it is, what would an appropriate policy response look like?â I think in your latest comment to Hayven thatâs where you strongest objections are (which makes sense to me, given your background and expertise), but again is diferent from the pure question of if AIXR concern is valid.
Framing those concerned with AIXR as âalarmistsâ - I think youâre perhaps overindexing on MIRI here as representative of AI Safety as a whole? From my vague sense, MIRI doesnât hold a dominant position in AI Safety space as it perhaps did 10â20 years ago. I donât think that ~90%+ belief in doom is an accurate depiction of EA, and similarly I donât think that an indefinite global pause is the default EA view of the policies that ought to be adopted. Like you mention Anthropic and CHAI as two good institutions, and theyâre both highly EA-coded and sincerely concerend about AIXR. I think a potential disambiguation here is between âconcern about AIXRâ and âcertain doom about AIXRâ?
But also some bad ones:
saying that EAâs focus on x-risk lacks âcommon senseââI actually think x-risk is something which the general public would think makes a lot of sense, but theyâd think that EA gets the source of that risk wrong (though empirical data). I think a lot of people would say that trying to reduce the risk of human extinction from Nuclear War or Climate Change is an unambiguously good cause and potentially good use of marginal resources.
Viewing EA, let alone AIXR, as motivated by ânonsense utilitarianismâ about âtrillions of theoretical future peopleâ. Most EA spending goes to Global Health Causes in the present. Many AIXR advocates donât identify as longtermists at all. Theyâre often, if not mostly, concerned about risk to humans alive today, themselves, those they care about. Concern about AIXR could also be motivated through non-utilitarian frameworks, though Iâd concede that this probably isnât the standard EA position
I know this is a super long comment, so feel free to only respond to the bits you find useful or even not at all. Alternatively we could try out the new dialogue feature to talk through this a bit more? In any case, thanks again for the post, it got me thinking about where and why I disagree both with AI âdoomersâ as well as your position in this post.
roughly 0.4% for superforecasters vs 2.1% for AI experts by 2100
Currently March 14th 2026 at time of writing
Love this thoughtful response!
Good feedbackâI see the logic of your points, and donât find faults with any of them.
On AIXR as valid and what the response would be, youâre right; I emphasize the practical nature of the policy recommendation because otherwise, the argument can veer into the metaphysical. To use an analogy, if I claim thereâs a 10% chance another planet could collide with Earth and destroy the planet in the next decade, you might begrudgingly accept the premise to move the conversation on to the practical aspect of my forecast. Even if that were true, what would my policy intervention look like? Build interstellar lifeboats? Is that feasible in the absence of concrete evidence?
Agreeâarmchair psychoanalysis isnât really useful. What is useful is understanding how heuristics and biases work on a population level. If we know that, in general, projects run over budget and take longer than expected, we can adjust our estimates. If we know experts mis-forecast x-risk, we can adjust for that too. Thatâs far from psychoanalysis.
I donât really know what the median view on AIXR within EA communities truly is. One thingâs for certain: the public narrative around the issue highly tilts towards the âpause AIâ camp and the Yudkowskys out there.
On the common sense of X-riskâone of the neat offices that few people know of at the State Department is the Nuclear Risk Reduction Center or NRRC. Itâs staffed 24â7 and has foreign language designated positions, meaning at least someone in the room speaks Russian, etc. The office is tasked with staying in touch with other nations to reduce the odds of a miscalculation and nuclear war. That makes tons of sense. Thinking about big problems that could end the world makes sense in generalâdisease, asteroids, etc.
What I find troubling is the propensity to assign odds to distant right-tail events. And then to take the second step of recommending costly and questionable policy recommendations. I donât think these are EA consensus positions, but they certainly receive outsized attention.
Iâm glad you found my comment useful. I think then, with respect, you should consider retracting some of your previous comments, or at least reframing them to be more circumspect and be clear youâre taking issue with a particular framing/âsubset of the AIXR community as opposed to EA as a whole.
As for the points in your comment, thereâs a lot of good stuff here. I think a post about the NRRC, or even an insiderâs view into how the US administration thinks about and handles Nuclear Risk, would be really useful content on the Forum, and also incredibly interesting! Similarly, I think how a community handles making âright-tail recommendationsâ when those recommendations may erode its collective and institutional legitimacy[1] would be really valuable. (Not saying that you should write these posts, theyâre just examples off the top of my head. In general I think you have a professional perspective a lot of EAs could benefit from)
I think one thing where we agree is that thereâs a need to ask and answer a lot more questions, some of which you mention here (beyond âis AIXR validâ):
What policy options do we have to counteract AIXR if true?
How do the effectiveness of these policy options change as we change our estimation of the risk?
What is the median view in the AIXR/âbroader EA/âbroader AI communities on risk?
And so on.
Some people in EA might write this off as âopticsâ, but I think thatâs wrong
These are all great suggestions! As for my objections to EA as a whole versus a subset, it reminds me a bit of a defense that folks employ whenever a larger organization is criticised. Defenses that one hears from Republicans in the US for example. âItâs not all of us, just a vocal subset!â That might be true, but I think it misses the point. Itâs hard to soul-search and introspect as an organization or a movement if we collectively say, ânot all-EAâ when someone points to the enthusiasm around SBF and ideas like buying up coal mines.
Hey, welcome to the EA forum! I hope you stick around.
I pretty much agree with this post. The argument put forward by AI risk doomers is generally flimsy and weak, with core weaknesses involving unrealistic assumptions about what AGI would actually be capable of, given limitations of computational complexity and the physical difficulty of technological advancements, and also a lack of justification for assuming AI will be fanatical utility function maximisers. I think the chances of human extinction from AI are extremely low, and that estimates around here are inflated by subtle groupthink, poor probabilistic treatment of speculative events, and a few just straight up wrong ideas that were made up a long time ago and not updated sufficiently for the latest events in AI.
That being said, AI advancements could have a significant effect on the world. I think itâs fairly likely that if AI is misused, there may be a body count, perhaps a significant one. I donât think itâs a bad idea to be proactive and think ahead about how to manage the risks involved. There is a middle ground between no regulation and bombing data centers.
Iâm curious as to the somewhat hedging word choices in the second paragraph like could and perhaps. The case for great, even extreme, harm from AI misuse seems a lot more straightforward than AI doom. Misuse of new, very powerful technologies has caused at least significant harm (including body counts) in the past with some consistency, so I would assume the pattern would follow with AI as well.
Iâm allowing for the possibility that we hit another AI winter, and the new powerful technology just doesnât arrive in our lifetime. Or that the technology is powerful for some things, but remains too unreliable for use in life-critical situations and is kept out of them.
I think itâs likely that AI will have at least an order or magnitude or two greater body count than it has now, but I donât know how high it will be.
I once worked on a program with DoD to help buy up loose MANPADS in Libya. Thereâs a linear causal relationship between portable air defense systems and harm. Other ordnance has a similar relationship.
The relationship is tenuous when we move from the world of atoms to bits. I struggle to see how new software could pose novel risks to life and limb. That doesnât mean developers of self-driving vehicles or autopilot functions in aircraft should ignore safety in their software design, what Iâm suggesting is that those considerations are not novel.
If someone advocates that we treat neural networks unlike any other system in existence today, I would imagine the burden of proof would be on them to justify this new approach.
Hi Zed! Thanks for your post. A couple of responses:
âAs critics of the long-termist viewpoint have noted, the base-rate for human extinction is zero.â
Yes, but this is tautologically true: Only in worlds where humanity hasnât gone extinct could you make that observation in the first place. (For a discussion of this and some tentative probabilities, see https://ââwww.nature.com/ââarticles/ââs41598-019-47540-7)
âInstead of outlandish ideas of a new global government capable of unilaterally curtailing compute power or some other factor through force, we should focus on what is practically achievable today. Encouraging firms like OpenAI to red-team their models before release, for example, is practical and limits negative externalities.â
Why are the two mutually exclusive? I think youâre opening a false dichotomyâas far as I know, x-risk oriented folks are amongst the leading voices calling for red teams or even engaging in this work themselves. (See also: https://ââforum.effectivealtruism.org/ââposts/ââQ4rg6vwbtPxXW6ECj/ââwe-are-fighting-a-shared-battle-a-call-for-a-different)
âLetâs assume for a moment that domain experts who warn of imminent threats to humanityâs survival from AI are acting in good faith and are sincere in their convictions.â
The way you phrase this makes it sound like we have reason to doubt their sincerity. Iâd love to hear what makes you think we do!
âFor example, a global pause in model training that many advocated for made no reference to the ideaâs inherent weaknessâthat is, it sets up a prisonerâs dilemma in which the more AI firms voluntarily agree to pause research, the greater the incentive for any one group to defect from the agreement and gain a competitive edge. It makes no mention of practical implementation, nor does it explain how it arrived on its pause time-duration; nor does it recognize the improbability of enforcing a global treaty on AI.â
My understanding is that even strong advocates of a pause are aware of its shortcomings and communicate these uncertainties rather transparentlyâI have yet to meet someone who sees them as a panacea. Granted, the questions you ask need to be answered, but the fact that an idea is thorny and potentially difficult to implement doesnât make it a bad one per sĂ©.
âA strict international regime dedicated to preventing proliferation still failed to prevent India, Israel, Pakistan, North Korea, and, likely, Iran from acquiring weapons.â
Are you talking about the NPT or the IAEA here? My expertise on this is limited (~90 hours of engagement), but I authored a case study on IAEA safeguards this summer and my overall takeaway was that domain experts like Carl Robichaud still consider these regimes success stories. Iâd be curious to hear where you disagree! :)
Thanks for the thoughtful response.
On background extinction rates, rather than go down that rabbit hole, I think my point still stands, any estimation of human extinction needs to be rooted in some historical analysis. Whether that is one in 87,000 of homo sapiens going extinct in any given year as the Nature piece suggests, or something revised up or down from there.
On false dichotomiesâIâd set aside individual behavior for a moment and look at the macro picture. We know from political science basics that elites can meaningfully shift public opinion on issues of low salience. According to Pew, weâve seen a 15-point shift in the general public expressing âmore concern than excitementâ over AI in the United States. Rarely do we see such a marked shift in opinion on any particular issue in such a divided electorate.
Letâs put it this wayâin a literal sense, yes, one could loudly espouse a belief that AI could destroy humanity within a decade and at the same time, advocate for rudimentary red-teaming to keep napalm recipes out of an LLMâs response, but, in practice, this seems to defy common sense and ignores the effect on public opinion.
Imagine weâre engineers at a new electric vehicle company. At an all hands meeting, we discuss one of the biggest issues with the design, the automatic trunk release. Weâre afraid people might get their hands caught in it. An engineer pipes up and says, âwhile weâre talking about flaws, I think thereâs a chance that the car might explode and take out a city block.â Now, thereâs nothing stopping us from looking at the trunk release and investigating spontaneous combustion, but in practice, I struggle to imagine those processes happening in parallel in a meaningful way.
Coming back to public opinion, weâve seen what happens when novel technology gains motivated opponents, from nuclear fission to genetic engineering, to geoengineering, to stem cell research, to gain of function research, to autonomous vehicles, and on. Government policy responds to voter sentiment, not elite opinion. And fear of the unknown is a much more powerful driver of behavior than a vague sense of productivity gains. My sense is that if we continue to see elites writing op-eds on how the world will end soon, weâll see public opinion treat AI like it treats GMO fruits and veg.
My default is to assume folks are sincere in their convictions (and data shows most people are)--I should have clarified that line; it was in reference to claims that outfits calling for AI regulation are cynically putting up barriers to entry and on a path to rent-seeking.
On the pause being a bad idea: my point here is that the very conception is foolish at the strategic level, not that it has practical implementation difficulties. First, what would change in six months? And second, why would creating a prisonerâs dilemma lead to better outcomes? It would be like soft drink makers asking for a non-binding pause on advertisingâit only works if thereâs consensus and an enforcement mechanism that would impose a penalty on defectors; otherwise, itâs even better for me if you stop advertising, and I continue, stealing your market share.
The IAEA and NPT are their own can of worms, but in general, my broader point here is that even a global attempt to limit the spread of nuclear weapons failed. What is the likelihood of imposing a similar regime on a technology that is much simpler to work with? No centrifuges, no radiation, just code and compute power? I struggle to see how creating an IAEA for AI would have a different outcome.
Do you think a permanent* ban on AI research and development would be a better path than a pause? I agree a six-month pause is likely not to do anything, but far-reaching government legislation banning AI just mightâespecially if we can get the U.S., China, EU, and Russia all on board (easier said than done!).
*nothing is truly permanent, but I would feel much more comfortable with a more socially just and morally advanced human society having the AI discussion ~200 years from now, than for the tech to exist today. Humanity today shouldnât be trusted to develop AI for the same reason 10-year-olds shouldnât be trusted to drive trucks: it lacks the knowledge, experience, and development to do it safely.
Letâs look at the history of global bans:
- They donât work for doping in the Olympics.
- They donât work for fissile material.
- They donât prevent luxury goods from entering North Korea.
- They donât work against cocaine or heroine.
We could go on. And those examples are much easier to implementâthereâs global consensus and law enforcement trying to stop the drug trade, but the economics of the sector mean an escalating war with cartels only leads to greater payoffs for new market entrants.
Setting aside practical limitations, we ought to think carefully before weaponizing the power of central governments against private individuals. When we can identify a negative externality, we have some justification to internalize it. No one wants firms polluting rivers or scammers selling tainted milk.
Generative AI hasnât shown externalities that would necessitate something like a global ban.
Trucks: we know what the externalities of a poorly piloted vehicle are. So we minimize those risks by requiring competence.
And on a morally advanced societyâyes, Iâm certain a majority of folks if asked would say theyâd like a more moral and ethical world. But thatâs not the questionâthe question is who gets to decide what we can and cannot do? And what criteria are they using to make these decisions? Real risk, as demonstrated by data, or theoretical risk? The latter was used to halt interest in nuclear fission. Should we expect the same for generative AI?
The question of âwho gets to do whatâ is fundamentally political, and I really try to stay away from politics especially when dealing with the subject of existential risk. This isnât to discount the importance of politics, only to say that while political processes are helpful in determining how we manage x-risk, they donât in and of themselves directly relate to the issue. Global bans would also be political, of course.
You may well be right that the existential risk iof generative AI, and eventually AGI, is low or indeterminate, and theoretical rather than actual. I donât think we should wait until we have an actual x-risk on our hands to act â because then it may be too late.
Youâre also likely correct on AI development being unstoppable at this point. Mitigation plans are needed should unfriendly outcomes occur especially with an AGI, and I think we can both agree on that.
Maybe Iâm too cautious when it comes to the subject of AI, but part of what motivates me is the idea that, should the catastrophic occur, I could at least know that I did everything in my power to oppose that risk.
These are all very reasonable positions, and one would struggle to find fault with them.
Personally, Iâm glad there are smart folks out there thinking about what sorts of risks we might face in the near future. Biologists have been talking about the next big pandemic for years. It makes sense to think these issues through.
Where I vehemently object is on the policy side. To use the pandemic analogy, itâs the difference between a research-led investigation into future pandemics and a call to ban the use of CRISPR. Itâs impractical and, from a policy perspective, questionable.
The conversation around AI within EA is framed as âwe need to stop AI progress before we all die.â It seems tough to justify such an extreme policy position.
Welcome to the EA Forum, and thanks for the post, Zed!
The link you have here is broken.
ah yes sorry, here it is: https://ââlink.springer.com/ââchapter/ââ10.1007/ââ978-3-662-45704-7_2