Executive summary: The author argues that If Anyone Builds It Everyone Dies overstates the certainty of AI-driven human extinction, contending instead that while AI takeover risk is serious, there are multiple plausible points where catastrophe could be averted, leading them to assign a low (≈2%) but still alarming probability of extinction from misaligned AI.
Key points:
The author rejects Yudkowsky and Soares’ claim of near-certain doom, arguing that uncertainty compounds across multiple necessary steps such as building superintelligent agents, failing at alignment, missing warning shots, and AI being able to kill everyone.
They assign substantial probability to “alignment by default,” suggesting that reinforcement learning and current training practices may often produce broadly friendly behavior rather than catastrophic misalignment.
Even if alignment is not achieved by default, the author argues there is a significant chance that deliberate alignment research, potentially aided by AI systems themselves, could succeed.
The author expects credible “warning shots” from misaligned AI before full takeover, which would likely trigger shutdowns or bans rather than being ignored.
They question whether intelligence alone guarantees the ability to exterminate humanity, noting physical, experimental, and infrastructural constraints on what AI could actually do.
While rejecting near-certainty of doom, the author still views AI risk as extremely serious and argues that believing doom is inevitable leads to worse strategic thinking than an “everything and the kitchen sink” risk-reduction approach.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
Executive summary: The author argues that If Anyone Builds It Everyone Dies overstates the certainty of AI-driven human extinction, contending instead that while AI takeover risk is serious, there are multiple plausible points where catastrophe could be averted, leading them to assign a low (≈2%) but still alarming probability of extinction from misaligned AI.
Key points:
The author rejects Yudkowsky and Soares’ claim of near-certain doom, arguing that uncertainty compounds across multiple necessary steps such as building superintelligent agents, failing at alignment, missing warning shots, and AI being able to kill everyone.
They assign substantial probability to “alignment by default,” suggesting that reinforcement learning and current training practices may often produce broadly friendly behavior rather than catastrophic misalignment.
Even if alignment is not achieved by default, the author argues there is a significant chance that deliberate alignment research, potentially aided by AI systems themselves, could succeed.
The author expects credible “warning shots” from misaligned AI before full takeover, which would likely trigger shutdowns or bans rather than being ignored.
They question whether intelligence alone guarantees the ability to exterminate humanity, noting physical, experimental, and infrastructural constraints on what AI could actually do.
While rejecting near-certainty of doom, the author still views AI risk as extremely serious and argues that believing doom is inevitable leads to worse strategic thinking than an “everything and the kitchen sink” risk-reduction approach.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.