As someone who does lots of biorisk work, I disagree that this is the only likely catastrophic risk, as I note in my response, but event more strongly disagree that this is actually a direct extinction risk—designed diseases that kill everyone in the face of humans actually trying to stop them aren’t obviously possible, much less findable by near-human or human level AI systems.
Of course, combined with systemic fragility, intentional disinformation, and other attack modes enabled by AI, it seems plausible that a determined adversary with tremendous resources could create an extinction event—though it’s unclear any such group exists. But that doesn’t route only through biorisk, and even robustly solving biorisk probably wouldn’t eliminate this risk given other vulnerabilities and determined adversaries. (So it’s a good thing no large and undetected group of people or major government with implausibly good opsec wants to spend a decade and billions of dollars to commit omnicide, while no-one notices or does anything to stop them.)
As someone who does lots of biorisk work, I disagree that this is the only likely catastrophic risk, as I note in my response, but event more strongly disagree that this is actually a direct extinction risk—designed diseases that kill everyone in the face of humans actually trying to stop them aren’t obviously possible, much less findable by near-human or human level AI systems.
Of course, combined with systemic fragility, intentional disinformation, and other attack modes enabled by AI, it seems plausible that a determined adversary with tremendous resources could create an extinction event—though it’s unclear any such group exists. But that doesn’t route only through biorisk, and even robustly solving biorisk probably wouldn’t eliminate this risk given other vulnerabilities and determined adversaries. (So it’s a good thing no large and undetected group of people or major government with implausibly good opsec wants to spend a decade and billions of dollars to commit omnicide, while no-one notices or does anything to stop them.)