The main question of the debate week is: “On the margin, it is better to work on reducing the chance of our extinction than increasing the value of the future where we survive”.
Where “our” is defined in a footnote as “earth-originating intelligent life (i.e. we aren’t just talking about humans because most of the value in expected futures is probably in worlds where digital minds matter morally and are flourishing)”.
I’m interested to hear from the participants how likely they think extinction of “earth-originating intelligent life” really is this century. Note this is not the same as asking what your p(doom) is, or what likelihood you assign to existential catastrophe this century.
My own take is that literal extinction of intelligent life, as defined, is (much) less than 1% likely to happen this century, and this upper-bounds the overall scale of the “literal extinction” problem (in ITN terms). I think this partly because the definition counts AI survival as non-extinction, and I truly struggle to think of AI-induced catastrophes leaving only charred ruins, without even AI survivors. Other potential causes of extinction, like asteroid impacts, seem unlikely on their own terms. As such, I also suspect that most work directed at existential risk is just already not in practice targeting extinction as defined, though of course it is also not explicitly focusing on “better futures” instead — more like “avoiding potentially terrible global outcomes”.
(This became more a comment than a question… my question is: “thoughts?”)
I think what we should be talking about is whether we hit the “point of no return” this century for extinction of Earth-originating intelligent life. Where that could mean: Homo sapiens and a most other mammals get killed off in an extinction event this century; then technologically-capable intelligence never evolves again on Earth; so all life dies off within a billion years or so. (In a draft post that you saw of mine, this is what I had in mind.)
The probability of this might be reasonably high. There I’m at idk 1%-5%.
I think the “earth-originating intelligent life” term should probably include something that indicates sentience/ moral value. Perhaps you could read that into “intelligent” but that feels like a stretch. But I didn’t want to imply that a world with no humans but many non-conscious AI systems would count as anything but an extinction scenario—that’s one of the key extinction scenarios.
I think it hinges on whether our AI successors would be counted as “life”, or whether they “matter morally”. I think the answer is likely no to both[1]. Therefore the risk of extinction boils down to risk of misaligned ASI wiping out the biosphere. Which I think is ~90% likely this century on the default trajectory, absent a well enforced global moratorium on ASI development.
Or at least “no” to the latter; if we consider viruses to be life that don’t matter morally, or are in fact morally negative, we can consider (default rogue) ASI to be similar.
Yeah, do you have other proposed reconceptualisations of the debate?
One shower thought I’ve had is that maybe we should think of the debate as about whether to focus on ensuring that humans have final control over AI systems or ensuring that humans do good things with that control. But this is far from perfect.
If we assume that natural risks are negligible, I would guess that this probably reduces to something like the question of what probability you put on extinction or existential catastrophe due to anthropogenic biorisk? Since biorisk is likely to leave much of the rest of Earthly life unscathed, it also hinges on what probability you assign to something like “human level intelligence” evolving anew. I find it reasonably plausible that a cumulative technological culture of the kind that characterizes human beings is unlikely to be a convergent evolutionary outcome (for the reasons given in Powell, Contingency and Convergence), and thus if human beings are wiped out, there is very little probability of similar traits emerging in other lineages. So human extinction due to a bioengineered pandemic strikes me as maybe the key scenario for the extinction of earth-originating intelligent life. Does that seem plausible?
I would add the (in my view far more likely) possibility of Yudkowskian* paperclipping via non-sentient AI, which given our currently incredibly low level of control of AI systems, and the fact that we don’t know how to create sentience, seems like the most likely default.
*) Specifically, the view that paperclipping occurs by default from any complex non-satiable implicit utility function, rather than the Bostromian paperclipping risk of accidentally giving a smart AI a dumb goal.
The main question of the debate week is: “On the margin, it is better to work on reducing the chance of our extinction than increasing the value of the future where we survive”.
Where “our” is defined in a footnote as “earth-originating intelligent life (i.e. we aren’t just talking about humans because most of the value in expected futures is probably in worlds where digital minds matter morally and are flourishing)”.
I’m interested to hear from the participants how likely they think extinction of “earth-originating intelligent life” really is this century. Note this is not the same as asking what your p(doom) is, or what likelihood you assign to existential catastrophe this century.
My own take is that literal extinction of intelligent life, as defined, is (much) less than 1% likely to happen this century, and this upper-bounds the overall scale of the “literal extinction” problem (in ITN terms). I think this partly because the definition counts AI survival as non-extinction, and I truly struggle to think of AI-induced catastrophes leaving only charred ruins, without even AI survivors. Other potential causes of extinction, like asteroid impacts, seem unlikely on their own terms. As such, I also suspect that most work directed at existential risk is just already not in practice targeting extinction as defined, though of course it is also not explicitly focusing on “better futures” instead — more like “avoiding potentially terrible global outcomes”.
(This became more a comment than a question… my question is: “thoughts?”)
I think what we should be talking about is whether we hit the “point of no return” this century for extinction of Earth-originating intelligent life. Where that could mean: Homo sapiens and a most other mammals get killed off in an extinction event this century; then technologically-capable intelligence never evolves again on Earth; so all life dies off within a billion years or so. (In a draft post that you saw of mine, this is what I had in mind.)
The probability of this might be reasonably high. There I’m at idk 1%-5%.
Notably, the extinction event in this scenario is non-AI related I assume? And needs to occur before we have created self-sufficient AIs.
I think the “earth-originating intelligent life” term should probably include something that indicates sentience/ moral value. Perhaps you could read that into “intelligent” but that feels like a stretch. But I didn’t want to imply that a world with no humans but many non-conscious AI systems would count as anything but an extinction scenario—that’s one of the key extinction scenarios.
I think it hinges on whether our AI successors would be counted as “life”, or whether they “matter morally”. I think the answer is likely no to both[1]. Therefore the risk of extinction boils down to risk of misaligned ASI wiping out the biosphere. Which I think is ~90% likely this century on the default trajectory, absent a well enforced global moratorium on ASI development.
Or at least “no” to the latter; if we consider viruses to be life that don’t matter morally, or are in fact morally negative, we can consider (default rogue) ASI to be similar.
Yeah, do you have other proposed reconceptualisations of the debate?
One shower thought I’ve had is that maybe we should think of the debate as about whether to focus on ensuring that humans have final control over AI systems or ensuring that humans do good things with that control. But this is far from perfect.
If we assume that natural risks are negligible, I would guess that this probably reduces to something like the question of what probability you put on extinction or existential catastrophe due to anthropogenic biorisk? Since biorisk is likely to leave much of the rest of Earthly life unscathed, it also hinges on what probability you assign to something like “human level intelligence” evolving anew. I find it reasonably plausible that a cumulative technological culture of the kind that characterizes human beings is unlikely to be a convergent evolutionary outcome (for the reasons given in Powell, Contingency and Convergence), and thus if human beings are wiped out, there is very little probability of similar traits emerging in other lineages. So human extinction due to a bioengineered pandemic strikes me as maybe the key scenario for the extinction of earth-originating intelligent life. Does that seem plausible?
I would add the (in my view far more likely) possibility of Yudkowskian* paperclipping via non-sentient AI, which given our currently incredibly low level of control of AI systems, and the fact that we don’t know how to create sentience, seems like the most likely default.
*) Specifically, the view that paperclipping occurs by default from any complex non-satiable implicit utility function, rather than the Bostromian paperclipping risk of accidentally giving a smart AI a dumb goal.