You raise some interesting points. Some quick notes/counterpoints:
Not all existential risk is extinction risk.
Existential risk doesn’t have an extremely clean definition, but in the simple extinction/doom/non-utopia ontology, most longtermist EA’s intuitive conception of “existential risk” is closer to risk of “doom” than “extinction”
Nuclear war may not be a large direct existential risk, but it’s an existential risk factor.
The world could be made scarier after large-scale nuclear war, and thus less hospitable for altruistic values (plus other desiderata)
AI may or may not kill us all. But this point is academic and only mildly important, because if unaligned AI takes over, we (humanity and our counterfactual descendants) have lost control of the future.
Almost all moral value in the future is in the tails (extremely good and extremely bad outcomes).
Those outcomes likely require optimization for, and it seems likely that our spiritual descendants optimize heavily for good stuff than bad stuff.
Bad stuff might happen incidentally (historical analogues include factory farming and slavery), but they aren’t being directly optimized for, so will be a small fraction of the badness of maximally bad outcomess.
Yeah I think I have the most problem with (4), something that I probably should have expressed more in the post.
It’s true that humans are in theory trying to optimize for good outcomes, and this is a reason to expect utility to diverge to infinity. However, there are in my view equally good reasons utility to diverge to negative infinity- that being that the world is not designed for humans. We are inherently fragile creatures, only suitable to live in a world with specific temperature, air composition, etc. There are a lot of large-scale phenomenon causing these factors to change—s-risks—that could send utility plunging. This, plus the fact that current utility is below 0, means that I think existential risk is probably a moral benefit.
I also agree that this whole thing is pretty pedantic, especially in cases like AI domination.
I think the main question here is: What can we do today to make the world better in the future? If you believe AI could make the world a lot worse, or even just lock in the already existing state, it seems really valuable to do work on that not happening. If you additionally believe AI could solve problems such as wild animal suffering or unhappy humans then it seems like an even more area problem to spend your time on.
(I think this might be less clear for biorisk where the main concern really is extinction.)
One final thing: Generally, I think people make a distinction between existential risk (roughly: permanent, irreversible, and drastic loss of value of the future) and extinction risk (extinction of humans), where extinction risk is just one type of existential risk.
Even if you think all sentient life is net negative, extinction is not a wise choice. Unless you completely destroy Earth, animal life will probably evolve again, so there will be suffering in the future.
Moreover, what if there are sentient aliens somewhere? What if some form of panpsychism is true and there is consciousness embedded in most systems? What if some multiverse theory is true?
If you want to truly end suffering, your best bet would be something like creating a non sentient AGI that transforms everything into some nonsentient matter, and then spends eternity thinking and experimenting to determine if there are other universes or other pockets of suffering, and how to influence them.
Of course this would entail human extinction too, but it’s a very precise form of extinction. Even if you create an AGI, it would have to be aligned with your suffering-minimizing ethics.
So for now, even if you think life is net negative, preventing ourselves from losing control of the future is a very important instrumental goal. And anything that threatens that control, even if it’s not an existential threat, should be avoided.
Congrats on your first post! I appreciate reading your perspective on this – it’s well articulated.
I think I disagree about how likely existential risk from advanced AI is. You write:
Given that life is capable of thriving all on its own via evolution, AI would have to see the existence of any life as a threat for it to actively pursue extinction
In my view, an AGI (artificial general intelligence) is a self-aware agent with a set of goals and the capability to pursue those goals very well. Sure, if such an agent views humans as a threat to its own existence it would wipe us out. It might also wipe us out because we slightly get in the way of some goal it’s pursuing. Humans have very complex values, and it is quite difficult to match an AI’s values to human values. I am somewhat worried that an AI would kill us all not because it hates us but because we are a minor nuisance to its pursuit of unrelated goals.
When humans bulldoze an ant hill in order to make a highway, it’s not because we hate the ants or are threatened by them. It’s because they’re in the way of what we’re trying to do. Humans tend to want to control the future, so if I were an advanced AI trying to optimize for some values, and they weren’t the same exact values humans have, it might be easiest to just get rid of the competition – we’re not that hard to kill.
I think this is one story of why AI poses existential risk, but there are many more. For further reading, I quite like Carlsmith’s piece! Again, welcome to the forum!
Existential risk might be worth talking about because of normative uncertainty. Not all EAs are necessarily hedonists, and perhaps the ones who are shouldn’t be, for reasons to be discovered later. So, if we don’t know what “value” is, or, as a movement, EA doesn’t “know” what “value” is, a priori, we might want to keep our options open, and if everyone is dead, then we can’t figure out what “value” really is or ought to be.
Hi, welcome to the forum.
You raise some interesting points. Some quick notes/counterpoints:
Not all existential risk is extinction risk.
Existential risk doesn’t have an extremely clean definition, but in the simple extinction/doom/non-utopia ontology, most longtermist EA’s intuitive conception of “existential risk” is closer to risk of “doom” than “extinction”
Nuclear war may not be a large direct existential risk, but it’s an existential risk factor.
The world could be made scarier after large-scale nuclear war, and thus less hospitable for altruistic values (plus other desiderata)
AI may or may not kill us all. But this point is academic and only mildly important, because if unaligned AI takes over, we (humanity and our counterfactual descendants) have lost control of the future.
Almost all moral value in the future is in the tails (extremely good and extremely bad outcomes).
Those outcomes likely require optimization for, and it seems likely that our spiritual descendants optimize heavily for good stuff than bad stuff.
Bad stuff might happen incidentally (historical analogues include factory farming and slavery), but they aren’t being directly optimized for, so will be a small fraction of the badness of maximally bad outcomess.
Thank you for the response!
Yeah I think I have the most problem with (4), something that I probably should have expressed more in the post.
It’s true that humans are in theory trying to optimize for good outcomes, and this is a reason to expect utility to diverge to infinity. However, there are in my view equally good reasons utility to diverge to negative infinity- that being that the world is not designed for humans. We are inherently fragile creatures, only suitable to live in a world with specific temperature, air composition, etc. There are a lot of large-scale phenomenon causing these factors to change—s-risks—that could send utility plunging. This, plus the fact that current utility is below 0, means that I think existential risk is probably a moral benefit.
I also agree that this whole thing is pretty pedantic, especially in cases like AI domination.
“the world is not designed for humans”
I think our descendants will unlikely be flesh-and-blood humans but rather digital forms of sentience: https://www.cold-takes.com/how-digital-people-could-change-the-world/
I think the main question here is: What can we do today to make the world better in the future? If you believe AI could make the world a lot worse, or even just lock in the already existing state, it seems really valuable to do work on that not happening. If you additionally believe AI could solve problems such as wild animal suffering or unhappy humans then it seems like an even more area problem to spend your time on.
(I think this might be less clear for biorisk where the main concern really is extinction.)
Here is my take on the value of extinction risk reduction, from some years ago: https://forum.effectivealtruism.org/posts/NfkEqssr7qDazTquW/the-expected-value-of-extinction-risk-reduction-is-positive
This posts also contains links to many other posts related to the topic.
Some other posts, that come to different conclusions:
https://forum.effectivealtruism.org/posts/BY8gXSpGijypbGitT/why-i-prioritize-moral-circle-expansion-over-artificial
https://forum.effectivealtruism.org/posts/RkPK8rWigSAybgGPe/a-longtermist-critique-of-the-expected-value-of-extinction-2
One final thing: Generally, I think people make a distinction between existential risk (roughly: permanent, irreversible, and drastic loss of value of the future) and extinction risk (extinction of humans), where extinction risk is just one type of existential risk.
Even if you think all sentient life is net negative, extinction is not a wise choice. Unless you completely destroy Earth, animal life will probably evolve again, so there will be suffering in the future.
Moreover, what if there are sentient aliens somewhere? What if some form of panpsychism is true and there is consciousness embedded in most systems? What if some multiverse theory is true?
If you want to truly end suffering, your best bet would be something like creating a non sentient AGI that transforms everything into some nonsentient matter, and then spends eternity thinking and experimenting to determine if there are other universes or other pockets of suffering, and how to influence them.
Of course this would entail human extinction too, but it’s a very precise form of extinction. Even if you create an AGI, it would have to be aligned with your suffering-minimizing ethics.
So for now, even if you think life is net negative, preventing ourselves from losing control of the future is a very important instrumental goal. And anything that threatens that control, even if it’s not an existential threat, should be avoided.
Congrats on your first post! I appreciate reading your perspective on this – it’s well articulated.
I think I disagree about how likely existential risk from advanced AI is. You write:
In my view, an AGI (artificial general intelligence) is a self-aware agent with a set of goals and the capability to pursue those goals very well. Sure, if such an agent views humans as a threat to its own existence it would wipe us out. It might also wipe us out because we slightly get in the way of some goal it’s pursuing. Humans have very complex values, and it is quite difficult to match an AI’s values to human values. I am somewhat worried that an AI would kill us all not because it hates us but because we are a minor nuisance to its pursuit of unrelated goals.
When humans bulldoze an ant hill in order to make a highway, it’s not because we hate the ants or are threatened by them. It’s because they’re in the way of what we’re trying to do. Humans tend to want to control the future, so if I were an advanced AI trying to optimize for some values, and they weren’t the same exact values humans have, it might be easiest to just get rid of the competition – we’re not that hard to kill.
I think this is one story of why AI poses existential risk, but there are many more. For further reading, I quite like Carlsmith’s piece! Again, welcome to the forum!
Existential risk might be worth talking about because of normative uncertainty. Not all EAs are necessarily hedonists, and perhaps the ones who are shouldn’t be, for reasons to be discovered later. So, if we don’t know what “value” is, or, as a movement, EA doesn’t “know” what “value” is, a priori, we might want to keep our options open, and if everyone is dead, then we can’t figure out what “value” really is or ought to be.