I think one problem with your re-definition (that makes it imperfect IMO also) is apparent when thinking about the following questions: How likely is it that Earth-originating intelligent life eventually reaches >90% of its potential? How likely is it that it eventually reaches >0.1% of its potential? >0.0001% of its potential? >10^20 the total value of the conscious experiences of all humans with net-positive lives during the year 2020? My answers to these questions increase with each respective question, and my answer to the last question is several times higher than my answer to the first question.
Our cosmic potential is potentially extremely large, and there are many possible “very long-lasting, positive futures” (to use Holden’s language from here) that seem “extremely good” from our limited perspective today (e.g. the futures we imagine when we read Bostrom’s Letter from Utopia). But these futures potentially differ in value tremendously.
Okay, I just saw Zach’s comment that he thinks value is roughly binary. I currently don’t think I agree with him (see his first paragraph at that link, and my reply clarifying my view). Maybe my view is unusual?
I’d be interested in seeing operationalizations at some subset of {1%, 10%, 50%, 90, 99%}.* I can imagine that most safety researchers will give nearly identical answers to all of them, but I can also imagine that large divergences, so decent value of information here.
I’d give similar answers for all 5 of those questions because I think most of the “existential catastrophes” (defined vaguely) involve wiping out >>99% of our potential (e.g. extinction this century before value/time increases substantially). But my independent impression is that there are a lot of “extremely good” outcomes in which we have a very long-lasting, positive future with value/year much, much greater than the value per year on Earth today, that also falls >99% short of our potential (and even >99.9999% of our potential).
Great point. Ideally “existential risk” should be an entirely empirical thing that we can talk about independent of our values / moral beliefs about what future is optimal.
This is impossible if you consider “unrecoverable dystopia”, “stable totalitarianism” etc as existential risks, as these things are implicitly values judgments.
Though I’m open to the idea that we should maybe talk about extinction risks instead of existential risks instead, given that this is empirically most of what xrisk people work on.
(Though I think some AI risk people think, as an empirical matter, that some AI catastrophes would entail humanity surviving while completely losing control for the lightcone, and both they and I would consider this basically as bad as all of our descendants dying).
I think one problem with your re-definition (that makes it imperfect IMO also) is apparent when thinking about the following questions: How likely is it that Earth-originating intelligent life eventually reaches >90% of its potential? How likely is it that it eventually reaches >0.1% of its potential? >0.0001% of its potential? >10^20 the total value of the conscious experiences of all humans with net-positive lives during the year 2020? My answers to these questions increase with each respective question, and my answer to the last question is several times higher than my answer to the first question.
Our cosmic potential is potentially extremely large, and there are many possible “very long-lasting, positive futures” (to use Holden’s language from here) that seem “extremely good” from our limited perspective today (e.g. the futures we imagine when we read Bostrom’s Letter from Utopia). But these futures potentially differ in value tremendously.
Okay, I just saw Zach’s comment that he thinks value is roughly binary. I currently don’t think I agree with him (see his first paragraph at that link, and my reply clarifying my view). Maybe my view is unusual?
Linch here:
I’d give similar answers for all 5 of those questions because I think most of the “existential catastrophes” (defined vaguely) involve wiping out >>99% of our potential (e.g. extinction this century before value/time increases substantially). But my independent impression is that there are a lot of “extremely good” outcomes in which we have a very long-lasting, positive future with value/year much, much greater than the value per year on Earth today, that also falls >99% short of our potential (and even >99.9999% of our potential).
It’s also possible that different people have different views of what “humanity’s potential” really means!
Great point. Ideally “existential risk” should be an entirely empirical thing that we can talk about independent of our values / moral beliefs about what future is optimal.
This is impossible if you consider “unrecoverable dystopia”, “stable totalitarianism” etc as existential risks, as these things are implicitly values judgments.
Though I’m open to the idea that we should maybe talk about extinction risks instead of existential risks instead, given that this is empirically most of what xrisk people work on.
(Though I think some AI risk people think, as an empirical matter, that some AI catastrophes would entail humanity surviving while completely losing control for the lightcone, and both they and I would consider this basically as bad as all of our descendants dying).