A risk of catastrophe where an adverse outcome would permanently cause Earth-originating intelligent life’s astronomical value to be <50% of what it would otherwise be capable of.
I’m not a fan of this definition, because I find it very plausible that the expected value of the future is less than 50% of what humanity is capable of. Which e.g. raises the question: does even extinction fulfil the description? Maybe you could argue “yes”: but the mix of causing an actual outcome compared with what intelligent life is “capable of” makes all of this unnecessarily dependant on both definitions and empirics about the future.
For purposes of the original question, I don’t think we need to deal with all the complexity around “curtailing potential”. You can just ask: How much should a funder be willing to pay to remove an 0.01% risk of extinction that’s independent from all other extinction risks we’re facing. (Eg., a giganormous asteroid is on its way to Earth and has an 0.01% probability of hitting us, causing guaranteed extinction. No on else will notice this in time. Do we pay $X to redirect it?)
This seems closely analogous to questions that funders are facing (are we keen to pay to slightly reduce one, contemporary extinction risk). For non-extinction x-risk reduction, this extinction-estimate will be informative as a comparison point, and it seems completely appropriate that you should also check “how bad is this purported x-risk compared to extinction” as a separate exercise.
Seems better than the previous one, though imo still worse than my suggestion, for 3 reasons:
it’s more complex than asking about immediate extinction. (Why exactly 100 year cutoff? why 50%?)
since the definition explicitly allows for different x-risks to be differently bad, the amount you’d pay to reduce them would vary depending on the x-risk. So the question is underspecified.
The independence assumption is better if funders often face opportunities to reduce a Y%-risk that’s roughly independent from most other x-risk this century. Your suggestion is better if funders often face opportunities to reduce Y percentage points of all x-risk this century (e.g. if all risks are completely disjunctive, s.t. if you remove a risk, you’re guaranteed to not be hit by any other risk).
For your two examples, the risks from asteroids and climate change are mostly independent from the majority of x-risk this century, so there the independence assumption is better.
The disjunctive assumption can happen if we e.g. study different mutually exclusive cases, e.g. reducing risk from worlds with fast AI take-off vs reducing risk from worlds with slow AI take-off.
I weakly think that the former is more common.
(Note that the difference only matters if total x-risk this century is large.)
Edit: This is all about what version of this question is the best version, independent of inertia. If you’re attached to percentage points because you don’t want to change to an independence assumption after there’s already been some discussion on the post, then this your latest suggestion seems good enough. (Though I think most people have been assuming low total amount of x-risk, so probably independence or not doesn’t matter that much for the existing discussion.)
Currently, the post says:
I’m not a fan of this definition, because I find it very plausible that the expected value of the future is less than 50% of what humanity is capable of. Which e.g. raises the question: does even extinction fulfil the description? Maybe you could argue “yes”: but the mix of causing an actual outcome compared with what intelligent life is “capable of” makes all of this unnecessarily dependant on both definitions and empirics about the future.
For purposes of the original question, I don’t think we need to deal with all the complexity around “curtailing potential”. You can just ask: How much should a funder be willing to pay to remove an 0.01% risk of extinction that’s independent from all other extinction risks we’re facing. (Eg., a giganormous asteroid is on its way to Earth and has an 0.01% probability of hitting us, causing guaranteed extinction. No on else will notice this in time. Do we pay $X to redirect it?)
This seems closely analogous to questions that funders are facing (are we keen to pay to slightly reduce one, contemporary extinction risk). For non-extinction x-risk reduction, this extinction-estimate will be informative as a comparison point, and it seems completely appropriate that you should also check “how bad is this purported x-risk compared to extinction” as a separate exercise.
How do people feel about a proposed new definition:
Seems better than the previous one, though imo still worse than my suggestion, for 3 reasons:
it’s more complex than asking about immediate extinction. (Why exactly 100 year cutoff? why 50%?)
since the definition explicitly allows for different x-risks to be differently bad, the amount you’d pay to reduce them would vary depending on the x-risk. So the question is underspecified.
The independence assumption is better if funders often face opportunities to reduce a Y%-risk that’s roughly independent from most other x-risk this century. Your suggestion is better if funders often face opportunities to reduce Y percentage points of all x-risk this century (e.g. if all risks are completely disjunctive, s.t. if you remove a risk, you’re guaranteed to not be hit by any other risk).
For your two examples, the risks from asteroids and climate change are mostly independent from the majority of x-risk this century, so there the independence assumption is better.
The disjunctive assumption can happen if we e.g. study different mutually exclusive cases, e.g. reducing risk from worlds with fast AI take-off vs reducing risk from worlds with slow AI take-off.
I weakly think that the former is more common.
(Note that the difference only matters if total x-risk this century is large.)
Edit: This is all about what version of this question is the best version, independent of inertia. If you’re attached to percentage points because you don’t want to change to an independence assumption after there’s already been some discussion on the post, then this your latest suggestion seems good enough. (Though I think most people have been assuming low total amount of x-risk, so probably independence or not doesn’t matter that much for the existing discussion.)