I think this quantitative model has some potential and itās a great addition to the growing literature on cause selection. Thanks for taking the time Michael :)
One aspect of this model which I find problematic, and I feel is something that may be often overlooked when calculating the EV of the far future, is that there is some Cumulative Probability Of Non Existence (CPONE) that is currently not accounted for in the probabilities listed in the EV of the far future spreadsheet.
The CPONE relies on the following:
The extinction risk probability per year is always greater than 0 because extinction is possible in any year.
Extinction in any year means extinction in all future years. This property is what makes the probability of non-existence cumulative. By cumulative I mean it increases each additional year it is forecast into the future.
It follows that the probability of value existing x years in the future decreases as x increases.
I donāt have a great sense of what I feel the probability for extinction of humans or their descendants into the far future is but assigning zero probability to this outcome in the far future spreadsheet conflicts with my initial thoughts on this topic. For instance, available estimates seem to put the current annual probability of extinction at ~10^-4 and it seems that even much smaller annual extinction probabilities accumulate over large timescales to become significant.
These probabilities of extinction matter because future EV comes from the ā(estimated value at future time point multiplied by the probability of value existing at that future time point) for all future time points. If we feel that, say, 10^10 years into the future thereās a 0.5 probability humans or their descendants are extinct then all estimated values after that time point have to be multiplied by <0.5 in order to find their EV.
Given this, I think thereās some chance that the inclusion of reasonable CPONE models into far future EV calculations can cause orders of magnitude difference relative to not including CPONE models.
Please note, I am not sure the points I made in this comment are correct. I havenāt thought about/ā researched this much and as such thereās certainly a chance that I will update in future. Itās unclear to me what impact including CPONE on EV of the far future has, maybe one day I will attempt some calculations myself. I currently assign significant probability to it causing orders of magnitude difference and that makes me feel like CPONE should be attempted to be included in models like this. Another solution would be to make it clearer how the model is dealing with extinction probabilities into the far future and how this may conflict with some peopleās views.
The CPONE model looks interesting. It does seem somewhat reasonable to say that if we assign some non-trivial probability to extinction each year, then this probability accumulates over time such that we are almost guaranteed to go extinct eventually. Ultimately, though, I donāt believe this is a correct assumption.
First, just to clarify, I donāt believe that the human species going extinct per se is bad. If humans go extinct but some intelligent benevolent[^1] species continues to exist, that would be good too. So the real concern is that valuable beings stop existing, not that humans stop existing. (Some people might claim that humans are the only valuable beings; I think this is pretty wrong but thereās not much to argue here.)
If we successfully start colonizing other planets, our probability of extinction decreases as we grow. While itās true that we have a nonzero probability of extinction on any given year, this probability decreases over time. I believe a plausible cumulative distribution function for the probability of extinction would have an asymptoteāor else something like an asymptote, e.g., the probability of extinction between 100 and 1000 years from now is about the same as the probability of extinction between 1000 and 10,000 years from now, etc.
Perhaps we can model the probability of extinction by looking at history. Almost all individual species have gone extinct so if we treat humans as a regular species then our extinction looks pretty likely. But I believe this is the wrong reference class. We shouldnāt just care about humans, we should care about good happy beings in general. I think multi-cellular life on earth makes for a better reference class. Multi-cellular life has taken several big hits over the years, but itās always bounced back. Now, itās plausible that life so far has been net bad, but the point is that this gives us good theoretical reason to believe that we can avoid extinction for potentially billions of years.
Still, maybe youād say weāve have like a 90% chance of going extinct in the next 10^10 years conditional on making it through the next century. That sounds pretty defensible to me. That doesnāt actually change the outputs of the model as much as you might think. When youāre looking at interventionsā effects on the far future, the numbers are so big that the prior does a lot of workā10^54 and 10^55 expected utility donāt look that different after updating on the prior. (I believe this is the correct model: if you thought GiveDirectly was a little better than AI risk, but then I told you that I updated my estimated utility of the far future from 10^54 to 10^55, would that change your mind? But it would have a bigger impact if you used a much wider prior distribution.) Plus far-future interventionsā effects tend to be correlated with each other, so this changes how good they all look but doesnāt do as much to change how you prioritize them.
[^1] Itās sort of unclear how benevolent humans are; we do have factory farming after all. Iām hoping we get rid of that before too long.
I believe a plausible cumulative distribution function for the probability of extinction would have an asymptoteāor else something like an asymptote, e.g., the probability of extinction between 100 and 1000 years from now is about the same as the probability of extinction between 1000 and 10,000 years from now, etc.
Using that example the probability of value existing could be roughly modelled as:
p(n)= (1-r)^(log(n-1))
Where p is the probability of value existing n years into the future, r is the extinction probability between 10 and 100 years, log means the log base 10 and n is the number of years in the future. This relationship works for n>2.
I was curious about what the average of p(n) for that type of function would be over the next 10^11 years. Some available extinction estimates put r between 10% and 50%. I imagine thereās also similar variance within EAsā r value. Using r= 10% the average of p(n) over 10^11 years seems like it would be 3 10^-1 . Using r= 50% the average of p(n) over 10^11 years would be ~7 10^-4. I used Wolfram Alphaās integral calculator for these calculations and I am not that itās performing the calculation correctly. These averages for p(n) could make the impact on far future EV significant.
I donāt have strong views on which CPONE model is best and the ones I mention here may be flawed. I softly lean towards including CPONE models because the posterior then more closely reflects the userās view of reality, itās not too difficult to include CPONE models, reasonable people may have different CPONE models, and the addition of a CPONE model may result in different cause prioritization conclusions.
I think multi-cellular life on earth makes for a better reference class. Multi-cellular life has taken several big hits over the years, but itās always bounced back.
Interesting. I hadnāt thought of that reference class before :)
When youāre looking at interventionsā effects on the far future, the numbers are so big that the prior does a lot of workā10^54 and 10^55 expected utility donāt look that different after updating on the prior.
Excellent point :) I wasnāt fully taking that into consideration. Updates me towards thinking that CPONE models are less important than previously thought. I think reasonable people could have a CPONE model which causes more than one order of magnitude difference in EV and therefore cause a more significant difference after updating on the prior.
[edited originally I accidentally used the natural logarithm instead of log base 10 when calculating the average of the probability function over 10^11 years]
The constant (irreducible) extinction risk model is a good one, but hereās the catch. You donāt really know how low the irreducible risk is once all risk-reducing interventions are made. It could go down to 0.1% annually, (suppose there are some unavoidable sun flares or other cosmic disasters) or it could go down to near-enough to zero. We donāt know which kind of world weād end up in, but if you care a lot about the long-run future, itās the latter family of worlds that youāre aiming to save, and on expected value, the notion of those first worlds (whose cosmic endowments are kinda doomed :p) donāt really matter. So the point if that the constant irreducable extinction risk model is just good enough that it only needs a slight upgrade to get the right (opposite) answer!
With uncertainty about that extinction rate this Weitzman paperās argument is relevant:
A critical feature of the distant future is currently unresolvable uncertainty about what will
then be the appropriate rate of return on capital to use for discounting. This paper shows that
there is a well-defined sense in which the āālowest possibleāā interest rate should be used for
discounting the far-distant future part of any investment project. Some implications are
discussed for evaluating long-term environmental projects or activities, like measures to
mitigate the possible effects of global climate change
Why āonce all risk-reducing measures are madeā? Presumably what we care about is the marginal risk-reduction measure we can make on the margin?
I see no reason to think returns here are close to linear, since a reduction in the extinction rate from 0.2% to 0.1% (500 years ā 1000 years) delivers half the benefits of going from 0.1% to 0.05% (1000 years ā 2000 years) which is half of the benefits of going from 0.05% to 0.025%, etc. So my very weak prior on āmarginal return on effort spent reducing extinction riskā would be that they would be roughly exponentially increasing with the overall magnitude of resources thrown at the problem.
Which means I donāt think you can take the usual shortcut of saying āif 10% of world resources were spent on this it would be a great return on investment, and diminishing returns, so me spending 0.00001% of the worlds resources is also a great returnā.
With that said, massively increasing returns is extremely unusual and feels intuitively odd so Iām very open to alternative models; this came up recently at a London EA discussion as a major objection to some of the magnitudes thrown around in x-risk causes, but I still donāt have a great sense of what alternative models might look like.
Yeah introducing diminishing returns into a model could change the impact by an order of magnitude but Iām trying to answer a more binary question: which
What Iām trying to look at is will an intervention to x-risk has a ālong-run impactā e.g. either approx. the cosmic endowment or approx. the current milleneum. If you use a constant discount or an exponential discount, thatās going to make all of the difference. And if you think thereās some amount of existential risk thatās irreducible, that forces you to include some exponential discounting. So itās kind-of different from where youāre trying to lead things.
I think this quantitative model has some potential and itās a great addition to the growing literature on cause selection. Thanks for taking the time Michael :)
One aspect of this model which I find problematic, and I feel is something that may be often overlooked when calculating the EV of the far future, is that there is some Cumulative Probability Of Non Existence (CPONE) that is currently not accounted for in the probabilities listed in the EV of the far future spreadsheet.
The CPONE relies on the following:
The extinction risk probability per year is always greater than 0 because extinction is possible in any year.
Extinction in any year means extinction in all future years. This property is what makes the probability of non-existence cumulative. By cumulative I mean it increases each additional year it is forecast into the future.
It follows that the probability of value existing x years in the future decreases as x increases.
I donāt have a great sense of what I feel the probability for extinction of humans or their descendants into the far future is but assigning zero probability to this outcome in the far future spreadsheet conflicts with my initial thoughts on this topic. For instance, available estimates seem to put the current annual probability of extinction at ~10^-4 and it seems that even much smaller annual extinction probabilities accumulate over large timescales to become significant.
These probabilities of extinction matter because future EV comes from the ā(estimated value at future time point multiplied by the probability of value existing at that future time point) for all future time points. If we feel that, say, 10^10 years into the future thereās a 0.5 probability humans or their descendants are extinct then all estimated values after that time point have to be multiplied by <0.5 in order to find their EV.
Given this, I think thereās some chance that the inclusion of reasonable CPONE models into far future EV calculations can cause orders of magnitude difference relative to not including CPONE models.
Please note, I am not sure the points I made in this comment are correct. I havenāt thought about/ā researched this much and as such thereās certainly a chance that I will update in future. Itās unclear to me what impact including CPONE on EV of the far future has, maybe one day I will attempt some calculations myself. I currently assign significant probability to it causing orders of magnitude difference and that makes me feel like CPONE should be attempted to be included in models like this. Another solution would be to make it clearer how the model is dealing with extinction probabilities into the far future and how this may conflict with some peopleās views.
[Edited to master bullet point formatting]
The CPONE model looks interesting. It does seem somewhat reasonable to say that if we assign some non-trivial probability to extinction each year, then this probability accumulates over time such that we are almost guaranteed to go extinct eventually. Ultimately, though, I donāt believe this is a correct assumption.
First, just to clarify, I donāt believe that the human species going extinct per se is bad. If humans go extinct but some intelligent benevolent[^1] species continues to exist, that would be good too. So the real concern is that valuable beings stop existing, not that humans stop existing. (Some people might claim that humans are the only valuable beings; I think this is pretty wrong but thereās not much to argue here.)
If we successfully start colonizing other planets, our probability of extinction decreases as we grow. While itās true that we have a nonzero probability of extinction on any given year, this probability decreases over time. I believe a plausible cumulative distribution function for the probability of extinction would have an asymptoteāor else something like an asymptote, e.g., the probability of extinction between 100 and 1000 years from now is about the same as the probability of extinction between 1000 and 10,000 years from now, etc.
Perhaps we can model the probability of extinction by looking at history. Almost all individual species have gone extinct so if we treat humans as a regular species then our extinction looks pretty likely. But I believe this is the wrong reference class. We shouldnāt just care about humans, we should care about good happy beings in general. I think multi-cellular life on earth makes for a better reference class. Multi-cellular life has taken several big hits over the years, but itās always bounced back. Now, itās plausible that life so far has been net bad, but the point is that this gives us good theoretical reason to believe that we can avoid extinction for potentially billions of years.
Still, maybe youād say weāve have like a 90% chance of going extinct in the next 10^10 years conditional on making it through the next century. That sounds pretty defensible to me. That doesnāt actually change the outputs of the model as much as you might think. When youāre looking at interventionsā effects on the far future, the numbers are so big that the prior does a lot of workā10^54 and 10^55 expected utility donāt look that different after updating on the prior. (I believe this is the correct model: if you thought GiveDirectly was a little better than AI risk, but then I told you that I updated my estimated utility of the far future from 10^54 to 10^55, would that change your mind? But it would have a bigger impact if you used a much wider prior distribution.) Plus far-future interventionsā effects tend to be correlated with each other, so this changes how good they all look but doesnāt do as much to change how you prioritize them.
[^1] Itās sort of unclear how benevolent humans are; we do have factory farming after all. Iām hoping we get rid of that before too long.
Using that example the probability of value existing could be roughly modelled as: p(n)= (1-r)^(log(n-1)) Where p is the probability of value existing n years into the future, r is the extinction probability between 10 and 100 years, log means the log base 10 and n is the number of years in the future. This relationship works for n>2.
I was curious about what the average of p(n) for that type of function would be over the next 10^11 years. Some available extinction estimates put r between 10% and 50%. I imagine thereās also similar variance within EAsā r value. Using r= 10% the average of p(n) over 10^11 years seems like it would be 3 10^-1 . Using r= 50% the average of p(n) over 10^11 years would be ~7 10^-4. I used Wolfram Alphaās integral calculator for these calculations and I am not that itās performing the calculation correctly. These averages for p(n) could make the impact on far future EV significant.
I donāt have strong views on which CPONE model is best and the ones I mention here may be flawed. I softly lean towards including CPONE models because the posterior then more closely reflects the userās view of reality, itās not too difficult to include CPONE models, reasonable people may have different CPONE models, and the addition of a CPONE model may result in different cause prioritization conclusions.
Interesting. I hadnāt thought of that reference class before :)
Excellent point :) I wasnāt fully taking that into consideration. Updates me towards thinking that CPONE models are less important than previously thought. I think reasonable people could have a CPONE model which causes more than one order of magnitude difference in EV and therefore cause a more significant difference after updating on the prior.
[edited originally I accidentally used the natural logarithm instead of log base 10 when calculating the average of the probability function over 10^11 years]
The constant (irreducible) extinction risk model is a good one, but hereās the catch. You donāt really know how low the irreducible risk is once all risk-reducing interventions are made. It could go down to 0.1% annually, (suppose there are some unavoidable sun flares or other cosmic disasters) or it could go down to near-enough to zero. We donāt know which kind of world weād end up in, but if you care a lot about the long-run future, itās the latter family of worlds that youāre aiming to save, and on expected value, the notion of those first worlds (whose cosmic endowments are kinda doomed :p) donāt really matter. So the point if that the constant irreducable extinction risk model is just good enough that it only needs a slight upgrade to get the right (opposite) answer!
With uncertainty about that extinction rate this Weitzman paperās argument is relevant:
Why āonce all risk-reducing measures are madeā? Presumably what we care about is the marginal risk-reduction measure we can make on the margin?
I see no reason to think returns here are close to linear, since a reduction in the extinction rate from 0.2% to 0.1% (500 years ā 1000 years) delivers half the benefits of going from 0.1% to 0.05% (1000 years ā 2000 years) which is half of the benefits of going from 0.05% to 0.025%, etc. So my very weak prior on āmarginal return on effort spent reducing extinction riskā would be that they would be roughly exponentially increasing with the overall magnitude of resources thrown at the problem.
Which means I donāt think you can take the usual shortcut of saying āif 10% of world resources were spent on this it would be a great return on investment, and diminishing returns, so me spending 0.00001% of the worlds resources is also a great returnā.
With that said, massively increasing returns is extremely unusual and feels intuitively odd so Iām very open to alternative models; this came up recently at a London EA discussion as a major objection to some of the magnitudes thrown around in x-risk causes, but I still donāt have a great sense of what alternative models might look like.
Yeah introducing diminishing returns into a model could change the impact by an order of magnitude but Iām trying to answer a more binary question: which
What Iām trying to look at is will an intervention to x-risk has a ālong-run impactā e.g. either approx. the cosmic endowment or approx. the current milleneum. If you use a constant discount or an exponential discount, thatās going to make all of the difference. And if you think thereās some amount of existential risk thatās irreducible, that forces you to include some exponential discounting. So itās kind-of different from where youāre trying to lead things.