Iām a bit confused by this response. Are you just saying that high expected value is not sufficient for actual value because we might get unlucky?
Some possible extinction-averting events merely extend life for 1 second, and so provide little value. Other possibilities offer far greater extensions. Obviously the latter possibilities are the ones that ground high expected value estimates.
I think Shulmanās points give us reason to think thereās a non-negligible chance of averting extinction (extending civilization) for a long time. Pointing out that other possibilities are also possible doesnāt undermine this claim.
Are you just saying that high expected value is not sufficient for actual value because we might get unlucky?
No. I agree high expected value does not ensure high actual value, but I only care about the expected (not the actual) increase in welfare.
Other possibilities offer far greater extensions. Obviously the latter possibilities are the ones that ground high expected value estimates.
This is far from obvious to me, and I wonder what makes you so confident. I am not aware of any reasonably quantitative and empirical modelling of the expected increase in future welfare under a longtermist perspective. I expect what David Thorstad calls rapid diminution. I see the difference between the probability density function (PDF) after and before an intervention reducing the nearterm risk of human extinction as quickly decaying to 0, thus making the increase in the expected value of the astronomically valuable worlds negligible. For instance:
If the difference between the PDF after and before the intervention decays exponentially with the value of the future v, the increase in the expected value density caused by the intervention will be proportional to v*e^-v.
The above rapidly goes to 0 as v increases. For an expected value of the future equal to my estimate of 1.40*10^52 human lives, the increase in the expected value density will involve a factor of 1.40*10^52*e^(-1.40*10^52) = 10^(log10(1.40) + 52 - log10(e)*1.40*10^52) = 10^(-6.08*10^51), i.e. it will be basically 0.
I think Shulmanās points give us reason to think thereās a non-negligible chance of averting extinction (extending civilization) for a long time. Pointing out that other possibilities are also possible doesnāt undermine this claim.
It seems that you are inferring one could meaningfully increase the probability of astronomically valuable futures because these are plausible. However, astronomically valuable futures could be plausible while having a probability that is very hard to change.
You keep describing possible scenarios in which the actual value of averting extinction would be low. Do you not understand the scenario conditions under which the actual value would be high? Iām suggesting that those kinds of scenario (in which astronomically valuable futures are reliably securable so long as we avoid extinction this century) seem reasonably credible to me, which is what grounds my background belief in x-risk mitigation having high expected value.
Edited to add: You may in a sense be talking about āexpected valueā, but insofar as it is based on a single model of how you expect the future to go (ārapid diminutionā), it seems problematically analogous to actual value ā like the fallacy I diagnose in āRule High Stakes In, Not Outā. We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.
The point of RHSINO is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldnāt establish that our all-models-considered EV must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/āscenarios.
Do you not understand the scenario conditions under which the actual value would be high?
I do. In theory, there could be worlds which i) will actually be astronomically valuable given some efforts, but that ii) will be practically actually neutral without such efforts. In this case, the efforts would be astronomically valuable due to meaningfully increasing the probability of astronomically valuable futures. I just think such worlds are super implausible. I can see many worlds satisfying i), but not i) and ii) simultaneously.
We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.
I agree. I illustrated my point with a single model in my past comment. However, I believe appropriately weighted sets of models lead to the same conclusion that the expected increase in welfare decreases with actual welfare when this is astronomical (because I think the increase in the probability of the actual welfare decreases faster than the actual welfare when this is astronomical).
The point of RHSINO [āRule High Stakes In, Not Outā] is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldnāt establish that our all-models-considered EV [expected value] must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/āscenarios.
I understand what you have in mind. If EV is 1ā0.99 under model A with 99 % weight, and 1 M under B with 1 % weight, A contributes 1 (= 0.99*1/ā0.99) to the EV considering both models, and B contributes 10 k (= 0.01*1*10^6). The contribution of A is 0.01 % (= 1/ā(1*10^4)) that of B. So changing by a given percentage the EV under B, or weight of B changes the EV considering both models much more that changing by the same percentage the EV under A, or weight of A. For example, doubling (increasing by 100 %) the EV under B would increase the EV considering both models by 1 (= 1*1), whereas doubling the EV under A would increase the EV considering both models by 10 k (= 1*10^4), 10 k (= 1*10^4/ā1) times the increase resulting from increasing the EV under B. The change in the EV considering all models resulting from changing the EV under a model, or its weight is proportional to the contribution of the model to the EV considering all models, and the relative change in the EV under the model, or its weight.
However, I think your presentation of RHSINO assumes the conclusion it is trying to support. High-stakes models are the ones that matter if they are the major driver of the EV under all models. However, the question is whether they are the major driver of the EV under all models. This has to be determined based on empirical evidence, not on points that are mathematically true.
People usually give weights that are at least 0.1/āānumber of modelsā, which easily results in high-stakes models dominating. However, giving weights which are not much smaller than the uniform weight of 1/āānumber of modelsā could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to ādistanceā^-2 (correct answer), ādistanceā^-20, or ādistanceā^-200, I imagine I would get a significant fraction picking the exponents of ā20 and ā200. Assuming 60 % picked ā2, 20 % picked ā20, and 20 % picked ā200, a respondant may naively conclude the mean exponent of ā45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Alternatively, a respondant may naively conclude an exponent of ā9.19 (= 0.933*(-2) + 0.0333*(-20) + 0.0333*(-200)) is reasonable giving a weight of 3.33 % (= 0.1/ā3) to each of the 2 wrong exponents, equal to 10 % of the uniform weight, and the remaining weight of 93.3 % (= 1 ā 2*0.0333) to the correct exponent. Yet, there is lots of empirical evidence against the exponents of ā45.2 and ā9.19 which the respondants are not aware of. The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks. This is also why I am sceptical that the absolute value of the welfare per unit time of animals is bound to be relatively close to that of humans, as one may naively infer from the welfare ranges Rethink Priorities (RP) initially presented, or the ones in Bob Fischerās book about comparing welfare across species, where there seems to be only 1 line about the weights. āWe assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive modelā.
Mistakes like the one illustrated above happen when the weights of models are guessed independently of their stakes. People are often sensitive to astronomically high stakes, but not to the astronomically low weights they imply.
Ok, thanks for expanding upon your view! It sounds broadly akin to how Iām inclined to address Pascalās Mugging cases (treat the astronomical stakes as implying proportionately negligible probability). Astronomical stakes from x-risk mitigation seems much more substantively credible to me, but I donāt have much to add at this point if you donāt share that substantive judgment!
It sounds broadly akin to how Iām inclined to address Pascalās Mugging cases (treat the astronomical stakes as implying proportionately negligible probability).
Makes sense. I see Pascalās muggings as instances where the probability of the offers is assessed indepently of their outcomes. In contrast, for any distribution with a finite expected value, the expected value density (product between the PDF and value) always ends up decaying to 0 as the outcome increases. In meta-analyses, effect sizes, which can be EVs under a given model, are commonly weighted by the reciprocal of their variance. Variance tends to increase with effect size, and therefore larger effect sizes are usually weighted less heavily.
Iām a bit confused by this response. Are you just saying that high expected value is not sufficient for actual value because we might get unlucky?
Some possible extinction-averting events merely extend life for 1 second, and so provide little value. Other possibilities offer far greater extensions. Obviously the latter possibilities are the ones that ground high expected value estimates.
I think Shulmanās points give us reason to think thereās a non-negligible chance of averting extinction (extending civilization) for a long time. Pointing out that other possibilities are also possible doesnāt undermine this claim.
Thanks for the follow-up, Richard.
No. I agree high expected value does not ensure high actual value, but I only care about the expected (not the actual) increase in welfare.
This is far from obvious to me, and I wonder what makes you so confident. I am not aware of any reasonably quantitative and empirical modelling of the expected increase in future welfare under a longtermist perspective. I expect what David Thorstad calls rapid diminution. I see the difference between the probability density function (PDF) after and before an intervention reducing the nearterm risk of human extinction as quickly decaying to 0, thus making the increase in the expected value of the astronomically valuable worlds negligible. For instance:
If the difference between the PDF after and before the intervention decays exponentially with the value of the future v, the increase in the expected value density caused by the intervention will be proportional to v*e^-v.
The above rapidly goes to 0 as v increases. For an expected value of the future equal to my estimate of 1.40*10^52 human lives, the increase in the expected value density will involve a factor of 1.40*10^52*e^(-1.40*10^52) = 10^(log10(1.40) + 52 - log10(e)*1.40*10^52) = 10^(-6.08*10^51), i.e. it will be basically 0.
It seems that you are inferring one could meaningfully increase the probability of astronomically valuable futures because these are plausible. However, astronomically valuable futures could be plausible while having a probability that is very hard to change.
You keep describing possible scenarios in which the actual value of averting extinction would be low. Do you not understand the scenario conditions under which the actual value would be high? Iām suggesting that those kinds of scenario (in which astronomically valuable futures are reliably securable so long as we avoid extinction this century) seem reasonably credible to me, which is what grounds my background belief in x-risk mitigation having high expected value.
Edited to add: You may in a sense be talking about āexpected valueā, but insofar as it is based on a single model of how you expect the future to go (ārapid diminutionā), it seems problematically analogous to actual value ā like the fallacy I diagnose in āRule High Stakes In, Not Outā. We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.
The point of RHSINO is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldnāt establish that our all-models-considered EV must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/āscenarios.
I do. In theory, there could be worlds which i) will actually be astronomically valuable given some efforts, but that ii) will be practically actually neutral without such efforts. In this case, the efforts would be astronomically valuable due to meaningfully increasing the probability of astronomically valuable futures. I just think such worlds are super implausible. I can see many worlds satisfying i), but not i) and ii) simultaneously.
I agree. I illustrated my point with a single model in my past comment. However, I believe appropriately weighted sets of models lead to the same conclusion that the expected increase in welfare decreases with actual welfare when this is astronomical (because I think the increase in the probability of the actual welfare decreases faster than the actual welfare when this is astronomical).
I understand what you have in mind. If EV is 1ā0.99 under model A with 99 % weight, and 1 M under B with 1 % weight, A contributes 1 (= 0.99*1/ā0.99) to the EV considering both models, and B contributes 10 k (= 0.01*1*10^6). The contribution of A is 0.01 % (= 1/ā(1*10^4)) that of B. So changing by a given percentage the EV under B, or weight of B changes the EV considering both models much more that changing by the same percentage the EV under A, or weight of A. For example, doubling (increasing by 100 %) the EV under B would increase the EV considering both models by 1 (= 1*1), whereas doubling the EV under A would increase the EV considering both models by 10 k (= 1*10^4), 10 k (= 1*10^4/ā1) times the increase resulting from increasing the EV under B. The change in the EV considering all models resulting from changing the EV under a model, or its weight is proportional to the contribution of the model to the EV considering all models, and the relative change in the EV under the model, or its weight.
However, I think your presentation of RHSINO assumes the conclusion it is trying to support. High-stakes models are the ones that matter if they are the major driver of the EV under all models. However, the question is whether they are the major driver of the EV under all models. This has to be determined based on empirical evidence, not on points that are mathematically true.
People usually give weights that are at least 0.1/āānumber of modelsā, which easily results in high-stakes models dominating. However, giving weights which are not much smaller than the uniform weight of 1/āānumber of modelsā could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to ādistanceā^-2 (correct answer), ādistanceā^-20, or ādistanceā^-200, I imagine I would get a significant fraction picking the exponents of ā20 and ā200. Assuming 60 % picked ā2, 20 % picked ā20, and 20 % picked ā200, a respondant may naively conclude the mean exponent of ā45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Alternatively, a respondant may naively conclude an exponent of ā9.19 (= 0.933*(-2) + 0.0333*(-20) + 0.0333*(-200)) is reasonable giving a weight of 3.33 % (= 0.1/ā3) to each of the 2 wrong exponents, equal to 10 % of the uniform weight, and the remaining weight of 93.3 % (= 1 ā 2*0.0333) to the correct exponent. Yet, there is lots of empirical evidence against the exponents of ā45.2 and ā9.19 which the respondants are not aware of. The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks. This is also why I am sceptical that the absolute value of the welfare per unit time of animals is bound to be relatively close to that of humans, as one may naively infer from the welfare ranges Rethink Priorities (RP) initially presented, or the ones in Bob Fischerās book about comparing welfare across species, where there seems to be only 1 line about the weights. āWe assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive modelā.
Mistakes like the one illustrated above happen when the weights of models are guessed independently of their stakes. People are often sensitive to astronomically high stakes, but not to the astronomically low weights they imply.
You may be interested in my chat with Matthew Adelstein. We discussed my scepticism of longtermism.
Ok, thanks for expanding upon your view! It sounds broadly akin to how Iām inclined to address Pascalās Mugging cases (treat the astronomical stakes as implying proportionately negligible probability). Astronomical stakes from x-risk mitigation seems much more substantively credible to me, but I donāt have much to add at this point if you donāt share that substantive judgment!
You are welcome!
Makes sense. I see Pascalās muggings as instances where the probability of the offers is assessed indepently of their outcomes. In contrast, for any distribution with a finite expected value, the expected value density (product between the PDF and value) always ends up decaying to 0 as the outcome increases. In meta-analyses, effect sizes, which can be EVs under a given model, are commonly weighted by the reciprocal of their variance. Variance tends to increase with effect size, and therefore larger effect sizes are usually weighted less heavily.
People sometimes point to Holden Karnofskyās post Why we canāt take expected value estimates literally (even when theyāre unbiased) to justify not relying on EVs (here are my notes on it from 4 years ago). However, the post does not broadly argue against using EVs. I see it as a call for not treating all EVs the same, and weighting them appropriately.