Do you not understand the scenario conditions under which the actual value would be high?
I do. In theory, there could be worlds which i) will actually be astronomically valuable given some efforts, but that ii) will be practically actually neutral without such efforts. In this case, the efforts would be astronomically valuable due to meaningfully increasing the probability of astronomically valuable futures. I just think such worlds are super implausible. I can see many worlds satisfying i), but not i) and ii) simultaneously.
We need to abstract from that specific model and ask how confident we should be in that one model compared to competing ones, and thus reach a kind of higher-order (or all models considered) expected value estimate.
I agree. I illustrated my point with a single model in my past comment. However, I believe appropriately weighted sets of models lead to the same conclusion that the expected increase in welfare decreases with actual welfare when this is astronomical (because I think the increase in the probability of the actual welfare decreases faster than the actual welfare when this is astronomical).
The point of RHSINO [‘Rule High Stakes In, Not Out’] is that the probability you assign to low stakes models (like rapid diminution) makes surprisingly little difference: you could assign them 99% probability and that still wouldn’t establish that our all-models-considered EV [expected value] must be low. Our ultimate judgment instead depends more on what credibility we assign to the higher-stakes models/scenarios.
I understand what you have in mind. If EV is 1⁄0.99 under model A with 99 % weight, and 1 M under B with 1 % weight, A contributes 1 (= 0.99*1/0.99) to the EV considering both models, and B contributes 10 k (= 0.01*1*10^6). The contribution of A is 0.01 % (= 1/(1*10^4)) that of B. So changing by a given percentage the EV under B, or weight of B changes the EV considering both models much more that changing by the same percentage the EV under A, or weight of A. For example, doubling (increasing by 100 %) the EV under B would increase the EV considering both models by 1 (= 1*1), whereas doubling the EV under A would increase the EV considering both models by 10 k (= 1*10^4), 10 k (= 1*10^4/1) times the increase resulting from increasing the EV under B. The change in the EV considering all models resulting from changing the EV under a model, or its weight is proportional to the contribution of the model to the EV considering all models, and the relative change in the EV under the model, or its weight.
However, I think your presentation of RHSINO assumes the conclusion it is trying to support. High-stakes models are the ones that matter if they are the major driver of the EV under all models. However, the question is whether they are the major driver of the EV under all models. This has to be determined based on empirical evidence, not on points that are mathematically true.
People usually give weights that are at least 0.1/”number of models”, which easily results in high-stakes models dominating. However, giving weights which are not much smaller than the uniform weight of 1/”number of models” could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to “distance”^-2 (correct answer), “distance”^-20, or “distance”^-200, I imagine I would get a significant fraction picking the exponents of −20 and −200. Assuming 60 % picked −2, 20 % picked −20, and 20 % picked −200, a respondant may naively conclude the mean exponent of −45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Alternatively, a respondant may naively conclude an exponent of −9.19 (= 0.933*(-2) + 0.0333*(-20) + 0.0333*(-200)) is reasonable giving a weight of 3.33 % (= 0.1/3) to each of the 2 wrong exponents, equal to 10 % of the uniform weight, and the remaining weight of 93.3 % (= 1 − 2*0.0333) to the correct exponent. Yet, there is lots of empirical evidence against the exponents of −45.2 and −9.19 which the respondants are not aware of. The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks. This is also why I am sceptical that the absolute value of the welfare per unit time of animals is bound to be relatively close to that of humans, as one may naively infer from the welfare ranges Rethink Priorities (RP) initially presented, or the ones in Bob Fischer’s book about comparing welfare across species, where there seems to be only 1 line about the weights. “We assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive model”.
Mistakes like the one illustrated above happen when the weights of models are guessed independently of their stakes. People are often sensitive to astronomically high stakes, but not to the astronomically low weights they imply.
Ok, thanks for expanding upon your view! It sounds broadly akin to how I’m inclined to address Pascal’s Mugging cases (treat the astronomical stakes as implying proportionately negligible probability). Astronomical stakes from x-risk mitigation seems much more substantively credible to me, but I don’t have much to add at this point if you don’t share that substantive judgment!
It sounds broadly akin to how I’m inclined to address Pascal’s Mugging cases (treat the astronomical stakes as implying proportionately negligible probability).
Makes sense. I see Pascal’s muggings as instances where the probability of the offers is assessed indepently of their outcomes. In contrast, for any distribution with a finite expected value, the expected value density (product between the PDF and value) always ends up decaying to 0 as the outcome increases. In meta-analyses, effect sizes, which can be EVs under a given model, are commonly weighted by the reciprocal of their variance. Variance tends to increase with effect size, and therefore larger effect sizes are usually weighted less heavily.
I do. In theory, there could be worlds which i) will actually be astronomically valuable given some efforts, but that ii) will be practically actually neutral without such efforts. In this case, the efforts would be astronomically valuable due to meaningfully increasing the probability of astronomically valuable futures. I just think such worlds are super implausible. I can see many worlds satisfying i), but not i) and ii) simultaneously.
I agree. I illustrated my point with a single model in my past comment. However, I believe appropriately weighted sets of models lead to the same conclusion that the expected increase in welfare decreases with actual welfare when this is astronomical (because I think the increase in the probability of the actual welfare decreases faster than the actual welfare when this is astronomical).
I understand what you have in mind. If EV is 1⁄0.99 under model A with 99 % weight, and 1 M under B with 1 % weight, A contributes 1 (= 0.99*1/0.99) to the EV considering both models, and B contributes 10 k (= 0.01*1*10^6). The contribution of A is 0.01 % (= 1/(1*10^4)) that of B. So changing by a given percentage the EV under B, or weight of B changes the EV considering both models much more that changing by the same percentage the EV under A, or weight of A. For example, doubling (increasing by 100 %) the EV under B would increase the EV considering both models by 1 (= 1*1), whereas doubling the EV under A would increase the EV considering both models by 10 k (= 1*10^4), 10 k (= 1*10^4/1) times the increase resulting from increasing the EV under B. The change in the EV considering all models resulting from changing the EV under a model, or its weight is proportional to the contribution of the model to the EV considering all models, and the relative change in the EV under the model, or its weight.
However, I think your presentation of RHSINO assumes the conclusion it is trying to support. High-stakes models are the ones that matter if they are the major driver of the EV under all models. However, the question is whether they are the major driver of the EV under all models. This has to be determined based on empirical evidence, not on points that are mathematically true.
People usually give weights that are at least 0.1/”number of models”, which easily results in high-stakes models dominating. However, giving weights which are not much smaller than the uniform weight of 1/”number of models” could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to “distance”^-2 (correct answer), “distance”^-20, or “distance”^-200, I imagine I would get a significant fraction picking the exponents of −20 and −200. Assuming 60 % picked −2, 20 % picked −20, and 20 % picked −200, a respondant may naively conclude the mean exponent of −45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Alternatively, a respondant may naively conclude an exponent of −9.19 (= 0.933*(-2) + 0.0333*(-20) + 0.0333*(-200)) is reasonable giving a weight of 3.33 % (= 0.1/3) to each of the 2 wrong exponents, equal to 10 % of the uniform weight, and the remaining weight of 93.3 % (= 1 − 2*0.0333) to the correct exponent. Yet, there is lots of empirical evidence against the exponents of −45.2 and −9.19 which the respondants are not aware of. The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks. This is also why I am sceptical that the absolute value of the welfare per unit time of animals is bound to be relatively close to that of humans, as one may naively infer from the welfare ranges Rethink Priorities (RP) initially presented, or the ones in Bob Fischer’s book about comparing welfare across species, where there seems to be only 1 line about the weights. “We assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive model”.
Mistakes like the one illustrated above happen when the weights of models are guessed independently of their stakes. People are often sensitive to astronomically high stakes, but not to the astronomically low weights they imply.
You may be interested in my chat with Matthew Adelstein. We discussed my scepticism of longtermism.
Ok, thanks for expanding upon your view! It sounds broadly akin to how I’m inclined to address Pascal’s Mugging cases (treat the astronomical stakes as implying proportionately negligible probability). Astronomical stakes from x-risk mitigation seems much more substantively credible to me, but I don’t have much to add at this point if you don’t share that substantive judgment!
You are welcome!
Makes sense. I see Pascal’s muggings as instances where the probability of the offers is assessed indepently of their outcomes. In contrast, for any distribution with a finite expected value, the expected value density (product between the PDF and value) always ends up decaying to 0 as the outcome increases. In meta-analyses, effect sizes, which can be EVs under a given model, are commonly weighted by the reciprocal of their variance. Variance tends to increase with effect size, and therefore larger effect sizes are usually weighted less heavily.
People sometimes point to Holden Karnofsky’s post Why we can’t take expected value estimates literally (even when they’re unbiased) to justify not relying on EVs (here are my notes on it from 4 years ago). However, the post does not broadly argue against using EVs. I see it as a call for not treating all EVs the same, and weighting them appropriately.