I’m writing quickly because I think this is a tricky issue and I’m trying not to spend too long on it. If I don’t make sense, I might have misspoken or made a reasoning error.
One way I thought about the problem (quite different to yours, very rough): variation in existential risk rate depends mostly on technology. At a wide enough interval (say, 100 years of tech development at current rates), change in existential risk with change in technology is hard to predict, though following Aschenbrenner and Xu’s observations it’s plausible that it tends to some equilibrium in the long run. You could perhaps model a mixture of a purely random walk and walks directed towards uncertain equilibria.
Also, technological growth probably has an upper limit somewhere, though quite unclear where, so even the purely random walk probably settles down eventually.
There’s uncertainty over a) how long it takes to “eventually” settle down, b) how much “randomness” there is as we approach an equilibrium c) how quickly equilibrium is approached, if it is approached.
I don’t know what you get if you try to parametrise that and integrate it all out, but I would also be surprised if it put and overwhelmingly low credence in a short sharp time of troubles.
I think “one-off displacement from equilibrium” probably isn’t a great analogy for tech-driven existential risk.
I think “high and sustained risk” seems weird partly because surviving for a long period under such conditions is weird, so conditioning on survival usually suggests that risk isn’t so high after all—so in many cases risk really does go down for survivors. But this effect only applies to survivors, and the other possibility is that we underestimated risk and we die. So I’m not sure that this effect changes conclusions. I’m also not sure how this affects your evaluation of your impact on risk—probably makes it smaller?
I think this observation might apply to your thought experiment, which conditions on survival.
(As an aside, it might not make a difference mathematically, but numerically one possible difference between us is that I think of the underlying unit to be ~logarithmic rather than linear)
Also, technological growth probably has an upper limit somewhere, though quite unclear where, so even the purely random walk probably settles down eventually.
Agreed, an important part of my model is something like nontrivial credence in a) technological completion conjecture and b) there aren’t “that many” technologies laying around to be discovered. So I zoom in and think about technological risks, a lot of my (proposed) model is thinking about the a) underlying distribution of scary vs worldsaving technologies and b) whether/how much the world is prepared for each scary technology as they appears, c) how high is the sharpness of dropoff of lethality for survival from each new scary technology conditional upon survival in the previous timestep.
I think “high and sustained risk” seems weird partly because surviving for a long period under such conditions is weird, so conditioning on survival usually suggests that risk isn’t so high after all—so risk really does go down for survivors. But this effect only applies to the fraction who survive, so I’m not sure that it changes conclusions
I think I probably didn’t make the point well enough, but roughly speaking, you only care about worlds where you survive, so my guess is that you’ll systematically overestimate longterm risk if your mixture model doesn’t update on survival at each time step to be evidence that survival is more likely on future time steps. But you do have to be careful here.
I’m also not sure how this affects your evaluation of your impact on risk—probably makes it smaller?
Yeah I think this is true. A friend brought up this point, roughly, the important parts of your risk reduction comes from temporarily vulnerable worlds. But if you’re not careful, you might “borrow” your risk-reduction from permanently vulnerable worlds (given yourself credit for high microextinctions averted), and also “borrow” your EV_of_future from permanently invulnerable worlds (given yourself credit for a share of an overwhelmingly large future). But to the extent those are different and anti-correlated worlds (which accords with David’s original point, just a bit more nuanced), then your actual EV can be a noticeably smaller slice.
I’m writing quickly because I think this is a tricky issue and I’m trying not to spend too long on it. If I don’t make sense, I might have misspoken or made a reasoning error.
One way I thought about the problem (quite different to yours, very rough): variation in existential risk rate depends mostly on technology. At a wide enough interval (say, 100 years of tech development at current rates), change in existential risk with change in technology is hard to predict, though following Aschenbrenner and Xu’s observations it’s plausible that it tends to some equilibrium in the long run. You could perhaps model a mixture of a purely random walk and walks directed towards uncertain equilibria.
Also, technological growth probably has an upper limit somewhere, though quite unclear where, so even the purely random walk probably settles down eventually.
There’s uncertainty over a) how long it takes to “eventually” settle down, b) how much “randomness” there is as we approach an equilibrium c) how quickly equilibrium is approached, if it is approached.
I don’t know what you get if you try to parametrise that and integrate it all out, but I would also be surprised if it put and overwhelmingly low credence in a short sharp time of troubles.
I think “one-off displacement from equilibrium” probably isn’t a great analogy for tech-driven existential risk.
I think “high and sustained risk” seems weird partly because surviving for a long period under such conditions is weird, so conditioning on survival usually suggests that risk isn’t so high after all—so in many cases risk really does go down for survivors. But this effect only applies to survivors, and the other possibility is that we underestimated risk and we die. So I’m not sure that this effect changes conclusions. I’m also not sure how this affects your evaluation of your impact on risk—probably makes it smaller?
I think this observation might apply to your thought experiment, which conditions on survival.
Appreciate your comments!
(As an aside, it might not make a difference mathematically, but numerically one possible difference between us is that I think of the underlying unit to be ~logarithmic rather than linear)
Agreed, an important part of my model is something like nontrivial credence in a) technological completion conjecture and b) there aren’t “that many” technologies laying around to be discovered. So I zoom in and think about technological risks, a lot of my (proposed) model is thinking about the a) underlying distribution of scary vs worldsaving technologies and b) whether/how much the world is prepared for each scary technology as they appears, c) how high is the sharpness of dropoff of lethality for survival from each new scary technology conditional upon survival in the previous timestep.
I think I probably didn’t make the point well enough, but roughly speaking, you only care about worlds where you survive, so my guess is that you’ll systematically overestimate longterm risk if your mixture model doesn’t update on survival at each time step to be evidence that survival is more likely on future time steps. But you do have to be careful here.
Yeah I think this is true. A friend brought up this point, roughly, the important parts of your risk reduction comes from temporarily vulnerable worlds. But if you’re not careful, you might “borrow” your risk-reduction from permanently vulnerable worlds (given yourself credit for high microextinctions averted), and also “borrow” your EV_of_future from permanently invulnerable worlds (given yourself credit for a share of an overwhelmingly large future). But to the extent those are different and anti-correlated worlds (which accords with David’s original point, just a bit more nuanced), then your actual EV can be a noticeably smaller slice.