This indeed seems like an interesting implication of Tarsney’s model, and indeed updates me towards placing a bit less emphasis on reducing non-extinction existential risks—e.g., reducing the chance of lock-in of a bad governmental system or set of values.
Surely “lock-in” implies stability and persistence?
Greaves and MacAskill introduce the concept of the ‘non-extinction attractor state’ to capture interventions that can achieve the persistence Tarsney says is so important, but that don’t rely on extinction to do so.
This includes institutional reform:
But once such institutions were created, they might persist indefinitely. Political institutions often change as a result of conflict or competition with other states. For strong world governments, this consideration would not apply (Caplan 2008). In the past, governments have also often changed as a result of civil war or internal revolution. However, advancing technology might make that far less likely for a future world government: modern and future surveillance technologies could prevent insurrection, and AI-controlled police and armies could be controlled by the leaders of the government, thereby removing the possibility of a military coup (Caplan 2008; Smith 2014).
Surely “lock-in” implies stability and persistence?
Yeah, definitely. I see now that I didn’t clearly explain what I meant. It’s not that I changed my views on how how important the difference between lock-in of a bad governmental system or set of values and a future without such a lock-in is.
It’s more like I somewhat updated my views regarding:
how likely such a lock-in is
and in particular how likely it is that a state that looks like it might be a lock-in would actually be a lock-in
and in particular how much the epistemic challenge to longtermism might undermine a focus on this type of potential lock-in in particular
And as a result, I somewhat updated me views regarding how much we should focus on preventing these outcomes. Analogous to how I’d update my prioritisation of biorisk if I learned the relevant catastrophes were less likely than I thought, even if no less bad.
(I’m still not sure that explanation is 100% clear.)
And yeah, Greaves and MacAskill’s “non-extinction attractor state” concept is relevant here, and I liked that section of their paper :)
OK that’s clearer, although I’m not immediately sure why the paper would have achieved the following:
I somewhat updated my views regarding:
how likely such a lock-in is
and in particular how likely it is that a state that looks like it might be a lock-in would actually be a lock-in
...
I think Tarsney implies that institutional reform is less likely to be a true lock-in, but he doesn’t really back this up with much argument. He just implies that this point is somewhat obvious. Under this assumption, I can understand why his model would lead to the following update:
...
...
and in particular how much the epistemic challenge to longtermism might undermine a focus on this type of potential lock-in in particular
In other words, if Tarsney had engaged in a discussion about why institutional change isn’t actually likely to be stable/persistent, providing object-level reasons for why (which may involve disagreeing with Greaves and MacAskill’s points), I think I too would update away from thinking institutional change is that important, but I don’t think he really engages in this discussion.
I should say that I haven’t properly read through the whole paper (I have mainly relied on watching the video and skimming through the paper), so it’s possible I’m missing some things.
I think it makes sense to be a bit confused about what claim I’m making and why. I read the paper and made the initial version of these note a few weeks ago, so my memory of what the paper said and how it changed my views is slightly hazy.
But I think the key point is essentially the arguably obvious point that the rate of ENEs can be really important, and that that rate seems likely to be much higher when the target state is something like “a very good system of government or set of values” or “a very bad system of government or set of values” (compared to when the target state is whether an intelligent civilization exists). It does seem much more obvious that extinction or non-extinction are each stronger attractor states that particularly good or particularly bad non-extinction outcomes are.
This is basically something I already knew, but I think Tarsney’s models and analysis made the point a bit more salient, and also made it clearer how important it is (since the rate of ENEs seems like probably one of the most important factors influencing the case for longtermism).
But what I’ve said above kind-of implicitly accepts Tarsney’s focus (for the sake of his working example) on simply whether there is an intelligent civilization around, rather than what it’s doing. In reality, I think that what the civilization is doing is likely also very important.[1] So the above point about particularly good or particularly bad non-extinction outcomes maybe being only weak attractor states might also undermine the significance of keeping an intelligent civilization around.
But here’s one way that might not be true: Maybe we think it’s easier to have a lock-in of—or natural trends that maintain—a good non-extinction outcome than a bad non-extinction outcome. (I think Ord essentially implies this in The Precipice. I might soon post something related to this. It’s also been discussed in some other places, e.g. here.) If so, then the point about the rate of ENEs suggests the case for avoiding unrecoverable dystopias and unrecoverable collapses might be weak, but it wouldn’t as strongly suggest the case for avoiding extinction is weak.
...but this all seems rather complicated, and I’m still not sure my thinking is clear, and even less sure my explanation is clear!
[1] Tarsney does acknowledge roughly this point later in the paper:
Additionally, there are other potential sources of epistemic resistance to longtermism besides Weak Attractors that this paper has not addressed. In particular, these include:
Neutral Attractors To entertain small values of r [the rate of ENEs], we must assume that the state S targeted by a longtermist intervention, and its complement ¬S, are both at least to some extent “attractor” states: Once a system is in state S, or state ¬S, it is unlikely to leave that state any time soon. But to justify significant values of ve and vs, it must also be the case that the attractors we are able to target differ significantly in expected value. And it’s not clear that we can assume this. For instance, perhaps “large interstellar civilization exists in spatial region X” is an attractor state, but “large interstellar civilization exists in region X with healthy norms and institutions that generate a high level of value” is not. If civilizations tend to “wander” unpredictably between high-value and low-value states, it could be that despite their astronomical potential for value, the expected value of large interstellar civilizations is close to zero. In that case, we can have persistent effects on the far future, but not effects that matter (in expectation).
Surely “lock-in” implies stability and persistence?
Greaves and MacAskill introduce the concept of the ‘non-extinction attractor state’ to capture interventions that can achieve the persistence Tarsney says is so important, but that don’t rely on extinction to do so.
This includes institutional reform:
Yeah, definitely. I see now that I didn’t clearly explain what I meant. It’s not that I changed my views on how how important the difference between lock-in of a bad governmental system or set of values and a future without such a lock-in is.
It’s more like I somewhat updated my views regarding:
how likely such a lock-in is
and in particular how likely it is that a state that looks like it might be a lock-in would actually be a lock-in
and in particular how much the epistemic challenge to longtermism might undermine a focus on this type of potential lock-in in particular
And as a result, I somewhat updated me views regarding how much we should focus on preventing these outcomes. Analogous to how I’d update my prioritisation of biorisk if I learned the relevant catastrophes were less likely than I thought, even if no less bad.
(I’m still not sure that explanation is 100% clear.)
And yeah, Greaves and MacAskill’s “non-extinction attractor state” concept is relevant here, and I liked that section of their paper :)
OK that’s clearer, although I’m not immediately sure why the paper would have achieved the following:
I think Tarsney implies that institutional reform is less likely to be a true lock-in, but he doesn’t really back this up with much argument. He just implies that this point is somewhat obvious. Under this assumption, I can understand why his model would lead to the following update:
In other words, if Tarsney had engaged in a discussion about why institutional change isn’t actually likely to be stable/persistent, providing object-level reasons for why (which may involve disagreeing with Greaves and MacAskill’s points), I think I too would update away from thinking institutional change is that important, but I don’t think he really engages in this discussion.
I should say that I haven’t properly read through the whole paper (I have mainly relied on watching the video and skimming through the paper), so it’s possible I’m missing some things.
[Writing this comment quickly]
I think it makes sense to be a bit confused about what claim I’m making and why. I read the paper and made the initial version of these note a few weeks ago, so my memory of what the paper said and how it changed my views is slightly hazy.
But I think the key point is essentially the arguably obvious point that the rate of ENEs can be really important, and that that rate seems likely to be much higher when the target state is something like “a very good system of government or set of values” or “a very bad system of government or set of values” (compared to when the target state is whether an intelligent civilization exists). It does seem much more obvious that extinction or non-extinction are each stronger attractor states that particularly good or particularly bad non-extinction outcomes are.
This is basically something I already knew, but I think Tarsney’s models and analysis made the point a bit more salient, and also made it clearer how important it is (since the rate of ENEs seems like probably one of the most important factors influencing the case for longtermism).
But what I’ve said above kind-of implicitly accepts Tarsney’s focus (for the sake of his working example) on simply whether there is an intelligent civilization around, rather than what it’s doing. In reality, I think that what the civilization is doing is likely also very important.[1] So the above point about particularly good or particularly bad non-extinction outcomes maybe being only weak attractor states might also undermine the significance of keeping an intelligent civilization around.
But here’s one way that might not be true: Maybe we think it’s easier to have a lock-in of—or natural trends that maintain—a good non-extinction outcome than a bad non-extinction outcome. (I think Ord essentially implies this in The Precipice. I might soon post something related to this. It’s also been discussed in some other places, e.g. here.) If so, then the point about the rate of ENEs suggests the case for avoiding unrecoverable dystopias and unrecoverable collapses might be weak, but it wouldn’t as strongly suggest the case for avoiding extinction is weak.
...but this all seems rather complicated, and I’m still not sure my thinking is clear, and even less sure my explanation is clear!
[1] Tarsney does acknowledge roughly this point later in the paper:
OK thanks I think that is clearer now.