OK that’s clearer, although I’m not immediately sure why the paper would have achieved the following:
I somewhat updated my views regarding:
how likely such a lock-in is
and in particular how likely it is that a state that looks like it might be a lock-in would actually be a lock-in
...
I think Tarsney implies that institutional reform is less likely to be a true lock-in, but he doesn’t really back this up with much argument. He just implies that this point is somewhat obvious. Under this assumption, I can understand why his model would lead to the following update:
...
...
and in particular how much the epistemic challenge to longtermism might undermine a focus on this type of potential lock-in in particular
In other words, if Tarsney had engaged in a discussion about why institutional change isn’t actually likely to be stable/persistent, providing object-level reasons for why (which may involve disagreeing with Greaves and MacAskill’s points), I think I too would update away from thinking institutional change is that important, but I don’t think he really engages in this discussion.
I should say that I haven’t properly read through the whole paper (I have mainly relied on watching the video and skimming through the paper), so it’s possible I’m missing some things.
I think it makes sense to be a bit confused about what claim I’m making and why. I read the paper and made the initial version of these note a few weeks ago, so my memory of what the paper said and how it changed my views is slightly hazy.
But I think the key point is essentially the arguably obvious point that the rate of ENEs can be really important, and that that rate seems likely to be much higher when the target state is something like “a very good system of government or set of values” or “a very bad system of government or set of values” (compared to when the target state is whether an intelligent civilization exists). It does seem much more obvious that extinction or non-extinction are each stronger attractor states that particularly good or particularly bad non-extinction outcomes are.
This is basically something I already knew, but I think Tarsney’s models and analysis made the point a bit more salient, and also made it clearer how important it is (since the rate of ENEs seems like probably one of the most important factors influencing the case for longtermism).
But what I’ve said above kind-of implicitly accepts Tarsney’s focus (for the sake of his working example) on simply whether there is an intelligent civilization around, rather than what it’s doing. In reality, I think that what the civilization is doing is likely also very important.[1] So the above point about particularly good or particularly bad non-extinction outcomes maybe being only weak attractor states might also undermine the significance of keeping an intelligent civilization around.
But here’s one way that might not be true: Maybe we think it’s easier to have a lock-in of—or natural trends that maintain—a good non-extinction outcome than a bad non-extinction outcome. (I think Ord essentially implies this in The Precipice. I might soon post something related to this. It’s also been discussed in some other places, e.g. here.) If so, then the point about the rate of ENEs suggests the case for avoiding unrecoverable dystopias and unrecoverable collapses might be weak, but it wouldn’t as strongly suggest the case for avoiding extinction is weak.
...but this all seems rather complicated, and I’m still not sure my thinking is clear, and even less sure my explanation is clear!
[1] Tarsney does acknowledge roughly this point later in the paper:
Additionally, there are other potential sources of epistemic resistance to longtermism besides Weak Attractors that this paper has not addressed. In particular, these include:
Neutral Attractors To entertain small values of r [the rate of ENEs], we must assume that the state S targeted by a longtermist intervention, and its complement ¬S, are both at least to some extent “attractor” states: Once a system is in state S, or state ¬S, it is unlikely to leave that state any time soon. But to justify significant values of ve and vs, it must also be the case that the attractors we are able to target differ significantly in expected value. And it’s not clear that we can assume this. For instance, perhaps “large interstellar civilization exists in spatial region X” is an attractor state, but “large interstellar civilization exists in region X with healthy norms and institutions that generate a high level of value” is not. If civilizations tend to “wander” unpredictably between high-value and low-value states, it could be that despite their astronomical potential for value, the expected value of large interstellar civilizations is close to zero. In that case, we can have persistent effects on the far future, but not effects that matter (in expectation).
OK that’s clearer, although I’m not immediately sure why the paper would have achieved the following:
I think Tarsney implies that institutional reform is less likely to be a true lock-in, but he doesn’t really back this up with much argument. He just implies that this point is somewhat obvious. Under this assumption, I can understand why his model would lead to the following update:
In other words, if Tarsney had engaged in a discussion about why institutional change isn’t actually likely to be stable/persistent, providing object-level reasons for why (which may involve disagreeing with Greaves and MacAskill’s points), I think I too would update away from thinking institutional change is that important, but I don’t think he really engages in this discussion.
I should say that I haven’t properly read through the whole paper (I have mainly relied on watching the video and skimming through the paper), so it’s possible I’m missing some things.
[Writing this comment quickly]
I think it makes sense to be a bit confused about what claim I’m making and why. I read the paper and made the initial version of these note a few weeks ago, so my memory of what the paper said and how it changed my views is slightly hazy.
But I think the key point is essentially the arguably obvious point that the rate of ENEs can be really important, and that that rate seems likely to be much higher when the target state is something like “a very good system of government or set of values” or “a very bad system of government or set of values” (compared to when the target state is whether an intelligent civilization exists). It does seem much more obvious that extinction or non-extinction are each stronger attractor states that particularly good or particularly bad non-extinction outcomes are.
This is basically something I already knew, but I think Tarsney’s models and analysis made the point a bit more salient, and also made it clearer how important it is (since the rate of ENEs seems like probably one of the most important factors influencing the case for longtermism).
But what I’ve said above kind-of implicitly accepts Tarsney’s focus (for the sake of his working example) on simply whether there is an intelligent civilization around, rather than what it’s doing. In reality, I think that what the civilization is doing is likely also very important.[1] So the above point about particularly good or particularly bad non-extinction outcomes maybe being only weak attractor states might also undermine the significance of keeping an intelligent civilization around.
But here’s one way that might not be true: Maybe we think it’s easier to have a lock-in of—or natural trends that maintain—a good non-extinction outcome than a bad non-extinction outcome. (I think Ord essentially implies this in The Precipice. I might soon post something related to this. It’s also been discussed in some other places, e.g. here.) If so, then the point about the rate of ENEs suggests the case for avoiding unrecoverable dystopias and unrecoverable collapses might be weak, but it wouldn’t as strongly suggest the case for avoiding extinction is weak.
...but this all seems rather complicated, and I’m still not sure my thinking is clear, and even less sure my explanation is clear!
[1] Tarsney does acknowledge roughly this point later in the paper:
OK thanks I think that is clearer now.