Hi Fin, sorry I’m a bit late with my question, I was rereading parts of the Better Futures series. First of all, I have to say it’s one of my favorite article series I’ve ever read, and I’ll be citing it in my own work going forward. The easygoing-versus-fussy distinction in particular is something I’m finding really interesting to dig into. :) Would love to discuss it in more detail at some point.
I wanted to push on the metaphor of sailing to an island, which appears at the start of No Easy Eutopia, but my question is going to take some preamble explanation (sorry!).
I find myself preferring a slightly different picture. Rather than thinking of eutopia as an island we’re navigating to, I tend to think of society as the ship itself, drifting through a sea of value over time (a topography of better and worse regions we’re already moving through). Societal change feels to me more like a search through uncharted moral territories than an expedition to a specific destination. On that picture, the priority seems more likely to be “how do we improve the ship, so that society reliably moves toward better regions of the sea?”
A couple of clarifications. First, I grant fussiness, I agree most plausible axiologies locate near-best futures in a very narrow region (I lean towards total hedonistic utilitarianism, myself). Second, I’m not a a quietist, in my own work I’m defending what I call moral niche construction, a fairly interventionist view on which we should actively reshape institutions, technologies, and even our own moral psychology (through things like AI moral decisionmakers or bioenhancement) to push society toward better regions. So the disagreement isn’t really about ambition, either.
Where I want to press is the following. In the ship-improvement picture, I can grant openly that we probably will never reach eutopia. We end up in a high-value region of the sea (in a local optima), much better than where we are now, plausibly very good in absolute terms, but not the narrow island.
That sounds like a concession, but on rereading Convergence and Compromise, it looks to me like the target-pursuit picture probably doesn’t reach the island either: you mention how WAM-convergence is unlikely, partial convergence plus trade faces serious obstacles, value-destroying threats can eat most of the value… So the comparison isn’t “guaranteed eutopia versus probably-not-eutopia”, since you yourself seem pretty pessimistic. So it’s two orientations that both probably miss the island, where one delivers reliable improvements to our current region of the sea along the way, and the other keeps optimizing toward a target it probably won’t hit. And, well, if you miss the moon, you don’t really land upon the stars… you drift in empty space and die, haha.
(There are similar points on Jerry Gaus’ The Tyranny of the Ideal, and on recent debates between ideal theory and non-ideal theory in moral and political philosophy)
So, finally, my question is: given that target-pursuit probably doesn’t reach eutopia either, on the series’ own analysis, why is the practical orientation toward the narrow target rather than toward improving our current region of the sea (e.g. pursuing very high + plausibly easy to reach and resilient local optima)? What’s the case for target-pursuit as a practical orientation, once we factor in that we will probably fail? Is it a case akin to fanaticism, where, if we land in the island, the payoff would be huge?
(Apologies in advance if this is addressed somewhere in the series, my memory context window isn’t large enough to hold the whole essay series at once!)
It suggests that the task of society is to reach some very particular kind of state(s), and otherwise it (presumably) flounders.
One way that’s inaccurate is that you can ask how well things are going on the ship, before it reaches the island, or after it misses the island.
It’s also unclear that the landscape of possible futures looks like a relatively discrete region of “success” and its complement, a region of failure.
Finally, the value of the expedition, whether or not it reaches the island, might depend on the course which the ship took. In jargon, the value of society at a time might be stateful with respect to its history, or the value over time might not be time-separable. In words, it’s about the journey…
So if that’s what you had in mind, I agree.
And then as I read you, the worry is something like: “Better Futures argues that great futures are likely to be difficult to reach. So even if great futures are somehow many times better than mediocre futures, shouldn’t we plan to make mediocre futures slightly better (or less bad), rather than throw a Hail Mary at the best futures?”
I wonder if there are two versions of this worry. The first might be: “the strategy which maximises the chance of passing some (or any) ‘great’ threshold of value is meaningfully worse, in expected value terms, than other strategies”. In particular it could be that the max(p(great future)) strategy involves doing common-sensically bad stuff with a slim chance of paying off, and which otherwise does harm. For example, strategies which involve massively centralising power and then hoping that whoever holds all the power makes the right decisions.
I think some version of this worry is decently plausible. But I don’t think anyone thinks that you absolutely should take whatever course maximises the chance of a great future. Rather, I think the rough idea in Better Futures is that, as a heuristic, it seems generally worthwhile to choose actions in light of whether they make great futures more likely. This is similar to Bostrom’s “maxipok” principle, which Will and Guive Assadi argue against here. But both principles are derived from asking “what heuristic seems like it does a good job at approximating trying to take the max-EV option in a more granular way”; rather than suggesting a direct alternative. So the question is whether it’s a good approximation of max-EV, rather than a good alternative to it.
The second version of the worry is something like: “on the Better Futures worldview, the max-EV strategy may well involve a Hail Mary with a slim chance of a great future, and otherwise a very likely outcome which is mediocre or bad. And this seems fanatical, or otherwise wrong.”
Here again I would be pretty sympathetic, and it might be useful to distinguish which actions are EV maximising, from which actions are right. If you are a consequentialist (with caveats and asterisks) you think the right action is the EV-maximising one. But we don’t try to argue that you should think right action is always EV-maximising and vice-versa (and IIRC there is a footnote trying to make this clear-ish). As with other cases of fanaticism, you could think that the plan which results in the most expected value is not the right plan!
Practically speaking, my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses. For example, it seems like power concentration looks pretty bad for both, in practice.
That’s really great! You posed my question better than I did, and answered it more detail than I was expecting. Thanks a lot. :)
RE: “my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses.”
For what it’s worth, it might be worth your time to check Gerald Gauss’ “The Tyranny of the Ideal”, particularly his framework around page 82, with what he calls “The Choice”: that sometimes we might face scenarios in which the local optimum is in the opposite direction from where the global optimum is. I discovered it recently and I thought it was pretty good for thinking about these kinds of issues.
Hi Fin, sorry I’m a bit late with my question, I was rereading parts of the Better Futures series. First of all, I have to say it’s one of my favorite article series I’ve ever read, and I’ll be citing it in my own work going forward. The easygoing-versus-fussy distinction in particular is something I’m finding really interesting to dig into. :) Would love to discuss it in more detail at some point.
I wanted to push on the metaphor of sailing to an island, which appears at the start of No Easy Eutopia, but my question is going to take some preamble explanation (sorry!).
I find myself preferring a slightly different picture. Rather than thinking of eutopia as an island we’re navigating to, I tend to think of society as the ship itself, drifting through a sea of value over time (a topography of better and worse regions we’re already moving through). Societal change feels to me more like a search through uncharted moral territories than an expedition to a specific destination. On that picture, the priority seems more likely to be “how do we improve the ship, so that society reliably moves toward better regions of the sea?”
A couple of clarifications. First, I grant fussiness, I agree most plausible axiologies locate near-best futures in a very narrow region (I lean towards total hedonistic utilitarianism, myself). Second, I’m not a a quietist, in my own work I’m defending what I call moral niche construction, a fairly interventionist view on which we should actively reshape institutions, technologies, and even our own moral psychology (through things like AI moral decisionmakers or bioenhancement) to push society toward better regions. So the disagreement isn’t really about ambition, either.
Where I want to press is the following. In the ship-improvement picture, I can grant openly that we probably will never reach eutopia. We end up in a high-value region of the sea (in a local optima), much better than where we are now, plausibly very good in absolute terms, but not the narrow island.
That sounds like a concession, but on rereading Convergence and Compromise, it looks to me like the target-pursuit picture probably doesn’t reach the island either: you mention how WAM-convergence is unlikely, partial convergence plus trade faces serious obstacles, value-destroying threats can eat most of the value… So the comparison isn’t “guaranteed eutopia versus probably-not-eutopia”, since you yourself seem pretty pessimistic. So it’s two orientations that both probably miss the island, where one delivers reliable improvements to our current region of the sea along the way, and the other keeps optimizing toward a target it probably won’t hit. And, well, if you miss the moon, you don’t really land upon the stars… you drift in empty space and die, haha.
(There are similar points on Jerry Gaus’ The Tyranny of the Ideal, and on recent debates between ideal theory and non-ideal theory in moral and political philosophy)
So, finally, my question is: given that target-pursuit probably doesn’t reach eutopia either, on the series’ own analysis, why is the practical orientation toward the narrow target rather than toward improving our current region of the sea (e.g. pursuing very high + plausibly easy to reach and resilient local optima)? What’s the case for target-pursuit as a practical orientation, once we factor in that we will probably fail? Is it a case akin to fanaticism, where, if we land in the island, the payoff would be huge?
(Apologies in advance if this is addressed somewhere in the series, my memory context window isn’t large enough to hold the whole essay series at once!)
Thanks for the kind words, Rafael!
I can see how the island analogy is confusing:
It suggests that the task of society is to reach some very particular kind of state(s), and otherwise it (presumably) flounders.
One way that’s inaccurate is that you can ask how well things are going on the ship, before it reaches the island, or after it misses the island.
It’s also unclear that the landscape of possible futures looks like a relatively discrete region of “success” and its complement, a region of failure.
Finally, the value of the expedition, whether or not it reaches the island, might depend on the course which the ship took. In jargon, the value of society at a time might be stateful with respect to its history, or the value over time might not be time-separable. In words, it’s about the journey…
So if that’s what you had in mind, I agree.
And then as I read you, the worry is something like: “Better Futures argues that great futures are likely to be difficult to reach. So even if great futures are somehow many times better than mediocre futures, shouldn’t we plan to make mediocre futures slightly better (or less bad), rather than throw a Hail Mary at the best futures?”
I wonder if there are two versions of this worry. The first might be: “the strategy which maximises the chance of passing some (or any) ‘great’ threshold of value is meaningfully worse, in expected value terms, than other strategies”. In particular it could be that the max(p(great future)) strategy involves doing common-sensically bad stuff with a slim chance of paying off, and which otherwise does harm. For example, strategies which involve massively centralising power and then hoping that whoever holds all the power makes the right decisions.
I think some version of this worry is decently plausible. But I don’t think anyone thinks that you absolutely should take whatever course maximises the chance of a great future. Rather, I think the rough idea in Better Futures is that, as a heuristic, it seems generally worthwhile to choose actions in light of whether they make great futures more likely. This is similar to Bostrom’s “maxipok” principle, which Will and Guive Assadi argue against here. But both principles are derived from asking “what heuristic seems like it does a good job at approximating trying to take the max-EV option in a more granular way”; rather than suggesting a direct alternative. So the question is whether it’s a good approximation of max-EV, rather than a good alternative to it.
The second version of the worry is something like: “on the Better Futures worldview, the max-EV strategy may well involve a Hail Mary with a slim chance of a great future, and otherwise a very likely outcome which is mediocre or bad. And this seems fanatical, or otherwise wrong.”
Here again I would be pretty sympathetic, and it might be useful to distinguish which actions are EV maximising, from which actions are right. If you are a consequentialist (with caveats and asterisks) you think the right action is the EV-maximising one. But we don’t try to argue that you should think right action is always EV-maximising and vice-versa (and IIRC there is a footnote trying to make this clear-ish). As with other cases of fanaticism, you could think that the plan which results in the most expected value is not the right plan!
Practically speaking, my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses. For example, it seems like power concentration looks pretty bad for both, in practice.
That’s really great! You posed my question better than I did, and answered it more detail than I was expecting. Thanks a lot. :)
RE: “my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses.”
For what it’s worth, it might be worth your time to check Gerald Gauss’ “The Tyranny of the Ideal”, particularly his framework around page 82, with what he calls “The Choice”: that sometimes we might face scenarios in which the local optimum is in the opposite direction from where the global optimum is. I discovered it recently and I thought it was pretty good for thinking about these kinds of issues.