It suggests that the task of society is to reach some very particular kind of state(s), and otherwise it (presumably) flounders.
One way that’s inaccurate is that you can ask how well things are going on the ship, before it reaches the island, or after it misses the island.
It’s also unclear that the landscape of possible futures looks like a relatively discrete region of “success” and its complement, a region of failure.
Finally, the value of the expedition, whether or not it reaches the island, might depend on the course which the ship took. In jargon, the value of society at a time might be stateful with respect to its history, or the value over time might not be time-separable. In words, it’s about the journey…
So if that’s what you had in mind, I agree.
And then as I read you, the worry is something like: “Better Futures argues that great futures are likely to be difficult to reach. So even if great futures are somehow many times better than mediocre futures, shouldn’t we plan to make mediocre futures slightly better (or less bad), rather than throw a Hail Mary at the best futures?”
I wonder if there are two versions of this worry. The first might be: “the strategy which maximises the chance of passing some (or any) ‘great’ threshold of value is meaningfully worse, in expected value terms, than other strategies”. In particular it could be that the max(p(great future)) strategy involves doing common-sensically bad stuff with a slim chance of paying off, and which otherwise does harm. For example, strategies which involve massively centralising power and then hoping that whoever holds all the power makes the right decisions.
I think some version of this worry is decently plausible. But I don’t think anyone thinks that you absolutely should take whatever course maximises the chance of a great future. Rather, I think the rough idea in Better Futures is that, as a heuristic, it seems generally worthwhile to choose actions in light of whether they make great futures more likely. This is similar to Bostrom’s “maxipok” principle, which Will and Guive Assadi argue against here. But both principles are derived from asking “what heuristic seems like it does a good job at approximating trying to take the max-EV option in a more granular way”; rather than suggesting a direct alternative. So the question is whether it’s a good approximation of max-EV, rather than a good alternative to it.
The second version of the worry is something like: “on the Better Futures worldview, the max-EV strategy may well involve a Hail Mary with a slim chance of a great future, and otherwise a very likely outcome which is mediocre or bad. And this seems fanatical, or otherwise wrong.”
Here again I would be pretty sympathetic, and it might be useful to distinguish which actions are EV maximising, from which actions are right. If you are a consequentialist (with caveats and asterisks) you think the right action is the EV-maximising one. But we don’t try to argue that you should think right action is always EV-maximising and vice-versa (and IIRC there is a footnote trying to make this clear-ish). As with other cases of fanaticism, you could think that the plan which results in the most expected value is not the right plan!
Practically speaking, my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses. For example, it seems like power concentration looks pretty bad for both, in practice.
That’s really great! You posed my question better than I did, and answered it more detail than I was expecting. Thanks a lot. :)
RE: “my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses.”
For what it’s worth, it might be worth your time to check Gerald Gauss’ “The Tyranny of the Ideal”, particularly his framework around page 82, with what he calls “The Choice”: that sometimes we might face scenarios in which the local optimum is in the opposite direction from where the global optimum is. I discovered it recently and I thought it was pretty good for thinking about these kinds of issues.
Thanks for the kind words, Rafael!
I can see how the island analogy is confusing:
It suggests that the task of society is to reach some very particular kind of state(s), and otherwise it (presumably) flounders.
One way that’s inaccurate is that you can ask how well things are going on the ship, before it reaches the island, or after it misses the island.
It’s also unclear that the landscape of possible futures looks like a relatively discrete region of “success” and its complement, a region of failure.
Finally, the value of the expedition, whether or not it reaches the island, might depend on the course which the ship took. In jargon, the value of society at a time might be stateful with respect to its history, or the value over time might not be time-separable. In words, it’s about the journey…
So if that’s what you had in mind, I agree.
And then as I read you, the worry is something like: “Better Futures argues that great futures are likely to be difficult to reach. So even if great futures are somehow many times better than mediocre futures, shouldn’t we plan to make mediocre futures slightly better (or less bad), rather than throw a Hail Mary at the best futures?”
I wonder if there are two versions of this worry. The first might be: “the strategy which maximises the chance of passing some (or any) ‘great’ threshold of value is meaningfully worse, in expected value terms, than other strategies”. In particular it could be that the max(p(great future)) strategy involves doing common-sensically bad stuff with a slim chance of paying off, and which otherwise does harm. For example, strategies which involve massively centralising power and then hoping that whoever holds all the power makes the right decisions.
I think some version of this worry is decently plausible. But I don’t think anyone thinks that you absolutely should take whatever course maximises the chance of a great future. Rather, I think the rough idea in Better Futures is that, as a heuristic, it seems generally worthwhile to choose actions in light of whether they make great futures more likely. This is similar to Bostrom’s “maxipok” principle, which Will and Guive Assadi argue against here. But both principles are derived from asking “what heuristic seems like it does a good job at approximating trying to take the max-EV option in a more granular way”; rather than suggesting a direct alternative. So the question is whether it’s a good approximation of max-EV, rather than a good alternative to it.
The second version of the worry is something like: “on the Better Futures worldview, the max-EV strategy may well involve a Hail Mary with a slim chance of a great future, and otherwise a very likely outcome which is mediocre or bad. And this seems fanatical, or otherwise wrong.”
Here again I would be pretty sympathetic, and it might be useful to distinguish which actions are EV maximising, from which actions are right. If you are a consequentialist (with caveats and asterisks) you think the right action is the EV-maximising one. But we don’t try to argue that you should think right action is always EV-maximising and vice-versa (and IIRC there is a footnote trying to make this clear-ish). As with other cases of fanaticism, you could think that the plan which results in the most expected value is not the right plan!
Practically speaking, my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses. For example, it seems like power concentration looks pretty bad for both, in practice.
That’s really great! You posed my question better than I did, and answered it more detail than I was expecting. Thanks a lot. :)
RE: “my hope is that aiming for truly great futures, and just trying to improve incrementally on ‘default’ futures, recommend quite similar and compatible courses.”
For what it’s worth, it might be worth your time to check Gerald Gauss’ “The Tyranny of the Ideal”, particularly his framework around page 82, with what he calls “The Choice”: that sometimes we might face scenarios in which the local optimum is in the opposite direction from where the global optimum is. I discovered it recently and I thought it was pretty good for thinking about these kinds of issues.