My central objection to Thorstadās work on this is the failure to properly account for uncertainty. Attempting to exclusively model a most-plausible scenario, and draw dismissive conclusions about longtermist interventions based solely on that, fails to reflect best practices about how to reason under conditions of uncertainty. (Iāve also raised this criticism against Schwitzgebelās negligibility argument.) You need to consider the full range of possible models /ā scenarios!
Itās essentially fallacious to think that āplausibly incorrect modeling assumptionsā undermine expected value reasoning. High expected value can still result from regions of probability space that are epistemically unlikely (or reflect āplausibly incorrectā conditions or assumptions). If thereās even a 1% chance that the relevant assumptions hold, just discount the output value accordingly. Astronomical stakes are not going to be undermined by lopping off the last two zeros.
Tarsneyās Epistemic Challenge to Longtermism is so much better at this. As he aptly notes, as long as youāre on board with orthodox decision theory (and so donāt disproportionately discount or neglect low-probability possibilities), and not completely dogmatic in refusing to give any credence at all to the longtermist-friendly assumptions (robust existential security after time of perils, etc.), reasonable epistemic worries ultimately arenāt capable of undermining the expected value argument for longtermism.
(These details can still be helpful for getting better-refined EV estimates, of course. But thatās very different from presenting them as an objection to the whole endeavor.)
This feels like an isolated demand for rigour, since as far as I can see Thorstadās[1] central argument isnāt that a particular course of the future is more plausible, but that [popular representations of] longtermist arguments themselves donāt consider the full range of possibilities, donāt discount for uncertainty, and that apparently modest-sounding claims that existential risk is non-zero and that humanity could last a long time if we survive near-term threats are compatible only if one makes strong claims about the hinginess of history[2]
I donāt see him trying to build a more accurate model of the future[3] so much as pointing out how very simple changes completely change longtermist models. As such, his models are intentionally simple and Owenās expansion above adds more value for anyone actively trying to model a range of future scenarios. But Iām not sure why it would be incumbent on the researcher arguing against choosing a course of action based on long term outcomes to be the one who explicitly models the entire problem space. Iād turn that around and question why longtermists who donāt consider the whole endeavour of predicting the long term future in our decision theory to be futile generally dogmatically reject low probability outcomes with Pascalian payoffs that favour the other option, or to simply assume the asymmetry of outcomes works in their favour.
Now personally Iām fine with āno, actually I think catastrophes are badā, but thatās because Iām focused on the near term where it really is obvious that nuclear holocausts arenāt going to have a positive welfare impact. Once weāre insisting that our decisions ought to be guided by tiny subjective credences in far future possibilities with uncertain likelihood but astronomic payoffs and that itās an error not to factor unlikely interstellar civilizations into our calculations of what we should do if theyāre big enough, it seems far less obvious that the astronomical stakes skew in favour of humanity.
The Tarsney paper even explicitly models the possibility of non-human galactic colonization, but with the unjustified assumption that no non-humans will be of converting resources to utility at a higher rate than [post]humans, so their emergence as competitors for galactic resources merely ānullifiesā the beneficial effects of humanity surviving. But from a total welfarist perspective, the problem here isnāt just that maximizing the possible welfare across the history of the universe may not be contingent on the long term survival of the human species,[4] but that humans surviving to colonise galaxies might diminish galactic welfare. Schweitzgebelās argument that actually human extinction might be net good for total welfare is only a mad hypothetical if you reject fanaticism: otherwise itās the logical consequence of accepting the possibility, however small, that a nonhuman species might convert resources to welfare much more efficiently than us.[5] Now a future of decibillions of aliens building Dyson Spheres all over the galaxy because thereās no pesky humans in their way sounds extremely unlikely, and perhaps even less likely than a galaxy filled with the same fantastic tech to support quadrillions of humansāa species we at least know exists and has some interest in inventing Dyson Spheresābut despite this the asymmetry of possible payoff magnitudes may strongly favour not letting us survive to colonise the galaxy.[6]
In the absence of any particular reason for confidence that the EV of one set of futures is definitely higher than the others, it seems like you end up reaching for heuristics like ābut letting everyone die would be insaneā. I couldnāt agree more, but the least arbitrary way to do that is to adjust the framework to privilege the short term with discount rates sufficiently high to neuter payoffs so speculative and astronomical we canāt rule out the possibility they exceed the payoff from [not] letting eight billion humans die[7] Since that discount rate reflects extreme uncertainty about what might happen and what payoffs might look like, it also feels more epistemically humble than basing an entire worldview on the long tail outcome of some low probability far futures whilst dismissing other equally unsubstantiated hypotheticals because their implications are icky. And Iām pretty sure this is what Thorstad wants us to do, not to place high credence in his point estimates or give up on X-risk mitigation altogether.
which doesnāt of course mean that hinginess is untrue, but does make it less a general principle of caring about the long term and more a relatively bold and specific claim about the distribution of future outcomes.
in the arguments referenced here anyway. He has also written stuff which attempts to estimate different XR base rates from those posited by Ord et al, which I find just as speculative as the longtermistsā
there are of course ethical frameworks other than maximizing total utility across all species which give us reason to prefer 10^31 humans over a similarly low probability von Neumann civilization involving 10^50 aliens or a single AI utility monster (I actually prefer them, so no proposing destroying humanity as a cause area from me!) but theyāre a different from the framework Tarsney and most longtermists use, and open the door to other arguments for weighting current humans over far future humans.
Weāre a fairly stupid, fragile and predatory species capable of experiencing strongly negative pain and emotional valences at regular intervals over fairly short lifetimes, with competitive social dynamics, very specific survival needs and a wasteful approach to consumption, so it doesnāt seem obvious or even likely that humanity and its descendants will be even close to the upper bound for converting resources to welfare...
Of course, if you reject fanaticism, the adverse effects of humans not dying in a nuclear holocaust on alien utility monsters are far too remote and unlikely and frankly a bit daft to worry about. But if you accept fanaticism (and species-neutral total utility maximization), it seems as inappropriate to disregard the alien Dyson spheres as the human ones...
Disregarding very low probabilities which are subjective credences applied to future scenarios we have too little understanding of to exclude (rather than frequencies inferred from actual observation of their rarity) is another means to the same end, of course.
The point about accounting for uncertainty is very well taken. I had not considered possible asymmetries in the effects of uncertainty when writing this.
On longtermism generally, I think my language in the post was probably less favorable to longtermism than I would ultimately endorse. As you say, the value of the future remains exceptionally large even after reduction by a few orders of magnitude, a fact that should hopefully be clear from the units (trillions of life-years) used in the graphs above.
If I have time in future, I may try to create new graphs for sensitivity and value that take into account uncertainty.
My central objection to Thorstadās work on this is the failure to properly account for uncertainty. Attempting to exclusively model a most-plausible scenario, and draw dismissive conclusions about longtermist interventions based solely on that, fails to reflect best practices about how to reason under conditions of uncertainty. (Iāve also raised this criticism against Schwitzgebelās negligibility argument.) You need to consider the full range of possible models /ā scenarios!
Itās essentially fallacious to think that āplausibly incorrect modeling assumptionsā undermine expected value reasoning. High expected value can still result from regions of probability space that are epistemically unlikely (or reflect āplausibly incorrectā conditions or assumptions). If thereās even a 1% chance that the relevant assumptions hold, just discount the output value accordingly. Astronomical stakes are not going to be undermined by lopping off the last two zeros.
Tarsneyās Epistemic Challenge to Longtermism is so much better at this. As he aptly notes, as long as youāre on board with orthodox decision theory (and so donāt disproportionately discount or neglect low-probability possibilities), and not completely dogmatic in refusing to give any credence at all to the longtermist-friendly assumptions (robust existential security after time of perils, etc.), reasonable epistemic worries ultimately arenāt capable of undermining the expected value argument for longtermism.
(These details can still be helpful for getting better-refined EV estimates, of course. But thatās very different from presenting them as an objection to the whole endeavor.)
This feels like an isolated demand for rigour, since as far as I can see Thorstadās[1] central argument isnāt that a particular course of the future is more plausible, but that [popular representations of] longtermist arguments themselves donāt consider the full range of possibilities, donāt discount for uncertainty, and that apparently modest-sounding claims that existential risk is non-zero and that humanity could last a long time if we survive near-term threats are compatible only if one makes strong claims about the hinginess of history[2]
I donāt see him trying to build a more accurate model of the future[3] so much as pointing out how very simple changes completely change longtermist models. As such, his models are intentionally simple and Owenās expansion above adds more value for anyone actively trying to model a range of future scenarios. But Iām not sure why it would be incumbent on the researcher arguing against choosing a course of action based on long term outcomes to be the one who explicitly models the entire problem space. Iād turn that around and question why longtermists who donāt consider the whole endeavour of predicting the long term future in our decision theory to be futile generally dogmatically reject low probability outcomes with Pascalian payoffs that favour the other option, or to simply assume the asymmetry of outcomes works in their favour.
Now personally Iām fine with āno, actually I think catastrophes are badā, but thatās because Iām focused on the near term where it really is obvious that nuclear holocausts arenāt going to have a positive welfare impact. Once weāre insisting that our decisions ought to be guided by tiny subjective credences in far future possibilities with uncertain likelihood but astronomic payoffs and that itās an error not to factor unlikely interstellar civilizations into our calculations of what we should do if theyāre big enough, it seems far less obvious that the astronomical stakes skew in favour of humanity.
The Tarsney paper even explicitly models the possibility of non-human galactic colonization, but with the unjustified assumption that no non-humans will be of converting resources to utility at a higher rate than [post]humans, so their emergence as competitors for galactic resources merely ānullifiesā the beneficial effects of humanity surviving. But from a total welfarist perspective, the problem here isnāt just that maximizing the possible welfare across the history of the universe may not be contingent on the long term survival of the human species,[4] but that humans surviving to colonise galaxies might diminish galactic welfare. Schweitzgebelās argument that actually human extinction might be net good for total welfare is only a mad hypothetical if you reject fanaticism: otherwise itās the logical consequence of accepting the possibility, however small, that a nonhuman species might convert resources to welfare much more efficiently than us.[5] Now a future of decibillions of aliens building Dyson Spheres all over the galaxy because thereās no pesky humans in their way sounds extremely unlikely, and perhaps even less likely than a galaxy filled with the same fantastic tech to support quadrillions of humansāa species we at least know exists and has some interest in inventing Dyson Spheresābut despite this the asymmetry of possible payoff magnitudes may strongly favour not letting us survive to colonise the galaxy.[6]
In the absence of any particular reason for confidence that the EV of one set of futures is definitely higher than the others, it seems like you end up reaching for heuristics like ābut letting everyone die would be insaneā. I couldnāt agree more, but the least arbitrary way to do that is to adjust the framework to privilege the short term with discount rates sufficiently high to neuter payoffs so speculative and astronomical we canāt rule out the possibility they exceed the payoff from [not] letting eight billion humans die[7] Since that discount rate reflects extreme uncertainty about what might happen and what payoffs might look like, it also feels more epistemically humble than basing an entire worldview on the long tail outcome of some low probability far futures whilst dismissing other equally unsubstantiated hypotheticals because their implications are icky. And Iām pretty sure this is what Thorstad wants us to do, not to place high credence in his point estimates or give up on X-risk mitigation altogether.
For the avoidance of doubt I am a different David T ;-)
which doesnāt of course mean that hinginess is untrue, but does make it less a general principle of caring about the long term and more a relatively bold and specific claim about the distribution of future outcomes.
in the arguments referenced here anyway. He has also written stuff which attempts to estimate different XR base rates from those posited by Ord et al, which I find just as speculative as the longtermistsā
there are of course ethical frameworks other than maximizing total utility across all species which give us reason to prefer 10^31 humans over a similarly low probability von Neumann civilization involving 10^50 aliens or a single AI utility monster (I actually prefer them, so no proposing destroying humanity as a cause area from me!) but theyāre a different from the framework Tarsney and most longtermists use, and open the door to other arguments for weighting current humans over far future humans.
Weāre a fairly stupid, fragile and predatory species capable of experiencing strongly negative pain and emotional valences at regular intervals over fairly short lifetimes, with competitive social dynamics, very specific survival needs and a wasteful approach to consumption, so it doesnāt seem obvious or even likely that humanity and its descendants will be even close to the upper bound for converting resources to welfare...
Of course, if you reject fanaticism, the adverse effects of humans not dying in a nuclear holocaust on alien utility monsters are far too remote and unlikely and frankly a bit daft to worry about. But if you accept fanaticism (and species-neutral total utility maximization), it seems as inappropriate to disregard the alien Dyson spheres as the human ones...
Disregarding very low probabilities which are subjective credences applied to future scenarios we have too little understanding of to exclude (rather than frequencies inferred from actual observation of their rarity) is another means to the same end, of course.
The point about accounting for uncertainty is very well taken. I had not considered possible asymmetries in the effects of uncertainty when writing this.
On longtermism generally, I think my language in the post was probably less favorable to longtermism than I would ultimately endorse. As you say, the value of the future remains exceptionally large even after reduction by a few orders of magnitude, a fact that should hopefully be clear from the units (trillions of life-years) used in the graphs above.
If I have time in future, I may try to create new graphs for sensitivity and value that take into account uncertainty.