I feel like the discussion of AI is heavily underemphasized in this problem profile (in fact, in this post it is the last thing mentioned).
I used to casually think “sure, space governance seems like it could be a good idea to start on soon; space exploration needs to happen eventually, I guess,” but once I started to consider the likelihood and impact of AI development within the next 200 or even ~60 years, I very heavily adjusted my thinking towards skepticism/pessimism.
That question of AI development seems like a massive gatekeeper/determinant to this overall question: I’m unclear how any present efforts towards long-term space governance and exploration matter in the case where AI 1) is extraordinarily superintelligent and agentic, and 2) operates effectively as a “singleton”—which itself seems like a likely outcome from (1).
Some scenarios that come to my mind regarding AI development (with varying degrees of plausibility):
We create a misaligned superintelligence which leads to extinction or other forms of significant curtailment of humanity’s cosmic potential, which renders all of our efforts towards space exploration unimportant for the long term.
We create an aligned, agentic superintelligent singleton which basically renders all of our efforts towards space exploration unimportant for the long term (because it will very very quickly surpass all of our previous reasoning and work).
We somehow end up with multiple highly intelligent agents (e.g., national AIs) that are somewhat “aligned” with certain values, but their intelligence does not enable them to identify/commit to positive-sum cooperation strategies (e.g., they cannot converge towards a singleton) and this curtails space expansion capabilities, but having developed norms in advance helps to (slightly?) mitigate this curtailment.
We determine that the alignment problem is unsolvable or too inherently risky to try to develop a superintelligence—at least for a few centuries—but we are also somehow able to prevent individual/unilateral actors from trying to create superintelligent agents, and so it may be worthwhile to get a “headstart” on space exploration, even if it only equates to improving our long term future by some (0.00...1%)
Creating superintelligence/AGI proves far more elusive than we expect currently (and/or the alignment problem is difficult, see above) and thus takes many decades or even centuries longer, while at the same time space becomes a domain that could trigger hostilities or tensions that undermine coordination on existential risks (including AI).
??
Ultimately, I’d really like to see:
More up-front emphasis on the importance of AI alignment as a potential determinant.
Examination of the scenarios in which work on space governance would have been useful, given the first point, including how likely those scenarios appear to be.
This sentiment seems like a fully general objection to every intervention not directly related to AI safety (or TAI).
As presented currently, many TAI or AI safety related scenarios blow out all other considerations—it won’t matter how far you get to Alpha Centauri with prosaic spaceships, TAI will track you down.
It seems like you would need to get “altitude” to give this consideration proper thought (pardon the pun). My guess is that the OP has done that.
This is partially an accurate objection (i.e., I do think that x-risks and other longtermist concerns tend to significantly outweigh near-term problems such as in health and development), but there is an important distinction to make with my objections to certain aspects of space governance:
Contingent on AI timelines, there is a decent chance that none of our efforts will even have a significantly valuable near-term effect (i.e., we won’t achieve our goals by the time we get AGI). Consider the following from the post/article:
If the cost of travelling to other planetary bodies continues the trend in the chart above and falls by an order of magnitude or so, then we might begin to build increasingly permanent and self-sustaining settlements. Truly self-sustaining settlements are a long way off, but both NASA and China have proposed plans for a Moon base, and China recently announced plans to construct a base on Mars.
Suppose that it would take ~80 years to develop meaningful self-sustaining settlements on Mars without AGI or similar forms of superintelligence. But suppose that we get AGI/superintelligence in ~60 years: we might get misaligned AGI and all the progress (and humanity) is erased and it fails to achieve its goals; we might create aligned AGI which might obsolesce all ~60 years of progress within 5 or so years (I would imagine even less time); or we might get something unexpected or in between, in which case maybe it does matter?
In contrast, at least with health and development causes you can argue “I let this person live another ~50 years… and then the AGI came along and did X.”
Furthermore, this all is based on developing self-sustaining settlements being a valuable endeavor, which I think is often justified with ideas that we’ll use those settlements for longer-term plans and experimentation for space exploration, which requires an even longer timeline.
Thanks for this, I think I agree with the broad point you’re making.
That is, I agree that basically all the worlds in which space ends up really mattering this century are worlds in which we get transformative AI (because scenarios in which we start to settle widely and quickly are scenarios in which we get TAI). So, for instance, I agree that there doesn’t seem to be much value in accelerating progress on space technology. And I also agree that getting alignment right is basically a prerequisite to any of the longer-term ‘flowthrough’ considerations.
If I’m reading you right I don’t think your points apply to near-term considerations, such as from arms control in space.
It seems like a crux is something like: how much precedent-setting or preliminary research now on ideal governance setups doesn’t get washed out once TAI arrives, conditional on solving alignment? And my answer is something like: sure, probably not a ton. But if you have a reason to be confident that none of it ends up being useful, it feels like that must be a general reason for scepticism that any kind of efforts at improving governance, or even values change, are rendered moot by the arrival of TAI. And I’m not fully sceptical about those efforts.
Suppose before TAI arrived we came to a strong conclusion: e.g. we’re confident we don’t want to settle using such-and-such a method, or we’re confident we shouldn’t immediately embark on a mission to settle space once TAI arrives. What’s the chance that work ends up making a counterfactual difference, once TAI arrives? Notquite zero, it seems to me.
So I am indeed on balance significantly less excited about working on long-term space governance things than on alignment and AI governance, for the reasons you give. But not so much that they don’t seem worth mentioning.
Ultimately, I’d really like to see [...] More up-front emphasis on the importance of AI alignment as a potential determinant.
This seems like a reasonable point, and one I was/am cognisant of — maybe I’ll make an addition if I get time.
(Happy to try saying more about any of above if useful)
If I’m reading you right I don’t think your points apply to near-term considerations, such as from arms control in space.
That is mostly correct: I wasn’t trying to respond to near-term space governance concerns, such as how to prevent space development or space-based arms races, which I think could indeed play into long-term/x-risk considerations (e.g., undermining cooperation in AI or biosecurity), and may also have near-term consequences (e.g., destruction of space satellites which undermines living standards and other issues).
But if you have a reason to be confident that none of it ends up being useful, it feels like that must be a general reason for scepticism that any kind of efforts at improving governance, or even values change, are rendered moot by the arrival of TAI. And I’m not fully sceptical about those efforts.
To summarize the point I made in response to Charles (which I think is similar, but correct me if I’m misunderstanding): I think that if an action is trying to improve things now (e.g., health and development, animal welfare, improving current institutional decision-making or social values), it can be justified under neartermist values (even if it might get swamped by longtermist calculations). But it seems that if one is trying to figure out “how do we improve governance of space settlements and interstellar travel that could begin 80–200 years from now,” they run the strong risk of their efforts having effectively no impact on affairs 80–200 years from now because AGI might develop before their efforts ever matter towards the goal, and humanity either goes extinct or the research is quickly obsolesced.
Ultimately, any model of the future needs to take into account the potential for transformative AI, and many of the pushes such as for Mars colonization just do not seem to do that, presuming that human-driven (vs. AI-driven) research and efforts will still matter 200 years from now. I’m not super familiar with these discussions, but to me this point stands out so starkly as 1) relatively easy to explain (although it may require introductions to superintelligence for some people); 2) substantially impactful on ultimate conclusions/recommendations, and 3) frequently neglected in the discussions/models I’ve heard so far. Personally, I would put points like this among the top 3–5 takeaway bullet points or in a summary blurb—unless there are image/optics reasons to avoid doing this (e.g., causing a few readers to perhaps-unjustifiably roll their eyes and disregard the rest of the problem profile).
Suppose before TAI arrived we came to a strong conclusion: e.g. we’re confident we don’t want to settle using such-and-such a method, or we’re confident we shouldn’t immediately embark on a mission to settle space once TAI arrives. What’s the chance that work ends up making a counterfactual difference, once TAI arrives? Notquite zero, it seems to me.
This is an interesting point worth exploring further, but I think that it’s helpful to distinguish—perhaps crudely?—between two types of problems:
Technical/scientific problems and “moral problems” which are really just “the difficulty of understanding how our actions will relate to our moral goals, including what sub-goals we should have in order to achieve our ultimate moral goals (e.g., maximizing utility, maximizing virtue/flourishing).”
Social moral alignment—i.e., getting society to want to make more-moral decisions instead of being self-interested at the expense of others.
It seems to me that an aligned superintelligence would very likely be able to obsolesce every effort we make towards the first problem fairly quickly: if we can design a human-aligned superintelligent AI, we should be able to have it automate or at least inform us on everything from “how do we solve this engineering problem” to “will colonizing this solar system—or even space exploration in general—be good per [utilitarianism/etc.]?”
However, making sure that humans care about other extra-terrestrial civilizations/intelligence—and that the developers of AI care about other humans (and possibly animals)—might require some preparation such as via moral circle expansion. Additionally, I suppose it might be possible that a TAI’s performance on the first problem is not as good as we expect (perhaps due to the second problems), and of course there are other scenarios I described where we can’t rely as much on a (singleton) superintelligence, but my admittedly-inexperienced impression is that such scenarios seem unlikely.
I feel like the discussion of AI is heavily underemphasized in this problem profile (in fact, in this post it is the last thing mentioned).
I used to casually think “sure, space governance seems like it could be a good idea to start on soon; space exploration needs to happen eventually, I guess,” but once I started to consider the likelihood and impact of AI development within the next 200 or even ~60 years, I very heavily adjusted my thinking towards skepticism/pessimism.
That question of AI development seems like a massive gatekeeper/determinant to this overall question: I’m unclear how any present efforts towards long-term space governance and exploration matter in the case where AI 1) is extraordinarily superintelligent and agentic, and 2) operates effectively as a “singleton”—which itself seems like a likely outcome from (1).
Some scenarios that come to my mind regarding AI development (with varying degrees of plausibility):
We create a misaligned superintelligence which leads to extinction or other forms of significant curtailment of humanity’s cosmic potential, which renders all of our efforts towards space exploration unimportant for the long term.
We create an aligned, agentic superintelligent singleton which basically renders all of our efforts towards space exploration unimportant for the long term (because it will very very quickly surpass all of our previous reasoning and work).
We somehow end up with multiple highly intelligent agents (e.g., national AIs) that are somewhat “aligned” with certain values, but their intelligence does not enable them to identify/commit to positive-sum cooperation strategies (e.g., they cannot converge towards a singleton) and this curtails space expansion capabilities, but having developed norms in advance helps to (slightly?) mitigate this curtailment.
We determine that the alignment problem is unsolvable or too inherently risky to try to develop a superintelligence—at least for a few centuries—but we are also somehow able to prevent individual/unilateral actors from trying to create superintelligent agents, and so it may be worthwhile to get a “headstart” on space exploration, even if it only equates to improving our long term future by some (0.00...1%)
Creating superintelligence/AGI proves far more elusive than we expect currently (and/or the alignment problem is difficult, see above) and thus takes many decades or even centuries longer, while at the same time space becomes a domain that could trigger hostilities or tensions that undermine coordination on existential risks (including AI).
??
Ultimately, I’d really like to see:
More up-front emphasis on the importance of AI alignment as a potential determinant.
Examination of the scenarios in which work on space governance would have been useful, given the first point, including how likely those scenarios appear to be.
This sentiment seems like a fully general objection to every intervention not directly related to AI safety (or TAI).
As presented currently, many TAI or AI safety related scenarios blow out all other considerations—it won’t matter how far you get to Alpha Centauri with prosaic spaceships, TAI will track you down.
It seems like you would need to get “altitude” to give this consideration proper thought (pardon the pun). My guess is that the OP has done that.
This is partially an accurate objection (i.e., I do think that x-risks and other longtermist concerns tend to significantly outweigh near-term problems such as in health and development), but there is an important distinction to make with my objections to certain aspects of space governance:
Contingent on AI timelines, there is a decent chance that none of our efforts will even have a significantly valuable near-term effect (i.e., we won’t achieve our goals by the time we get AGI). Consider the following from the post/article:
Suppose that it would take ~80 years to develop meaningful self-sustaining settlements on Mars without AGI or similar forms of superintelligence. But suppose that we get AGI/superintelligence in ~60 years: we might get misaligned AGI and all the progress (and humanity) is erased and it fails to achieve its goals; we might create aligned AGI which might obsolesce all ~60 years of progress within 5 or so years (I would imagine even less time); or we might get something unexpected or in between, in which case maybe it does matter?
In contrast, at least with health and development causes you can argue “I let this person live another ~50 years… and then the AGI came along and did X.”
Furthermore, this all is based on developing self-sustaining settlements being a valuable endeavor, which I think is often justified with ideas that we’ll use those settlements for longer-term plans and experimentation for space exploration, which requires an even longer timeline.
Thanks for this, I think I agree with the broad point you’re making.
That is, I agree that basically all the worlds in which space ends up really mattering this century are worlds in which we get transformative AI (because scenarios in which we start to settle widely and quickly are scenarios in which we get TAI). So, for instance, I agree that there doesn’t seem to be much value in accelerating progress on space technology. And I also agree that getting alignment right is basically a prerequisite to any of the longer-term ‘flowthrough’ considerations.
If I’m reading you right I don’t think your points apply to near-term considerations, such as from arms control in space.
It seems like a crux is something like: how much precedent-setting or preliminary research now on ideal governance setups doesn’t get washed out once TAI arrives, conditional on solving alignment? And my answer is something like: sure, probably not a ton. But if you have a reason to be confident that none of it ends up being useful, it feels like that must be a general reason for scepticism that any kind of efforts at improving governance, or even values change, are rendered moot by the arrival of TAI. And I’m not fully sceptical about those efforts.
Suppose before TAI arrived we came to a strong conclusion: e.g. we’re confident we don’t want to settle using such-and-such a method, or we’re confident we shouldn’t immediately embark on a mission to settle space once TAI arrives. What’s the chance that work ends up making a counterfactual difference, once TAI arrives? Notquite zero, it seems to me.
So I am indeed on balance significantly less excited about working on long-term space governance things than on alignment and AI governance, for the reasons you give. But not so much that they don’t seem worth mentioning.
This seems like a reasonable point, and one I was/am cognisant of — maybe I’ll make an addition if I get time.
(Happy to try saying more about any of above if useful)
That is mostly correct: I wasn’t trying to respond to near-term space governance concerns, such as how to prevent space development or space-based arms races, which I think could indeed play into long-term/x-risk considerations (e.g., undermining cooperation in AI or biosecurity), and may also have near-term consequences (e.g., destruction of space satellites which undermines living standards and other issues).
To summarize the point I made in response to Charles (which I think is similar, but correct me if I’m misunderstanding): I think that if an action is trying to improve things now (e.g., health and development, animal welfare, improving current institutional decision-making or social values), it can be justified under neartermist values (even if it might get swamped by longtermist calculations). But it seems that if one is trying to figure out “how do we improve governance of space settlements and interstellar travel that could begin 80–200 years from now,” they run the strong risk of their efforts having effectively no impact on affairs 80–200 years from now because AGI might develop before their efforts ever matter towards the goal, and humanity either goes extinct or the research is quickly obsolesced.
Ultimately, any model of the future needs to take into account the potential for transformative AI, and many of the pushes such as for Mars colonization just do not seem to do that, presuming that human-driven (vs. AI-driven) research and efforts will still matter 200 years from now. I’m not super familiar with these discussions, but to me this point stands out so starkly as 1) relatively easy to explain (although it may require introductions to superintelligence for some people); 2) substantially impactful on ultimate conclusions/recommendations, and 3) frequently neglected in the discussions/models I’ve heard so far. Personally, I would put points like this among the top 3–5 takeaway bullet points or in a summary blurb—unless there are image/optics reasons to avoid doing this (e.g., causing a few readers to perhaps-unjustifiably roll their eyes and disregard the rest of the problem profile).
This is an interesting point worth exploring further, but I think that it’s helpful to distinguish—perhaps crudely?—between two types of problems:
Technical/scientific problems and “moral problems” which are really just “the difficulty of understanding how our actions will relate to our moral goals, including what sub-goals we should have in order to achieve our ultimate moral goals (e.g., maximizing utility, maximizing virtue/flourishing).”
Social moral alignment—i.e., getting society to want to make more-moral decisions instead of being self-interested at the expense of others.
It seems to me that an aligned superintelligence would very likely be able to obsolesce every effort we make towards the first problem fairly quickly: if we can design a human-aligned superintelligent AI, we should be able to have it automate or at least inform us on everything from “how do we solve this engineering problem” to “will colonizing this solar system—or even space exploration in general—be good per [utilitarianism/etc.]?”
However, making sure that humans care about other extra-terrestrial civilizations/intelligence—and that the developers of AI care about other humans (and possibly animals)—might require some preparation such as via moral circle expansion. Additionally, I suppose it might be possible that a TAI’s performance on the first problem is not as good as we expect (perhaps due to the second problems), and of course there are other scenarios I described where we can’t rely as much on a (singleton) superintelligence, but my admittedly-inexperienced impression is that such scenarios seem unlikely.