Astrobiologist @ Imperial College London
Lead of the SGAC Cosmic Futures project.
Partnerships Coordinator @ SGAC Space Safety and Sustainability Project Group
Interested in: Space Governance, Great Power Conflict, Existential Risk, Cosmic threats, Academia, International policy
I didn’t write this post with the intention of criticising the importance of space governance, so I wouldn’t go as far as you. I think reframing space governance in the context of how it supports other cause areas reveals how important it really is. But space governance also has its own problems to deal with, so it’s not just a tool or a background consideration. Some (pressing) stuff that could be very bad in the 2030s (or earlier) without effective space governance:
China/Russia and the USA disagree over how to claim locations for a lunar base, and they both want to build one on the south pole. High potential for conflict in space (would also increase tensions on Earth). Really bad precedent for the long term future.
I think space mining companies have a high chance of accidentally changing the orbits of multiple asteroids, increasing the risk of short warning times from asteroids with suddenly altered orbits (or creation of lots of fragments that could damage satellites). No policy exists to protect against this risk.
Earth’s orbit is getting very full of debris and satellites. Another few anti-satellite weapons tests or a disaster involving a meteroid shower may trigger Kessler syndrome. Will Elon Musk de-orbit all of his thousands of Starlink satellites?
The footprints of the first humans to ever set foot on another celestial body still exist on the moon. They will be destroyed by lunar plumes caused by mining in the 2030s—this will be a huge blow to the long term future (I think it could even be the greatest cultural heritage of all time to a spacefaring civilisation and we’re gonna lose it). All it takes is one small box around some of the footprints to protect 90% of the value.
Earth’s orbit is filled with debris. The moon’s orbit is smaller and we can’t just get rid of satellites by burning them in the atmosphere. No policy exists to set a good precedent around that yet so the moon’s orbit will probably end up being even worse than Earth’s—people are already dodging each other’s satellites around the moon, and ESA & NASA want to build whole networks for moon internet.
My conclusions are different throughout the post including in the title! I’m still not sure whether space governance is more like international policy, or more like EA community building—maybe its a mix of the two, where it’s actually like international policy but we should treat it more like EA community building.
So either space governance is a “meta-cause area” or an “area of expertise”, but not a “cause area” in the sense that the term is most often used (i.e. a cause to address).
I disagree with the suggestion but I upvoted as I think it is an important discussion to have on the forum. Especially with the Musk example, longtermism gets a lot of criticism for ideas that aren’t associated with it (even in the space policy literature). But I agree with @Davidmanheim’s comment. Thanks for making the post!
Thanks for your reply, lots of interesting points :)
Consciousness may not be binary, in that case, we don’t know if humans are low, medium, or high consciousness, I only know that I am not at zero. We should then likely assume we are average. Then, the relevant comparison is no longer between P(humanity is “conscious”) and P(aliens creating SFCs are “conscious”) but between P(humanity’s consciousness > 0) and P(aliens-creating-SFC’s consciousness > 0)
I particularly appreciate that reframing of consciousness. I think it’s probably both binary and continuous though. Binary in the sense that you need a “machinery” that’s capable of producing consciousness i.e. neurons in a brain seem to work. And then if you have that capable machinery, you then have the range from low to high consciousness, like we see on Earth. If intelligence is related to consciousness level as it seems to be on Earth, then I would expect that any alien with “capable machinery” that’s intelligent enough to become spacefaring would have consciousness high enough to satisfy my worries (though not necessarily at the top of the range).
So then any alien civilisation would either be “conscious enough” or “not conscious at all”, conditional on (a) the machinery of life being binary in its ability to produce a scale of consciousness and (b) consciousness being correlated with intelligence.
So I’m not betting on it. The stakes are so high (a universe devoid of sentience) that I would have to meet and test the consciousness of aliens with a ‘perfect’ theory of consciousness before I updated any strategy towards reducing P(ancestral-human SFC) even if there’s an extremely high probability of Civ-Similarity Hypothesis being true.
Thanks David, that makes sense :)
The validity of this hypothesis can be studied using models estimating the frequency of Space-Faring Civilizations (SFCs) in the universe (Sandberg 2018, Finnveden 2019, Olson 2020, Hanson 2021, Snyder-Beattie 2021, Cook 2022). The validity will also depend on which decision theory we use and on our beliefs behind these
I’m very speculative about making moral decisions concerning the donations of potentially millions of dollars based on something so speculative. I think it’s too far down the EA crazy train to prioritise different causes based on the density of alien civilisations. It’s probably more speculative than the simulation hypothesis (which, if true, significantly increases the likelihood that you are the only sentient being in this universe), but we don’t make moral decisions based on that.
I get that there’s been a lot of work on this and that we can make progress on it (I know, I’m an astrobiologist), but I’m sure there are so many unknown unknowns associated with the origin of life, development of sentience, and spacefaring civilisation that we just aren’t there yet. The universe is so enormous and bonkers and our brains are so small—we can make numerical estimates sure, but creating a number doesn’t necessarily mean we have more certainty.
How much counterfactual value Humanity creates then depends entirely on the utility Humanity’s spacefaring civilisation creates relative to all spacefaring civilisations.
I’ve got a big moral circle (all sentient beings and their descendants), but it does not extend to aliens because of cluelessness.
I think you’re posing a post-understanding of consciousness question. Consciousness might be very special or it might be an emergent property of anything that synthesises information, we just don’t know. But it’s possible to imagine aliens with complex behaviour similar to us, but without evolving the consciousness aspect, like superintelligent AI probably will be like. For now, the safe assumption is that we’re the only conscious life, and I think it’s very important that we act like it until proven otherwise.
So for now, I’m quite confident that if we’re thinking about the moral utility of spacefaring civilisation, we should at least limit our scope to our own civilisation, more specifically, our own sentience and its descendants (I personally prefer to limit that scope even further to the next few thousand years, or just our Solar System to reduce the ambiguity a bit—longtermism still stands strong with this huge limitation). I think the main value in looking into the potential density of aliens in the universe helps figure out what our own future might look like. Even if humans only colonise the Solar System because alien SFCs colonise the galaxy, that’s still 10^27 potential future lives (1.2 sextillion over the next 6000 years; future life equivalents based on the Solar System’s carrying capacity; as opposed to 100 trillion if we stay on Earth till its destruction). We can control and predict that to an extent, and there’s enough ambiguity and cluelessness already associated with how to make human civilisation’s future in space good in the context of AI—but we can at least make some concrete decisions (e.g. work by Simon Institute & CLR).
Very interesting post though! Lots to think about and I can see that this could be the most important moral consideration… maybe… I look forward to your series and I definitely think it’s worthwhile to try and figure out what that consideration might be.
Other currently neglected agendas may increase P(Alignment | Humanity creates an SFC) while not increasing P(Alignment AND Humanity creates an SFC). Those include agendas aiming at decreasing P(Humanity creates an SFC | Misalignment). An example of intervention in such an agenda is overriding instrumental goals for space colonization and replacing them with an active desire not to colonize space. This defensive preference could be removed later, conditional on achieving corrigibility.
What’s the difference between “P(Alignment | Humanity creates an SFC)” and “P(Alignment AND Humanity creates an SFC)”?
I don’t get it either. Can you maybe run us through 2 worked examples for bullet point 2? Like what is someone currently doing (or planning to do) that you think should be deprioritised? And presumably, there might be something that you think should be prioritised instead?
I’m imagining here that you want to deprioritise an AI safety regime if it is focusing on making AIs that create technology that can be used for spacefaring civilisation, but aren’t aligned? That wouldn’t be an AI safety regime would it? That’s just creating AI that wants to leave Earth
I would be really interested in a post that outlined 1-3 different scenarios for post-AGI x-risk based on increasingly strict assumptions. So the first one would assume that misaligned superintelligent AI would almost instantly emerge from AGI, and describe the x-risks associated with that. Then the assumptions become stricter and stricter, like AGI would only be able to improve itself slowly, we would be able to align it to our goals etc.
I think this could be a valuable post to link people to, as a lot of debates around whether AI poses an x-risk seem to fall on accepting or rejecting potential scenarios, but it’s usually unproductive because everyone has different assumptions about what AI will be capable of.
So with this post, to say that AI x-risk is not tangible, then for each AI development scenario (with increasingly strict assumptions), you have to either:
reject at least one of the listed assumptions (e.g. argue that computer chips are a limit on exponential intelligence increases)
or argue that all proposed existential risks in that scenario are so impossible that even an AI wouldn’t be able to make any of them work.
If you can’t do either of those, you accept AI is an x-risk. If you can, you move on to the next scenario with stricter assumptions. Eventually you find the assumptions you agree with, and have to reject all proposed x-risks in that scenario to say that AI x-risk isn’t real.
The post might also help with planning for different scenarios if it’s more detailed than I’m anticipating.
I accept that I don’t know actual procedure for firing a nuclear weapon. And no one in the west knows what North Korea’s nuclear weapons cybersecurity is like, and ChatGPT tells me its connected to digital networks. So there’s definitely some uncertainty and I wouldn’t dismiss the possibility outright that nuclear weapons would be more likely to be hacked if superintelligence existed. So I’d guess maybe a 10-20% chance that it’s possible to hack nuclear weapons based on what I know.
And I agree that it may be impossible to create drexlerian style nanotech. Maybe a 0.5% chance an ASI could do something like that?
But I don’t think the debate here is about any particular scenario that I came up with.
I think if I tried really hard I could come up with about 20 scenarios where an artificial superintelligence might be able to destroy humanity (if you really want me to I can try and list them). And I guess my proposed scenarios would have an average chance of actually working of 1-2%, so maybe around 10% chance that one of my proposed scenarios would work.
But are you saying that the chance of ASI being able to kill us is 0%? In which case every conceivable scenario (including any plan that an ASI could come up with) would have to have a 0% chance of working? I just don’t find that possible, human civilisation isn’t that robust. It must be at least a 10% chance that one of those plans could work right? In which case significant efforts in AI safety to mitigate this risk are warranted.
Not an AI safety expert, but I think you might be under-estimating the imagination a superintelligent AI may have. Or maybe you’re under-estimating what a post-AGI intelligence may be capable of?
With the “killing humans through machines” option, a superintelligent AI would probably be smart enough to kill us all without taking the time to build a robot army, which would definitely raise my suspicions! Maybe it would hack nuclear weapons and blow us all up, invent and release an airborne super-toxin, or make a self-replicating nanobot—wouldn’t see it coming, over as soon as we realised it wasn’t aligned.
And for “making humans kill themselves”, it might destabilise Governments with misinformation or breakdown communication channels, leading to global conflicts where we kill each other. To stay on the nuclear weapons route, maybe it tricks the USA’s nuclear detection system into thinking that nuclear weapons have been fired, and the USA retaliates causing a nuclear war.
I think the fact that I can use my limited human intelligence to imagine multiple scenarios where AI could kill us all makes me very confident that a superintelligent AI would find additional and more effective methods. The question from my perspective isn’t “can AI kill us all?” but “how likely is it that AI can kill us all?”, and my answer is I don’t know but definitely not 0%. So here’s a huge problem that needs to be solved.
Full disclosure, I have no idea how AI alignment will overcome that problem, but I’m very glad they’re working on it.
No problem :)
Best of luck with your A-levels!
I don’t know about impactful paths related to art and music. Have you considered asking 80,000 hours for a free careers advising call? https://80000hours.org/speak-with-us/
There’s also some useful content on the 80,000 hours website on impactful careers in arts: https://80000hours.org/topic/careers/other-careers/art-entertainment/
From my perspective, I wouldn’t worry too much about what A-levels you choose at this stage. You’re not going to pigeonhole yourself by just choosing what you find interesting and are likely to succeed in—maybe try and pick a range of humanities and sciences to cover your bases.
I’d say if you’re really unsure then try and pick something broad like international politics (or geography; where you could specialise into x-risks, AI safety, climate change or any other global problem), physics (which gives you broad science/maths skills you can apply to a range of cause areas), maths, statistics, or philosophy (which definitely has broad applications, but maybe more debatable transferrable skills). This applies to doing a degree too, and what A levels you did won’t matter very much after you do the degree.
Thanks for this analysis! I was thinking about this issue a lot recently and I’ll definitely refer to your post going forward.
I wonder if large scale space resource activities might incidentally increase the risk of an asteroid impact in the long term though? Our Solar System has had billions of years for asteroids to settle into orbits. Asteroid mining and small alterations of asteroid trajectories could build up in the long term into chaotic effects. So we might get a situation where asteroid orbits are way more chaotic and difficult to predict across the Solar System, posing threats (or, more likely, extreme costs for deflection activities) to humans that might live in space or to anything with a gravity well. This might be similar to sending our Solar System into a situation more similar to the Late Heavy Bombardment 4bya, where large asteroid impacts were extremely common.
I think this threat is addressed by the same recommendations you make though for asteroid weaponization though, I’m particularly hopeful about advocating for more equitable governance of space resources. These policies will start to be locked in during the 2030s with the ISRU for lunar bases—unfortunately many nations like USA and Luxembourg appear to have little intention of sharing space resources and are using unilateral space policy to incentivize private industry.
Interestingly, the singularity could actually have the opposite effect. Where originally human exploration of the Solar System was decades away, extremely intelligent AI could speed up technology to where it’s all possible within a decade.
The space policy landscape is not ready for that at all. There is no international framework for governing the use of space resources, and human exploration is still technically illegal on Mars due to contamination of the surface (and the moon! Yes we still care a lot).
So I lean more towards superintelligent AI being a reason to care more about space, not less. Will Macaskill discusses it in more detail here.
Haha good point, that’s precisely why I asked.
I’ve just put together the trial version of “Actions for Impact”, the newsletter to leverage the effective altruism (EA) community’s size to complete quick high impact tasks to support EA cause areas. I’m getting feedback on the first version at the moment.
DM me on the forum if you’re interested and I’ll send you the first version—very welcome to feedback!
That’s fair