Global moratorium on AGI, now (Twitter). Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn ⏸️
Karma: 5,776
It therefore seems better to prioritize our descendant moral patients conditional on our survival because there are far far more of them.
I think in practical terms this isn’t mutually exclusive with ensuring our survival. The immediate way to secure our survival, at least for the next decade or so, is a global moratorium on ASI. This also reduces s-risks from ASI, and keeps our options open for reducing human-caused s-risk (i.e. we can still avoid factory farming in space colonization).
If alignment is hopeless (and I think it is), we should work on preventing ASI from ever being built! That’s what I’m doing.
Personally, I think p(ASI in the next 5 years)>70%, and p(death|ASI)~90%. And this is wholly unacceptable just in terms of my own survival, let alone everyone else’s. Philosophically justifying such a risk of death does not help when it’s becoming so viscerally real. See also my comments on this post.
Not be conscious: shares no evolutionary history or biology with us (I guess it’s possible it could find a way to upload itself into biology though..)
I think Will MacAskill and Finn Morehouse’s paper rests on the crucial consideration that aligning ASI is possible (by anyone at all). They haven’t established this (EDIT: by this I mean they don’t cite to any supporting arguments for this, rather than personally coming up with the arguments themselves. But as far as I know, there aren’t any supporting arguments for the assumption, and in fact there are good arguments on the other side for why aligning ASI is fundamentally impossible).
Singleton takeover seems very likely simply down to the speed advantage of the first mover (at the sharp end of the intelligence explosion it will be able to do subjective decades of R&D before the second mover gets off the ground, even if the second mover is only hours behind).
Interesting. What makes you confident about AI consciousness?
>suppose that the best attainable futures are 1000 times better than the default non-extinction scenario
This seems rather arbitrary. Why woiuld preventing extinction now guarantee that we (forever) lose that 1000x potential?
>In this toy model, you should only allocate your resources to reducing extinction if it is 10 times more tractable than ensuring we are on track to get the best possible future, at the current margin.
I think it is. Gaining the best possible future requires aligning an ASI, which has not been proven to be even theoretically possible afaik.
I think it rests a lot on conditional value, and that is very unsatisfactory from a simple moral perspective of wanting to personally survive and have my friends and family survive. If extinction risk is high, and near (and I think it is!) we should be going all out to prevent it (i.e. pushing for a global moratorium on ASI). We can then work out the other issues once we have more time to think about them (rather than hastily punting on a long shot of surviving just because it appears higher EV now).
How do these considerations affect what you are doing / spending resources on? Does it change the calculus if extinction is likely to happen sooner? (see also comment here).
Question: what level of extinction risk are people personally willing to accept in order to realise higher expected value in the futures where we survive? How much would the extinction coming in the next 5 years effect this? Or the next 1 year? How is this reflected in terms of what you are working on / spending resources on?
So in the debate week statement (footnote 2) it says “earth-originating intelligent life”. What if you disagree that AI counts as “life”? I expect that a singleton ASI will take over and will not be sentient or conscious, or value anything that humans value (i.e. the classic Yudkowskian scenario).
Yes, this is yet another reason for a moratorium on further-AGI development imo. If everyone has a genie with unlimited wishes, and are all pushing the world in different directions, the result will be chaos. Yampolskiy’s solution to this is everyone having their own private solipsistic universe simulations...
My position is that Timelines are short, p(doom) is high: a global stop to frontier AI development until x-safety consensus is our only reasonable hope (this post needs updating, to factor in things like inference time compute scaling, but my conclusions remain the same).
The problem is that no one has even established whether aligning or controlling ASI is theoretically, let alone practically, possible. Everything else (whether there is a human future at all past the next few years) is downstream of that.
And they are worse than big oil companies and big tobacco companies.
They are big AGI companies.
After nearly 7 years, I intend to soon step down as Executive Director of CEEALAR, founded by me as the EA Hotel in 2018. I will remain a Trustee, but take more of a back seat role. This is in order to focus more of my efforts on slowing down/pausing/stoping AGI/ASI, which for some time now I’ve thought of as being the most important, neglected and urgent cause.
We are hiring for my replacement. Please apply if you think you’d be good in the role! Or send on to others you’d like to see in the role. I’m hoping that we find someone who is highly passionate about CEEALAR, and able to take it to the next level (possibly even franchising the model to other locations? Something that has been talked about a lot for various locations but has still yet to happen.)
Nothing short of a global non-proliferation treaty on ASI (or a Pause, for short) is going to save us. So we have to make it realistic. We have to always be bringing comms back to that.
In terms of explaining the problem to public audience, lethalintelligence.ai is great.
I don’t think it’s discount rate (esp given short timelines); I think it’s more that people haven’t really thought about why their p(doom|ASI) is low. But people seem remarkably resistant to actually tackle the cruxes of the object level arguments, or fully extrapolate the implications of what they do agree on. When they do, they invariably come up short.
Or more basic things like religion, nationalism. People will want to shape their utopias in the image of their religious concept of heaven, and the idealised versions of their countries.