Global moratorium on AGI, now (Twitter). Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn ⏸️
Karma: 5,713
AI Safety work is starting to make real progress for avoiding the worst outcomes
What makes you think this? Every technique there is is statistical in nature (due to the nature of the deep learning paradigm), and none are even approaching 3 9s of safety and we need something like 13 9s if we are going to survive more than a few years of ASI.
AI capabilities advancement seems to be on a relatively good path (less foomy)
I also don’t see how it’s less foomy. SWE bench and ML researcher automation are still improving—what happens when the models are drop in replacements for top researchers?
Yet gradual disempowerment risks seem extremely hard to mitigate
What is the eventual end result after total disempowerment? Extinction, right?
Wait, how does your 93% disagree tie in with your support for PauseAI?
Does this change if our chance of extinction in the next few years is high? (Which I think it is, from AI).
Vinding says:
There is a key point on which I agree strongly with advocates for an AI pause: there is a massive moral urgency in ensuring that we do not end up with horrific AI-controlled outcomes. Too few people appreciate this insight, and even fewer seem to be deeply moved by it.
At the same time, I think there is a similarly massive urgency in ensuring that we do not end up with horrific human-controlled outcomes. And humanity’s current trajectory is unfortunately not all that reassuring with respect to either of these broad classes of risks …
The upshot for me is that there is a roughly equal moral urgency in avoiding each of these categories of worst-case risksBut he does not justify this equality. It seems highly likely to me that ASI-induced s-risks are on a much larger scale than human-induced ones (down to ASI being much more powerful than humanity), creating a (massive) asymmetry in favour of preventing ASI.
Agree. But I’m sceptical that we could robustly align or control a large population of such AIs (and how would we cap the population?), especially considering the speed advantage they are likely to have.
Yeah, I think a lot of the overall debate—including what is most ethical to focus on(!) -- depends on AI trajectories and control.
What level of intelligence are you imagining such a system as being at? Some percentile on the scale of top performing humans? Somewhat above the most intelligent humans?
Not sure why this is downvoted, it isn’t a rhetorical question—I genuinely want to know the answer.
Fair point. But AI is indeed unlikely to top out at merely “slighlty more” intelligent. And it has the potential for a massive speed/numbers advantage too.
Why do you think this? What make you think that it’s possible at all?[1] And what do you mean by “large minority”? Can you give an approximate percentage?
- ^
Or to paraphrase Yampolskiy: what makes it possible for a less intelligent species to indefinitely control a more intelligent species (when this has never happened before)?
- ^
Thinking about it some more, I think I mean something more like “subjective decades of strategising and preparation at the level of intelligence of the second mover”, so it would be able to counter anything the second mover does to try and gain power.
But also there would be software intelligence explosion effects (I think the figures you have in your footnote 37 are overly conservative—human level is probably closer to “GPT-5″).
I don’t think this is likely to happen though, absent something like moral realism being true, centred around sentient experiences, and the AI discovering this.
I think it hinges on whether our AI successors would be counted as “life”, or whether they “matter morally”. I think the answer is likely no to both[1]. Therefore the risk of extinction boils down to risk of misaligned ASI wiping out the biosphere. Which I think is ~90% likely this century on the default trajectory, absent a well enforced global moratorium on ASI development.
- ^
Or at least “no” to the latter; if we consider viruses to be life that don’t matter morally, or are in fact morally negative, we can consider (default rogue) ASI to be similar.
- ^
Fair point. But this applies to a lot of things in EA. We give what we can.
Any realistic Pause would not be lifted absent a global consensus on proceeding with whatever risk remains.
the AI capability level that poses a meaningful risk of human takeover comes earlier than the AI capability level that poses a meaningful risk of AI takeover.
I don’t think it comes meaningfully earlier. It might only be a few months (an AI capable of doing the work of a military superpower would be capable of doing most work involved in AI R&D, precipitating an intelligence explosion). And the humans wielding the power will lose it to the AI too, unless they halt all further development of AI (which seems unlikely, due to hubris/complacency, if nothing else).
starting off with almost none (which will be true of the ASI)
Any ASI worthy of the name would probably be able to go straight for an unstoppable nanotech computronium grey goo scenario.
To bank on that we would need to have established at least some solid theoretical grounds for believing it’s possible—do you know of any? I think in fact we are closer to having the opposite: solid theoretical grounds for believing it’s impossible!
I could believe consciousness arose more than once in the tree of life (convergent evolution has happened for other things like eyes and flight).
But also, it’s probably a sliding scale, and the simple ancestor may well be at least minimally conscious.in which case consciousness would seem easy to reproduce (wire together some ~1,000 nerves).
Fair point. AI could well do this (and go as far as uploading into much larger biological structures, as I pointed to above).
I think that without knowing people’s assessment of extinction risk (e.g. chance of extinction over the next 5, 10, 20, 50, 100 years)[1], the answers here don’t provide a lot of information value.
I think a lot of people on the disagree side would change their mind if they believed (as I do) that there is a >50% chance of extinction in the next 5 years (absent further intervention).
Would be good if there was a short survey to establish such background assumptions to people’s votes.
And their assessment of the chance that AI successors will be morally valuable, as per footnote 2 of the statement.