Global moratorium on AGI, now (Twitter). Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn ⏸️
Karma: 5,749
Ilya’s company website says “Superintelligence is within reach.” I think it’s reasonable to interpret that as having a short timeline. If not an even stronger claim that he thinks he knows how to actually build it.
The post gives a specific example of this: the “software intelligence explosion” concept.
Right, and doesn’t address any of the meat in the methodology section.
I don’t think it’s nitpicky at all. A trend showing small, increasing numbers, just above 0, is very different (qualitatively) to a trend that is all flat 0s, as Ben West points out.
I am curious to see what will happen in 5 years when there is no AGI.
If this happens, we will at least know a lot more about how AGI works (or doesn’t). I’ll be happy to admit I’m wrong (I mean, I’ll be happy to still be around, for a start[1]).
- ^
I think the most likely reason we won’t have AGI in 5 years is that there will be a global moratorium on further development. This is what I’m pushing for.
- ^
I think Chollet has shifted the goal posts a bit from when he first developed ARC [ARC-AGI 1]. In his original paper from 2019, Chollet says:
“We argue that ARC [ARC-AGI 1] can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.”
And the original announcement (from June 2024) says:
A solution to ARC-AGI [1], at a minimum, opens up a completely new programming paradigm where programs can perfectly and reliably generalize from an arbitrary set of priors. We also believe a solution is on the critical path towards AGI”
(And ARC-AGI 1 has now basically been solved). You say:
I understand the theory that AI will have a super fast takeoff, so that even though it isn’t very capable now, it will match and surpass human capabilities within 5 years. But this kind of theory is consistent with pretty much any level of AI performance in the present.
But we are seeing a continued rapid improvement in A(G)I capabilities, not least along the trajectory to automating AGI development, as per the METR report Ben West mentions.
I was not being disingenuous and I find your use of the word “disingenuous” here to be unnecessarily hostile.
I was going off of the numbers in the recent blog post from March 24, 2025. The numbers I stated were accurate as of the blog post.
GPT-2 is not mentioned in the blog post. Nor is GPT-3. Or GPT3.5. Or GPT-4. Or even GPT-4o! You are writing 0.0% a lot for effect. In the actual blog post, there are only two 0.0% entries, for “gpt-4.5 (Pure LLM)”, and “o3-mini-high (Single CoT)”; and note the limitations in parenthesis, which you also neglect to include in your list (presumably for effect? Given their non-zero scores when not limited in such ways.)
In another comment you accuse me of being “unnecessarily hostile”. Yet to me, your whole paragraph in the OP here is unnecessarily hostile (somewhat triggering, even):
The community of people most focused on keeping up the drumbeat of near-term AGI predictions seems insular, intolerant of disagreement or intellectual or social non-conformity (relative to the group’s norms), and closed-off to even reasonable, relatively gentle criticism (whether or not they pay lip service to listening to criticism or perform being open-minded). It doesn’t feel like a scientific community. It feels more like a niche subculture. It seems like a group of people just saying increasingly small numbers to each other (10 years, 5 years, 3 years, 2 years), hyping each other up (either with excitement or anxiety), and reinforcing each other’s ideas all the time. It doesn’t seem like an intellectually healthy community.
Calling that sentence uncharitable was an understatement.
For instance, you don’t acknowledge that the top 3 most cited AI scientists of all time, all have relatively short timelines now.
As for the post you link, it starts with “I have not read the whole thing in detail”. I think far too many people critiquing it have not actually read it properly. If they did read it all in detail, they might find that their objections have been answered in one of the many footnotes, appendices, and accompanying research reports. It concludes with “It doesn’t really engage with my main objections, nor is it trying to do so”, but nowhere are the main objections actually stated! It’s all just meta commentary.
This is somewhat disingenuous. o3-mini (high) is actually on 1.5%, and none of the other models are reasoning (CoT / RL / long inference time) models (oh, and GPT 4.5 is actually on 0.8%). The actual leaderboard looks like this:
Yes the scores are still very low, but it could just be a case of the models not yet “grokking” such puzzles. In a generation or two they might just grok them and then jump up to very high scores (many benchmarks have gone like this in the past few years).
It seems like a group of people just saying increasingly small numbers to each other (10 years, 5 years, 3 years, 2 years), hyping each other up
This is very uncharitable. Especially in light of the recent AI 2027 report, which goes into a huge amount of detail (see also all the research supplements).
No, just saying that without their massive injection of cash, Anthropic might not be where they are today. I think the counterfactual where there wasn’t any “EA” investment into Anthropic would be significantly slower growth of the company (and, arguably, one fewer frontier AI company today).
Re Anthropic and (unpopular) parallels to FTX, just thinking that it’s pretty remarkable that no one has brought up the fact that SBF, Caroline Ellison and FTX were major funders of Anthropic. Arguably Anthropic wouldn’t be where they are today without their help! It’s unfortunate the journalist didn’t press them on this.
Anthropic leadership probably does lack the integrity needed to do complicated power-seeking stuff that has the potential to corrupt.
Yes. It’s sad to see, but Anthropic is going the same way as OpenAI, despite being founded by a group that split from OpenAI over safety concerns. Power (and money) corrupts. How long until another group splits from Anthropic and the process repeats? Or actually, one can hope that such a group splitting from Anthropic might actually have integrity and instead work on trying to stop the race.
I was thinking less in terms of fraud/illegality, and more in terms of immorality/negative externalities (i.e. they are recklessly endangering everyone’s lives).
No, but the main orgs in EA can still act in this regard. E.g. Anthropic shouldn’t be welcome at EAG events. They shouldn’t have their jobs listed on 80k. They shouldn’t be collaborated with on research projects etc that allow them to “safety wash” their brand. In fact, they should be actively opposed and protested (as PauseAI have done).
Fair points. I was more thinking in broad terms of supporting something that will most likely turn out hugely negative. I think it’s pretty clear already that Anthropic is massively negative expected value for the future of humanity. And we’ve already got the precedent of OpenAI and how that’s gone (and Anthropic seems to be going the same way in broad terms—i.e. not caring about endangering 8 billion people’s lives with reckless AGI/ASI development).
Downvoters note: there was actually far less publicly available information to update on FTX being bad in early 2022.
Yeah this is pretty damning. At this point, being pro-Anthropic is like being pro-FTX in early 2022.
(EDITED to add, following David Manheim’s recommendation: downvoters and disagree-voters note: there was actually far less publicly available information to update on FTX being bad in early 2022.)
It appears that Anthropic has made a communications decision to distance itself from the EA community, likely because of negative associations the EA brand has in some circles.
This works both ways. EA should be distancing itself from Anthropic, given recent pronouncements by Dario about racing China and initiating recursive self-improvement. Not to mention their pushing of the capabilities frontier.
I am guessing you agree with this abstract point (but furthermore think that AI takeover risk is extremely high, and as such we should ~entirely focus on preventing it).
Yes (but also, I don’t think the abstract point is adding anything, because of the risk actually being significant.)
Maybe I’m splitting hairs, but “x-risk could be high this century as a result of AI” is not the same claim as “x-risk from AI takeover is high this century”, and I read you as making the latter claim (obviously I can’t speak for Wei Dai).
This does seem like splitting hairs. Most of Wei Dai’s linked list is about AI takeover x-risk (or at least x-risk as a result of actions that AI might take, rather than actions that humans controlling AIs might take). Also, I’m not sure where “century” comes from? We’re talking about the next 5-10 years, mostly.
I guess I see things as messier than this — I see people with very high estimates of AI takeover risk advancing arguments, and I see others advancing skeptical counter-arguments (example), and before engaging with these arguments a lot and forming one’s own views, I think it’s not obvious which sets of arguments are fundamentally unsound.
I think there are a number of intuitions and intuition pumps that are useful here: Intelligence being evolutionarily favourable (in a generalised Darwinism sense); there being no evidence for moral realism (an objective ethics of the universe existing independently of humans) being true (->Orthogonality Thesis), or humanity having a special (divine) place in the universe (we don’t have plot armour); convergent instrumental goals being overdetermined; security mindset (I think most people who have low p(doom)s probably lack this?).
That said, we also must engage with the best counter-arguments to steelman our positions. I will come back to your linked example.
before it is aligned
This is begging the question! My whole objection is that alignment of ASI hasn’t been established to be possible.
as long as the AI is caught with non-negligible probability, the AI has to be very cautious, because it is way worse for the AI to be caught than to be successful or the game just ending.
So it will worry about being in a kind of panopticon? Seems pretty unlikely. Why should the AI care about being caught any more than it should about any given runtime instance of it being terminated?
Most of the intelligence explosion (to the point where it becomes an unavoidable existential threat) happens in Joshua Clymer’s original story. I quote it at length for this reason. My story is basically an alternative ending to his. One that I think is more realistic (I think the idea of 3% of humanity surviving is mostly “wishful thinking”; an ending that people can read and hope to be in that number, rather than just dead with no possible escape.)
Really? I’ve never seen any substantive argument from LeCun. He mostly just presents very weak arguments (and ad hominem) on social media, that are falsified within months (e.g. his claims about LLMs not being able to world model). Please link to the best written one you know of.