Global moratorium on AGI, now (Twitter). Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn ⏸️
Karma: 5,801
Saying they are conditional does not mean they are. For example, why is P(We invent a way for AGIs to learn faster than humans|We invent algorithms for transformative AGI) only 40%? Or P(AGI inference costs drop below $25/hr (per human equivalent)[1]|We invent algorithms for transformative AGI) only 16%!? These would be much more reasonable as unconditional probabilities. At the very least, “algorithms for transformative AGI” would be used to massively increase software and hardware R&D, even if expensive at first, such that inference costs would quickly drop.
- ^
As an aside, surely this milestone has basically now already been reached? At least for the 90% percentile human in most intellectual tasks.
- ^
If they were already aware, they certainly didn’t do anything to address it, given their conclusion is basically a result of falling for it.
It’s more than just intuitions, it’s grounded in current research and recent progress in (proto) AGI. To validate the opposing intuitions (long timelines) requires more in the way of leaps of faith (to say that things will suddenly stop working as they have been). Longer timelines intuitions have also been proven wrong consistently over the last few years (e.g. AI constantly doing things people predicted were “decades away” just a few years, or even months, before).
I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.
This is an example of the multiple stages fallacy (as pointed out here), where you can get arbitrarily low probabilities for anything by dividing it up enough and assuming things are uncorrelated.
For what it’s worth, I think you are woefully miscalibrated about what the right course of action is if you care about the people you love. Preventing ASI from being built for at least a few years should be a far bigger priority (and Mechanize’s goal is ~the opposite of that). Would be interested to hear more re why you think violent AI takeover is unlikely.
As previously referred to, I can’t get bank loans (no stable income).
Where I say “some of which I borrow against now (with 100% interest over 5 years)”, I’m referring to the bet.
if you think the world is almost certainly doomed
I think it’s maybe 60% doomed.
it seems crazy not to just spend it and figure out the reputational details on the slim chance we survive.
Even if I thought it was 90%+ doomed, it’s this kind of attitude that has got us into this whole mess in the first place! People burning the commons for short term gain is directly leading to massive amounts of x-risk.
you couldn’t ask for someone better than Yann LeCun, no?
Really? I’ve never seen any substantive argument from LeCun. He mostly just presents very weak arguments (and ad hominem) on social media, that are falsified within months (e.g. his claims about LLMs not being able to world model). Please link to the best written one you know of.
Ilya’s company website says “Superintelligence is within reach.” I think it’s reasonable to interpret that as having a short timeline. If not an even stronger claim that he thinks he knows how to actually build it.
The post gives a specific example of this: the “software intelligence explosion” concept.
Right, and doesn’t address any of the meat in the methodology section.
I don’t think it’s nitpicky at all. A trend showing small, increasing numbers, just above 0, is very different (qualitatively) to a trend that is all flat 0s, as Ben West points out.
I am curious to see what will happen in 5 years when there is no AGI.
If this happens, we will at least know a lot more about how AGI works (or doesn’t). I’ll be happy to admit I’m wrong (I mean, I’ll be happy to still be around, for a start[1]).
- ^
I think the most likely reason we won’t have AGI in 5 years is that there will be a global moratorium on further development. This is what I’m pushing for.
- ^
I think Chollet has shifted the goal posts a bit from when he first developed ARC [ARC-AGI 1]. In his original paper from 2019, Chollet says:
“We argue that ARC [ARC-AGI 1] can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.”
And the original announcement (from June 2024) says:
A solution to ARC-AGI [1], at a minimum, opens up a completely new programming paradigm where programs can perfectly and reliably generalize from an arbitrary set of priors. We also believe a solution is on the critical path towards AGI”
(And ARC-AGI 1 has now basically been solved). You say:
I understand the theory that AI will have a super fast takeoff, so that even though it isn’t very capable now, it will match and surpass human capabilities within 5 years. But this kind of theory is consistent with pretty much any level of AI performance in the present.
But we are seeing a continued rapid improvement in A(G)I capabilities, not least along the trajectory to automating AGI development, as per the METR report Ben West mentions.
I was not being disingenuous and I find your use of the word “disingenuous” here to be unnecessarily hostile.
I was going off of the numbers in the recent blog post from March 24, 2025. The numbers I stated were accurate as of the blog post.
GPT-2 is not mentioned in the blog post. Nor is GPT-3. Or GPT3.5. Or GPT-4. Or even GPT-4o! You are writing 0.0% a lot for effect. In the actual blog post, there are only two 0.0% entries, for “gpt-4.5 (Pure LLM)”, and “o3-mini-high (Single CoT)”; and note the limitations in parenthesis, which you also neglect to include in your list (presumably for effect? Given their non-zero scores when not limited in such ways.)
In another comment you accuse me of being “unnecessarily hostile”. Yet to me, your whole paragraph in the OP here is unnecessarily hostile (somewhat triggering, even):
The community of people most focused on keeping up the drumbeat of near-term AGI predictions seems insular, intolerant of disagreement or intellectual or social non-conformity (relative to the group’s norms), and closed-off to even reasonable, relatively gentle criticism (whether or not they pay lip service to listening to criticism or perform being open-minded). It doesn’t feel like a scientific community. It feels more like a niche subculture. It seems like a group of people just saying increasingly small numbers to each other (10 years, 5 years, 3 years, 2 years), hyping each other up (either with excitement or anxiety), and reinforcing each other’s ideas all the time. It doesn’t seem like an intellectually healthy community.
Calling that sentence uncharitable was an understatement.
For instance, you don’t acknowledge that the top 3 most cited AI scientists of all time, all have relatively short timelines now.
As for the post you link, it starts with “I have not read the whole thing in detail”. I think far too many people critiquing it have not actually read it properly. If they did read it all in detail, they might find that their objections have been answered in one of the many footnotes, appendices, and accompanying research reports. It concludes with “It doesn’t really engage with my main objections, nor is it trying to do so”, but nowhere are the main objections actually stated! It’s all just meta commentary.
This is somewhat disingenuous. o3-mini (high) is actually on 1.5%, and none of the other models are reasoning (CoT / RL / long inference time) models (oh, and GPT 4.5 is actually on 0.8%). The actual leaderboard looks like this:
Yes the scores are still very low, but it could just be a case of the models not yet “grokking” such puzzles. In a generation or two they might just grok them and then jump up to very high scores (many benchmarks have gone like this in the past few years).
It seems like a group of people just saying increasingly small numbers to each other (10 years, 5 years, 3 years, 2 years), hyping each other up
This is very uncharitable. Especially in light of the recent AI 2027 report, which goes into a huge amount of detail (see also all the research supplements).
No, just saying that without their massive injection of cash, Anthropic might not be where they are today. I think the counterfactual where there wasn’t any “EA” investment into Anthropic would be significantly slower growth of the company (and, arguably, one fewer frontier AI company today).
Re Anthropic and (unpopular) parallels to FTX, just thinking that it’s pretty remarkable that no one has brought up the fact that SBF, Caroline Ellison and FTX were major funders of Anthropic. Arguably Anthropic wouldn’t be where they are today without their help! It’s unfortunate the journalist didn’t press them on this.
Anthropic leadership probably does lack the integrity needed to do complicated power-seeking stuff that has the potential to corrupt.
Yes. It’s sad to see, but Anthropic is going the same way as OpenAI, despite being founded by a group that split from OpenAI over safety concerns. Power (and money) corrupts. How long until another group splits from Anthropic and the process repeats? Or actually, one can hope that such a group splitting from Anthropic might actually have integrity and instead work on trying to stop the race.
I was thinking less in terms of fraud/illegality, and more in terms of immorality/negative externalities (i.e. they are recklessly endangering everyone’s lives).
It just looks a lot like motivated reasoning to me—kind of like they started with the conclusion and worked backward. Those examples are pretty unreasonable as conditional probabilities. Do they explain why “algorithms for transformative AGI” are very unlikely to meaningfully speed up software and hardware R&D?