I should add, fairly belatedly, another point of comparison. Two Turing Award-winning AI researchers, Yann LeCun and Richard Sutton, each have novel fundamental ideas â not based on scaling LLMs or other comparably mainstream ideas â for how to get to AGI. (A few days ago, I wrote a comment about this here.)
In a 2024 interview, Yann LeCun said he thought it would take âat least a decade and probably much moreâ to get to AGI or human-level AI by executing his research roadmap. Trying to pinpoint when ideas first started is a fraught exercise. If we say the start time is the 2022 publication of LeCunâs position paper âA Path Towards Autonomous Machine Intelligenceâ, then by LeCunâs own estimate, the time from publication to human-level AI is at least 12 years and âprobably much moreâ.
In another 2024 interview, Richard Sutton said he thinks thereâs a 25% chance by 2030 weâll âunderstand intelligenceâ, although itâs unclear to me if he imagines by 2030 thereâs a 25% chance weâll actually build AGI (or be in a position to do so straightforwardly) or just have the fundamental theoretical knowledge required to do so. The equivalent paper co-authored by Sutton is âThe Alberta Plan for AI Researchâ, coincidentally also published in 2022. So, Suttonâs own estimate is a 25% chance of success in 8 years, although itâs not clear if success here means actually building AGI or a different goal.
But, crucially, I also definitely donât think we should just automatically accept these numbers. (I also discussed this in my previous comment about this here.) Researchers like Yann LeCun and Richard Sutton have a very high level of self-belief, which I think is psychologically healthy and rational. It is good to be this ambitious. But we shouldnât think of these as predictions or forecasts, but rather as goals.
LeCun himself has explicitly said you should be skeptical of anyone who says they have found the secret to AGI and will deliver it ten years, including him (as I discussed here). Which of course is very reasonable!
I think we should strive for, like, you know, 2030, and knowing that we probably wonât succeed, but you have to try.
This was in response to one of the interviewers noting that Sutton had said âdecadesâ, plural, when he said âthese are the decades when weâre going to figure out how the mind works.â
We have good reason to be skeptical if we look at predictions from people in AI that have now come false, such as Dario Amodeiâs incorrect prediction about AI writing 90% of code by mid-September 2025 or, for that matter, his prediction made 2 years and 2 months ago that we could have something that sounds a lot like AGI in 2 or 3 years, which still has 10 months left to go but looks extremely dubious. As I mentioned in the post, thereâs also Geoffrey Hintonâs prediction about radiology getting automated and various wrong predictions from various people in AI about widespread fully autonomous driving.
So, to summarize: what Yann LeCun and Richard Sutton are saying is already much more conservative than a trajectory from publishing a paper to building AGI within 7 years. They both tell us to be skeptical of even the timelines they lay out. And, independent of whether they tell us to be skeptical or not, based on the track record of similar predictions, we have good reason to be skeptical.
To me, this seems to be the much more apt point of comparison than the progress of LLMs from 2018 to 2025.
In a 2024 interview, Yann LeCun said he thought it would take âat least a decade and probably much moreâ to get to AGI or human-level AI by executing his research roadmap. Trying to pinpoint when ideas first started is a fraught exercise. If we say the start time is the 2022 publication of LeCunâs position paper âA Path Towards Autonomous Machine Intelligenceâ, then by LeCunâs own estimate, the time from publication to human-level AI is at least 12 years and âprobably much moreâ.
Hereâs why I donât think âstart time for LeCunâs research program is 2022â is true in any sense relevant to this conversation.
IIUC, the subtext of your OP and this whole conversation is that you think people shouldnât be urgently trying to prepare for AGI /â ASI right now.
In that context, one could say that the two relevant numbers are â(A) how far in advance should we be preparing for AGI /â ASI?â and â(B) how far away is AGI /â ASI?â. And you should start preparing when (A)=(B).
I think thatâs a terrible model, because we donât and wonât know either (A) or (B) until itâs too late, and thereâs plenty of work we can be doing right now, so itâs nuts not to be doing that work ASAP. Indeed, I think itâs nuts that we werenât doing more work on AGI x-risk in 2015, and 2005, and 1995 etc.
As bad as I think that âstart when (A)=(B)â model is, Iâm concerned that your implicit model is even worse. You seem to be acting as if (A) is less than 7 years, but you havenât justified that, and I donât think you can. I am concerned that what youâre actually thinking is more like: âAGI doesnât feel imminent, therefore (B)<(A)â.
Does the clock start in 2022 when LeCun published A Path Towards Autonomous Machine Intelligence (APTAMI)? That was 3 years ago. Yet you still, right now, donât seem to feel like we should be urgently preparing for AGI. If LeCun et al. keep making progress, maybe someday you will start feeling that sense of urgency about imminent LeCun-style AGI. And when that day comes, thatâs when the relevant clock starts. And I think that clock will leave very little time indeed until AGI and ASI. (My own guess would be 0â2 years, if your sense of urgency will be triggered by obvious signals of impressiveness like using language and solving problems beyond current LLMs. If you have some other trigger that youâre looking for, what is it?)
What would it look like to feel a sense of urgency starting from the moment that APTAMI was published? It would look like what I did, which was write the response: LeCunâs âA Path Towards Autonomous Machine Intelligenceâ has an unsolved technical alignment problem. Iâm pretty sure LeCun knows that this post exists, but he has not responded, and to this day he continues to insist that he has a great plan for AI alignment. Anyway, here I am, arguably the only person on Earth who is working on solving the technical alignment problem for APTAMI. LeCun and his collaborators have not shown the slightest interest in helping, and I donât expect that situation to change as they get ever closer to AGI /â ASI (on the off-chance that their research program is headed towards AGI /â ASI).
(If you think we should be urgently preparing for AGI /â ASI x-risk right now, despite AGI being extremely unlikely by 2032, then great, we would be in much more agreement than I assumed. If thatâs the situation, then I think your post does not convey that mood, and I think that almost all readers will interpret it as having that subtext unless you explicitly say otherwise.)
I find this comment fairly confusing, so Iâm going to try to hopefully clear up some of the confusion.
Hereâs why I donât think âstart time for LeCunâs research program is 2022â is true in any sense relevant to this conversation.
Was the intention of the comment I made about Yann LeCunâs and Richard Suttonâs research roadmaps unclear? It has nothing to do with the question of how far in advance we should start preparing for AGI. I was just giving a different point of comparison than your example of the progress in LLMs from 2018 to 2025. These were examples of how two successful AI researchers think about the amount of time between formulating the fundamental concepts â or at least the fundamental research directions â necessary to build AGI in a paper and actually building AGI. How much in advance of AGI youâd want to prepare is a separate question.
Similarly, I donât think your example of the amount of progress in LLMs from 2018 to 2025 was intended to make an argument about how long in advance of AGI to start preparing, was it? I thought you were simply trying to argue that the time between a novel AI paradigm being conceptualized and AGI being created could indeed be 7 years, contrary to what I asserted in the conclusion to my post.
Am I misunderstanding something? This response doesnât seem to be a response of what I was trying to say in the comment itâs responding to. Am I missing the point?
IIUC, the subtext of your OP and this whole conversation is that you think people shouldnât be urgently trying to prepare for AGI /â ASI right now.
The topic of how much in advance we should be preparing for AGI and what, specifically, we should be doing to prepare is, of course, related to the topic of when we think AGI is likely to happen, but someone could make the argument that itâs important to start preparing for AGI now even if itâs 50 or 100 years away. The correctness or incorrectness of that argument wouldnât depend on whether AGI by 2032 is extremely unlikely. My post is about whether AGI by 2032 is extremely unlikely and isnât intended to comment on the question of how far in advance of AGI we should prepare, or what we should do to prepare.
If we really should be preparing for AGI 50 or 100 years in advance, then whether I think we should start preparing for AGI now really doesnât depend on whether I think AGI is likely within 7 years.
I think itâs nuts that we werenât doing more work on AGI x-risk in 2015, and 2005, and 1995 etc.
If you think there is a strong argument for doing work on AGI safety or alignment 35+ years in advance of when AGI is expected to be created, then you can make that argument without arguing that AGI is likely to be created within 7 years, so that argument could be correct even if my thesis is correct that AGI by 2032 is extremely unlikely. Forgive me if Iâm repeating myself here.
You seem to be acting as if (A) is less than 7 years, but you havenât justified that, and I donât think you can.
I didnât say anything about that in the post. As I said just above, if itâs true, as you say, that we should start preparing for AGI long before we think itâs likely to arrive, then this wouldnât be a logical inference from what Iâve argued.
I am concerned that what youâre actually thinking is more like: âAGI doesnât feel imminent, therefore (B)<(A)â.
Is âfeelâ supposed to be pejorative here? Is âAGI doesnât feel imminentâ supposed to mean something other than âI donât think AGI is imminentâ? Are your opinions about AGI timelines also something you âfeelâ?
Does the clock start in 2022 when LeCun published A Path Towards Autonomous Machine Intelligence (APTAMI)?
Are you asking me whether I think Yann LeCun has published the roadmap that will, in fact, lead to AGI? I brought up LeCunâs roadmap as an example. I brought up Richard Suttonâs Alberta Plan as another example. As far as I can tell, these are mutually incompatible roadmaps to AGI. They could also both be wrong. But I just brought these up as examples. I wasnât saying one of them will actually lead to the invention of AGI.
...if your sense of urgency will be triggered by obvious signals of impressiveness like using language and solving problems beyond current LLMs. If you have some other trigger that youâre looking for, what is it?
In the post, I mentioned a few different broad areas where I think current AI systems do poorly and used this as evidence to argue that AGI is unlikely within 7 years. It would stand to reason, therefore, that I think if AI systems started significantly improving in these areas, it would be a reason for me to believe AGI is closer than I currently think it is.
I would at least be curious to know what you think about the reasons I gave in the post, even if I disagree.
(If you think we should be urgently preparing for AGI /â ASI x-risk right now, despite AGI being extremely unlikely by 2032, then great, we would be in much more agreement than I assumed. If thatâs the situation, then I think your post does not convey that mood, and I think that almost all readers will interpret it as having that subtext unless you explicitly say otherwise.)
How far in advance of AGI we should start preparing for it is logically independent from the thesis of this post â which is about the likelihood of near-term AGI â and I didnât say anything in this post about whether we should start preparing now or not. I would prefer to discuss that in the context of a post that does make an argument about how far in advance we should start preparing (and, if so, what kind of preparation would be useful or even possible).
That topic depends on a lot of things other than AGI timelines, e.g., hard takeoff vs. soft takeoff, the âMIRI worldviewâ on AI alignment vs. other views, and the scientific/âtechnological paradigm used to build AGI.
I made this post because I had certain ideas I wanted to talk about that I wanted to hear what people thought about. If you have thoughts about what I said in this post, I would be curious to hear them. If Iâm wrong about what I said in the post, why am I wrong? Tell me!
(âŚBut I still think your post has a connotation in context that âAGI by 2032 is extremely unlikely [therefore AGI x-risk work is not an urgent priority]â, and that it would be worth clarifying that you are just arguing the narrow point.)
If someone in 1900 had looked at everyone before the Wright brothers saying that theyâll get heavier-than-air flight soon, all those predictions would have been falsified, and they might have generalized to âWe have good reason to be skeptical if we look at predictions from people in [inventing airplanes] that have now come falseâ. But that generalization would have then failed when the Wright brothers came along.
Sutton does not seem to believe that âAGI by 2032 is extremely unlikelyâ so Iâm not sure how thatâs evidence on your side. Youâre saying that heâs over-optimistic, and maybe he is, but we donât know that. If you want examples of AI researchers and experts being over-pessimistic about the speed of progress, they are very easy to find (e.g.).
Youâve heard of Sutton & LeCun. There are a great many other research programs that you havenât heard of, toiling away and writing obscure arxiv papers. Some of those people have been writing obscure arxiv papers for many years already, even decades. We both agree that it takes >>7 years for an R&D pathway to get from its first obscure arxiv paper to ASI. What Iâm pushing back on the claim that it takes >>7 years to get from the final obscure arxiv paper (after which point the R&D pathway is impressive enough to stop being obscure) to ASI.
OK, hereâs the big picture of this discussion as I see it.
As someone who doesnât think LLMs will scale to AGI, I skipped over pretty much all of your OP as off-topic from my perspective, until I got to the sentences:
Eventually, there will be some AI paradigm beyond LLMs that is better at generality or generalization. However, we donât know what that paradigm is yet and thereâs no telling how long it will take to be discovered. Even if, by chance, it were discovered soon, itâs extremely unlikely it would make it all the way from conception to working AGI system within 7 years.
(Plus the subsequent couple paragraphs about brain computation, which I responded to briefly in my top-level comment.)
So that excerpt is what I was responding to originally, and thatâs what weâve been discussing pretty much this whole time. Right?
My claim is that, in the context of this paragraph, âextremely unlikelyâ (as in â<0.1%â) is way way too confident. Technological forecasting is hard, a lot can happen in seven years ⌠I think thereâs just no way to justify such an extraordinarily high confidence [conditioned on LLMs not scaling to AGI as always].
If you had said â<20%â instead of â<0.1%â, then OK sure, I would have been in close-enough agreement with you, that I wouldnât have bothered replying.
Does that help? Sorry if Iâm misunderstanding.
Hmm, reading what you wrote again, I think part of your mistake is saying ââŚconception to working AGI systemâ. Whoâs to say that this âAI paradigm beyond LLMsâ hasnât already been discovered ten years ago or more? There are a zillion speculative non-LLM AI paradigms that have been under development for years or decades. Nobody has heard of them because theyâre not doing impressive things yet. That doesnât mean that there hasnât already been a lot of development progress.
As someone who doesnât think LLMs will scale to AGI, I skipped over pretty much all of your OP as off-topic from my perspective
Okay, good to know.
I know that there are different views, but it seems like a lot of people in EA have started taking near-term AGI a lot more seriously since ChatGPT was released, and those people generally donât give the other views â the views on which LLMs arenât evidence of near-term AGI â much credence. Thatâs why the focus on LLMs.
The other views tend to be highly abstract, highly theoretical, highly philosophical and so to argue about them you basically have to write the whole Encyclopedia Britannica and you canât point to clear evidence from tests, studies, economic or financial indicators, and practical performance to make a case about AGI timelines within about 2,000 words.
Trying to argue those other views is not something I want to do, but I do want to argue about near-term AGI in a context where people are using LLMs as their key evidence for it.
Because my brain works that way, Iâm tempted to argue about the other views as well, but I never find those kinds of discussions satisfying. It feels like by the time you get a few exchanges deep into those discussions (either me personally or people in general), it gets into âHow many angels can dance on the head of a pin?â territory. For any number of sub-questions under that very abstract AGI discussion, maybe the answer is this, maybe itâs that, but nobody actually knows, thereâs no firm evidence, thereâs no theoretical consensus, and in fact the theorizing is very loose and pre-paradigmatic. (This is my impression after 15-20 years observing these discussions online and occasionally participating in them.) I think my response to these ideas should be, âYeah. Maybe. Who knows?â because I donât think thereâs much to say beyond that.
My claim is that, in the context of this paragraph, âextremely unlikelyâ (as in â<0.1%â) is way way too confident. Technological forecasting is hard, a lot can happen in seven years ⌠I think thereâs just no way to justify such an extraordinarily high confidence [conditioned on LLMs not scaling to AGI as always].
If you had said â<20%â instead of â<0.1%â, then OK sure, I would have been in close-enough agreement with you, that I wouldnât have bothered replying.
Does that help? Sorry if Iâm misunderstanding.
I didnât actually give a number for what I think are the chances of going from conception of a new AI paradigm to a working AGI system in 7 years. I did say itâs extremely unlikely, which is the same language I used for AGI within 7 years overall. I said I think the overall chances of AGI within 7 years is significantly less than 0.1%, so itâs understandable you might think by saying going from a new paradigm to working AGI in 7 years is extremely unlikely, I also mean I think that has a significantly less than 0.1% chance of success, or a similar number.
The relationship between the overall chance of AGI within 7 years and the chance of AGI conditional on the right paradigm being conceived isnât clear because that depends on a third variable, which is the chance that the right paradigm has already been conceived (or soon will be) â and also how long ago it was conceived (or how soon it will be). That seems basically unknowable to me.
I havenât really thought about what number I would assign to that specific outcome: a new AI paradigm going from conception to a working AGI system within 7 years. It seems very unlikely to me. In general, I donât like the practice of just thinking up numbers to assign to things like that. It could be an okay practice if people didnât take these numbers as literally and seriously as they do. Then it wouldnât really matter. But people take these numbers really seriously and I think thatâs unwise, and I donât like contributing to that practice if I can help it.
I do think where guessing a number is helpful is when it helps convey an intuition that might be otherwise hard to express. If you just had a first date and your ask asks how it went, and you say, âIt was a 7 out of 10,â that isnât a rigorous scale, your friend isnât expecting that all first dates of that quality will always be given a 7 rather than a 6 or an 8, but it helps convey a sense of somewhere between bad and fantastic. I think giving a number to a probability can be helpful like that. I think it can also be helpful to compare the probability of an event, like AGI being created within 7 years, to the probability of another event, which is why I came up with the Jill Stein example. (The problem is for this to work your interlocutor or your audience has to share your intuitive sense of how probable the other event is.)
I donât know how you would try to rigorously estimate how long it would take to go from the right idea about AGI to a working AGI system. This depends largely on what the right idea is, which is precisely what we donât know. So, there is irreducible uncertainty here.
We can come up with points of comparison. You used LLMs from 2018 to 2025 as as an example â 7 years. I brought up backpropagation in 1970 to AlexNet in 2011 as another potential point of comparison â 41 years. You could also choose the conception of connectionism in 1943 to AlphaGo beating Lee Seedol in 2016 as another comparison â 73 years. Or you can take Yann LeCunâs guess of at least 12 years and probably much more from his position paper to human-level AI, or Richard Suttonâs guess of a 25% chance of âunderstanding the mindâ (still not sure if that implies the ability to build AGI) in 8 years after publishing the Alberta Plan for AI Research. Who knows which of these points of comparison is most apt? Maybe none of them are particularly apt. Who knows.
The other thing I tried was considering the computation required for AGI in comparison to the human brain. This is almost as fraught as the above. We donât know for sure how much computation the human brain uses. We donât know at all whether AGI will require as much computation, or much less, or much more. Who knows?
In principle, almost anything could happen at almost any time, even if it goes against how we thought the world works, and this is uncomfortable, but itâs true. (I donât just mean with AI, I mean with everything. Volcanoes, aliens, physics, cosmology, the fabric of society â everything.)
What to do in the face of that uncertainty is a discussion that I think belongs in and under another post. For example, if we assume at least for the sake of argument that we have no idea which of several various ideas for building AGI will turn out to be correct, such as program synthesis, LeCunâs energy-based models, the Alberta Plan, Numentaâs Thousand Brains approach, whole brain emulation, and so on â and also if we have no idea whether all of these ideas will turn out to be the wrong ones â is there a strongly defensible course of action for preparing for AGI? Is there, indeed, a strongly defensible case for why AGI would be dangerous?
I worry that such a discussion would quickly get into the âHow many angels can dance on the head of a pin?â territory I said I donât like. But I would be impressed if someone could make a strong case for some course of action that makes sense even under a high level of irreducible uncertainty about which theoretical ideas will underpin the design of AGI and about when it will ultimately arrive.
I imagine this would be hard to do, however. For example, suppose Scenario A is that: the MIRI worldview on AI alignment is correct, there will be a hard takeoff, and AGI will be designed with a combination of deep learning and symbolic AI. Suppose Scenario B is: the MIRI worldview is false, whole brain emulation is the fastest possible path to AGI, and it will slowly scale up from a mouse brain emulation around 2065 to a human brain emulation around 2125,[1] and gradually from 2125 to 2165 it (or, more accurately, they) will become like AlphaGo for everything â a world champion at all tasks. Is there any strongly defensible course of action that makes sense if we donât know whether Scenario A or Scenario B is true (or many other possible scenarios I could describe) and if we canât even cogently assign probabilities to these scenarios? That sounds like a very tall order.
Itâs especially a tall order if part of the required defense is arguing why the proposed course of action wouldnât backfire and make things worse.
Whoâs to say that this âAI paradigm beyond LLMsâ hasnât already been discovered ten years ago or more? There are a zillion speculative non-LLM AI paradigms that have been under development for years or decades. Nobody has heard of them because theyâre not doing impressive things yet. That doesnât mean that there hasnât already been a lot of development progress.
I should add, fairly belatedly, another point of comparison. Two Turing Award-winning AI researchers, Yann LeCun and Richard Sutton, each have novel fundamental ideas â not based on scaling LLMs or other comparably mainstream ideas â for how to get to AGI. (A few days ago, I wrote a comment about this here.)
In a 2024 interview, Yann LeCun said he thought it would take âat least a decade and probably much moreâ to get to AGI or human-level AI by executing his research roadmap. Trying to pinpoint when ideas first started is a fraught exercise. If we say the start time is the 2022 publication of LeCunâs position paper âA Path Towards Autonomous Machine Intelligenceâ, then by LeCunâs own estimate, the time from publication to human-level AI is at least 12 years and âprobably much moreâ.
In another 2024 interview, Richard Sutton said he thinks thereâs a 25% chance by 2030 weâll âunderstand intelligenceâ, although itâs unclear to me if he imagines by 2030 thereâs a 25% chance weâll actually build AGI (or be in a position to do so straightforwardly) or just have the fundamental theoretical knowledge required to do so. The equivalent paper co-authored by Sutton is âThe Alberta Plan for AI Researchâ, coincidentally also published in 2022. So, Suttonâs own estimate is a 25% chance of success in 8 years, although itâs not clear if success here means actually building AGI or a different goal.
But, crucially, I also definitely donât think we should just automatically accept these numbers. (I also discussed this in my previous comment about this here.) Researchers like Yann LeCun and Richard Sutton have a very high level of self-belief, which I think is psychologically healthy and rational. It is good to be this ambitious. But we shouldnât think of these as predictions or forecasts, but rather as goals.
LeCun himself has explicitly said you should be skeptical of anyone who says they have found the secret to AGI and will deliver it ten years, including him (as I discussed here). Which of course is very reasonable!
In the 2024 interview, Sutton said:
This was in response to one of the interviewers noting that Sutton had said âdecadesâ, plural, when he said âthese are the decades when weâre going to figure out how the mind works.â
We have good reason to be skeptical if we look at predictions from people in AI that have now come false, such as Dario Amodeiâs incorrect prediction about AI writing 90% of code by mid-September 2025 or, for that matter, his prediction made 2 years and 2 months ago that we could have something that sounds a lot like AGI in 2 or 3 years, which still has 10 months left to go but looks extremely dubious. As I mentioned in the post, thereâs also Geoffrey Hintonâs prediction about radiology getting automated and various wrong predictions from various people in AI about widespread fully autonomous driving.
So, to summarize: what Yann LeCun and Richard Sutton are saying is already much more conservative than a trajectory from publishing a paper to building AGI within 7 years. They both tell us to be skeptical of even the timelines they lay out. And, independent of whether they tell us to be skeptical or not, based on the track record of similar predictions, we have good reason to be skeptical.
To me, this seems to be the much more apt point of comparison than the progress of LLMs from 2018 to 2025.
Hereâs why I donât think âstart time for LeCunâs research program is 2022â is true in any sense relevant to this conversation.
IIUC, the subtext of your OP and this whole conversation is that you think people shouldnât be urgently trying to prepare for AGI /â ASI right now.
In that context, one could say that the two relevant numbers are â(A) how far in advance should we be preparing for AGI /â ASI?â and â(B) how far away is AGI /â ASI?â. And you should start preparing when (A)=(B).
I think thatâs a terrible model, because we donât and wonât know either (A) or (B) until itâs too late, and thereâs plenty of work we can be doing right now, so itâs nuts not to be doing that work ASAP. Indeed, I think itâs nuts that we werenât doing more work on AGI x-risk in 2015, and 2005, and 1995 etc.
As bad as I think that âstart when (A)=(B)â model is, Iâm concerned that your implicit model is even worse. You seem to be acting as if (A) is less than 7 years, but you havenât justified that, and I donât think you can. I am concerned that what youâre actually thinking is more like: âAGI doesnât feel imminent, therefore (B)<(A)â.
Does the clock start in 2022 when LeCun published A Path Towards Autonomous Machine Intelligence (APTAMI)? That was 3 years ago. Yet you still, right now, donât seem to feel like we should be urgently preparing for AGI. If LeCun et al. keep making progress, maybe someday you will start feeling that sense of urgency about imminent LeCun-style AGI. And when that day comes, thatâs when the relevant clock starts. And I think that clock will leave very little time indeed until AGI and ASI. (My own guess would be 0â2 years, if your sense of urgency will be triggered by obvious signals of impressiveness like using language and solving problems beyond current LLMs. If you have some other trigger that youâre looking for, what is it?)
What would it look like to feel a sense of urgency starting from the moment that APTAMI was published? It would look like what I did, which was write the response: LeCunâs âA Path Towards Autonomous Machine Intelligenceâ has an unsolved technical alignment problem. Iâm pretty sure LeCun knows that this post exists, but he has not responded, and to this day he continues to insist that he has a great plan for AI alignment. Anyway, here I am, arguably the only person on Earth who is working on solving the technical alignment problem for APTAMI. LeCun and his collaborators have not shown the slightest interest in helping, and I donât expect that situation to change as they get ever closer to AGI /â ASI (on the off-chance that their research program is headed towards AGI /â ASI).
(If you think we should be urgently preparing for AGI /â ASI x-risk right now, despite AGI being extremely unlikely by 2032, then great, we would be in much more agreement than I assumed. If thatâs the situation, then I think your post does not convey that mood, and I think that almost all readers will interpret it as having that subtext unless you explicitly say otherwise.)
I find this comment fairly confusing, so Iâm going to try to hopefully clear up some of the confusion.
Was the intention of the comment I made about Yann LeCunâs and Richard Suttonâs research roadmaps unclear? It has nothing to do with the question of how far in advance we should start preparing for AGI. I was just giving a different point of comparison than your example of the progress in LLMs from 2018 to 2025. These were examples of how two successful AI researchers think about the amount of time between formulating the fundamental concepts â or at least the fundamental research directions â necessary to build AGI in a paper and actually building AGI. How much in advance of AGI youâd want to prepare is a separate question.
Similarly, I donât think your example of the amount of progress in LLMs from 2018 to 2025 was intended to make an argument about how long in advance of AGI to start preparing, was it? I thought you were simply trying to argue that the time between a novel AI paradigm being conceptualized and AGI being created could indeed be 7 years, contrary to what I asserted in the conclusion to my post.
Am I misunderstanding something? This response doesnât seem to be a response of what I was trying to say in the comment itâs responding to. Am I missing the point?
The topic of how much in advance we should be preparing for AGI and what, specifically, we should be doing to prepare is, of course, related to the topic of when we think AGI is likely to happen, but someone could make the argument that itâs important to start preparing for AGI now even if itâs 50 or 100 years away. The correctness or incorrectness of that argument wouldnât depend on whether AGI by 2032 is extremely unlikely. My post is about whether AGI by 2032 is extremely unlikely and isnât intended to comment on the question of how far in advance of AGI we should prepare, or what we should do to prepare.
If we really should be preparing for AGI 50 or 100 years in advance, then whether I think we should start preparing for AGI now really doesnât depend on whether I think AGI is likely within 7 years.
If you think there is a strong argument for doing work on AGI safety or alignment 35+ years in advance of when AGI is expected to be created, then you can make that argument without arguing that AGI is likely to be created within 7 years, so that argument could be correct even if my thesis is correct that AGI by 2032 is extremely unlikely. Forgive me if Iâm repeating myself here.
I didnât say anything about that in the post. As I said just above, if itâs true, as you say, that we should start preparing for AGI long before we think itâs likely to arrive, then this wouldnât be a logical inference from what Iâve argued.
Is âfeelâ supposed to be pejorative here? Is âAGI doesnât feel imminentâ supposed to mean something other than âI donât think AGI is imminentâ? Are your opinions about AGI timelines also something you âfeelâ?
Are you asking me whether I think Yann LeCun has published the roadmap that will, in fact, lead to AGI? I brought up LeCunâs roadmap as an example. I brought up Richard Suttonâs Alberta Plan as another example. As far as I can tell, these are mutually incompatible roadmaps to AGI. They could also both be wrong. But I just brought these up as examples. I wasnât saying one of them will actually lead to the invention of AGI.
In the post, I mentioned a few different broad areas where I think current AI systems do poorly and used this as evidence to argue that AGI is unlikely within 7 years. It would stand to reason, therefore, that I think if AI systems started significantly improving in these areas, it would be a reason for me to believe AGI is closer than I currently think it is.
I would at least be curious to know what you think about the reasons I gave in the post, even if I disagree.
How far in advance of AGI we should start preparing for it is logically independent from the thesis of this post â which is about the likelihood of near-term AGI â and I didnât say anything in this post about whether we should start preparing now or not. I would prefer to discuss that in the context of a post that does make an argument about how far in advance we should start preparing (and, if so, what kind of preparation would be useful or even possible).
That topic depends on a lot of things other than AGI timelines, e.g., hard takeoff vs. soft takeoff, the âMIRI worldviewâ on AI alignment vs. other views, and the scientific/âtechnological paradigm used to build AGI.
I made this post because I had certain ideas I wanted to talk about that I wanted to hear what people thought about. If you have thoughts about what I said in this post, I would be curious to hear them. If Iâm wrong about what I said in the post, why am I wrong? Tell me!
I am desperate to hear good counterarguments.
OK, sorry for getting off track.
(âŚBut I still think your post has a connotation in context that âAGI by 2032 is extremely unlikely [therefore AGI x-risk work is not an urgent priority]â, and that it would be worth clarifying that you are just arguing the narrow point.)
Wilbur Wright overestimated how long it would take him to fly by a factor of 25âhe said 50 years, it was actually 2. This is an example of how even researchers estimating their own very-near-term progress on their own R&D pathway can absolutely suck at timelines, including in the over-pessimistic direction.
If someone in 1900 had looked at everyone before the Wright brothers saying that theyâll get heavier-than-air flight soon, all those predictions would have been falsified, and they might have generalized to âWe have good reason to be skeptical if we look at predictions from people in [inventing airplanes] that have now come falseâ. But that generalization would have then failed when the Wright brothers came along.
Sutton does not seem to believe that âAGI by 2032 is extremely unlikelyâ so Iâm not sure how thatâs evidence on your side. Youâre saying that heâs over-optimistic, and maybe he is, but we donât know that. If you want examples of AI researchers and experts being over-pessimistic about the speed of progress, they are very easy to find (e.g.).
Youâve heard of Sutton & LeCun. There are a great many other research programs that you havenât heard of, toiling away and writing obscure arxiv papers. Some of those people have been writing obscure arxiv papers for many years already, even decades. We both agree that it takes >>7 years for an R&D pathway to get from its first obscure arxiv paper to ASI. What Iâm pushing back on the claim that it takes >>7 years to get from the final obscure arxiv paper (after which point the R&D pathway is impressive enough to stop being obscure) to ASI.
Do you have any response to the arguments made in the post? I would be curious to hear if you have any interesting counterarguments.
As for the rest, I think itâs been addressed at sufficient length already.
OK, hereâs the big picture of this discussion as I see it.
As someone who doesnât think LLMs will scale to AGI, I skipped over pretty much all of your OP as off-topic from my perspective, until I got to the sentences:
(Plus the subsequent couple paragraphs about brain computation, which I responded to briefly in my top-level comment.)
So that excerpt is what I was responding to originally, and thatâs what weâve been discussing pretty much this whole time. Right?
My claim is that, in the context of this paragraph, âextremely unlikelyâ (as in â<0.1%â) is way way too confident. Technological forecasting is hard, a lot can happen in seven years ⌠I think thereâs just no way to justify such an extraordinarily high confidence [conditioned on LLMs not scaling to AGI as always].
If you had said â<20%â instead of â<0.1%â, then OK sure, I would have been in close-enough agreement with you, that I wouldnât have bothered replying.
Does that help? Sorry if Iâm misunderstanding.
Hmm, reading what you wrote again, I think part of your mistake is saying ââŚconception to working AGI systemâ. Whoâs to say that this âAI paradigm beyond LLMsâ hasnât already been discovered ten years ago or more? There are a zillion speculative non-LLM AI paradigms that have been under development for years or decades. Nobody has heard of them because theyâre not doing impressive things yet. That doesnât mean that there hasnât already been a lot of development progress.
Okay, good to know.
I know that there are different views, but it seems like a lot of people in EA have started taking near-term AGI a lot more seriously since ChatGPT was released, and those people generally donât give the other views â the views on which LLMs arenât evidence of near-term AGI â much credence. Thatâs why the focus on LLMs.
The other views tend to be highly abstract, highly theoretical, highly philosophical and so to argue about them you basically have to write the whole Encyclopedia Britannica and you canât point to clear evidence from tests, studies, economic or financial indicators, and practical performance to make a case about AGI timelines within about 2,000 words.
Trying to argue those other views is not something I want to do, but I do want to argue about near-term AGI in a context where people are using LLMs as their key evidence for it.
Because my brain works that way, Iâm tempted to argue about the other views as well, but I never find those kinds of discussions satisfying. It feels like by the time you get a few exchanges deep into those discussions (either me personally or people in general), it gets into âHow many angels can dance on the head of a pin?â territory. For any number of sub-questions under that very abstract AGI discussion, maybe the answer is this, maybe itâs that, but nobody actually knows, thereâs no firm evidence, thereâs no theoretical consensus, and in fact the theorizing is very loose and pre-paradigmatic. (This is my impression after 15-20 years observing these discussions online and occasionally participating in them.) I think my response to these ideas should be, âYeah. Maybe. Who knows?â because I donât think thereâs much to say beyond that.
I didnât actually give a number for what I think are the chances of going from conception of a new AI paradigm to a working AGI system in 7 years. I did say itâs extremely unlikely, which is the same language I used for AGI within 7 years overall. I said I think the overall chances of AGI within 7 years is significantly less than 0.1%, so itâs understandable you might think by saying going from a new paradigm to working AGI in 7 years is extremely unlikely, I also mean I think that has a significantly less than 0.1% chance of success, or a similar number.
The relationship between the overall chance of AGI within 7 years and the chance of AGI conditional on the right paradigm being conceived isnât clear because that depends on a third variable, which is the chance that the right paradigm has already been conceived (or soon will be) â and also how long ago it was conceived (or how soon it will be). That seems basically unknowable to me.
I havenât really thought about what number I would assign to that specific outcome: a new AI paradigm going from conception to a working AGI system within 7 years. It seems very unlikely to me. In general, I donât like the practice of just thinking up numbers to assign to things like that. It could be an okay practice if people didnât take these numbers as literally and seriously as they do. Then it wouldnât really matter. But people take these numbers really seriously and I think thatâs unwise, and I donât like contributing to that practice if I can help it.
I do think where guessing a number is helpful is when it helps convey an intuition that might be otherwise hard to express. If you just had a first date and your ask asks how it went, and you say, âIt was a 7 out of 10,â that isnât a rigorous scale, your friend isnât expecting that all first dates of that quality will always be given a 7 rather than a 6 or an 8, but it helps convey a sense of somewhere between bad and fantastic. I think giving a number to a probability can be helpful like that. I think it can also be helpful to compare the probability of an event, like AGI being created within 7 years, to the probability of another event, which is why I came up with the Jill Stein example. (The problem is for this to work your interlocutor or your audience has to share your intuitive sense of how probable the other event is.)
I donât know how you would try to rigorously estimate how long it would take to go from the right idea about AGI to a working AGI system. This depends largely on what the right idea is, which is precisely what we donât know. So, there is irreducible uncertainty here.
We can come up with points of comparison. You used LLMs from 2018 to 2025 as as an example â 7 years. I brought up backpropagation in 1970 to AlexNet in 2011 as another potential point of comparison â 41 years. You could also choose the conception of connectionism in 1943 to AlphaGo beating Lee Seedol in 2016 as another comparison â 73 years. Or you can take Yann LeCunâs guess of at least 12 years and probably much more from his position paper to human-level AI, or Richard Suttonâs guess of a 25% chance of âunderstanding the mindâ (still not sure if that implies the ability to build AGI) in 8 years after publishing the Alberta Plan for AI Research. Who knows which of these points of comparison is most apt? Maybe none of them are particularly apt. Who knows.
The other thing I tried was considering the computation required for AGI in comparison to the human brain. This is almost as fraught as the above. We donât know for sure how much computation the human brain uses. We donât know at all whether AGI will require as much computation, or much less, or much more. Who knows?
In principle, almost anything could happen at almost any time, even if it goes against how we thought the world works, and this is uncomfortable, but itâs true. (I donât just mean with AI, I mean with everything. Volcanoes, aliens, physics, cosmology, the fabric of society â everything.)
What to do in the face of that uncertainty is a discussion that I think belongs in and under another post. For example, if we assume at least for the sake of argument that we have no idea which of several various ideas for building AGI will turn out to be correct, such as program synthesis, LeCunâs energy-based models, the Alberta Plan, Numentaâs Thousand Brains approach, whole brain emulation, and so on â and also if we have no idea whether all of these ideas will turn out to be the wrong ones â is there a strongly defensible course of action for preparing for AGI? Is there, indeed, a strongly defensible case for why AGI would be dangerous?
I worry that such a discussion would quickly get into the âHow many angels can dance on the head of a pin?â territory I said I donât like. But I would be impressed if someone could make a strong case for some course of action that makes sense even under a high level of irreducible uncertainty about which theoretical ideas will underpin the design of AGI and about when it will ultimately arrive.
I imagine this would be hard to do, however. For example, suppose Scenario A is that: the MIRI worldview on AI alignment is correct, there will be a hard takeoff, and AGI will be designed with a combination of deep learning and symbolic AI. Suppose Scenario B is: the MIRI worldview is false, whole brain emulation is the fastest possible path to AGI, and it will slowly scale up from a mouse brain emulation around 2065 to a human brain emulation around 2125,[1] and gradually from 2125 to 2165 it (or, more accurately, they) will become like AlphaGo for everything â a world champion at all tasks. Is there any strongly defensible course of action that makes sense if we donât know whether Scenario A or Scenario B is true (or many other possible scenarios I could describe) and if we canât even cogently assign probabilities to these scenarios? That sounds like a very tall order.
Itâs especially a tall order if part of the required defense is arguing why the proposed course of action wouldnât backfire and make things worse.
Yeah, maybe. Who knows?
2065 for a mouse brain and 2125 for a human brain are real guesses from an expert survey:
Zeleznikow-Johnston A, Kendziorra EF, McKenzie AT (2025) What are memories made of? A survey of neuroscientists on the structural basis of long-term memory. PLoS One 20(6): e0326920. https://ââdoi.org/ââ10.1371/ââjournal.pone.0326920