Pronouns: she/her or they/them.
I got interested in EA back before it was called EA, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now I’m trying to figure out where EA can fit into my life these days and what it means to me.
I used to feel so strongly about effective altruism. But my heart isn’t in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven’t been able to sustain a vegan diet for more than a short time. And so on.
But there isn’t a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI’s takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk has dominated the EA Forum for a while. It feels like AGI is what the movement is mostly about now, so now I just disagree with most of what effective altruism is about.
-The extent to which LessWrong culture has taken over or “colonized” effective altruism culture is such a bummer. I know there’s been at least a bit of overlap for a long time, but ten years ago it felt like effective altruism had its own, unique culture and nowadays it feels like the LessWrong culture has almost completely taken over. I have never felt good about LessWrong or “rationalism” and the more knowledge and experience of it I’ve gained, the more I’ve accumulated a sense of repugnance, horror, and anger toward that culture and ideology. I hate to see that become what effective altruism is like.
-The stories about sexual harassment are so disgusting. They’re really, really bad and crazy. And it’s so annoying how many comments you see on EA Forum posts about sexual harassment that make exhausting, unempathetic, arrogant, and frankly ridiculous statements, if not borderline incomprehensible in some cases. You see these stories of sexual harassment in the posts and you see evidence of the culture that enables sexual harassment in the comments. Very, very, very bad. Not my idea of a community I can wholeheartedly feel I belong to.
-Kind of a similar story with sexism, racism, and transphobia. The level of underreaction I’ve seen to instances of racism has been crazymaking. It’s similar to the comments under the posts about sexual harassment. You see people justifying or downplaying clearly immoral behaviour. It’s sickening.
-A lot of the response to the Nonlinear controversy was disheartening. It was disheartening to see how many people were eager to enable, justify, excuse, downplay, etc. bad behaviour. Sometimes aggressively, arrogantly, and rudely. It was also disillusioning to see how many people were so… easily fooled.
-Nobody talks normal in this community. At least not on this forum, in blogs, and on podcasts. I hate the LessWrong lingo. To the extent the EA Forum has its own distinct lingo, I probably hate that too. The lingo is great if you want to look smart. It’s not so great if you want other people to understand what the hell you are talking about. In a few cases, it seems like it might even be deliberate obscurantism. But mostly it’s just people making poor choices around communication and writing style and word choice, maybe for some good reasons, maybe for some bad reasons, but bad choices either way. I think it’s rare that writing with a more normal diction wouldn’t enhance people’s understanding of what you’re trying to say, even if you’re only trying to communicate with people who are steeped in the effective altruist niche. I don’t think the effective altruist sublanguage is serving good thinking or good communication.
-I see a lot of interesting conjecture elevated to the level of conventional wisdom. Someone in the EA or LessWrong or rationalist subculture writes a creative, original, evocative blog post or forum post and then it becomes a meme, and those memes end up taking on a lot of influence over the discourse. Some of these ideas are probably promising. Many of them probably contain at least a grain of truth or insight. But they become conventional wisdom without enough scrutiny. Just because an idea is “homegrown” it takes on the force of a scientific idea that’s been debated and tested in peer-reviewed journals for 20 years, or a widely held precept of academic philosophy. That seems just intellectually the wrong thing to do and also weirdly self-aggrandizing.
-An attitude I could call “EA exceptionalism”, where people assert that people involved in effective altruism are exceptionally smart, exceptionally wise, exceptionally good, exceptionally selfless, etc. Not just above the average or median (however you would measure that), but part of a rare elite and maybe even superior to everyone else in the world. I see no evidence this is true. (In these sorts of discussions, you also sometimes see the lame argument that effective altruism is definitionally the correct approach to life because effective altruism means doing the most good and if something isn’t doing the most good, then it isn’t EA. The obvious implication of this argument is that what’s called “EA” might not be true EA, and maybe true EA looks nothing like “EA”. So, this argument is not a defense of the self-identified “EA” movement or community or self-identified “EA” thought.)
-There is a dark undercurrent to some EA thought, along the lines of negative utilitarianism, anti-natalism, misanthropy, and pessimism. I think there is a risk of this promoting suicidal ideation because it basically is suicidal ideation.
-Too much of the discourse seems to revolve around how to control people’s behaviours or beliefs. It’s a bit too House of Cards. I recently read about the psychologist Kurt Lewin’s study on the most effective ways to convince women to use animal organs (e.g. kidneys, livers, hearts) in their cooking during meat shortages during World War II. He found that a less paternalistic approach that showed more respect for the women’s was more effective in getting them to incorporate animal organs into their cooking. The way I think about this is: you didn’t have to be manipulated to get to the point where you are in believing what you believe or caring this much about this issue. So, instead of thinking of how to best manipulate people, think about how you got to the point where you are and try to let people in on that in an honest, straightforward way. Not only is this probably more effective, it’s also more moral and shows more epistemic humility (you might be wrong about what you believe and that’s one reason not to try to manipulate people into believing it).
-A few more things but this list is already long enough.
Put all this together and the old stuff I cared about (charity effectiveness, giving what I can, expanding my moral circle) is lost in a mess of other stuff that is antithetical to what I value and what I believe. I’m not even sure the effective altruism movement should exist anymore. The world might be better off if it closed down shop. I don’t know. It could free up a lot of creativity and focus and time and resources to work on other things that might end up being better things to work on.
I still think there is value in the version of effective altruism I knew around 2015, when the primary focus was on global poverty and the secondary focus was on animal welfare, and AGI was on the margins. That version of effective altruism is so different from what exists today — which is mostly about AGI and has mostly been taken over by the rationalist subculture — that I have to consider those two different things. Maybe the old thing will find new life in some new form. I hope so.
I think you make an important point that I’m inclined to agree with.
Most of the discourse, theories, intuitions, and thought experiments about AI alignment was formed either before the popularization of deep learning (which started circa 2012) or before the people talking and writing about AI alignment started really caring about deep learning.
In or around 2017, I had an exchange with Eliezer Yudkowsky in an EA-related or AI-related Facebook group where he said he didn’t think deep learning would lead to AGI and thought symbolic AI would instead. Clearly, at some point since then, he changed his mind.
For example, in his 2023 TED Talk, he said he thinks deep learning is on the cusp of producing AGI. (That wasn’t the first time, but it was a notable instance and an instance where he was especially clear on what he thought.)
I haven’t been able to find anywhere where Eliezer talks about changing his mind or explains why he did. It would probably be helpful if he did.
All the pre-deep learning (or pre-caring about deep learning) ideas about alignment have been carried into the ChatGPT era and I’ve seen a little bit of discourse about this, but only a little. It seems strange that ideas about AI itself would change so much over the last 13 years and ideas about alignment would apparently change so little.
If there are good reasons why those older ideas about alignment should still apply to deep learning-based systems, I haven’t seen much discussion about that, either. You would think there would be more discussion.
My hunch is that AI alignment theory could probably benefit from starting with a fresh sheet of paper. I suspect there is promise in the approach of starting from scratch in 2025 without trying to build on or continue from older ideas and without trying to be deferential toward older work.
I suspect there would also be benefit in getting out of the EA/Alignment Forum/LessWrong/rationalist bubble.
Yesterday, I watched this talk by François Chollet, which provides support for a few of the assertions I made in this post.
I think it’s still very relevant! I don’t think this talk’s relevance has diminished. It’s just important to also have that more recent information about o3 in addition to what’s in this talk. (That’s why I linked the other talk at the bottom of this post.)
By the way, I think it’s just o3 and not o1 that achieves the breakthrough results on ARC-AGI-1. It looks like o1 only gets 32% on ARC-AGI-1, whereas the lower-compute version of o3 gets around 76% and the higher-compute version gets around 87%.
The lower-compute version of o3 only gets 4% on ARC-AGI-2 in partial testing (full testing has not yet been done) and the higher-compute version has not yet been tested.
Chollet speculates in this blog post about how o3 works (I don’t think OpenAI has said much about this) and how that fits in to his overall thinking about LLMs and AGI:
Why does o3 score so much higher than o1? And why did o1 score so much higher than GPT-4o in the first place? I think this series of results provides invaluable data points for the ongoing pursuit of AGI.
My mental model for LLMs is that they work as a repository of vector programs. When prompted, they will fetch the program that your prompt maps to and “execute” it on the input at hand. LLMs are a way to store and operationalize millions of useful mini-programs via passive exposure to human-generated content.
This “memorize, fetch, apply” paradigm can achieve arbitrary levels of skills at arbitrary tasks given appropriate training data, but it cannot adapt to novelty or pick up new skills on the fly (which is to say that there is no fluid intelligence at play here.) This has been exemplified by the low performance of LLMs on ARC-AGI, the only benchmark specifically designed to measure adaptability to novelty – GPT-3 scored 0, GPT-4 scored near 0, GPT-4o got to 5%. Scaling up these models to the limits of what’s possible wasn’t getting ARC-AGI numbers anywhere near what basic brute enumeration could achieve years ago (up to 50%).
To adapt to novelty, you need two things. First, you need knowledge – a set of reusable functions or programs to draw upon. LLMs have more than enough of that. Second, you need the ability to recombine these functions into a brand new program when facing a new task – a program that models the task at hand. Program synthesis. LLMs have long lacked this feature. The o series of models fixes that.
For now, we can only speculate about the exact specifics of how o3 works. But o3′s core mechanism appears to be natural language program search and execution within token space – at test time, the model searches over the space of possible Chains of Thought (CoTs) describing the steps required to solve the task, in a fashion perhaps not too dissimilar to AlphaZero-style Monte-Carlo tree search. In the case of o3, the search is presumably guided by some kind of evaluator model. To note, Demis Hassabis hinted back in a June 2023 interview that DeepMind had been researching this very idea – this line of work has been a long time coming.
So while single-generation LLMs struggle with novelty, o3 overcomes this by generating and executing its own programs, where the program itself (the CoT) becomes the artifact of knowledge recombination. Although this is not the only viable approach to test-time knowledge recombination (you could also do test-time training, or search in latent space), it represents the current state-of-the-art as per these new ARC-AGI numbers.
Effectively, o3 represents a form of deep learning-guided program search. The model does test-time search over a space of “programs” (in this case, natural language programs – the space of CoTs that describe the steps to solve the task at hand), guided by a deep learning prior (the base LLM). The reason why solving a single ARC-AGI task can end up taking up tens of millions of tokens and cost thousands of dollars is because this search process has to explore an enormous number of paths through program space – including backtracking.
There are however two significant differences between what’s happening here and what I meant when I previously described “deep learning-guided program search” as the best path to get to AGI. Crucially, the programs generated by o3 are natural language instructions (to be “executed” by a LLM) rather than executable symbolic programs. This means two things. First, that they cannot make contact with reality via execution and direct evaluation on the task – instead, they must be evaluated for fitness via another model, and the evaluation, lacking such grounding, might go wrong when operating out of distribution. Second, the system cannot autonomously acquire the ability to generate and evaluate these programs (the way a system like AlphaZero can learn to play a board game on its own.) Instead, it is reliant on expert-labeled, human-generated CoT data.
It’s not yet clear what the exact limitations of the new system are and how far it might scale. We’ll need further testing to find out. Regardless, the current performance represents a remarkable achievement, and a clear confirmation that intuition-guided test-time search over program space is a powerful paradigm to build AI systems that can adapt to arbitrary tasks.
I’m curious what you both think of my impression that the focus on near-term AGI has completely taken over EA and sucked most of the oxygen out of the room.
I was probably one of the first 1,000 people to express an interest in organized effective altruism, back before it was called “effective altruism”. I remember being in the Giving What We Can group on Facebook when it was just a few hundred members, when they were still working on making a website. The focus then was exclusively on global poverty.
Later, when I was involved in a student EA group from around 2015 to 2017, global poverty was still front and centre, animal welfare and vegetarianism/veganism/reducetarianism was secondary, and the conversation about AI was nipping at the margins.
Fast forward to 2025 and it seems like EA is now primarily a millennialist intellectual movement focused on AGI either causing the apocalypse or creating utopia within the next 3-10 years (with many people believing it will happen within 5 years), or possibly as long as 35 years if you’re way far out on the conservative end of the spectrum.
This change has nothing to do with FTX and probably wouldn’t be a reason for anyone at Anthropic to distance themselves from EA, since Anthropic is quite boldly promoting a millennialist discourse around very near-term AGI.
But it is a reason for me not to feel an affinity with the EA movement anymore. It has fundamentally changed. It’s gone from tuberculosis to transhumanism. And that’s just not what I signed up for.
The gentle irony is that I’ve been interested in AGI, transhumanism, the Singularity, etc. for as long as I’ve been interested in effective altruism, if not a little longer. In principle, I endorse some version of many of these ideas.
But when I see the kinds of things that, for example, Dario Amodei and others at Anthropic are saying about AGI within 2 years, I feel unnerved. It feels like I’m at the boundary of the kind of ideas that it makes sense to try to argue against or rationally engage with. Because it doesn’t really feel like a real intellectual debate. It feels closer to someone experiencing some psychologically altered state, like mania or psychosis, where attempting to rationally persuade someone feels inappropriate and maybe even unkind. What do you even do in that situation?
I recently wrote here about why these super short AGI timelines make no sense to me. I read an article today that puts this into perspective. Apple is planning to eventually release a version of Siri that merges the functionality of the old, well-known version of Siri and the new soon-to-be-released version that is based on an LLM. The article says Apple originally wanted to release the merged version of Siri sooner, but now this has been delayed to 2027. Are we going to have AGI before Apple finishes upgrading Siri? These ideas don’t live in the same reality.
To put a fine point on it, I would estimate the probability of AGI being created by January 1, 2030 to be significantly less than the odds of Jill Stein winning the U.S. presidential election in 2028 as the Green Party candidate (not as the leader of either the Democratic or Republican primary), which, to be clear, I think will be roughly as likely as her winning in 2024, 2020, or 2016 was. I couldn’t find any estimates of Stein’s odds of winning either the 2028 election or past elections from prediction markets or election forecast models. At one point, electionbettingodds.com gave her 0.1%, but I don’t know if they massively rounded up or if those odds were distorted by a few long-shot bets on Stein. Regardless, I think it’s safe to say the odds of AGI being developed by January 1, 2030 are significantly less than 0.1%.
If I am correct (and I regret to inform you that I am correct), then I have to imagine the credibility of EA will diminish significantly over the next 5 years. Because, unlike FTX scamming people, belief in very near-term AGI is something that many people in EA have consciously, knowingly, deliberately signed up for. Whereas many of the warning signs about FTX were initially only known to insiders, the evidence against very near-term AGI is out in the open, meaning that deciding to base the whole movement on it now is a mistake that is foreseeable and… I’m sorry to say… obvious.
I feel conflicted saying things like this because I can see how it might come across as mean and arrogant. But I don’t think it’s necessarily unkind to try to give someone a reality check under unusual, exceptional circumstances like these.
I think EA has become dangerously insular and — despite the propaganda to the contrary — does not listen to criticism. The idea that EA has abnormal or above-average openness to criticism (compared to what? the evangelical church?) seems only to serve the function of self-licensing. That is, people make token efforts at encouraging or engaging with criticism, and then, given this demonstration of their open-mindedness, become more confident in what they already believed, and feel licensed to ignore or shut down criticism in other instances.
It also bears considering what kind of criticism or differing perspectives actually get serious attention. Listening to someone who suggests that you slightly tweak your views is, from one perspective, listening to criticism, but, from another perspective, it’s two people who already agree talking to each other in an echo chamber and patting themselves on the back for being open-minded. (Is that too mean? I’m really trying not to be mean.)
On the topic of near-term AGI, I see hand-wavey dismissal of contrary views, whether they come from sources like Turing Prize winner and FAIR Chief AI Scientist Yann LeCun, surveys of AI experts, or superforecasters. Some people predict AGI will be created very soon and seemingly a much larger number think it will take much longer. Why believe the former and not the latter? I see people being selective in this way, but I don’t see them giving principled reasons for being selective.
Crucially, AGI forecasts are a topic where intuition plays a huge role, and where intuitions are contagious. A big part of the “evidence” for near-term AGI that people explicitly base their opinion on is what person X, Y, and Z said about when they think AGI will happen. Someone somewhere came up with the image of some people sitting in a circle just saying ever-smaller numbers to each other, back and forth. What exactly would prevent that from being the dynamic?
When it comes to listening to differing perspectives on AGI, what I have seen more often than engaging with open-mindedness and curiosity is a very unfortunate, machismo/hegemonic masculinity-style impulse to degrade or humiliate a person for disagreeing. This is the far opposite of “EA loves criticism”. This is trying to inflict pain on someone you see as an opponent. This is the least intellectually healthy way of engaging in discourse, besides, I guess, I don’t know, shooting someone with a gun if they disagree with you. You might as well just explicitly forbid and censor dissent.
I would like to believe that, in 5 years, the people in EA who have disagreed with me about near-term AGI will snap out of it and send me a fruit basket. But they could also do like Elon Musk, who, after predicting fully autonomous Teslas would be available in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, and 2024, and getting it wrong 9 years in a row, now predicts fully autonomous Teslas will be available in 2025.
In principle, you could predict AGI within 5 years and just have called it a few years too soon. If you can believe in very near-term AGI today, you will probably be able to believe in very near-term AGI when 2030 rolls around, since AI capabilities will only improve.
Or they could go the Ray Kurzweil route. In 2005, Kurzweil predicted that we would have “high-resolution, full-immersion, visual-auditory virtual reality” by 2010. In 2010, when he graded his own predictions, he called this prediction “essentially correct”. This was his explanation:
The computer game industry is rapidly moving in this direction. Technologies such as Microsoft’s Kinect allows players to control a videogame without requiring controllers by detecting the player’s body motions. Three-dimensional high-definition television is now available and will be used by a new generation of games that put the user in a full-immersion, high-definition, visual-auditory virtual reality environment.
Kurzweil’s gradings of his own predictions are largely like that. He finds a way to give himself a rating of “correct” or “essentially correct”. Even though he was fully incorrect. I wonder if Dario Amodei will do the same thing in 2030.
In 2030, there will be the option of doubling down on near-term AGI. Either the Elon Musk way — kick the can down the road — or the Ray Kurzweil way — revisionist history. And the best option will be some combination of both.
When people turn out to be wrong, it is not guaranteed to increase their humility or lead to soul searching. People can easily increase their defensiveness and their aggression toward people who disagree with them.
And, so, I don’t think merely being wrong will be enough on its own for EA to pull out of being a millennialist near-term AGI community. That can continue indefinitely even if AGI is over 100 years away. There is no guarantee that EA will self-correct in 5 years.
For these reasons, I don’t feel an affinity toward EA any more — it’s nothing like what it was 10 or 15 years ago — and I don’t feel much hope for it changing back, since I can imagine a scenario where it only gets worse 5 years from now.
When I was most involved in in-person EA organizing/outreach/activism around 2015 to 2017, it seemed to me at the time that the focus of the movement was something roughly like this:
Global poverty: 80%
Animal welfare: 15%
AI and existential risk: 5%
Now, organizations like 80,000 Hours and individuals like Will MacAskill are saying they’re pivoting to focusing exclusively on AGI. Judging from what I see online, it seems like the focus of the movement is currently something like:
AGI: 95%
Global poverty and animal welfare: 5%
It seems like the movement has really changed. It started as a movement around charity effectiveness in the cause area of global poverty, and now it’s a movement or community devoted to talking about AGI. This is a big change and it has probably alienated a lot of people who formerly felt an affinity for the EA movement, as well as put off people who would have been interested in what the EA movement used to be but who aren’t on board with the current movement’s beliefs about AGI.
I recently wrote about why I don’t believe in the predictions of AGI within 5 years here. For me, at this point, the EA movement has almost completely killed off its credibility. I don’t think there is any way to undo what has been done. The “effective altruism” label is now owned by people who think AGI is coming soon and as the years tick on and it becomes increasingly clear AGI is not coming soon, the term “effective altruism” will be seen by more mainstream parts of the world as even more fringe, weird, and dubious than it is today.
I’m not sure what people who don’t believe in near-term AGI and who want to focus on global poverty and/or animal welfare should do. Maybe there would be value in creating some kind of spin-off term? A term that creates a clear distinction between “effective altruism the movement about AGI” and “a movement that focuses on charity effectiveness in the cause area(s) of global poverty and/or animal welfare”. The benefit of coining and popularizing such a term would be to draw a clear distinction between what EA used to be ten years ago and what EA is now.
Using such a term wouldn’t necessarily require disavowing and distancing yourself or your organization from the EA movement or from self-identified EA organizations. Maybe some people would want to do that (I don’t know), but the primary purpose would just be to clearly differentiate your beliefs and your focus from the people who believe in a relatively imminent Singularity and who treat that as the most important thing in the world to focus on right now.
Society has long operated on a simple bargain: we contribute our labor and, in return, gain income, security, and a stake in the economy. Governments tax our wages to fund public services; corporations rely on human workers to create value; and in return, workers expect their efforts to be rewarded with opportunity and security. This centuries-old implicit bargain will soon come under strain.
Maybe this is a small detail to focus on, but I often see a problem when people try to tell a story along the lines of “society was stable and harmonious since time immemorial and then this new, disruptive, dangerous technology came along”. I see a problem with this story.
Slavery wasn’t abolished in the United States until 1866. Two centuries ago would be 1825. So, is anything resembling a fair bargain exchanging labour for “income, security, and a stake in the economy” really “centuries-old”?
North America and other parts of the world also have a history of indentured servitude.
Even well into the 1900s, workers were treated in a way that was exploitative and violent. For example, Ford’s management and security tried to violently suppress union activities, in at least one case killing some of the workers.
The brief overview of the history of labour under the heading “Revisiting Our Social Contract” also doesn’t mention slavery, indentured servitude, or other ugly, violent parts of this history.
AGI or transformative AI would, in theory, cause fundamental changes to the way society organizes itself around productive labour, capital investment, government revenue, the welfare state, and so on. Yes. This possibility does raise social, political, and ethical questions. Yes. I get that when you’re writing an article like this, you often just need to quickly put together a framing device to get the conversation off the ground.
But this framing device just seemed a little too whitewashed for my taste.
Thanks.
Sorry for replying to this ancient post now. (I was looking at my old EA Forum posts after not being active on the forum for about a year.)
Here’s why this answer feels unsatisfying to me. An incredibly mainstream view is to care about everyone alive today and everyone who will be born in the next 100 years. I have to imagine over 90% of people in the world would agree to that view or a view very close to that if you asked them.
That’s already a reason to care about existential risks and a reason people do care about what they perceive as existential risks or global catastrophic risks. It’s the reason most people who care about climate change care about climate change.
I don’t really know what the best way to express the most mainstream view(s) would be. I don’t think most people have tried to form a rigorous view on the ethics of far future people. (I have a hard enough time translating my own intuitions into a rigorous view, even with exposure to academic philosophy and to these sorts of ideas.) But maybe we could conjecture that most people mentally apply a “discount rate” to future lives, so that they care less and less about future lives as the centuries stretch into the future, and at some point it reaches zero.
Future lives in the distant future (i.e. people born significantly later than 100 years from now) only make an actionable different to existential risk when the estimated risk is so low that it changes the expected value math to account for 10^16 or 10^52 or whatever it is hypothetical future lives. That feels like an important insight to me, but its applicability feels limited.
So: people who don’t take a longtermist view of existential risk already have a good reason to care about existential risk.
Also: people who take a longtermist view of ethics don’t seem to have a good reason to think differently about any other subject than existential risk. At least, that’s the impression I get from trying to engage open-mindedly and charitably with this new idea of “longtermism”.
Ultimately, I’m still kind of annoyed (or at least perplexed) by “longtermism” being promoted as if it’s a new idea with broad applicability, when:
A longtermist view of existential risk has been promoted in discussions of existential risk for a very long time. Like, decades.
If longtermism is actionable for anything, it’s for existential risk and very little (if anything) else.
Most people are already bought in to caring about existential risk for relatively “neartermist” reasons.
When I heard the hype about longtermism, I was expecting there to be more meat on the bone.
What AI model did you use to generate these summaries?
Looking at the methodology section you linked to, this really just confirms the accuracy of nostalgebraist’s critique, for me. (nostalgebraist is the Tumblr blogger.) There are a lot of guesses and intuitions. Such as:
Overall we’d guess that this is the sort of limitation that would take years to overcome—but not decades; just look at the past decade of progress and consider how many similar barriers have been overcome. E.g. in the history of game-playing RL AIs, we went from AlphaGo to EfficientZero in about a decade.
Remember, we are assuming SC is reached in Mar 2027. We think that most possible barriers that would block SAR from being feasible in 2027 would also block SC from being feasible in 2027.
So in this case we guess that with humans doing the AI R&D, it would take about 2-15 years.
Okay? I’m not necessarily saying this is an unreasonable opinion. I don’t really know. But this is fundamentally a process of turning intuitions into numbers and turning numbers into a mathematical model. The mathematical model doesn’t make the intuitions any more (or less) correct.
Why not 2-15 months? Why not 20-150 years? Why not 4-30 years? It’s ultimately about what the authors intuitively find plausible. Other well-informed people could reasonably find very different numbers plausible.
And if you swap out more of the authors’ intuitions for other people’s intuitions, the end result might be AGI in 2047 or 2077 or 2177 instead of 2027.
Edit: While looking up something else, I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.
A trend showing small, increasing numbers, just above 0, is very different (qualitatively) to a trend that is all flat 0s
Then it’s a good thing I didn’t claim there was “a trend that is all flat 0s” in the comment you called “disingenuous”. I said:
It’s only with the o3-low and o1-pro models we see scores above 0% — but still below 5%. Getting above 0% on ARC-AGI-2 is an interesting result and getting much higher scores on the previous version of the benchmark, ARC-AGI, is an interesting result. There’s a nuanced discussion to be had about that topic.
This feels like such a small detail to focus on. It feels ridiculous.
I don’t think it’s a good idea to engage with criticism of an idea in the form of meme videos from Reddit designed to dunk on the critic. Is that intellectually healthy?
I don’t think the person who made that video or other people who want to dunk on Yann LeCun for that quote understand what he was trying to say. (Benjamin Todd recently made the same mistake here.) I think people are interpreting this quote hyper-literally and missing the broader point LeCun was trying to make.
Even today, in April 2025, models like GPT-4o and o3-mini don’t have a robust understanding of things like time, causality, and the physics of everyday objects. They will routinely tell you absurd things like that an event that happened in 2024 was caused by an event in 2025, while listing the dates of the events. Why don’t LLMs, still, in April 2025 consistently understand that causes precede effects and not vice versa?
If anything, this makes it seem like what LeCun said in January 2022 seem prescient. Despite a tremendous amount of scaling of training data and training compute, and, more recently, significant scaling of test-time compute, the same fundamental flaw LeCun called out over 3 years ago remains a flaw in the latest LLMs.
All that being said… I think even if LeCun had made the claim that I think people are mistakenly interpreting him as making and he had turned out to have been wrong about that, discrediting him based on him being wrong about that one thing would be ridiculously uncharitable.
That seems like a non-sequitur and it seems like a calculated insult and not a good faith effort to engage in the substance of my argument.
I disagree with what you said about ARC-AGI and ARC-AGI-2, but it doesn’t seem worth getting into.
I think OpenAI revenue is, on average, more aggressive than Moore’s law.
I tried to frame the question to avoid counting the revenue or profit of AI companies that sell AI as a product or service. I said:
the ability of AI systems to generate profit for their users by displacing human labour.
Generating profit for users is different from generating profit for vendors. Generating profit for users would mean, for example, that OpenAI’s customers are generating more profit for themselves by using OpenAI’s models than they were before using LLMs.
I’d guess that LM ability to automate intellectual work is more aggressive than Moore’s law, too, but it started from a very low baseline, so it’s hard to see.
I realized in some other comments on this post (here and here) that trying to compare these kinds of things to Moore’s law is a mistake. As you mentioned, if you start from a low enough baseline, all kinds of things are faster than Moore’s law, at least for a while. Also, if you measure all kinds of normal trends within a selective window of time (e.g. number of sandwiches eaten per day from Monday to Tuesday increased from 1 to 2, indicating an upward trajectory many orders of magnitude faster than Moore’s law), then you can get a false picture of astronomically fast growth.
Back to the topic of profit… In an interview from sometime in the past few years, Demis Hassabis said that LLMs are mainly being used for “entertainment”. I was so surprised by this because you wouldn’t expect a statement that sounds so dismissive from someone in his position.
And yet, when I thought about it, that does accurately characterize a lot of what people have used LLMs for, especially initially in 2022 and 2023.
So, to try to measure the usefulness of LLMs, we have to exclude entertainment use cases. To me, one simple, clean way to do that is to measure the profit that people generate by using LLMs. If a corporation, a small business, or a self-employed person pays to use (for example) OpenAI’s models, for example, can they increase their profits? And, if so, how much has that increase in profitability changed (if it all) over time, e.g., from 2023 to 2025?
(We would still have to close some loopholes. For example, if a company pays to use OpenAI’s API and then just re-packages OpenAI’s models for entertainment purposes, then that shouldn’t count, since that’s the same function I wanted to exclude from the beginning and the only thing that’s different is an intermediary has been added.)
I haven’t seen much hard data on changes in firm-level profitability or firm-level productivity among companies that adopt LLMs. One of the few sources of data I can find is this study about customer support agents: https://academic.oup.com/qje/article/140/2/889/7990658 The paper is open access.
Here’s an interesting quote:
In Figure III, Panels B–E we show that less skilled agents consistently see the largest gains across our other outcomes as well. For the highest-skilled workers, we find mixed results: a zero effect on AHT [Average Handle Time] (Panel B); a small but positive effect for CPH [Chats Per Hour] (Panel C); and, interestingly, small but statistically significant decreases in RRs [Resolution Rates] and customer satisfaction (Panels D and E).
These results are consistent with the idea that generative AI tools may function by exposing lower-skill workers to the best practices of higher-skill workers. Lower-skill workers benefit because AI assistance provides new solutions, whereas the best performers may see little benefit from being exposed to their own best practices. Indeed, the negative effects along measures of chat quality—RR and customer satisfaction—suggest that AI recommendations may distract top performers or lead them to choose the faster or less cognitively taxing option (following suggestions) rather than taking the time to come up with their own responses. Addressing this outcome is potentially important because the conversations of top agents are used for ongoing AI training.
My main takeaway from this study is that this seems really underwhelming. Maybe worse than underwhelming.
Unless I misinterpreted what Steven was trying to say, this supports my observation in the OP about insularity:
There were a number of people, all quite new to the fields of AI and AI safety / alignment, for whom it seems to have never crossed their mind until they talked to me that maybe foundation models won’t scale to AGI, and likewise who didn’t seem to realize that the field of AI is broader than just foundation models.
How could you possibly never encounter the view that “foundation models won’t scale to AGI”? How could an intellectually healthy community produce this outcome?
Let me repeat something I said in the OP:
The community of people most focused on keeping up the drumbeat of near-term AGI predictions seems insular, intolerant of disagreement or intellectual or social non-conformity (relative to the group’s norms), and closed-off to even reasonable, relatively gentle criticism (whether or not they pay lip service to listening to criticism or perform being open-minded). It doesn’t feel like a scientific community. It feels more like a niche subculture. It seems like a group of people just saying increasingly small numbers to each other (10 years, 5 years, 3 years, 2 years), hyping each other up (either with excitement or anxiety), and reinforcing each other’s ideas all the time. It doesn’t seem like an intellectually healthy community.
My impression is that a lot of people who believe in short AGI timelines (e.g. AGI by January 1, 2030) and who believe in some strong version of the scaling hypothesis (e.g. LLMs will scale to AGI with relatively minor fundamental changes but with greatly increased training compute, inference compute, and/or training data) are in an echo chamber where they just reinforce each other’s ideas all the time.
What might look like vigorous disagreement is, in many cases, when you zoom out, people with broadly similar views arguing around the margins (e.g. AGI in 3 years vs. 7 years; minimal non-scaling innovations on LLMs vs. modest non-scaling innovations on LLMs).
If people stop to briefly consider what a well-informed critic like Yann LeCun has to say about the topic, it’s usually to make fun of him and move on.
It will seem more obvious that you’re right if the people you choose to listen to are the people who broadly agree with you and if you meet well-informed disagreement from people like Yann Lecun or François Chollet with dismissal, ridicule, or hostility. This is a recipe for overconfidence. Taken to an extreme, this approach can lead people down a path where they end up deeply misguided.
Those two examples you just gave do a lot to clarify what you’re talking about!
A formative experience for me around examples was reading Immanuel Kant’s Critique of Judgment for an undergraduate philosophy class. Kant wrote this 450-page book that has received a lot of attention from scholars and students trying to understand what he was trying to say. This whole class was devoted just to this book. And yet, since Kant was apparently averse to giving examples — I’m not sure there’s a single example in the whole book — what he was trying to say might be forever lost.
Examples are fundamental!
So, what I originally wrote is:
Progress does not seem like a fast exponential trend, faster than Moore’s law and laying the groundwork for an intelligence explosion. Progress seems actually pretty slow and incremental, with a moderate improvement from GPT-3.5 to GPT-4, and another moderate improvement from GPT-4 to o3-mini. The decline in the costs of running the models or the increase in the compute used to be train models is probably happening faster than Moore’s law, but not the actual intelligence of the models.
To remove the confusing part about Moore’s law, I could re-word it like this:
Progress toward AGI does not seem very fast, not fast enough to lay the groundwork for an intelligence explosion within anything like 5 years. Progress seems actually pretty slow and incremental, with a moderate improvement from GPT-3.5 to GPT-4, and another moderate improvement from GPT-4 to o3-mini. The decline in the costs of running the models or the increase in the compute used to be train models is probably very large, but the actual intelligence of the models seems to be increasing only a bit with each new major version.
I think this conveys my meaning better than what I wrote originally, and it avoids getting into the Moore’s law topic.
The Moore’s law topic is a bit of an unnecessary rabbit hole. A lot of things increase faster than Moore’s law during a short window of time, but few increase at a CAGR of 41% (or whatever Moore’s law’s CAGR is) for decades. There’s all kinds of ways to mis-apply the analogy of Moore’s law.
People have made jokes about this kind of thing before, like The Economist sarcastically forecasting in 2006 based on then-recent trends that a 14-blade razor would be released by 2010.
I also think of David Deutsch’s book The Beginning of Infinity, in which he rails against the practice of uncritically extrapolating past trends forward, and his TED Talk where he does a bit of the same.
Good grief. Maybe we can retrospectively say that AGI was created in the 1990s. Maybe Babbage’s Analytical Engine was AGI.
This is an absurd and discrediting statement. Sorry, Tyler Cowen, you’ve said some things I thought were interesting or insightful on blogs or podcasts before, and Ezra Klein respects you a lot, which goes a long way for me because I respect Ezra a lot. But this is sheer madness.
As Andrew Connor points out in the quoted text in this post, there are simple tests of fairly rudimentary intelligence like ARC-AGI-2 that, as far as we know (the full test results are still forthcoming), o3 isn’t human-level on — and it’s not like ARC-AGI-2 is a test of AGI, nor will ARC-AGI-3 be. Getting human-level on ARC-AGI-2 or ARC-AGI-3 is necessary, but not sufficient, for AGI. It’s a very low bar to clear.
And that’s just for starters. I recently posted a great talk by François Chollet here that points out many weaknesses in LLMs, and I can’t imagine o3 overcomes all or even most of them.
There are all kinds of tests we could come up with for AGI, such as a deliberately challenging, adversarial Turing test with no time limits and with judges who are savvy about LLMs. Or the ability to fully and autonomously replace a highly skilled knowledge worker such as a journalist, an academic research assistant, or a paralegal with only the same level of input and supervision that a human in that role would require. Can current frontier AI systems like o3 do long-term, autonomous hierarchical planning at that level of complexity? No. Absolutely not. Not even close.
I think people should interpret Tyler Cowen making such a ridiculous statement as a signal that the AGI discourse is rotten and off the rails. This sucks.