Pronouns: she/her or they/them.
I got interested in EA back before it was called EA, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now I’m trying to figure out where EA can fit into my life these days and what it means to me.
Sorry for replying to this ancient post now. (I was looking at my old EA Forum posts after not being active on the forum for about a year.)
Here’s why this answer feels unsatisfying to me. An incredibly mainstream view is to care about everyone alive today and everyone who will be born in the next 100 years. I have to imagine over 90% of people in the world would agree to that view or a view very close to that if you asked them.
That’s already a reason to care about existential risks and a reason people do care about what they perceive as existential risks or global catastrophic risks. It’s the reason most people who care about climate change care about climate change.
I don’t really know what the best way to express the most mainstream view(s) would be. I don’t think most people have tried to form a rigorous view on the ethics of far future people. (I have a hard enough time translating my own intuitions into a rigorous view, even with exposure to academic philosophy and to this sorts of ideas.) But maybe we could conjecture that most people mentally apply a “discount rate” to future lives, so that they care less and less about future lives as the centuries stretch into the future, and at some point it reaches zero.
Future lives in the distant future (i.e. people born significantly later than 100 years from now) only make an actionable different to existential risk when the estimated risk is so low that it changes the expected value math to account for 10^16 or 10^52 or whatever it is hypothetical future lives. That feels like an important insight to me, but its applicability feels limited.
So: people who don’t take a longtermist view of existential risk already have a good reason to care about existential risk.
Also: people who take a longtermist view of ethics don’t seem to have a good reason to think differently about any other subject than existential risk. At least, that’s the impression I get from trying to engage open-mindedly and charitably with this new idea of “longtermism”.
Ultimately, I’m still kind of annoyed (or at least perplexed) by “longtermism” being promoted as if it’s a new idea with broad applicability, when:
A longtermist view of existential risk has been promoted in discussions of existential risk for a very long time. Like, decades.
If longtermism is actionable for anything, it’s for existential risk and very little (if anything) else.
Most people are already bought in to caring about existential risk for relatively “neartermist” reasons.
When I heard the hype about longtermism, I was expecting there to be more meat on the bone.
What AI model did you use to generate these summaries?
Looking at the methodology section you linked to, this really just confirms the accuracy of nostalgebraist’s critique, for me. (nostalgebraist is the Tumblr blogger.) There are a lot of guesses and intuitions. Such as:
Overall we’d guess that this is the sort of limitation that would take years to overcome—but not decades; just look at the past decade of progress and consider how many similar barriers have been overcome. E.g. in the history of game-playing RL AIs, we went from AlphaGo to EfficientZero in about a decade.
Remember, we are assuming SC is reached in Mar 2027. We think that most possible barriers that would block SAR from being feasible in 2027 would also block SC from being feasible in 2027.
So in this case we guess that with humans doing the AI R&D, it would take about 2-15 years.
Okay? I’m not necessarily saying this is an unreasonable opinion. I don’t really know. But this is fundamentally a process of turning intuitions into numbers and turning numbers into a mathematical model. The mathematical model doesn’t make the intuitions any more (or less) correct.
Why not 2-15 months? Why not 20-150 years? Why not 4-30 years? It’s ultimately about what the authors intuitively find plausible. Other well-informed people could reasonably find very different numbers plausible.
And if you swap out more of the authors’ intuitions for other people’s intuitions, the end result might be AGI in 2047 or 2077 or 2177 instead of 2027.
Edit: While looking up something else, I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.
A trend showing small, increasing numbers, just above 0, is very different (qualitatively) to a trend that is all flat 0s
Then it’s a good thing I didn’t claim there was “a trend that is all flat 0s” in the comment you called “disingenuous”. I said:
It’s only with the o3-low and o1-pro models we see scores above 0% — but still below 5%. Getting above 0% on ARC-AGI-2 is an interesting result and getting much higher scores on the previous version of the benchmark, ARC-AGI, is an interesting result. There’s a nuanced discussion to be had about that topic.
This feels like such a small detail to focus on. It feels ridiculous.
I don’t think it’s a good idea to engage with criticism of an idea in the form of meme videos from Reddit designed to dunk on the critic. Is that intellectually healthy?
I don’t think the person who made that video or other people who want to dunk on Yann LeCun for that quote understand what he was trying to say. (Benjamin Todd recently made the same mistake here.) I think people are interpreting this quote hyper-literally and missing the broader point LeCun was trying to make.
Even today, in April 2025, models like GPT-4o and o3-mini don’t have a robust understanding of things like time, causality, and the physics of everyday objects. They will routinely tell you absurd things like that an event that happened in 2024 was caused by an event in 2025, while listing the dates of the events. Why don’t LLMs, still, in April 2025 consistently understand that causes precede effects and not vice versa?
If anything, this makes it seem like what LeCun said in January 2022 seem prescient. Despite a tremendous amount of scaling of training data and training compute, and, more recently, significant scaling of test-time compute, the same fundamental flaw LeCun called out over 3 years ago remains a flaw in the latest LLMs.
All that being said… I think even if LeCun had made the claim that I think people are mistakenly interpreting him as making and he had turned out to have been wrong about that, discrediting him based on him being wrong about that one thing would be ridiculously uncharitable.
That seems like a non-sequitur and it seems like a calculated insult and not a good faith effort to engage in the substance of my argument.
I disagree with what you said about ARC-AGI and ARC-AGI-2, but it doesn’t seem worth getting into.
I think OpenAI revenue is, on average, more aggressive than Moore’s law.
I tried to frame the question to avoid counting the revenue or profit of AI companies that sell AI as a product or service. I said:
the ability of AI systems to generate profit for their users by displacing human labour.
Generating profit for users is different from generating profit for vendors. Generating profit for users would mean, for example, that OpenAI’s customers are generating more profit for themselves by using OpenAI’s models than they were before using LLMs.
I’d guess that LM ability to automate intellectual work is more aggressive than Moore’s law, too, but it started from a very low baseline, so it’s hard to see.
I realized in some other comments on this post (here and here) that trying to compare these kinds of things to Moore’s law is a mistake. As you mentioned, if you start from a low enough baseline, all kinds of things are faster than Moore’s law, at least for a while. Also, if you measure all kinds of normal trends within a selective window of time (e.g. number of sandwiches eaten per day from Monday to Tuesday increased from 1 to 2, indicating an upward trajectory many orders of magnitude faster than Moore’s law), then you can get a false picture of astronomically fast growth.
Back to the topic of profit… In an interview from sometime in the past few years, Demis Hassabis said that LLMs are mainly being used for “entertainment”. I was so surprised by this because you wouldn’t expect a statement that sounds so dismissive from someone in his position.
And yet, when I thought about it, that does accurately characterize a lot of what people have used LLMs for, especially initially in 2022 and 2023.
So, to try to measure the usefulness of LLMs, we have to exclude entertainment use cases. To me, one simple, clean way to do that is to measure the profit that people generate by using LLMs. If a corporation, a small business, or a self-employed person pays to use (for example) OpenAI’s models, for example, can they increase their profits? And, if so, how much has that increase in profitability changed (if it all) over time, e.g., from 2023 to 2025?
(We would still have to close some loopholes. For example, if a company pays to use OpenAI’s API and then just re-packages OpenAI’s models for entertainment purposes, then that shouldn’t count, since that’s the same function I wanted to exclude from the beginning and the only thing that’s different is an intermediary has been added.)
I haven’t seen much hard data on changes in firm-level profitability or firm-level productivity among companies that adopt LLMs. One of the few sources of data I can find is this study about customer support agents: https://academic.oup.com/qje/article/140/2/889/7990658 The paper is open access.
Here’s an interesting quote:
In Figure III, Panels B–E we show that less skilled agents consistently see the largest gains across our other outcomes as well. For the highest-skilled workers, we find mixed results: a zero effect on AHT [Average Handle Time] (Panel B); a small but positive effect for CPH [Chats Per Hour] (Panel C); and, interestingly, small but statistically significant decreases in RRs [Resolution Rates] and customer satisfaction (Panels D and E).
These results are consistent with the idea that generative AI tools may function by exposing lower-skill workers to the best practices of higher-skill workers. Lower-skill workers benefit because AI assistance provides new solutions, whereas the best performers may see little benefit from being exposed to their own best practices. Indeed, the negative effects along measures of chat quality—RR and customer satisfaction—suggest that AI recommendations may distract top performers or lead them to choose the faster or less cognitively taxing option (following suggestions) rather than taking the time to come up with their own responses. Addressing this outcome is potentially important because the conversations of top agents are used for ongoing AI training.
My main takeaway from this study is that this seems really underwhelming. Maybe worse than underwhelming.
Unless I misinterpreted what Steven was trying to say, this supports my observation in the OP about insularity:
There were a number of people, all quite new to the fields of AI and AI safety / alignment, for whom it seems to have never crossed their mind until they talked to me that maybe foundation models won’t scale to AGI, and likewise who didn’t seem to realize that the field of AI is broader than just foundation models.
How could you possibly never encounter the view that “foundation models won’t scale to AGI”? How could an intellectually healthy community produce this outcome?
Let me repeat something I said in the OP:
The community of people most focused on keeping up the drumbeat of near-term AGI predictions seems insular, intolerant of disagreement or intellectual or social non-conformity (relative to the group’s norms), and closed-off to even reasonable, relatively gentle criticism (whether or not they pay lip service to listening to criticism or perform being open-minded). It doesn’t feel like a scientific community. It feels more like a niche subculture. It seems like a group of people just saying increasingly small numbers to each other (10 years, 5 years, 3 years, 2 years), hyping each other up (either with excitement or anxiety), and reinforcing each other’s ideas all the time. It doesn’t seem like an intellectually healthy community.
My impression is that a lot of people who believe in short AGI timelines (e.g. AGI by January 1, 2030) and who believe in some strong version of the scaling hypothesis (e.g. LLMs will scale to AGI with relatively minor fundamental changes but with greatly increased training compute, inference compute, and/or training data) are in an echo chamber where they just reinforce each other’s ideas all the time.
What might look like vigorous disagreement is, in many cases, when you zoom out, people with broadly similar views arguing around the margins (e.g. AGI in 3 years vs. 7 years; minimal non-scaling innovations on LLMs vs. modest non-scaling innovations on LLMs).
If people stop to briefly consider what a well-informed critic like Yann LeCun has to say about the topic, it’s usually to make fun of him and move on.
It will seem more obvious that you’re right if the people you choose to listen to are the people who broadly agree with you and if you meet well-informed disagreement from people like Yann Lecun or François Chollet with dismissal, ridicule, or hostility. This is a recipe for overconfidence. Taken to an extreme, this approach can lead people down a path where they end up deeply misguided.
Those two examples you just gave do a lot to clarify what you’re talking about!
A formative experience for me around examples was reading Immanuel Kant’s Critique of Judgment for an undergraduate philosophy class. Kant wrote this 450-page book that has received a lot of attention from scholars and students trying to understand what he was trying to say. This whole class was devoted just to this book. And yet, since Kant was apparently averse to giving examples — I’m not sure there’s a single example in the whole book — what he was trying to say might be forever lost.
Examples are fundamental!
So, what I originally wrote is:
Progress does not seem like a fast exponential trend, faster than Moore’s law and laying the groundwork for an intelligence explosion. Progress seems actually pretty slow and incremental, with a moderate improvement from GPT-3.5 to GPT-4, and another moderate improvement from GPT-4 to o3-mini. The decline in the costs of running the models or the increase in the compute used to be train models is probably happening faster than Moore’s law, but not the actual intelligence of the models.
To remove the confusing part about Moore’s law, I could re-word it like this:
Progress toward AGI does not seem very fast, not fast enough to lay the groundwork for an intelligence explosion within anything like 5 years. Progress seems actually pretty slow and incremental, with a moderate improvement from GPT-3.5 to GPT-4, and another moderate improvement from GPT-4 to o3-mini. The decline in the costs of running the models or the increase in the compute used to be train models is probably very large, but the actual intelligence of the models seems to be increasing only a bit with each new major version.
I think this conveys my meaning better than what I wrote originally, and it avoids getting into the Moore’s law topic.
The Moore’s law topic is a bit of an unnecessary rabbit hole. A lot of things increase faster than Moore’s law during a short window of time, but few increase at a CAGR of 41% (or whatever Moore’s law’s CAGR is) for decades. There’s all kinds of ways to mis-apply the analogy of Moore’s law.
People have made jokes about this kind of thing before, like The Economist sarcastically forecasting in 2006 based on then-recent trends that a 14-blade razor would be released by 2010.
I also think of David Deutsch’s book The Beginning of Infinity, in which he rails against the practice of uncritically extrapolating past trends forward, and his TED Talk where he does a bit of the same.
Both this post and the paper would benefit from some specific examples.
By some accounts, growth from 0.0 to 4.0 is infinite growth, which is infinitely faster than Moore’s law!
More seriously, I didn’t really think through precisely whether artificial intelligence could be increasing faster than Moore’s law. I guess in theory it could. I forgot that Moore’s law speed actually isn’t that impressive on its own. It has to compound over decades to be impressive.
If I eat a sandwich today and eat two sandwiches tomorrow, the growth rate in my sandwich consumption is astronomically faster than Moore’s law. But what matters is if the growth rate continues and compounds long-term.
The bigger picture is how to measure general intelligence or “fluid intelligence” in a way that makes sense. The Elo rating of AlphaGo probably increased faster than Moore’s law from 2014 to 2017. But we don’t see the Elo rating of AlphaGo as a measure of AGI, or else AGI would have already been achieved in 2015.
I think essentially all of these benchmarks and metrics for LLM performance are like the Elo rating of AlphaGo in this respect. They are measuring a narrow skill.
It depends what you want ARC-AGI-2 to tell you. For one, it tells you that current frontier models lack the general intelligence or “fluid intelligence” to solve simple puzzles that pretty much any person can solve. Why is that? Isn’t that interesting?
Why should it matter whether new models have been released after the reveal of ARC-AGI-2? If models have to be specifically fine-tuned for these tasks, doesn’t that show they are lacking in the capability to generalize to novel problems? If they don’t have to be specifically fine-tuned, then the timing shouldn’t matter. A model with good generalization capability should be able to do well whether it happens to be released before or after the reveal of the ARC-AGI-2 benchmark.
Another “benchmark” I mused about is the ability of AI systems to generate profit for their users by displacing human labour. It seems like improvement on that “benchmark” has been much, much slower than Moore’s law, but, then again, I don’t know if anyone’s been able to accurately measure that.
The bigger picture is that LLMs have extremely meagre capabilities in many cognitive domains and I haven’t seen signs of anything but modest improvement over the last ~2.5 years. I also don’t see many people trying to quantify those things.
On one level, that makes sense because it takes time, money/labour, and expertise to create a good benchmark and there is no profit in it. You don’t seem to get much acclaim, either. Also, you might feel like you wasted your time if you made a benchmark that frontier AI models got ~0% on and, a year later, they still got ~0%…
On another level, measuring AGI progress carefully and thoughtfully seems important and it’s a bit surprising/disappointing that the status quo for benchmarks is so poor.
In his interview with Dwarkesh Patel in June 2024 to talk about the launch of the ARC Prize, Chollet emphasized how easy the ARC-AGI tasks were for humans, saying that even children could do them. This is not something he’s saying only now in retrospect that the ARC-AGI tasks have been mostly solved.
That first quote, from the 2019 paper, is consistent with Chollet’s January 2025 Bluesky post. That second quote is not from Chollet, but from Mike Knoop. I don’t know what the first sentence is supposed to mean, but the second sentence is also consistent with the Bluesky post.
In response to the graph… Just showing a graph go up does not amount to a “trajectory to automating AGI development”. The kinds of tasks AI systems can do today are very limited in their applicability to AGI research and development. That has only changed modestly between ChatGPT’s release in November 2022 and today.
In 2018, you could have shown a graph of go performance increasing from 2015 to 2017 and that also would not have been evidence of a trajectory toward automating AGI development. Nor would AlphaZero’s tripling of the games a single AI system can master from go to go, chess, and shogi. Measuring improved performance on tasks only provides evidence for AGI progress if the tasks you are measuring test for general intelligence.
It seems like you are really zeroing in on nitpicky details that make barely any difference to the substance of what I said in order to accuse me of being intentionally deceptive. This is not a cool behaviour.
I am curious to see what will happen in 5 years when there is no AGI. How will people react? Will they just kick their timelines 5 years down the road and repeat the cycle? Will some people attempt to resolve the discomfort by defining AGI as whatever exists in 5 years? Will some people be disillusioned and furious?
I hope that some people engage in soul searching about why they believed AGI was imminent when it wasn’t. And near the top of the list of reasons why will be (I believe) intolerance of disagreement about AGI and hostility to criticism of short AGI timelines.
Thank you for sharing your experience.
The good: it sounds like you talked to a lot of people who were eager to hear a differing opinion.
The bad: it sounds like you talked to a lot of people who had never even heard a differing opinion before and hadn’t even considered that a differing opinion could exist.
I have to say, the bad part supports my observation!
When I talk about paying lip service to the idea of being open-minded vs. actually being open-minded, ultimately how you make that distinction is going to be influenced by what opinions you hold. I don’t think there is a 100% impartial, objective way of making that distinction.
What I have in mind in this context when I talk about lip service vs. actual open-mindedness is stuff like how a lot of people who believe in the scaling hypothesis and short AGI timelines have ridiculed and dismissed Yann LeCun (for example here, but also so many other times before that) for saying that autoregressive LLMs will never attain AGI. If you want to listen to a well-informed, well-qualified critic, you couldn’t ask for someone better than Yann LeCun, no? So, why is the response dismissal and ridicule rather than engaging with the substance of his arguments, “steelmanning”, and all that?
Also, when you set the two poles of the argument as people who have 1-year AGI timelines at one pole and people who have 20-year AGI timelines at the opposite pole, you really constrain the diversity of perspectives you are hearing. If you have vigorous debates with people who already broadly agree you on the broad strokes, you are hearing criticism about the details but not about the broad strokes. That’s a problem with insularity.
I think you are misusing the concept of charity. Or maybe we just disagree on what it means to be charitable or uncharitable in this context because we strongly disagree on the subject matter.
You linked to the website for Ilya Sutskever’s company as a citation for the claim that Ilya Sutskever has a relatively short AGI timeline. The website doesn’t mention a timeline and I can’t find an instance of Ilya Sutskever mentioning a specific timeline.
Yoshua Bengio gave a timeline of 5 to 20 years in 2023, so that’s 3 to 18 years now. He says he’s 95% confident in this prediction. Okay.
Geoffrey Hinton also says 5 to 20 years, but only with 50% confidence. Hmm. Well, 95% vs. 50% is a big discrepancy, right? Also, he’s been saying “5 to 20 years” since 2023, which, if we just take that at face value, means he’s actually been pushing back his timeline by about 1-2 years over the past 1-2 years.
I think the person who wrote the Tumblr post is pretty clear on what their problem with the AI 2027 report is. To treat the report as an actual prediction about the future, it requires you to be on board with a lot of modelling assumptions. And if you’re not already on board with those modelling assumptions, the report doesn’t do much to try to convince you. The post gives a specific example of this: the “software intelligence explosion” concept.
Thanks.