Benjamin_Todd

Karma: 8,924

Benjamin_Todd Apr 6, 2025, 8:09 PM
2 points
0 ∶ 0
in reply to: titotal’s comment on: The case for AGI by 2030
Thank you!
I am roughly in agreement with this post by an AI expert responding to the other (less good) short- timeline article going around.
This post just points out that the AI 2027 article is an attempt to flesh out a particular scenario, rather than an argument for short timelines, which the authors of AI 2027 would agree with.
I thought instead of critiquing the parts that I’m not an expert in, I might take a look at the part of this post that intersects with my field, when you mention material science discovery, and pour just a little bit of cold water on it.
So, an important thing to note is that this was not an LLM (neither was alphafold), but a specially designed deep learning model for generating candidate material structures.
Yes, I explicitly wanted to point out that AI can be useful to science beyond LLMs.
I covered a bit about them in my last article, this is a nice bit of evidence for their usefulness. The possibility space for new materials is ginormous and humans are not that good at generating new ones: the paper showed that this tool boosted productivity by making that process significantly easier. I don’t like how the paper described this as “idea generation”: it evokes the idea that the AI is making it’s own newtonian flashes of scientific insight, but actually it’s just mass generating candidate materials that an experienced professional can sift through.
I agree it’s not having flashes of insight, but I also think people under-estimate how useful brute force problem solving could be. I expect AI to become useful to science well before it has ‘novel insights’ in the way we imagine genius humans to have them.
I think your quoted statement is technically true, but it’s worth mentioning that the 80% faster figure was just for the people previously in the top decile of performance (ie the best researchers), for people who were not performing well there was not evidence of a real difference.
I do say it increased the productivity of ‘top’ researchers, and it’s also clarified through the link. (To my mind, it makes it more impressive, since it was adding value even to the best researchers.)
In practice the effect of the tool on progress was less than this: it was plausibly attributed to increasing the number of new patents at a firm by roughly 40%, and increasing the number of actual prototypes by 20%. You can also see that the productivity is not continuing to increase: they got their boost from the improved generation pipeline, and now the bottleneck is somewhere else.
20% more prototypes and 40% new patents sounds pretty meaningful.
I was just trying to illustrate that AI is already starting to contribute to scientific productivity in the near-term.
Productivity won’t continually increase until something more like a fully automated-scientist is created (which we clearly don’t already have).
To be clear, this is still great, and a clear deep learning success story, but it’s not really in line with colonizing the mars in 2035 or whatever the ASI people are saying now.
I’m not sure I follow. No-one is claiming that AI can already do these things – the claim is that if progress continues, then you could reach a point where AI accelerates AI research, and from there you get to something like ASI, and from there space colonisation. To argue against that you need to show the rate of progress is insufficient to get there.

Benjamin_Todd Apr 6, 2025, 3:42 PM
4 points
1 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: The case for AGI by 2030
I think Ege is one of the best proponents of longer timelines, and link to that episode in the article.
I don’t put much stock in the forecast of AI researchers the graph is from. I see the skill of forecasting as very different from the skill of being a published AI researcher. A lot of their forecasts also seem inconsistent. A bit more discussion here: https://80000hours.org/2025/03/when-do-experts-expect-agi-to-arrive/
Financially, I’m already heavily exposed to short AI timelines via my investments.

Benjamin_Todd Apr 6, 2025, 3:11 PM
4 points
0 ∶ 0
in reply to: emre kaplan🔸’s comment on: The case for AGI by 2030
The next few years, I expect AI revenues to continue to increase 2-4x per year, like they have recently, which gets you to those kinds of numbers in 2027.

There won’t be widespread automation, rather AI will make money from a few key areas with few barriers, especially programming.

You could then reach an inflection point where AI starts to help with AI research. AI inference gets mostly devoted to that task for a while. Major progress is made, perhaps reaching AGI, without further external deployment.

Revenues would then explode after that point, but OpenAI aren’t going to put that in their investor deck right now.
You could also see an acceleration in revenues when agents start to work. And in general I expect revenues to strongly lag capabilities. (Revenue also depends on the diff between the leading model and best free model.)
Overall I see the near term revenue figures as consistent with an AGI soon scenario. I agree 100bn in 2029 is harder to square, but I think that’s in part because OpenAI thinks investors won’t believe higher figures.

Benjamin_Todd Feb 22, 2025, 1:04 AM
6 points
0 ∶ 0
in reply to: Tim Hua’s comment on: Teaching AI to reason: this year’s most important story
Thanks this is helpful.

Benjamin_Todd Feb 18, 2025, 6:37 PM
11 points
3 ∶ 0
in reply to: ClimateDoc’s comment on: Teaching AI to reason: this year’s most important story
This is my understanding too – some crucial questions going forward:
1. How useful are AIs that are mainly good at these verifiable tasks?
2. How much does getting better at reasoning on these verifiable tasks generalise to other domains? (It seems like at least a bit e.g. o1 improved at law)
3. How well will reinforcement learning work when applied at scale to areas with weaker reward signals?

Benjamin_Todd Feb 18, 2025, 6:33 PM
6 points
0 ∶ 0
in reply to: Tim Hua’s comment on: Teaching AI to reason: this year’s most important story
Pretty sure o1 and Gemini have access to the internet.
The main way it’s potentially misleading is that it’s not a log plot (most benchmark results will look like exponentials on a linear scale) – however, I expect Deep Research would still seem above trend even if it was. I also think it’s helpful to new readers to see some of the charts on linear scales, since in some ways it’s more intuitive.

Benjamin_Todd Feb 14, 2025, 6:56 PM
4 points
0 ∶ 0
in reply to: tobycrisford 🔸’s comment on: Teaching AI to reason: this year’s most important story
Glad it’s useful! I categorise RL on chain of thought as a type of post-training, rather than test time compute. (Sometimes people lump them together as both ‘inference scaling’, but I think that’s confusing.) I agree RL opens up novel capabilities you can’t get just from next token prediction on the internet.

Benjamin_Todd Feb 13, 2025, 11:55 PM
4 points
0 ∶ 0
in reply to: tobycrisford 🔸’s comment on: Teaching AI to reason: this year’s most important story
For test time compute, you need to do logarithmic increases of compute to get linear increases in accuracy on the benchmark. It’s similar to the pretraining scaling law.
I agree test time compute isn’t especially explosive – it mainly serves to “pull forward” more advanced capabilities by 1-2 years.
More broadly, you can swap training for inference: https://epoch.ai/blog/trading-off-compute-in-training-and-inference
On brute force, I mainly took Toby’s thread to be saying we don’t clearly have enough information to know how effective test time compute is vs. brute force.

Benjamin_Todd Dec 10, 2024, 4:21 AM
7 points
0 ∶ 0
on: AI stocks could crash. And that could have implications for AI safety
Spitballing: EA entrepreneurs should be preparing for one of two worlds:
i) Short timelines with 10x funding of today
ii) Longer timelines with relatively scarce funding

Benjamin_Todd Sep 5, 2024, 11:46 AM
2 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: Are there diseconomies of scale in the reputation of communities?
The response to Michael is an interesting point, but it only concerns diminishing returns in individual capabilities of new members.
Diminishing returns are mainly driven by the quality of opportunities being used up, rather than the capabilities.
IIRC a 10x in resources to get a 3x in impact was a typical response in the old coordination forum survey responses.
In the past at 80k I’d often assume a 3x increase in inputs (e.g. advising calls) to get a 2x increase in outputs (impact-adjusted plan changes), and that seemed to be roughly consistent with the data (though the data don’t tell us that much). In some cases, returns seem to diminish a lot faster than that. And you often face diminishing returns at several levels (e.g. 3x as much marketing to get 2x as many applicants to advising).
I agree returns are more linear in areas where EA resources are a small fraction of the total, like global health, but that’s not the case in areas like AI safety, GCBRs, new causes like digital sentience, or promoting EA.
And even in global health, if GiveWell only had $100m to allocate, average cost-effectiveness would be a lot higher (maybe 3-10x higher?) than where the marginal dollar goes today. If GiveWell had to allocate $10bn, I’d guess returns would be at least several fold lower again on the marginal spending.

Benjamin_Todd Aug 24, 2024, 8:34 AM
4 points
0 ∶ 0
in reply to: Benjamin_Todd’s comment on: Are there diseconomies of scale in the reputation of communities?
Aside: A more compelling argument against growth in this area to me is something like “EA should focus on improving its brand and comms skills, and on making reforms & changing its messaging to significantly reduce the chance of something like FTX happening again, before trying to grow aggressively again”; rather than “the possibility of scandals means it should never grow”.
Another one is “it’s even more high priority to grow X others movements than EA” rather than “EA is net negative to grow”.

Benjamin_Todd Aug 24, 2024, 8:27 AM
2 points
0 ∶ 0
in reply to: Benjamin_Todd’s comment on: Are there diseconomies of scale in the reputation of communities?
Less importantly, I also feel less confident coordination benefits would mean impact per member goes up with the number of members.
I understand that the value of a social network like Facebook grows with the number of members. But many forms of coordination become much harder with the number of members.
As an analogy, it’s significantly easier for 2 people to decide where to go to dinner than for 3 people to decide. And 10 people in a group discussion can take ages to come to consensus.
Or, it’s much harder to get a new policy adopted in an organisation of 100 than an organisation of 10, because there are more stakeholders to consult and compromise with, and then more people to train in the new policy etc. And large organisations are generally way more bureaucratic than smaller ones.
I think these analogies might be closer than the analogy of Facebook.
You also get effects like in a movement of under 1000, it’s possible to have met in person most of the people, and know many of them well; while in a movement of 10,000, coordination has to be based on institutional mechanisms, which tend to involve a lot of overhead and not be as good.
Overall it seems to me that movement growth means more resources and skills, more shared knowledge, infrastructure and brand effects, but also many ways that it becomes harder to work together, and the movement becoming less nimble. I feel unsure which effect wins, but I put a fair bit of credence on the term decreasing rather than increasing.
If it were decreasing, and you also add in diminishing returns, then impact per member could be going down quite fast.

Benjamin_Todd Aug 24, 2024, 8:14 AM
2 points
0 ∶ 0
on: Are there diseconomies of scale in the reputation of communities?
Thanks for the analysis! I think it makes sense to me, but I’m wondering if you’ve missed an important parameter: diminishing returns to resources.
If there are 100 community members they can take the 100 most impactful opportunities (e.g. writing DGB, publicising that AI safety is even a thing), while if there are 1000 people, they will need to expand into opportunities 101-1000, which will probably be lower impact than the first 100 (e.g. becoming the 50th person working on AI safety).
I’d guess a 10x increase to labour or funding working on EA things (even setting aside coordination and reputation issues) only increases impact by ~3x.
It seems like that might make significant difference to the model—if I’ve understood, currently the impact of marginal members in the model is actually increasing due coordination benefits, whereas this could mean it’s decreasing.
I’d still guess marginal growth is net positive, but I feel less confident than the post suggests.

Benjamin_Todd Aug 21, 2024, 5:31 PM
3 points
0 ∶ 0
in reply to: JWS 🔸’s comment on: JWS’s Shortform
Thank you, I appreciate that.

Benjamin_Todd Aug 21, 2024, 12:17 PM
28 points
1 ∶ 0
in reply to: JWS 🔸’s comment on: JWS’s Shortform
Hey JWS,

These comments were off-hand and unconstructive, have been interpreted in ways I didn’t intend, and twitter isn’t the best venue for them, so I apologise for posting, and I’m going to delete them. My more considered takes are here. Hopefully I can write more in the future.

Benjamin_Todd Aug 13, 2024, 8:31 PM
2 points
0 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: On the Value of Advancing Progress
Thank you!

Benjamin_Todd Aug 11, 2024, 9:05 AM
10 points
1 ∶ 0
on: On the Value of Advancing Progress
I’d be interested to hear a short explanation of why this seems like a different result from Leopold’s paper, especially the idea that it could be better to accelerate through the time of perils.

Benjamin_Todd May 22, 2024, 8:11 PM
5 points
0 ∶ 0
in reply to: Geoffrey Miller’s comment on: AI stocks could crash. And that could have implications for AI safety
I should maybe have been more cautious—how messaging will pan out is really unpredictable.
However, the basic idea is that if you’re saying “X might be a big risk!” and then X turns out to be a damp squib, it looks like you cried wolf.
If there’s a big AI crash, I expect there will be a lot of people rubbing their hands saying “wow those doomers were so wrong about AI being a big deal! so silly to worry about that!”
That said, I agree if your messaging is just “let’s end AI!”, then there’s some circumstances under which you could look better after a crash e.g. especially if it looks like your efforts contributed to it, or it failed due to reasons you predicted / the things you were protesting about (e.g. accidents happening, causing it to get shut down).
However, if the AI crash is for unrelated reasons (e.g. the scaling laws stop working, it takes longer to commercialise than people hope), then I think the Pause AI people could also look silly – why did we bother slowing down the mundane utility we could get from LLMs if there’s no big risk?

Benjamin_Todd May 20, 2024, 8:43 AM
4 points
0 ∶ 0
in reply to: Ozzie Gooen’s comment on: Project idea: AI for epistemics
Thanks that’s helpful!

Benjamin_Todd May 17, 2024, 8:57 AM
2 points
0 ∶ 0
in reply to: Nathan Young’s comment on: Nuclear security seems like an interesting funding gap
https://www.macfound.org/media/article_pdfs/nuclear-challenges-synthesis-report_public-final-1.29.21.pdf[16].pdf