Toby_Ord

Karma: 4,425

Toby_Ord 16 Feb 2026 19:40 UTC
2 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: The Scaling Series Discussion Thread: with Toby Ord
And I’ll add that RL training (and to a lesser degree inference scaling) is limited to a subset of capabilities (those with verifiable rewards and that the AI industry care enough about to run lots of training on). So progress on benchmarks has been less representative of how good they are at things that aren’t being benchmarked than it was in the non-reasoning-model era. So I think the problems of the new era are somewhat bigger than the effects that show up in benchmarks.

Toby_Ord 16 Feb 2026 19:37 UTC
4 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: The Scaling Series Discussion Thread: with Toby Ord
That’s a great question. I’d expect a bit of slowdown this year, though not necessarily much. e.g. I think there is a 10x or so possible for RL before RL-training-compute reaches the size of pre-training compute, and then we know they have enough to 10x again beyond that (since GPT-4.5 was already 10x more), so there are some gains still in the pipe there. And I wouldn’t be surprised if METR timelines keep going up in part due to increased inference spend (i.e. my points about inference scaling not being that good are to do with costs exploding, so if a cost-insensitive benchmark is going on, it might not register on it all that much). There is also room for more AI-research or engineering improvements to these things, and a lump of new compute coming in, making it a bit messy.
Overall, I’d say my predictions are more about appreciable slowing in 2027+ rather than 2026.

Toby_Ord 12 Feb 2026 8:09 UTC
4 points
0 ∶ 0
in reply to: Ryan Greenblatt’s comment on: Distinguish between inference scaling and “larger tasks use more compute”
Good point about the METR curves not being Pareto frontiers.

Toby_Ord 11 Feb 2026 22:20 UTC
14 points
0 ∶ 0
on: Distinguish between inference scaling and “larger tasks use more compute”
Interesting ideas! A few quick responses:
1. The data for the early ‘linear’ regime for these models actually appears to be even better than you suggest here. They have a roughly straight line (on a log-log plot), but at a slope that is better than 1. Eyeballing it, I think some are slope 5 or higher (i.e. increasing returns, with time horizon growing as the 5th power of compute). See my 3rd chart here. If anything, this would strengthen your case for talking about that regime separately from the poorly scaling high compute regime later on.
2. I’d also suspected that when you apply extra RL to a model (e.g. o3 compared to o1) that it would have a curve that dominated the earlier model. But that doesn’t seem to be the case. See the curves in the final chart here, where o1-preview is dominated, but the other OpenAI models curves all cross each other (being cheaper for the same horizon at some horizons and more expensive at others).
3. Even when they do dominate each other neatly like in your fake data, I noticed that the ‘sweet spots’ and the ‘saturation points’ can still be getting more expensive, both in terms of $ and in terms of $/hr. I’m not sure what to make of that though!
4. I think you’re on to something with the idea that there is a problematic kind of inference scaling and a fine kind, though I’m not sure if you’ve quite put your finger on how to distinguish them. I suppose we can definitely talk about the super-linear scaling regime and the sub-linear regime (which meet at what I call the sweet spot), but I’m not sure these are the two types you refer to in qualitative terms near the top.

Toby_Ord 10 Feb 2026 21:10 UTC
2 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: The Scaling Series Discussion Thread: with Toby Ord
Yeah, it isn’t just like a constant factor slow-down, but is fairly hard to describe in detail. Pre-training, RL, and inference all have their own dynamics, and we don’t know if there will be new good scaling ideas that breathe new life into them or create a new thing on which to scale. I’m not trying to say the speed at any future point is half what it would have been, but that you might have seen scaling as a big deal, and going forward it is a substantially smaller deal (maybe half as big a deal).

Toby_Ord 9 Feb 2026 8:51 UTC
2 points
0 ∶ 0
in reply to: Denkenberger🔸’s comment on: Inference Scaling Reshapes AI Governance
Thanks for catching that — a lot of symbols in the appendix were lost when converting the post for the forum, so I’ve edited it to add them back in.

Toby_Ord 6 Feb 2026 15:56 UTC
4 points
0 ∶ 0
in reply to: Margot Stakenborg’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
That’s an interesting way to connect these. I suppose one way to view your model is as making clear the point that you can’t cost-effectively use models on tasks that much longer than their 50% horizons — even if you are willing to try multiple times — and that trend of dramatic price improvements over time isn’t enough to help with this. Instead you need the continuation of the METR trend of exponentially growing horizons. Moreover, you give a nice intuitive explanation of why that is.
One thing to watch out for is Gus Hamilton’s recent study suggesting that there isn’t a constant hazard rate. I share my thoughts on it here, but my basic conclusion is that he is probably right. In particular, he has a functional form estimating how their success probability declines. You could add this to your model (it is basically 1 minus the CDF of a Weibull distribution with K=0.6). I think this survival function tail is a power law rather than an exponential, making the ‘just run it heaps of times’ thing slightly more tenable. It may mean that it is the cost of human verification that gets you, rather than it being untenable even on AI costs alone.
What links here?

Toby_Ord 4 Feb 2026 19:26 UTC
4 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
And one of the authors of the METR timelines paper has his own helpful critique/clarifications of their results.

Toby_Ord 4 Feb 2026 19:23 UTC
4 points
1 ∶ 0
in reply to: Charlie_Guthmann’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
Good points. I’m basically taking METR’s results at face value and showing that people are often implicitly treating costs (or cost per ‘hour’) as constant (especially when extrapolating them), but show that these costs appear to be growing substantially.
Re the quality / generalisability of the METR timelines, there is quite a powerful critique of it by Nathan Witkin. I wouldn’t go as far as he does, but he’s got some solid points.

Toby_Ord 4 Feb 2026 19:16 UTC
6 points
0 ∶ 0
in reply to: basil.halperin’s comment on: The Scaling Series Discussion Thread: with Toby Ord
Thanks Basil! That’s an interesting idea. The constant hazard rate model is just comparing two uses of the same model over different task lengths, so if using that to work out the 99% time horizon, it should cost 1/70th as much ($1.43). Over time, I think these 99% tasks should rise in cost in roughly the same way as the 50%-horizon ones (as they are both increasing in length in proportion). But estimating how that will change in practice is especially dicey as there is too little data.
Also, note that Gus Hamilton has written a great essay that takes the survival analysis angle I used in my constant hazard rates piece and extended it to pretty convincingly show that the hazard rates are actually decreasing. I explain it in more detail here. One upshot is that it gives a different function for estimating the 99% horizon lengths and he also shows that these are poorly constrained by the data and his model disagrees with METR’s by a factor of 20 on how long they are, with even more disagreement for shorter lengths.

Toby_Ord 4 Feb 2026 16:53 UTC
4 points
0 ∶ 0
on: Is there a Half-Life for the Success Rates of AI Agents?
Some great new analysis by Gus Hamilton shows that AI agents probably don’t obey a constant hazard rate / half-life after all. Instead their hazard rates systematically decline as the task goes on.
This means that their success rates on tasks beyond their 50%-horizon are better than the simple model suggests, but those for tasks shorter than the 50% horizon are worse.
I had suggested a constant hazard rate was a good starting assumption for how their success rate at tasks decays with longer durations. It is the simplest model and fits the data OK. But Gus used the standard second-simplest model from survival analysis (the Weibull distribution rather than the exponential distribution). It has a second parameter, K, which represents how the hazard rate changes with time (if at all). If K=1, there is a constant hazard rate, so the exponential distribution is a special case of the Weibull. But if K<1, then hazard decreases over time (like the Lindy effect), and if it is greater, hazard increases (like aging).
Gus found that the estimated values for K were below 1 for all the models, showing that *all* of them had decreasing hazard rates.
A distribution that generalises another is always going to get a better fit of the data than my exponential distribution, so fit alone wouldn’t be decisive. But the way that every single model has K statistically significantly below 1 convinces me he is right.
So what does this mean?
One thing is that it gives very different estimated success rates for tasks much shorter or longer than the 50% horizon (which METR focuses on because it is easier to reliably estimate). e.g. use the Weibull to estimate the 99% horizon (or 10% horizon).
Another thing is that the AI agents mainly have a K of about 0.6, while the human value of K is significantly lower, at about 0.4. This means even if they have the same 50% horizon, humans can do better on really long tasks (and worse on really short ones).
As this shows, for a fixed 50% horizon length, it isn’t clearly better or worse to have a lower value of K. Lower values are good for the low success rate really long tasks, but worse for the high reliability thresholds.
As a word of warning, I was quite sure before METR released its Opus 4.5 results that it was going to have a more human-like value of K, since it had a great 50% horizon, but only an average showing at 80%. But the estimates are that its value of K is similar. I’m not sure why that is, but it might be due to the fact that there isn’t much data to go on here and things are quite noisy for any individual model.
So, from Gus’s results, it still looks like there is some important gap between how human success rates drop off at longer tasks versus how AI agents do.
Gus also compares his two-parameter Weibull model of the data to METR’s two-parameter log-logistic model. He finds that they are similar, but with the log-logistic fitting slightly better. So it isn’t clear which of these to use if you have the choice. They differ quite a lot in the tails of the distribution (i.e. in estimated success rates for very short or very long tasks). e.g. the Weibull says the 99% horizon is 1/20th as long as the log-logistic predicts. That’s a big deal and the data doesn’t tell us which to favour! I’d slightly favour the Weibull, on the grounds that it is more plausible ex ante. But maybe the bigger lesson is that it is unknown which is right, and thus the 99% horizons (necessary for much useful work) are deeply uncertain.

Toby_Ord 4 Feb 2026 16:42 UTC
5 points
0 ∶ 0
in reply to: Mo Putera’s comment on: The Scaling Series Discussion Thread: with Toby Ord
I agree — a bunch of the arguments read like marketing that is greatly simplifying the real picture and not seeming very interested in digging deeper once a convenient story was found.

Toby_Ord 4 Feb 2026 16:38 UTC
21 points
3 ∶ 0
on: Thoughts on Toby Ords AI Scaling Series
That’s a good summary and pretty in-line with my own thoughts on the overall upshots. I’d say that absent new scaling approaches the strong tailwind to AI progress from compute increases will soon weaken substantially. But it wouldn’t completely disappear, there may be new scaling approaches, and there remains progress via AI research. Overall, I’d say it lengthens timelines somewhat, makes raw compute/finances less of an overwhelming advantage, and may require different approaches to compute governance.

Toby_Ord 3 Feb 2026 10:43 UTC
42 points
0 ∶ 0
on: The Scaling Series Discussion Thread: with Toby Ord
A few points to clarify my overarching view:
1. All kinds of compute scaling are quite inefficient on most standard metrics. There are steady gains, but they are coming from exponentially increasing inputs. These can’t continue forever, so all these kinds of gains from compute scaling are naturally time-limited. The exponential growth in inputs may also be masking fundamental deficiencies in the learning algorithms.
2. By ‘compute scaling’ I’m generally referring to the strategy of adding more GPUs to get more practically useful capabilities. I think this is running out of steam for pretraining and will soon start running out of steam for RL and inference scaling. This is possible even if the official ‘scaling laws’ of pretraining continue to hold (I’m generally neutral on whether they will).
3. It is possible that there will always be a new paradigm of compute scaling to take over when old ones run out of steam. If so, then like Moore’s Law, the longterm upward trend might be made out of a series of stacked S-curves. I’m mainly pointing out the limits of the current scaling paradigms, not denying the possibility of future ones.
4. I don’t think that companies are likely to completely stop scaling any of these forms of compute-scaling. The maths tends to recommend balancing the shares of compute that go to all of them in proportion to how much they improve capabilities per doubling of compute. e.g. perhaps a 3:1:2 ratio between pre-training, RL, and inference (though I expect the 3 to decline due to running out of high-quality text for pretraining).
5. But given that we don’t know if there will always be more paradigms delivered on time to save scaling, the limits on the current approaches should increase our credence that the practical process of scaling will provide less of a tail-wind to AI progress going forward. Overall, my view is something like: the strength of this tail-wind that has driven much of AI progress since 2020 will halve. (So it would still be important, but no longer the main determinant of who is in front, or of the pace of progress.)
6. As well as implications for the pace of progress, changes in what determines progress has implications for strategy and governance of AI. For example, AI researchers will be comparatively more important than in recent years, and if inference-scaling becomes the main form of scaling, that has big implications for compute governance and for the business model of AI companies.

Toby_Ord 3 Feb 2026 10:19 UTC
14 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: The Scaling Series Discussion Thread: with Toby Ord
That is quite a surprising graph — the annual tripling and the correlation between the compute and revenue are much more perfect than I think anyone would have expected. Indeed they are so perfect that I’m a bit skeptical of what is going on.
One thing to note is that it isn’t clear what the compute graph is of (e.g. is it inference + training compute, but not R&D?). Another thing to note is that it is year-end figures vs year total figures on the right, but given they are exponentials with the same doubling time and different units, that isn’t a big deal.
There are a number of things I disagree with in the post. The main one relevant to this graph is the implication that the graph on the left causes the graph on the right. That would be genuinely surprising. We’ve seen that the slope on the famous scaling law graphs is about −0.05 for compute — so you need to double compute 20 times to get log-loss to halve. Whereas this story of 3x compute leading to 3x the revenue implies that the exponent for a putative scaling law of compute vs R&D is extremely close to 1.0. And that it remains flukishly close to that magic number despite the transition from pretraining scaling to RL+inference scaling. I could believe a power law exponent of 1.0 for some things that are quite mathematical of physical, but not for the extremely messy relationship of compute to total revenue which depends on details of:
- the changing relationship between compute and intelligence,
- the utility of more intelligence to people, the market dynamics between competitors,
- running out of new customers and having to shift to more revenue per customer,
- the change from a big upfront cost (training compute) to mostly per-customer charges (inference compute)
More likely is something like reverse causation — that the growth in revenue is driving the amount of compute they can afford. Or it could be that the prices they need to charge increase with the amount of investment they received in order to buy compute — so they are charging the minimum they can in order to allow revenue growth to match investment growth.
Overall, I’d say that I believe these are real numbers, but I don’t believe the implied model. e.g. I don’t believe this trend will continue in the long run and I don’t think that if they had been able to 10x compute in one of those years that the revenue would have also jumped by 10x (unless that is because they effectively choose how much revenue to take by trading market growth for revenue in order to make this graph work to convince investors).

Toby_Ord 3 Feb 2026 9:50 UTC
2 points
0 ∶ 0
in reply to: Douglas Knight’s comment on: Inference Scaling and the Log-x Chart
Comparing AI scaling laws to Wright’s law is an interesting idea. That is still a power law rather than logarithmic returns, but usefully comparable to both the pretraining and inference scaling behaviours.

Toby_Ord 3 Feb 2026 9:08 UTC
3 points
0 ∶ 0
in reply to: Sharmake’s comment on: Evidence that Recent AI Gains are Mostly from Inference-Scaling
Thanks for the comments. The idea that pretraining has slowed/stalled is in the background in many posts in my series and it is unfortunate I didn’t write one where I addressed it head-on. I don’t disagree with Vladimir Nesov as much as you may think. Some of this is that the terms are slippery.
I think there are three things under discussion:
1. Scaling Laws. The empirical relationship between model size (or training data or compute) and the log-loss when predicting tokens from randomly chosen parts of the same data distribution that it hadn’t trained on.
2. Training Compute Increases. The annual increase in the amount of compute used to pretrain a frontier model.
3. Value of scaling. The practical returns from each 10x to the amount of pre-training compute.
In my view, the scaling laws (1) may well hold, and I wouldn’t be surprised if there isn’t even a kink in the curve. This is what Nesov is mainly discussing and I don’t disagree with him about it. My view is that the annual training compute scaleup for frontier models has declined (from more than 10x to less than 3x) and that the value per 10x has also declined (possibly due to having already trained on all books, leaving only marginal gains in Reddit comments etc).
As witness to this, consider that the Epoch estimates for total training of OpenAI’s leading model are just about 2x as high as original GPT-4, released almost 3 years ago. They did have a version (GPT-4.5) that was 10x as high, but were disappointed by it and quickly sunsetted it. xAI has a version with even more than that, but it is only the about the 5th best model and isn’t widely acclaimed, despite having scaled the most.
I see the fact that companies can’t economically serve large pretrained models as part of an explanation for the stalling of pretraining, rather than as a counterargument.
Note that I’m not saying pre-training scaling is dead (or anything about Scaling Laws). I’m saying something more like:
Pretraining scaling has returned substantially fewer practical benefits in the last 2.5 years than people in industry expected and is no longer the determinate of who has the best model. This is at least a temporary slump, and may be permanent. Compared to the days before GPT-4, I think the tailwind for AI progress provided by pre-training scaling has roughly halved.
Finally, I’ll say that I’m talking about scaling as the process of ‘just adding more GPUs’. If smart AI researchers improve pretraining efficiency, then that is great for pretraining even at the same scale, and is not the thing I’m critiquing. It is more like progress driven by ‘just adding more AI researchers’ and has different dynamics to the compute scaling that drove everything in the GPT 1 → GPT 4 era.

Toby_Ord 2 Feb 2026 16:18 UTC
6 points
0 ∶ 0
in reply to: Paolo Bova’s comment on: The Scaling Series Discussion Thread: with Toby Ord
Thanks Paolo,
I was only able to get weak evidence of a noisy trend from the limited METR data, so it is hard to draw many conclusions from that. Moreover, METR’s desire to measure the exponentially growing length of useful work tasks is potentially more exposed to an exponential rise in compute costs than more safety-related tasks. But overall, I’d think that the year-on-year growth in the amount of useful compute you can use on safety evaluations is probably growing faster than one can sustainably grow the number of staff.
I’m not sure how the dynamics will shake out for safety evals over a few years. e.g. a lot of recent capability gain has come from RL, which I think isn’t sustainable, and I also think the growth in ability via inference compute will limit both ability to serve the model and ability for people to afford it, so I suspect we’ll see some returning to eke what they can out of more pretraining. i.e. that the reasoning era saw them shift to finding capabilities in new areas with comparably low costs of scaling, but once they reach the optimal mix, we’ll see a mixture of all three going forward. So the future might look a bit less like 2025 and more like a mix of that and 2022-24.
Unfortunately I don’t have much insight on the question of ideal mixes of safety evals!

Toby_Ord 2 Feb 2026 16:06 UTC
5 points
1 ∶ 0
in reply to: titotal’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
Yes, that is a big limitation. Even more limiting is that it is only based on a subset of METR’s data on this. That’s enough to raise the question and illustrate what an answer might look like in data like this, but not to really answer it.
I’m not aware of others exploring this question, but I haven’t done much looking.

Are the Costs of AI Agents Also Rising Exponentially?

Toby_Ord2 Feb 2026 8:45 UTC

80 points

10 comments8 min readEA link

(www.tobyord.com)

Toby_Ord

Are the Costs of AI Agents Also Ris­ing Ex­po­nen­tially?

Are the Costs of AI Agents Also Rising Exponentially?