Toby_Ord

Karma: 4,921

The Dynamics of Intelligence Explosions

Toby_Ord28 Aug 2026 9:41 UTC

21 points

0 comments38 min readEA link

(arxiv.org)

Toby_Ord 23 Aug 2026 20:11 UTC
9 points
1 ∶ 0
in reply to: titotal’s comment on: Moral Imagination & Effective Altruism
Interesting take!
Re your final question, I definitely don’t think Michael was trying to say that it has to reach that standard to count as moral imagination — just that they are some of the best and clearest examples of moral imagination and how important it can be.

Moral Imagination & Effective Altruism

Toby_Ord23 Aug 2026 15:47 UTC

150 points

5 comments6 min readEA link

Toby_Ord 24 Mar 2026 11:40 UTC
5 points
1 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: Which questions can’t we punt?
Thanks Owen. I also agree with your maths.
Re conditioning, I agree that this is the technically correct thing to do and that it isn’t clear what difference it makes to the more simple analysis. In some cases it is fairly easy to condition (e.g. if working on a late-stage topic, one can do the project imagining that there isn’t lots of advanced AI advice in time when it arrives), while at the prioritisation stage it feels a bit harder to do. Oh, and I very much agree that it could be important to act to change whether such AI analysis happens (something that is, if anything, a bit easier to see on a view that treats whether this happens as uncertain).
Re maximal reasonable probabilities, I still genuinely feel like it is hard to get >90% credence that very large amounts of AI analysis on a key issue will happen prior to the issue coming to a head. I think one could get there for some things, but not that many. This is due to there being a variety of defeaters for such high amounts of AI analysis, such as external people like us not having access to the tools, needing the analysis earlier than expected (e.g. due to the need to socialise the ideas), jaggedness in the AI capabilities (e.g. where its engineering abilities take off substantially before more conceptual, philosophical abilities). I think you are onto something re what you are imagining as default vs what I am.

Toby_Ord 23 Mar 2026 10:54 UTC
19 points
3 ∶ 0
on: Which questions can’t we punt?
Thanks for this in-depth writeup of what is clearly a very important factor in prioritising our work aimed at the AI transition. Your piece has built the argument for such prioritisation clearly enough that it has allowed me to put some previously inchoate responses into a more crisp form:
If we could tell with certainty which topics would receive >100x as much work as we could put in prior to when that work is needed, then I think your argument goes through. But I have a lot of uncertainty about that and such uncertainty weakens the prioritisation effect substantially.
To see the effect easily, suppose for simplicity that for some piece of apparently late-stage strategy there is a 50% chance that >100x as much work gets done on it, obviating the need for us to work on it now, and a 50% chance that there is no appreciable extra work done (e.g. because the intelligence explosion is happening in a particular lab that doesn’t do this work, or because the work requires aspects of cognition that are improving more slowly, or because it turns out it was needed earlier in the explosion than expected).
In this case, the expected value of marginal work on that late-stage strategy gets roughly halved compared to if there weren’t going to be this AI-driven work later (50% chance of the naive estimate + 50% chance of <1% of that estimate). Given the fairly extreme distribution in the value of a particular person working on different topics, it isn’t that rare for the best thing of one category to work on being >2x as good as the best thing from another category, such that you shouldn’t switch category even after downgrading the EV.
That would mean early-stage vs late-stage would be an important factor in choosing what to work on, but not any kind of filter as to what to work on. As the chance of large amounts of AI work on that topic increases, the factor gets stronger. e.g. it reaches 10x at a 90% chance, which is quite strong (though I think it is hard to reach or exceed a 90% chance here).
So I think this can have a substantial effect on the choice of what to work on on the margins, but isn’t a filter.
What about its effect on the portfolio of research work aimed at the AI transition?
Suppose that there are logarithmic returns to the research work (which means that the marginal value of extra work is inversely proportional to aggregate work so far, which is a common neglectedness assumption). In that case, we should do 50% as much total work on equally-important things that we estimate to have a 50% chance of being obviated later, and 10% as much total work on those we estimate to have a 90% chance of being obviated later.
So that is still quite a lot of the share of our total work into late-stage things even when we don’t think they are intrinsically more important. In the piece you suggested that we do at least some work on these topics, to avoid the possibility of being caught completely flat-footed if the anticipated AI-work on those topics doesn’t happen, and I think the maths above suggests a larger amount of work than that (especially on topics that appear to be more important or more tractable).
(Note that my simplifying assumption of no appreciable AI help vs an overwhelming amount might be doing some work here. I’m not sure what the best way to relax it is.)

Toby_Ord 22 Mar 2026 17:18 UTC
8 points
2 ∶ 0
in reply to: Matthew_Barnett’s comment on: Broad Timelines
I agree that these are vague and could come apart from each other. But I don’t see any crisp, verifiable definitions that I could replace them with and serve the same purpose. I’m interested in forecasting transformative AI for the main purpose of forecasting when one has to have one’s AI-related impact by. e.g. by when do we need to have solved alignment (or to have paused AI development)?
If I instead used a verifiable definition here, such as the “In what year would AI systems be able to replace 99% of current fully remote jobs?” that I cite in the essay, then you have to do further forecasting of how that time relates to the key things that matter (such as the deadline on AI alignment). Also, for crisp concrete definitions, one tends to then get hung up on estimating exactly how hard the final 1% of current fully remote jobs are, because that is central to the prediction. For example, are there 1% of current fully remote jobs that we only let a human do, e.g. for reasons of legal responsibility or personal relationships? Maybe? But that isn’t relevant to the central features we care about.
I’m sure my definition could be improved (the focus of my essay isn’t on my prediction but on the wider points about everyone’s timelines), but I hope this explains why being “measurable and uncontroversial” need not make for the best thing to forecast.

Toby_Ord 20 Mar 2026 17:30 UTC
54 points
2 ∶ 1
in reply to: Angelina Li’s comment on: Broad Timelines
Thanks Angelina!
Annoying because: the core premise is so obvious, and yet I found spelling out the implications surprisingly clarifying!
This is actually the story of almost everything I’ve ever done:
- You should give to charities that do more good with your money (and find out which ones these are)
- Because you can save multiple lives with your donations, they are deeply morally serious and a key part of living a morally good life
- It would be really bad if humanity’s potential were destroyed, so we need to prioritise making sure that never happens
- …
It’s a byproduct of trying to find the most important, unarguably true, yet neglected things. I then work very hard on finding the deepest explanations until I find the ways of presenting each claim that make it effortless to see.
I like to think of it as working at the border of the trivial and the profound.
What links here?
- Mo Putera's comment on leogao’s Shortform by leogao (LessWrong; 27 Mar 2026 5:32 UTC; 2 points)

Toby_Ord 20 Mar 2026 10:33 UTC
7 points
0 ∶ 0
in reply to: NickLaing’s comment on: Broad Timelines
Thanks Nick!
How to communicate this is a good question, and I don’t yet know the best answer. I think admitting uncertainty is generally good — it is both honest and actually appreciated by many audiences. But there is still the question of how to do it. The scientific norm is usually to stay silent until the evidence for something (or some piece of the puzzle) is strong enough (e.g. reaching p = 0.05). I don’t think that is the right norm here. We are in a very high stake situation and policy-makers need the partial evidence that we do have. But communicating it is hard.
I think your expression is pretty good, and could be made a little better. e.g. “Leading AI forecasters can’t rule out it happening before 2030 and think it will probably happen before 2040.”
Re AI 2027, there is a good explanation of how their views have changed here.

Broad Timelines

Toby_Ord19 Mar 2026 19:05 UTC

227 points

26 comments16 min readEA link

Toby_Ord 16 Feb 2026 19:40 UTC
2 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: The Scaling Series Discussion Thread: with Toby Ord
And I’ll add that RL training (and to a lesser degree inference scaling) is limited to a subset of capabilities (those with verifiable rewards and that the AI industry care enough about to run lots of training on). So progress on benchmarks has been less representative of how good they are at things that aren’t being benchmarked than it was in the non-reasoning-model era. So I think the problems of the new era are somewhat bigger than the effects that show up in benchmarks.

Toby_Ord 16 Feb 2026 19:37 UTC
4 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: The Scaling Series Discussion Thread: with Toby Ord
That’s a great question. I’d expect a bit of slowdown this year, though not necessarily much. e.g. I think there is a 10x or so possible for RL before RL-training-compute reaches the size of pre-training compute, and then we know they have enough to 10x again beyond that (since GPT-4.5 was already 10x more), so there are some gains still in the pipe there. And I wouldn’t be surprised if METR timelines keep going up in part due to increased inference spend (i.e. my points about inference scaling not being that good are to do with costs exploding, so if a cost-insensitive benchmark is going on, it might not register on it all that much). There is also room for more AI-research or engineering improvements to these things, and a lump of new compute coming in, making it a bit messy.
Overall, I’d say my predictions are more about appreciable slowing in 2027+ rather than 2026.

Toby_Ord 12 Feb 2026 8:09 UTC
6 points
0 ∶ 0
in reply to: Ryan Greenblatt’s comment on: Distinguish between inference scaling and “larger tasks use more compute”
Good point about the METR curves not being Pareto frontiers.

Toby_Ord 11 Feb 2026 22:20 UTC
14 points
0 ∶ 0
on: Distinguish between inference scaling and “larger tasks use more compute”
Interesting ideas! A few quick responses:
1. The data for the early ‘linear’ regime for these models actually appears to be even better than you suggest here. They have a roughly straight line (on a log-log plot), but at a slope that is better than 1. Eyeballing it, I think some are slope 5 or higher (i.e. increasing returns, with time horizon growing as the 5th power of compute). See my 3rd chart here. If anything, this would strengthen your case for talking about that regime separately from the poorly scaling high compute regime later on.
2. I’d also suspected that when you apply extra RL to a model (e.g. o3 compared to o1) that it would have a curve that dominated the earlier model. But that doesn’t seem to be the case. See the curves in the final chart here, where o1-preview is dominated, but the other OpenAI models curves all cross each other (being cheaper for the same horizon at some horizons and more expensive at others).
3. Even when they do dominate each other neatly like in your fake data, I noticed that the ‘sweet spots’ and the ‘saturation points’ can still be getting more expensive, both in terms of $ and in terms of $/hr. I’m not sure what to make of that though!
4. I think you’re on to something with the idea that there is a problematic kind of inference scaling and a fine kind, though I’m not sure if you’ve quite put your finger on how to distinguish them. I suppose we can definitely talk about the super-linear scaling regime and the sub-linear regime (which meet at what I call the sweet spot), but I’m not sure these are the two types you refer to in qualitative terms near the top.

Toby_Ord 10 Feb 2026 21:10 UTC
2 points
0 ∶ 0
in reply to: Ben_West🔸’s comment on: The Scaling Series Discussion Thread: with Toby Ord
Yeah, it isn’t just like a constant factor slow-down, but is fairly hard to describe in detail. Pre-training, RL, and inference all have their own dynamics, and we don’t know if there will be new good scaling ideas that breathe new life into them or create a new thing on which to scale. I’m not trying to say the speed at any future point is half what it would have been, but that you might have seen scaling as a big deal, and going forward it is a substantially smaller deal (maybe half as big a deal).

Toby_Ord 9 Feb 2026 8:51 UTC
2 points
0 ∶ 0
in reply to: Denkenberger🔸’s comment on: Inference Scaling Reshapes AI Governance
Thanks for catching that — a lot of symbols in the appendix were lost when converting the post for the forum, so I’ve edited it to add them back in.

Toby_Ord 6 Feb 2026 15:56 UTC
4 points
0 ∶ 0
in reply to: Margot Stakenborg’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
That’s an interesting way to connect these. I suppose one way to view your model is as making clear the point that you can’t cost-effectively use models on tasks that much longer than their 50% horizons — even if you are willing to try multiple times — and that trend of dramatic price improvements over time isn’t enough to help with this. Instead you need the continuation of the METR trend of exponentially growing horizons. Moreover, you give a nice intuitive explanation of why that is.
One thing to watch out for is Gus Hamilton’s recent study suggesting that there isn’t a constant hazard rate. I share my thoughts on it here, but my basic conclusion is that he is probably right. In particular, he has a functional form estimating how their success probability declines. You could add this to your model (it is basically 1 minus the CDF of a Weibull distribution with K=0.6). I think this survival function tail is a power law rather than an exponential, making the ‘just run it heaps of times’ thing slightly more tenable. It may mean that it is the cost of human verification that gets you, rather than it being untenable even on AI costs alone.
What links here?

Toby_Ord 4 Feb 2026 19:26 UTC
4 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
And one of the authors of the METR timelines paper has his own helpful critique/clarifications of their results.

Toby_Ord 4 Feb 2026 19:23 UTC
4 points
1 ∶ 0
in reply to: Charlie_Guthmann’s comment on: Are the Costs of AI Agents Also Rising Exponentially?
Good points. I’m basically taking METR’s results at face value and showing that people are often implicitly treating costs (or cost per ‘hour’) as constant (especially when extrapolating them), but show that these costs appear to be growing substantially.
Re the quality / generalisability of the METR timelines, there is quite a powerful critique of it by Nathan Witkin. I wouldn’t go as far as he does, but he’s got some solid points.

Toby_Ord 4 Feb 2026 19:16 UTC
6 points
0 ∶ 0
in reply to: basil.halperin’s comment on: The Scaling Series Discussion Thread: with Toby Ord
Thanks Basil! That’s an interesting idea. The constant hazard rate model is just comparing two uses of the same model over different task lengths, so if using that to work out the 99% time horizon, it should cost 1/70th as much ($1.43). Over time, I think these 99% tasks should rise in cost in roughly the same way as the 50%-horizon ones (as they are both increasing in length in proportion). But estimating how that will change in practice is especially dicey as there is too little data.
Also, note that Gus Hamilton has written a great essay that takes the survival analysis angle I used in my constant hazard rates piece and extended it to pretty convincingly show that the hazard rates are actually decreasing. I explain it in more detail here. One upshot is that it gives a different function for estimating the 99% horizon lengths and he also shows that these are poorly constrained by the data and his model disagrees with METR’s by a factor of 20 on how long they are, with even more disagreement for shorter lengths.

Toby_Ord 4 Feb 2026 16:53 UTC
4 points
0 ∶ 0
on: Is there a Half-Life for the Success Rates of AI Agents?
Some great new analysis by Gus Hamilton shows that AI agents probably don’t obey a constant hazard rate / half-life after all. Instead their hazard rates systematically decline as the task goes on.
This means that their success rates on tasks beyond their 50%-horizon are better than the simple model suggests, but those for tasks shorter than the 50% horizon are worse.
I had suggested a constant hazard rate was a good starting assumption for how their success rate at tasks decays with longer durations. It is the simplest model and fits the data OK. But Gus used the standard second-simplest model from survival analysis (the Weibull distribution rather than the exponential distribution). It has a second parameter, K, which represents how the hazard rate changes with time (if at all). If K=1, there is a constant hazard rate, so the exponential distribution is a special case of the Weibull. But if K<1, then hazard decreases over time (like the Lindy effect), and if it is greater, hazard increases (like aging).
Gus found that the estimated values for K were below 1 for all the models, showing that *all* of them had decreasing hazard rates.
A distribution that generalises another is always going to get a better fit of the data than my exponential distribution, so fit alone wouldn’t be decisive. But the way that every single model has K statistically significantly below 1 convinces me he is right.
So what does this mean?
One thing is that it gives very different estimated success rates for tasks much shorter or longer than the 50% horizon (which METR focuses on because it is easier to reliably estimate). e.g. use the Weibull to estimate the 99% horizon (or 10% horizon).
Another thing is that the AI agents mainly have a K of about 0.6, while the human value of K is significantly lower, at about 0.4. This means even if they have the same 50% horizon, humans can do better on really long tasks (and worse on really short ones).
As this shows, for a fixed 50% horizon length, it isn’t clearly better or worse to have a lower value of K. Lower values are good for the low success rate really long tasks, but worse for the high reliability thresholds.
As a word of warning, I was quite sure before METR released its Opus 4.5 results that it was going to have a more human-like value of K, since it had a great 50% horizon, but only an average showing at 80%. But the estimates are that its value of K is similar. I’m not sure why that is, but it might be due to the fact that there isn’t much data to go on here and things are quite noisy for any individual model.
So, from Gus’s results, it still looks like there is some important gap between how human success rates drop off at longer tasks versus how AI agents do.
Gus also compares his two-parameter Weibull model of the data to METR’s two-parameter log-logistic model. He finds that they are similar, but with the log-logistic fitting slightly better. So it isn’t clear which of these to use if you have the choice. They differ quite a lot in the tails of the distribution (i.e. in estimated success rates for very short or very long tasks). e.g. the Weibull says the 99% horizon is 1/20th as long as the log-logistic predicts. That’s a big deal and the data doesn’t tell us which to favour! I’d slightly favour the Weibull, on the grounds that it is more plausible ex ante. But maybe the bigger lesson is that it is unknown which is right, and thus the 99% horizons (necessary for much useful work) are deeply uncertain.

Toby_Ord

The Dy­nam­ics of In­tel­li­gence Explosions

Mo­ral Imag­i­na­tion & Effec­tive Altruism

Broad Timelines

The Dynamics of Intelligence Explosions

Moral Imagination & Effective Altruism