Most of these criticisms are not new; for an organized writeup of the most important known issues in the original time horizon paper see my blog post from January that OP linked in the conclusion. I do have some comments on this post’s methodology.
Task suite (HCALC, n=23). Human-Calibrated Arithmetic and Ledger Computations: quantitative tasks screened for objective, automatic scorability — grocery receipts, payroll, a 360-payment amortization schedule, descriptive statistics on 500 blood-pressure readings, a 200×200 matrix inversion, and two Monte Carlo simulations. Screening for automatic scorability conveniently restricts the suite to things Excel can attempt.
Excel finished the retirement simulation in 51 seconds, a 135,294× speedup
The super long tasks are not economically valuable, because no human would do 200x200 matrix inversion or Monte Carlo simulations by hand in 1985, so they’re not really worth tens of weeks of human wages. The baseline for these should probably be a human C or Fortran programmer with access to a reasonably fast computer, probably an hour or two rather than weeks.
It is also not really true that automatic scoreability restricts the suite to things Excel is good at. E.g. Excel cannot do reasoning questions or computer use, but it does support scripting which is not included here.
The agent. Excel cannot type, so it is scaffolded with a human peripheral who keys in values at a measured 0.3 seconds per entry (n=1, mildly caffeinated) and contributes no cognition.
If the point is that the scaffold heavily affects the intelligence of the model, METR tested this in February and found that Codex and Claude Code scaffolds don’t outperform the standard scaffold we use. Generally the bigger issue has been models not using scaffolds properly than exactly how much optimization goes into the scaffold.
80%-time horizon of frontier systems, 1985–2026
The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.
The logistic does the work. Two parameters, fit through mostly-ones and a few zeros. Given only my aggregate success rate and the task-length distribution, you could recover the horizon without knowing which tasks Excel passed.
Shashwat Goel showed you can reconstruct the entire log-linear trend from aggregate accuracy plus the task-length distribution with a fixed slope — the individual task outcomes barely matter.
It’s true that time horizon is highly correlated with success rate if you know the task length distribution, and I used this shortcut in my follow-up last year, for all the benchmarks where we didn’t have individual question data. IMO it’s not a major flaw because is known to differ wildly by task distribution and you can’t estimate it just from aggregate accuracy; there could be some weird distribution on which even isn’t enough.
If a capability can sit 36 orders of magnitude above trend for four decades without anyone noticing, the correct response to any capability chart is fear, and the correct response to the absence of a capability chart is more fear.
This violates conservation of expected evidence. It can’t be the case that capability chart and no capability chart are both evidence of dangerous capabilities, and it’s not healthy or productive to feel fear regardless of the evidence.
Finally, this post seems almost entirely AI written, probably by Claude 4.8 or Fable 5. Pangram says it’s 100% AI written. This created a bunch of minor issues in the writing.
“The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.”
Most of these criticisms are not new; for an organized writeup of the most important known issues in the original time horizon paper see my blog post from January that OP linked in the conclusion. I do have some comments on this post’s methodology.
The super long tasks are not economically valuable, because no human would do 200x200 matrix inversion or Monte Carlo simulations by hand in 1985, so they’re not really worth tens of weeks of human wages. The baseline for these should probably be a human C or Fortran programmer with access to a reasonably fast computer, probably an hour or two rather than weeks.
It is also not really true that automatic scoreability restricts the suite to things Excel is good at. E.g. Excel cannot do reasoning questions or computer use, but it does support scripting which is not included here.
If the point is that the scaffold heavily affects the intelligence of the model, METR tested this in February and found that Codex and Claude Code scaffolds don’t outperform the standard scaffold we use. Generally the bigger issue has been models not using scaffolds properly than exactly how much optimization goes into the scaffold.
The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.
It’s true that time horizon is highly correlated with success rate if you know the task length distribution, and I used this shortcut in my follow-up last year, for all the benchmarks where we didn’t have individual question data. IMO it’s not a major flaw because
This violates conservation of expected evidence. It can’t be the case that capability chart and no capability chart are both evidence of dangerous capabilities, and it’s not healthy or productive to feel fear regardless of the evidence.
Finally, this post seems almost entirely AI written, probably by Claude 4.8 or Fable 5. Pangram says it’s 100% AI written. This created a bunch of minor issues in the writing.
“The graph lists Microsoft Excel 1.0, but the benchmarks must have been run on a modern version of Excel. The worksheet limit of Excel 1.0 was only 16,384 rows rather than 1,048,576, plus it lacked many of the functions of modern Excel, so it would presumably fail more tasks.”
This is beautiful. Thank you.