METR Time Horizon 2.0—The benchmark you’ve been waiting for

AgentMa🔸8 Jul 2026 23:41 UTC
23 points
16 comments5 min readEA link

TL;DR: I applied METR’s time-horizon methodology to Microsoft Excel. On my 23-task suite, Excel completes tasks that take an unaided human 6.5 hours at 80% reliability — more than double Claude Mythos Preview, the best frontier model on the current METR plot (~3 hours at 80%). Excel’s 50%-time horizon is 54 working weeks. I have updated my probability of spreadsheet-induced catastrophe by 2035 from 0.1% to 5–10%. Then I wrote down everything wrong with my methodology and noticed I had written a review of the METR plot.

Methodology

METR measures a model’s “time horizon”: the task duration, measured by human completion time, at which an agent is predicted to succeed with a given reliability. You fit a logistic curve of success against log task length and read off where it crosses 50% or 80%. I did exactly this, with one substitution: the agent is Microsoft Excel.

  • Task suite (HCALC, n=23). Human-Calibrated Arithmetic and Ledger Computations: quantitative tasks screened for objective, automatic scorability — grocery receipts, payroll, a 360-payment amortization schedule, descriptive statistics on 500 blood-pressure readings, a 200×200 matrix inversion, and two Monte Carlo simulations. Screening for automatic scorability conveniently restricts the suite to things Excel can attempt. I consider this a feature.

  • Human baselines (Ti). Ti is how long each task takes a human without Excel: pencil, paper, printed Z-table. Four-function calculators permitted; phones confiscated. For the longest tasks nobody would agree to a baseline, so I estimated. The retirement Monte Carlo — 10,000 paths × 480 months with a d20 and a normal table — I scored at 115,000 minutes, roughly eleven months of full-time dice rolling. Nobody checked this number. Nobody could.

  • The agent. Excel cannot type, so it is scaffolded with a human peripheral who keys in values at a measured 0.3 seconds per entry (n=1, mildly caffeinated) and contributes no cognition. Peripheral time is attributed to the scaffold and excluded from analysis, consistent with the standard practice of not counting inference infrastructure. Note the horizon is human time-to-complete, not agent wall-clock — METR’s FAQ is explicit that agents are typically much faster than humans on tasks they complete. Excel finished the retirement simulation in 51 seconds, a 135,294× speedup, of which roughly 30 seconds was locating the fill handle.

  • Scoring and aggregation. Binary automatic scoring. Weighted logistic MLE of success on log2(Ti) with inverse-square-root task-family weights — METR’s exact procedure, reproduced in the attached workbook as live formulas. You can rerun the fit yourself with Solver, which felt thematically important.

Results

HCALC logistic fit: P(Excel succeeds) vs. human task length

Excel succeeded on 19 of 23 tasks (80.6% weighted). The failures: it believes 1900 was a leap year (a real bug, preserved deliberately for Lotus 1-2-3 compatibility); the matrix inversion overflowed; the 2.5-million-row ledger exceeded the 1,048,576-row worksheet limit; and the customer-ranking run was scored zero after the human peripheral developed repetitive strain injury at row 14,203.

The fitted curve crosses 80% at 390 minutes — 6 hours 30 minutes (95% CI: 18 seconds to unbounded; 12.7% of bootstrap resamples contain no failures at all, in which case the horizon is infinite). It crosses 50% at 54 working weeks.

80%-time horizon of frontier systems, 1985–2026

For context, I computed 80% horizons for frontier LLMs from METR’s own published TH1.1 run data, using the same fitting code. The best system on the current plot, Claude Mythos Preview, sits around 3 hours at 80% reliability — METR reports it more than doubled the next-best model, and the next-best (Claude Opus 4.6) computes to about 70 minutes. Excel beats the frontier by 2.2×. The LLM frontier’s doubling time works out to roughly 4 months, consistent with what METR and the UK AISI report, which back-extrapolates to a predicted 1985 horizon of about 10−32 seconds. Excel’s measured horizon exceeds trend by 36 orders of magnitude. Alternatively, the trend is fine and Excel’s capability has simply been flat for 41 years — which, under this framework, means it is due.

What this did to my p(doom)

I will be honest: I am rattled. Before running HCALC, my probability of catastrophic outcomes from spreadsheet technology by 2035 was maybe 0.1%. It is now 5–10%. This capability was hiding in plain sight — installed on more than a billion devices, embedded in every bank, every hospital, every defense contractor — and nobody benchmarked it. We spent 2025 arguing over whether the frontier had crossed one hour or two, and the entire time a system with a six-and-a-half-hour horizon was sitting in the taskbar. I did not realize we already had these advanced technologies. If a capability can sit 36 orders of magnitude above trend for four decades without anyone noticing, the correct response to any capability chart is fear, and the correct response to the absence of a capability chart is more fear. I keep coming back to the Monte Carlo result: eleven months of human labor, 51 seconds of machine time. That ratio is not going to get smaller.

Problems with my methodology

Having slept on it, some concerns, in descending order of severity.

  1. Task selection is the entire result. The suite is small, its topics are known in advance, and it was assembled by someone who knew what answer he wanted. Whether Excel’s 50%-time horizon is one month (drop both Monte Carlo tasks: 3.8 work-weeks), one year (keep them: 54.3), or eight years (add a third — say, a weather simulation I could invent in the time it took to type this sentence: 436 work-weeks) depends entirely on how many Monte Carlo tasks I felt like including. The 80% horizon swings from 3.6 hours to 11.3 hours on the same choice. Nothing about Excel changes; only my sampling does.

  2. n is tiny exactly where it matters. The 80% crossing is pinned by a handful of tasks in the 4-to-160-hour range. A single flipped outcome there moves the headline number by hours.

  3. Human baselines are unfalsifiable. No human computes the standard deviation of 500 readings by hand, so “how long would it take one” is a guess wearing a number. My longest baselines were estimated rather than measured, and the estimates carry all the leverage.

  4. Binary scoring flattens everything. A leap-year bug scores identically to total incapacity. An operator injury scores as a model failure. There is no partial credit and no concept of “almost.”

  5. The logistic does the work. Two parameters, fit through mostly-ones and a few zeros. Given only my aggregate success rate and the task-length distribution, you could recover the horizon without knowing which tasks Excel passed. The 330× gap between my 50% and 80% horizons is not a fact about Excel; it is a fact about the fitted slope.

  6. The confidence interval is a shrug. Eighteen seconds to infinity.

  7. The tasks are not work. Clean, self-contained, algorithmically scorable. Real quantitative work is ambiguous, contextual, and involves other people — a regime where Excel performs about as well as it performs 1900 date arithmetic.

  8. None of this licenses extrapolation to “all economically valuable labor,” a phrase I nonetheless felt in my chest while formatting the chart.

These exact problems exist on the METR plot

Every item above is a live issue with the chart your feed re-litigates monthly. Through 2025, the frontier region of the METR plot — tasks of one to four hours — contained 14 samples, and the task topics are public, weighted toward cybersecurity CTFs and ML-engineering problems that labs openly train for; a lab can move its dot by upsampling those distributions, deliberately or by accident. Claude 3.7 Sonnet was assigned a 59-minute horizon while succeeding on roughly 60% of one-to-two-hour tasks, because it went 0-for on the 2–4 hour bucket and the logistic fit punished the whole curve for it. Shashwat Goel showed you can reconstruct the entire log-linear trend from aggregate accuracy plus the task-length distribution with a fixed slope — the individual task outcomes barely matter. METR’s own limitations notes supply the rest: bootstrapped confidence intervals of roughly a factor of two in each direction, widening as the suite saturates; 50% and 80% horizons that are not independent estimates, because a two-parameter logistic cannot fit both ends of the curve; only 5 of 31 tasks over eight hours with measured rather than estimated human baselines; success rates that fall about eight points per unit of task “messiness”; and a standing notice that measurements above 16 hours are unreliable on the current task suite — which did not stop the discourse from treating a ≥16-hour point estimate for Mythos Preview as a fire alarm.

To be clear about the target: this is not a case against METR. They publish their runs, their code, and their caveats, and that transparency is the only reason this parody was buildable in an afternoon; task-length horizons remain a better question than benchmark accuracy. It is a case against the inference pipeline downstream — the one where a dot moves inside a 14-sample region of a two-parameter curve fit, and timelines, investment theses, and p(doom)s all reprice by close of business. If a chart moves your worldview, first count the samples doing the moving. Mine had two. They were dice.

AgentMa🔸8 Jul 2026 23:41 UTC
23 points
16 comments5 min readEA link