I don’t have high familiarity with METR but I think it is probably not great data for this type of analysis. Few issues or clarifications would be needed (and anyone who understands METR better bear with me/or correct me on my mistakes plz).
1. How does METR handle context windows? Are we doing a rolling window? Compact? something else? How much of this inverse quadratic relationship is just caused by longer tasks having a larger used context window for the back half of the run? How much is caused by a lack of a default information management system that persists?
2. What is the exact harness(es) METR is using?
Harness/enviornment engineering/information management might control more of the cost of long running SWE projects than iq (past a point).
3. Does METR allow repo forking? routing? In the future no 180 iq ai is building the ORM and buttons for a crud app. They are either forking a boilerplate or routing it to a cheaper model.
etc.
It is said that the current iteration of models suffer from retrograde amnesia. Whether or not this will get bitter lesson pilled is a separate question, but for this class of memento models, version control, information management, context management, and the meta process of improving and routing through the best versions of these combos for a specific task is not some side quest but in fact the main route to making long tasks cheaper. Even as we enter the next paradigm of models that don’t have such profound short term memory loss, a huge part of cost reduction will come from the orchestrator meta planning about how much to explore the space of options/build out the software factory / vs actually starting the work.
I’m not denying the core question OP is raising — costs could plausibly be rising and could matter a lot. I’m just not convinced this specific curve cleanly isolates “AI economics” from “how expensive a particular scaffold/set of arbitrary constraints makes long-context work.”
Good points. I’m basically taking METR’s results at face value and showing that people are often implicitly treating costs (or cost per ‘hour’) as constant (especially when extrapolating them), but show that these costs appear to be growing substantially.
Re the quality / generalisability of the METR timelines, there is quite a powerful critique of it by Nathan Witkin. I wouldn’t go as far as he does, but he’s got some solid points.
Thank you for doing this, love to see some data.
I don’t have high familiarity with METR but I think it is probably not great data for this type of analysis. Few issues or clarifications would be needed (and anyone who understands METR better bear with me/or correct me on my mistakes plz).
1. How does METR handle context windows? Are we doing a rolling window? Compact? something else?
How much of this inverse quadratic relationship is just caused by longer tasks having a larger used context window for the back half of the run? How much is caused by a lack of a default information management system that persists?
2. What is the exact harness(es) METR is using?
Harness/enviornment engineering/information management might control more of the cost of long running SWE projects than iq (past a point).
3. Does METR allow repo forking? routing?
In the future no 180 iq ai is building the ORM and buttons for a crud app. They are either forking a boilerplate or routing it to a cheaper model.
etc.
It is said that the current iteration of models suffer from retrograde amnesia. Whether or not this will get bitter lesson pilled is a separate question, but for this class of memento models, version control, information management, context management, and the meta process of improving and routing through the best versions of these combos for a specific task is not some side quest but in fact the main route to making long tasks cheaper. Even as we enter the next paradigm of models that don’t have such profound short term memory loss, a huge part of cost reduction will come from the orchestrator meta planning about how much to explore the space of options/build out the software factory / vs actually starting the work.
I’m not denying the core question OP is raising — costs could plausibly be rising and could matter a lot. I’m just not convinced this specific curve cleanly isolates “AI economics” from “how expensive a particular scaffold/set of arbitrary constraints makes long-context work.”
Good points. I’m basically taking METR’s results at face value and showing that people are often implicitly treating costs (or cost per ‘hour’) as constant (especially when extrapolating them), but show that these costs appear to be growing substantially.
Re the quality / generalisability of the METR timelines, there is quite a powerful critique of it by Nathan Witkin. I wouldn’t go as far as he does, but he’s got some solid points.
And one of the authors of the METR timelines paper has his own helpful critique/clarifications of their results.