If LLMs are adopting poor learning heuristics and not generalizing, AI2027 is predicting a weaker kind of “superhuman” coder — one that can reliably solve software tasks with clean feedback loops but will struggle on open-ended tasks!
No, AI 2027 is predicting a kind of superhuman coder that can automate even messy open ended research engineering tasks. The forecast attempts to account for gaps between automatically-scoreable, relatively clean + green-field software tasks and all tasks. (Though the adjustment might be too small in practice.)
If LLMs can’t automate such tasks and nothing else can automate such tasks, then this wouldn’t count as superhuman coder happening.
No, AI 2027 is predicting a kind of superhuman coder that can automate even messy open ended research engineering tasks. The forecast attempts to account for gaps between automatically-scoreable, relatively clean + green-field software tasks and all tasks. (Though the adjustment might be too small in practice.)
If LLMs can’t automate such tasks and nothing else can automate such tasks, then this wouldn’t count as superhuman coder happening.