I donât think they are designed to be a low bar to clear.
Based on what?
This is what François Chollet said about ARC-AGI in a post on Bluesky from January 6, 2025:
I donât think people really appreciate how simple ARC-AGI-1 was, and what solving it really means.
It was designed as the simplest, most basic assessment of fluid intelligence possible.
Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. Passing it means your system exhibits non-zero fluid intelligenceâyouâre finally looking at something that isnât pure memorized skill. But it says rather little about how intelligent your system is, or how close to human intelligence it is.
On Dwarkesh Patelâs podcast, Chollet emphasized that pretty much anybody can solve ARC-AGI puzzles, even children.
It seems pretty hard to make a timelines update from ARC-AGI unless you are very confident in the importance of abstract shape rotation problems for much more concrete problems, or you care about some notion of âintelligenceâ much more than automating intellectual labour.
Youâve got to measure something and the most commonly cited benchmarks for LLMs mostly seem to measure memorizing large quantities of text with very limited generalization to novel chunks of text. Thatâs cool, but I donât think itâs measuring general intelligence.
ARC-AGI and the new and improved ARC-AGI-2 are specifically designed to measure progress toward AGI by focusing on capabilities that humans have and AI doesnât. I donât know if it succeeds in measuring general intelligence, but I find it a lot more interesting than the benchmarks that reward memorizing text.
I think it would be a good idea for others to take inspiration from ARC-AGI-2 and design new benchmarks that specifically focus on what humans can do ~100% of the time and what AI can do ~0% of the time. If you donât try to measure this, and you arenât really careful and thoughtful in how you measure it, you risk ending up with distorted conclusions about AGI progress.
Based on what?
This is what François Chollet said about ARC-AGI in a post on Bluesky from January 6, 2025:
On Dwarkesh Patelâs podcast, Chollet emphasized that pretty much anybody can solve ARC-AGI puzzles, even children.
Youâve got to measure something and the most commonly cited benchmarks for LLMs mostly seem to measure memorizing large quantities of text with very limited generalization to novel chunks of text. Thatâs cool, but I donât think itâs measuring general intelligence.
ARC-AGI and the new and improved ARC-AGI-2 are specifically designed to measure progress toward AGI by focusing on capabilities that humans have and AI doesnât. I donât know if it succeeds in measuring general intelligence, but I find it a lot more interesting than the benchmarks that reward memorizing text.
I think it would be a good idea for others to take inspiration from ARC-AGI-2 and design new benchmarks that specifically focus on what humans can do ~100% of the time and what AI can do ~0% of the time. If you donât try to measure this, and you arenât really careful and thoughtful in how you measure it, you risk ending up with distorted conclusions about AGI progress.