N=1, but I looked at an ARC puzzle https://​​arcprize.org/​​play?task=e3721c99, and I couldn’t just do it in a few minutes, and I have a PhD from the University of Oxford. I don’t doubt that most of the puzzles are trivial for some humans, and some of the puzzles are trivial for most humans or that I could probably outscore any AI across the whole ARC-2 data set. But at the same time, I am a general intelligence, so being able to solve all ARC puzzles doesn’t seem like a necessary criteria. Maybe this is the opposite of how doing well on benchmarks doesn’t always generalize to real world tasks, and I am just dumb at these but smart overall, and the same could be true for an LLM.
Ah, okay, that is tricky! I totally missed one of the rules that the examples are telling us about. Once you see it, it seems simple and obvious, but it’s easy to miss. If you want to see the solution, it’s here.
I believe all ARC-AGI-2 puzzles contain (at least?) two different rules that you have to combine. I forgot about that part! I was trying to solve the puzzle as if there was just one rule to figure out.
I tried the next puzzle and was able to solve it right away, on the first try, keeping in mind the ‘two rules’ thing. These puzzles are actually pretty fun, I might do more.
N=1, but I looked at an ARC puzzle https://​​arcprize.org/​​play?task=e3721c99, and I couldn’t just do it in a few minutes, and I have a PhD from the University of Oxford. I don’t doubt that most of the puzzles are trivial for some humans, and some of the puzzles are trivial for most humans or that I could probably outscore any AI across the whole ARC-2 data set. But at the same time, I am a general intelligence, so being able to solve all ARC puzzles doesn’t seem like a necessary criteria. Maybe this is the opposite of how doing well on benchmarks doesn’t always generalize to real world tasks, and I am just dumb at these but smart overall, and the same could be true for an LLM.
Ah, okay, that is tricky! I totally missed one of the rules that the examples are telling us about. Once you see it, it seems simple and obvious, but it’s easy to miss. If you want to see the solution, it’s here.
I believe all ARC-AGI-2 puzzles contain (at least?) two different rules that you have to combine. I forgot about that part! I was trying to solve the puzzle as if there was just one rule to figure out.
I tried the next puzzle and was able to solve it right away, on the first try, keeping in mind the ‘two rules’ thing. These puzzles are actually pretty fun, I might do more.