Note that the old[1] o3-high that was tested on ARC-AGI-1:
Was 10x more expensive than initially estimated, taking it over the $10,000 limit, so is no longer included on the leaderboard
Took 1,024 attempts at each task, writing about 137 pages of text for each attempt, or about 43 million words total
OpenAI have stated that the newly-released o3 is not the same one as was evaluated on ARC-AGI-1 in December
Good Lord! Thanks for this information!
The Twitter thread by Toby Ord is great. Thanks for linking that. This tweet helps put things in perspective:
For reference, these are simple puzzles that my 10-year-old child can solve in about 4 minutes.
Note that the old[1] o3-high that was tested on ARC-AGI-1:
Was 10x more expensive than initially estimated, taking it over the $10,000 limit, so is no longer included on the leaderboard
Took 1,024 attempts at each task, writing about 137 pages of text for each attempt, or about 43 million words total
OpenAI have stated that the newly-released o3 is not the same one as was evaluated on ARC-AGI-1 in December
Good Lord! Thanks for this information!
The Twitter thread by Toby Ord is great. Thanks for linking that. This tweet helps put things in perspective: