While you can use o1 and gemini with internet access, I think they almost certainly evaluated it without such access (see the original paper here).
I really really do not think you should put the plot there. It’s like comparing two different students performance except one of them has access to the internet. I think it’s extremely misleading. If you want to illustrate progress you could just use the FrontierMath/GPQA results or even ARC-AGI.
While you can use o1 and gemini with internet access, I think they almost certainly evaluated it without such access (see the original paper here).
I really really do not think you should put the plot there. It’s like comparing two different students performance except one of them has access to the internet. I think it’s extremely misleading. If you want to illustrate progress you could just use the FrontierMath/GPQA results or even ARC-AGI.
Thanks this is helpful.
(Just adding the FrontierMath/GPQA and ARC-AGI charts you mentioned for my own benefit, and others)