I think it can be a fun exercise is to just interpret CEOs statements literally and see what they imply.
If Dario Amodei claims they don’t want to push capabilities, I think an interesting question to ask is in what sense releasing the world’s best LLM isn’t pushing capabilities.
One option that seems possible to me, could be that they no longer consider releasing improved LLMs to meaningfully push the frontier. If Claude-3 spurs OpenAI to push a quicker release of GPT-4.5, this would not be an issue as releasing ever more refined LLMs doesn’t meaningfully increase any specific risk.
In that case they would still be true to their word, Anthropic just no longer considers LLMs on their own to be the frontier of capabilities. This seems reasonable to me?
It certainly does seem to push capabilities, although one could argue about whether the extent of it is very significant or not.
Being confused and skeptical about their adherence to their stated philosophy seems justified here, and it is up to them to explain their reasoning behind this decision.
On the margin, this should probably update us towards believing they don’t take their stated policy of not advancing the SOTA too seriously.
I think the answer is ‘yes’ for a general layperson’s understanding of ‘pushing capabilities’, but the emerging EA discourse on this seems to be at risk on inflating several questions:
Has Claude-3 shown better capability than other models? Yes under certain specific conditions and benchmarks
Do those benchmarks matter/​actually capture performance of interest? No, in my opinion. I’d recommend reading Melanie Mitchell’s takes on this.
Does Claude-3′s extra capabilities make it more likely to cause an x-risk event? No, or at least the probability that the current frontier AI model will cause an x-risk event has gone from ~epsilon to ~epsilon
Will Claude-3′s release increase or decrease x-risk? Very difficult to say, I don’t know how people get over cluelessness objections to these questions.
So I guess in your post ‘frontier’ is covering 2 separate concepts, the ‘frontier’ in terms of published benchmarks and the ‘fronitier’ in terms of marginal x-risk increase. In my opinion, Claude-3 may be an interesting case where these come apart.
Does Claude-3 push capabilities?
I think it can be a fun exercise is to just interpret CEOs statements literally and see what they imply.
If Dario Amodei claims they don’t want to push capabilities, I think an interesting question to ask is in what sense releasing the world’s best LLM isn’t pushing capabilities.
One option that seems possible to me, could be that they no longer consider releasing improved LLMs to meaningfully push the frontier. If Claude-3 spurs OpenAI to push a quicker release of GPT-4.5, this would not be an issue as releasing ever more refined LLMs doesn’t meaningfully increase any specific risk.
In that case they would still be true to their word, Anthropic just no longer considers LLMs on their own to be the frontier of capabilities. This seems reasonable to me?
It certainly does seem to push capabilities, although one could argue about whether the extent of it is very significant or not.
Being confused and skeptical about their adherence to their stated philosophy seems justified here, and it is up to them to explain their reasoning behind this decision.
On the margin, this should probably update us towards believing they don’t take their stated policy of not advancing the SOTA too seriously.
I think the answer is ‘yes’ for a general layperson’s understanding of ‘pushing capabilities’, but the emerging EA discourse on this seems to be at risk on inflating several questions:
Has Claude-3 shown better capability than other models? Yes under certain specific conditions and benchmarks
Do those benchmarks matter/​actually capture performance of interest? No, in my opinion. I’d recommend reading Melanie Mitchell’s takes on this.
Does Claude-3′s extra capabilities make it more likely to cause an x-risk event? No, or at least the probability that the current frontier AI model will cause an x-risk event has gone from ~epsilon to ~epsilon
Will Claude-3′s release increase or decrease x-risk? Very difficult to say, I don’t know how people get over cluelessness objections to these questions.
So I guess in your post ‘frontier’ is covering 2 separate concepts, the ‘frontier’ in terms of published benchmarks and the ‘fronitier’ in terms of marginal x-risk increase. In my opinion, Claude-3 may be an interesting case where these come apart.