On “It’s very important that AI companies have accurate views on how dangerous their models are”. I would agree its important to the companies so they can prevent near-term harm and increase long-term acelleration.
I would argue that if mythos had slipped through a month ago, and lets say a bank and a government were hacked then we would have our biggest warning shot yet. If Claude had released mythos prematurely, I think it would have reduced AI risk long term because it probably would have freaked out governments and the public, which might then have legislated and put brakes on.
In this case, if Anthropic had prematurely released it, that would have slowed the race more than the real world scenario where they didn’t. The slowing due to not releasing is IMO almost negligable.
I would argue similar for biorisk evals. A warning shot now might trigger the kind of public/government reaction we need before risks get existential. Hiccups now while models aren’t takeover/existential risk ready might slow the race down in a meaningful way. Preventing lower-level biorisk events now could increase existential risk later.
But it’s obviously really difficult to tell if this kind of short term pain might be worth the longer term gain. But if the labs want the safety now, its for the purpose of continued scaling more than the safety itself. That should give us pause.
Also we’ve already seen Anthropic and Open AI back down on their safety docs red-lines, “knowing” doesn’t mean “slowing”. Its entirely possible all of the evals come in, the knowledge is there and everyone just plows on.
I would be like 51% sure on this (so barely at all), but at the very least its daft to automatically think that safety work now is necessarily a good thing for the world long term. There’s a lot of complexity there. I think there are strong arguments for and against near-term safety.
On “It’s very important that AI companies have accurate views on how dangerous their models are”. I would agree its important to the companies so they can prevent near-term harm and increase long-term acelleration.
I would argue that if mythos had slipped through a month ago, and lets say a bank and a government were hacked then we would have our biggest warning shot yet. If Claude had released mythos prematurely, I think it would have reduced AI risk long term because it probably would have freaked out governments and the public, which might then have legislated and put brakes on.
In this case, if Anthropic had prematurely released it, that would have slowed the race more than the real world scenario where they didn’t. The slowing due to not releasing is IMO almost negligable.
I would argue similar for biorisk evals. A warning shot now might trigger the kind of public/government reaction we need before risks get existential. Hiccups now while models aren’t takeover/existential risk ready might slow the race down in a meaningful way. Preventing lower-level biorisk events now could increase existential risk later.
But it’s obviously really difficult to tell if this kind of short term pain might be worth the longer term gain. But if the labs want the safety now, its for the purpose of continued scaling more than the safety itself. That should give us pause.
Also we’ve already seen Anthropic and Open AI back down on their safety docs red-lines, “knowing” doesn’t mean “slowing”. Its entirely possible all of the evals come in, the knowledge is there and everyone just plows on.
I would be like 51% sure on this (so barely at all), but at the very least its daft to automatically think that safety work now is necessarily a good thing for the world long term. There’s a lot of complexity there. I think there are strong arguments for and against near-term safety.