You want to give a regulator the power to decide which large training runs are safe. I think this policy’s effects depend tremendously on the regulator—if it’s great at distinguishing safe stuff from dangerous stuff and it makes great choices, the policy is great; if not, it’s not. I feel pretty uncertain about how good it would be, and I suspect some disagreements about this policy are actually disagreements about how good the regulator would be. It feels hard to evaluate a proposal that leaves so much up to the regulator.
(Maybe it would help to have a concrete illustrative line to help readers get a sense of what you think the regulator would ideally do, like “LLMs and bio stuff with training compute > 1e24 FLOP are banned, everything else is not.” Ideally the regulator would be more sophisticated than that, of course.)
This is a good insight—I definitely feel like lack of trust (due partly to uncertainty) in the proposed regulator is a big blocker for me feeling at all on board with pause/regulation more broadly. Especially relevant given that I think the original CAIP proposals missed the mark by some margin. I acknowledge that Thomas is writing in his personal capacity, but I think that the link is still relevant.
Their original criteria for “frontier AI” was very broad, but an expansive definition makes sense if you think the regulator will be great—you give it lots of discretion to reject unsafe stuff but it can quickly approve safe stuff. I think disagreements about CAIP’s central proposal come down to different intuitions about how good the regulator would be—I think Thomas thinks the regulator would quickly approve almost all clearly-safe stuff, so an expansive scope does little harm.
Yeah, this sounds right to me. At present I feel like a regulator would end up massively overrepresenting at least one of (a) the EA community and (b) large tech corporations with pretty obviously bad incentives.
Hmm, I don’t see what goes wrong if the regulator overrepresents EA. And overrepresenting the major labs is suboptimal but I’d guess it’s better than no regulation—it decreases multipolarity among labs and (insofar as major labs are relatively safe and want to require others to be safe) improves safety directly.
A regulator overrepresenting EA seems bad to me (not an EA) because:
I don’t agree with a lot of the beliefs of the EA community on this subject and so I’d expect an EA-dominated regulator to take actions I don’t approve of.
Dominance by a specific group makes legitimacy much harder.
The EA community is pretty strongly intertwined with the big labs so most of the concerns from there carry over.
I don’t expect (1) to be particularly persuasive for you but maybe (2) and (3) are. I find some of the points in Ways I Expect AI Regulation To Increase X-Risk relevant to issues with overrepresentation of big labs. I think the overrepresentation of big labs would lead to a squashing of open-source, for instance, which I think is currently beneficial and would remain beneficial on the margin for a while.
More generally, I don’t particularly like the flattening of specific disagreements on matters of fact (and thus subsequent actions) to “wants people to be safe”/”doesn’t want people to be safe”. I expect that most people who disagree about the right course of action aren’t doing so out of some weird desire to see people harmed/replaced by AI (I’m certainly not) and it seems a pretty unfair dismissal.
Re “want to require others to be safe”—that was poorly worded, I meant wants to require everyone to follow specific safety practices they already follow, possibly to slow competitors in addition to safety reasons.
Cool, apologies if that came across a bit snarky (on rereading it does to me). I think this was instance N+1 of this phrasing and I’d gotten a bit annoyed by instances 1 through N which you obviously bear no responsibility for! I’m happy to have pushed back on the phrasing but hope I didn’t cause offence.
A more principled version of (1) would be to appeal to moral uncertainty, or to the idea that a regulator should represent all the stakeholders and I worry than an EA-dominated regulator would fail to do so.
Naively I would trade a lot of clearly-safe stuff being delayed or temporarily prohibited for even a minor decrease in chance of safe-seeming-but-actually-dangerous stuff going through, which pushes me towards favoring a more expansive scope of regulation.
(in my mind the potential loss of decades of life improvements currently pale vs potential non-existence of all lives in the longterm future)
Don’t know how to think about it when accounting for public opinion though, I expect a larger scope will gather more opposition to regulation, which could be detrimental in various ways, the most obvious being decreased likelihood of such regulation being passed/upheld/disseminated to other places.
Good post.
You want to give a regulator the power to decide which large training runs are safe. I think this policy’s effects depend tremendously on the regulator—if it’s great at distinguishing safe stuff from dangerous stuff and it makes great choices, the policy is great; if not, it’s not. I feel pretty uncertain about how good it would be, and I suspect some disagreements about this policy are actually disagreements about how good the regulator would be. It feels hard to evaluate a proposal that leaves so much up to the regulator.
(Maybe it would help to have a concrete illustrative line to help readers get a sense of what you think the regulator would ideally do, like “LLMs and bio stuff with training compute > 1e24 FLOP are banned, everything else is not.” Ideally the regulator would be more sophisticated than that, of course.)
This is a good insight—I definitely feel like lack of trust (due partly to uncertainty) in the proposed regulator is a big blocker for me feeling at all on board with pause/regulation more broadly. Especially relevant given that I think the original CAIP proposals missed the mark by some margin. I acknowledge that Thomas is writing in his personal capacity, but I think that the link is still relevant.
Their original criteria for “frontier AI” was very broad, but an expansive definition makes sense if you think the regulator will be great—you give it lots of discretion to reject unsafe stuff but it can quickly approve safe stuff. I think disagreements about CAIP’s central proposal come down to different intuitions about how good the regulator would be—I think Thomas thinks the regulator would quickly approve almost all clearly-safe stuff, so an expansive scope does little harm.
Yeah, this sounds right to me. At present I feel like a regulator would end up massively overrepresenting at least one of (a) the EA community and (b) large tech corporations with pretty obviously bad incentives.
Hmm, I don’t see what goes wrong if the regulator overrepresents EA. And overrepresenting the major labs is suboptimal but I’d guess it’s better than no regulation—it decreases multipolarity among labs and (insofar as major labs are relatively safe and want to require others to be safe) improves safety directly.
A regulator overrepresenting EA seems bad to me (not an EA) because:
I don’t agree with a lot of the beliefs of the EA community on this subject and so I’d expect an EA-dominated regulator to take actions I don’t approve of.
Dominance by a specific group makes legitimacy much harder.
The EA community is pretty strongly intertwined with the big labs so most of the concerns from there carry over.
I don’t expect (1) to be particularly persuasive for you but maybe (2) and (3) are. I find some of the points in Ways I Expect AI Regulation To Increase X-Risk relevant to issues with overrepresentation of big labs. I think the overrepresentation of big labs would lead to a squashing of open-source, for instance, which I think is currently beneficial and would remain beneficial on the margin for a while.
More generally, I don’t particularly like the flattening of specific disagreements on matters of fact (and thus subsequent actions) to “wants people to be safe”/”doesn’t want people to be safe”. I expect that most people who disagree about the right course of action aren’t doing so out of some weird desire to see people harmed/replaced by AI (I’m certainly not) and it seems a pretty unfair dismissal.
OK.
Re “want to require others to be safe”—that was poorly worded, I meant wants to require everyone to follow specific safety practices they already follow, possibly to slow competitors in addition to safety reasons.
Cool, apologies if that came across a bit snarky (on rereading it does to me). I think this was instance N+1 of this phrasing and I’d gotten a bit annoyed by instances 1 through N which you obviously bear no responsibility for! I’m happy to have pushed back on the phrasing but hope I didn’t cause offence.
A more principled version of (1) would be to appeal to moral uncertainty, or to the idea that a regulator should represent all the stakeholders and I worry than an EA-dominated regulator would fail to do so.
Naively I would trade a lot of clearly-safe stuff being delayed or temporarily prohibited for even a minor decrease in chance of safe-seeming-but-actually-dangerous stuff going through, which pushes me towards favoring a more expansive scope of regulation.
(in my mind the potential loss of decades of life improvements currently pale vs potential non-existence of all lives in the longterm future)
Don’t know how to think about it when accounting for public opinion though, I expect a larger scope will gather more opposition to regulation, which could be detrimental in various ways, the most obvious being decreased likelihood of such regulation being passed/upheld/disseminated to other places.