[ETA: I’m worried this comment is being misinterpreted. I’m not saying we should have no regulation. I’m challenging the point about where the burden of proof lies for showing whether a new technology is harmful.]
My broad goal for AI Safety advocacy is to shift the burden of proof to its rightful place– onto AI companies to prove their product is safe–, rather than where it currently seems to be– on the rest of us to prove that AGI is potentially dangerous. [...]
The below poll from AI Policy Institute and YouGov (release 8/11/23) shows comfortable majorities among US adults on questions about AI x-risk (76% worry about extinction risks from machine intelligence), slowing AI (82% say we should go slowly and deliberately), and government regulation of the AI industry (82% say tech executives can’t be trusted to self-regulate).
Can you speak a little more about why you think this is the “rightful place” of the burden of proof? When I think back to virtually every new technology in human history, I don’t think the burden of proof was generally considered to be on the inventors to prove that their technology was safe before developing it.
In the vast majority of cases, the way we’ve dealt with technologies in the past is by allowing essentially laissez faire for inventors at first. Then, for many technologies, after they’ve been adopted by a substantial fraction of the population for a while, and we’ve empirically observed the dangers, we place controls on who can produce, sell, and use the technology. For example, we did that with DDT, PCBs, leaded gasoline, and asbestos.
There might be good reasons why we don’t want to deal with AI this way, but I generally still think the burden of proof is on other people to show why AI is different than other technologies, rather than being on developers to prove that AI can or will be developed safely.
For particular classes of technologies, like food and medicine, our society thinks that it’s too risky to allow companies to sell completely new goods without explicit approval. That’s perhaps more in line with what you’re proposing for AI. But it’s worth noting that the FDA only started requiring proof of safety in 1938. Our current regime in which we require that companies prove that their products are safe before they are allowed to sell them is a distinctly modern and recent phenomenon, rather than some universal norm in human societies.
Moreover, I can’t think of a single example in which we’ve waited for democratic approval for new technologies. My guess is that, if we had done that in the past, then many essential modern day technologies would have never been developed. In a survey covering 20 countries around the world, in almost every single nation, there are more people who say that GM foods are “unsafe” than people who say that GM foods are “safe”:
Majorities in places such as Russia (70%), Italy (62%), India (58%) and South Korea (57%) view GM foods as generally unsafe to eat. The balance of opinion tilts negative even in places where sizable shares say they don’t know enough about GM foods to offer a view. For example, 47% of Spaniards say GM foods are unsafe, while just 13% say they are safe to eat. Australia is the only place surveyed where at least as many view GM foods as safe as view them to be unsafe (31% to 31%).
This is despite the fact that GM foods have been studied for decades now, with a strong scientific consensus about their safety. On matters of safety, the general public is typically uninformed and frequently misinformed. I think it is correct to rely far more on credible assessments of safety from experts than public opinion.
Just as a partial reply, it seems weird to me to claim that the groups both best able to demonstrate safety and most technically capable of doing so—the groups making the systems—should get a free pass to tell other people to prove what they are doing is unsafe. That’s a really bad incentive.
And I think basically everywhere in the western world, for the past half century or so, we require manufacturers and designers to ensure their products are safe, implicitly or explicitly. Houses, bridges, consumer electronics, and children’s toys all get certified for safety. Hell, we even license engineers in most countries and make it illegal for non-licensed engineers to do things like certify building safety. That isn’t a democratic control, but it’s clearly putting the burden of proof on the makers, not those claiming it might be unsafe.
And I think basically everywhere in the western world, for the past half century or so, we require manufacturers and designers to ensure their products are safe, implicitly or explicitly. Houses, bridges, consumer electronics, and children’s toys all get certified for safety.
Sure, there are regulations on manufacturing products. But these regulations are generally based on decades of experience with the technologies, and were only put in place after people started to see harm. They weren’t conceived a priori before the technologies had any sizable impact.
(But if we had decades of experience with computer-based systems not reliably doing exactly what we wanted, you’d admit that this degree of caution on systems we expect to be powerful would be reasonable?)
That’s not how modern risk assessment works. Risk registers and mitigation planning are based on proactively identifying risk. To the extent that this doesn’t occur before something is built and/or deployed, at the very least, it’s a failure of the engineering process. (It also seems somewhat perverse to argue that we need to protect innovation in a specific domain by sticking to the way regulation happened long in the past.)
And in the cases where engineering and scientific analysis has identified risks in advance, but no regulatory system is in place, the legal system has been clear that there is liability on the part of the producers. And given those widely acknowledged dangers, it seems clear that if model developers ignores a known or obvious risk, they are criminally liable for negligence. This isn’t the same as restricting by-default-unsafe technologies like drugs and buildings, but at the very least, I think you should agree that one needs to make an argument for why ML models should be treated differently than other technologies with widely acknowledged dangers.
The frame of “burden of proof is on you to show AI is different from other technologies” is bizarre to me.
It’s a bit like if we’re talking about transporting live stock and someone is like “prove transporting dragons is differently than other livestock”. They’re massive, can fly, can breath fire and in many stories are very intelligent.
Where’s this assumption of sameness coming from? And how do you miss all the differences?
Similarly, I don’t know how you can look at AI and think “just another technology”.
AI can invent other technologies, provide strategical advice, act autonomously, self-replicate, ect. It feels like the default should very much be that it needs its own analysis.
It’s a bit like if we’re talking about transporting live stock and someone is like “prove transporting dragons is differently than other livestock. They’re massive, can fly, can breath fire and in many stories are very intelligent.
Those facts provide a reasonable basis for why we should treat dragons differently than livestock when transporting them. I don’t think that is really a shifting of the burden of proof, but rather an argument that dragons have met the burden of proof. Do we see that with AI yet? Perhaps. But I think so far most of the arguments for AI risks have been abstract and rely heavily upon theoretical evidence rather than concrete foreseeable harms. I think this type of argument is notoriously unreliable.
I’m also not saying “AI is the same”. I’m saying “We shouldn’t just assume AI is different a priori”.
AI can invent other technologies, provide strategical advice, act autonomously, self-replicate, ect. It feels like the default should very much be that it needs its own analysis.
I agree that AI will eventually be able to do those things, and so we should probably regulate it pretty heavily eventually. But a “pause” would probably include stopping a bunch of harmless AI products too. For example, a lot of people want to stop GPT-5. I’m skeptical, as a practical matter, that OpenAI should have to prove to us that GPT-5 will be safe before releasing it. I think we should probably instead wait until the concrete harms from AI become clearer before controlling it heavily.
My point regarding burden of proof is that something has gone wrong if you think dragons are in the same reference class as pigs, cows, even lions in terms of transportation challenges. And the fact that someone needs to ask for an explicit list is indicative of a mistake somewhere.
I’m not saying that you can’t argue that they are the same. Just that a more reasonable framing would then be more along the lines of, “here’s my surprising conclusion that we can regulate it the same way”.
That appears to assume the conclusion (“AI is dangerous”) to explain why the burden of proof is on the inventors to prove AI is safe. But I’m asking about where the burden of proof should lie prior to us already having the answer! If we had used this burden for every prior technology, it’s likely that a giant amount of innovation would have been stifled.
Now, you could alternatively think that the burden of proof is on other people to show that a new technology is dangerous, and this burden has already been met for AI. But I think that’s a different claim. I was responding to the idea that the burden of proof is on AI developers to prove that their product is safe.
We might be talking past each other. I think the burden of proof that AI-in-principle could be dangerous is on non-inventors, and that has already been met[1]. I think the burden of proof that specific-AI-tech-in-practice is safe should then be on AI manufacturers.
Similarly, if we know no details about nuclear power plants or nuclear weapons, the burden of proof about how scary an abstract “power plant” or “taking some ore from the ground and refining it” should be on concerned people. But after the theoretical case for nuclear scariness is demonstrated, we shouldn’t have had to wait until Hiroshima or Nagasaki, or even the Trinity Test, before the burden of proof falls on nuclear weapons manufacturers/states to demonstrate that their potential for accident is low.
I think the burden of proof that AI-in-principle could be dangerous is on non-inventors, and that has already been met. I think the burden of proof that specific-AI-tech-in-practice is safe should then be on AI manufacturers.
I think that makes sense. But I also think that the idea of asking for a “pause” looks a lot more like asking AI developers to prove that AI in the abstract can be safe, whereas an “FDA for new AIs” looks more like asking developers to prove their specific implementations are safe. The distinction between the two ideas here is blurry though, admittedly.
But insofar as this distinction makes sense, I think it should likely push us against a generic pause, and in favor of specific targeted regulations.
Nuclear chain reactions leading to massive explosions are dangerous. We don’t have separate prohibition treaties on each specific model of nuke.
Impenetrable multi-trillion-parameter neural networks are dangerous. I think it does make sense for AI developers to prove that AI (as per the current foundation model neural network paradigm) in the abstract can be safe.
“I mean a global, indefinite moratorium on the development of frontier models until it is safe to proceed.”
I think this is distinctly different from what you claim. In any actual system implementing a pause, the model developer is free “to prove their specific implementations are safe,” and go ahead. The question is what the default is—and you’ve implied elsewhere in this thread that you think that developers should be treated like pre-1938 drug manufacturers, with no rules.
If what you’re proposing is instead that there needs to be a regulatory body like the FDA to administer the rules and review cases when a company claims to have sufficient evidence of safety when planning to create a model, with a default rule that it’s illegal to develop the model until reviewed, instead of a ban with a review for exceptions when a company claims to have sufficient evidence of safety when planning to create a model, I think the gap between our positions is less blurry than it is primarily semantic.
you’ve implied elsewhere in this thread that you think that developers should be treated like pre-1938 drug manufacturers, with no rules.
I think you misread me. I’ve said across multiple comments that I favor targeted regulations that are based on foreseeable harms after we’ve gotten more acquainted with the technology. I don’t think that’s very similar to an indefinite pause, and it certainly isn’t the same as “no rules”.
That makes sense—I was confused, since you said different things, and some of them were subjunctive, and some were speaking about why you disagree with proposed analogies.
Given your perspective, is loss-of-control from more capable and larger models not a foreseeable harm? If we see a single example of this, and we manage to shut it down, would you then be in favor of a regulate-before-training approach?
[ETA: I’m worried this comment is being misinterpreted. I’m not saying we should have no regulation. I’m challenging the point about where the burden of proof lies for showing whether a new technology is harmful.]
Can you speak a little more about why you think this is the “rightful place” of the burden of proof? When I think back to virtually every new technology in human history, I don’t think the burden of proof was generally considered to be on the inventors to prove that their technology was safe before developing it.
In the vast majority of cases, the way we’ve dealt with technologies in the past is by allowing essentially laissez faire for inventors at first. Then, for many technologies, after they’ve been adopted by a substantial fraction of the population for a while, and we’ve empirically observed the dangers, we place controls on who can produce, sell, and use the technology. For example, we did that with DDT, PCBs, leaded gasoline, and asbestos.
There might be good reasons why we don’t want to deal with AI this way, but I generally still think the burden of proof is on other people to show why AI is different than other technologies, rather than being on developers to prove that AI can or will be developed safely.
For particular classes of technologies, like food and medicine, our society thinks that it’s too risky to allow companies to sell completely new goods without explicit approval. That’s perhaps more in line with what you’re proposing for AI. But it’s worth noting that the FDA only started requiring proof of safety in 1938. Our current regime in which we require that companies prove that their products are safe before they are allowed to sell them is a distinctly modern and recent phenomenon, rather than some universal norm in human societies.
Moreover, I can’t think of a single example in which we’ve waited for democratic approval for new technologies. My guess is that, if we had done that in the past, then many essential modern day technologies would have never been developed. In a survey covering 20 countries around the world, in almost every single nation, there are more people who say that GM foods are “unsafe” than people who say that GM foods are “safe”:
This is despite the fact that GM foods have been studied for decades now, with a strong scientific consensus about their safety. On matters of safety, the general public is typically uninformed and frequently misinformed. I think it is correct to rely far more on credible assessments of safety from experts than public opinion.
Just as a partial reply, it seems weird to me to claim that the groups both best able to demonstrate safety and most technically capable of doing so—the groups making the systems—should get a free pass to tell other people to prove what they are doing is unsafe. That’s a really bad incentive.
And I think basically everywhere in the western world, for the past half century or so, we require manufacturers and designers to ensure their products are safe, implicitly or explicitly. Houses, bridges, consumer electronics, and children’s toys all get certified for safety. Hell, we even license engineers in most countries and make it illegal for non-licensed engineers to do things like certify building safety. That isn’t a democratic control, but it’s clearly putting the burden of proof on the makers, not those claiming it might be unsafe.
Sure, there are regulations on manufacturing products. But these regulations are generally based on decades of experience with the technologies, and were only put in place after people started to see harm. They weren’t conceived a priori before the technologies had any sizable impact.
...and this risk isn’t predictable on priors?
(But if we had decades of experience with computer-based systems not reliably doing exactly what we wanted, you’d admit that this degree of caution on systems we expect to be powerful would be reasonable?)
That’s not how modern risk assessment works. Risk registers and mitigation planning are based on proactively identifying risk. To the extent that this doesn’t occur before something is built and/or deployed, at the very least, it’s a failure of the engineering process. (It also seems somewhat perverse to argue that we need to protect innovation in a specific domain by sticking to the way regulation happened long in the past.)
And in the cases where engineering and scientific analysis has identified risks in advance, but no regulatory system is in place, the legal system has been clear that there is liability on the part of the producers. And given those widely acknowledged dangers, it seems clear that if model developers ignores a known or obvious risk, they are criminally liable for negligence. This isn’t the same as restricting by-default-unsafe technologies like drugs and buildings, but at the very least, I think you should agree that one needs to make an argument for why ML models should be treated differently than other technologies with widely acknowledged dangers.
The frame of “burden of proof is on you to show AI is different from other technologies” is bizarre to me.
It’s a bit like if we’re talking about transporting live stock and someone is like “prove transporting dragons is differently than other livestock”. They’re massive, can fly, can breath fire and in many stories are very intelligent.
Where’s this assumption of sameness coming from? And how do you miss all the differences?
Similarly, I don’t know how you can look at AI and think “just another technology”.
AI can invent other technologies, provide strategical advice, act autonomously, self-replicate, ect. It feels like the default should very much be that it needs its own analysis.
I call this the technology bucket error
Those facts provide a reasonable basis for why we should treat dragons differently than livestock when transporting them. I don’t think that is really a shifting of the burden of proof, but rather an argument that dragons have met the burden of proof. Do we see that with AI yet? Perhaps. But I think so far most of the arguments for AI risks have been abstract and rely heavily upon theoretical evidence rather than concrete foreseeable harms. I think this type of argument is notoriously unreliable.
I’m also not saying “AI is the same”. I’m saying “We shouldn’t just assume AI is different a priori”.
I agree that AI will eventually be able to do those things, and so we should probably regulate it pretty heavily eventually. But a “pause” would probably include stopping a bunch of harmless AI products too. For example, a lot of people want to stop GPT-5. I’m skeptical, as a practical matter, that OpenAI should have to prove to us that GPT-5 will be safe before releasing it. I think we should probably instead wait until the concrete harms from AI become clearer before controlling it heavily.
My point regarding burden of proof is that something has gone wrong if you think dragons are in the same reference class as pigs, cows, even lions in terms of transportation challenges. And the fact that someone needs to ask for an explicit list is indicative of a mistake somewhere.
I’m not saying that you can’t argue that they are the same. Just that a more reasonable framing would then be more along the lines of, “here’s my surprising conclusion that we can regulate it the same way”.
Almost no past technology have a claim to be a non-Pascalian existential risk to humanity; especially one that’s known in advance.
The only exceptions I could think of are nukes, certain forms of bioweapons research, and maybe CFCs in refrigeration.
That appears to assume the conclusion (“AI is dangerous”) to explain why the burden of proof is on the inventors to prove AI is safe. But I’m asking about where the burden of proof should lie prior to us already having the answer! If we had used this burden for every prior technology, it’s likely that a giant amount of innovation would have been stifled.
Now, you could alternatively think that the burden of proof is on other people to show that a new technology is dangerous, and this burden has already been met for AI. But I think that’s a different claim. I was responding to the idea that the burden of proof is on AI developers to prove that their product is safe.
We might be talking past each other. I think the burden of proof that AI-in-principle could be dangerous is on non-inventors, and that has already been met[1]. I think the burden of proof that specific-AI-tech-in-practice is safe should then be on AI manufacturers.
Similarly, if we know no details about nuclear power plants or nuclear weapons, the burden of proof about how scary an abstract “power plant” or “taking some ore from the ground and refining it” should be on concerned people. But after the theoretical case for nuclear scariness is demonstrated, we shouldn’t have had to wait until Hiroshima or Nagasaki, or even the Trinity Test, before the burden of proof falls on nuclear weapons manufacturers/states to demonstrate that their potential for accident is low.
As demonstrated by eg, surveys.
I think that makes sense. But I also think that the idea of asking for a “pause” looks a lot more like asking AI developers to prove that AI in the abstract can be safe, whereas an “FDA for new AIs” looks more like asking developers to prove their specific implementations are safe. The distinction between the two ideas here is blurry though, admittedly.
But insofar as this distinction makes sense, I think it should likely push us against a generic pause, and in favor of specific targeted regulations.
Nuclear chain reactions leading to massive explosions are dangerous. We don’t have separate prohibition treaties on each specific model of nuke.
Impenetrable multi-trillion-parameter neural networks are dangerous. I think it does make sense for AI developers to prove that AI (as per the current foundation model neural network paradigm) in the abstract can be safe.
I think this is distinctly different from what you claim. In any actual system implementing a pause, the model developer is free “to prove their specific implementations are safe,” and go ahead. The question is what the default is—and you’ve implied elsewhere in this thread that you think that developers should be treated like pre-1938 drug manufacturers, with no rules.
If what you’re proposing is instead that there needs to be a regulatory body like the FDA to administer the rules and review cases when a company claims to have sufficient evidence of safety when planning to create a model, with a default rule that it’s illegal to develop the model until reviewed, instead of a ban with a review for exceptions when a company claims to have sufficient evidence of safety when planning to create a model, I think the gap between our positions is less blurry than it is primarily semantic.
I think you misread me. I’ve said across multiple comments that I favor targeted regulations that are based on foreseeable harms after we’ve gotten more acquainted with the technology. I don’t think that’s very similar to an indefinite pause, and it certainly isn’t the same as “no rules”.
That makes sense—I was confused, since you said different things, and some of them were subjunctive, and some were speaking about why you disagree with proposed analogies.
Given your perspective, is loss-of-control from more capable and larger models not a foreseeable harm? If we see a single example of this, and we manage to shut it down, would you then be in favor of a regulate-before-training approach?