Concerns over AI safety and calls for government control over the technology are highly correlated but they should not be.
There are two major forms of AI risk: misuse and misalignment. Misuse risks come from humans using AIs as tools in dangerous ways. Misalignment risks arise if AIs take their own actions at the expense of human interests.
Governments are poor stewards for both types of risk. Misuse regulation is like the regulation of any other technology. There are reasonable rules that the government might set, but omission bias and incentives to protect small but well organized groups at the expense of everyone else will lead to lots of costly ones too. Misalignment regulation is not in the Overton window for any government. Governments do not have strong incentives to care about long term, global, costs or benefits and they do have strong incentives to push the development of AI forwards for their own purposes.
Noticing that AI companies put the world at risk is not enough to support greater government involvement in the technology. Government involvement is likely to exacerbate the most dangerous parts of AI while limiting the upside.
Default government incentives
Governments are not social welfare maximizers. Government actions are an amalgam of the actions of thousands of personal welfare maximizers who are loosely aligned and constrained. In general, governments have strong incentives for myopia, violent competition with other governments, and negative sum transfers to small, well organized groups. These exacerbate existential risk and limit potential upside.
The vast majority of the costs of existential risk occur outside of the borders of any single government and beyond the election cycle for any current decision maker, so we should expect governments to ignore them.
We see this expectation fulfilled in governments reactions to other long term or global externalities e.g debt and climate change. Governments around the world are happy to impose trillions of dollars in direct cost and substantial default risk on future generations because costs and benefits on these future generations hold little sway in the next election. Similarly, governments spend billions subsidizing fossil fuel production and ignore potential solutions to global warming, like a carbon tax or geoengineering, because the long term or extraterritorial costs and benefits of climate change do not enter their optimization function.
AI risk is no different. Governments will happily trade off global, long term risk for national, short term benefits. The most salient way they will do this is through military competition. Government regulations on private AI development will not stop them from racing to integrate AI into their militaries. Autonomous drone warfare is already happening in Ukraine and Israel. The US military has contracts with Palantir and Andruil which use AI to augment military strategy or to power weapons systems. Governments will want to use AI for predictive policing, propaganda, and other forms of population control.
The case of nuclear tech is informative. This technology was strictly regulated by governments, but they still raced with each other and used the technology to create the most existentially risky weapons mankind has ever seen. Simultaneously, they cracked down on civilian use. Now, we’re in a world where all the major geopolitical flashpoints have at least one side armed with nuclear weapons and where the nuclear power industry is worse than stagnant.
Government’s military ambitions mean that their regulation will preserve the most dangerous misuse risks from AI. They will also push the AI frontier and train larger models, so we will still face misalignment risks. These may be exacerbated if governments are less interested or skilled in AI safety techniques. Government control over AI development is likely to slow down AI progress overall. Accepting the premise that this is good is not sufficient to invite regulation, though, because government control will cause a relative speed up of the most dystopian uses for AI.
In short term, governments are primarily interested in protecting well-organized groups from the effects of AI. E.g copyright holders, drivers unions, and other professional lobby groups. Here’s a summary of last years congressional AI hearing from Zvi Mowshowitz.
The Senators care deeply about the types of things politicians care deeply about. Klobuchar asked about securing royalties for local news media. Blackburn asked about securing royalties for Garth Brooks. Lots of concern about copyright violations, about using data to train without proper permission, especially in audio models. Graham focused on section 230 for some reason, despite numerous reminders it didn’t apply, and Howley talked about it a bit too.
This kind of regulation has less risk than misaligned killbots, but it does limit the potential upside from the technology.
Private incentives for AI development are far from perfect. There are still large externalities and competitive dynamics that may push progress too fast. But identifying this problem is not enough to justify government involvement. We need a reason to believe that governments can reliably improve the incentives facing private organizations. Government’s strong incentives for myopia, military competition, and rent-seeking make it difficult to find such a reason.
Negative Spillovers
The default incentives of both governments and profit seeking companies are imperfect. But the whole point of AI safety advocacy is to change these incentives or to convince decision makers to act despite them, so you can buy that governments are imperfect and still support calls for AI regulation. The problem with this is that even extraordinarily successful advocacy in government can be redirected into opposite and catastrophic effects.
Consider Sam Altman’s testimony in congress last May. No one was convinced of anything except the power of AI fear for their own pet projects. Here is a characteristic quote:
Senator Blumenthal addressing Sam Altman: I think you have said, in fact, and I’m gonna quote, ‘Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.’ You may have had in mind the effect on jobs. Which is really my biggest nightmare in the long term.
A reasonable upper bound for the potential of AI safety lobbying is the environmental movement of the 1970s. It was extraordinarily effective. Their advocacy led to a series of laws, including the National Environmental Policy Act (NEPA) that are among the most comprehensive and powerful regulations ever passed. These laws are not clearly in service of some pre-existing government incentive. Indeed, they regulate the federal government more strictly than anything else and often got it in its way. The cultural and political advocacy of the environmental movement made a large counterfactual impact with laws that still have massive influence today.
This success has turned sour, though, because the massive influence of these laws is now a massive barrier to decarbonization. NEPA has exemptions for oil and gas but not for solar or windfarms. Exemptions for highways but not highspeed rail. The costs of compliance with NEPA’s bureaucratic proceduralism hurts Terraform Industries a lot more than Shell. The standard government incentives for concentrating benefits to large legible groups and diffusing costs to large groups and the future redirected the political will and institutional power of the environmental movement into some of the most environmentally damaging and economically costly laws ever.
AI safety advocates should not expect to do much better than this, especially since many of their proposals are specifically based on permitting AI models like NEPA permits construction projects.
Belief in the potential for existential risk from AI does not imply that governments should have greater influence over its development. Government’s incentives make them misaligned with the goal of reducing existential risk. They are not rewarded or punished for costs or benefits outside of their borders or term limits and this is where nearly all of the importance of existential risk lies. Governments are rewarded for rapid development of military technology that empowers them over their rivals. They are also rewarded for providing immediate benefits to well-organized, legible groups, even when these rewards come at great expense to larger or more remote groups of people. These incentives exacerbate the worst misuse and misalignment risks of AI and limit the potential economic upside.
I think this is too pessimistic: why did one of Biden’s cabinet ask for Christiano in one of the top positions at the US gov’s AI safety org if the government will reliably prioritize the sort of factors you cite here to the exclusion of safety?: https://www.nist.gov/news-events/news/2024/04/us-commerce-secretary-gina-raimondo-announces-expansion-us-ai-safety
I also think that whether or not the government regulates private AI has little to do with whether it militarizes AI. It’s not like there is one dial with “amount of government” and it just gets turned up or down. Government can do very little to restrict what Open AI/DeepMind/Anthropic do, but then also spend lots and lots of money on military AI projects. So worries about militarization are not really a reason not to want the government to restrict Open AI/DeepMind/Anthropic.
Not to mention that insofar as the basic science here is getting done for commercial reasons, any regulations which slow down the commercial development of frontier modes will actually slow down the progress of AI for military applications too, whether or not that is what the US gov intends, and regardless of whether those regulations are intended to reduce X-risk, or to protect the jobs of voice actors in cartoons facing AI replacement.
This post correctly identifies some of the major obstacles to governing AI, but ultimately makes an argument for “by default, governments will not regulate AI well,” rather than the claim implied by its title, which is that advocating for (specific) AI regulations is net negative—a type of fallacious conflation I recognize all too well from my own libertarian past.
I do make the “by default” claim but I also give reasons why advocating for specific regulations can backfire. E.g the environmentalist success with NEPA. Environmentalists had huge success in getting the specific legal powers and constraints on govt that they asked for but those have been repurposed in service of default govt incentives. Also, advocacy for a specific set of regulations has spillovers onto others. When AI safety advocates make the case for fearing AI progress they provide support for a wide range of responses to AI including lots of nonsensical ones.
Yes, some regulations backfire, and this is a good flag to keep in mind when designing policy, but to actually make the reference-class argument here work, you’d have to show that this is what we should expect from AI policy, which would include showing that failures like NEPA are either much more relevant for the AI case or more numerous than other, more successful regulations, like (in my opinion) the Clean Air Act, Sarbanes-Oxley, bans on CFCs or leaded gasoline, etc. I know it’s not quite as simple as “I would simply design good regulations instead of bad ones,” but it’s also not as simple as “some regulations are really counterproductive, so you shouldn’t advocate for any.” Among other things, this assumes that nobody else will be pushing for really counterproductive regulations!
I think you’ve failed to think on the margin here. I agree that the broad classes of regulation you point to here have *netted out* badly, but this says little about what the most thoughtful and determined actors in these spaces have achieved.
Classically, Germany’s early 2000s investments in solar R&D had enormous positive externalities on climate and the people who pushed for those didn’t have to support restricting nuclear power also. The option space for them was not “the net-bad energy policy that emerged” vs “libertarian paradise;” it was: “the existing/inevitable bad policies with a bet on solar R&D” vs “the existing/inevitable bad policies with no bet on solar R&D.”
I believe most EAs treat their engagement with AI policy as researching and advocating for narrow policies tailored to mitigate catastrophic risk. In this sense, they’re acting as an organized/expert interest group motivated by a good, even popular per some polls, view of the public interest. They are competing with rather than complementing the more selfishly motivated interest groups seeking the kind of influence the oil & gas industry did in the climate context. On your model of regulation, this seems like a wise strategy, perhaps the only viable one. Again the alternative is not no regulation, but regulation that leaves out the best, most prosocial ideas.
To the extent you’re trying to warn EAs not to indiscriminately cheer any AI policy proposal assuming it will help with x-risk, I agree with you. I don’t however agree that’s reflective of how they’re treating the issue.
Yes that’s fair. I do think that even specific advocacy can have risks though. Most advocacy is motivated by AI fear which can be picked up and used to support lots of other bad policies, e.g how Sam Altaman was received in congress.
FWIW I work at the AI Safety Institute UK and we’re considering a range of both misuse and misalignment threats, and there are a lot of smart folks on board taking things pretty seriously. I admit I… don’t fully understand how we ended up in this situation and it feels contingent and precious, as does the tentative international consensus on the value of cooperation on safety (e.g. the Bletchley declaration). Some people in government are quite good, actually!
Executive summary: Government regulation of AI is likely to exacerbate the risks of AI misuse and misalignment while limiting the potential benefits, due to governments’ incentives for myopia, military competition, and protecting special interests.
Key points:
AI risks come in two forms: misuse by humans and misalignment of AI systems with human interests.
Governments have poor incentives to mitigate long-term, global risks and strong incentives to use AI for military advantage and domestic control.
Government regulation is likely to preserve the most dangerous misuse risks, potentially exacerbate misalignment risks, and slow down beneficial AI progress.
Even successful AI safety advocacy can be redirected by government incentives, as seen with environmental regulations now hindering decarbonization efforts.
Private incentives for AI development, while imperfect, are better aligned with reducing existential risk than government incentives.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.