On December 8, EU policymakers announced an agreement on the AI Act. This post aims to briefly explain the context and implications for the governance of global catastrophic risks from advanced AI. My portfolio on Open Philanthropy’s AI Governance and Policy Team includes EU matters (among other jurisdictions), but I am not an expert on EU policy or politics and could be getting some things in this post wrong, so please feel free to correct it or add more context or opinions in the comments!
If you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via this short Google form.
Context
The AI Act was first introduced in April 2021, and for the last ~8 months, it has been in the “trilogue” stage. The EU Commission, which is roughly analogous to the executive branch (White House or 10 Downing Street), drafted the bill; then, the European Parliament (sort of like the U.S. House of Representatives, with seats assigned to each country by a population-based formula) and the Council of the EU (sort of like the pre-17th-Amendment U.S. Senate, with each country’s government getting one vote in a complicated voting system)[1] each submitted proposed revisions; then, representatives from each body negotiated to land on a final version (analogous to conference committees in the US Congress).
In my understanding, AI policy folks who are worried about catastrophic risk were hoping that the Act would include regulations on all sufficiently capable GPAI (general-purpose AI) systems, with no exemptions for open-source models (at least for the most important regulations from a safety perspective), and ideally additional restrictions on “very capable foundation models” (those above a certain compute threshold), an idea floated by some negotiators in October. In terms of the substance of the hoped-for regulations, my sense is that the main hope was that the legislation would give the newly-formed AI Office substantial leeway to require things like threat assessments/dangerous capabilities evaluations and cybersecurity measures, with a lot of the details to be figured out later by that Office and by standard-setting bodies like CEN-CENELEC’s JTC-21.
GPAI regulations appeared in danger of being excluded after Mistral, Aleph Alpha, and the national governments of France, Germany, and Italy objected to what they perceived as regulatory overreach and threatened to derail the Act in November. There was also some reporting that the Act would totally exempt open-source models from regulation.
What’s in it?
Sabrina Küspert, an AI policy expert working at the EU Commission, summarized the results on some of these questions in a thread on X:
The agreement does indeed include regulations on “general-purpose AI,” or GPAI.
There does appear to be a version of the “very capable foundation models” idea in the form of “GPAI models with systemic risks,” which are based on capabilities and “reach,” which I think means how widely deployed they are.
Küspert also says “no exemptions,” which I interpret to mean “no exemptions to the systemic-risk rules for open-source systems.” Other reporting suggests there are wide exemptions for open-source models, but the requirements kick back in if the models pose systemic risks. However, Yann LeCun is celebrating based on this part of a Washington Post article: “The legislation ultimately included restrictions for foundation models but gave broad exemptions to “open-source models,” which are developed using code that’s freely available for developers to alter for their own products and tools. The move could benefit open-source AI companies in Europe that lobbied against the law, including France’s Mistral and Germany’s Aleph Alpha, as well as Meta, which released the open-source model LLaMA.” So it’s currently unclear to me where the Act lands on this question, and I think a close review by someone with legal or deep EU policy expertise might help illuminate.
The Commission’s blog post says: “For very powerful models that could pose systemic risks, there will be additional binding obligations related to managing risks and monitoring serious incidents, performing model evaluation and adversarial testing. These new obligations will be operationalised through codes of practices developed by industry, the scientific community, civil society and other stakeholders together with the Commission.” (I’m guessing this means JTC-21 and similar, but if people with more European context can better read the tea leaves, let me know.)
Parliament’s announcement notes that GPAI systems and models will “have to adhere to transparency requirements” including “technical documentation, complying with EU copyright law and disseminating detailed summaries about the content used for training.” I think these transparency requirements are the main opportunity to develop strong requirements for evaluations.
Enforcement will be up to both national regulators and the new European AI Office, which, as the Commission post notes, will be “the first body globally that enforces binding rules on AI and is therefore expected to become an international reference point.” Companies that fail to comply with these rules face fines up to “35 million euro or 7% of global revenue,” whichever is higher. (Not sure whether this would mean 7% of e.g. Alphabet’s global revenue or DeepMind’s).
The Act also does what some people have called the obvious thing of requiring that AI-generated content be labeled as such in a machine-readable format, with fines for noncompliance. (Seems easy to do for video/audio, much harder for text, but at least requiring that AI chatbots notify users that they’re AI systems rather than humans would be a first step.)
This post focuses on the most relevant parts of the Act to frontier models and catastrophic risk, but most of the Act is focused on the application layer. It bans the use of AI for:
“biometric categorisation systems that use sensitive characteristics (e.g. political, religious, philosophical beliefs, sexual orientation, race);
untargeted scraping of facial images from the internet or CCTV footage to create facial recognition databases;
emotion recognition in the workplace and educational institutions;
social scoring based on social behaviour or personal characteristics;
AI systems that manipulate human behaviour to circumvent their free will;
AI used to exploit the vulnerabilities of people (due to their age, disability, social or economic situation).”
The Act will start being enforced at the end of “a transitional period,” which the NYT says will be 12-24 months. In the meantime, the Commission is launching the cleverly titled “AI Pact,” which seeks voluntary commitments to start implementing the Act’s requirements before the legal deadline. EU Commission president Ursula von der Leyen says “around 100 companies have already expressed their interest to join” the Pact.
How big of a deal is this?
A few takeaways for me so far:
Despite the frontier AI companies being American and English, the EU has what I’d describe as a moderate amount of leverage on AI companies by being a large market (~17% of global GDP in dollar terms).
If the regulations they impose aren’t terribly costly and make enough sense, companies will likely comply with them. If they’re poorly executed or super costly, companies may fight them in court (as they did with the GDPR) or pull out of the EU market. So, European regulators will have a not-tiny budget of cost imposition that they could spend wisely and get a decent amount of safety (but of course can’t unilaterally pause AI development).
This effect will be especially important insofar as the EU’s regulations apply to training, rather than applications: AI companies might release EU-compliant versions of chatbots within the EU while deploying non-EU-compliant versions elsewhere, but due to the costs of training foundation models, they’re unlikely to train entirely separate models.
The EU and this GovAI paper (summarized here) are both very fond of the “de jure” Brussels Effect, where other jurisdictions imitate EU regulation. (They contrast this with the “de facto” Brussels Effect, which includes the kind of direct effects in the previous bullet.) So far, I haven’t seen many signs of the US or UK imitating the EU, but it’s possible that China’s approach will be informed by the EU. Other countries with less leverage over frontier AI might also be influenced, but this is less of a big deal.
There’s also an effect that Markus Anderljung pointed out to me, which is that even with no imitation by policymakers, regulators themselves might be influenced by the object-level outputs of the AI Office: if Europe rules that a particular AI system was not sufficiently evaluated or secured before release, some regulators might defer, as many countries’ pharma regulators apparently defer to the FDA.
The EU adopting pretty strong regulations even after industry and their allies in government were seriously activated is a good sign for the politics of AI regulation in similar polities. (This is less true to the extent that even very powerful/risky/expensive-to-train models are exempt if their weights get released.)
The multilateral EU regulating like this is also a step towards international agreements; a robust international regime needs to include the US, China, EU, and UK at bare minimum, and the EU might be important connective tissue between a mutually distrustful China and US/UK.
Making the AI Act effective for catastrophic risk reduction
The Act appears to stake out a high-level approach to Europe’s AI policy, but will very likely task the AI Office, standard-setting organizations (SSOs) like JTC-21, and EU member states with fleshing out a lot of detail and implementing the policies. Depending on the standardization and implementation phases over the next few years, the Act could wind up strongly incentivizing AI developers to act more safely, or it could wind up insufficiently detailed, captured by industry, bogged down in legal challenges, or so onerous that AI companies withdraw from the EU market and ignore the law.
To achieve outcomes more like the former, people who would like to reduce global catastrophic risks from future AI systems could consider doing the following:
Joining the SSOs. These bodies tend to include a mix of industry lobbyists and civil society representatives, and the civil society folks have a huge range of priorities, so there are very few people at these who are focused on frontier model safety, and you could make a big difference by joining.
Working for the European AI Office or member state implementation bodies. By default, I think these offices (like most tech regulators) will have difficulty recruiting technically knowledgeable people. (The bar for “technically knowledgeable” in government tends to be pretty low; if you’re familiar with lots of the material on the AI Safety Fundamentals governance syllabus I think you’re in decent shape.) Having a few more such people in these offices could make them more informed about the risks and governance opportunities, both specifically regarding models with “systemic risks” and in general.
Policy research that aims to inform catastrophic-risk-focused people in these groups or other European institutions. Think tanks and research institutions tend to be on a spectrum from “policy/strategy development,” where they write reports about what policymakers should be aiming for, and advocacy, where they mostly take ideas developed by others and talk a lot with people in governments to turn them into policy outcomes. The people I’ve spoken with who are currently working closely with the EU (in the Commission, SSOs, or think tanks on the latter side of that spectrum) said that additional “upstream” (i.e. on the former side of the spectrum) policy work like that done by e.g. IAPS and GovAI would be really useful.
Generally, I think AI policy and governance folks should invest some time in understanding what’s going on in the EU (though some people might have strong comparative advantage reasons not to), and relatedly probably encourage catastrophic-risk-focused Europeans to try to do useful work in the EU rather than coming to the US. At the very least, in my view, the picture has changed in an EU-favoring direction in the last year (despite lots of progress in US AI policy), and this should prompt a re-evaluation of the conventional wisdom (in my understanding) that the US has enough leverage over AI development such that policy careers in DC are more impactful even for Europeans.
And once again: if you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via this short Google form.
EU policymakers reach an agreement on the AI Act
On December 8, EU policymakers announced an agreement on the AI Act. This post aims to briefly explain the context and implications for the governance of global catastrophic risks from advanced AI. My portfolio on Open Philanthropy’s AI Governance and Policy Team includes EU matters (among other jurisdictions), but I am not an expert on EU policy or politics and could be getting some things in this post wrong, so please feel free to correct it or add more context or opinions in the comments!
If you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via this short Google form.
Context
The AI Act was first introduced in April 2021, and for the last ~8 months, it has been in the “trilogue” stage. The EU Commission, which is roughly analogous to the executive branch (White House or 10 Downing Street), drafted the bill; then, the European Parliament (sort of like the U.S. House of Representatives, with seats assigned to each country by a population-based formula) and the Council of the EU (sort of like the pre-17th-Amendment U.S. Senate, with each country’s government getting one vote in a complicated voting system)[1] each submitted proposed revisions; then, representatives from each body negotiated to land on a final version (analogous to conference committees in the US Congress).
In my understanding, AI policy folks who are worried about catastrophic risk were hoping that the Act would include regulations on all sufficiently capable GPAI (general-purpose AI) systems, with no exemptions for open-source models (at least for the most important regulations from a safety perspective), and ideally additional restrictions on “very capable foundation models” (those above a certain compute threshold), an idea floated by some negotiators in October. In terms of the substance of the hoped-for regulations, my sense is that the main hope was that the legislation would give the newly-formed AI Office substantial leeway to require things like threat assessments/dangerous capabilities evaluations and cybersecurity measures, with a lot of the details to be figured out later by that Office and by standard-setting bodies like CEN-CENELEC’s JTC-21.
GPAI regulations appeared in danger of being excluded after Mistral, Aleph Alpha, and the national governments of France, Germany, and Italy objected to what they perceived as regulatory overreach and threatened to derail the Act in November. There was also some reporting that the Act would totally exempt open-source models from regulation.
What’s in it?
Sabrina Küspert, an AI policy expert working at the EU Commission, summarized the results on some of these questions in a thread on X:
The agreement does indeed include regulations on “general-purpose AI,” or GPAI.
There does appear to be a version of the “very capable foundation models” idea in the form of “GPAI models with systemic risks,” which are based on capabilities and “reach,” which I think means how widely deployed they are.
It looks like GPAI models are presumed to have these capabilities if they’re trained on 10^25 FLOP, which is one order of magnitude smaller than the October 30 Biden executive order’s cutoff for reporting requirements (and which would probably include GPT-4 and maybe Gemini, but no other current models as far as I know).
Küspert also says “no exemptions,” which I interpret to mean “no exemptions to the systemic-risk rules for open-source systems.” Other reporting suggests there are wide exemptions for open-source models, but the requirements kick back in if the models pose systemic risks. However, Yann LeCun is celebrating based on this part of a Washington Post article: “The legislation ultimately included restrictions for foundation models but gave broad exemptions to “open-source models,” which are developed using code that’s freely available for developers to alter for their own products and tools. The move could benefit open-source AI companies in Europe that lobbied against the law, including France’s Mistral and Germany’s Aleph Alpha, as well as Meta, which released the open-source model LLaMA.” So it’s currently unclear to me where the Act lands on this question, and I think a close review by someone with legal or deep EU policy expertise might help illuminate.
The Commission’s blog post says: “For very powerful models that could pose systemic risks, there will be additional binding obligations related to managing risks and monitoring serious incidents, performing model evaluation and adversarial testing. These new obligations will be operationalised through codes of practices developed by industry, the scientific community, civil society and other stakeholders together with the Commission.” (I’m guessing this means JTC-21 and similar, but if people with more European context can better read the tea leaves, let me know.)
Parliament’s announcement notes that GPAI systems and models will “have to adhere to transparency requirements” including “technical documentation, complying with EU copyright law and disseminating detailed summaries about the content used for training.” I think these transparency requirements are the main opportunity to develop strong requirements for evaluations.
Enforcement will be up to both national regulators and the new European AI Office, which, as the Commission post notes, will be “the first body globally that enforces binding rules on AI and is therefore expected to become an international reference point.” Companies that fail to comply with these rules face fines up to “35 million euro or 7% of global revenue,” whichever is higher. (Not sure whether this would mean 7% of e.g. Alphabet’s global revenue or DeepMind’s).
The Act also does what some people have called the obvious thing of requiring that AI-generated content be labeled as such in a machine-readable format, with fines for noncompliance. (Seems easy to do for video/audio, much harder for text, but at least requiring that AI chatbots notify users that they’re AI systems rather than humans would be a first step.)
This post focuses on the most relevant parts of the Act to frontier models and catastrophic risk, but most of the Act is focused on the application layer. It bans the use of AI for:
“biometric categorisation systems that use sensitive characteristics (e.g. political, religious, philosophical beliefs, sexual orientation, race);
untargeted scraping of facial images from the internet or CCTV footage to create facial recognition databases;
emotion recognition in the workplace and educational institutions;
social scoring based on social behaviour or personal characteristics;
AI systems that manipulate human behaviour to circumvent their free will;
AI used to exploit the vulnerabilities of people (due to their age, disability, social or economic situation).”
The Act will start being enforced at the end of “a transitional period,” which the NYT says will be 12-24 months. In the meantime, the Commission is launching the cleverly titled “AI Pact,” which seeks voluntary commitments to start implementing the Act’s requirements before the legal deadline. EU Commission president Ursula von der Leyen says “around 100 companies have already expressed their interest to join” the Pact.
How big of a deal is this?
A few takeaways for me so far:
Despite the frontier AI companies being American and English, the EU has what I’d describe as a moderate amount of leverage on AI companies by being a large market (~17% of global GDP in dollar terms).
If the regulations they impose aren’t terribly costly and make enough sense, companies will likely comply with them. If they’re poorly executed or super costly, companies may fight them in court (as they did with the GDPR) or pull out of the EU market. So, European regulators will have a not-tiny budget of cost imposition that they could spend wisely and get a decent amount of safety (but of course can’t unilaterally pause AI development).
This effect will be especially important insofar as the EU’s regulations apply to training, rather than applications: AI companies might release EU-compliant versions of chatbots within the EU while deploying non-EU-compliant versions elsewhere, but due to the costs of training foundation models, they’re unlikely to train entirely separate models.
The EU and this GovAI paper (summarized here) are both very fond of the “de jure” Brussels Effect, where other jurisdictions imitate EU regulation. (They contrast this with the “de facto” Brussels Effect, which includes the kind of direct effects in the previous bullet.) So far, I haven’t seen many signs of the US or UK imitating the EU, but it’s possible that China’s approach will be informed by the EU. Other countries with less leverage over frontier AI might also be influenced, but this is less of a big deal.
There’s also an effect that Markus Anderljung pointed out to me, which is that even with no imitation by policymakers, regulators themselves might be influenced by the object-level outputs of the AI Office: if Europe rules that a particular AI system was not sufficiently evaluated or secured before release, some regulators might defer, as many countries’ pharma regulators apparently defer to the FDA.
The EU adopting pretty strong regulations even after industry and their allies in government were seriously activated is a good sign for the politics of AI regulation in similar polities. (This is less true to the extent that even very powerful/risky/expensive-to-train models are exempt if their weights get released.)
The multilateral EU regulating like this is also a step towards international agreements; a robust international regime needs to include the US, China, EU, and UK at bare minimum, and the EU might be important connective tissue between a mutually distrustful China and US/UK.
Making the AI Act effective for catastrophic risk reduction
The Act appears to stake out a high-level approach to Europe’s AI policy, but will very likely task the AI Office, standard-setting organizations (SSOs) like JTC-21, and EU member states with fleshing out a lot of detail and implementing the policies. Depending on the standardization and implementation phases over the next few years, the Act could wind up strongly incentivizing AI developers to act more safely, or it could wind up insufficiently detailed, captured by industry, bogged down in legal challenges, or so onerous that AI companies withdraw from the EU market and ignore the law.
To achieve outcomes more like the former, people who would like to reduce global catastrophic risks from future AI systems could consider doing the following:
Joining the SSOs. These bodies tend to include a mix of industry lobbyists and civil society representatives, and the civil society folks have a huge range of priorities, so there are very few people at these who are focused on frontier model safety, and you could make a big difference by joining.
Working for the European AI Office or member state implementation bodies. By default, I think these offices (like most tech regulators) will have difficulty recruiting technically knowledgeable people. (The bar for “technically knowledgeable” in government tends to be pretty low; if you’re familiar with lots of the material on the AI Safety Fundamentals governance syllabus I think you’re in decent shape.) Having a few more such people in these offices could make them more informed about the risks and governance opportunities, both specifically regarding models with “systemic risks” and in general.
Policy research that aims to inform catastrophic-risk-focused people in these groups or other European institutions. Think tanks and research institutions tend to be on a spectrum from “policy/strategy development,” where they write reports about what policymakers should be aiming for, and advocacy, where they mostly take ideas developed by others and talk a lot with people in governments to turn them into policy outcomes. The people I’ve spoken with who are currently working closely with the EU (in the Commission, SSOs, or think tanks on the latter side of that spectrum) said that additional “upstream” (i.e. on the former side of the spectrum) policy work like that done by e.g. IAPS and GovAI would be really useful.
Generally, I think AI policy and governance folks should invest some time in understanding what’s going on in the EU (though some people might have strong comparative advantage reasons not to), and relatedly probably encourage catastrophic-risk-focused Europeans to try to do useful work in the EU rather than coming to the US. At the very least, in my view, the picture has changed in an EU-favoring direction in the last year (despite lots of progress in US AI policy), and this should prompt a re-evaluation of the conventional wisdom (in my understanding) that the US has enough leverage over AI development such that policy careers in DC are more impactful even for Europeans.
And once again: if you have useful skills, networks, or other resources that you might like to direct toward an impactful implementation of the AI Act, you can indicate your interest in doing so via this short Google form.
Thanks to the commenter Sherrinford for correcting me on these.