Reducing AGI-related X-risk might require regulations in many countries that prevent firms from building AGI models. This is difficult because governments are resistant to shutting down businesses and damaging their economies. This post discusses a similar event in which the government of Jordan attempted to shutdown some farms to preserve an aquifer. I use Jordan’s experience to illustrate important considerations for the effectiveness of AGI regulation. I argue that possibilities for regulation depend partly on two factors. If AGI regulation is accepted as necessary by software engineers, peer enforcement would make regulation possible. Alternatively, regulation is possible if building a dangerous-level AGI model has high startup costs, which would reduce entrants into the market and increase the penalty for discovery.
Introduction
It seems plausible that preventing x-risk from AGI could depend on creating anti-AGI regulations in multiple countries. If AGI took off quickly, there might be little time to figure out how to influence countries to establish effective regulations that prevent AGI development. It would be better to have an idea of how to build such regulations now, rather than wait to come up with them in the future.
Building such regulations would be especially challenging if AGI were profitable, for example if AGI could replace expensive human lawyers in advising clients. In that case governments would have a financial and popularity incentive to protect their AGI industry, and private agents would have a financial incentive to evade such regulations. Why would governments ever voluntarily hurt their economies, and under what circumstances would governments succeed at that?
It turns out that I am an expert on that exact question. My name is Tim and I’m a political scientist at Georgetown where I study why non-democracies shut down, seize and regulate companies out of existence. This is often called “expropriation”. I have one paper already out about why Jordan expropriated some farmers (more papers are in the pipeline). A bunch of farmers probably seems pretty far from AGI, but this case study tells us a lot about how regulating AGI would and wouldn’t work.
This blog post starts with a few headline lessons for how to influence AGI-regulation overseas. I restrict my lessons to subnational implementation because there are too many lessons for one blog post. Then I tell the story of a battle between the King Abdullah II, a bunch of civil engineers, the US government, the World Bank and a lot of pissed of farmers. Along the way, I give examples of the lessons to learn about AGI regulation.
I am not an expert on AGI or on AGI-specific regulation. For creativity I have not yet read other people’s work on AGI regulation. If people are interested in this work, I could become more involved.
The Scenario
I’m writing for a possible 2027 in which AGI has been developed and a portion of global elites are convinced it is a danger. This might be the result of X-risk ideas expanding, the EA movement greatly expanding, or the issue becoming higher profile. Perhaps the most plausible scenario is an issue-defining event in which an AI produced some noticeable disruption but was subsequently contained. For example, a misaligned AI intentionally disabled a pipeline to achieve its function, creating a news cycle on AI x-risks. As a result some set of global elites (US congress, a large foundation) are aware that AI safety is an important public good. However, the good needs to cover the entire world because an AI created anywhere in the world poses a risk to the whole world.
I won’t address scenarios where few people care about AGI, since no policies will be implemented then.
I will also assume that AGI has some economic value over its non-general counterparts. This seems pretty likely because currend ML applications are constrained by the human common sense needed to design and use their outputs.
How to close an industry in a non-democracy
First a definition: A regime is the group of people who run a state who have solved their internal collective action problems. They might use peer enforcement ([this is good actually](https://www.youtube.com/watch?v=v8oDEwvec1c)), they might have fought a war together, or the leader might spy on them to keep them in line. Anyway, think of the regime as a bunch of interest groups sitting at a table hashing out their differences and figuring out which policy hurts their collective interests the least. The state is an enormous organization with many thousands of employees, and the regime tries to control the state. The lowly field officer enforcing the AGI regulation has a boss, and his boss’s boss’s boss is a regime member. The field officer likely cares more about keeping the local bigwigs happy than those guys at the table. The state is too big to efficiently solve its collective action problems.
This model helps us see that we have not one problem but two.
We must convince the regime to decide to regulate AGI, given its collective interests.
We must ensure that the regulations actually succeed in changing the behavior of private actors.
These are two separate and difficult tasks; Success requires accomplishing both. But explaining them all is beyond one blog post. We need to focus on a set of issues small enough for one blog post.
This post asks “How can we ensure non-democracies can implement and AGI regulation”. If possible I will do separate posts on how to convince non-democracies to regulate, then later on how democracies are different (in general democratic regimes are harder to influence but better at implementing the regulations).
When does regulation succeed?
Profit-making industries like farming and AGI businesses are difficult to regulate because private actors can start them without support from the government. If Joe and Jane decide to start a forbidden business, no one needs to tell the bigwigs in the capital. Moreover, if a local inspector finds the forbidden business, he doesn’t need to tell the bigwigs. Shutting down a local business would piss off the inspector’s host community, and might even cause protests. Non-democracies tend to be paranoid about protests, so the inspector has an incentive to let things slide.
So the big problem is getting information up the chain to the regime. There can be a whole industry the regime is unaware of. In China, enforcing pollution regulations got so bad the CCP simply ordered whole factory districts shut for non-compliance, because local inspectors refused to enforce. Without ground truth the regime could only monitor if the factories were open or closed, so they used that leverage.
Peer enforcement is the most powerful mechanism. If a regulation is well-supported by the locals, they will punish each other for misbehavior. Peers are the only group that always knows about this economic activity, and is positioned to report. Ever noticed that narcotics syndicates mostly kill members of their own communities, not state enforcers? They do so because their community members have the information that state enforcers need.
In the absence of peer-enforcement, top-down can work when starting a business is capital intensive. The main risk of enforcement is losing your investment. If starting a firm requires a lot of investment, the risk of enforcement is highly dissuasive. So few firms enter the illegal business. Because few firms enter, enforcement is less costly and the entrants get caught more. So high capital costs reduce violation through two mechanisms, creating a large effect.
Okay, so to review
The regime needs information about who is doing the activity
Peer informants are the most effective information source, but requires grass-roots legitimacy
Top-down monitoring is possible when startup costs are high, so fewer entrants facing greater punishment
The politics of water in Jordan
Here I will illustrate these points with an extended example from Jordan. <summary here>
Jordan has two main sources of water. Every year a few hundred MCM (million cubic meters) can be captured in the Northern Highlands by an expensive system of dams and canals. We call this surface water. The eastern desert has water in underground aquifers, and sometimes is refilled by rare rainfall in the northeast. We call this ground water.
A satellite photo of Jordan during a rainy day in the rainy season
Meanwhile, Jordan has two main consumers of water. In the northeastern mountains are Jordan’s thirsty metropolises, which need water for washing, drinking, gardening and industry. All over the highlands and in the desert there are farmers that also need water.
The problem is that the cities have outgrown the surface water, so they started to have periodic droughts in the summer. The droughts caused riots in poor neighborhoods, which scared the regime. This meant the cities needed a new constant source of water, like the aquifers in the desert.
But by the 2000′s farmers throughout north Jordan were consuming groundwater at an unsustainable rate. The desert was big and poorly regulated, so from 1970 to 2000 any old person could drill a well and start a farm. Small farms dotted the northern desert, and all these farmers were consuming water faster than the rain refilled it. The GoJ could build new desert wells and pump them to the desert, but those wells would not last.
Despite the need, the GoJ was reluctant to enforce any policies that would close farms. Growth makes governing much easier. Closing farms would create unemployed people, who might move to cities and participate in riots. The losers from the change protested in Jordan’s parliament to protect their communities livelihoods. At one point a Jordanian MP got in a shootout with the Minister of Water. Once a community depends on a technology for their livelihoods, governments are loathe to take it away.
In around 2008, the pro-conservation camp got the upper hand, even appointing their ally to the Minister of Water and Irrigation. The US played a big role. USAID saw this unsolved urban-drought-riot problem as a threat to their one friend in an unfriendly region. They had a powerful ally in the government engineers who actually supplied the city, for whom farmer competition was a major headache. In the next post I can explain how they got the regime on side.
This pro-urban coalition launched the following policies
Farmers had to pay a fee for the volume of water they consumed
New farms were effectively banned, or subject to steep fines
A small aquifer in the deep south would be closed to farming completely
Regulation in the North
The new pro-conservation Minister of Water instructed his agents to go implement the laws. Make sure these farmers are paying fees on each cubic meter of water. Find newly constructed illegal wells and close them. Send a map of each farm and its location to the headquarters. The Minister even asked them to fill the dynamiting of illegal wells and send it to him.
In practice things went poorly. Each district had hundreds of farmers spread across the desert, so monitoring them was expensive. The GoJ had to hire tens of inspectors to criss cross the desert inspecting farms. The local inspection office often declined to share their information upward. One visiting academic found that they had a whole list of wells in the office which they refused to share with the capital.
Meanwhile, the farmers quickly found ways to evade these regulations. They damaged their wells to appear old and wore, and thus grandfathered by previous laws. They hid new wells on their properties. They pumped water in plastic piping for kilometres to evade regulation. Sure these tricks were pretty obvious to people on the ground, but the minister was off in the capital, so only the inspectors could relay the information.
As a result, the actually cropped area in the desert continued to expand not contract in this period. The farmers pretty much succeeded in evading regulation.
Regulation in the South
In South Jordan there is a small area called Mudawarra with a deep aquifer. Mudawarra is about as literally the furthest point in Jordan the thirsty cities. Tapping that aquifer to feed the metropolises would ultimately require $1.2 billion pipeline: a princely sum for a small principality (3.5% of Jordan’s GDP/year). To the World Bank, this Mudawarra scheme seemed completely hair brained. Why spend $1.2 billion for water when there are larger aquifers right next to the cities!
The GoJ had realized that Mudawarra was the only place in Jordan where they could actually regulate agriculture. Because Mudawarra was isolated, a new farmer had to invest in transporting equipment and connecting with global supply chains. There was no local community to sell to or to start recreational farms. As a result Mudawarra had just three major commercial farms.
The GoJ realized that once the existing farms closed, they just had to stop new farms from starting. A farm is quite distinctive in the desert. And any farmer would have to buy expensive circular-irrigation equipment and tractors, then truck them far away. If the government called that farmer’s bluff he would lose the investment. And with no other famers around, even the regime could catch him with just google maps.
Lessons for AI regulation
What does all this say about AI regulation? Actually, it tells us a lot!
Popular legitimacy among software engineers makes enforcement possible
Jordan’s regulation process was so difficult because there was no peer enforcement. Farmers never snitched on new entrants and inspectors covered each other’s backs.
If the engineers who build AGI software think that x-risks are impossible and the regulation is silly, they will not peer enforce. “These servers officer? Oh they’re just running a basic random forest, best move along.”
But if regulations are seen as legitimate, enforcement becomes really cheap! This would require a wider education campaign on AGI risks. The global software engineering community is diverse, and not all reading the same news. In Thunderball the villain builds a private nuclear arsenal to blackmail the city of Miami, but his engineer quietly disables the devise before he can activate it (making Bond useless). The global legitimacy of the chemicals weapons ban made disarming the post-soviet states possible.
The cost to build and run an AGI is important. Higher is better!
In north Jordan starting a small farm is easy, in Muddawarra it is hard. With hundreds of small farms regulation became completely intractable and the punishment for being caught was a few thousand dollars in lost equipment/wages.
Imagine a world where running a smart AGI requires a warehouse full of servers or requires 100,000 person hours to design and train. In this world, only individuals with lots of capital can get involved. That’s a smaller group of people, who are easier to educate and regulate. For an extreme example, consider nuclear weapons. It cost North Korea about $ 1-2 billion to build their nuclear arsenal. Nuclear nonproliferation is cheap because only a tiny set of organizations can front the capital to build a weapon.
Conclusion
We need to take a big, complicated task “preventing AGI x-risk” and turn it into a set of smaller, simpler tasks. This blog post argued for two out of many subtasks.
Convincing engineers globally to care about AGI x-risk
Keeping the costs of a new AGI system high
Final note on funding
I would like to finish this project. I have not dug into how to achieve those subtasks, nor built out the full set of tasks we would require. There’s plenty of other research on these questions to describe. Also, I need to improve my writing ability on this topic, and collect ideas into a single organization. I need to read what others have written about the topic.
The thing is I lack the free time to do that. I learned about why governments shut down businesses to market that skill to investors in industries like mining. They privately benefit from avoiding expropriation so they might pay me for this information. I was planning to spend my summer writing op-eds in mining magazines to promote my brand.
I could redirect that energy toward writing about hypothetical AGI regulation schemes, but only for money. Can anyone advise me on possible funding schemes for this? I could write a report and then charge people to download it.
Also if these seems completely irrelevant please tell me so I can allocate my time more effectively. Also tell me if this is confusingly written, I did not have time to edit much.
AGI Risk: How to internationally regulate industries in non-democracies
Abstract
Reducing AGI-related X-risk might require regulations in many countries that prevent firms from building AGI models. This is difficult because governments are resistant to shutting down businesses and damaging their economies. This post discusses a similar event in which the government of Jordan attempted to shutdown some farms to preserve an aquifer. I use Jordan’s experience to illustrate important considerations for the effectiveness of AGI regulation. I argue that possibilities for regulation depend partly on two factors. If AGI regulation is accepted as necessary by software engineers, peer enforcement would make regulation possible. Alternatively, regulation is possible if building a dangerous-level AGI model has high startup costs, which would reduce entrants into the market and increase the penalty for discovery.
Introduction
It seems plausible that preventing x-risk from AGI could depend on creating anti-AGI regulations in multiple countries. If AGI took off quickly, there might be little time to figure out how to influence countries to establish effective regulations that prevent AGI development. It would be better to have an idea of how to build such regulations now, rather than wait to come up with them in the future.
Building such regulations would be especially challenging if AGI were profitable, for example if AGI could replace expensive human lawyers in advising clients. In that case governments would have a financial and popularity incentive to protect their AGI industry, and private agents would have a financial incentive to evade such regulations. Why would governments ever voluntarily hurt their economies, and under what circumstances would governments succeed at that?
It turns out that I am an expert on that exact question. My name is Tim and I’m a political scientist at Georgetown where I study why non-democracies shut down, seize and regulate companies out of existence. This is often called “expropriation”. I have one paper already out about why Jordan expropriated some farmers (more papers are in the pipeline). A bunch of farmers probably seems pretty far from AGI, but this case study tells us a lot about how regulating AGI would and wouldn’t work.
This blog post starts with a few headline lessons for how to influence AGI-regulation overseas. I restrict my lessons to subnational implementation because there are too many lessons for one blog post. Then I tell the story of a battle between the King Abdullah II, a bunch of civil engineers, the US government, the World Bank and a lot of pissed of farmers. Along the way, I give examples of the lessons to learn about AGI regulation.
I am not an expert on AGI or on AGI-specific regulation. For creativity I have not yet read other people’s work on AGI regulation. If people are interested in this work, I could become more involved.
The Scenario
I’m writing for a possible 2027 in which AGI has been developed and a portion of global elites are convinced it is a danger. This might be the result of X-risk ideas expanding, the EA movement greatly expanding, or the issue becoming higher profile. Perhaps the most plausible scenario is an issue-defining event in which an AI produced some noticeable disruption but was subsequently contained. For example, a misaligned AI intentionally disabled a pipeline to achieve its function, creating a news cycle on AI x-risks. As a result some set of global elites (US congress, a large foundation) are aware that AI safety is an important public good. However, the good needs to cover the entire world because an AI created anywhere in the world poses a risk to the whole world.
I won’t address scenarios where few people care about AGI, since no policies will be implemented then.
I will also assume that AGI has some economic value over its non-general counterparts. This seems pretty likely because currend ML applications are constrained by the human common sense needed to design and use their outputs.
How to close an industry in a non-democracy
First a definition: A regime is the group of people who run a state who have solved their internal collective action problems. They might use peer enforcement ([this is good actually](https://www.youtube.com/watch?v=v8oDEwvec1c)), they might have fought a war together, or the leader might spy on them to keep them in line. Anyway, think of the regime as a bunch of interest groups sitting at a table hashing out their differences and figuring out which policy hurts their collective interests the least. The state is an enormous organization with many thousands of employees, and the regime tries to control the state. The lowly field officer enforcing the AGI regulation has a boss, and his boss’s boss’s boss is a regime member. The field officer likely cares more about keeping the local bigwigs happy than those guys at the table. The state is too big to efficiently solve its collective action problems.
This model helps us see that we have not one problem but two.
We must convince the regime to decide to regulate AGI, given its collective interests.
We must ensure that the regulations actually succeed in changing the behavior of private actors.
These are two separate and difficult tasks; Success requires accomplishing both. But explaining them all is beyond one blog post. We need to focus on a set of issues small enough for one blog post.
This post asks “How can we ensure non-democracies can implement and AGI regulation”. If possible I will do separate posts on how to convince non-democracies to regulate, then later on how democracies are different (in general democratic regimes are harder to influence but better at implementing the regulations).
When does regulation succeed?
Profit-making industries like farming and AGI businesses are difficult to regulate because private actors can start them without support from the government. If Joe and Jane decide to start a forbidden business, no one needs to tell the bigwigs in the capital. Moreover, if a local inspector finds the forbidden business, he doesn’t need to tell the bigwigs. Shutting down a local business would piss off the inspector’s host community, and might even cause protests. Non-democracies tend to be paranoid about protests, so the inspector has an incentive to let things slide.
So the big problem is getting information up the chain to the regime. There can be a whole industry the regime is unaware of. In China, enforcing pollution regulations got so bad the CCP simply ordered whole factory districts shut for non-compliance, because local inspectors refused to enforce. Without ground truth the regime could only monitor if the factories were open or closed, so they used that leverage.
Peer enforcement is the most powerful mechanism. If a regulation is well-supported by the locals, they will punish each other for misbehavior. Peers are the only group that always knows about this economic activity, and is positioned to report. Ever noticed that narcotics syndicates mostly kill members of their own communities, not state enforcers? They do so because their community members have the information that state enforcers need.
In the absence of peer-enforcement, top-down can work when starting a business is capital intensive. The main risk of enforcement is losing your investment. If starting a firm requires a lot of investment, the risk of enforcement is highly dissuasive. So few firms enter the illegal business. Because few firms enter, enforcement is less costly and the entrants get caught more. So high capital costs reduce violation through two mechanisms, creating a large effect.
Okay, so to review
The regime needs information about who is doing the activity
Peer informants are the most effective information source, but requires grass-roots legitimacy
Top-down monitoring is possible when startup costs are high, so fewer entrants facing greater punishment
The politics of water in Jordan
Here I will illustrate these points with an extended example from Jordan. <summary here>
Jordan has two main sources of water. Every year a few hundred MCM (million cubic meters) can be captured in the Northern Highlands by an expensive system of dams and canals. We call this surface water. The eastern desert has water in underground aquifers, and sometimes is refilled by rare rainfall in the northeast. We call this ground water.
Meanwhile, Jordan has two main consumers of water. In the northeastern mountains are Jordan’s thirsty metropolises, which need water for washing, drinking, gardening and industry. All over the highlands and in the desert there are farmers that also need water.
The problem is that the cities have outgrown the surface water, so they started to have periodic droughts in the summer. The droughts caused riots in poor neighborhoods, which scared the regime. This meant the cities needed a new constant source of water, like the aquifers in the desert.
But by the 2000′s farmers throughout north Jordan were consuming groundwater at an unsustainable rate. The desert was big and poorly regulated, so from 1970 to 2000 any old person could drill a well and start a farm. Small farms dotted the northern desert, and all these farmers were consuming water faster than the rain refilled it. The GoJ could build new desert wells and pump them to the desert, but those wells would not last.
Despite the need, the GoJ was reluctant to enforce any policies that would close farms. Growth makes governing much easier. Closing farms would create unemployed people, who might move to cities and participate in riots. The losers from the change protested in Jordan’s parliament to protect their communities livelihoods. At one point a Jordanian MP got in a shootout with the Minister of Water. Once a community depends on a technology for their livelihoods, governments are loathe to take it away.
In around 2008, the pro-conservation camp got the upper hand, even appointing their ally to the Minister of Water and Irrigation. The US played a big role. USAID saw this unsolved urban-drought-riot problem as a threat to their one friend in an unfriendly region. They had a powerful ally in the government engineers who actually supplied the city, for whom farmer competition was a major headache. In the next post I can explain how they got the regime on side.
This pro-urban coalition launched the following policies
Farmers had to pay a fee for the volume of water they consumed
New farms were effectively banned, or subject to steep fines
A small aquifer in the deep south would be closed to farming completely
Regulation in the North
The new pro-conservation Minister of Water instructed his agents to go implement the laws. Make sure these farmers are paying fees on each cubic meter of water. Find newly constructed illegal wells and close them. Send a map of each farm and its location to the headquarters. The Minister even asked them to fill the dynamiting of illegal wells and send it to him.
In practice things went poorly. Each district had hundreds of farmers spread across the desert, so monitoring them was expensive. The GoJ had to hire tens of inspectors to criss cross the desert inspecting farms. The local inspection office often declined to share their information upward. One visiting academic found that they had a whole list of wells in the office which they refused to share with the capital.
Meanwhile, the farmers quickly found ways to evade these regulations. They damaged their wells to appear old and wore, and thus grandfathered by previous laws. They hid new wells on their properties. They pumped water in plastic piping for kilometres to evade regulation. Sure these tricks were pretty obvious to people on the ground, but the minister was off in the capital, so only the inspectors could relay the information.
As a result, the actually cropped area in the desert continued to expand not contract in this period. The farmers pretty much succeeded in evading regulation.
Regulation in the South
In South Jordan there is a small area called Mudawarra with a deep aquifer. Mudawarra is about as literally the furthest point in Jordan the thirsty cities. Tapping that aquifer to feed the metropolises would ultimately require $1.2 billion pipeline: a princely sum for a small principality (3.5% of Jordan’s GDP/year). To the World Bank, this Mudawarra scheme seemed completely hair brained. Why spend $1.2 billion for water when there are larger aquifers right next to the cities!
The GoJ had realized that Mudawarra was the only place in Jordan where they could actually regulate agriculture. Because Mudawarra was isolated, a new farmer had to invest in transporting equipment and connecting with global supply chains. There was no local community to sell to or to start recreational farms. As a result Mudawarra had just three major commercial farms.
The GoJ realized that once the existing farms closed, they just had to stop new farms from starting. A farm is quite distinctive in the desert. And any farmer would have to buy expensive circular-irrigation equipment and tractors, then truck them far away. If the government called that farmer’s bluff he would lose the investment. And with no other famers around, even the regime could catch him with just google maps.
Lessons for AI regulation
What does all this say about AI regulation? Actually, it tells us a lot!
Popular legitimacy among software engineers makes enforcement possible
Jordan’s regulation process was so difficult because there was no peer enforcement. Farmers never snitched on new entrants and inspectors covered each other’s backs.
If the engineers who build AGI software think that x-risks are impossible and the regulation is silly, they will not peer enforce. “These servers officer? Oh they’re just running a basic random forest, best move along.”
But if regulations are seen as legitimate, enforcement becomes really cheap! This would require a wider education campaign on AGI risks. The global software engineering community is diverse, and not all reading the same news. In Thunderball the villain builds a private nuclear arsenal to blackmail the city of Miami, but his engineer quietly disables the devise before he can activate it (making Bond useless). The global legitimacy of the chemicals weapons ban made disarming the post-soviet states possible.
The cost to build and run an AGI is important. Higher is better!
In north Jordan starting a small farm is easy, in Muddawarra it is hard. With hundreds of small farms regulation became completely intractable and the punishment for being caught was a few thousand dollars in lost equipment/wages.
Imagine a world where running a smart AGI requires a warehouse full of servers or requires 100,000 person hours to design and train. In this world, only individuals with lots of capital can get involved. That’s a smaller group of people, who are easier to educate and regulate. For an extreme example, consider nuclear weapons. It cost North Korea about $ 1-2 billion to build their nuclear arsenal. Nuclear nonproliferation is cheap because only a tiny set of organizations can front the capital to build a weapon.
Conclusion
We need to take a big, complicated task “preventing AGI x-risk” and turn it into a set of smaller, simpler tasks. This blog post argued for two out of many subtasks.
Convincing engineers globally to care about AGI x-risk
Keeping the costs of a new AGI system high
Final note on funding
I would like to finish this project. I have not dug into how to achieve those subtasks, nor built out the full set of tasks we would require. There’s plenty of other research on these questions to describe. Also, I need to improve my writing ability on this topic, and collect ideas into a single organization. I need to read what others have written about the topic.
The thing is I lack the free time to do that. I learned about why governments shut down businesses to market that skill to investors in industries like mining. They privately benefit from avoiding expropriation so they might pay me for this information. I was planning to spend my summer writing op-eds in mining magazines to promote my brand.
I could redirect that energy toward writing about hypothetical AGI regulation schemes, but only for money. Can anyone advise me on possible funding schemes for this? I could write a report and then charge people to download it.
Also if these seems completely irrelevant please tell me so I can allocate my time more effectively. Also tell me if this is confusingly written, I did not have time to edit much.