I think we should generally have a prior that social dynamics of large groups of people end up pushing heavily towards conformity, and that those pressures towards conformity can cancel out many orders of magnitude of growth of the number of people who could theoretically explore different directions.
The world has mostly copied bad US approaches to over-regulating planes as well. We also see regulatory convergence in topics like human cloning; many had speculated that China would be defy the consensus elsewhere against it, but that turned out not to be true. Public prediction markets on interesting topics seems to be blocked by regulations almost everywhere, and insider trading laws are most everywhere an obstacle to internal corporate markets.
Back in February we saw a dramatic example of world regulatory coordination. Around the world public health authorities were talking about treating this virus like they had treated all the others in the last few decades. But then world elites talked a lot, and suddenly they all agreed that this virus must be treated differently, such as with lockdowns and masks. Most public health authorities quickly caved, and then most of the world adopted the same policies. Contrarian alternatives like variolation, challenge trials, and cheap fast lower-reliability tests have also been rejected everywhere; small experiments have not even been allowed.
One possible explanation for all this convergence is that regulators are just following what is obviously the best policy. But if you dig into the details you will quickly see that the usual policies are not at all obviously right. Often, they seem obviously wrong. And having all the regulatory bodies suddenly change at once, even when no new strong evidence appeared, seems especially telling.
It seems to me that we instead have a strong world culture of regulators, driven by a stronger world culture of elites. Elites all over the world talk, and then form a consensus, and then authorities everywhere are pressured into following that consensus. Regulators most everywhere are quite reluctant to deviate from what most other regulators are doing; they’ll be blamed far more for failures if they deviate. If elites talk some more, and change their consensus, then authorities must then change their polices. On topic X, the usual experts on X are part of that conversation, but often elites overrule them, or choose contrarians from among them, and insist on something other than what most X experts recommend.
The number of nations, as well as the number of communities and researchers that were capable of doing innovative things in response to COVID was vastly greater in 2020 than for any previous pandemic. But what we saw was much less global variance and innovation in pandemic responses. I think there was scientific innovation, and that innovation was likely greater than for previous pandemics, but overall, despite the vastly greater number of nations and people in the international community of 2020, this only produced more risk-aversion in stepping out of line with elite consensus.
I think by-default we should expect similar effects in fields like AI Alignment. I think maintaining a field that is open to new ideas and approaches is actively difficult. If you grow the field without trying to preserve the concrete and specific mechanisms that are in place to allow innovation to grow, more people will not result in more innovation, it will result in less, even from the people that have previously been part of the same community.
In the case of COVID, the global research community spent a substantial fraction of its effort on actively preventing people from performing experiments like variolation or challenge trials, and we see the same in fields like Psychology research where a substantial fraction of energy is spent on ever-increasing ethical review requirements.
We see the same in the construction industry (a recent strong interest of mine), which despite its quickly growing size, is performing substantially fewer experiments than it was 40 years ago, and is spending most of its effort actively regulating what other people in the industry can do, and limiting the type of allowable construction materials and approaches to smaller and smaller sets.
I think by-default, I expect fast growth of the AI Alignment community to reduce innovation for the same reasons. I expect a larger community will increase pressures towards forming an elite consensus, and that consensus will be enforced via various legible and illegible means. Most of the world is really not great at innovation, and the default outcome of large groups of people, even when pointed towards a shared goal, is not innovation, but conformity, and if we recklessly grow, I think we will default towards the same common outcome.
This is a good point, and a wake up call for us to do better with AI Alignment. Given the majority of funding for AGI x-safety is coming from within EA right now, and as a community we are accutely aware of the failings with Covid, we should be striving to do better.
Back in February we saw a dramatic example of world regulatory coordination. Around the world public health authorities were talking about treating this virus like they had treated all the others in the last few decades. But then world elites talked a lot, and suddenly they all agreed that this virus must be treated differently, such as with lockdowns and masks. Most public health authorities quickly caved, and then most of the world adopted the same policies.
There was some deviation (e.g. no lockdowns in Sweden), but the most telling thing was no human challenge trials anywhere in the world. That alone was a tragedy that prolonged the pandemic by months (by delaying roll-out of vaccines) and caused millions of deaths.
I think we should generally have a prior that social dynamics of large groups of people end up pushing heavily towards conformity, and that those pressures towards conformity can cancel out many orders of magnitude of growth of the number of people who could theoretically explore different directions.
As a concrete case study, I like this Robin Hanson post “The World Forager Elite”:
The number of nations, as well as the number of communities and researchers that were capable of doing innovative things in response to COVID was vastly greater in 2020 than for any previous pandemic. But what we saw was much less global variance and innovation in pandemic responses. I think there was scientific innovation, and that innovation was likely greater than for previous pandemics, but overall, despite the vastly greater number of nations and people in the international community of 2020, this only produced more risk-aversion in stepping out of line with elite consensus.
I think by-default we should expect similar effects in fields like AI Alignment. I think maintaining a field that is open to new ideas and approaches is actively difficult. If you grow the field without trying to preserve the concrete and specific mechanisms that are in place to allow innovation to grow, more people will not result in more innovation, it will result in less, even from the people that have previously been part of the same community.
In the case of COVID, the global research community spent a substantial fraction of its effort on actively preventing people from performing experiments like variolation or challenge trials, and we see the same in fields like Psychology research where a substantial fraction of energy is spent on ever-increasing ethical review requirements.
We see the same in the construction industry (a recent strong interest of mine), which despite its quickly growing size, is performing substantially fewer experiments than it was 40 years ago, and is spending most of its effort actively regulating what other people in the industry can do, and limiting the type of allowable construction materials and approaches to smaller and smaller sets.
I think by-default, I expect fast growth of the AI Alignment community to reduce innovation for the same reasons. I expect a larger community will increase pressures towards forming an elite consensus, and that consensus will be enforced via various legible and illegible means. Most of the world is really not great at innovation, and the default outcome of large groups of people, even when pointed towards a shared goal, is not innovation, but conformity, and if we recklessly grow, I think we will default towards the same common outcome.
Re conformity, I wonder if related arguments could help shift the Future Funds’ worldview?
This is a good point, and a wake up call for us to do better with AI Alignment. Given the majority of funding for AGI x-safety is coming from within EA right now, and as a community we are accutely aware of the failings with Covid, we should be striving to do better.
Is there any legible evidence for this?
There was some deviation (e.g. no lockdowns in Sweden), but the most telling thing was no human challenge trials anywhere in the world. That alone was a tragedy that prolonged the pandemic by months (by delaying roll-out of vaccines) and caused millions of deaths.