Thanks, Rudolf, I think this is a very important point, and probably the best argument against PauseAI. It’s true in general that The Ends Do Not Justify the Means (Among Humans).
My primary response is that you are falling for status-quo bias. Yes this path might be risky, but the default path is more risky. My perception is the current governance of AI is on track to let us run some terrible gambles with the fate of humanity.
Consider environmentalism. It seems quite uncertain whether the environmentalist movement has been net positive (!).
We can play reference class tennis all day but I can counter with the example of the Abolitionists, the Suffragettes, the Civil Rights movement, Gay Pride or the American XL Bully.
It seems to me that people overstate the track record of populist activism at solving complicated problems
...
the science is fairly straightforward, environmentalism is clearly necessary, and the movement has had huge wins
As I argue in the post, I think this is an easier problem than climate change. Just as most people don’t need a detailed understanding of the greenhouse effect, most people don’t need a detailed understanding of the alignment problem (“creating something smarter than yourself is dangerous”).
The advantage with AI is that there is a simple solution that doesn’t require anyone to make big sacrifices, unlike with climate change. With PauseAI, the policy proposal is right there in the name, so it is harder to become distorted than vaguer goals of “environmental justice”.
fighting Moloch rather than sacrificing our epistemics to him for +30% social clout
I think to a significant extent it is possible for PauseAI leadership to remain honest while still having broad appeal. Most people are fine if you say that “I in particular care mostly about x-risk, but I would like to form a coalition with artists who have lost work to AI.”
There is a spirit here, of truth-seeking and liberalism and building things, of fighting Moloch rather than sacrificing our epistemics to him for +30% social clout. I admit that this is partly an aesthetic preference on my part. But I do believe in it strongly.
I’m less certain about this but I think the evidence is much less strong than rationalists would like to believe. Consider: why has no successful political campaign ever run on actually good, nuanced policy arguments? Why do advertising campaigns not make rational arguments for why should prefer their product, instead appealing to your emotions? Why did it take until 2010 for people to have the idea of actually trying to figure out which charities are effective? The evidence is overwhelming that emotional appeals are the only way to persuade large numbers of people.
If we make the conversation about AIS more thoughtful, reasonable, and rational, it increases the chances that the right thing (whatever that ends up being—I think we should have a lot of intellectual humility here!) ends up winning.
Again, this seems like it would be good, but the evidence is mixed. People were making thoughtful arguments for why pandemics are a big risk long before Covid, but the world’s institutions were sufficiently irrational that they failed to actually do anything. If there had been an emotional, epistemically questionable mass movement calling for pandemic preparedness, that would have probably been very helpful.
Most economists seem to agree that European monetary policy is pretty bad and significantly harms Europe, but our civilization is too inadequate to fix the problem. Many people make great arguments about why aging sucks and it should really be a top priority to fix, but it’s left to Silicon Valley to actually do something. Similarly for shipping policy, human challenge trials and starting school later. There is long list of preventable, disastrous policies which society has failed to fix due lack of political will, not lack of sensible arguments.
I agree this is slightly hyperbolic. If you include the disappearance of Ilya Sutskever, there’s three. And I know of one more not widely reported. Depending on how narrow your definition of a “safety-focused researcher” is, four people leaving in less than 6 months is fairly significant.