I think using “unsafe” in a very broad way like this is misleading overall and generally makes the AI safety community look like miscalibrated alarmists.
I agree that when there’s no memetic fitness/calibration trade-off, it’s always better to be calibrated. But here there is a trade-off. How should we take it?
My sense is that there’s never been any epistemically calibrated social movement and so that it would be playing against odds to impose that constraint. Even someone like Henry Spira who was very thoughtful personally used very unnuanced communication to achieve social change.
Richard, do you think that being miscalibrated has hurt or benefited the ability of past movements to cause social change? E.g. climate change and animal welfare.
My impression is that probably not? They caused entire chunks of society to be miscalibrated on climate change (maybe less in the US but in Europe it’s pretty big), and that’s not good, but I would guess that the alarmism helped them succeed?
As long as there also exists a moderate faction & and there still exists background debates on the object-level, I feel like having a standard social activism movement wd be overall very welcome.
Curious if anyone here knows the relevant literature on the topic, e.g. details in the radical flank literature.
I agree with the general underlying point.
I also think that another important issue is that reasoning on counterfactuals makes people more prone to do things that are unusual AND is more prone to errors (e.g. by not taking into account some other effects).
Both combined make counterfactual reasoning without empirical data pretty perilous on average IMO.
In the case of Ali in your example above for instance, Ali could neglect that the performance he’ll have will determine the opportunities & impact he has 5y down the line and so that being excited/liking the job is a major variable. Without counterfactual reasoning, Ali would have intuitively relied much more on excitement to pick the job but by doing counterfactual reasoning which seemed convincing, he neglected this important variable and made a bad choice.
I think that counterfactual reasoning makes people very prone to ignoring Chesterton’s fence.