“They are both unsafe now for the things they can be used for and releasing model weights in the future will be more unsafe because of things the model could do.”
I think using “unsafe” in a very broad way like this is misleading overall and generally makes the AI safety community look like miscalibrated alarmists. I do not want to end up in a position where, in 5 or 10 years’ time, policy proposals aimed at reducing existential risk come with 5 or 10 years worth of baggage in the form of previous claims about model harms that have turned out to be false. I expect that the direct effects of the Llama models that have been released so far will be net positive by a significant margin (for all the standard reasons that open source stuff is net positive). Maybe you disagree with this, but a) it seems better to focus on the more important claim, for which there’s a consensus in the field, and b) even if you’re going to make both claims, using the same word (“unsafe”) in these two very different senses is effectively a motte and bailey.
It’s more like people think “open source” is good because of the history of open source software, but this is a pretty different thing. The linked article describes how model weights are not software and Meta’s ToS are arguably anti-competitive, which undermines any claim to just wanting to share tools and accelerate progress.
The policy you are suggesting is far further away from “open source” than this is. It is totally reasonable for Meta to claim that doing something closer to open source has some proportion of the benefits of full open source.
The policy you are suggesting is far further away from “open source” than this is. It is totally reasonable for Meta to claim that doing something closer to open source has some proportion of the benefits of full open source.
Suppose meta was claiming that their models were curing cancer. It probably is the case that their work is more likely to cure cancer than if they took Holly’s preferred policy, but nonetheless it feels legitimate to object to them generating goodwill by claiming to cure cancer.
In your hypothetical, if Meta says “OK you win, you’re right, we’ll henceforth take steps to actually cure cancer”, onlookers would assume that this is a sensible response, i.e. that Meta is responding appropriately to the complaint. If the protester then gets back on the news the following week and says “no no no this is making things even worse”, I think onlookers would be very confused and say “what the heck is wrong with that protester?”
I think using “unsafe” in a very broad way like this is misleading overall and generally makes the AI safety community look like miscalibrated alarmists.
I agree that when there’s no memetic fitness/calibration trade-off, it’s always better to be calibrated. But here there is a trade-off. How should we take it?
My sense is that there’s never been any epistemically calibrated social movement and so that it would be playing against odds to impose that constraint. Even someone like Henry Spira who was very thoughtful personally used very unnuanced communication to achieve social change.
Richard, do you think that being miscalibrated has hurt or benefited the ability of past movements to cause social change? E.g. climate change and animal welfare.
My impression is that probably not? They caused entire chunks of society to be miscalibrated on climate change (maybe less in the US but in Europe it’s pretty big), and that’s not good, but I would guess that the alarmism helped them succeed? As long as there also exists a moderate faction & and there still exists background debates on the object-level, I feel like having a standard social activism movement wd be overall very welcome.
Curious if anyone here knows the relevant literature on the topic, e.g. details in the radical flank literature.
The analogy here would be climate scientists and climate protesters. Afaik climate protesters have not delegitimised climate scientists or made them seem like miscalibrated alarmists (perhaps even the opposite).
“They are both unsafe now for the things they can be used for and releasing model weights in the future will be more unsafe because of things the model could do.”
I think using “unsafe” in a very broad way like this is misleading overall and generally makes the AI safety community look like miscalibrated alarmists. I do not want to end up in a position where, in 5 or 10 years’ time, policy proposals aimed at reducing existential risk come with 5 or 10 years worth of baggage in the form of previous claims about model harms that have turned out to be false. I expect that the direct effects of the Llama models that have been released so far will be net positive by a significant margin (for all the standard reasons that open source stuff is net positive). Maybe you disagree with this, but a) it seems better to focus on the more important claim, for which there’s a consensus in the field, and b) even if you’re going to make both claims, using the same word (“unsafe”) in these two very different senses is effectively a motte and bailey.
The policy you are suggesting is far further away from “open source” than this is. It is totally reasonable for Meta to claim that doing something closer to open source has some proportion of the benefits of full open source.
Suppose meta was claiming that their models were curing cancer. It probably is the case that their work is more likely to cure cancer than if they took Holly’s preferred policy, but nonetheless it feels legitimate to object to them generating goodwill by claiming to cure cancer.
In your hypothetical, if Meta says “OK you win, you’re right, we’ll henceforth take steps to actually cure cancer”, onlookers would assume that this is a sensible response, i.e. that Meta is responding appropriately to the complaint. If the protester then gets back on the news the following week and says “no no no this is making things even worse”, I think onlookers would be very confused and say “what the heck is wrong with that protester?”
It is a confusing point, maybe too subtle for a protest. I am learning!
It was a difficult point to make and we ended up removing it where we could.
This is a good point and feels persuasive, thanks!
I agree that when there’s no memetic fitness/calibration trade-off, it’s always better to be calibrated. But here there is a trade-off. How should we take it?
My sense is that there’s never been any epistemically calibrated social movement and so that it would be playing against odds to impose that constraint. Even someone like Henry Spira who was very thoughtful personally used very unnuanced communication to achieve social change.
Richard, do you think that being miscalibrated has hurt or benefited the ability of past movements to cause social change? E.g. climate change and animal welfare.
My impression is that probably not? They caused entire chunks of society to be miscalibrated on climate change (maybe less in the US but in Europe it’s pretty big), and that’s not good, but I would guess that the alarmism helped them succeed?
As long as there also exists a moderate faction & and there still exists background debates on the object-level, I feel like having a standard social activism movement wd be overall very welcome.
Curious if anyone here knows the relevant literature on the topic, e.g. details in the radical flank literature.
How much do you anticipate protests characterizing the AI Safety community, and why is that important to you?
The analogy here would be climate scientists and climate protesters. Afaik climate protesters have not delegitimised climate scientists or made them seem like miscalibrated alarmists (perhaps even the opposite).