I intend to go even though I’m vaguely against trying to stop AI development right now. I think it’s true that:
The benefit/harm ratio for open sourcing AI seems much different than traditional software, and I don’t think a heuristic of “open sourcing is always better” is reasonable.
If you have access to the model weights, then you can relatively easily bypass the safety measures. I don’t think the Llama models are dangerous right now, but it is good to push back against the idea that making a model “safe” is simply a matter of making the original weights safe.
If Meta continues to open source their models, then this will eventually enable terrorists to use the models to asymmetrically harm the world. I think a single bio-terrorist attack would likely reverse all the positive gains from Meta open sourcing their models.
Meta’s chief AI scientist, Yann LeCun, mostly has terrible arguments against taking AI safety seriously.
What would it take for a model to be capable of supporting bioterrorism? Or simply to get consistently useful results, similar to human research scientists, in technical domains.
The LLM model is f(fuzzy map of human text, prompt) = [distribution of tokens proportional the probabilities a human might emit the token].
You might assume this would have median human intelligence but since “errors are noisy” while “correct answers repeat again and again”, an LLM emitting the most probable token is somewhat above median intelligence at tasks that can be reflected this way.
This does not necessarily scale to [complex technical situations that are not necessarily available in text form and require vision, smell, and touch as well as fine robotic control], [actions that evolve the state of a laboratory towards an end goal].
It seems like you would need actual training data from actual labs, right? So as long as Meta doesn’t actually train on that kind of information, the model won’t be better at helping a bioterrorist with their technical challenges than google? Or am I badly wrong somewhere?
I intend to go even though I’m vaguely against trying to stop AI development right now. I think it’s true that:
The benefit/harm ratio for open sourcing AI seems much different than traditional software, and I don’t think a heuristic of “open sourcing is always better” is reasonable.
If you have access to the model weights, then you can relatively easily bypass the safety measures. I don’t think the Llama models are dangerous right now, but it is good to push back against the idea that making a model “safe” is simply a matter of making the original weights safe.
If Meta continues to open source their models, then this will eventually enable terrorists to use the models to asymmetrically harm the world. I think a single bio-terrorist attack would likely reverse all the positive gains from Meta open sourcing their models.
Meta’s chief AI scientist, Yann LeCun, mostly has terrible arguments against taking AI safety seriously.
What would it take for a model to be capable of supporting bioterrorism? Or simply to get consistently useful results, similar to human research scientists, in technical domains.
The LLM model is f(fuzzy map of human text, prompt) = [distribution of tokens proportional the probabilities a human might emit the token].
You might assume this would have median human intelligence but since “errors are noisy” while “correct answers repeat again and again”, an LLM emitting the most probable token is somewhat above median intelligence at tasks that can be reflected this way.
This does not necessarily scale to [complex technical situations that are not necessarily available in text form and require vision, smell, and touch as well as fine robotic control], [actions that evolve the state of a laboratory towards an end goal].
It seems like you would need actual training data from actual labs, right? So as long as Meta doesn’t actually train on that kind of information, the model won’t be better at helping a bioterrorist with their technical challenges than google? Or am I badly wrong somewhere?
Great to see you getting behind this Matthew. Being against the open sourcing of frontier AI is something that we can have a broader coalition on.