Stuart—I’ve seen many replies along these lines on X/Twitter in response to this paper.
I think such replies underestimate the cognitive diversity (IQ spread) of bad actors, terrorists, misanthropes, etc., compared to typical LessWrong/Rationalist/EA people. What seems like ‘easily accessible’ knowledge in textbooks, scientific papers, Wikipedia, etc to some of us might not be at all easily accessible to certain kinds of bad actors. But those folks might find it relatively to easy to use ‘Spicy’ versions of LLMs to research weapons of mass destruction—especially if the LLMs can walk them, step by step, through recipes for mayhem.
What about the majority of my comment showing that by the paper’s own account, LLMs cannot (at least not yet) walk anyone through a recipe for mayhem, unless they are already enough of an expert to know when to discard hallucinatory answers, reprompt the LLM, etc.?
Also, if you’re worried about low-IQ people being able to create mayhem, I think the least of our worries should be that they’d get their hands on a detailed protocol for creating a virus or anything similar (see, e.g., https://www.nature.com/articles/nprot.2007.135) -- hardly anyone would be able to understand it anyway, let alone have the real-world skills or equipment to do any of it.
Yes, the information is available on Google. The question is, in our eyes, more about whether a future model could successfully walk an unskilled person through the process without the person needing to understand it at all.
The paper is an attempt to walk a careful line of warning the world that the same information in more capable models could be quite dangerous, but not actually increasing the likelihood of someone using the current open source models (which it is too late to control!) for making biological weapons.
If there are specific questions you have, I’d be happy to answer.
“future model could successfully walk an unskilled person through the process without the person needing to understand it at all.”
Seems very doubtful. Could an unskilled person be “walked through” this process just by slightly more elaborate instructions? https://www.nature.com/articles/nprot.2007.135? Seems that the real barriers to something as complex as synthesizing a virus are 1) lack of training/skill/tacit knowledge, 2) lack of equipment or supplies. Detailed instructions are already out there.
My interpretation of the Gopal paper is that LLMs do meaningfully change the risks:
They’ll allow you to make progress without understanding, say, the Luo paper or the technology involved.
They’ll tell you what equipment you’d need, where to get it, how to get it, and how to operate it. Or they’ll tell you how to pay someone else to do bits for you without arousing suspicion.
Perhaps model this as having access to a helpful amoral virologist?
Stuart—I’ve seen many replies along these lines on X/Twitter in response to this paper.
I think such replies underestimate the cognitive diversity (IQ spread) of bad actors, terrorists, misanthropes, etc., compared to typical LessWrong/Rationalist/EA people. What seems like ‘easily accessible’ knowledge in textbooks, scientific papers, Wikipedia, etc to some of us might not be at all easily accessible to certain kinds of bad actors. But those folks might find it relatively to easy to use ‘Spicy’ versions of LLMs to research weapons of mass destruction—especially if the LLMs can walk them, step by step, through recipes for mayhem.
What about the majority of my comment showing that by the paper’s own account, LLMs cannot (at least not yet) walk anyone through a recipe for mayhem, unless they are already enough of an expert to know when to discard hallucinatory answers, reprompt the LLM, etc.?
For one answer to this question, see https://www.lesswrong.com/posts/ytGsHbG7r3W3nJxPT/will-releasing-the-weights-of-large-language-models-grant?commentId=FCTuxs43vtqLMmG2n
For lots more discussion, see the other LessWrong comments at: https://www.lesswrong.com/posts/ytGsHbG7r3W3nJxPT/will-releasing-the-weights-of-large-language-models-grant
And also check out my rather unpopular question here: https://www.lesswrong.com/posts/dL3qxebM29WjwtSAv/would-it-make-sense-to-bring-a-civil-lawsuit-against-meta
I am genuinely interested in gathering valid critiques on my work so that I can do better in the future.
Also, if you’re worried about low-IQ people being able to create mayhem, I think the least of our worries should be that they’d get their hands on a detailed protocol for creating a virus or anything similar (see, e.g., https://www.nature.com/articles/nprot.2007.135) -- hardly anyone would be able to understand it anyway, let alone have the real-world skills or equipment to do any of it.
Yes, the information is available on Google. The question is, in our eyes, more about whether a future model could successfully walk an unskilled person through the process without the person needing to understand it at all.
The paper is an attempt to walk a careful line of warning the world that the same information in more capable models could be quite dangerous, but not actually increasing the likelihood of someone using the current open source models (which it is too late to control!) for making biological weapons.
If there are specific questions you have, I’d be happy to answer.
“future model could successfully walk an unskilled person through the process without the person needing to understand it at all.”
Seems very doubtful. Could an unskilled person be “walked through” this process just by slightly more elaborate instructions? https://www.nature.com/articles/nprot.2007.135? Seems that the real barriers to something as complex as synthesizing a virus are 1) lack of training/skill/tacit knowledge, 2) lack of equipment or supplies. Detailed instructions are already out there.
My interpretation of the Gopal paper is that LLMs do meaningfully change the risks:
They’ll allow you to make progress without understanding, say, the Luo paper or the technology involved.
They’ll tell you what equipment you’d need, where to get it, how to get it, and how to operate it. Or they’ll tell you how to pay someone else to do bits for you without arousing suspicion.
Perhaps model this as having access to a helpful amoral virologist?