While I’ve only worked in biosecurity for about a year and my computer security background consists of things I picked up while working on other aspects of software engineering, the cultures seem incredibly different. Some examples of good computer security culture that would be bad biosecurity culture:
Openness and full disclosure. Write blog posts with deep detail on how vulnerabilities were found, with the goal of teaching others how to find similar ones in the future. Keep details quiet for a few months if need be to give vendors time to fix but after, say, 90 days go public.
Breaking things to fix them. Given a new system, of course you should try to compromise it. If you succeed manually, make a demo that cracks it in milliseconds. Make (and publish!) fuzzers and other automated vulnerability search tools.
Enthusiastic curiosity and exploration. Noticing hints of vulnerabilities and digging into them to figure out how deep they go is great. If someone says “you don’t need to know that” ignore them and try to figure it out for yourself.
This is not how computer security has always been, or how it is everywhere, and people in the field are often fiercely protective of these ideals against vendors that try to hide flaws or silence researchers. And overall my impression is that this culture has been tremendously positive in computer security.
Which means that if you come into the effective altruism corner of biosecurity with a computer security background and see all of these discussions of “information hazards”, people discouraging trying to find vulnerabilities, and people staying quiet about dangerous things they’ve discovered it’s going to feel very strange, and potentially rotten.
So here’s a framing that might help see things from this biosecurity perspective. Imagine that the Morris worm never happened, nor Blaster, nor Samy. A few people independently discovered SQL injection but kept it to themselves. Computer security never developed as a field, even as more and more around us became automated. We have driverless cars, robosurgeons, and simple automated agents acting for us, all with the security of original Sendmail. And it’s all been around long enough that the original authors have moved on and no one remembers how any of it works. Someone who put in some serious effort could cause immense destruction, but this doesn’t happen because the people who have the expertise to cause havoc have better things to do. Introducing modern computer security culture into this hypothetical world would not go well!
Most of the cultural differences trace back to what happens once a vulnerability is known. With computers:
The companies responsible for software and hardware are in a position to fix their systems, and disclosure has helped build a norm that they should do this promptly.
People who are writing software can make changes to their approach to avoid creating similar vulnerabilities in the future.
End users have a wide range of effective and reasonably cheap options for mitigation once the vulnerability is known.
But with biology there is no vendor, a specific fix can take years, a fully general fix may not be possible, and mitigation could be incredibly expensive. The culture each field needs is downstream from these key differences.
Overall this is sad: we could move faster if we could all just talk about what we’re most concerned about, plus cause prioritization would be simpler. I wish we were in a world where we could apply the norms from computer security! But different constraints lead to different solutions, and the level of caution I see in biorisk seems about right given these constraints.
(Note that when I talk about “good biosecurity culture” I’m describing a set of norms that I see as the right ones for the situation we’re in, and that are common among effective altruists and other people with a similar view of the world. There’s another set of norms within biology, however, that developed when the main threats were natural. Since there’s no risk of nature using public knowledge to cause harm, this older approach is even more open than computer security culture, and in my opinion is a very poor fit for the environment we’re in now.)
Biosecurity Culture, Computer Security Culture
While I’ve only worked in biosecurity for about a year and my computer security background consists of things I picked up while working on other aspects of software engineering, the cultures seem incredibly different. Some examples of good computer security culture that would be bad biosecurity culture:
Openness and full disclosure. Write blog posts with deep detail on how vulnerabilities were found, with the goal of teaching others how to find similar ones in the future. Keep details quiet for a few months if need be to give vendors time to fix but after, say, 90 days go public.
Breaking things to fix them. Given a new system, of course you should try to compromise it. If you succeed manually, make a demo that cracks it in milliseconds. Make (and publish!) fuzzers and other automated vulnerability search tools.
Enthusiastic curiosity and exploration. Noticing hints of vulnerabilities and digging into them to figure out how deep they go is great. If someone says “you don’t need to know that” ignore them and try to figure it out for yourself.
This is not how computer security has always been, or how it is everywhere, and people in the field are often fiercely protective of these ideals against vendors that try to hide flaws or silence researchers. And overall my impression is that this culture has been tremendously positive in computer security.
Which means that if you come into the effective altruism corner of biosecurity with a computer security background and see all of these discussions of “information hazards”, people discouraging trying to find vulnerabilities, and people staying quiet about dangerous things they’ve discovered it’s going to feel very strange, and potentially rotten.
So here’s a framing that might help see things from this biosecurity perspective. Imagine that the Morris worm never happened, nor Blaster, nor Samy. A few people independently discovered SQL injection but kept it to themselves. Computer security never developed as a field, even as more and more around us became automated. We have driverless cars, robosurgeons, and simple automated agents acting for us, all with the security of original Sendmail. And it’s all been around long enough that the original authors have moved on and no one remembers how any of it works. Someone who put in some serious effort could cause immense destruction, but this doesn’t happen because the people who have the expertise to cause havoc have better things to do. Introducing modern computer security culture into this hypothetical world would not go well!
Most of the cultural differences trace back to what happens once a vulnerability is known. With computers:
The companies responsible for software and hardware are in a position to fix their systems, and disclosure has helped build a norm that they should do this promptly.
People who are writing software can make changes to their approach to avoid creating similar vulnerabilities in the future.
End users have a wide range of effective and reasonably cheap options for mitigation once the vulnerability is known.
But with biology there is no vendor, a specific fix can take years, a fully general fix may not be possible, and mitigation could be incredibly expensive. The culture each field needs is downstream from these key differences.
Overall this is sad: we could move faster if we could all just talk about what we’re most concerned about, plus cause prioritization would be simpler. I wish we were in a world where we could apply the norms from computer security! But different constraints lead to different solutions, and the level of caution I see in biorisk seems about right given these constraints.
(Note that when I talk about “good biosecurity culture” I’m describing a set of norms that I see as the right ones for the situation we’re in, and that are common among effective altruists and other people with a similar view of the world. There’s another set of norms within biology, however, that developed when the main threats were natural. Since there’s no risk of nature using public knowledge to cause harm, this older approach is even more open than computer security culture, and in my opinion is a very poor fit for the environment we’re in now.)