Thanks for the write-up. Just adding a note on how this distinction has practical implications for how to design databases containing hazardous sequences that are required for gene synthesis screening systems.
With gene synthesis screening, companies want to stop bad actors from getting access to the physical DNA or RNA of potential pandemic pathogens. Now, let’s say researchers find the sequence of a novel pathogen that would likely spark a pandemic if released. Most would want this sequence to be added to synthesis screening databases. But some also want this database to be public. The information hazards involved in making such information publicly available could be large, especially if there is attached discussion of how exactly these sequences are dangerous.
(I expected to see your comment link to SecureDNA, which has a cryptographic solution to screening synthesis without either (a) sending hazards to synthesizers or (b) sending reconstructible synthesis orders to others.)
Hi slg — great point about synthesis screening being a very concrete example where approaches to security can make a big difference.
One quibble I have: Your hyperlink seems to suggest that Diggans and Leproust advocate for a fully “public” database of annotated hazard sequences. But I think it’s worth noting that although they do use the phrase “publicly available” a couple of times, they also pretty explicitly discuss the idea of having such a database be accessible to synthesis providers only, which is a much smaller set and seems to carry significantly lower risks for misuse than truly public access. Relevant quote:
“Sustained funding and commitment will be required to build and maintain a database of risk-associated sequences, their known mechanisms of pathogenicity and the biological contexts in which these mechanisms can cause harm. This database (or at a minimum a screening capability making use of this database), to have maximum impact on global DNA synthesis screening, must be available to both domestic and international providers.”
Also worth noting the parenthetical about having providers use a screening mechanism with access to the database without having such direct access themselves, which seems like a nod to some of the features in, eg, SecureDNA’s approach.
Yes, benchtop devices have significant ramifications!
Agreed, storing the database on-device does sound much harder to secure than some kind of distributed storage. Though, I can imagine that some customers will demand airgapped on-device solutions, where this challenge could present itself anyway.
Agreed, sending exact synthesis orders from devices to screeners seems undesirable/unviable, for a host of reasons.
But that’s consistent with my comment, which just meant to emphasise that I don’t read Diggans and Leproust as advocating for a fully “public” hazard database, as slg’s comment could be read to imply.
If your benchtop device user can modify the hardware to attempt to defeat the screening mechanism, the problem becomes orders of magnitude harder. I imagine that making a DNA sequence generating device that can’t be modified to make smallpox even if it’s in the middle of Pyongyang and the malicious user is the North Korean government is an essentially unsolvable problem—if nothing else, they can try to reverse engineer the device and build a similar one without any screening mechanism at all.
A bit tangential, but this raises an important point: in general, you’re looking for things that raise the bar for causing harm. If you can take smallpox synthesis from something where anyone who works in a lab with a benchtop can synthesize it without even opening in the machine to one where they would have to disassemble it, but that already increases the chance that someone else in the lab would notice.
It would be great to get to a place where we have systems that will provide reliable protection even from well-funded state actors, but (a) a lot of the risk comes from much easier cases like it becoming easier for an individual to cause harm and (b) we are so far from having that kind of protection that efforts to improve the situation there should be much lower priority than ones that handle the easier cases.
To be clear, I definitely think there’s a spectrum of attitudes towards security, centralisation, and other features of hazard databases, so I think you’re pointing to an important area of meaningful substantive disagreement!
Thanks for the write-up. Just adding a note on how this distinction has practical implications for how to design databases containing hazardous sequences that are required for gene synthesis screening systems.
With gene synthesis screening, companies want to stop bad actors from getting access to the physical DNA or RNA of potential pandemic pathogens. Now, let’s say researchers find the sequence of a novel pathogen that would likely spark a pandemic if released. Most would want this sequence to be added to synthesis screening databases. But some also want this database to be public. The information hazards involved in making such information publicly available could be large, especially if there is attached discussion of how exactly these sequences are dangerous.
Yup!
(I expected to see your comment link to SecureDNA, which has a cryptographic solution to screening synthesis without either (a) sending hazards to synthesizers or (b) sending reconstructible synthesis orders to others.)
Hi slg — great point about synthesis screening being a very concrete example where approaches to security can make a big difference.
One quibble I have: Your hyperlink seems to suggest that Diggans and Leproust advocate for a fully “public” database of annotated hazard sequences. But I think it’s worth noting that although they do use the phrase “publicly available” a couple of times, they also pretty explicitly discuss the idea of having such a database be accessible to synthesis providers only, which is a much smaller set and seems to carry significantly lower risks for misuse than truly public access. Relevant quote:
“Sustained funding and commitment will be required to build and maintain a database of risk-associated sequences, their known mechanisms of pathogenicity and the biological contexts in which these mechanisms can cause harm. This database (or at a minimum a screening capability making use of this database), to have maximum impact on global DNA synthesis screening, must be available to both domestic and international providers.”
Also worth noting the parenthetical about having providers use a screening mechanism with access to the database without having such direct access themselves, which seems like a nod to some of the features in, eg, SecureDNA’s approach.
I think benchtop synthesizers would change this quite a bit? Because then you need one of:
Ship the database on every benchtop, where it is at much higher risk of compromise.
Have benchtops send each synthesis request out for screening.
Something like Secure DNA’s approach, where the benchtop sends the order out for screening in a format that does not disclose it’s contents.
Yes, benchtop devices have significant ramifications!
Agreed, storing the database on-device does sound much harder to secure than some kind of distributed storage. Though, I can imagine that some customers will demand airgapped on-device solutions, where this challenge could present itself anyway.
Agreed, sending exact synthesis orders from devices to screeners seems undesirable/unviable, for a host of reasons.
But that’s consistent with my comment, which just meant to emphasise that I don’t read Diggans and Leproust as advocating for a fully “public” hazard database, as slg’s comment could be read to imply.
If your benchtop device user can modify the hardware to attempt to defeat the screening mechanism, the problem becomes orders of magnitude harder. I imagine that making a DNA sequence generating device that can’t be modified to make smallpox even if it’s in the middle of Pyongyang and the malicious user is the North Korean government is an essentially unsolvable problem—if nothing else, they can try to reverse engineer the device and build a similar one without any screening mechanism at all.
A bit tangential, but this raises an important point: in general, you’re looking for things that raise the bar for causing harm. If you can take smallpox synthesis from something where anyone who works in a lab with a benchtop can synthesize it without even opening in the machine to one where they would have to disassemble it, but that already increases the chance that someone else in the lab would notice.
It would be great to get to a place where we have systems that will provide reliable protection even from well-funded state actors, but (a) a lot of the risk comes from much easier cases like it becoming easier for an individual to cause harm and (b) we are so far from having that kind of protection that efforts to improve the situation there should be much lower priority than ones that handle the easier cases.
That’s a good pointer, thanks! I’ll drop the reference to Diggans and Leproust for now.
To be clear, I definitely think there’s a spectrum of attitudes towards security, centralisation, and other features of hazard databases, so I think you’re pointing to an important area of meaningful substantive disagreement!