Relatedly, an area where I think arXiv could have a huge impact (in both biosecurity and AI) would be setting standards for easy-to-implement manged access to algorithms and datasets.
Given the misuse potential of research objects like code, datasets, and protocols, approaches for risk mitigation are needed. Across digital research objects, there appears to be a trend towards increased modularisation, i.e., sharing information in dedicated, purpose built repositories, in contrast to supplementary materials. This modularisation may allow differential access to research products according to the risk that they represent. Curated repositories with greater access control could be used that allow reuse and verification when full public disclosure of a research object is inadvisable. Such repositories are already critical for life sciences that deal with personally identifiable information.
+1 to this. arXiv could play a big role in contributing to a norm around not publishing dual use bio research. There are challenges of screening large numbers of papers, but they can be met. See here for an example from ASM. bioRxiv may or may not be screening already, but they aren’t sharing information about their practices. It would be helpful if they were more vocal about the importance of not publishing dangerous information.
I think implementation is going to be very hard. TL;DR: arxiv can’t just reject papers:
If arxiv simply rejects some paper, the author might, as a naive example, tweet about “even arxiv think this is such a big deal that they won’t publish my paper! but science should be FREE!” and it might get even more traction.
I think a better way to handle this would be to reply to the authors and talk to them nicely about why we ask them to keep this secret, even though they worked really hard on it.
Replying to authors seems good to me, but I recommend talking more to biosecurity experts at FHI / CSER / SERI for advice (like Daniel Green, Tessa or ASB) because I think information security is complicated and many actions can backfire
I think it’s vastly worse if people don’t publish. It’s much better if the entire world can contribute research on how to handle a threat, than if we just count on nobody else having the same (potentially harmful) idea while we do our secret research, and hitting an unprepared world.
So don’t publish an explicit list of new pathogen genomes? Sure; don’t publish anything describing them or the technology used to create them? Bad.
I think a very strong reason is needed for something not to be published. I don’t think it’s that complicated—better to err on the side of publishing something, than hinder the world’s preparation for threats (and prevent the positive impact the same technologies can have).
Edit: linking to Tessa’s comment for a more cautious and nuanced direction that I still agree with.
Keeping an eye out for dual use biotechnology research, metascience stuff
Relatedly, an area where I think arXiv could have a huge impact (in both biosecurity and AI) would be setting standards for easy-to-implement manged access to algorithms and datasets.
This is something called for in Biosecurity in an Age of Open Science:
This sort of idea also appears in New ideas for mitigating biotechnology misuse under responsible access to genetic sequences and in Dual use of artificial-intelligence-powered drug discovery as a proposal for managing risks from algorithmically designed toxins.
Do you mean “biotechnology that could lead to a pandemic and it’s better if nobody publishes”?
+1 to this. arXiv could play a big role in contributing to a norm around not publishing dual use bio research. There are challenges of screening large numbers of papers, but they can be met. See here for an example from ASM. bioRxiv may or may not be screening already, but they aren’t sharing information about their practices. It would be helpful if they were more vocal about the importance of not publishing dangerous information.
I agree with the goal, +1.
I think implementation is going to be very hard. TL;DR: arxiv can’t just reject papers:
If arxiv simply rejects some paper, the author might, as a naive example, tweet about “even arxiv think this is such a big deal that they won’t publish my paper! but science should be FREE!” and it might get even more traction.
I think a better way to handle this would be to reply to the authors and talk to them nicely about why we ask them to keep this secret, even though they worked really hard on it.
Any thoughts about this?
Anyway, it is worth trying.
Replying to authors seems good to me, but I recommend talking more to biosecurity experts at FHI / CSER / SERI for advice (like Daniel Green, Tessa or ASB) because I think information security is complicated and many actions can backfire
Thanks!
I don’t know who ASB is, would you somehow connect us (for example, forward the post to them)?
(Daniel Greene and Tessa replied to this post)
ASB commented too—https://forum.effectivealtruism.org/posts/cxcPrwMQCh5tg2JLm/?commentId=ZQGutoEyKEBgurCg4
Thanks!
yeah
I think it’s vastly worse if people don’t publish. It’s much better if the entire world can contribute research on how to handle a threat, than if we just count on nobody else having the same (potentially harmful) idea while we do our secret research, and hitting an unprepared world.
So don’t publish an explicit list of new pathogen genomes? Sure; don’t publish anything describing them or the technology used to create them? Bad.
How about publishing the instructions for building a “3d printer” that prints pathogens, and that people can use at home?
This is complicated, figuring out what is more dangerous than not.
I think a very strong reason is needed for something not to be published. I don’t think it’s that complicated—better to err on the side of publishing something, than hinder the world’s preparation for threats (and prevent the positive impact the same technologies can have).
Edit: linking to Tessa’s comment for a more cautious and nuanced direction that I still agree with.
Would you defer you opinion to whatever the people working fulltime on preparing for these threats think?
Maybe if you ask a wide enough group of them? But for sure not “what a small group of them selected to agree with infohazard assumptions think”.