I evaluate biological language models for general capabilities and safety risk. PhD candidate in Mohammed AlQuraishi’s lab at Columbia. Incoming MATS fellow with Fourth Eon Bio, current Pivotal Frontier Biodefense Fellow, former Center for Health Security fellow.
Noga Aharony
Karma: 179
That is true. I’m a part of a team building a genome language model. It’s really unclear even to us which models are the best models! and a lot of these fundamental questions can also guide model design.
In that way, if we do it right, and thoughtfully, it’s possible to both become a resource for model construction and improve their safeguards and security.
We need more empirical research on biological AI model safety
I would advise against. It is too likely to be used for terror purposes and be directly counterproductive. There is plenty of evidence as Hamas is not shy about it. Hamas has historically taxed anything coming into Gaza (especially through the tunnels). It also repurposed aid for military use many times (it has a propaganda video where it digs up pipes installed by the EU and used them as rockets). Today, Ismail Haniyeh, the head of Hamas, called for a “money Jihad” where people donate to Gaza so that Hamas can keep waging war. Generally the exact number is unknown but it is estimated Hamas used billions of dollars in aid to purchase and smuggle arms, and many of its leaders also embezzle much of the money. In addition there is already a backup of hundreds of trucks with humanitarian aid waiting to arrive in Gaza, so aid is not the bottleneck. All we can hope for is that the fighting finishes as soon as possible. So I think there are 2 options: I believe I saw an initiative to set up aid airdrops over Gaza. It was by Ahmed Alkhoutib at the FDD I believe? Maybe there is a way to support it. Additionally, Donating to rebuild Gaza afterwards though (specially after most of the world has moved on to the next thing) could be valuable.
I have a million arguments against this post and maybe in the past I’d have engaged with each of your points and arguments more thoroughly but I lost my patience for these things. I’ll just say this: know that this post (as well as the comments), how they are completely ignorant of the emotional and economic ramifications of unplanned pregnancies, how they completely ignore self-autonomy, are the last straw. I’ve been in this movement for seven years, modeled my career after it, but especially in light of recent events it made me realize I just can’t keep justifying the direction it’s taking. This is the tip of the iceberg of why this movement is so male-dominated. And no, I am not outraged of this because it’s not ‘woke’ to discuss these things, or because it’s taboo. These things never bothered me. It’s because by adopting this perspective you consider to be more encompassing you’re really just falling to the same pitfall as the majority of people in human history and ignoring half of us, whether we currently live or will live in the future. Signing off.
This hurts but it checks out.
Okay I’ll address the rest of the argument. You’re also not giving a lot of context. It’s hard to understand but based on your whole comment I can also see it being possible that you bumped into situations where people were trying to sort out interpersonal issues privately, and you got wind of it and tried to make it public.
There is a world of difference between those situations and situations where people are not intellectually honest, which is most of the situations OP describes and discusses.
And it makes the last part of your comment even more uncalled for.
[Question] How should I keep mentoring in light of the FTX scandal?
I’m sorry I just disagree. We are an applied ethics movement. Maybe the only one in the world. We should hold ourselves to the highest ethical standards. And yet we benefitted from a scheme that, legal or not, ruined a lot of people’s lives. Utilitarianism or not, we need to do everything we can to atone. If we don’t, it could ruin our psyches, our ethical standards, our perception and our trajectory.
(1) Effects of cybersecurity on geopolitics, or individual privacy. These are two different areas and they seem to me like one bad actor can cause a lot of suffering or lead to suboptimal futures, but I don’t know of any EAs who looked deeply into it.
(2) Reproductive health and the costs of childbearing, possibly from a policy angle. I think as a community we decided to bite the bullet and become total utilitarians, and I see some discussions on how it should play out in terms of contraception and choosing to have more children but all of these come across to me as not very well-informed. There are only 2 posts I found about it are by isabel and they touch very specific topics. So I think an analysis of why people are having fewer children, what policies will help people choose to have more children, and a thorough analysis to settle all the discussions around contraception and abortion, including an attempt to quantify the suffering and counterfactual involved in childbearing and childcare, would be appreciated.
Tae so I guess now you can tell your friend that you’ve Called off the EAs :P
I don’t think this post made the strong assumptions about population ethics you assume.
More unplanned pregnancies does not necessarily equal larger population. In fact, at the very beginning the post highlights that there are twice as many abortions as unplanned births and more unsafe abortions than unplanned births. Including the still births, that is a lot of preventable human suffering. Is it worth those unplanned births?
I also think it’s a bit ignorant to deny sub-Saharan Africa a technology we enjoy - would you also be against birth control use in the US?
Probably not, because the introduction of birth control and family planning did a lot of good—it allowed women to take a lot more control over their lives, some of it translated into the flood of women entering the workplace in the 60′s. Roe vs. Wade alone was correlated with a much reduced crime rate in the US. Without contraception, I would have never dared to pursue a PhD and dedicate my career to EA. Is a world where half the population can’t plan their futures a better world?
Additionally, more birth control does not equal fewer children. Israel has more access to birth control and legal abortion than many western countries, and it has a higher birth rate than the global average. Secular women in Israel alone have a higher birth rate than any other OECD country. Similarly, Eastern Europe has a lower birth rate than Western Europe despite having more strict control over abortion and birth control.
So the counterfactual does not mean fewer live births necessarily. And maybe it’s wiser to try and shape a culture to be more pro-natalist rather than rob people of their choices. With the longtermist framework in mind and the timeframes of childbearing and raising we have the time to do these things, rather than reach for interventions that maximize the single metric that is population size in the short term but could have negative implications on culture and suffering.
I think in order to be against an intervention that gives people more choice you need to make a very strong argument. I also think that you’re probably reaching this conclusion because you’re underestimating the burden of childbearing. There is a reason why so many women choose to have unsafe abortions rather than give birth.
I actually have given artificial wombs a little thought. I do think they’d be great: they could eliminate a very common suffering, give more options to LGBTQ people, aid in civilizational resilience, and definitely increase the number of wanted children people have in practice. They make sense within many different ethical frameworks.
I also think we’re very, very far from them. I’m a systems biologist in a lab that also ventures into reproductive health, and we ostensibly know very little about the process of pregnancy. My lab is using the most cutting-edge methods to prove very specific and fundamental things. So at the same time, I am skeptical we will see it in our lifetimes, if ever.
(1) I never purported that communicating that monkeypox is transmitted mostly among MSM is tone-deaf in itself. Like I wrote at the end of my comment, I think this information is important. I think it is the way in which you communicated that made it come across as tone-deaf.
(2) the definition of an STI is:infections that are passed from one person to another through sexual contact. The contact is usually vaginal, oral, or anal sex. But sometimes they can spread through other intimate physical contact. This is because some STDs, like herpes and HPV, are spread by skin-to-skin contact. [link]
Based on this definition monkeypox is just as much an STI as herpes and HIV. A short google search about it seems to communicate that it’s not being called an STI right now since it can cause stigma and stigma can hurt response efforts. But regardless definition has nothing to do with it. You can include it on an STI panel even if it’s not strictly an STI, if it is more effective to do that to curb its spread.
(3) I do maintain that paying people to abstain is not an effective intervention. Sex isn’t exactly a rational pursuit, and it’d be impossible to reinforce that. Also the CDC does encourage condom use and there is an early study purporting to find monkeypox in semen.
I’m a microbiologist but not an expert in STIs and to me if this is being passed among mostly MSM, reducing the types of skin-to-skin contact that MSM have more often than others, should reduce the spread. Even if it won’t bring cases down to 0.
I think your post comes across as a little tone-deaf in a way that can be counter-productive.
“Is it worth worrying about?” “Basically no. The disease remains highly confined to the gay community.”
Comes across as a disregard to the LGBTQ community. Mostly because of the historical context in which we live.
This sentence echoes many things that were said during the AIDS pandemic, which prior to COVID was the closest we got to GCBR in the past 100 years and in many metrics is closer to a GBCR than COVID. Historically there was a very intentional disregard for the AIDS pandemic because its devastating effects were mostly confined within the LGBTQ community. [link] The Reagan administration famously told the CDC “act pretty and do as little as possible.” This policy lead to a lot of preventable deaths and suffering.
“simple solutions like paying gay men to abstain from sex becomes impractical”
Is it a simple solution? I never heard of any sort of successful policy where people were paid to stop having sex. I think abstinance-only sex-ed shows that it’s very hard to convince anyone to stop having sex even when people are taught to internalize to avoid it categorically and the stakes are high (other STIs, teen pregnancy...). I think you should either remove this sentence or give some thought & add something about encouraging using protection and STI testing that includes monkeypox. More practically, I do think these policies have a better track record too.
This information is important, and you spent significant time reading on it and writing it up. It’s important to communicate it without hurting a whole group of people, and I think that with a bit more thinking it can be done.
I have an undergraduate degree in Neuroscience and I am very skeptical that such a drug can be found. Can’t talk to these specific genes in particular but genes are often turned on in different ways and in different parts of the brain, and lead to different effects based on which genes are turned on or off with them. Now because of the gene interaction in the background, the same receptor can cause a reverse effect when activated in one part of the brain or another. Additionally, each neurotransmitter has upwards of 20 different receptors in the brain. Also, every single synapse has multiple different receptors.
Drugs on the other hand hit all the receptors of a certain type in all of the brain. It’s a very indiscriminant effect. The diseases that medications seem to be capable of fixing are diseases that lead to a brain-wide shortage of a neurotransmitter, which are fixed by prescribing the transmitter or a medication that causes its recycling and reuptake (like levo-dopa for parkinson’s).I’ve head a neuroscientist say once that treating disease with medication is like “fixing a car by pouring oil indiscriminantly under the hood,” and I agree. One example for that is SSRIs for depression. Serotonin does a lot of things in the brain - it also has an important role in motor functions. SSRIs help depression, but the effect isn’t very direct (people describe having some additional energy, not necessarily being ‘cured’) and has many, many side effects because they also affect all the other serotonin concentrations all over the brain.
Even when compared to mood control, sleep is also a very delicate, very carefully-orchestrated process in the brain. It’s controlled by similar centres as breathing, heart rate, and balance. Sure there are some rare genes that contribute to less sleep but I doubt you’ll find a drug that will not have severe side effects.
On a different note—I haven’t read her studies in particular but the main reason I left neuroscience was because the standard practices, especially in human and animal research, seemed to me so lacking. The recruitment process which you mentioned alone can be extremely biased, and is extremely hard to control for. The sample sizes are also always quite small. I wouldn’t be surprised if this group is skewed towards top 1/300th of people with this gene, and that they have numerous other genes and environmental factors that makes them more ‘protected’ from deprivation but also more likely to reach recruitment for this type of study.
TL;DR gene expression is tightly controlled over time and space and drugs are not. This is especially true for sleep. I am skeptic drugs will work at this fine scale since no current drugs do.
Only if you’re strictly total utilitarian. But won’t all these things drop us into a situation like in the repugnant conclusion, where we would just get more people (especially women) living in worse conditions, with fewer choices?
Women in fact already are having fewer children than they want. Me and a lot of women around me would want to have children earlier than we are planning on, but we couldn’t do it without dropping three levels down the socioeconomic ladder and having to give up on goals we’ve been investing in since elementary school. We won’t only be quashing our potential but that of the children we would raise once we do have the resources to invest in them. Is that really a better future?
If EA really wants to increase fertility at a global level I think some hard thought needs to be given to how to change the social structures and incentives so that women can have children without having to also disproportionately carry such a large burden through pregnancy, birth, and childcare.
So I took a class on sleep and I read some papers about it. Here are some thoughts:
I do think it’s a cause area with a very clear solution: train more imagery rehearsal therapists, and disperse them/make them available through telehealth. I read the papers and it does seem highly effective. I think a lot of people would have enrolled even at high cost if they knew it existed/they had access to it. And then after there were more therapists we could probably talk about raising awareness/providing these servies for free and at places where they are more needed (women’s shelters, refugee camps, etc)
I think you would have made a more powerful argument if you framed it from the point of view of PTSD. Nightmares are a staple of PTSD and IRT is probably the most effective intervention for PTSD! I don’t know the numbers but PTSD is common, and I think if you looked up PTSD you would get better stats about nightmares. I do realize that you didn’t frame it like this because you seem to have a very atypical type of nightmares (lots of variety, and the nightmares aren’t related to a life event that is replayed in your head) and I am sorry for what you have to go throug
Hi, I’m an EA working in a prominent antibiotics resistance lab. From my point of view, antibiotics resistance is a big issue, resistance is growing, HOWEVER, there are actually a lot of medications in the pipeline that are effective but weren’t brought to market because it’s not financially viable right now (that I heard in a talk by Floyd Romesberg). There are also other interesting therapies like antimicrobial peptides (explanation here: https://www.ncbi.nlm.nih.gov/pubmed/15761415 ). My lab developed an ML model that will help doctors in Israel select the most optimal antibiotic (https://www.nature.com/articles/s41591-019-0503-6 ) and also sequencing is getting so good that it’s not improbable that in 10 years, IF THERE WILL BE A NEED you’d come to the Doctor, he’ll swab you and get a full antibiotics resistance profile, and prescribe you a medication accordingly.
The reason that these medications and technologies aren’t implemented is that antibiotics resistance is still more or less manageable, it is not at a crisis level. Unlike pandemics, a global antibiotics resistance crisis doesn’t rise in one day (though it is quick! there was essentially very little 70 years ago) so that means that was the situation to become worse, it is possible to draw on all these technologies to form some sort of defense. However, it is true that many people die of antibiotics a year and it will likely affect developing countries disproportionately. All in all, it is true that antibiotics resistance is a threat, but it’s just not a GBCR.
This is a very creative idea. A few of my thoughts:
1) Like you pointed out, many diseases that can be very virulent in humans (Ebola, Nipah, Coronavirus) are not so virulent in bats, so there would be many instances where a vaccination program will be very valuable for humans but have very little (and maybe even negative, due to side effects) effect on the wild animal population.
2) Diseases tend to be harmful in dense, homogeneous populations. Like people, or livestock. I don’t know how much disease really impacts wild animal suffering—there could be surprisingly small infection rates and impact.
3) Before vaccines we need survellience, not only because it will answer the question of how much disease affects wild animal suffering, but also because it’s impossible to produce a vaccine without a sequence.
4) Take into account that vaccine development involves a LOT of test subjects—which in this case will have to be wild animals grown in laboratory conditions. Could be thousands or even more. Even though a vaccine is developed, it might not be implemented because it fell through so any process like this could have a large unintentional negative impact. I’d be interested to know how vaccines are developed for animals (especially non-domesticated ones) because my guess is that the standards for side effects would be lower than humans, even only because it’s harder to detect side effects in animals. I’m really curious about the gel that was developed for bat white-nose syndrome!
I think it’s a real conflict. I think the situation with BAIMs is different: current models don’t seem to generalize very well at all, and the consensus seems to be that it’s because we don’t have enough publicly available data right now. So in this way, it’s not clear if “the genie is out of the box.”
At the same time yes, actually stopping data collection is both unrealistic and will have major repercussions to a lot of serious improvements to human health in the short term. I think tiered access can mitigate some of that, not all.
I do think Biosecurity is very different than cybersecurity so unfortunately the same interventions won’t work, and we will need different thinking. The whole field is a lot more decentralized, there are ostensibly no security practices to patch against potential attacks. Chris Rodriguez described it better than I could here.