Some ideas for career paths that I think have a very low chance of terrible outcomes and a reasonable chance to do a ton of good for the long-term future (I’m not claiming that they definitely will be net-positive, I’m claiming they are more than 10x more likely to be net positive than to be net negative):
Developing early warning systems for future pandemics (and related work) (technical bio work)
Strengthening the bioweapons convention and building better enforcement mechanisms (bio policy)
Predicting how fast powerful AI is going to be developed to get strategic clarity (AI strategy)
Developing theories of how to align AI and reasoning about how they could fail (AI alignment research)
Building institutions that are ready to govern AI effectively once it starts being transformative (AI governance)
Besides these, I think that almost all work longtermists work on today has a positive expected value, even if it has large downsides. Your comparison to deworming isn’t perfect. Failed deworming is not causing direct harm. It is still better to give money to ineffective deworming than to do nothing.
This is valuable, thank you. I really like the point on early warning systems for pandemics.
Regarding the bioweapons convention, I intuitively agree. I do have some concerns about how it could tip power balances (akin to how abortion bans tend to increase illegal abortions and put women at risk, but that’s a weak analogy). There is also a historical example of how the Geneva Disarmament Conference inspired Japan’s bioweapons program.
Predicting how fast powerful AI is going to be developed: That one seems value-neutral to me. It could help regular AI as much as AI safety. Why do you think it’s 10x more likely to be beneficial?
AI alignment research and AI governance: I would like to agree with you, and part of me does… I’ve outlined my hesitations in the comment below.
Re: bioweapons convention: Good point, so maybe not as straightforward as I described.
Re: predicting AI: You can always not publish the research you are doing or only inform safety-focused institutions about it. I agree that there are some possible downsides to knowing more precisely when AI will be developed, but there seem to be much worse downsides to not knowing when AI will be developed (mainly that nobody is preparing for it policy- and coordination-wise) I think the biggest risk is getting governments too excited about AI. So I’m actually not super confident that any work on this is 10x more likely to be positive.
Re: policy & alignment: I’m very confident, that there is some form of alignment work that is not speeding up capabilities, especially the more abstract one. Though I agree on interpretability. On policy, I would also be surprised if every avenue of governance was as risky as you describe. Especially laying out big picture strategies and monitoring AI development seem pretty low-risk.
Overall, I think you have done a good job scrutinizing my claims and I’m much less confident now. Still, I’d be really surprised if every type of longtermist work was as risky as your examples—especially for someone as safety-conscious as you are. (Actually, one very positive thing might be criticizing different approaches and showing their downsides)
I share your sentiment: there must be some form of alignment work that is not speeding up capabilities, some form of longtermist work that isn’t risky… right?
Why are the examples so elusive? I think this is the core of the present forum post.
15 years ago, when GiveWell started, the search for good interventions was difficult. It required a lot of research, trials, reasoning etc. to find the current recommendations. We are at a similar point for work targeting the far future… except that we can’t do experiments, don’t have feedback, don’t have historical examples[1], etc. This makes the question a much harder one. It also means that “do research on good interventions” isn’t a good answer either, since this research is so intractable.
Failed deworming is not causing direct harm. It is still better to give money to ineffective deworming than to do nothing.
Apologies in advance for being nitpicky. But you could consider the counterfactual where the money would instead go to another effective charity. A similar point holds for AI safety outreach: it may cause people to switch careers and move away from other promising areas, or cause people to stop earning to give.
Apologies in advance for being nitpicky. But you could consider the counterfactual where the money would instead go to another effective charity. A similar point holds for AI safety outreach: it may cause people to switch careers and move away from other promising areas, or cause people to stop earning to give.
Sorry if your bar for “reliable good” entails being clearly better than counterfactuals with high confidence, then afaict literally nothing in EA clears that bar. Certainly none of the other Givewell charities clear this bar.
I don’t mean to set an unreasonably high bar. Sorry if my comment came across that way.
It’s important to use the right counterfactual because work for the long-term future competes with GiveWell-style charities. This is clearly the message of 80000hours.org, for example. After all, we want to do the most good we can, and it’s not enough to do better than zero.
It’s important to use the right counterfactual because work for the long-term future competes with GiveWell-style charities
I’m probably confused about what you’re saying, but how is this different from saying that work on Givewell-style charities compete with the long-term future, and also donations to Givewell-style charities compete with each other?
Some ideas for career paths that I think have a very low chance of terrible outcomes and a reasonable chance to do a ton of good for the long-term future (I’m not claiming that they definitely will be net-positive, I’m claiming they are more than 10x more likely to be net positive than to be net negative):
Developing early warning systems for future pandemics (and related work) (technical bio work)
Strengthening the bioweapons convention and building better enforcement mechanisms (bio policy)
Predicting how fast powerful AI is going to be developed to get strategic clarity (AI strategy)
Developing theories of how to align AI and reasoning about how they could fail (AI alignment research)
Building institutions that are ready to govern AI effectively once it starts being transformative (AI governance)
Besides these, I think that almost all work longtermists work on today has a positive expected value, even if it has large downsides. Your comparison to deworming isn’t perfect. Failed deworming is not causing direct harm. It is still better to give money to ineffective deworming than to do nothing.
This is valuable, thank you. I really like the point on early warning systems for pandemics.
Regarding the bioweapons convention, I intuitively agree. I do have some concerns about how it could tip power balances (akin to how abortion bans tend to increase illegal abortions and put women at risk, but that’s a weak analogy). There is also a historical example of how the Geneva Disarmament Conference inspired Japan’s bioweapons program.
Predicting how fast powerful AI is going to be developed: That one seems value-neutral to me. It could help regular AI as much as AI safety. Why do you think it’s 10x more likely to be beneficial?
AI alignment research and AI governance: I would like to agree with you, and part of me does… I’ve outlined my hesitations in the comment below.
Re: bioweapons convention: Good point, so maybe not as straightforward as I described.
Re: predicting AI: You can always not publish the research you are doing or only inform safety-focused institutions about it. I agree that there are some possible downsides to knowing more precisely when AI will be developed, but there seem to be much worse downsides to not knowing when AI will be developed (mainly that nobody is preparing for it policy- and coordination-wise)
I think the biggest risk is getting governments too excited about AI. So I’m actually not super confident that any work on this is 10x more likely to be positive.
Re: policy & alignment: I’m very confident, that there is some form of alignment work that is not speeding up capabilities, especially the more abstract one. Though I agree on interpretability. On policy, I would also be surprised if every avenue of governance was as risky as you describe. Especially laying out big picture strategies and monitoring AI development seem pretty low-risk.
Overall, I think you have done a good job scrutinizing my claims and I’m much less confident now. Still, I’d be really surprised if every type of longtermist work was as risky as your examples—especially for someone as safety-conscious as you are. (Actually, one very positive thing might be criticizing different approaches and showing their downsides)
Thanks a lot for your responses!
I share your sentiment: there must be some form of alignment work that is not speeding up capabilities, some form of longtermist work that isn’t risky… right?
Why are the examples so elusive? I think this is the core of the present forum post.
15 years ago, when GiveWell started, the search for good interventions was difficult. It required a lot of research, trials, reasoning etc. to find the current recommendations. We are at a similar point for work targeting the far future… except that we can’t do experiments, don’t have feedback, don’t have historical examples[1], etc. This makes the question a much harder one. It also means that “do research on good interventions” isn’t a good answer either, since this research is so intractable.
Ian Morris in this podcast episode discusses to what degree history is contingent, i.e., past events have influenced the future for a long time.
Apologies in advance for being nitpicky. But you could consider the counterfactual where the money would instead go to another effective charity. A similar point holds for AI safety outreach: it may cause people to switch careers and move away from other promising areas, or cause people to stop earning to give.
Sorry if your bar for “reliable good” entails being clearly better than counterfactuals with high confidence, then afaict literally nothing in EA clears that bar. Certainly none of the other Givewell charities clear this bar.
I don’t mean to set an unreasonably high bar. Sorry if my comment came across that way.
It’s important to use the right counterfactual because work for the long-term future competes with GiveWell-style charities. This is clearly the message of 80000hours.org, for example. After all, we want to do the most good we can, and it’s not enough to do better than zero.
I’m probably confused about what you’re saying, but how is this different from saying that work on Givewell-style charities compete with the long-term future, and also donations to Givewell-style charities compete with each other?