I agree with the statement “It’s OK for some people not to go into AI” but strongly disagree with the statement “It’s OK not to go into AI, even if you don’t have a good reason”. One can list increasingly unreasonable statements like:
AI safety is an important problem.
Every EA should either work on AI safety or have a good reason why they’re not.
EAs should introduce themselves with what cause area they’re working on, and their reason for not working on AI if applicable.
Literally everyone should work on AI safety; there are no excuses not to.
I want to remind people of (1) and defend something between (2) and (3).
Our goal as world optimizers is to find the best thing we could possibly be doing, subject to various constraints like non-consequentialist goals and limited information. This means that for every notable action worth considering, we should have a good reason why we’re not doing it. And (2) is just a special case of this, since working on alignment (technical or otherwise) is definitely a notable action. Because there are good reasons to work on AI safety, you need to have a better reason not to.
Having 100x more traction on the problem of making Malawi (0.25% of the world population) a developed country is not a good enough reason, because under most reasonable moral views, preventing human extinction is >100x better than raising the standard of living of 0.25% of the world population.
Note that there are many people who should not work on AI safety because they have >400x more traction on problems 400x smaller, or whatever.
Wanting to expand what EA thinks is possible is not a sufficient reason, because you also have to argue that the expected value of this is higher than investing into causes we already know about.
Holden Karnofsky makes the case against “cause X” here: AI risk is already really large in scale; they essentially say “this century we’re going to figure out what kind of civilization is going to tile the entire galaxy”, and it’s hard to find something larger in scale than that; x-risks are also neglected. It’s hard for tractability differences to overwhelm large differences in scale/neglectedness.
Not having thought about the arguments is not a good enough reason. Reality is the judge of your actions and takes no excuses.
Majoring in something other than math/CS is not a good enough reason, because your current skills or interest areas don’t completely determine your career comparative advantage
Finding the arguments for AI risk unconvincing is not a reason to just not work on AI risk, because if the arguments are wrong, this implies lots of effort on alignment is wasted and we need to shift billions of dollars away from it (and if they have nonessential flaws this could change research directions within alignment), so you should write counterarguments up to allow the EA community to correctly allocate its resources.
also, if working on alignment is your comparative advantage, it might make sense to work on even if the arguments have a 10% chance of being right.
Some potential sufficient reasons
“I tried 3 different kinds of AI safety research and was worse than useless at all of them, and have various reasons not to do longtermist community-building either”
“I have 100x more traction on biorisk and think biorisk is 20x smaller than AI risk”
“I have 100x more traction on making the entire continent of Africa as developed as the USA, plus person-affecting views, plus my AI timelines are long enough that I can make a difference before AGI happens”
“I think suffering-focused ethics are correct, so I would rather prevent suffering now than have a small chance of preventing human extinction”
“I can become literally the best X in the world, or a mediocre AI safety community-builder. I think the
“I have a really good story for why the arguments for AI risk are wrong and have spent the last month finding the strongest version of my counterarguments; this will direct lots of resources to preventing various moral atrocities in worlds where I am right”
edit: after thinking about it more I don’t endorse the below paragraph
I also want to defend (3) to some extent. Introducing yourself with your target cause area and reasons for working on it seems like a pretty natural and good thing. In particular it forces you to have a good reason for doing what you’re doing. But there are other benefits too: it’s an obvious conversation starter, and when half the people at the EA event are working on AI safety it just carries a lot of information.
Upvoted for explaining your stance clearly, though I’m unclear on what you see as the further implications of:
Because there are good reasons to work on AI safety, you need to have a better reason not to.
This is true about many good things a person could do. Some people see AI safety as a special case because they think it’s literally the most good thing, but other people see other causes the same way — and I don’t think we want to make any particular thing a default “justify if not X”.
(FWIW, I’m not sure you actually want AI to be this kind of default — you never say so — but that’s the feeling I got from this comment.)
Note that there are many people who should not work on AI safety because they have >400x more traction on problems 400x smaller, or whatever.
When someone in EA tells me they work on X, my default assumption is that they think their (traction on X * assumed size of X) is higher than the same number would be for any other thing. Maybe I’m wrong, because they’re in the process of retraining or got rejected from all the jobs in Y or something. But I don’t see it as my job to make them explain to me why they did X instead of Y, unless they’re asking me for career advice or something.
There may be exceptional cases where someone is working on something really unusual, but in those cases, I aim for a vibe of “curious and interested” rather than “expecting justification”. At a recent San Diego meetup, I met a dentist and was interested to learn how he chose dentistry; as it turns out, his reasoning was excellent (and I learned a lot about the dental business).
Finding the arguments for AI risk unconvincing is not a reason to just not work on AI risk, because if the arguments are wrong, this implies lots of effort on alignment is wasted and we need to shift billions of dollars away from it (and if they have nonessential flaws this could change research directions within alignment), so you should write counterarguments up to allow the EA community to correctly allocate its resources.
This point carries over to global health, right? If someone finds EA strategy in that area unconvincing, do they need to justify why they aren’t writing up their arguments?
In theory, maybe it applies more to global health, since the community spends much more money on global health than AI? (Possibly more effort, too, though I could see that going either way.)
This is true about many good things a person could do. Some people see AI safety as a special case because they think it’s literally the most good thing, but other people see other causes the same way — and I don’t think we want to make any particular thing a default “justify if not X”.
I’m unsure how much I want AI safety to be the default, there are a lot of factors pushing in both directions. But I think one should have a reason why one isn’t doing each of the top ~10 things one could, and for a lot of people AI safety (not necessarily technical research) should be on this list.
When someone in EA tells me they work on X, my default assumption is that they think their (traction on X * assumed size of X) is higher than the same number would be for any other thing. Maybe I’m wrong, because they’re in the process of retraining or got rejected from all the jobs in Y or something. But I don’t see it as my job to make them explain to me why they did X instead of Y, unless they’re asking me for career advice or something.
My guess is that the median person who filled out the EA survey isn’t being consistent in this way. I expect that they could have a one-hour 1-1 with a top community-builder that makes them realize they could be doing something at least 10% better. This is a crux for me.
Separately, I do feel a bit weird about making every conversation into a career advice conversation, but often this seems like the highest impact thing.
If someone finds EA strategy in [global health] unconvincing, do they need to justify why they aren’t writing up their arguments?
This was thought-provoking for me. I think existingposts of similar types were hugely impactful. If money were a bottleneck for AI safety and I thought money currently spent on global health should be reallocated to AI safety, writing up some document on this would be among the best things I could be doing. I suppose in general it also depends on one’s writing skill.
My guess is that the median person who filled out the EA survey isn’t being consistent in this way. I expect that they could have a one-hour 1-1 with a top community-builder that makes them realize they could be doing something at least 10% better. This is a crux for me.
I agree with most of this. (I think that other people in EA usually think they’re doing roughly the best thing for their skills/beliefs, but I don’t think they’re usually correct.)
I don’t know about “top community builder”, unless we tautologically define that as “person who’s really good at giving career/trajectory advice”. I think you could be great at building or running a group and also bad at giving advice. (There are several ways to be bad at giving advice — you might be ignorant of good options, bad at surfacing key features of a person’s situation, bad at securing someone’s trust, etc.)
Separately, I do feel a bit weird about making every conversation into a career advice conversation, but often this seems like the highest impact thing.
I’m thinking about conversations in the vein of an EAG speed meeting, where you’re meeting a new person and learning about what they do for a few minutes. If someone comes to EAG and all their speed meetings turn into career advice with an overtone of “you’re probably doing something wrong”, that seems exhausting/dispiriting and unlikely to help (if they aren’t looking for help). I’ve heard from a lot of people who had this experience at an event, and it often made them less interested in further engagement.
If I were going to have an hour-long, in-depth conversation with someone about their work, even if they weren’t specifically asking for advice, I wouldn’t be surprised if we eventually got into probing questions about how they made their choices (and I hope they’d challenge me about my choices, too!). But I wouldn’t try to ask probing questions unprompted in a brief conversation unless someone said something that sounded very off-base to me.
I disagree with 2) because I think the movement will be able to get more done with more diverse backgrounds of people who are really good at different things. Even if AI is the most important thing, we need people who understand communications, policy, organizing grassroots movements, and also people who are good at completely unrelated fields who can understand the impact of AI on their field (manufacturing, agricuture, shipping logistics, etc) though there aren’t those opportunities to do that work directly in AI right now.
I strong upvoted this because: 1) I think AI governance is a big deal (the argument for this has been fleshed out elsewhere by others in the community) and 2) I think this comment is directionally correct beyond the AI governance bit even if I don’t think it quite fully fleshes out the case for it (I’ll have a go at fleshing out the case when I have more time but this is a time-consuming thing to do and my first attempt will be crap even if there is actually something to it).
I think that strong upvoting was appropriate because: 1) stating beliefs that go against the perceived consensus view is hard and takes courage 2) the only way the effective altruism community develops new good ideas is if people feel they have permission to state views that are different from the community “accepted” view.
I think some example steps for forming new good ideas are: 1) someone states, without a fully fleshed out case, what they believe 2) others then think about whether that seems true to them and begin to flesh out reasons for their gut-level intuition 3) other people pushback on those reasons and point out the nuance 4) the people who initially have the gut-level hunch that the statement is true either change their minds or iterate their argument so it incorporates the nuance that others have pointed out for them. If the latter happens then, 5) More nuanced versions of the arguments are written up and steps 3 to 5 repeat themselves as much as necessary for the new good ideas to have a fleshed out case for them.
I agree with the statement “It’s OK for some people not to go into AI” but strongly disagree with the statement “It’s OK not to go into AI, even if you don’t have a good reason”. One can list increasingly unreasonable statements like:
AI safety is an important problem.
Every EA should either work on AI safety or have a good reason why they’re not.
EAs should introduce themselves with what cause area they’re working on, and their reason for not working on AI if applicable.
Literally everyone should work on AI safety; there are no excuses not to.
I want to remind people of (1) and defend something between (2) and (3).
Our goal as world optimizers is to find the best thing we could possibly be doing, subject to various constraints like non-consequentialist goals and limited information. This means that for every notable action worth considering, we should have a good reason why we’re not doing it. And (2) is just a special case of this, since working on alignment (technical or otherwise) is definitely a notable action. Because there are good reasons to work on AI safety, you need to have a better reason not to.
Having 100x more traction on the problem of making Malawi (0.25% of the world population) a developed country is not a good enough reason, because under most reasonable moral views, preventing human extinction is >100x better than raising the standard of living of 0.25% of the world population.
Note that there are many people who should not work on AI safety because they have >400x more traction on problems 400x smaller, or whatever.
Wanting to expand what EA thinks is possible is not a sufficient reason, because you also have to argue that the expected value of this is higher than investing into causes we already know about.
Holden Karnofsky makes the case against “cause X” here: AI risk is already really large in scale; they essentially say “this century we’re going to figure out what kind of civilization is going to tile the entire galaxy”, and it’s hard to find something larger in scale than that; x-risks are also neglected. It’s hard for tractability differences to overwhelm large differences in scale/neglectedness.
Not having thought about the arguments is not a good enough reason. Reality is the judge of your actions and takes no excuses.
Majoring in something other than math/CS is not a good enough reason, because your current skills or interest areas don’t completely determine your career comparative advantage
Finding the arguments for AI risk unconvincing is not a reason to just not work on AI risk, because if the arguments are wrong, this implies lots of effort on alignment is wasted and we need to shift billions of dollars away from it (and if they have nonessential flaws this could change research directions within alignment), so you should write counterarguments up to allow the EA community to correctly allocate its resources.
also, if working on alignment is your comparative advantage, it might make sense to work on even if the arguments have a 10% chance of being right.
Some potential sufficient reasons
“I tried 3 different kinds of AI safety research and was worse than useless at all of them, and have various reasons not to do longtermist community-building either”
“I have 100x more traction on biorisk and think biorisk is 20x smaller than AI risk”
“I have 100x more traction on making the entire continent of Africa as developed as the USA, plus person-affecting views, plus my AI timelines are long enough that I can make a difference before AGI happens”
“I think suffering-focused ethics are correct, so I would rather prevent suffering now than have a small chance of preventing human extinction”
“I can become literally the best X in the world, or a mediocre AI safety community-builder. I think the
“I have a really good story for why the arguments for AI risk are wrong and have spent the last month finding the strongest version of my counterarguments; this will direct lots of resources to preventing various moral atrocities in worlds where I am right”
edit: after thinking about it more I don’t endorse the below paragraph
I also want to defend (3) to some extent. Introducing yourself with your target cause area and reasons for working on it seems like a pretty natural and good thing. In particular it forces you to have a good reason for doing what you’re doing. But there are other benefits too: it’s an obvious conversation starter, and when half the people at the EA event are working on AI safety it just carries a lot of information.
Upvoted for explaining your stance clearly, though I’m unclear on what you see as the further implications of:
This is true about many good things a person could do. Some people see AI safety as a special case because they think it’s literally the most good thing, but other people see other causes the same way — and I don’t think we want to make any particular thing a default “justify if not X”.
(FWIW, I’m not sure you actually want AI to be this kind of default — you never say so — but that’s the feeling I got from this comment.)
When someone in EA tells me they work on X, my default assumption is that they think their (traction on X * assumed size of X) is higher than the same number would be for any other thing. Maybe I’m wrong, because they’re in the process of retraining or got rejected from all the jobs in Y or something. But I don’t see it as my job to make them explain to me why they did X instead of Y, unless they’re asking me for career advice or something.
There may be exceptional cases where someone is working on something really unusual, but in those cases, I aim for a vibe of “curious and interested” rather than “expecting justification”. At a recent San Diego meetup, I met a dentist and was interested to learn how he chose dentistry; as it turns out, his reasoning was excellent (and I learned a lot about the dental business).
This point carries over to global health, right? If someone finds EA strategy in that area unconvincing, do they need to justify why they aren’t writing up their arguments?
In theory, maybe it applies more to global health, since the community spends much more money on global health than AI? (Possibly more effort, too, though I could see that going either way.)
Thanks for the good reply.
I’m unsure how much I want AI safety to be the default, there are a lot of factors pushing in both directions. But I think one should have a reason why one isn’t doing each of the top ~10 things one could, and for a lot of people AI safety (not necessarily technical research) should be on this list.
My guess is that the median person who filled out the EA survey isn’t being consistent in this way. I expect that they could have a one-hour 1-1 with a top community-builder that makes them realize they could be doing something at least 10% better. This is a crux for me.
Separately, I do feel a bit weird about making every conversation into a career advice conversation, but often this seems like the highest impact thing.
This was thought-provoking for me. I think existing posts of similar types were hugely impactful. If money were a bottleneck for AI safety and I thought money currently spent on global health should be reallocated to AI safety, writing up some document on this would be among the best things I could be doing. I suppose in general it also depends on one’s writing skill.
I agree with most of this. (I think that other people in EA usually think they’re doing roughly the best thing for their skills/beliefs, but I don’t think they’re usually correct.)
I don’t know about “top community builder”, unless we tautologically define that as “person who’s really good at giving career/trajectory advice”. I think you could be great at building or running a group and also bad at giving advice. (There are several ways to be bad at giving advice — you might be ignorant of good options, bad at surfacing key features of a person’s situation, bad at securing someone’s trust, etc.)
I’m thinking about conversations in the vein of an EAG speed meeting, where you’re meeting a new person and learning about what they do for a few minutes. If someone comes to EAG and all their speed meetings turn into career advice with an overtone of “you’re probably doing something wrong”, that seems exhausting/dispiriting and unlikely to help (if they aren’t looking for help). I’ve heard from a lot of people who had this experience at an event, and it often made them less interested in further engagement.
If I were going to have an hour-long, in-depth conversation with someone about their work, even if they weren’t specifically asking for advice, I wouldn’t be surprised if we eventually got into probing questions about how they made their choices (and I hope they’d challenge me about my choices, too!). But I wouldn’t try to ask probing questions unprompted in a brief conversation unless someone said something that sounded very off-base to me.
I disagree with 2) because I think the movement will be able to get more done with more diverse backgrounds of people who are really good at different things. Even if AI is the most important thing, we need people who understand communications, policy, organizing grassroots movements, and also people who are good at completely unrelated fields who can understand the impact of AI on their field (manufacturing, agricuture, shipping logistics, etc) though there aren’t those opportunities to do that work directly in AI right now.
I strong upvoted this because:
1) I think AI governance is a big deal (the argument for this has been fleshed out elsewhere by others in the community) and
2) I think this comment is directionally correct beyond the AI governance bit even if I don’t think it quite fully fleshes out the case for it (I’ll have a go at fleshing out the case when I have more time but this is a time-consuming thing to do and my first attempt will be crap even if there is actually something to it).
I think that strong upvoting was appropriate because:
1) stating beliefs that go against the perceived consensus view is hard and takes courage
2) the only way the effective altruism community develops new good ideas is if people feel they have permission to state views that are different from the community “accepted” view.
I think some example steps for forming new good ideas are:
1) someone states, without a fully fleshed out case, what they believe
2) others then think about whether that seems true to them and begin to flesh out reasons for their gut-level intuition
3) other people pushback on those reasons and point out the nuance
4) the people who initially have the gut-level hunch that the statement is true either change their minds or iterate their argument so it incorporates the nuance that others have pointed out for them. If the latter happens then,
5) More nuanced versions of the arguments are written up and steps 3 to 5 repeat themselves as much as necessary for the new good ideas to have a fleshed out case for them.