I agree with the statement âItâs OK for some people not to go into AIâ but strongly disagree with the statement âItâs OK not to go into AI, even if you donât have a good reasonâ. One can list increasingly unreasonable statements like:
AI safety is an important problem.
Every EA should either work on AI safety or have a good reason why theyâre not.
EAs should introduce themselves with what cause area theyâre working on, and their reason for not working on AI if applicable.
Literally everyone should work on AI safety; there are no excuses not to.
I want to remind people of (1) and defend something between (2) and (3).
Our goal as world optimizers is to find the best thing we could possibly be doing, subject to various constraints like non-consequentialist goals and limited information. This means that for every notable action worth considering, we should have a good reason why weâre not doing it. And (2) is just a special case of this, since working on alignment (technical or otherwise) is definitely a notable action. Because there are good reasons to work on AI safety, you need to have a better reason not to.
Having 100x more traction on the problem of making Malawi (0.25% of the world population) a developed country is not a good enough reason, because under most reasonable moral views, preventing human extinction is >100x better than raising the standard of living of 0.25% of the world population.
Note that there are many people who should not work on AI safety because they have >400x more traction on problems 400x smaller, or whatever.
Wanting to expand what EA thinks is possible is not a sufficient reason, because you also have to argue that the expected value of this is higher than investing into causes we already know about.
Holden Karnofsky makes the case against âcause Xâ here: AI risk is already really large in scale; they essentially say âthis century weâre going to figure out what kind of civilization is going to tile the entire galaxyâ, and itâs hard to find something larger in scale than that; x-risks are also neglected. Itâs hard for tractability differences to overwhelm large differences in scale/âneglectedness.
Not having thought about the arguments is not a good enough reason. Reality is the judge of your actions and takes no excuses.
Majoring in something other than math/âCS is not a good enough reason, because your current skills or interest areas donât completely determine your career comparative advantage
Finding the arguments for AI risk unconvincing is not a reason to just not work on AI risk, because if the arguments are wrong, this implies lots of effort on alignment is wasted and we need to shift billions of dollars away from it (and if they have nonessential flaws this could change research directions within alignment), so you should write counterarguments up to allow the EA community to correctly allocate its resources.
also, if working on alignment is your comparative advantage, it might make sense to work on even if the arguments have a 10% chance of being right.
Some potential sufficient reasons
âI tried 3 different kinds of AI safety research and was worse than useless at all of them, and have various reasons not to do longtermist community-building eitherâ
âI have 100x more traction on biorisk and think biorisk is 20x smaller than AI riskâ
âI have 100x more traction on making the entire continent of Africa as developed as the USA, plus person-affecting views, plus my AI timelines are long enough that I can make a difference before AGI happensâ
âI think suffering-focused ethics are correct, so I would rather prevent suffering now than have a small chance of preventing human extinctionâ
âI can become literally the best X in the world, or a mediocre AI safety community-builder. I think the
âI have a really good story for why the arguments for AI risk are wrong and have spent the last month finding the strongest version of my counterarguments; this will direct lots of resources to preventing various moral atrocities in worlds where I am rightâ
edit: after thinking about it more I donât endorse the below paragraph
I also want to defend (3) to some extent. Introducing yourself with your target cause area and reasons for working on it seems like a pretty natural and good thing. In particular it forces you to have a good reason for doing what youâre doing. But there are other benefits too: itâs an obvious conversation starter, and when half the people at the EA event are working on AI safety it just carries a lot of information.
Upvoted for explaining your stance clearly, though Iâm unclear on what you see as the further implications of:
Because there are good reasons to work on AI safety, you need to have a better reason not to.
This is true about many good things a person could do. Some people see AI safety as a special case because they think itâs literally the most good thing, but other people see other causes the same way â and I donât think we want to make any particular thing a default âjustify if not Xâ.
(FWIW, Iâm not sure you actually want AI to be this kind of default â you never say so â but thatâs the feeling I got from this comment.)
Note that there are many people who should not work on AI safety because they have >400x more traction on problems 400x smaller, or whatever.
When someone in EA tells me they work on X, my default assumption is that they think their (traction on X * assumed size of X) is higher than the same number would be for any other thing. Maybe Iâm wrong, because theyâre in the process of retraining or got rejected from all the jobs in Y or something. But I donât see it as my job to make them explain to me why they did X instead of Y, unless theyâre asking me for career advice or something.
There may be exceptional cases where someone is working on something really unusual, but in those cases, I aim for a vibe of âcurious and interestedâ rather than âexpecting justificationâ. At a recent San Diego meetup, I met a dentist and was interested to learn how he chose dentistry; as it turns out, his reasoning was excellent (and I learned a lot about the dental business).
Finding the arguments for AI risk unconvincing is not a reason to just not work on AI risk, because if the arguments are wrong, this implies lots of effort on alignment is wasted and we need to shift billions of dollars away from it (and if they have nonessential flaws this could change research directions within alignment), so you should write counterarguments up to allow the EA community to correctly allocate its resources.
This point carries over to global health, right? If someone finds EA strategy in that area unconvincing, do they need to justify why they arenât writing up their arguments?
In theory, maybe it applies more to global health, since the community spends much more money on global health than AI? (Possibly more effort, too, though I could see that going either way.)
This is true about many good things a person could do. Some people see AI safety as a special case because they think itâs literally the most good thing, but other people see other causes the same way â and I donât think we want to make any particular thing a default âjustify if not Xâ.
Iâm unsure how much I want AI safety to be the default, there are a lot of factors pushing in both directions. But I think one should have a reason why one isnât doing each of the top ~10 things one could, and for a lot of people AI safety (not necessarily technical research) should be on this list.
When someone in EA tells me they work on X, my default assumption is that they think their (traction on X * assumed size of X) is higher than the same number would be for any other thing. Maybe Iâm wrong, because theyâre in the process of retraining or got rejected from all the jobs in Y or something. But I donât see it as my job to make them explain to me why they did X instead of Y, unless theyâre asking me for career advice or something.
My guess is that the median person who filled out the EA survey isnât being consistent in this way. I expect that they could have a one-hour 1-1 with a top community-builder that makes them realize they could be doing something at least 10% better. This is a crux for me.
Separately, I do feel a bit weird about making every conversation into a career advice conversation, but often this seems like the highest impact thing.
If someone finds EA strategy in [global health] unconvincing, do they need to justify why they arenât writing up their arguments?
This was thought-provoking for me. I think existingposts of similar types were hugely impactful. If money were a bottleneck for AI safety and I thought money currently spent on global health should be reallocated to AI safety, writing up some document on this would be among the best things I could be doing. I suppose in general it also depends on oneâs writing skill.
My guess is that the median person who filled out the EA survey isnât being consistent in this way. I expect that they could have a one-hour 1-1 with a top community-builder that makes them realize they could be doing something at least 10% better. This is a crux for me.
I agree with most of this. (I think that other people in EA usually think theyâre doing roughly the best thing for their skills/âbeliefs, but I donât think theyâre usually correct.)
I donât know about âtop community builderâ, unless we tautologically define that as âperson whoâs really good at giving career/âtrajectory adviceâ. I think you could be great at building or running a group and also bad at giving advice. (There are several ways to be bad at giving advice â you might be ignorant of good options, bad at surfacing key features of a personâs situation, bad at securing someoneâs trust, etc.)
Separately, I do feel a bit weird about making every conversation into a career advice conversation, but often this seems like the highest impact thing.
Iâm thinking about conversations in the vein of an EAG speed meeting, where youâre meeting a new person and learning about what they do for a few minutes. If someone comes to EAG and all their speed meetings turn into career advice with an overtone of âyouâre probably doing something wrongâ, that seems exhausting/âdispiriting and unlikely to help (if they arenât looking for help). Iâve heard from a lot of people who had this experience at an event, and it often made them less interested in further engagement.
If I were going to have an hour-long, in-depth conversation with someone about their work, even if they werenât specifically asking for advice, I wouldnât be surprised if we eventually got into probing questions about how they made their choices (and I hope theyâd challenge me about my choices, too!). But I wouldnât try to ask probing questions unprompted in a brief conversation unless someone said something that sounded very off-base to me.
I disagree with 2) because I think the movement will be able to get more done with more diverse backgrounds of people who are really good at different things. Even if AI is the most important thing, we need people who understand communications, policy, organizing grassroots movements, and also people who are good at completely unrelated fields who can understand the impact of AI on their field (manufacturing, agricuture, shipping logistics, etc) though there arenât those opportunities to do that work directly in AI right now.
I strong upvoted this because: 1) I think AI governance is a big deal (the argument for this has been fleshed out elsewhere by others in the community) and 2) I think this comment is directionally correct beyond the AI governance bit even if I donât think it quite fully fleshes out the case for it (Iâll have a go at fleshing out the case when I have more time but this is a time-consuming thing to do and my first attempt will be crap even if there is actually something to it).
I think that strong upvoting was appropriate because: 1) stating beliefs that go against the perceived consensus view is hard and takes courage 2) the only way the effective altruism community develops new good ideas is if people feel they have permission to state views that are different from the community âacceptedâ view.
I think some example steps for forming new good ideas are: 1) someone states, without a fully fleshed out case, what they believe 2) others then think about whether that seems true to them and begin to flesh out reasons for their gut-level intuition 3) other people pushback on those reasons and point out the nuance 4) the people who initially have the gut-level hunch that the statement is true either change their minds or iterate their argument so it incorporates the nuance that others have pointed out for them. If the latter happens then, 5) More nuanced versions of the arguments are written up and steps 3 to 5 repeat themselves as much as necessary for the new good ideas to have a fleshed out case for them.
I agree with the statement âItâs OK for some people not to go into AIâ but strongly disagree with the statement âItâs OK not to go into AI, even if you donât have a good reasonâ. One can list increasingly unreasonable statements like:
AI safety is an important problem.
Every EA should either work on AI safety or have a good reason why theyâre not.
EAs should introduce themselves with what cause area theyâre working on, and their reason for not working on AI if applicable.
Literally everyone should work on AI safety; there are no excuses not to.
I want to remind people of (1) and defend something between (2) and (3).
Our goal as world optimizers is to find the best thing we could possibly be doing, subject to various constraints like non-consequentialist goals and limited information. This means that for every notable action worth considering, we should have a good reason why weâre not doing it. And (2) is just a special case of this, since working on alignment (technical or otherwise) is definitely a notable action. Because there are good reasons to work on AI safety, you need to have a better reason not to.
Having 100x more traction on the problem of making Malawi (0.25% of the world population) a developed country is not a good enough reason, because under most reasonable moral views, preventing human extinction is >100x better than raising the standard of living of 0.25% of the world population.
Note that there are many people who should not work on AI safety because they have >400x more traction on problems 400x smaller, or whatever.
Wanting to expand what EA thinks is possible is not a sufficient reason, because you also have to argue that the expected value of this is higher than investing into causes we already know about.
Holden Karnofsky makes the case against âcause Xâ here: AI risk is already really large in scale; they essentially say âthis century weâre going to figure out what kind of civilization is going to tile the entire galaxyâ, and itâs hard to find something larger in scale than that; x-risks are also neglected. Itâs hard for tractability differences to overwhelm large differences in scale/âneglectedness.
Not having thought about the arguments is not a good enough reason. Reality is the judge of your actions and takes no excuses.
Majoring in something other than math/âCS is not a good enough reason, because your current skills or interest areas donât completely determine your career comparative advantage
Finding the arguments for AI risk unconvincing is not a reason to just not work on AI risk, because if the arguments are wrong, this implies lots of effort on alignment is wasted and we need to shift billions of dollars away from it (and if they have nonessential flaws this could change research directions within alignment), so you should write counterarguments up to allow the EA community to correctly allocate its resources.
also, if working on alignment is your comparative advantage, it might make sense to work on even if the arguments have a 10% chance of being right.
Some potential sufficient reasons
âI tried 3 different kinds of AI safety research and was worse than useless at all of them, and have various reasons not to do longtermist community-building eitherâ
âI have 100x more traction on biorisk and think biorisk is 20x smaller than AI riskâ
âI have 100x more traction on making the entire continent of Africa as developed as the USA, plus person-affecting views, plus my AI timelines are long enough that I can make a difference before AGI happensâ
âI think suffering-focused ethics are correct, so I would rather prevent suffering now than have a small chance of preventing human extinctionâ
âI can become literally the best X in the world, or a mediocre AI safety community-builder. I think the
âI have a really good story for why the arguments for AI risk are wrong and have spent the last month finding the strongest version of my counterarguments; this will direct lots of resources to preventing various moral atrocities in worlds where I am rightâ
edit: after thinking about it more I donât endorse the below paragraph
I also want to defend (3) to some extent. Introducing yourself with your target cause area and reasons for working on it seems like a pretty natural and good thing. In particular it forces you to have a good reason for doing what youâre doing. But there are other benefits too: itâs an obvious conversation starter, and when half the people at the EA event are working on AI safety it just carries a lot of information.
Upvoted for explaining your stance clearly, though Iâm unclear on what you see as the further implications of:
This is true about many good things a person could do. Some people see AI safety as a special case because they think itâs literally the most good thing, but other people see other causes the same way â and I donât think we want to make any particular thing a default âjustify if not Xâ.
(FWIW, Iâm not sure you actually want AI to be this kind of default â you never say so â but thatâs the feeling I got from this comment.)
When someone in EA tells me they work on X, my default assumption is that they think their (traction on X * assumed size of X) is higher than the same number would be for any other thing. Maybe Iâm wrong, because theyâre in the process of retraining or got rejected from all the jobs in Y or something. But I donât see it as my job to make them explain to me why they did X instead of Y, unless theyâre asking me for career advice or something.
There may be exceptional cases where someone is working on something really unusual, but in those cases, I aim for a vibe of âcurious and interestedâ rather than âexpecting justificationâ. At a recent San Diego meetup, I met a dentist and was interested to learn how he chose dentistry; as it turns out, his reasoning was excellent (and I learned a lot about the dental business).
This point carries over to global health, right? If someone finds EA strategy in that area unconvincing, do they need to justify why they arenât writing up their arguments?
In theory, maybe it applies more to global health, since the community spends much more money on global health than AI? (Possibly more effort, too, though I could see that going either way.)
Thanks for the good reply.
Iâm unsure how much I want AI safety to be the default, there are a lot of factors pushing in both directions. But I think one should have a reason why one isnât doing each of the top ~10 things one could, and for a lot of people AI safety (not necessarily technical research) should be on this list.
My guess is that the median person who filled out the EA survey isnât being consistent in this way. I expect that they could have a one-hour 1-1 with a top community-builder that makes them realize they could be doing something at least 10% better. This is a crux for me.
Separately, I do feel a bit weird about making every conversation into a career advice conversation, but often this seems like the highest impact thing.
This was thought-provoking for me. I think existing posts of similar types were hugely impactful. If money were a bottleneck for AI safety and I thought money currently spent on global health should be reallocated to AI safety, writing up some document on this would be among the best things I could be doing. I suppose in general it also depends on oneâs writing skill.
I agree with most of this. (I think that other people in EA usually think theyâre doing roughly the best thing for their skills/âbeliefs, but I donât think theyâre usually correct.)
I donât know about âtop community builderâ, unless we tautologically define that as âperson whoâs really good at giving career/âtrajectory adviceâ. I think you could be great at building or running a group and also bad at giving advice. (There are several ways to be bad at giving advice â you might be ignorant of good options, bad at surfacing key features of a personâs situation, bad at securing someoneâs trust, etc.)
Iâm thinking about conversations in the vein of an EAG speed meeting, where youâre meeting a new person and learning about what they do for a few minutes. If someone comes to EAG and all their speed meetings turn into career advice with an overtone of âyouâre probably doing something wrongâ, that seems exhausting/âdispiriting and unlikely to help (if they arenât looking for help). Iâve heard from a lot of people who had this experience at an event, and it often made them less interested in further engagement.
If I were going to have an hour-long, in-depth conversation with someone about their work, even if they werenât specifically asking for advice, I wouldnât be surprised if we eventually got into probing questions about how they made their choices (and I hope theyâd challenge me about my choices, too!). But I wouldnât try to ask probing questions unprompted in a brief conversation unless someone said something that sounded very off-base to me.
I disagree with 2) because I think the movement will be able to get more done with more diverse backgrounds of people who are really good at different things. Even if AI is the most important thing, we need people who understand communications, policy, organizing grassroots movements, and also people who are good at completely unrelated fields who can understand the impact of AI on their field (manufacturing, agricuture, shipping logistics, etc) though there arenât those opportunities to do that work directly in AI right now.
I strong upvoted this because:
1) I think AI governance is a big deal (the argument for this has been fleshed out elsewhere by others in the community) and
2) I think this comment is directionally correct beyond the AI governance bit even if I donât think it quite fully fleshes out the case for it (Iâll have a go at fleshing out the case when I have more time but this is a time-consuming thing to do and my first attempt will be crap even if there is actually something to it).
I think that strong upvoting was appropriate because:
1) stating beliefs that go against the perceived consensus view is hard and takes courage
2) the only way the effective altruism community develops new good ideas is if people feel they have permission to state views that are different from the community âacceptedâ view.
I think some example steps for forming new good ideas are:
1) someone states, without a fully fleshed out case, what they believe
2) others then think about whether that seems true to them and begin to flesh out reasons for their gut-level intuition
3) other people pushback on those reasons and point out the nuance
4) the people who initially have the gut-level hunch that the statement is true either change their minds or iterate their argument so it incorporates the nuance that others have pointed out for them. If the latter happens then,
5) More nuanced versions of the arguments are written up and steps 3 to 5 repeat themselves as much as necessary for the new good ideas to have a fleshed out case for them.