I think it’s interesting to explore far out ideas and I suppose it might makes sense from the perspective of someone focused on near-termism.
However, as someone more focused on AI safety, one of the cause areas that is more talent dependent and less immediately legible, this seems like this would be a mistake.
If the community is uncertain between the causes, I suggest that it probably wouldn’t be a good idea to dismantle the community now, at least if we think we might obtain more clarity over the next few years.
I think that AI safety needs to be promoted as a cause, not as a community. If you have personal moral uncertainty about whether to focus on animal suffering or AI risk, it might make sense to be a vegan AI researcher. But if you have moral uncertainty about what the priority is overall, you shouldn’t try to mix the two.
People in Machine learning are increasingly of the opinion that there is a risk, and it would be much better to educate them than to try to bring them in to a community which has goals they don’t, and don’t need to, care about.
But if we were to eliminate the EA community, an AI safety community would quickly replace it, as people are often attached to what they do. And this is even more likely if you add any moral connotation. People working at a charity, for example, are drawn to build an identity around it.
Honest question: isn’t an option for the AI Safety community being just the AI Safety community, independent of there being an EA community?
I understand the idea of the philosophy of effective altruism and longtermism being a motivation to work in AI Safety, but that could as well be a worry about modern ML systems, or just sheer intellectual interest. I don’t know if the current entanglement between both communities is that healthy.
EDIT: Corrected stupid wording mistakes. I wrote in a hurry.
I certainly think that having an academic discipline devoted to AI safety is an option, but I think it’s a bad idea for other reasons; if safety is viewed as separate from ML in general, you end up in a situation similar to cybersecurity, where everyone builds dangerous shit, and then the cyber people recoil in horror, and hopefully barely patch the most obvious problems.
That said, yes, I’m completely fine with having informal networks of people working on a goal—it exists regardless of efforts. But a centralized effort at EA community building in general is a different thing, and as I argued here, I tentatively think this are bad, at least at the margin.
I agree with you insofar as separating AI safety from ML is terrible, since the objective of AI safety, in the end, is not to only study safety but to actually implement it in ML systems, and that can only be done in close communication with the general ML community (and I really enjoyed your analogy with cybersecurity).
I don’t know what is the actual current state of this communication, nor who is working on improving it (although I know people are discussing it), but a thing I want to see at least are alignment papers published in NeurIPS, ICML, JMLR, and so on. My two-cent guess is that this would be easier if AI safety would be more dissociated with EA or even longtermism, although I could easily envision myself being wrong.
EDIT: One point important to clarify is that “more dissociated” does not mean “fully dissociated” here. It may be as well that EA donors support AI safety research, effective altruism as an idea makes people look into AI safety, and so on. My worry is AI safety being seen by a lot of people as “that weird idea coming from EA/rationalist folks”. No matter how fair this view actually is, the point is that AI safety should be popular, non-controversial, if safety techniques are to be adopted en masse (which is the end goal).
I’m in favour of direct AI safety movement building too, but the point still remains that the EA community is a vital talent pipeline for cause areas that are more talent dependent. And given the increasing prominence of these cause areas, it seems like it would be a mistake to optimise for the other cause, at least when it’s looking highly plausible that the community may shift even more in the longtermist/x-risk over the next few years.
The community is shifting more long-termist because of intentional decisions that were made—there’s no reason that these shifts have to be locked into place if there happens to be a good reason to shift away from them—not suggesting there is one! If the shift turns out to be a mistake in the future, we should be happy to move away from it, not say “oh but the community may shift towards it in the future”, especially when that shift is caused by intentional decisions in EA leadership.
Publishing What We Owe the Future was an intentional decision, but there’s a sense in which people read whatever people write and make up their own minds.
“Oh but the community may shift towards it in the future”—I guess some of these shifts are pretty predictable in advance, but that’s less important than the point I was making about maintaining option value especially for options that are looking increasingly high value.
Can you expand a bit on what you mean by why these ideas applying better to near – termism?
E.g. Out of ‘hey it seems like machine learning systems are getting scarily powerful, maybe we should do something to make sure they’re aligned with humans’ vs ‘you might think it’s most cost effective to help extremely poor people or animals but actually if you account for the far future it looks like existential risks are more important, and AI is one of the most credible existential risks so maybe you should work on that’, the first one seems like a more scalable/legible message or something. Obviously I’ve strawmaned the second one a bit to make a point but I’m curious what your perspective is!
I think it’s interesting to explore far out ideas and I suppose it might makes sense from the perspective of someone focused on near-termism.
However, as someone more focused on AI safety, one of the cause areas that is more talent dependent and less immediately legible, this seems like this would be a mistake.
If the community is uncertain between the causes, I suggest that it probably wouldn’t be a good idea to dismantle the community now, at least if we think we might obtain more clarity over the next few years.
I think that AI safety needs to be promoted as a cause, not as a community. If you have personal moral uncertainty about whether to focus on animal suffering or AI risk, it might make sense to be a vegan AI researcher. But if you have moral uncertainty about what the priority is overall, you shouldn’t try to mix the two.
People in Machine learning are increasingly of the opinion that there is a risk, and it would be much better to educate them than to try to bring them in to a community which has goals they don’t, and don’t need to, care about.
But if we were to eliminate the EA community, an AI safety community would quickly replace it, as people are often attached to what they do. And this is even more likely if you add any moral connotation. People working at a charity, for example, are drawn to build an identity around it.
I’d suggest that we need multiple paths for drawing talent and general EA community building has been surprisingly successful so far.
Honest question: isn’t an option for the AI Safety community being just the AI Safety community, independent of there being an EA community?
I understand the idea of the philosophy of effective altruism and longtermism being a motivation to work in AI Safety, but that could as well be a worry about modern ML systems, or just sheer intellectual interest. I don’t know if the current entanglement between both communities is that healthy.
EDIT: Corrected stupid wording mistakes. I wrote in a hurry.
I certainly think that having an academic discipline devoted to AI safety is an option, but I think it’s a bad idea for other reasons; if safety is viewed as separate from ML in general, you end up in a situation similar to cybersecurity, where everyone builds dangerous shit, and then the cyber people recoil in horror, and hopefully barely patch the most obvious problems.
That said, yes, I’m completely fine with having informal networks of people working on a goal—it exists regardless of efforts. But a centralized effort at EA community building in general is a different thing, and as I argued here, I tentatively think this are bad, at least at the margin.
I agree with you insofar as separating AI safety from ML is terrible, since the objective of AI safety, in the end, is not to only study safety but to actually implement it in ML systems, and that can only be done in close communication with the general ML community (and I really enjoyed your analogy with cybersecurity).
I don’t know what is the actual current state of this communication, nor who is working on improving it (although I know people are discussing it), but a thing I want to see at least are alignment papers published in NeurIPS, ICML, JMLR, and so on. My two-cent guess is that this would be easier if AI safety would be more dissociated with EA or even longtermism, although I could easily envision myself being wrong.
EDIT: One point important to clarify is that “more dissociated” does not mean “fully dissociated” here. It may be as well that EA donors support AI safety research, effective altruism as an idea makes people look into AI safety, and so on. My worry is AI safety being seen by a lot of people as “that weird idea coming from EA/rationalist folks”. No matter how fair this view actually is, the point is that AI safety should be popular, non-controversial, if safety techniques are to be adopted en masse (which is the end goal).
I’m in favour of direct AI safety movement building too, but the point still remains that the EA community is a vital talent pipeline for cause areas that are more talent dependent. And given the increasing prominence of these cause areas, it seems like it would be a mistake to optimise for the other cause, at least when it’s looking highly plausible that the community may shift even more in the longtermist/x-risk over the next few years.
The shift to longtermism/x-risk to me seems to have been an intentional one, but your comment makes it sound otherwise?
I don’t know what you mean by intentional or not.
But my guess is that the community will shift more long-termist after more people have had time to digest What We Owe the Future.
The community is shifting more long-termist because of intentional decisions that were made—there’s no reason that these shifts have to be locked into place if there happens to be a good reason to shift away from them—not suggesting there is one! If the shift turns out to be a mistake in the future, we should be happy to move away from it, not say “oh but the community may shift towards it in the future”, especially when that shift is caused by intentional decisions in EA leadership.
I guess this is why I asked what you meant.
Publishing What We Owe the Future was an intentional decision, but there’s a sense in which people read whatever people write and make up their own minds.
“Oh but the community may shift towards it in the future”—I guess some of these shifts are pretty predictable in advance, but that’s less important than the point I was making about maintaining option value especially for options that are looking increasingly high value.
Can you expand a bit on what you mean by why these ideas applying better to near – termism?
E.g. Out of ‘hey it seems like machine learning systems are getting scarily powerful, maybe we should do something to make sure they’re aligned with humans’ vs ‘you might think it’s most cost effective to help extremely poor people or animals but actually if you account for the far future it looks like existential risks are more important, and AI is one of the most credible existential risks so maybe you should work on that’, the first one seems like a more scalable/legible message or something. Obviously I’ve strawmaned the second one a bit to make a point but I’m curious what your perspective is!
Maybe I should have said global health and development, rather than near-termism.