Honest question: isn’t an option for the AI Safety community being just the AI Safety community, independent of there being an EA community?
I understand the idea of the philosophy of effective altruism and longtermism being a motivation to work in AI Safety, but that could as well be a worry about modern ML systems, or just sheer intellectual interest. I don’t know if the current entanglement between both communities is that healthy.
EDIT: Corrected stupid wording mistakes. I wrote in a hurry.
I certainly think that having an academic discipline devoted to AI safety is an option, but I think it’s a bad idea for other reasons; if safety is viewed as separate from ML in general, you end up in a situation similar to cybersecurity, where everyone builds dangerous shit, and then the cyber people recoil in horror, and hopefully barely patch the most obvious problems.
That said, yes, I’m completely fine with having informal networks of people working on a goal—it exists regardless of efforts. But a centralized effort at EA community building in general is a different thing, and as I argued here, I tentatively think this are bad, at least at the margin.
I agree with you insofar as separating AI safety from ML is terrible, since the objective of AI safety, in the end, is not to only study safety but to actually implement it in ML systems, and that can only be done in close communication with the general ML community (and I really enjoyed your analogy with cybersecurity).
I don’t know what is the actual current state of this communication, nor who is working on improving it (although I know people are discussing it), but a thing I want to see at least are alignment papers published in NeurIPS, ICML, JMLR, and so on. My two-cent guess is that this would be easier if AI safety would be more dissociated with EA or even longtermism, although I could easily envision myself being wrong.
EDIT: One point important to clarify is that “more dissociated” does not mean “fully dissociated” here. It may be as well that EA donors support AI safety research, effective altruism as an idea makes people look into AI safety, and so on. My worry is AI safety being seen by a lot of people as “that weird idea coming from EA/rationalist folks”. No matter how fair this view actually is, the point is that AI safety should be popular, non-controversial, if safety techniques are to be adopted en masse (which is the end goal).
I’m in favour of direct AI safety movement building too, but the point still remains that the EA community is a vital talent pipeline for cause areas that are more talent dependent. And given the increasing prominence of these cause areas, it seems like it would be a mistake to optimise for the other cause, at least when it’s looking highly plausible that the community may shift even more in the longtermist/x-risk over the next few years.
The community is shifting more long-termist because of intentional decisions that were made—there’s no reason that these shifts have to be locked into place if there happens to be a good reason to shift away from them—not suggesting there is one! If the shift turns out to be a mistake in the future, we should be happy to move away from it, not say “oh but the community may shift towards it in the future”, especially when that shift is caused by intentional decisions in EA leadership.
Publishing What We Owe the Future was an intentional decision, but there’s a sense in which people read whatever people write and make up their own minds.
“Oh but the community may shift towards it in the future”—I guess some of these shifts are pretty predictable in advance, but that’s less important than the point I was making about maintaining option value especially for options that are looking increasingly high value.
Honest question: isn’t an option for the AI Safety community being just the AI Safety community, independent of there being an EA community?
I understand the idea of the philosophy of effective altruism and longtermism being a motivation to work in AI Safety, but that could as well be a worry about modern ML systems, or just sheer intellectual interest. I don’t know if the current entanglement between both communities is that healthy.
EDIT: Corrected stupid wording mistakes. I wrote in a hurry.
I certainly think that having an academic discipline devoted to AI safety is an option, but I think it’s a bad idea for other reasons; if safety is viewed as separate from ML in general, you end up in a situation similar to cybersecurity, where everyone builds dangerous shit, and then the cyber people recoil in horror, and hopefully barely patch the most obvious problems.
That said, yes, I’m completely fine with having informal networks of people working on a goal—it exists regardless of efforts. But a centralized effort at EA community building in general is a different thing, and as I argued here, I tentatively think this are bad, at least at the margin.
I agree with you insofar as separating AI safety from ML is terrible, since the objective of AI safety, in the end, is not to only study safety but to actually implement it in ML systems, and that can only be done in close communication with the general ML community (and I really enjoyed your analogy with cybersecurity).
I don’t know what is the actual current state of this communication, nor who is working on improving it (although I know people are discussing it), but a thing I want to see at least are alignment papers published in NeurIPS, ICML, JMLR, and so on. My two-cent guess is that this would be easier if AI safety would be more dissociated with EA or even longtermism, although I could easily envision myself being wrong.
EDIT: One point important to clarify is that “more dissociated” does not mean “fully dissociated” here. It may be as well that EA donors support AI safety research, effective altruism as an idea makes people look into AI safety, and so on. My worry is AI safety being seen by a lot of people as “that weird idea coming from EA/rationalist folks”. No matter how fair this view actually is, the point is that AI safety should be popular, non-controversial, if safety techniques are to be adopted en masse (which is the end goal).
I’m in favour of direct AI safety movement building too, but the point still remains that the EA community is a vital talent pipeline for cause areas that are more talent dependent. And given the increasing prominence of these cause areas, it seems like it would be a mistake to optimise for the other cause, at least when it’s looking highly plausible that the community may shift even more in the longtermist/x-risk over the next few years.
The shift to longtermism/x-risk to me seems to have been an intentional one, but your comment makes it sound otherwise?
I don’t know what you mean by intentional or not.
But my guess is that the community will shift more long-termist after more people have had time to digest What We Owe the Future.
The community is shifting more long-termist because of intentional decisions that were made—there’s no reason that these shifts have to be locked into place if there happens to be a good reason to shift away from them—not suggesting there is one! If the shift turns out to be a mistake in the future, we should be happy to move away from it, not say “oh but the community may shift towards it in the future”, especially when that shift is caused by intentional decisions in EA leadership.
I guess this is why I asked what you meant.
Publishing What We Owe the Future was an intentional decision, but there’s a sense in which people read whatever people write and make up their own minds.
“Oh but the community may shift towards it in the future”—I guess some of these shifts are pretty predictable in advance, but that’s less important than the point I was making about maintaining option value especially for options that are looking increasingly high value.