While I agree that it’s not “elitist” in the sense that anyone can put forward ideas and be considered by significant people in the community (which I think is great!), I would say there’s still some expectations that need to be met in that the “good idea” generally must accept several commonly agreed up premises that represent what I’d call the “orthodoxy” of EA / rationality / AI safety.
For instance, I noticed way back when I first joined Less Wrong that the Orthogonality Thesis and Instrumental Convergence are more or less doctrines, and challenging them tends to receive a negative or at best lukewarm reception, or at least, demand a very strong case to be made.
In that sense, there is still a bit of elitism in the sense that some of the ideas of the sorta co-founders of the movements, like Eliezer Yudkowsky, Nick Bostrom, Will MacAskill and such, are likely to be treated with notably more deference. It used to be the case, for instance, that on Less Wrong, posts that challenged these views would lead to somewhat dismissive “Read The Sequences” responses, although I admit I haven’t seen that particular refrain in quite a while, so the community might have improved in this regard.
And admittedly, I do think compared to say, most political ideological movements and religions, the EA / rationality / AI safety communities are MUCH more open to dissenting views and good productive debates. This is one thing I do like about us.
In that sense, there is still a bit of elitism in the sense that some of the ideas of the sorta co-founders of the movements, like Eliezer Yudkowsky, Nick Bostrom, Will MacAskill and such, are likely to be treated with notably more deference.
I think I disagree with this.
It has been joked that disagreeing with Eliezer is the favorite activity of LessWrongers.
Will MacAskill has written some very well-reasoned and well-received EAF posts that nonetheless were met with strong disagreement, for example Are we living at the most influential time in history? I think this is a sign of good epistemics: people recognized that post as good and interesting, but they didn’t defer much—they mostly disagreed with its conclusion.
I think the main reason people are more likely to talk about those people’s ideas is because their ideas are genuinely really good.
There will inevitably be some people who have good ideas, and lots of people are persuaded, and from the outside this looks like deference.
I mean, from the inside it would look like what you said, good ideas that are persuasive. And I’m not saying they aren’t good ideas that are persuasive (I agree they are). I’m more just pointing out some examples of ideas that form core elements of our belief ecosystem, that have their source in the works of particular elites, in this case, being named and popularized by Bostrom’s book Superintelligence, and then further popularized by Yudkowsky in the Sequences. In the sense that this is elitism, it’s very mild, and I don’t mean to imply it’s a problem or anything. It’s natural for the more elite among us to be in a better position to come up with the good ideas. I think also that to the extent there is deference here, it is usually well deserved, and also very tame compared to other intellectual communities.
I think this is reasonable as a way for the community to reflexively react to things, to be honest. The question I’m trying to answer when I see someone making a post with an argument that seems worth engaging with is: what’s the probability that I’ll learn something new or change my mind as a result of engaging with this?
When there’s a foundational assumption disagreement, it’s quite difficult to have productive conversations. The conversation kind of needs to be about the disagreement about that assumption, which is a fairly specific kind of discussion. Eg if someone hasn’t really thought about AI alignment much, thinks it’s not an issue, but isn’t familiar with the reasons I believe it matters, then I put a much lower (though still non-zero) probability that I’ll make useful updates from talking to them. Because I have a bunch of standard arguments for the most obvious objections people sometimes raise, and don’t learn much from stating them. And I think there’s a lot of value to having high-context discussion spaces where people broadly agree on these foundational claims.
These foundational claims are pretty difficult to establish consensus on if people have different priors, and discussing them doesn’t really tend to move people either way. I get a lot of value from discussing technical details of what working on AI safety is like with people, much more so than I get from the average “does AI safety matter at all?” conversation.
Obviously, if someone could convince me that AI safety doesn’t matter, that would be a big deal. But I’d guess it’s only really worth the effort if I’m reasonably sure the person understands why I believe it does matter and disagrees anyway, in a way that isn’t stemming from some intractable foundational disagreements in worldviews
I want to clarify that I don’t think ideas like the Orthogonality Thesis or Instrumental Convergence are wrong. They’re strong predictive hypotheses that follow logically from very reasonable assumptions, and even the possibility that they could be correct is more than enough justification for AI safety work to be critical.
I was more just pointing out some examples of ideas that are very strongly held by the community, that happen to have been named and popularized by people like Bostrom and Yudkowsky, both of whom might be considered elites among us.
P.S. I’m always a bit surprised that the Neel Nanda of Google DeepMind has the time and desire to post so much on the EA Forums (and also Less Wrong). That probably says very good things about us, and also gives me some more hope that the folks at Google are actually serious about alignment. I really like your work, so it’s an honour to be able to engage with you here (hope I’m not fanboying too much).
If your complaint is that the default assumption is that these are more or less true, well, my claim is that even though normies tend to see this as a negative signal, it’s actually a positive signal for those with good epistemics.
That said, it’s important to keep in mind that these aren’t directly talking about actual systems (the Orthogonality Thesis is about possible systems and Instrumental Convergence is about incentives).
While I agree that it’s not “elitist” in the sense that anyone can put forward ideas and be considered by significant people in the community (which I think is great!), I would say there’s still some expectations that need to be met in that the “good idea” generally must accept several commonly agreed up premises that represent what I’d call the “orthodoxy” of EA / rationality / AI safety.
For instance, I noticed way back when I first joined Less Wrong that the Orthogonality Thesis and Instrumental Convergence are more or less doctrines, and challenging them tends to receive a negative or at best lukewarm reception, or at least, demand a very strong case to be made.
In that sense, there is still a bit of elitism in the sense that some of the ideas of the sorta co-founders of the movements, like Eliezer Yudkowsky, Nick Bostrom, Will MacAskill and such, are likely to be treated with notably more deference. It used to be the case, for instance, that on Less Wrong, posts that challenged these views would lead to somewhat dismissive “Read The Sequences” responses, although I admit I haven’t seen that particular refrain in quite a while, so the community might have improved in this regard.
And admittedly, I do think compared to say, most political ideological movements and religions, the EA / rationality / AI safety communities are MUCH more open to dissenting views and good productive debates. This is one thing I do like about us.
I think I disagree with this.
It has been joked that disagreeing with Eliezer is the favorite activity of LessWrongers.
Will MacAskill has written some very well-reasoned and well-received EAF posts that nonetheless were met with strong disagreement, for example Are we living at the most influential time in history? I think this is a sign of good epistemics: people recognized that post as good and interesting, but they didn’t defer much—they mostly disagreed with its conclusion.
I think the main reason people are more likely to talk about those people’s ideas is because their ideas are genuinely really good.
There will inevitably be some people who have good ideas, and lots of people are persuaded, and from the outside this looks like deference.
I mean, from the inside it would look like what you said, good ideas that are persuasive. And I’m not saying they aren’t good ideas that are persuasive (I agree they are). I’m more just pointing out some examples of ideas that form core elements of our belief ecosystem, that have their source in the works of particular elites, in this case, being named and popularized by Bostrom’s book Superintelligence, and then further popularized by Yudkowsky in the Sequences. In the sense that this is elitism, it’s very mild, and I don’t mean to imply it’s a problem or anything. It’s natural for the more elite among us to be in a better position to come up with the good ideas. I think also that to the extent there is deference here, it is usually well deserved, and also very tame compared to other intellectual communities.
I think this is reasonable as a way for the community to reflexively react to things, to be honest. The question I’m trying to answer when I see someone making a post with an argument that seems worth engaging with is: what’s the probability that I’ll learn something new or change my mind as a result of engaging with this?
When there’s a foundational assumption disagreement, it’s quite difficult to have productive conversations. The conversation kind of needs to be about the disagreement about that assumption, which is a fairly specific kind of discussion. Eg if someone hasn’t really thought about AI alignment much, thinks it’s not an issue, but isn’t familiar with the reasons I believe it matters, then I put a much lower (though still non-zero) probability that I’ll make useful updates from talking to them. Because I have a bunch of standard arguments for the most obvious objections people sometimes raise, and don’t learn much from stating them. And I think there’s a lot of value to having high-context discussion spaces where people broadly agree on these foundational claims.
These foundational claims are pretty difficult to establish consensus on if people have different priors, and discussing them doesn’t really tend to move people either way. I get a lot of value from discussing technical details of what working on AI safety is like with people, much more so than I get from the average “does AI safety matter at all?” conversation.
Obviously, if someone could convince me that AI safety doesn’t matter, that would be a big deal. But I’d guess it’s only really worth the effort if I’m reasonably sure the person understands why I believe it does matter and disagrees anyway, in a way that isn’t stemming from some intractable foundational disagreements in worldviews
I want to clarify that I don’t think ideas like the Orthogonality Thesis or Instrumental Convergence are wrong. They’re strong predictive hypotheses that follow logically from very reasonable assumptions, and even the possibility that they could be correct is more than enough justification for AI safety work to be critical.
I was more just pointing out some examples of ideas that are very strongly held by the community, that happen to have been named and popularized by people like Bostrom and Yudkowsky, both of whom might be considered elites among us.
P.S. I’m always a bit surprised that the Neel Nanda of Google DeepMind has the time and desire to post so much on the EA Forums (and also Less Wrong). That probably says very good things about us, and also gives me some more hope that the folks at Google are actually serious about alignment. I really like your work, so it’s an honour to be able to engage with you here (hope I’m not fanboying too much).
If your complaint is that the default assumption is that these are more or less true, well, my claim is that even though normies tend to see this as a negative signal, it’s actually a positive signal for those with good epistemics.
That said, it’s important to keep in mind that these aren’t directly talking about actual systems (the Orthogonality Thesis is about possible systems and Instrumental Convergence is about incentives).
I more or less agree. It’s not really a complaint from me. I probably was too provocative in my choice of wording earlier.