Hey! I liked certain parts of this post and not other parts of this post. I appreciate the thoughtfulness by which you critique EA through this post.
On your first point about the AI messiah:
I think the key distinction is that there are many reasons to believe this argument about the dangers of an AGI are correct, though. Even if many claims with a similar form are wrong, that doesn’t exclude this specific claim from being right.
“Climate scientists keep telling us about how climate change is going to be so disastrous and we need to be prepared. But humanity has seen so many claims of this form and they’ve all been so wrong!”
The key distinction is that there is a lot of reason to believe that AGI will be dangerous. There is also a lot of reason to support the claim that we are not prepared currently. Without addressing that chain of logic directly, I don’t think I’m convinced by this argument.
On your second point about the EA religious tendencies:
Because religious communities are one of the most common communities we see, there’s obviously going to be parallels that exist between religious communities and EA.
Some of these analogies hold, others not so much. We, too, want to community build, network, and learn from each other. I’d love for you to point at specific examples of things EA do, from conferences to holding EA university groups, that are ineffective or unnecessary.
To perhaps a greater point of EA perhaps becoming too groupthink-y, which I think may be warranted:
I think a key distinction is that EA has a healthy level of debate, disagreement, and skepticism—while religions tend to demand blind faith in believing something unprovable. This ongoing debate on how to do the most good I personally find the most valuable in the community—and I hope this spirit never dies.
Keep on critiquing EA; I think such critiques are extremely valuable. Thanks for writing this.
Maintaining that healthy level of debate, disagreement, and skepticism is critical, but harder to do when an idea becomes more popular. I believe most of the early “converts” to AI Safety have carefully weighed the arguments and made a decision based on analysis of the evidence. But as AI Safety becomes a larger portion of EA, the idea will begin to spread for other, more “religious” reasons (e.g., social conformity, $’s, institutionalized recruiting/evangelization, leadership authority).
As an example, I’d put the belief in prediction markets as an EA idea that tends towards the religious. Prediction markets may well be a beneficial innovation, but I personally don’t think we have good evidence one way or the other yet. But due to the idea’s connection to rationality and EA community leaders, it has gained many adherents who probably haven’t closely evaluated the supporting data. Again, maybe the idea is correct and this is a good thing. But I think it is better if EA had fewer of these canonized, insider signals, because it makes reevaluation of the ideas difficult.
This point has helped me understand the original post more.
I feel that too many times, many EAs take current EA frameworks and ways of thinking for granted instead of questioning those frameworks and actively trying to identify flaws and in-built assumptions. Thinking through and questioning those perspectives is a good exercise in general but also extremely helpful to contribute to the motivating worldview of the community.
Still don’t believe that this necessarily means EAs “tend toward the religious”—there are probably several layers of nuance that are missing in that statement.
All in all, I’d love to see more people critique EA frameworks and conventional EA ideas in this forum—I believe there are plenty of flaws to be found.
I’m pretty confused here. On the one hand, I think it’s probably good to have less epistemic deference and more independent thinking in EA. On the other, I think if I take your statements literally and extend them, I think they’re probably drawing the boundaries of “religious” way too broadly, in mostly-unhelpful ways.
As an example, I’d put the belief in prediction markets as an EA idea that tends towards the religious. Prediction markets may well be a beneficial innovation, but I personally don’t think we have good evidence one way or the other yet. But due to the idea’s connection to rationality and EA community leaders, it has gained many adherents who probably haven’t closely evaluated the supporting data. Again, maybe the idea is correct and this is a good thing. But I think it is better if EA had fewer of these canonized, insider signals, because it makes reevaluation of the ideas difficult.
I think people who study forecasting are usually aware of the potential limitations of prediction markets. See e.g. here, here, and here. And to the extent they aren’t, this is because “more research is needed”, not because of an unhealthy deference to authority.
People who don’t study forecasting may well overestimate the value of prediction markets, and some of this might be due to deference. But I don’t know, this just seems unavoidable as part of a healthy collective epistemic process, and categorizing it as “tends towards the religious” just seems to stretch the definition of “religious” way too far.
Analogously, many non-EAs also believe that a) handwashing stops covid-19, and b) the Earth orbits the Sun. In both cases, the epistemic process probably looks much more like some combination of “people I respect believe this”, “this seems to make sense”, and “the authorities believe this” rather than a deep principled understanding of the science. And this just seems...broadly not-religious to me? Of course, the main salient difference between a) and b) is that one of the above is probably false. But I don’t think it’d be appropriate to frame “have a mistaken belief because the apparent scientific consensus is incorrect” as “religious”
Re: your first point: I agree people have inside view reasons for believing in risk from AGI. My point was just that it’s quite remarkable to believe that, sure, all those other times the god-like figure didn’t show up, but that this time we’re right. I realize this argument will probably sound unsatisfactory to many people. My main goal was not to try to persuade people away from focusing on AI risks, it was to point out that the claims being made are very messianic and that that is kind of interesting sociologically.
Re: your second point: I should perhaps have been clearer: I am not making a parallel to religion as a way of criticizing EA. I think religions are kind of amazing. They’re one of the few human institutions that have been able to reproduce themselves and shape human behaviour in fairly consistent ways over thousands of years. That’s an incredible accomplishment. We could learn from them.
I have come to see the term ‘religion’ (as well as ‘ideology’) as unhelpful in these discussions. It might be helpful to taboo these words and start talking in terms of ‘motivating world-views’ instead.
Hey! I liked certain parts of this post and not other parts of this post. I appreciate the thoughtfulness by which you critique EA through this post.
On your first point about the AI messiah:
I think the key distinction is that there are many reasons to believe this argument about the dangers of an AGI are correct, though. Even if many claims with a similar form are wrong, that doesn’t exclude this specific claim from being right.
“Climate scientists keep telling us about how climate change is going to be so disastrous and we need to be prepared. But humanity has seen so many claims of this form and they’ve all been so wrong!”
The key distinction is that there is a lot of reason to believe that AGI will be dangerous. There is also a lot of reason to support the claim that we are not prepared currently. Without addressing that chain of logic directly, I don’t think I’m convinced by this argument.
On your second point about the EA religious tendencies:
Because religious communities are one of the most common communities we see, there’s obviously going to be parallels that exist between religious communities and EA.
Some of these analogies hold, others not so much. We, too, want to community build, network, and learn from each other. I’d love for you to point at specific examples of things EA do, from conferences to holding EA university groups, that are ineffective or unnecessary.
To perhaps a greater point of EA perhaps becoming too groupthink-y, which I think may be warranted:
I think a key distinction is that EA has a healthy level of debate, disagreement, and skepticism—while religions tend to demand blind faith in believing something unprovable. This ongoing debate on how to do the most good I personally find the most valuable in the community—and I hope this spirit never dies.
Keep on critiquing EA; I think such critiques are extremely valuable. Thanks for writing this.
Maintaining that healthy level of debate, disagreement, and skepticism is critical, but harder to do when an idea becomes more popular. I believe most of the early “converts” to AI Safety have carefully weighed the arguments and made a decision based on analysis of the evidence. But as AI Safety becomes a larger portion of EA, the idea will begin to spread for other, more “religious” reasons (e.g., social conformity, $’s, institutionalized recruiting/evangelization, leadership authority).
As an example, I’d put the belief in prediction markets as an EA idea that tends towards the religious. Prediction markets may well be a beneficial innovation, but I personally don’t think we have good evidence one way or the other yet. But due to the idea’s connection to rationality and EA community leaders, it has gained many adherents who probably haven’t closely evaluated the supporting data. Again, maybe the idea is correct and this is a good thing. But I think it is better if EA had fewer of these canonized, insider signals, because it makes reevaluation of the ideas difficult.
This point has helped me understand the original post more.
I feel that too many times, many EAs take current EA frameworks and ways of thinking for granted instead of questioning those frameworks and actively trying to identify flaws and in-built assumptions. Thinking through and questioning those perspectives is a good exercise in general but also extremely helpful to contribute to the motivating worldview of the community.
Still don’t believe that this necessarily means EAs “tend toward the religious”—there are probably several layers of nuance that are missing in that statement.
All in all, I’d love to see more people critique EA frameworks and conventional EA ideas in this forum—I believe there are plenty of flaws to be found.
I’m pretty confused here. On the one hand, I think it’s probably good to have less epistemic deference and more independent thinking in EA. On the other, I think if I take your statements literally and extend them, I think they’re probably drawing the boundaries of “religious” way too broadly, in mostly-unhelpful ways.
I think people who study forecasting are usually aware of the potential limitations of prediction markets. See e.g. here, here, and here. And to the extent they aren’t, this is because “more research is needed”, not because of an unhealthy deference to authority.
People who don’t study forecasting may well overestimate the value of prediction markets, and some of this might be due to deference. But I don’t know, this just seems unavoidable as part of a healthy collective epistemic process, and categorizing it as “tends towards the religious” just seems to stretch the definition of “religious” way too far.
Analogously, many non-EAs also believe that a) handwashing stops covid-19, and b) the Earth orbits the Sun. In both cases, the epistemic process probably looks much more like some combination of “people I respect believe this”, “this seems to make sense”, and “the authorities believe this” rather than a deep principled understanding of the science. And this just seems...broadly not-religious to me? Of course, the main salient difference between a) and b) is that one of the above is probably false. But I don’t think it’d be appropriate to frame “have a mistaken belief because the apparent scientific consensus is incorrect” as “religious”
Thanks for the kind words Richard.
Re: your first point: I agree people have inside view reasons for believing in risk from AGI. My point was just that it’s quite remarkable to believe that, sure, all those other times the god-like figure didn’t show up, but that this time we’re right. I realize this argument will probably sound unsatisfactory to many people. My main goal was not to try to persuade people away from focusing on AI risks, it was to point out that the claims being made are very messianic and that that is kind of interesting sociologically.
Re: your second point: I should perhaps have been clearer: I am not making a parallel to religion as a way of criticizing EA. I think religions are kind of amazing. They’re one of the few human institutions that have been able to reproduce themselves and shape human behaviour in fairly consistent ways over thousands of years. That’s an incredible accomplishment. We could learn from them.
I have come to see the term ‘religion’ (as well as ‘ideology’) as unhelpful in these discussions. It might be helpful to taboo these words and start talking in terms of ‘motivating world-views’ instead.