Thanks for this post. If true, it does describe a pretty serious concern.
One issue I’ve always had with the “highly engaged EA” metric is that it’s only a measure for alignment,* but the people who are most impactful within EA have both high alignment and high competence. If your recruitment selects only on alignment this suggests we’re at best neutral to competence and at worst (as this post describes) actively selecting against competence.
(I do think the elite university setting mitigates this harm somewhat, e.g. 25th percentile MIT students still aren’t stupid in absolute terms).
That said, I think the student group organizers I recently talked to are usually extremely aware of this distinction. (I’ve talked to a subset of student group organizers from Stanford, MIT, Harvard (though less granularity), UPenn (only one) and Columbia, in case this is helpful). And they tend to operationalize their targets more in terms of people who do good EA research, jobs, and exciting entrepreneurship projects, rather than in terms of just engagement/identification. Though I could be wrong about what they care about in general (as opposed to just when talking with me).
The pet theory I have for why people focus on “Highly Engaged EAs” as the metric most openly talked about as opposed to (e.g.) “Highly Engaged and Very Competent EAs” is that it looks more fair/egalitarian (at least in theory, anybody can become highly engaged, but very few people can become as smart as X or as good at operations as Y).
Some quick questions:
1. Can you define what you mean in your article by “Pascal’s Mugging?” I think you’re not using the formal definition but as a metonym for something else (which is totally fine, but I’m not sure what idea or cluster of ideas you’re exactly pointing to that’s more specific than “vaguely shady”).
2. Your critiques seem to mostly come from the LessWrong/rationality cluster. Do you have a sense of whether these critiques are shared (though expressed in different cultural languages) from people with good epistemics but from other cultural clusters?
*and of course, like all metrics it’s an imperfect measure for alignment, and can be gamed.
Regarding “Pascal’s Mugging”: I am not the author, so I might well be mistaken. But I think I can relate to the intended meaning more closely than “vaguely shady”
One paragraph is
EA may not in fact be a form of Pascal’s Mugging or fanaticism, but if you take certain presentations of longtermism and X-risk seriously, the demands are sufficiently large that it certainly pattern-matches pretty well to these.
which I read as: “Pascal’s mugging” describes a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities. I think that this in itself need not be problematic (there can be huge stakes which warrant change in behaviour), but if there is social pressure involved in forcing people to accept the premise of huge moral stakes, things become problematic.
One example is the “child drowning in a pond” thought experiment. It does introduce large moral stakes (the resources you use for conveniences in everyday life could in fact be used to help people in urgent need; and in the thought experiment itself you would decide that the latter is more important) and can be used to imply significant behavioural changes (putting a large fraction of one’s resources to helping worse-off people). If this argument is presented with strong social pressure to not voice objections, that would be a situation which fits under Pascal-mugging in my understanding.
If people are used to this type of rhetorical move, they will become wary as soon as anything along the lines of “there are huge moral stakes which you are currently ignoring and you should completely change your life-goals” is mentioned to them. Assuming this, I think the worry that
[...] the demands are sufficiently large that it certainly pattern-matches pretty well to these.
Thanks a lot for the explanation! It does make more sense in context of the text, though to be clear this is extremely far from the original meaning of the phrase, and also the phrase has very negative connotations in our community. So I’d prefer it if future community members don’t use “Pascal’s mugging” to mean “a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities,” unless maybe it’s locally-scoped and clearly defined in the text to mean something that does not have the original technical meaning.
It is unfortunate that I can’t think of a better term on the top of my head for this concept however, would be interested in good suggestions.
a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities
What is the definition you’d prefer people to stick to? Something like “being pushed into actions that have a very low probability of producing value, because the reward would be extremely high in the unlikely event they did work out”?
The Drowning Child argument doesn’t seem like an example of Pascal’s Mugging, but Wikipedia gives the example of:
“give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3 ↑↑↑↑ 3”
being pushed into actions that have a very low probability of producing value, because the reward would be extremely high in the unlikely event they did work out
I haven’t watched the video, but I assumed it’s going to say “AI Safety is not a Pascal’s Mugging because the probability of AI x-risk is nontrivially high.” So someone who comes into the video with the assumption that AI risk is a clear Pascal’s Mugging since they view it as “a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities” would be pretty unhappy with the video and think that there was a bait-and-switch.
I’m not sure the most impactful people need have high alignment. We’ve disagreed about Elon Musk in the past, but I still think he’s a better candidate for the world’s most counterfactually positive human than anyone else I can think of. Bill Gates is similarly important and similarly kinda-but-conspicuously-not-explicitly aligned.
Yes, if you rank all humans by counterfactual positive impact, most of them are not EA, because most humans are not EAs.
This is even more true if you are mostly selecting on people who were around long before EA started, or if you go by ex post rather than ex ante counterfactual impact (how much credit should we give to Bill Gates’ grandmother?)
(I’m probably just rehashing an old debate, but also Elon Musk is in the top 5-10 of contenders for “most likely to destroy the world,” so that’s at least some consideration against him specifically).
I don’t think background rate is relevant here. I was contesting your claim that ‘the people who are most impactful within EA have both high alignment and high competence’. It depends on what you mean ‘within EA’ I guess. If you mean ‘people who openly espouse EA ideas’, then the ‘high alignment’ seems uninterestingly true almost by definition. If you mean ‘people who are doing altruistic work effectively’ then Gates and Musk are , IMO, strong enough counterpoints to falsify the claim.
Thanks for this post. If true, it does describe a pretty serious concern.
One issue I’ve always had with the “highly engaged EA” metric is that it’s only a measure for alignment,* but the people who are most impactful within EA have both high alignment and high competence. If your recruitment selects only on alignment this suggests we’re at best neutral to competence and at worst (as this post describes) actively selecting against competence.
(I do think the elite university setting mitigates this harm somewhat, e.g. 25th percentile MIT students still aren’t stupid in absolute terms).
That said, I think the student group organizers I recently talked to are usually extremely aware of this distinction. (I’ve talked to a subset of student group organizers from Stanford, MIT, Harvard (though less granularity), UPenn (only one) and Columbia, in case this is helpful). And they tend to operationalize their targets more in terms of people who do good EA research, jobs, and exciting entrepreneurship projects, rather than in terms of just engagement/identification. Though I could be wrong about what they care about in general (as opposed to just when talking with me).
The pet theory I have for why people focus on “Highly Engaged EAs” as the metric most openly talked about as opposed to (e.g.) “Highly Engaged and Very Competent EAs” is that it looks more fair/egalitarian (at least in theory, anybody can become highly engaged, but very few people can become as smart as X or as good at operations as Y).
Some quick questions:
1. Can you define what you mean in your article by “Pascal’s Mugging?” I think you’re not using the formal definition but as a metonym for something else (which is totally fine, but I’m not sure what idea or cluster of ideas you’re exactly pointing to that’s more specific than “vaguely shady”).
2. Your critiques seem to mostly come from the LessWrong/rationality cluster. Do you have a sense of whether these critiques are shared (though expressed in different cultural languages) from people with good epistemics but from other cultural clusters?
*and of course, like all metrics it’s an imperfect measure for alignment, and can be gamed.
Regarding “Pascal’s Mugging”:
I am not the author, so I might well be mistaken. But I think I can relate to the intended meaning more closely than “vaguely shady”
One paragraph is
which I read as: “Pascal’s mugging” describes a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities. I think that this in itself need not be problematic (there can be huge stakes which warrant change in behaviour), but if there is social pressure involved in forcing people to accept the premise of huge moral stakes, things become problematic.
One example is the “child drowning in a pond” thought experiment. It does introduce large moral stakes (the resources you use for conveniences in everyday life could in fact be used to help people in urgent need; and in the thought experiment itself you would decide that the latter is more important) and can be used to imply significant behavioural changes (putting a large fraction of one’s resources to helping worse-off people).
If this argument is presented with strong social pressure to not voice objections, that would be a situation which fits under Pascal-mugging in my understanding.
If people are used to this type of rhetorical move, they will become wary as soon as anything along the lines of “there are huge moral stakes which you are currently ignoring and you should completely change your life-goals” is mentioned to them. Assuming this, I think the worry that
makes a lot of sense.
Thanks a lot for the explanation! It does make more sense in context of the text, though to be clear this is extremely far from the original meaning of the phrase, and also the phrase has very negative connotations in our community. So I’d prefer it if future community members don’t use “Pascal’s mugging” to mean “a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities,” unless maybe it’s locally-scoped and clearly defined in the text to mean something that does not have the original technical meaning.
It is unfortunate that I can’t think of a better term on the top of my head for this concept however, would be interested in good suggestions.
What is the definition you’d prefer people to stick to? Something like “being pushed into actions that have a very low probability of producing value, because the reward would be extremely high in the unlikely event they did work out”?
The Drowning Child argument doesn’t seem like an example of Pascal’s Mugging, but Wikipedia gives the example of:
and I think recent posts like The AI Messiah are gesturing at something like that (see, even, this video from the comments on that post: Is AI Safety a Pascal’s Mugging?).
Yes this is the definition I would prefer.
I haven’t watched the video, but I assumed it’s going to say “AI Safety is not a Pascal’s Mugging because the probability of AI x-risk is nontrivially high.” So someone who comes into the video with the assumption that AI risk is a clear Pascal’s Mugging since they view it as “a rhetorical move that introduces huge moral stakes into the world-view in order to push people into drastically altering their actions and priorities” would be pretty unhappy with the video and think that there was a bait-and-switch.
I’m not sure the most impactful people need have high alignment. We’ve disagreed about Elon Musk in the past, but I still think he’s a better candidate for the world’s most counterfactually positive human than anyone else I can think of. Bill Gates is similarly important and similarly kinda-but-conspicuously-not-explicitly aligned.
Yes, if you rank all humans by counterfactual positive impact, most of them are not EA, because most humans are not EAs.
This is even more true if you are mostly selecting on people who were around long before EA started, or if you go by ex post rather than ex ante counterfactual impact (how much credit should we give to Bill Gates’ grandmother?)
(I’m probably just rehashing an old debate, but also Elon Musk is in the top 5-10 of contenders for “most likely to destroy the world,” so that’s at least some consideration against him specifically).
I don’t think background rate is relevant here. I was contesting your claim that ‘the people who are most impactful within EA have both high alignment and high competence’. It depends on what you mean ‘within EA’ I guess. If you mean ‘people who openly espouse EA ideas’, then the ‘high alignment’ seems uninterestingly true almost by definition. If you mean ‘people who are doing altruistic work effectively’ then Gates and Musk are , IMO, strong enough counterpoints to falsify the claim.
There are many/most people who openly espouse EA ideas who I do not consider highly aligned.