Thanks for the post — There is definitely a certain fuzziness at times about value claims in the movement and I have been critical at times of similar things. Also chatgpt edited this but (nearly) all thoughts are my own hope that’s ok!
I see a few threads here that are easy to blur:
1) Metaethics (realism vs anti-realism) is mostly orthogonal to Ideal Reflection. You can be a realist or anti-realist and still endorse (or reject) a norm like “defer to what an idealized version of you would believe, holding evidence fixed.” Ideal Reflection doesn’t have to claim there’s a stance-independent EV “out there”; it can be a procedural claim about which internal standpoint is authoritative (idealized deliberation vs current snap judgment), and about how to talk when you’re trying to approximate that standpoint. I’m not saying you claimed the opposite exactly but language was a bit confusing to me at times.
2) Ideal Reflection is a metanormative framework; EA is basically a practical operationalization of it. Ideal Reflection by itself is extremely broad. But on its own it doesn’t tell you what you value, and it doesn’t even guarantee that you can map possible world-histories to an ordinal ranking. It might seem less hand-wavy but its lack of assumptions makes it hard to see what non trivial claims can follow. Once you add enough axioms/structure to make action-guiding comparisons possible (some consequentialist-ish evaluative ranking, plus willingness to act under uncertainty), then you can start building “upward” from reflection to action.
It also seems to me (and is part of what makes EA distinctive) that EA ecosystem was built by unusually self-reflective people — sometimes to a fault — who tried hard to notice when they were rationalizing, to systematize their uncertainty, and to actually let arguments change their minds.
On that picture, EA is a specifc operationalization/instance of Ideal Reflection for agents who (a) accept some ranking over world-states/world-histories, and (b) want scalable, uncertainty-aware guidance about what to do next.
3) But this mainly helps with the “upward” direction; it doesn’t make the “downward” direction easier. I think of philosophy as stacked layers: at the bottom are the rules of the game; at the top is “what should I do next.” EA (and the surrounding thought infrastructure) clarifies many paths upward once you’ve committed to enough structure to compare outcomes. But it’s not obvious why building effective machinery for action gives us privileged access to bedrock foundations. People have been trying to “go down” for a long time. So in practice a lot of EAs seem to do something like: “axiomatize enough to move, then keep climbing,” with occasional hops between layers when the cracks become salient.
4) At the community level, there’s a coordination story that explains the quasi-objective EV rhetoric and the sensitivity to hidden axioms. Even among “utilitarians,” the shape of the value function can differ a lot — and the best next action can be extremely sensitive to those details (population ethics, welfare weights across species, s-risk vs x-risk prioritization, etc.). Full transparency about deep disagreements can threaten cohesion, so the community ends up facilitating a kind of moral trade: we coordinate around shared methods and mid-level abstractions, and we get the benefits of specialization and shared infrastructure, even without deep convergence.
It’s true—Institutionally, founder effects + decentralization + concentrated resources (in a world with billionaires) create path dependence: once people find a lane and drive — building an org, a research agenda, a funding pipeline — they implicitly assume a set of rules and commit resources accordingly. As the work becomes more specific, certain foundational assumptions become increasingly salient, and it’s easy for implicit axioms to harden and complexify over time. To some extent you can say that is what happened, although on the object level it feels like we have picked pretty good stuff to work on in my view. And charitably, when 80k writes fuzzy definitions of the good, it isn’t necessarily that the employees and org don’t have more specific values, it’s that they think its better to leave it at the level of abstraction to build the best coalition right now. And also that they are trying to help you build up from what you have to making a decision.
Thanks for the post — There is definitely a certain fuzziness at times about value claims in the movement and I have been critical at times of similar things. Also chatgpt edited this but (nearly) all thoughts are my own hope that’s ok!
I see a few threads here that are easy to blur:
1) Metaethics (realism vs anti-realism) is mostly orthogonal to Ideal Reflection.
You can be a realist or anti-realist and still endorse (or reject) a norm like “defer to what an idealized version of you would believe, holding evidence fixed.” Ideal Reflection doesn’t have to claim there’s a stance-independent EV “out there”; it can be a procedural claim about which internal standpoint is authoritative (idealized deliberation vs current snap judgment), and about how to talk when you’re trying to approximate that standpoint. I’m not saying you claimed the opposite exactly but language was a bit confusing to me at times.
2) Ideal Reflection is a metanormative framework; EA is basically a practical operationalization of it.
Ideal Reflection by itself is extremely broad. But on its own it doesn’t tell you what you value, and it doesn’t even guarantee that you can map possible world-histories to an ordinal ranking. It might seem less hand-wavy but its lack of assumptions makes it hard to see what non trivial claims can follow. Once you add enough axioms/structure to make action-guiding comparisons possible (some consequentialist-ish evaluative ranking, plus willingness to act under uncertainty), then you can start building “upward” from reflection to action.
It also seems to me (and is part of what makes EA distinctive) that EA ecosystem was built by unusually self-reflective people — sometimes to a fault — who tried hard to notice when they were rationalizing, to systematize their uncertainty, and to actually let arguments change their minds.
On that picture, EA is a specifc operationalization/instance of Ideal Reflection for agents who (a) accept some ranking over world-states/world-histories, and (b) want scalable, uncertainty-aware guidance about what to do next.
3) But this mainly helps with the “upward” direction; it doesn’t make the “downward” direction easier.
I think of philosophy as stacked layers: at the bottom are the rules of the game; at the top is “what should I do next.” EA (and the surrounding thought infrastructure) clarifies many paths upward once you’ve committed to enough structure to compare outcomes. But it’s not obvious why building effective machinery for action gives us privileged access to bedrock foundations. People have been trying to “go down” for a long time. So in practice a lot of EAs seem to do something like: “axiomatize enough to move, then keep climbing,” with occasional hops between layers when the cracks become salient.
4) At the community level, there’s a coordination story that explains the quasi-objective EV rhetoric and the sensitivity to hidden axioms.
Even among “utilitarians,” the shape of the value function can differ a lot — and the best next action can be extremely sensitive to those details (population ethics, welfare weights across species, s-risk vs x-risk prioritization, etc.). Full transparency about deep disagreements can threaten cohesion, so the community ends up facilitating a kind of moral trade: we coordinate around shared methods and mid-level abstractions, and we get the benefits of specialization and shared infrastructure, even without deep convergence.
It’s true—Institutionally, founder effects + decentralization + concentrated resources (in a world with billionaires) create path dependence: once people find a lane and drive — building an org, a research agenda, a funding pipeline — they implicitly assume a set of rules and commit resources accordingly. As the work becomes more specific, certain foundational assumptions become increasingly salient, and it’s easy for implicit axioms to harden and complexify over time. To some extent you can say that is what happened, although on the object level it feels like we have picked pretty good stuff to work on in my view. And charitably, when 80k writes fuzzy definitions of the good, it isn’t necessarily that the employees and org don’t have more specific values, it’s that they think its better to leave it at the level of abstraction to build the best coalition right now. And also that they are trying to help you build up from what you have to making a decision.