Thanks for this post Aaron! I especially value the part about asymptotic structure instead of the sharp thresholds.
I think it’s useful for those interested in the topic to highlight the connection of your post to existing EA discussion and academic literature on ethics and decision theory. Here are some of the things I’m aware of:
As @Robi Rahman🔸 points out, the post is arguing for an established view: lexical (negative) utilitarianism. I think The Center on Reducing Suffering’s critique of Toby Ord’s blog post “Why I’m not a negative utilitarian” is a relevant discussion of negative utilitarianism that also touches on lexicality and can work as a “common misconceptions to avoid” for people new to the idea.
Like you, Academian takes a critical look at the vNM axioms and argues against continuity being required in a LessWrong post. They highlight the original lexicality paper: Melvin Hausner’s 1954 “Multidimensional utilities” that weakens the continuity axiom of vNM, producing lexicality.
Teo Ajantaival has an entire chapter in his book/sequence “Minimalist Axiologies” on “Doesn’t this endorse destroying the world?”.
On a more general note, vNM hasn’t been the favored framework in normative decision theory for a while. The frameworks of Savage and then Jeffrey-Bolker have provided increasingly realistic/reasonable setups while keeping the utilities and most of the properties of the axioms. I think Richard Bradley’s 2017 book “Decision Theory with a Human Face” provides the canonical background for current normative decision theory.
In contrast to what you argue in footnote 21, completeness and transitivity have both been challenged (I think successfully). Especially Suzumura consistency and representor models (more on incompleteness in belief in Anthony DiGiovanni’s comment) are attractive alternatives as they are more appropriate for real agents like us and still invulnerable to value pumps.
Also, I think that in a precise baysian framework, the strongest argument against lexicality is that it is irrelevant because the EV of two options will almost never be exactly the same. This changes if we incorporate imprecision because then it is quite possible that the primary utility doesn’t provide a preference over two options and instead gives comparative indeterminacy which one could consider similar to indifference for the purpose of lexicality. That is to say, I think lexicality is more action-guidance relevant and conceptually attractive in an, arguably better, imprecise framework.
Option 3 has been made precise with bracketing: overview and paper. I think this shows why it is not clear how “some subset” is defined even when we have a clear idea of what we want it to track (for top-down bracketing: the effects we’re not clueless about taken as widely as possible).
The problem with carving up effects is with choosing which locations of value to discount when they have determinate sign on their own (so not clueless) but not in aggregate (so clueless). Clifton gives a clear example in the overview post and this seems to apply to your CHA: an intervention reducing animal product consumption has positive EV for farmed animals, negative for wild animals, and indeterminate in aggregate. I do not think that the test you propose resolves this conflict since both groupings pass it (each has determinate sign on its own). So there may be multiple conflicting ways to draw the cluelessness horizon and picking some privileged way to draw them requires justifying it over all the others.