The sort of absolutism seems like it could be concerning. For example, things like eugenics, killing off mankind to save animals (I think that was a plot of a James bond film?), and other such clearly negative things all can come out of this type of optimization of trying to do the most good.
It’s kind of like AI-risk, where if you tell an AI to ‘save the most lives’ who knows what it will do. If your ideology is predicated on “doing the most good” it seems you have to be extremely careful to avoid falling into doing the most bad.
In contrast, if you only try to do “some good” or “more good” you can probably find ways to have minimal risk of doing bad and can almost definitely avoid ‘doing the most bad’.
The way to solve this is to look at actions in terms of both their positive and negative consequences. You want a way to reach some kind of summary conclusion about the set of consequences of a particular action (so that you can say, for example, “Overall, the action was altruistic”), but you also want a distinction between the summary and its components, so that individual consequences don’t go ignored.
EA folks think that they can find a useful probability for most any outcome, a probability that reflects something different than how well the outcome matches the consequences of a system of causes. They can’t, but they go on assigning epistemic confidence to their important decisions and assertions.
An EA person can say “I 70% believe taking action A will have consequence X, and 30% believe that it will have consequence Y”, and seem reasonable to their peers.
Suppose consequence X was “the most good” and consequence Y was “the most bad”. From this they calculate an “Expected Value”, multiplying their %’s of beliefs by the utility (the good) of their possible consequences.
So .70 times “the most good” (a positive value) plus .30 times “the most bad” (a negative value) , in the example, would give an expected value of “kinda good”.
The actual consequence, though, is either “the most good” or “the most bad”, not “kinda good”. When EA folks talk about doing “the most good” they are using estimates like expected value, and the calculation results are probably closer to “kinda good” or maybe “pretty good”, rather than “the most good”, which might have been one of the outcomes, but not what they were going for, ironically.
I think that’s a partial answer to your question. Your idea that incremental improvements can be more reliable than a big jump is a good one, I agree that it’s less risky and more wise, in some cases.