I don’t exactly claiming to have identified a problem with the counterfactual function, in itself. The counterfactual is perfectly well defined, and I like it, and it has done nothing wrong. I understand this. It is clear to me that it can’t be added just like that. The function, per se, is *fine*.

What I’m claiming is that, because it can’t be aggregated, it is not the right function to think about in terms of assigning impact to people in the context of groups. I am arguing about the area of applicability of the function, not about the function. I am claiming that, if you are optimizing for counterfactual impact in terms of groups, pitfalls may arise.

It’s like, when you first see for the same time: −1 = sqrt(-1)*sqrt(-1) = sqrt((-1)*(-1)) = sqrt(1) = 1, therefore −1 = 1, and you *can’t see the mistake*. It’s not that the sqrt function is wrong, it’s that you’re using it outside it’s limited fiefdom, so something breaks. I hope the example proved amusing.

I’m not only making statements about the counterfactual function, I’m also making statements about the concept which people have in your head which is called “impact”, and how that concept doesn’t map to counterfactual impact some of the time, and about how, if you had to map that concept to a mathematical function, the Shapley value is a better candidate.

Fair point re: uncertainty. The situation seems pretty symmetric, though: if a politician builds roads just to get votes, and an NGO steps in and does something valuable with that, the politician’s counterfactual impact is still the same as the NGO’s, so both the Shapley value and counterfactuals have that problem (?). Maybe one can exclude agents acording to how close their goals are to yours, e.g., totally exclude a paperclip maximizer from both counterfactual and Shapley value calculations, and apply order indifference to allies only (?). This is something I haven’t though about; thanks for pointing it out.

Fair point re: epistemic status. Changed my epistemic status.