The point about EA only requiring a small number of unselfish people is quite right and I actually write about these sorts of dilemmas elsewhere. (https://philpapers.org/rec/VENIAS)
One thing I’d say in this context, though, is that if the case against socialism is: “Well, a small minority can be trusted to have less selfish motivations, but the general public can’t—and we’re in the small minority”—this starts to look like a kind of elitism that the left often worries about EA encapsulating.
If EAs can do it, why think that (especially under different social conditions) most people couldn’t? This would require some thought that EAs are special, which, given the elite, white, rich, etc. skew of EA is a worrying one.
I think this is all really fair. I don’t have time to pursue these things much further right now, but I will try to circle back to this. Thanks for the thorough engagement!