Priors should matter! For example, early rationalists were (rightfully) criticized for being too open to arguments from white nationalists, believing they should only look at the argument itself rather than the source. It isn’t good epistemics to ignore the source of an argument and their potential biases (though it isn’t good epistemics to dismiss them out of hand either based on that, of course).
I don’t see a dichotomy between “ignoring the source of an argument and their potential biases” and downvoting a multi-paragraph comment on the grounds that it used less-than-charitable language about Silicone Valley billionaires.
Based on your final line I’m not sure we disagree?