I agree that ignoring psychological harms completely is arbitrary. Many people would prefer moderate physical pain to public humiliation and this seems pretty hard-wired in our psychology.
At the same time, in the current climate claims of psychological harm are clearly used strategically. People supposedly feel unsafe if a colleague has political views that they disagree with for example, which clearly is not some sort of universal fact of human psychology. Certain claims of emotional harm should be discounted not because they are necessarily false, but because indulging them leads to a bad equilibrium.
It is not obvious that non-extinction is an attractor state. If there is some minimal background risk of extinction that we can not get below (whether due to asteroids, false vacuum decay, nuclear war, everyone becoming a negative utilitarian and stops reproducing, whatever) then it is the nature of exponential discounting that the very long-term future can quickly become essentially unimportant.