Many of these concerns seem to be symmetric, and would also imply we should make it harder to upvote.
Yes, that’s a good point, I’ve been wondering about this as well. According to one (pretty common) approach to argumentation, an argument is acceptable unless challenged by a counterargument. From that perspective:
upvoting = an acknowledgement of the absence of a counterargument.
downvoting = an observation that there is a counterargument, in which case it should be stated.
This is just an idea from the top of my head, I’d be curious to discuss this in more detail since I find it genuinely curious :)