I do exactly that, engage with comments I think are wrong, or misunderstood, or even harassing, but reserve downvotes for stuff that is really in the wrong place or very obviously rude or offensive, or just mouthbreathing stupid, like hatred or revenge wanking.

Votes are for quality, not conformity.