I agree that you should let your dog off if you want to and I what I tend to do is keep Hamish away from dogs who are ON their leads. I think that the dog off can tease and torture the dog on the lease, who in turn feels it cannot defend itself and may act out aggressively because of that.
I dont put Hamish on the lead, but I keep him away or if the path is narrow, I put him on to avoid him 'worrying' the other dog.
But i have never been spoken to like that!