I wouldn't think about it all if you're hiking by the other way. I'd be out of hearing range in short enough time to not give it a thought. If you were passing me on a trail I'd probably (depending on what you're playing) just take a break and let you put some distance between us.

Avoiding those who do like music is also part of outdoor etiquette. I'd never hike for long nearby, or set up camp next to, someone who's listening to music and demand they shut it off.