If you sew enough patches on that it starts to look like one of those old trunks in the attic, yeah, people will smirk.

My buddy, who thru-hiked the AT, has that AT patch sewn on his pack. No one laughs. (Of course, he's also an ex-SEAL and looks the part.)

One patch, with some significance, is probably OK. Smaller patches don't look bad on a hat, though. (I've got a Scout Backpacking Merit Badge the aforementioned buddy gave me after I was his counseler for it. He told me, "You did everything I did; you earned it too." I wear it proudly on my hat, and have transferred it to a new hat a couple of times. That particular patch, I don't care if anyone does laugh. It means a lot to me.)