I like my hiking shirts to be a bit over-sized so there is good air circulation. If long sleeved - you can roll up the sleeves. I change clothes A LOT when hiking. It only takes a few minutes. One trip I did in late Fall, it turned unusually warm and I had only taken fleece! I did have two handkerchiefs so I tied them together and made a 70's hippy style top for modesty. You guys can just take off your shirt and hike in the buff.

And, another thought- if you are sweating and panting, maybe you are just going too fast. Slow down to a steady moderate pace and take few if any rest stops.