I usually wear clothes because (1) they are a part of my sleep system and they keep me warmer and (2) I don't want my dirty hide touching my relatively clean sleeping bag.

Exception: Summers in the desert when it's freakin' hot even at 1AM. I'm usually not hiking then. I'm kayaking. I won't use clothes......or a bag....or even a shelter....until the cooler predawn hours. That's when I'm solo blush
_________________________
If you only travel on sunny days you will never reach your destination.*

* May not apply at certain latitudes in Canada and elsewhere.