I prefer sleeping in the nude, especially since my guy and I zip our bags together, but there have been a couple of trips when it was so cold we kept our base layers on.
It all boils down to your own personal comfort. And no, that doesn't make you a freak. Anyone who thinks you a freak for that is the one with the problem.