I use "Dollar Store" nylon zip offs and fleece pajama pants underneath when it's cold. If it's freezing I'll add some long undies under the fleece.

That combo works from hot to cold for me and it's light enough to carry around and cheap enough to beat up without worrying about it.