I wear less or none on warmer nights and more and more on colder and colder nights. Long wool underwear are particularly good for capturing and recovering heat from body vapour, as long as you get them dry again the next day. Wool sweaters are particularly good for acting as an extra baffle and heat exchanger for any cold air or warm air finding its way down into or up out of the lower part of the bag from the head and neck area. Wool hats and neck tubes and insulted hoods in the head and neck area add alot of warmth on extra cold nights. Breathing through a knit wool neck tube or scarf is important for heat recovery when it gets really really cold, like sub zeroF.

All that said, there is no substitute for a well designed and well fitted bag built for the temperature range you need it for. Nothing like wearing next to nothing and getting a very good sleep with little effort on a very cold night. Still working on that. My bag is good to close to 0F if I have had a good meal and not too exausted, but below 0F it gets interesting, and a long night below -20F gets horrifying, even in my back yard. How folks manage -50F is beyond me.