Here's another thought, triggered by you mentioning that you're a big guy whose feet touch the end of the bag.

Could the bag be too small? If your feet press against the foot of the bag, or your chest and shoulders are broad enought that they just barely allow the zipper to close, you'll be pressing the bag lining against the outer shell of the bag. This will compress the fill, eliminating its insulating value.

This is one of the things you have to be careful with when relying on clothing to extend the range of a bag: too much clothing will have the same effect of compressing the insulation.

Kind of makes a fellow feel like Goldilocks: this bag is too big, this bag is too small - if only I could find the one that was just right!