How old is the sleeping bag? Is it designed so the down can be shifted from "top" to "bottom" within the baffles? (I'm not familiar with the "new" North Face product line. They used to be a hard-core outdoor gear company, back in the 80s. Now, not so much; I haven't heard many really good things about their newer products, and don't see them very much in the backcountry. In the 80s, everybody had their stuff.)

But I digress. If the down is shiftable, it could be that it is all in the bottom of the bag, which means that when you're on your back, all the down is underneath you, squashed, and you're getting no insulation on your belly area.

Also, if you choose to answer, do you have much body fat? I had a Boy Scout who had very little body fat (he couldn't float long enough to pass his swimming test.) He used to sleep cold on fairly warm nights, too; he usually had to wear clothing inside the bag.