It seems that you have slept on boughs before, You should use an insulated pad it does make a big difference, also why didn't you sleep in your fleece? That would make you warmer too. Contrary to old time thought, sleeping naked is not the warmest, and sleeping in cotton is the coldest. You could put a coat over you and be warmer too. Do you wear gloves in your sleeping bag? Light fleece gloves are nice to keep hands warm. and sleep in a hat as they say you lose half of your heat through your head.
Jim