Similar to the hot water bottle concept- you really need to get into the sleeping bag when you are warmed up. I usually do a intense small hike or run around a bit before I jump into the sleeping bag. The sleeping bag is an insulator- not a heater. YOU have to provide the initial heat. Standing out around a fire is really not a good way to warm up - one side will be warm, the back side will be cold.

When my sleeping pad is insufficient, I can really feel the cold from below and it is quite obvious. By the way, I sleep very cold, and my 5-degree bag is perfect for me at 30 degrees. You may want to borrow a similar rated sleeping bag from a friend, and see if it is the bag, the pad or you.