I'm not sure it would matter whether you're a freak. What works for you is what works for you and the only important thing is that it works.

As for me, when the temps dip near or below my bag's rating, I find I am warmer if I wear fleece inside it. It also helps immensely to do a bit of blood-moving exercise before getting into the bag, and being well-fed and hydrated.

A sleeping bag, as we all should know, can only trap our heat. If you are cold when you enter the bag, you'll stay cold quite a long time.