A great question!
This is why I hike in convertible pants/short and strip off after the first hour of hiking. Wind chill makes all the difference though. Even below freezing, I'll still unzip core covering clothes. I rarely ever remove gloves or hat. My fingers/head get cold around 40F and stay that way. Never been frostbit, but got close once on my bicycle with unprotected shoes.