Three factors come to mind, for me at least.

(1) The temperature literally does change. I've seen different figures for how much; this site says about 4 degrees F per 1000' of elevation.

(2) At higher elevations for me in the Sierras, there was snow, or snow melt and associated runoff streams. Sleeping below that level avoided issues with sleeping on snow or wet ground, and in particular, perhaps needing more ground insulation.

(3) Higher elevations can also be windy; more of a factor for me in my single-walled tent than someone with a beefier tent, but nevertheless potentially a factor. Of course higher doesn't have to be windier, but overall I think site selection to favor warmth can be easier lower down.

In terms of your comment about the same elevation seeming colder --- perhaps humidity has been a factor? A lot of folks (myself included) feel that the same temperature and wind conditions feel colder when it's damp (foggy, mist, whatever).

Of course there are other factors that could impact how you felt literally in your own backyard vs. in a place that you hiked to with all of your gear on your back ...

I'm certainly not saying you're wrong here (!), just musing out loud I guess.