In a lot of places with dusty trails, if you change clothes you'll be grubby again within an hour. Why bother? Everyone else out there stinks, too! I therefore don't worry about laundry, except for rinsing socks every day. And the only spare clothing I take is an extra pair or two of socks. My undies are quick-drying enough that I can go without for the short time it takes to dry them, or I can put them on damp and my body will dry them very quickly.

I sponge myself off (at least the critical areas) daily and I do keep my base layer (which I sleep in) clean and dry. If I wear the base layer outside the tent (usually on cold mornings), I keep it under my outer clothing and change out of it before I start hiking so I don't sweat in it.

An important note--even biodegradable soap kills aquatic life, and so do the residues of sunscreen and insect repellent left on your skin. Please rinse off at least 200 feet away from active or dormant water sources before swimming! Better yet, leave the soap at home!

I leave a package of moist towlettes and a change of clothing in my car at the trailhead to lessen the funk on my way home, at least if I'm going to be stopping at a restaurant or motel on the way.

EDIT (re being cold at night): The NeoAir is undoubtedly your problem. Either get an insulated air pad (those little baffles in the NeoAir that supposedly reflect your body heat back to you didn't work for me below 40*) or get a 1/3 to 1/2" thick CCF pad to use with the NeoAir when temps get below 35*. Having to use that supplemental CCF pad unfortunately eliminates any weight savings from that highly expensive and, IMHO, overrated piece of gear, which is why mine went back to REI.