My rule is to take no more clothing than I would wear all at one time in the worst conditions I might encounter on the particular trip. Exception: one extra pair of socks.

I don't take extra undies because I can rinse mine out and they dry in maybe 30-45 minutes--I go without while they're drying. Or I can put them back on wet and my body heat dries them in about that time. I fell in love with ExOfficio underwear several years ago and wear it all the time, not just for backpacking. They make men's underwear, too. I've found it far better than cotton for hot weather.

I don't worry about dirt because everyone else out there is dirty, too. Occasionally, though, there's the problem of food spills, something you probably don't want in the tent. With modern synthetic fabrics you can rinse them out and they'll dry as fast as the undies, whether hanging on a bush or on the body. Or you can just spot-clean. I take a 2-gallon plastic bag to use for washing. I don't use soap, but I still want to dump the water 200 feet away from any water source (I don't want anyone to have to drink my dirty socks!). My rain gear is always available to keep me covered during the laundry process.

My change of clothes stays in my car at the trailhead.