My trips in summer/fall are mostly Sierra. I take a midweight base layer plus a thick pair of wool socks, usually only for sleeping, and wear nylon lightweight pants, a light wool or synthetic shirt, a sun hat, and trail runners. I use lighter weight wool socks. I have with me a fleece beanie, liner gloves, possibly a pair of heavier gloves to go over them if forecasts are for subfreezing temps already, and the waterproof layer of choice - if I'm out for more than a couple days I take a rain jacket, rain pants, and emergency poncho. If just for a few days the jacket and poncho. My midlayer is a down jacket - Marmot Venus, really not a big puffy one for lower than freezing - or, a Montbell Thermawrap - synthetic but about the same warmth level of the Marmot jacket.

You don't need a big box baffled down jacket. Look at Eddie Bauer - they have a sub-100 down jacket, lower quality down than the Marmot jackets for sure, probably leaks feathers, but it will work. You can also find down jackets on sale elsewhere. My Marmot I found at REI on a 30% off sale - it was originally 120 I think. Check at Sierra Trading Post and other outlets. Or check backpacking forums - my Thermawrap was a used buy, from someone at backpackinglight, for half its full retail. Not a thing wrong with it.

Previous to the down jacket I took a 300 wt fleece - but that wasn't nearly as warm for me.

You could also (if the weather is fair and clear and you're not sitting at a fire) sit around camp in your sleeping bag.

One of the things I discovered as I began to hike more and more was that my metabolism shifted - I am able to withstand cold better than I used to. And I'm also better at not getting cold in the first place; layer management is a skill. Put on the hat first, with the liner gloves. Then think about a jacket if you're still cold. When I'm stopping for the day before sundown, I put on the base layer immediately and put the pants back on over them for wandering around camp. Gives me a chance to wipe down some, clean the feet, put on the clean camp socks, and get ready for bed well before the temps drop for the evening. (I got good at Wet Wipe bathing.)