If you do choose to bring a jacket, is there a reason why it has to be down? There are many good synthetic jackets out there that are reasonably priced. Many are warmer than 300 wt fleece but lighter and way less bulky.

I did the JMT (thruhike)in August and I really don't think you need a full blown down jacket to stay warm with the layers you already have. Temps will drop after the sun sets but it won't be terribly cold. If you want to sit around camp you can drape your sleeping bag over you.

Also, have you considered an insulated vest? Down or synthetic, they add an extra layer to warm your core without as much bulk or expense as a jacket.