Each of us is different. A vest just doesn't work for me; when it's cold I need covering on my arms and, above all, on my shoulders, which tend to get arthritic if not adequately insulated.

I also don't use down for clothing. For one thing, I prefer synthetic clothing since I have a down bag. For another, I am worried about body moisture condensing inside the outer layer of a down jacket in freezing weather. You can prevent this with a vapor barrier liner in a sleeping bag, but I sure wouldn't want to wear a vapor barrier during active hiking! I'd rather have several thinner synthetic layers. I like to fine-tune my insulation so I don't sweat when hiking and don't freeze when stopping, even if this means putting going through the onion peeling/unpeeling routine every time.

Your Mileage May (and probably will) Vary. You might consider some trials with an inexpensive fleece jacket and vest (which you should be able to get cheap at a thrift store) before investing in something more expensive).