I have a down vest, but it usually only gets used when temperatures drop below freezing. Sometimes, when it's that cold, my midweight longjohn top over my T-shirt isn't warm enough, even with a windbreaker over it, while I hike - but my hooded, "sweater" weight down jacket would be way too hot.
My down jacket is midweight ("sweater" means the same thing.) I rarely need a full-on heavy down jacket or coat, it just doesn't get that cold around here most of the time. But, on those rare occasions when it does, I can layer the vest under the jacket and be plenty warm.