I have the Patagonia down sweater (the "hoodie" pullover version from last year.) Over a 100-wt fleece top, it's quite warm down to freezing or so, just sitting around camp. Although it looks thin, this sort of insulated layer adds a lot of warmth. Putting a rain shell over it adds even more, if needed.

I also have a Patagonia Micropuff Pullover, which is several ounces lighter, and seems to be about the same warmth. But it doesn't have a hood, nor any useful pockets.