I use fleece! I've never had any trouble with moisture, at least not with high quality fleece. It can be fairly bulky and heavy, especially if you're comparing to down, but I've never had any trouble with moisture on my good fleece jackets. They both shed water fairly well, and even if they get soaked, they retain a lot of their insulation value. And they dry out quicker than I would expect.
If you can wear it under your rain layer, not a problem anyway.

My current fleece jacket is from Land's End - I'm not sure what style, I got it from my boss the summer I worked in Alaska (2007). Looks similar to the T-200 but it has interior pockets as well. The side tag says "Windcheck." It's very light and thin for as warm as it is. I wear it down to the 20's with only a T-shirt for a base layer (granted I'm a little bit more warm-blooded than most). If you're looking for something more reasonably priced and you're not planning on sub-zero temps, it would be something to consider.