I wear insulation such as wool or fleece under my wind layer. My down layers, however, I wear on the outside. They are made with a fairly wind resistant fabric.

Now to add one more caveat- when I wear down in wet conditions, I wear it underneath my rain layer. I normally am not wearing the wind layer in these situations.