I think this post sums things up well. The down will compress, especially under the straps and up the back, and at some point will become moist from sweat. I won't, however, be damaged and will bounce back.

Since I see you're in Minnesota I won't use my standard, "Why would you backpack in a down jacket?!?" but will suggest that fleece or even fiberfill are a better choice on the go, saving the down stuff for camp. But I don't hike in sub zero weather, so what do I know?

Whoops!--MN/NM/NH/HI--disgraphia kicking in today. Okay, unless you're above 12k feet, forget about backpacking wearing down entirely :-)