I actually have done this to a lesser degree. I have a set of Montbell Down Inner pants and jacket. They are pretty thin as far as down clothing goes. They are my winter "pajamas". I find that I can get away with a lighter bag when I use them, but I would not consider going without a bag at all for the reasons I stated above.
_________________________
If you only travel on sunny days you will never reach your destination.*

* May not apply at certain latitudes in Canada and elsewhere.