You might like to start with a pattern for insulated lined pants like this: http://www.jalie.com/sewingpatterns/patterncovers/958.html
You could then sew in channels for holding the down in place before completing the side seams.
I found down terrible to work with. I would strongly advise wearing a dust mask while stuffing the pants.
Just as a matter of curiosity, why down rather than a more easily handled and more easily washed insulation?