You might like to start with a pattern for insulated lined pants like this:

You could then sew in channels for holding the down in place before completing the side seams.

I found down terrible to work with. I would strongly advise wearing a dust mask while stuffing the pants.

Just as a matter of curiosity, why down rather than a more easily handled and more easily washed insulation?
Mad sewing witch!