I bought an early Gore-Tex bivy that was cordura on the bottom. One night on a winter trip in the White Mt.s of NH, I managed to roll around and put the nylon on the top. The bag got wet as the bivy trapped moisture. I think the bag was Hollowfill, so it didn't matter too much, and I dried it carefully with a Gaz backpacking tent heater. I think in certain conditions your down bag would get pretty damp and be hard to use a second night if you didn't have the time or conditions to dry it out. I'd also be highly skeptical of durability. Those things are meant for limited use. SOL makes a claimed breathable e-bivy, called a "Respirant" bivy, but you'd have to read the reviews. They produced it because of how notorious radiant bivy's are for trapping moisture.
I didn't mean to parrot what Oregon Mouse said, she posted hers while I was still composing mine. She hit the nail squarely, as did Aimless.