I would worry about the air draft. If I turn over with a sleeping bag, I can have it cinched so that no air gets in. If I turn on my side, the sleeping bag goes with me: it hugs around my body, whereas I think a quilt would be like what I have on my bed: something that separates from my back. How do you get around this?

If I could try it, and not have to deal with the time/expense if I didn't like it, then I would. I just don't have experience with it so it's too much a risk for me to try.