The other issue Kate brought up was the quilting. I didn't want to quilt it as it would cause seams and heat loss though it. I know sleepingbags have offset seams when quilting but I can't see that working for me. While I didn't want to pin or thread baste, compared to quilting I think it might be better. Both for keeping loft and my amateur skills.
Primaloft Sport is soft, has a good clo value (0.74), and drapes well. But in your quilt design, its requirement of unquilted areas no larger than two square feet is going to be a problem, especially as you have stated that you don't want to stabilize the interior panel of your design.
Primaloft One is even harder to work with. Those sub-denier fibers are so fine you're supposed to limit unquilted areas to 6 square inches.
Leaving either one unquilted over large spans is going to result in an insulation tear. You might get a little time out of sport before it tears, but Primaloft One without quilting on large panels is a real show-stopper.
You might consider a continuous filament insulation instead as they also have high clo values (0.82 for XP, 0.78 for Combat, 0.67 polarguard delta, 0.63 polarguard 3D, 0.65 climashield green). More importantly to your design, though, is that they are much better at spanning your big panels without interior quilting. You still might tear it with rough treatment, but it'd be far, far less likely than with Primaloft.
Primaloft's primary application is in apparel where things like softness, drape, and lack of bulk are of prime importance.