You are definitely correct that you don't need insulation under (except for your pad). But...the biggest issue with quilts is heat leaking around the sides as you shift in the night. I think you want some insulation near that interface. A bit of it may get compressed, but it could add some buffer as you shift in the night.