I cannot imagine sleeping directly on top of any insulating pad. With a sleeping bag, you do not do this. With a blanket, you do. I suspect that the cold is due to moisture. The pad is waterproof-airproof, so your body moisture will collect on it. Moisture/humidity creates on your skin acts like air conditioning.

Possibly you could sew (or tape together) a pad cover- space blanket on bottom and 100-wt fleece on top. Sew or tape three sides and slip your sleeping pad in the open end.