AFAIK, the only time color matters is when the sun is out. If you want your clothing to absorb heat from the sun, then your outermost layer should be black. If you want to stay cooler in the heat, then white clothing is more reflective and may assist in preventing heat absorption (in theory anyway).

Size, on the other hand, DOES matter. Insulation works by trapping air that is then warmed by heat radiating off your body. The "puffier" the garment, the warmer it is, so long as that which makes it puffy isn't too dense (thus displacing air).

Warm air is always going to move toward cold air, and in the process it will generate air currents. The air right at the surface of your body is going to be very warm, and the greater the gradient between the air temperature and your body, the more rapidly the heat will disperse. The "chill" you feel when you are cold is the movement of colder air toward your skin. The greater the distance between your warm body and the cold air (or surface), the less heat loss.

MNS