would it make any difference if the first layer (or any layer for that matter) is white or black?

This is an interesting theoretical question, but I have a hunch that any actual difference this makes would be so mind-bogglingly tiny that it would be dwarfed by the heat loss from simply breathing or shifting your body.