Conduction, convection, and evaporation are the primary mechanisms of heat transfer to and from the body. It is true that radiation becomes significant on a clear day when darker colors will absorb more energy radiated by the sun. It is primarily the outer layer that matters, although inner layers will determine how much of the heat gets conducted to your skin. To a lesser degree, in the middle of a clear night radiation from the body is more significant and darker colors will radiate more. But this will be on a much smaller scale than the energy absorbed by the sun.

It's more than just color, too. Texture, density, etc. can affect radiative heat transfer. The proverbial black body is not called that because of its color, BTW.