In the old days, before belay devices, we wore "belay gloves" to keep from getting rope burns. Good leather gloves also keep your hands warm. The gloves have to made of "grippy" material. Hands stay warmer if your body is warmer. I seldom have trouble keeping hand warm as long as my boldy is warm. A beefy down jacket works. You can wear a stocking cap under your helmet. Also, wear warm boots when belaying. Change into climbing shoes only when climbing. This works for top-roping but not for lead climbing. Bring a square of ensolite so you are not sitting directly on cold rock for those belays where you can sit. External warmth is always good too. Hand warmers and thermos both good ideas.
BUT, be willing to call it quits if the belayer gets so cold that he/she cannot do the job. Also, it helps to do shorter climbs so you can switch from belaying to climbing more often. One reason I really like Joshua Tree for winter climbing is that there are hundreds of shorter climbs.
There is an old saying - climbers have two time clocks - the climber's clock and the belayer's clock. Particularly when lead climbing, time flies. Each minute drags on and on for the poor belayer.