I think I see the problem here. The water was cold to start with and you had it in the pocket of your outer layer. If you move it a layer or two closer to your body you should be able to keep it warm.