Waffle,

If your wife carries the baby you would need to carry gear for three. If there is a "situation" you would have a kid too young to fend for themselves or even tell you they are too cold. Maybe car camping the first time would be better. You can then make as many trips to the car as you need for warm gear. Would your child sleep in a sleeping bag with an adult? If he didn't, how would you keep him in a sleeping bag? Sleeping in a hooded sleeping bag is not for someone who could suffocate. I think maybe a NO, in the interest of the babies safety.