Babies are easier to backpack with than toddlers. As a mom, feeding was no problem because they totally nursed. Used diapers were burned every night - we always had campfires back then. But from the 2-4 ages, we had horse packers bring in a base camp and take out the gear (we still walked/carried the kids). From about 5-6 we graduated to packers on the inbound trip and walking out carrying all our gear. I sewed all my kids backpack clothing from merino wool sweaters I bought at the second hand store. I also had access to a lot of free rip-stop nylon scraps.

I do not think little kids care if it is backpacking or car camping. They do not care if it is a small puddle of a lake or a huge lake in a spectacular setting. Just getting them accustomed to being outdoors, day and night will allow them to be comfortable backpacking later. I took my kids backpacking not because they preferred it, but I wanted it. And you have to want it really bad, because it is a LOT of work!

I got plenty of criticism for taking my kids out when they were infants. But as a nursing mom, they simply had to go where I went and I was not about to give up backpacking for several years. Like you said, they survived.