You are right OM! I am out backpacking. Just got back from trip #3 in the Wind Rivers. Have a week off then a 12-day trip.
I took my babies backpacking; by myself, with my husband, with girl friends. Best for me was with girlfriends. My husband was little help since once he saw a fish in a lake, I never saw HIM again for the rest of the day. My girlfriends, who had kids too, were very helpful. I tried two kids (2 and 4) by myself and it was a disaster. Did a trip alone with my 5 year old and that worked out very well- we had a horse packer take us in and then slowly walked out 20 miles (mostly downhill). My kids were small for their age so I could carry them in my pack until they were about 2 1/2. I remade my Kelty pack- added a canopy, sling seat and diaper carrier on top of the canopy.
I had no qualms about putting a harness on my little ones and tying a lanyard to a tree so they would not run off while I had to do camp chores.
A good friend of mine took all three of her kids for one-month long sojourns in the wilderness with her two pack horses. She hired a high school kid as a babysitter. Said she would never consider a trip with small children without a babysitter. You may be able to find a qualified babysitter who also backpacks.
One thing to really watch for with a kid in a backpack, is that they are sitting and you are walking. When it gets cold you may be warm but they could be subject to hypothermia. Let them out often and check their hand and feet temperature often.
You also should take a course in wilderness medicine and first aid as well as talk to your peditritian about specifics in first aid for children.
On and on-- but no, it is not crazy to take your kid backpacking.