I think the issue is not the kid but the parent. The right age is anytime the parent is willing to put in all the extra work it takes and make the trip a "kid" trip and you have means to carry the extra stuff. Nowadays with light gear this really is not a big problem. The baby backpacking stuff is harder, because you have to do the diaper thing.

I took my oldest daughter on her first backpack (actually 3-day wilderness canoe trip) at about 4 months old. Then a weeks backpack at 15 months. Two week trip at 16 months. My younger daughter (2 years younger) did her fist trip at 16 months, in fact, while I was in the process of weaning her. We did a family trip, horse packed in supplies, had kids age 16 mo. 4 yrs, 6 yrs, 10yrs 12 yrs. The major activity was fishing. Then I did a 5 day trip with my younger daughter when she was 5. She carried a pack with all the cook pots - loved to hear them clank as we went down the trail. We had a packer take us in and we walked out. Also did a solo weekend trip with both kids, age 2 and 4, while the younger was being potty trained. That I would not do again. I feel you minimally need one adult for each child.

I have seen plenty of 6-year olds backpacking and they do quite well. If your kid already enjoys the outdoors he will do just fine, as long as you keep in mind his interests and plan accordingly. Actually some kids will walk miles between point A and B, but most of that age get quite bored and do better at a few miles with plenty of in-camp activities.