Greetings! I have 3 kids ages 0-8. I frequently take them backpacking. In my experience, the earlier you start them, the better. But ease into it. Start with overnight camping trips including some hiking, then move into backpacking once you have the routine down. Kids need help with many of the routine tasks, so give yourself plenty of time.
Don't expect to crank out 18-mile-days, 6 miles is more realistic with little kids

I am in process of writing a series of articles about Backpacking with Kids (and other family outdoor activities), including hot-to guides, tips & tricks, etc. I'm posting the material here: http://outsidedads.com

Good Luck!