So two things:

1. You are right, the fit is what matters, and if your body changes a lot, that will change how pack fits.

2. The standard advice is to buy all the gear that goes in your pack first...as that will then tell how big a pack you need.

You don't say what you're going to be carrying in this pack, but we've day-hiked for years and years with a small daypack. The bigger the pack, the more likely you are to fill it with stuff you won't use.

Hope that helps.