Which reminds me: load distribution also matters. (The OP may already be aware of this; if so, my apologies - I wasn't trying to insult your intelligence.)

Lori's right about loading the packs when you try them on - I'd add to use the actual gear you plan to carry, if possible. (Pick a day when the shop you visit isn't too busy, because you will use up a lot of floor space and their time.) Each pack will probably have some features that will let you arrange your gear a bit differently - generally, loading heavy things into rear pockets will tend to pull the center of gravity back, and into side pockets will affect the side-to-side balance of the load. (Personally, I don't like a lot of pockets, but I do like the rear "corner" pockets on the Kestrel - they expand inward rather than out, and are a great place to isolate my rain gear without affecting the center of gravity.)

Using your actual gear to test-load will reveal pretty quickly any pack eccentricities that affect the fit.