Consider that just by switching to a lighter weight pack, you could offset the weight of the weapon. Since this is a lightweight backpacking forum, we can't help but suggest that!

I suspect that you are going to have to try various combinations to determine what works best for you as an individual. I would think that the pack's shoulder straps and sternum strap would interfere with the chest holster.
_________________________
May your trails be crooked, winding, lonesome, dangerous, leading to the most amazing view--E. Abbey