I totally agree! For me, the heavier the pack, the slower I go uphill, about same on the flat, but need more rest stops to reduce PAIN! The lighter the pack, the more enjoyment. The individual pack makes a difference; the way the weight is distributed in the pack makes a difference too. There is no magic formula. Buy light weight gear, do not take unnecessary stuff, be safe, and your pack simply will be what it will be. With experience, you can further lighten things based on specific conditions and past experience. Every region you travel thorugh is different.
By the way, the heaviest pack I carried was about 100 pounds for 8 miles - carrying out elk meat. I weigh 115 pounds. In the 1970's on NOLS 2-wk winter courses all our packs started at about 70 pounds. Our summer packs, as instructors, with 15 days rations, were always about 60 pounds. We all survived, but did not travel very fast. Our gear was bombproof but heavy. The lightest I have gone lately is 26 pounds for 8 days (had to take an ice axe). My 10-day trip packs (Sierra high country) usually start at 35 (summer) - 40 (shoulder season) pounds. Also, as a rule of thumb, if your food is over 2 pounds per day, you are probably taking too much or the wrong kind.
I do not claim to be an UL backpacker or even desire to be. I just want to enjoy myself and get by with the gear I have. When something wears out, I buy a lighter item. Gradually I am getting lighter.