I find that I can't talk meaningfully about someone else's gear if I'm not looking at a complete list. You list 2, arguably 3 items for clothing, and I'm pretty confident that you're carrying more than that. I'm sure there's significant other stuff that's not included --- toilet paper, toothbrush, etc etc (etc etc) that I don't see listed.

Pack up completely, or better yet, build a list the moment you get home from a trip so you have absolutely everything still together. Build a spreadsheet showing all of that stuff, list everything, and give the accurate weight for everything.

Now you can start to really analyze.

But the clothing sticks out for me. I find that amount of spare clothing carried is the number 1 thing that makes packs heavier and bulkier. Perhaps that's an issue for you too.
I'm not just talking rain gear here, but underwear, socks, mittens/gloves, warm hat, whatever.

It would also help to know your overall (gear) base weight. That at least you can get by just packing up --- again *everything* --- but leaving out food, water, and fuel. But include any food containers, water container(s), fuel containers, it's just the consumables omitted from base weight. Then stand on a bathroom scale with and without your pack, subtract to get the base weight.

Of course it's possible that at least part of the issue is in the amount of food and/or water and/or fuel that you're carrying. But separating out the consumables helps to narrow things down, and to compare to what others find comfortable.

Best of luck in getting this to where you're happy!