I find that I have a critical weight point. Under it, a pound lighter is not that big of a deal; over it, the pound does make a difference. This point also goes down as one ages! It also depends on your size and your pack. Although larger packs are heavier, their suspension system makes carrying the weight a bit more comfortable than a comparable pack with poorer suspension system. I have several packs for different length trips.
My personal breakpoints: If I can get my pack weight to about 20 pounds, going lighter does not make much difference. From 20-35 pounds two pounds lighter makes a big difference, but not 6 ounces. Over 35 pounds I suffer greatly - it really is the straw that breaks the back. On my longer trips I usually start with 35-40 pounds and suffer for the first 3-4 days. But the weight goes down as I eat up the food. I just do a bit less miles or elevation gain the first few days. It is not so much that I cannot make the miles, but suffer vs. enjoyment.
Lighter is better as long as you stay safe and have enough of your stuff you need to do whatever activity you enjoy most. I would NEVER leave my camera behind to save 8 oz!
Compare what it "costs" in to delete a pound of gear vs. a pound of water carried, or food. One pound food is a short day ration- on a 12 day trip, I just take 11 days food and so be it if I have to walk out the last day on an empty stomach. Most people carry too much water. If you are careful with planning with respect to water sources you need not carry so much. I rarely carry over a liter of water- most the time 1/2 to 2/3 a liter. And as others have said, each extra pound on your belly is like dead weight in your pack.