Smaller usually IS better for a couple of reasons.

One is torque. A larger pack typically has its center of gravity further from the hiker's back. The distance of that center of gravity increases torque at an exponential rate. It WILL have an impact on comfort and energy expended. If you doubt this, fill your pack with pillows, then attach 2 gallons of water to the back of the pack. Next, move the water to driectly against the back inside the pack and repeat.

The second is more important if you bushwhack. Smaller packs are easier to live with when forced to crawl under downed trees, through brush, etc.

For those alone, I would offer up that smaller packs generally ARE better.