Guess what?!?

Sleeping pads don't need to be rolled. All a tight roll does is give you a hard, cylindrical shape that doesn't pack well with other items. Just fold it in half or thirds and stick it in your pack before stuffing other items in. It will conform to whatever you are packing and take up less effective volume in the end. And you don't have to worry about messing around with a stuff sack.

The fundamental concept is that bags are good for packing items together that you want keep together and keep from disappearing into the far reaches of your pack. But using a stuff sack for a single, non-losable item? Neither necessary nor lightweight.

(Same goes for a sleeping bag, too.)