Depends on where you are. In a national park or protected area, no. But then you are digging through top soil that took many years to form, for your cat hole.
Around here, plants like lambs ear grow like weeds and pulling off a leaf or two isn't a problem. Use what you want, but don't kid yourself about TP being an 'environmental friendly' solution.