A couple thoughts, scottyb...

I think you will likely be capable of hauling your own pack out of the canyon. You'll want to pack relatively light anyway. It's different than backpacking and very plush in terms of food, drink, etc., but most of the heavy gear is group stuff, and your personal stuff can be reasonably light if you check around and follow the outfitters general suggestions.

On the other hand, if you have anxiety about your ability to do the Bright Angel hike comfortably, you should inquire further about the duffle service. You are right that the boat will drop you off at Pipe Creek, which makes sense for walking, but the boats will pass by Phantom. Honestly, the logistics of getting someone's gear to the stables are not that difficult.

It's your trip, and I assume you've paid good money for it. Don't let them bully you into an option you don't want.

I sure you'll have a great trip in any case!