You're right - you may find that a gender-specific pack will fit you better than a "unisex" pack designed for both genders. (Then again, you may not.)

Just remember: the gender of the pack doesn't always have to be the same as your gender.

There should be some women in the forums who find that a man's pack fits them better, and some men who find that a woman's pack fits them better. Pack builders will shoot for a "typical" man or woman, and build some features (angle of pack to hipbelt, shape and padding of hip belt and shoulder straps, etc.) to fit that typical build. However, if you happen to be built differently (say, a man with narrow shoulders, or a woman with narrow hips) you may find that the other gender's pack fits you better. And, since packs have moved somewhat away from stereotypical "ladies" and "mens" colors, there's no real stigma or "girly-man/manly-girl" issues to deal with.

Fit, not "gender" is the key.