Yup.... Lot's of people take infants backpacking and lots of "non outdoorsy-types" find people like us crazy. What is actually shocking to me in your description is it sounds like your husband is actually concerned with the dangerous part of this trip... namely you getting hurt. It seems like a PLB (or similar device as recommended by OM) would be a good compromise.

It doesn't matter who is right or wrong. The important thing is to talk it out and make sure both parents are on board with what is going on. Best of luck!