Am I crazy for taking a baby out into the wilderness alone?

Each of your positions has merit and can be reasonably defended. It's not a matter of who is right and who is wrong, or whether your plan is crazy or his alternative of having another adult is unreasonable. What's missing is your both taking the final steps to a compromise solution you can each agree upon. This impasse may be based on the feeling that this one trip sets an important precedent for many future trips, so giving up on any point now means giving up on it for a very long time. I can sympathize with that.

If you do have some anxiety over this arrangement being permanent, try to let it go, along with any resentment you might have over your SO appearing to thwart and obstruct your plans. He's doing his best and so are you.

You may need to get your alone time camping in the wilderness and your together time camping or hiking with your son on separate trips, either camping together within cell range, or else with another adult along. Reassure yourself that all these arrangements and compromises can be changed as circumstances change. The accommodations you make now don't have to be chiseled in stone.

My wife and I have had to make many similar compromises over the decades in regard to my need to solo backpack, our need to hike and camp together, while balancing these with the urgent care requirements of a medically fragile child and all the other responsibilities that come with a family. You just have to stay flexible and work together to meet everyone's needs. Not every good choice feels good.

Good luck!