I am in agreement with you. I do not call it lucky. I just think that no matter where you are there are a few kooks out there. But I do not get all worried about that. Look at the actual statistics. Do you stay inside your house because someone in town got murdered? You are more likely to die of hypothermia, a fall or other issues. I have never carried a gun or mace or anything. I have, although, thought of my ice axe as a weapon! I have never met anyone other than really nice folks in the wilderness.

As far as off-trail - I think it is almost safer. I do not worry about kooky people because I do not see anyone - except bears.
On the less used, less popular trails, you will meet a lot less people, if that is what you are worried about.

I think once you get a number of miles from the trailhead, away from drunks, you will be fine. Any place with easy access is more of a worry.

Life is a risk. Reduce the risk to zero and you have no life worth living.

If you do not feel comfortable about the risk, real or perceived, you need to find a backpack partner. I do not think you will enjoy your trip if you cannot come to the point where you feel safe. There is no "shame" in not going alone. Some people do not like to go solo for a host of other reasons too - too lonely, too many "bumps in the night", nobody to share the good moments with.