I take my car key with me when I backpack. I consider my car key to be part of my critical emergency/survival gear. Like the rest of my critical emergency gear, I always know where it is and keep it with me always.

Generally, I place it inside my pack inside a small drawstring pouch filled with my valuables, such as my driver's license, debit card, and a 3x5 index card with some important phone numbers written on it. If I go more than a very short distance from my pack or my camp, it comes with me.