If all you want is to mark a spot and go back to it, there are devices that do just that. I've seen them at REI.

If you eventually want to actually learn navigation, start with a map and compass, learn those skills, and think about how fancy you want to get with the GPS and data collection. If you're apt with a map and compass, which you should always have anyway, a basic unit without mapset works great for navigational aid. If you want to upload and download from a computer, have color topo maps, mark waypoints and routes, see elevation profiles, know your average speed, etc. then a more fully featured model is called for.

I prefer a good feature set, good satellite connection, and find that a Garmin 60csx with frequent calibration works pretty well.