Yes, you can use them as an adult. (I once had a summer bag from LL Bean that was a fleece blanket, with an optional zip-on nylon shell; I found the shell added a little warmth, so you could always pick up some wind-resistant nylon to sew on as a shell. You're right about diminishing returns, though; I was comfortable in that bag, wearing long johns and a fleece jacket, into the mid-50's if memory serves correctly. It was hoodless, so I also wore a stocking cap or balaclava - thick polyester, sold for about $25 at the outdoor store as a "balaclava," or for about $5 at Wallyworld as a "ski mask." (That was many years ago, when I was young, no money left over for gear, and by need adopted the attitude of "screw the weight.") My totally uninformed guess is that the second layer of fleece might work well, but any additional layers wouldn't be worth bothering with - and that a cheap polyester fleece jacket and stocking cap might work better than the second layer.

I'm not sure I'd do cold weather trips until I had about a year's experience - it may take that long to get the rest of your setting-up-camp and cooking routines down to where they're a habit, and reliably weatherproof.