You may have had so many clothes on everything was compressed and not giving much insulation.
Personally, I use the much maligned Alpine 20 sleeping bag. You can usually find one on sale for around $30. I use it without a pad and stay warm down to the mid-20's. It weighs 2 lb 4 oz including the stuff sack. That puts it close to the same weight as a down bag. The tradeoff is it is bulky. I solve that by using an external frame pack.
I'd suggest taking some of those extra clothes and putting them under you instead of on you. The part of your body that needs extra insulation below is from the waist up.
I find I'm warmest sleeping on my side and not moving. This takes some practice as people new to sleeping on the ground get uncomfortable.
If you get cold during the night, get up and take a leak. Then you will be warm for a couple of hours.
I'm also not afraid to pull the bag over my head if it is near the end of the night and I'm getting cold. Yes, some moisture will accumulate, but an hour or so of breathing in the bag isn't going to hurt anything. Just let it air out sometime during the day.
The coldest part of the night is usually during the half hour after the sun starts to come up. So get up when it starts to get light and move on.
I sleep warmer if I eat about 600 calories of high fat food just before going to sleep. I also try to make it high protein. This pretty much means some kind of nuts.
Until you can afford a really good sleeping system, being a little cold sometimes comes with the territory.