When it's that warm, I also leave off the rain gear and just get wet. I wear a lightweight synthetic base layer top and lightweight supplex hiking pants. While I'm actively moving, my body heat will dry them in about 15-20 minutes. If it keeps pouring, I change into my base layer at bedtime and put the wet shirt and pants into a large plastic ziplock bag inside my sleeping bag. No, they won't dry, but at least they won't be c-o-o-l-d when I put them on in the morning!

It is extremely important, however, to cover up when you stop moving, to prevent evaporative cooling when your internal thermostat slows down. If it's in the 60's and just drizzling, a lightweight wind shirt will do the job. If it's raining hard, I'll use my rain jacket. In the case of something really breathable like Frogg Toggs or DriDucks, you'll find that while you're wearing it over your wet clothing around camp, your body will push the moisture from your wet clothes through the rain gear so that your clothing will dry.

If it gets a little cooler, rain gear is essential. It's especially essential when you need to wear an insulation layer, since you need to keep that insulation dry at all costs. It only takes a few degrees' temperature drop while you're taking a rest stop or sitting around camp for that insulation to be needed. For that reason I would never go out without the rain gear.

It takes considerable experience with thermoregulation (adjusting your clothing to your body temperature) to recognize that very narrow gap between sweating and the beginnings of hypothermia.