Whether to remove the stays is best answered by removing them and then carrying the load you expect to carry for a few miles around the neighborhood. You'll feel for yourself how much or how little difference they make in your general comfort and can decide if they're worth it.

Using a contractor bag inside your pack is an excellent idea. If you put everything in it that needs to stay dry, you can probably do without a rain cover on your bag - although it is possible that without a rain cover your bag might absorb enough water weight from rain that it exceeds the weight of the cover. That's not easily discoverable without further experimentation.