For most of my life I wore leather boots and forded streams barefoot whenever I couldn't cross them dry-shod. For the past five or six years I've switched over to mostly fabric boots or trail runners and I consistently wear them when fording.
The great advantage to wearing them is safety. I can place my feet with much greater confidence. No stone bruises on my feet or small cuts or gouges from unseen sharp rocks or sticks. Better traction, too. It makes river crossings that much easier and lets you concentrate on the current, not your feet.
I sometimes remove my sox and the insoles. Sometimes not. I almost always pour out any residual water, unless I know I must ford again in a half mile or less.
As others have noted, blisters are not a problem from wet footwear nearly so much as from heat and friction. Stream crossings tend to cool your feet and your boots, so this is not a problem on warm days.
The whole problem of cold, wet feet is another matter on cold, wet days. That's when I break out the Sealskinz, and keep my feet isolated from the wet footwear as much as I can.
Try it. You will probably come to like it.