Were the panniers on the rear of the bike? If so, that was part of the problem, you need some weight up front as well. When I used to tour by bike I used four panniers, two on the front and two on the back. Worked well.