Unfortunately, you will have to redo, or start over on your hip belt with my suggestion.
I have found the best way is to use spray adhesive and stick the foam to the fabric of the hip belt. 3M Super 77 is the best, but not the only option.
Here is a video I did making a backpack. I talk about the hip belt starting around minute 3:48. video