As a general rule if it has been more than a day (12-18hr) between uses I will dump the 1st liter of filtered water. These filters have nothing in them to discourage bacteria growth and it does not take long to grow a crop if you leave the filter damp.
During a normal trip I use my filter 2x's per day so filter growth is not a big issue.