Do you use skimmer socks?
When I have what you describe, it is pollen or some plant debris like it and the skimmer socks grab it right up.