Sometimes, brown gunk is just . . . brown gunk. Our pools provide a very large surface area on which dirt collects: airborne dust particles, pollens, whatever. Not all of it gets swept into the skimmer and trapped by the filter. Much of it sinks and, in due time, makes brown gunk. I dust my furniture every so often without really giving much thought to what that fine grey layer of stuff is or where it comes from. It's just dust: dander, smoke, oils, the detritus of life inside a home.

Since you haven't had an active algae outbreak and your numbers look pretty good and your water is otherwise clear you could just accept the brown gunk as part of having a pool that gets dirty. Not unsanitary. Just dirty. I have those same spots in mine and I would bet they're pretty common. I vacuum occasionally, I brush other times, I use a skimmer sock. The fine debris comes back, sometimes overnight, sometimes it takes a few days. I look at it as my pool's circadian rhythm