Brushing exposes the algae to the chlorine; when you leave it on the walls it can form a protective slime layer. So, yes, brush it when you see it.