Instead of brushing the stuff and just stirring it around, vac it to waste. There is no sense in leaving it in there, especially if the chlorine is still fighting it. I had small amounts of that brown residue when I did my conversion last year. The key is keeping the bleach hammered to it 3x a day to your proper shock level. If your CYA is 100, then you need to be adding more bleach than you are now, like mentioned above. Are you usine the bleach calc? If not, it is a must have, so download it. Don't give up, you will actually enjoy your pool this year instead of fight algae like I did all last year.
Good Luck!