You should be brushing the pool at least daily. As far as the stuff on the bottom, vacuuming it, if you can vacuum to waste, will get rid of it much quicker than brushing it up into the water. If it's big enough, you might also try a fine-meshed net to scoop some of it off the bottom to get it out. If all else fails, then brush it up into the water. The filter will eventually get it, but it will take a good bit longer.

Janet