What are you doing as far as vacuuming, brushing and cleaning? Is there anything settling out onto the floor? Any sand on the pool bottom or coming out the backwash hose? (Might indicate filter problems)

Also, make sure you are backwashing often enough. Don't just rely on the gauge, as they sometimes stick. Make sure you have enough flow, and that your skimmer and pump baskets are clean.