Could the "dirt" be metal or silt? This stuff is much finer than sand and will go right through the filter and back into the pool. I've had a few people give me really good feeback on this item when they've used ponds or other questionable sources to get their water.
http://www.cleanerpools.net/
Iron out may also be a good idea - get a sample tested to find out where you're at and go from there.
Bookmarks