Sounds like a low level of iron -- possibly from old city water pipes.

1. Keep your pH low. (7.0 - 7.4)
2. Run your filter as much as possible.

If it's iron, you'll see it on your filter as a rust colored deposit.

You can swim, as long as your chlorine is OK. Just don't wear any white or very light colored suits that iron would stain.