Only if water is physically removed before you add fresh water. Otherwise, with evaporation only water gets removed and not the water's contents (CYA, salt, etc.) that simply get more concentrated. So when you add more water it does get diluted, but back to where it started.

Imaging half of the water evaporating. In that case, a CYA of 50 ppm would become 100 ppm because there is the same amount of CYA but now in half as much water (so the concentration gets doubled). When you add fresh water, you dilute this back to 50 ppm.