If you have no green left, it's time to add CYA. However, since you are working on your pool, do NOT put it in the skimmer. Instead, either use dichlor to chlorinate AND raise CYA for awhile OR hand about 4lbs in a sock in front of a return.

Regarding calcium leftovers, from sand algae . . . do NOT scrub too much; liners aren't made to take it.

Instead, the way to do it is to lower pH to 6.8 - 7.0 AND to aerate, which you can do by re-directing one of those jets to ruffle the water somewhat. You'll want to get your alkalinity as low as possible, which will make the pool water re-dissolve the scale. It will take REPEATED additions of acid to keep the pH low, since it will rise as the alkalinity diminishes.

By the way, it would be a good idea to verify that the scale is sand algae or calcium carbonate. If you can get a small piece or a few bits, and put them in a small glass container, and cover them with white vinegar. Cover that, air tight (Saran wrap) and leave it overnight. Calcium carbonate scale will dissolve; other things wont.