My answer? When it's back within the residual levels suggested in the BG table for that CYA level or below 10--whichever is higher.

So if my CYA is 40, I'll swim when the FC is 10 or less (resid is 3 to 6)

If my CYA is 150 I'll swim when the FC is up to 15 (Resid is 8 to 15).

You don't have to follow my advice--that's simply what I do.