You are correct...in a 20K gallon pool, one gallon of 6% bleach will raise your chlorine by 3 ppm. With a CYA of 50, you need to maintain a chlorine level at all times between 3 and 6 ppm. You don't have to wait long at all after a bleach addition to swim--just make sure that the bleach is dispersed by stirring it around or pouring it slowly into a return stream, you just don't want bleach "pockets". In fact, last night I added some bleach to the deep end while the kids were in the shallow end, but by adding it into the return it mixed quickly. I did not let them swim with open eyes underwater, but otherwise it didn't bother them. It is not what I would recommend for you or any others, but at the time the situation called for it.
Having said that, I would think that 15-30 minutes would be plenty of time to wait.