Welcome to The Forum, sorry I've missed answering this til now!

Spongy areas of vermiculite are an indication of groundwater flowing through or under it Those areas should be removed and new material troweled in (there are some other precautions you may want to take).

In the shallow end, you can either patch each little spot or do the entire floor - it's probably easier to do the 'spot patching'.

If you want more info on this, just ask and I'll try to get back more quickly.