If it is just vermiculite, I'd just use a big screwdriver or something to chisel it out.
But something about your description doesn't make sense: vermiculite + cement used under liner pools is pretty soft and weak. The fact that you are talking about a concrete saw makes me wonder what you really have.
Bookmarks