If it's dry, you can grind it.

Underwater, you can use a rubbing stone:
Goldblatt G06183 6-Inch Rub Brick
but, it will take a LOT of rubbing to do more than knock off the high spots or sharp edges.

And, do NOT leave that sucker in your pool. The screws that attach the abrasive to the handle will rust and leave some terrible stains on your new plaster!