It's entirely possible your kids have abraded things enough to produce improvements.

You might try a concrete rubbing block, like these from Amazon:
Concrete Rubbing Block
Concrete Rubbing Block

You can use them underwater, but do NOT leave them sitting around in the pool or on your deck: the screws that hold the handle are carbon steel and will rust. If you leave the wet block on your deck or in your pool, you WILL leave stains.

I should add that colored plaster, like yours appears to be, rarely looks as uniform as white plaster.

Ben