What is going on with my pool?

Hi all. I am new to pools so please forgive my ignorance!
A few months ago I let the water level in my pool drop in order to diagnose a leak ( it is in the skimmer) as the water level dropped, the outer layer of plaster (about 1/8" thick) dried up and literally started to fall off! It exposed the grayish aggregate below which appears to be in good condition, although a bit of a rough texture. I assumed this was the gunite and that my pool needs to be replastered.
I had 2 different contractors come out to give me an estimate. The first guy saw my pool and said " I have never seen anything like this ( the peeling thin plaster layer) before." I asked if the newly exposed gray aggregate was gunite, he said no, it is plaster. Interesting... it looks to be in good shape. Can I just scrape the rest of this peeling white stuff off (which flakes off effortlessly) clean the pool and fill it back up (and save $7500)? He says "no, this gray stuff is shot too". Ok.... but why??? It is rough.
Next guy comes over and also says the gray is plaster, the white stuff is just an outer layer of plaster. I asked him the same question- Can I just peel the white stuff, etc. He said I could, but the gray is really rough and abrasive. I need a replaster.
Ok... So I bet these guys are correct.... but I do realize they want to sell the new plaster job. I don't know if someone in the past did a cheap replaster and that is what is peeling off or what. I have done some research and learned you can actually sand rough plaster to smooth it out.
Can anyone offer any insight? Is it feasible that I could just peel off this top layer of peeling plaster and sand the stuff underneath??? Or is that a totally bonkers idea ??? Thanks. Please see the pic.