Help with elevation and build

Planning a new pool build and still making the decision between a vinyl and fiberglass. I have done the research between the two so I am pretty educated on the pros and cons of each. I am leaning towards the fiberglass, but there is a element to my build that may make the already higher costs even greater.

My house is located at the top of a hill, and there is quite a bit of elevation change from one end of the property to the other. The former owner leveled the grade, but for whatever reason stopped short by 11.5 feet of my property line. I currently have a fence and 4 foot wall where the grade stops. Based on the town setbacks and layout of the yard, the deep end of the pool will be right where the wall currently sits. I would like to have the entire yard be at the same grade, and have the fence pushed back to the property line.

My question is this. In order to install a fiberglass pool they need to dig out a hole to set the pool in correct? In my case there would only be enough earth to dig for half the pool. Would I first need to fill in the area where the property was stopped short, only to then dig it back out for the fiberglass pool? Can a fiberglass be installed by digging out half, then filling around the other half? Would it be easier and cheaper to put in the walls for a vinyl pool and fill around them, or would I still have to fill, then dig out the fill? Seems like a waste of fill and money if I have to go the route of filling in the yard only to then dig a good portion of it out.

The survey company suggested I maybe lower the grade some, but he was thinking more from perspective of the town and permitting. Where I live if you build a wall 4 feet and about you need it engineered and a permit. Maybe lowering the grade would help with the pool install as well, but that seems pricey as they would have to remove from a lot of square footage. Not sure what is in my best interest for terms of cost.