Julie,
I'm sorry but you're now beyond what I can help you with. You need a lawyer, a good one, I believe. Complain to the town that the inspector hasn't protected you--that's his job, not protecting the builder.
$60,000 for a fiberglass pool? It must be GIGANTIC.

Wish I could help more.