Leaking only when pump is off.. 20 x 40 inground vinyl. New to me 13 year old pool

My pool water level will drop 2 inches overnight but only when pump is off. I haven't notice any drop when pump is running. I have a crack in the skimmer so when the level was down I dried it out and siliconed the crack but the level seemed to keep dropping. After repair still leaking. Main drain plumbing leak? Maybe doesn't leak when there is suction on it? How can I possibly find and fix this? I may just leave pump running all summer. How often is normal to run pump?