We had a similar problem in our pool. If we hooked up the vac to the skimmer, the water level had to be pretty high or we would lose prime. I always thought it was weird since it should have been pulling water from the bottom of the pool so why would it matter what the water level was? We had to vacuum daily as there was always dirt in the pool.

Found out the fitting in the bottom of the skimmer was broken. When the pump was off, water would leak out. When the pump kicked on, it would suck that muddy water back into the pool. I discovered the leak when I winterized myself (paid to have it winterized the first year). When I was blowing out the lines, I put a gizmo in the skimmer and then air and muddy water blew in around the top of the skimmer. Had to cut the deck to get to the fitting. The broken piece had silicone around it so it was broken at install and not fixed properly. We bought the house with the pool already there and it was several years old. I guess the previous owners just dealt with it.