I don't really want to partially drain the pool, so I'll try to get my chlorine levels up. The chart says I need between 8-15 ppm with 25 ppm to shock.
That'll work.

That's a lot of chlorine, I assume it's safe to swim with it that high?
Your assumption is correct.

Also will the CYA go down on it's own if I don't use any more di-chlor or will I be battling this every year unless I drain it?
It'll eventually go down....might take 'til next Spring.

I'll order the test kit today so that I can keep track of all this.
That would be the smartest thing you can do for you and your pool!