Alison, welcome to the forum!

Bad news first - if the floor is that bad, you might want to put a ~ 1/4 -1/2" 'skimcoat' on the entire floor (Please remember that this will raise the pool floor, so have the next liner cut to accommodate the shallower pool!)

For area patching, I mix up 1/2 full 5 gallon buckets of the vermiculite. The mix is (by weight) 1 part vermiculite to 2 parts cement (Portland, type I or II). Premix the dry materials and then add water - go slow with the water and thoroughly mix, then add a little more water, mix and see what the consistency is. What you're looking for (for patching) is a little standing water after the 1/2 bucket is well mixed - a wet batch will feather in a lot better than a dry one! (TIP- use a 'margin trowel' to do the mixing, it gets into the corners at the bottom of the bucket and doesn't take as much muscle to move around) (TIP #2 - 'parfait' the vermiculite and cement - put some verm in the bucket, then a layer of cement, a layer of verm and then another layer of cement - it speeds the dry mixing, considerably!)

If you have to do a large area or many smaller ones, you can mix 1 bag of vermiculite (16 - 20 lb) to 1/2 bag (92 - 94 lb) cement in a wheelbarrow - you'd need 10 - 14 gallons of water for a 'regular' mix (if you're doing large areas, keep the mix drier than for patching - what you want is a mix that, when you grab a handful and squeeze it - water trickles out of your closed fingers.

For applying, have a wooden 'float' to do the rough finish/ put it where it needs to go, then a metal 'pool trowel' (it's oval shaped, regular rectangle 'mason's trowels' tend to dig in and mar the vermiculite surface). The pool trowel passes come out better with wider sweeps and slight! pressure on the back edge - the float has more pressure on it, when you're laying the verm down to pack the stuff in, but the trowel is only to ~ smooth the surface. (well finished vermiculite looks ~ like a corkboard - IT SHOULD NOT be be fully sealed - unlike concrete, we're looking to have a semi-porous surface.)

If I went too fast or didn't correctly explain what I've tried to express here, or if there are any questions I failed to answer - just ask

1 other thing, sweep the floor at least 3 times after the vermiculite has cured, before installing the liner!!!!!!!! (as you sweep, you'll also notice places that need a little more patching - patch it and sweep again)

Good luck and well wishes to all the DIY vermiculiters!