As a parent with children, I make sure that when we aren't in our pool, I always remove the ladder and place it a good distance from the pool. We have many children right next to us and I feel most comfortable knowing that we are safe having our yard fenced in.
Perhaps if it's not in the budget to fence your entire yard, look into the cost of fencing the perimeter of your pool. Seems your safest bet.