Well I dug it up, and turns out there used to be an old tree stump a couple of feet down. It was the spongy culprit. I dug it out, put a few shovel fulls of dirt in at a time compacted, put a few more, compacted, and so on until it was full.