I lost a contact lens several years ago while swimming underwater with my eyes open. While they make a tight seal on your eye, I suspect all that water can wash them loose.

And while it is relatively easy to find a contact on the bathroom floor, it is impossible to find one in a pool!