Do not leave pucks in your skimmer with pump shut off. The pucks will still dissolve in the water and highly chlorinated water will be in the skimmer now. If some of that water leeches out around the mouth of the skimmer you risk bleaching your liner around the skimmer.

If you have an old, faded liner or don't care if it gets damaged you can leave them in. Personally, I would not do that.