After due thought. I think I have found the downside of this idea.

Which is that a house is an enclosed space with finite BTU's to take.

Once you have the house cooled down, your heat source is gone, or at best limited to the heat gain of the building.

Not as effecient as the limitless outdoor heat.

And it sounded so good on paper too !!
.