WIth both a in-floor system and a spa, reducing the HP of the pump would reduce the performance of both. So unless you are willing to give those up entirely, I would keep the same size pump. Anyway most of your energy savings will come running on low speed so downsizing the pump isn't nearly as important as the low speed. Low speed will save about 50% in energy while each 1/2 HP drop in pump size will save about 10% more including turnover changes. Doing both will save more but again, you will sacrifice the spa and in-floor.

Also, the SWG should work fine on low speed of any size pump.