Panels are rated to withstand much higher pressures than that. The increase you see at the filter is going to be higher (don't know by how much) than the pressure in the panels due to the backflow pressure of all of the extra piping & fittings as well. At a minimum, you are forcing the water to go through 4 90 degree fittings, most likely much more.

My panel set causes about an 8 to 10 psi increase at high speed. If you have a diverter, you can come close to optimizing your system by fully closing the diverter while the panels are on, noting the pressure increase, then slowly opening the diverter until the point where the pressure just starts to drop (assuming that the increase is "reasonable" to begin with, say less than 12psi). At this point, you should be getting close to maximum flow through the panels.