Simple suggestion: with wizard towers, your magic is affected by how many towers you have and their level, while temple swap cooldown times are both extremely high compared to tower ones (we're talking 21 hours for a full refill) and cannot be upgraded with sugar lumps. At this point in time, it's just absolutely not viable at all to drop bellow 2/3 swaps because the cooldown will make it not worth it at all. Add to this the fact that Spirits like to unswap themselves as they please, this makes this mechanic kinda frustrating to use.
Edit: to be clear, I'm not talking about a huge boost or something overpowered. Just an incentive to level up Temples beyond just "getting the achievement"