The best quickenings are done by those at higher levels. Same goes for espers who are strongest when summoned by stronger characters. So, unbalancing them isn't such a good option in my opinion. And it doesn't hurt to have them equipped to a character so it wouldn't matter if you had espers on your strong party unless you used them when you had your reserve (the way you used your reserve party may be different than what I did so excuse the ignorance). I do see where you're coming from though.
Personally I only used espers for the fun of it. Matching them up with certain characters was just fun. Be it because of the plot or the design of them, it was cool matching them up for random reasons.