I agree with @CLWheeljack that you do seem to have to prioritize in a way that’s less than optimal. This is why many JRPGs will only allow you to use three dudes/dudettes but still give the XP to your whole squad whether they participate or not. You can keep everyone together on abilities that way.
I’m sure if they make another one they might find some way to address that. I mean, playing the challenges does get you extra dough to keep them up too though, I suppose? It’s just more gameplay to get there which isn’t a bad thing really.