The brake on expansion is that colonizers get progressively more expensive.
I don’t “cram them in” as you say, but I do try to fill up available space as resources allow. Keep in mind that murder is econ in AoW, since you get such huge boosts from clearing neutrals and dungeons and digs, and make sure you prioritize keeping your main stack (and later, multiple stacks) in the field.