They start out weak, but they end up with quite a few numbers depending on the structure.
By level 3 or so they’re on par with a lot of armies and with the help of guard towers it’s fairly common to win against chaff armies in my experience.
Towns and cities are a different matter. They basically have no garrison to speak of unless you build a specific military structure in them, but they do have walls and a ton of towers which can help quite a bit.