Man I so feel your pain. It’s normally the first thing I ditch in Bethesda games. “We should part ways.” That plus they tend to force storylines on you per follower, or force you bending to their likes/dislikes, which is a pain in the ass.
That being said, I think the mod followers are MUCH better in every respect. They are very configurable to stay the hell out of the way if that is what you need.