That’s pretty much on the money @FinnegansFather. People still find awesome (or scary) the bots in ancient games like F.E.A.R., but the technology that made that possible didn’t catch up. The kind of emergent behaviours that resulted were hard to predict for a designer, and as a result, in my opinion, the mainstream veered towards relying on behaviour trees which are easy to understand and debug, but result in behaviours like NPCs walking into walls or being nonplussed by somebody putting a cauldron on their heads.
Interestingly, games like the Sims were so successful because they ditched scripting altogether or for the most part (thinking here of the first one, didn’t play later ones).
So I don’t know what to make out of that gap - looks to me that people want their cake (feel like they are sitting around that table next to something that seems intelligent/interact with seemingly intelligent npc’s that go about their lives) and eat it too (have a decent chance of actually come on top/feel part of a story that makes “sense”). As Mark Twain (?) said: “Truth is Stranger than Fiction because Fiction Has to Make Sense”.