This has been discussed many times. On higher levels, monsters will ignore the AI more and more. It's WAI.

Yeah and it's not that I want to make the game easier but this really kills the game for me. Tired of attacking enemy AI and seeing epic monster stacks hanging out right next to their cities.

That's a separate issue, which Frogboy addressed here.  And I think it's fixed in the 1.3 patch (which is still in beta).  That post shows how complex debugging the AI can be.

From a project management standpoint, I think they really shot themselves in the foot by making the monster aggressiveness variable depending on the difficulty level.  In theory, I can see why they'd think it was a neat idea, but only under the assumption that the AI code will be flawless.  In practice, debugging AI code is a Sisyphean task, and it just made it difficult to get accurate player reports and reproduce bugs that were reported "in the wild".  At the very least they should have made "Monsters aggressiveness towards the AI" it's own option.

Basically, despite all the feedback and bug reports and writeups on the forums, we had to wait until one of Frogboy's playthroughs generated the appropriate edge case while he was looking for it.

I suspect that part of the reason that option got tied in with world difficulty is because the AI is really reckless (or stupid) about monsters.  In 1.2 at least it does not appear to pathfind to avoid monsters.  Also, Frogboy has stated previously that the game generates a nice safe starting position for the player, but not for the AI.  Without passive monsters, the AI would likely struggle to expand at all, which would result in a rather lackluster challenge.

If you dont want the pioneers running around your permanent residence just kill them.  Makes for a much more challenging game.  The AI will "care" then.