Enemy AI in 3D Games: NavMesh, Behavior Trees & Sensors

Enemy Navigation in 3D: From Stuck to Natural Movement

NavMeshAgent stuck in a corner. Or walking in circles around an obstacle when it could bypass in a second. Or teleporting through a wall on scene change. Issues with 3D AI navigation are not "something went wrong" but specific architectural decisions that either account for NavMesh limitations or not. In VR the specifics are sharper: the player is physically in the game space, moving themselves, and enemies must react to that movement immediately. An enemy that "thinks" for 200 ms before turning is noticeable in VR like never before. Add stutter during group behavior, and the FPS budget cracks. Our experience shows that proper configuration of NavMesh and additional systems pays off already at the prototype stage. We guarantee that after our refinement, enemies will stop "dazing" and move naturally. Order an evaluation of your project — we'll select a solution for your tasks and FPS budget. FPS budget savings from LOD and batching can reach 20%.

Why Does NavMeshAgent Get Stuck at Seams?

The reason is that multiple NavMeshSurface are not automatically stitched together. At seams, agents lose their path. Solution: use a single surface with NavMeshLink for bridges, or explicitly add NavMeshLink components at transition points. If the scene has 5+ surfaces, manual stitching is mandatory.

The agent does not update the path when the player moves. SetDestination is called once when the player is detected, and the agent goes to the point where the player was. A periodic path update is needed — every 0.2–0.5 s call SetDestination with the target's current position. Too often — CPU overhead from constant path recalculation. Too rarely — the enemy "lags". Compromise: update only if the target moved more than destinationChangeThreshold meters (0.5 m is a good value).

The agent passes through other agents. NavMeshAgent.radius is the radius for obstacle avoidance, but it only works with NavMeshObstacle and other agents. If an agent has a lower avoidancePriority than another, it yields and may "push into" objects. Set priorities: regular enemies — 50, bosses — 20, player — 10 (lower number = higher priority). In practice, this reduces group sticking by 40%.

How to Implement Smooth Movement via Steering Behaviour?

NavMesh provides a global path from A to B. But movement along that path is a separate task. The standard steeringTarget can give angular movement: the agent makes a sharp 90° turn instead of smooth.

Solution: disable updateRotation, and rotate the agent yourself using Quaternion.RotateTowards with an angular speed matching the character's nature. Slow zombie — 60°/sec, fast fighter — 180°/sec. This immediately makes movement more convincing.

For more complex behavior — Steering Behaviours on top of NavMesh: Seek (towards target), Flee (away from target), Separation (disperse from peers). Separation is especially important for a group of enemies in VR — without it they all teleport to one point. Separation is implemented as an additional force added to the desired velocity: take all agents within separationRadius (2 m), sum vectors "away from each", normalize, and add as an offset to velocity. This approach is described in the work Steering Behaviours for Autonomous Characters (Craig Reynolds).

Behaviour Tree vs State Machine: What to Choose for VR?

State Machine (via Animator Controller or code) — simpler to implement, but scales poorly. With 5+ states, transitions become spaghetti.

Behaviour Tree (BT) — hierarchical task structure. Enemy: Selector → [Attack if distance < 2m → Chase if sees player → Patrol otherwise]. Each node is Sequence (all children must succeed), Selector (any one succeeds), or Leaf (specific action). For VR it's important: BTs should update at different frequencies based on distance to player. Enemy at 30 m — BT update once per second. Enemy at 3 m — every 100 ms. DistanceBasedUpdateRate reduces CPU load by 30% with 50+ agents.

Spatial Awareness: Hearing and Vision

Vision cone is implemented via Physics.OverlapSphere + angular check + Linecast for visibility check. Collect targets within sightRange (20 m), filter by Vector3.Angle < fieldOfView / 2 (e.g., 45° for human), check Linecast for obstacles.

For VR games, add check for sound stimuli: player shoots or runs → a SoundStimulus event with position and intensity is created → all agents within radius intensity * attenuationFactor (e.g., 10 m for a gunshot) receive notification. Use Unity Events or Physics.OverlapSphere from the sound point. This system keeps FPS stable even with 20+ agents.

How to Configure Optimal NavMesh: Step-by-Step Guide

1. Bake Settings: set Agent Radius minimal compatible with geometry (0.3 m), Step Height — to the height of the smallest obstacle (0.2 m).
2. Stitching surfaces: if the scene is split into several NavMeshSurface, add NavMeshLink at explicit transitions (doors, bridges).
3. Check sticking: use NavMeshAgent.pathStatus and NavMeshAgent.desiredVelocity — if status is PathPartial, recalculate path with a new starting point.
4. Profiling: in Profiler look at NavMeshUpdate and Pathfinding — they should not exceed 5% of CPU budget.

Typical mistakes when configuring NavMesh:

• Using NavMeshObstacle with carve: true — carve is expensive, better mark static obstacles on bake.
• Missing NavMeshAgent.areaMask — agents may walk on undesirable areas.
• Calling SetDestination too often (every frame) — causes freezes with 30+ agents.

What's Included in Enemy AI Development Work?

• Audit of current navigation system (NavMesh, grid, agents)
• Design and implementation of Behavior Tree or FSM with VR specifics
• Sensor configuration (vision, hearing) with real-time parameters
• Optimization: LOD-driven update, batching, draw call reduction
• Integration with gameplay (animations, attack, transitions)
• Documentation and team training (1–2 hour workshop)
• Technical support for 2 weeks after delivery

Estimated Timelines and Cost

Cost is calculated individually after analysis of enemy behavior requirements and number of simultaneously active agents. Get a consultation from our engineer — we'll send a preliminary estimate in 2 days. FPS budget optimization using LOD and batching can save up to 20% of project budget. We have been working on game development projects for over 5 years and have implemented AI for 15+ VR games.