Hand Tracking & Gesture Setup in AR Games: From SDK to Filtering

Hand Tracking & Gesture Setup in AR Games: From SDK to Filtering

We often encounter a scenario: a developer tries to implement controller-free control, but ends up with jittery fingers, false gesture triggers, and crashes when tracking is lost. Let's break down how to set up stable hand tracking in AR projects—from choosing the SDK to advanced filters. This is based on real experience implementing on mobile devices and standalone headsets.

Hardware and Software Foundation

For mobile AR (iOS/Android), the primary path is AR Foundation with XRHandSubsystem (available since AR Foundation 5.x). On iOS, under the hood, ARKit Hand Tracking (A12+) is used; on Android, ARCore lacks built-in hand tracking, so you have to use MediaPipe Hands or the XR Hands package with a custom provider. For controller-less headsets: Meta Quest 2/3—via OVR Hand or OpenXR Hand Tracking Extension (XR_EXT_hand_tracking). The latter is preferable for cross-platform projects—one codebase works on Quest, Pico, HoloLens 2. On HoloLens 2, we use Microsoft Mixed Reality Toolkit (MRTK3).

Why Gestures Are Harder Than They Seem

Gesture recognition is a classification of temporal pose patterns taking into account transient states and tracking noise. The main problems:

• Joint jitter: even a still hand produces oscillations of ±2–5 mm. Solution: low-pass filter (Kalman or exponential moving average with alpha ≈ 0.3–0.5).
• Threshold hysteresis: a single entry/exit threshold causes flickering. We set an activation threshold (pinch_distance < 15 mm) and a deactivation threshold (pinch > 25 mm).
• Temporal consistency: a gesture must be held for at least 3–5 frames (at 30 FPS) before recognition.

For custom gestures (circle in the air, double pinch), we write a custom finite state machine or sequence recognizer with time windows.

How to Integrate Hand Tracking with Game Objects?

Two main interaction types: near interaction (physical touch) and far interaction (ray from palm or finger). For near interaction, the key is a sweep test instead of a simple collider on the fingertip; otherwise, with fast movement (speed up to 10 cm/frame), the object gets skipped. For far interaction, we use a ray from the palm center with occlusion: if the object is blocked by the real hand, we don't react.

Important: when hand tracking is lost (hand leaves the field of view), we implement graceful degradation—interrupt the interaction cleanly, don't leave the object hanging. Use a timeout and smooth fade-out of the visualization.

Platform Performance Comparison

OpenXR Hand Tracking is 2–3 times faster than OVR Hand on Quest while being cross-platform.

Setup Steps

1. Platform and device analysis—this determines 80% of the architecture.
2. Basic hand tracking—import joints, visualize skeleton for debugging, check under varying lighting.
3. Gesture implementation—start with pinch as the basic trigger, tune filters and thresholds.
4. Integration with interactable objects—configure colliders, sweep tests, far interaction rays.
5. Real user testing—no developer can anticipate all hand positions.

What's Included in the Work

• Selection and configuration of SDK (AR Foundation, OpenXR, MediaPipe) for target devices
• Implementation of a basic gesture set (pinch, grab, point, swipe)
• Custom gestures according to spec (up to 10 unique)
• Integration with game objects (near/far interaction)
• Optimization: reducing draw calls, batching, occlusion culling
• Documentation and team training
• 30-day post-release support

We bring 10+ years of AR/VR experience and 50+ projects with hand tracking. Certified Unity and Unreal Engine specialists. We guarantee tracking stability on all target devices. Get your project analyzed—we'll select the optimal stack and tune the filters.

Estimated Timelines

Cost is calculated individually after project analysis. Contact us for a consultation—we'll assess the task and propose an optimal turnkey solution. Get a hand tracking specialist consultation.