Professional AR Hand Gesture Recognition Setup for Unity and Unreal

Professional AR Hand Gesture Recognition Setup for Unity and Unreal

False positive gestures are the scourge of AR games. The hand shakes, pinch triggers accidentally, a fist is recognized as an open palm. In a Meta Quest project, we spent two weeks just filtering the noise from 26 joint positions from the Meta Hand Tracking SDK. Here is how we solved this problem without overcomplicating things.

What the Platform Provides and What You Need to Build Yourself

Meta Quest (via Meta Hand Tracking SDK or OpenXR Hand Interaction Extension) provides 26 joint positions of the fingers and wrist in world coordinates, updated at 30–60 Hz. These are raw data—bone positions. ARKit (iOS, via AR Foundation) with Vision framework gives a similar set of hand landmarks through ARHandTrackingConfiguration, available since iOS 18. On Android, ARCore does not have a direct Hand Tracking API—third-party solutions are used (MediaPipe via ML Kit) or Google ARCore Geospatial + a custom ML model. Out of the box, you get data, but not gestures. Gesture recognition is your task.

How to Build a Gesture Recognizer

Each gesture is a set of conditions on joint positions:

• Pinch: Euclidean distance between ThumbTip and IndexTip < threshold (usually 2–3 cm in world units). Plus an additional check: MiddleTip, RingTip, PinkyTip are far from the thumb (open). Without the second condition, a fist is falsely recognized as a pinch.
• Open Palm: all fingertip joints are at a significant distance from the Palm joint. We check Vector3.Distance(fingertip, palm) > openThreshold for all five fingers. Additionally, the palm normal (vector from Palm to Middle Metacarpal) should roughly face the camera, otherwise an open palm from behind will also trigger.
• Point: IndexTip extended (large distance from IndexMetacarpal), other fingers curled (small distance tip→metacarpal). Plus the angle between the IndexProximal → IndexTip vector and the palm vector.

Why Debounce and Confidence Filters Are Important

The hand naturally trembles—ThumbTip and IndexTip can randomly get close and move apart faster than 1 frame. Without debounce, a pinch fires 5 times per second when trying to perform it once. Standard technique: a state machine with a time threshold. A gesture is considered active only if the condition has been met continuously for at least N frames (or T seconds). Values: 3–5 frames for quick gestures, 10–15 frames (≈ 0.2 s at 60 Hz) for static poses like Open Palm. Additionally, a confidence filter. The Meta Hand Tracking SDK provides OVRHand.HandConfidence—when tracking confidence is low (hand partially out of view, poor lighting), gestures are not processed. This is critical for AR on smartphones where shooting conditions are unpredictable.

How to Integrate Gestures into AR Foundation?

In AR Foundation (Unity), Hand Tracking is connected via XRHandSubsystem (package com.unity.xr.hands). XRHandJoint for each joint provides TryGetPose()—position and rotation in space. The gesture recognizer subscribes to the XRHandSubsystem.handsUpdated event and processes data in the callback. It is important not to do heavy computation in this callback—it may be called off the main thread. Either buffer the data and process in Update, or use the Job System with IJobParallelFor for multi-hand recognition. For MediaPipe on Android—separate integration via Native Plugin or ready-made wrappers (mediapipe-unity-plugin), data arrives via callback with ML results.

We apply Kalman filtering for joint smoothing and quaternion angle checks for orientation-based gestures. This eliminates jitter and ensures robust detection.

Typical Gestures and Their Thresholds

What's Included in Gesture Recognition Setup?

Timeline and Budget

Timeline: from 3–5 working days for a basic prototype to 2–3 weeks for a full system with 10+ gestures, debounce, and confidence filters. The project budget is calculated individually, but typical costs range from $500 for a basic prototype to $5,000 for a comprehensive system with full testing and documentation. We guarantee reliable performance based on 7+ years of experience and provide a 30-day warranty on integration.

How to Proceed

1. Contact us with your requirements and target platform.
2. We analyze your project and propose a solution within 1-2 business days.
3. We deliver a prototype for testing and feedback.
4. After approval, we finalize integration and provide documentation.

Experience and Trust

With over 7 years of expertise and 15+ successful projects on Meta Quest and iOS, we have a proven track record in hand gesture recognition for AR. Our clients include gaming studios and enterprise AR developers. We provide a reliable, production-ready system backed by a warranty.

Learn more about Meta Hand Tracking SDK and OpenXR Hand Interaction. Contact us to get a consultation from an engineer.