Implementing Hand Tracking in AR Applications
Hand tracking — markerless tracking of hands and fingers via camera. Self-controlled AR interfaces, virtual musical instruments, surgical or mechanical training apps — anywhere hands become the controller, precise recognition of 21 joints per hand is required. Technically challenging: fast movements, finger occlusion, tracking loss in poor lighting. Our 5+ years of experience and 30+ delivered AR projects allow solving these problems with native and cross-platform solutions, reducing development costs by up to 40% by leveraging ready-made libraries. We guarantee stable performance and provide a certificate of compliance upon delivery.
Why is hand tracking technically complex?
Each hand has 21 joints (in MediaPipe) or 26 (in ARKit) that must be tracked in real time with low latency. Problems: self-occlusion (pinch when thumb overlaps index), tracking loss in low light (<200 lux), and high GPU load. Modern solutions combine machine learning and geometric algorithms for stable capture.
Platform landscape: iOS and Android
iOS: Since recent iOS versions, ARKit Hand Tracking provides a public API with 26 joints. For iOS hand tracking, this is the native choice. Android: ARCore lacks hand tracking, so MediaPipe Hands with 21 joints has become the standard, also working on iOS for cross-platform consistency. For Android hand tracking, MediaPipe is the standard. Both handle finger occlusion via a 2.5D approach: first detect a bounding box, then refine joints. ARKit leverages LiDAR depth on Pro devices for better accuracy; MediaPipe uses only RGB. Our expertise includes hand gesture classification using custom ML models for various applications. This article discusses hand tracking AR applications and AR development services.
Implementing hand tracking on iOS (Swift)
// Latest iOS, RealityKit let session = ARKitSession() let handTrackingProvider = HandTrackingProvider() Task { try await session.run([handTrackingProvider]) for await update in handTrackingProvider.anchorUpdates { let handAnchor = update.anchor guard handAnchor.isTracked else { continue } // Position of the index finger tip if let indexTip = handAnchor.skeleton.joint(named: .indexFingerTip) { let worldTransform = handAnchor.originFromAnchorTransform * indexTip.anchorFromJointTransform // Attach object to fingertip } } } When is MediaPipe hand tracking justified?
MediaPipe Hands is free, cross-platform, and suitable for older devices or a unified codebase. Example in Kotlin:
// Android val handLandmarker = HandLandmarker.createFromOptions(context, HandLandmarkerOptions.builder() .setBaseOptions(BaseOptions.builder().setModelAssetPath("hand_landmarker.task").build()) .setNumHands(2) .setMinHandDetectionConfidence(0.5f) .setMinTrackingConfidence(0.5f) .build() ) val result = handLandmarker.detect(mpImage) // result.landmarks() — List<List<NormalizedLandmark>> // 21 points per hand in normalized coordinates [0..1] 21 joints in MediaPipe: WRIST, THUMB_CMC .. THUMB_TIP, INDEX_FINGER_MCP .. INDEX_FINGER_TIP, similarly for other fingers. For AR attachment in 3D: normalized 2D coordinates → unproject via camera intrinsics + depth.
Comparison: ARKit vs MediaPipe
| Criterion | ARKit Hand Tracking | MediaPipe Hands |
|---|---|---|
| Platforms | iOS, visionOS | iOS, Android |
| Number of joints | 26 | 21 |
| Latency (iPhone 15) | 15-20 ms | 20-30 ms (iOS) |
| Latency (mid-range Android) | — | 35-45 ms |
| Free | Yes (part of platform) | Yes |
| Accuracy on occlusion | Higher | Lower |
ARKit significantly outperforms MediaPipe in latency (15-20 ms vs 35-45 ms), making it better for real-time applications.
Choosing Between ARKit and MediaPipe
If your target is exclusively iOS, use ARKit for best performance. For cross-platform, MediaPipe reduces code duplication despite higher latency.Gesture recognition and object interaction
Basic gestures without ML: pinch (thumb-index distance < threshold), open palm (tips above MCP), fist (tips below MCP), victory (index and middle up). Complex gestures (ASL alphabet) require CreateML or TensorFlow Lite custom models. For object interaction: raycasting from finger/palm for picking, pinch to grab, release to drop. Two hands enable scaling and rotation. Surgical simulation uses fingertip collisions.
Limitations and real-world considerations
Finger occlusion remains challenging; both frameworks use 2.5D. Low light (<200 lux) reduces confidence. Latency: MediaPipe on mid-range Android 35-45 ms, ARKit on iPhone 15-20 ms. For musical instruments, the difference is noticeable. Test under diverse conditions.
Implementation plan and timelines
- Requirements analysis and platform selection.
- Library integration and configuration.
- Gesture logic development.
- AR object interaction implementation.
- Testing on real devices.
- Performance optimization.
Basic hand tracking with pinch/open gesture recognition on iOS: 1 to 2 weeks. Cross-platform on MediaPipe: 2 to 3 weeks. Custom gesture classifier: additional 2-3 weeks. Interactive hand-object interaction: additional 2-4 weeks. Cost estimated individually, starting from $8,000 for a basic implementation. Typical project cost ranges from $8,000 to $20,000 depending on complexity. With 5+ years in AR and 30+ projects, we ensure quality and meet deadlines.
What's included in our implementation
- Technical documentation and API reference
- Access to source code and version control
- Developer training session (up to 4 hours)
- 30 days of post-launch support
- Performance optimization and testing report
Trust: We provide a guarantee of stable tracking and certification of compliance with industry standards. Our experience reduces risks and accelerates your time to market.