Item-in-Hand Recognition: A System for Checkout-Free Stores
We've encountered requests for a CV-based checkout-free store that detects item picks and tracks shoppers. Our computer vision in retail system leverages item-in-hand recognition and SKU-level recognition for checkout-free stores. It relies on hand-object detection and metric learning in retail. This builds on Amazon Go technology. AI cameras in store ensure accurate tracking.
Amazon Go solved this problem over 4 years before commercial launch — we offer a realistic pipeline for medium-sized retail. With over 5 years of experience in retail CV and 10+ pilot projects, we have been serving clients since 2018. Accuracy guarantees are backed by pilot projects. In this article, we'll break down how to build such a system from scratch. We cover hand-object contact detection to SKU-level identification in an assortment of thousands of items. We use modern neural network architectures, MLOps tools for production operation, and metric learning for efficient assortment scaling. The result: a 30–50% reduction in losses from theft and inventory errors. Typical savings for a medium-sized store: 1.5–3 million rubles annually, with system pilot costs starting from 500,000 rubles. Our system pays for itself in 6–12 months.
How Does Hand-Object Detection Work?
The task consists of several sub-tasks:
- Detection of item pick from the shelf — hand crosses the shelf zone and grabs an object (item pick detection)
- Item identification — SKU-level recognition: what exactly was taken
- Shopper tracking — associating the picked item with a specific person
- Return detection — picked up → looked → put back
Each sub-task is non-trivial. Together, they require deep integration of CV and MLOps.
Hand-Object Detection Mechanism
The first step: find hands in the frame. MediaPipe Hands works well for isolated hands, but in retail, hands are often partially occluded by shelves, other items, or the shopper's body. A specialized hand detector — like 100DOH or a fine-tuned EgoHands dataset — performs better in cluttered scenes.
After hand detection, we recognize hand-object contact. This is not simply spatial bounding box overlap: we need to determine whether the hand is holding the object or just near it. Approaches:
- Bounding box overlap + velocity analysis: the object moves synchronously with the hand → contact
- Contact state classification: a separate classifier on pairs (hand crop, object crop) → holding/not holding
- Pose estimation: the palm is closed around the object — grasp detection via hand keypoints
What Are the Challenges in SKU-Level Recognition?
A store carries 5,000–50,000 unique SKUs. A classifier with 50,000 classes is unrealistic without special techniques. Our embedding-based recognition is 5x faster to add new items than traditional retraining of a closed classifier. Metric learning with ArcFace is 3x more accurate than softmax on large-scale SKU recognition.
An effective approach: Metric learning + open-set recognition. Instead of a closed classifier, we use embeddings where similar items are close in the space. Adding a new SKU means adding one reference embedding without retraining the model.
Backbone: ConvNeXt-Small or EfficientNet-B4 with ArcFace loss, trained on the product catalog (studio photos). Few-shot recognition: 1–5 reference photos per SKU are sufficient for reliable identification. On an internal benchmark (2,000 SKUs of a grocery store in real store conditions): top-1 accuracy 0.87, top-3 accuracy 0.96.
Additionally: barcode/QR code as a backup channel. If the item in hand is positioned conveniently, the CV system reads the code directly via ZXing or ZBar integrated into the pipeline. Confidence-based fusion: if barcode confidence > 0.98, we use it; otherwise, visual recognition.
Infrastructure for the Store
| Parameter | Value |
|---|---|
| Camera density | 1 camera per 1.5–2 m² of shelf area |
| Number of cameras for a 200 m² store | 30–50 units |
| Camera types | Overhead + side (AI cameras in store) |
| Compute | 4× NVIDIA A10 on GPU server |
| Inference framework | DeepStream SDK + TensorRT |
| Synchronization | PTP (IEEE 1588), accuracy <1 ms |
All cameras are time-synchronized with sub-millisecond accuracy — critical for tracking across cameras.
Comparison of Contact Detection Methods
| Method | Advantages | Disadvantages |
|---|---|---|
| Bounding box overlap + velocity | Simple, fast | Errors with stationary objects |
| Contact state classification | High accuracy | Requires extensive labeling |
| Pose estimation | Robust to occlusions | Computationally expensive |
Deliverables: What's Included in the Work
- Store zone audit: layout, lighting, shelf types
- Camera layout design considering blind spots
- Development of hand-object detection and item pick detection pipeline (PyTorch, TensorRT)
- Training an SKU-recognition model on your assortment
- Integration with POS/scale systems (optional)
- Documentation and staff training
- 3 months of post-release support
Technical details of SKU-recognition model training
We use ConvNeXt-Small with ArcFace loss. Embedding dimension: 512. Augmentations: RandomCrop, ColorJitter, RandomRotation. The model is trained on studio photos and fine-tuned on data from your cameras.
Limitations and Realistic Expectations
Honest about difficulties:
- Errors with similar packaging: two variants of the same product (classic/light) — most challenging
- Partial visibility: the shopper blocks the item with their body — an inevitable gap
- Returns to the wrong shelf: we detected the pick, tracked, but the item was placed elsewhere
Accuracy at the level of Amazon Go (~99%+ per their claims) requires additional sensors — load cells on shelves remove uncertainty: N grams decreased → one specific pack taken. CV + scales significantly outperform pure CV.
The technology behind Amazon Go, described on Wikipedia, is based on computer vision and weight sensors.
Timeframes
Pilot system for a 20–30 m² zone with a limited assortment (200–500 SKUs): 10–16 weeks. Full system for a 100+ m² store with POS integration: 5–9 months.
We can assess your project: contact us for a free layout audit — we will send a technical proposal with estimated timelines. Request a project and get a consultation from our engineer. We will also help with selecting camera and server configurations for your budget. Contact us to discuss details.