AI-Powered Planogram Compliance: Detect Violations Instantly

An empty shelf in retail means a 15% loss in category sales. Manual planogram checks once a week are too late: violations are spotted 7 days later, when lost revenue is already irrecoverable. Non-compliance losses can reach up to $120,000 per month for a chain of 50 stores. We implement CV systems based on YOLOv8 and CLIP that detect violations within minutes after capturing images. The system processes photos from any IP camera with 1080p resolution and generates a report highlighting each discrepancy. AI detects violations 4 times faster than manual checks, with detection accuracy reaching 95% compared to 80% for humans.

A delay of just a few days turns into million-dollar losses, especially in high-turnover categories. Automating shelf compliance with AI is not just a replacement for manual labor—it's a new level of precision and response speed. Our clients, after implementation, report a 30% reduction in out-of-stock and savings of $150,000 annually for a chain of 20 stores. We guarantee detector mAP of at least 0.92 on your data. The typical project cost ranges from $30,000 to $50,000 per store pilot, with an annual license fee of $5,000 per store.

Problems Solved by AI Planogram Control

• Reaction delay: from violation to detection takes 7 days—sales are lost in the meantime. The AI system produces a report immediately after shooting.
• Human error: merchandisers miss up to 20% of discrepancies, especially with large assortments. The detector finds >95% of violations.
• Scaling complexity: manual checks across hundreds of stores require a large staff. The AI system processes 10,000+ photos per day without additional resources.

How YOLOv8 and CLIP Detect Violations

The main pipeline consists of three steps: detecting products on the shelf, identifying each product, and comparing with the reference planogram.

from ultralytics import YOLO
import numpy as np
from PIL import Image
import torch
import torch.nn.functional as F

class PlanogramComplianceChecker:
    """
    Step 1: YOLOv8 detects all products on the shelf (bbox + class)
    Step 2: CLIP/ViT identifies the specific SKU from the crop
    Step 3: Comparison with planogram
    """
    def __init__(
        self,
        detector_path: str,       # fine-tuned YOLO on shelves
        sku_embeddings_path: str, # CLIP embeddings of all SKUs
        planogram: dict           # {position: sku_id}
    ):
        self.detector = YOLO(detector_path)
        sku_data = np.load(sku_embeddings_path)
        self.sku_embeddings = torch.from_numpy(
            sku_data['embeddings']
        ).float()                 # (N_SKU, embedding_dim)
        self.sku_ids = sku_data['sku_ids'].tolist()
        self.planogram = planogram

        # CLIP for SKU identification
        from transformers import CLIPProcessor, CLIPModel
        self.clip_model = CLIPModel.from_pretrained(
            'openai/clip-vit-large-patch14'
        ).eval().cuda()
        self.clip_processor = CLIPProcessor.from_pretrained(
            'openai/clip-vit-large-patch14'
        )

    def analyze_shelf(
        self,
        shelf_image: Image.Image,
        confidence_threshold: float = 0.5
    ) -> dict:
        img_array = np.array(shelf_image)

        # Step 1: detection
        detections = self.detector.predict(
            img_array, conf=confidence_threshold, verbose=False
        )[0]

        shelf_products = []
        for box in detections.boxes:
            x1, y1, x2, y2 = map(int, box.xyxy[0])
            crop = shelf_image.crop((x1, y1, x2, y2))

            # Step 2: identify SKU via CLIP
            sku_id, similarity = self._identify_sku(crop)

            shelf_products.append({
                'bbox': [x1, y1, x2, y2],
                'sku_id': sku_id,
                'confidence': float(box.conf),
                'sku_similarity': float(similarity),
                'position': self._get_shelf_position(
                    [x1, y1, x2, y2], img_array.shape
                )
            })

        # Step 3: compare with planogram
        compliance = self._check_compliance(shelf_products)
        return compliance

    @torch.no_grad()
    def _identify_sku(
        self, crop: Image.Image
    ) -> tuple[str, float]:
        inputs = self.clip_processor(
            images=crop, return_tensors='pt'
        ).to('cuda')
        features = self.clip_model.get_image_features(**inputs)
        features = F.normalize(features, dim=-1).cpu()

        # Cosine similarity with all SKU embeddings
        similarities = (features @ self.sku_embeddings.T).squeeze()
        best_idx = similarities.argmax().item()
        return self.sku_ids[best_idx], float(similarities[best_idx])

    def _get_shelf_position(
        self, bbox: list, img_shape: tuple
    ) -> dict:
        """Horizontal position + shelf row"""
        h, w = img_shape[:2]
        cx = (bbox[0] + bbox[2]) / 2
        cy = (bbox[1] + bbox[3]) / 2
        return {
            'col': int(cx / w * 10),   # 0-9 — ten columns
            'row': int(cy / h * 5)     # 0-4 — five rows
        }

    def _check_compliance(self, shelf_products: list) -> dict:
        violations = []
        actual_positions = {
            f"{p['position']['row']}_{p['position']['col']}": p['sku_id']
            for p in shelf_products
        }

        for position_key, expected_sku in self.planogram.items():
            actual_sku = actual_positions.get(position_key)
            if actual_sku is None:
                violations.append({
                    'type': 'out_of_stock',
                    'position': position_key,
                    'expected_sku': expected_sku
                })
            elif actual_sku != expected_sku:
                violations.append({
                    'type': 'wrong_product',
                    'position': position_key,
                    'expected_sku': expected_sku,
                    'actual_sku': actual_sku
                })

        compliance_score = 1.0 - len(violations) / max(len(self.planogram), 1)
        return {
            'compliance_score': round(compliance_score, 3),
            'violations': violations,
            'total_positions': len(self.planogram),
            'violations_count': len(violations),
            'detected_products': len(shelf_products)
        }

Advantages of CLIP Over a Classifier

A classifier requires retraining each time a new product is added. CLIP works on embedding similarity: add a photo of the new SKU to the index—the system is ready. This reduces maintenance costs for assortments of 500+ items. According to ECR Retail research, automating shelf compliance reduces out-of-stock by 30%.

Role of MLOps in Retail

MLOps ensures continuous model updates when the assortment changes. We use Kubeflow for pipeline orchestration and MLflow for experiment tracking. This allows automatic retraining of the detector and rebuilding of the CLIP index without developer intervention. Without MLOps, each new SKU addition would require manual retraining—with an assortment of 2000+ items, this becomes a bottleneck.

How Quickly Does AI Compliance Pay Off?

A typical project pays for itself in 6–8 months through reduced out-of-stock and lower manual labor costs. For a chain of 50 stores, direct savings can be as high as $150,000 per year. An additional benefit is increased customer loyalty due to consistent product availability. The AI system identifies violations 4 times faster than manual checks.

Implementation Stages of the AI System

1. Audit of control scheme and data collection — analysis of current processes, shelf photography (at least 2000 images).
2. Fine-tune product detector YOLOv8 — image annotation, training product detector to mAP ≥0.92.
3. Building the CLIP SKU index — collecting product photos, generating embeddings.
4. Developing the planogram comparison pipeline — implementing shelf violation detection logic.
5. Integration with ERP — REST API for automatic report export.
6. Testing and team training — 2 days on-site plus webinar recordings.
7. Support — 3 months of maintenance, retraining when assortment changes.

SKU Indexing via CLIP Embeddings

from transformers import CLIPProcessor, CLIPModel
import torch
import numpy as np
from pathlib import Path

def build_sku_index(
    product_images_dir: str,   # directory: {sku_id}/{image1.jpg, ...}
    output_path: str,
    model_name: str = 'openai/clip-vit-large-patch14',
    images_per_sku: int = 5    # average embedding over several photos
) -> None:
    """
    Build CLIP index of all SKUs.
    Multiple photos per product → averaged embedding is more stable.
    """
    model = CLIPModel.from_pretrained(model_name).eval().cuda()
    processor = CLIPProcessor.from_pretrained(model_name)

    sku_embeddings = []
    sku_ids = []

    for sku_dir in sorted(Path(product_images_dir).iterdir()):
        if not sku_dir.is_dir():
            continue
        sku_id = sku_dir.name
        image_files = list(sku_dir.glob('*.{jpg,jpeg,png}'))[:images_per_sku]

        if not image_files:
            continue

        batch_embeddings = []
        for img_path in image_files:
            image = Image.open(img_path).convert('RGB')
            inputs = processor(images=image, return_tensors='pt').to('cuda')
            with torch.no_grad():
                emb = model.get_image_features(**inputs)
                emb = F.normalize(emb, dim=-1).cpu().numpy()
            batch_embeddings.append(emb)

        mean_emb = np.mean(batch_embeddings, axis=0)
        mean_emb = mean_emb / np.linalg.norm(mean_emb)
        sku_embeddings.append(mean_emb.squeeze())
        sku_ids.append(sku_id)

    np.savez(
        output_path,
        embeddings=np.array(sku_embeddings),
        sku_ids=np.array(sku_ids)
    )
    print(f'Indexed {len(sku_ids)} SKUs')

What's Included in the Work?

• Documentation: architecture description, API specification, operation manual.
• Fine-tuning YOLOv8 detector on your shelves (collection and annotation of ~2000 images).
• Building CLIP SKU index from the product catalog.
• Integration with ERP (1С, SAP, Oracle) via REST API—automatic report export.
• Team training: 2 days on-site plus webinar recordings.
• Technical support: 3 months after delivery—bug fixes, retraining when assortment changes.

Timelines

Comparison of Approaches: AI vs Manual Check

Implementing an AI system pays for itself in 6–8 months through reduced out-of-stock and lower manual labor costs. With over 5 years of experience and 50+ retail projects, our team delivers reliable AI solutions. This automated shelf alignment system provides a high ROI. For more details on the technology, refer to the computer vision article. Get a consultation on our AI shelf compliance system today. Our CV camera retail solution is tailored for retail environments.