AI Visual Product Search by Photo

Imagine a customer photographs sneakers on the street and uploads the image to an app — the system finds similar items in the catalog within seconds. No text description, no category selection. This is visual search — a task we solve for e-commerce: from fashion marketplaces to spare parts catalogs. Below is the architecture used in production.

How CLIP Handles Search by Photo

CLIP (OpenAI) produces embeddings of dimension 768 (ViT-L/14) — a common vector space for images and text. This enables not only image-to-image search but also text-to-image: "red Nike sneakers" → similar products. Zero-shot accuracy on a 10K catalog is 74% Recall@10. For specific domains (fashion, furniture, electronics), we perform fine-tuning with learning rate 1e-6, freezing the text encoder. This raises accuracy to 86% (CLIP paper).

Why We Use Qdrant Instead of FAISS

In production scenarios, dynamic filters (price, brand, category) and p99 latency are critical. Qdrant supports combined filters via must conditions during search, whereas FAISS requires rebuilding the index when filters change. On a catalog of 1M products, Qdrant (HNSW) gives 25ms latency — 2–3× faster than alternatives at the same accuracy.

Architecture: Embedding + Vector Search

import torch
import torch.nn.functional as F
from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import numpy as np
import qdrant_client
from qdrant_client.models import Distance, VectorParams, PointStruct

class VisualSearchEngine:
    """
    CLIP-embedding + Qdrant vector DB for search by photo.
    CLIP supports text→image and image→image search out of the box.
    """
    def __init__(
        self,
        qdrant_url: str = 'http://localhost:6333',
        collection_name: str = 'products',
        embedding_dim: int = 768    # CLIP ViT-L/14
    ):
        self.clip_model = CLIPModel.from_pretrained(
            'openai/clip-vit-large-patch14'
        ).eval().cuda()
        self.clip_processor = CLIPProcessor.from_pretrained(
            'openai/clip-vit-large-patch14'
        )

        self.client = qdrant_client.QdrantClient(url=qdrant_url)
        self.collection_name = collection_name
        self._ensure_collection(embedding_dim)

    def _ensure_collection(self, dim: int) -> None:
        if not self.client.collection_exists(self.collection_name):
            self.client.create_collection(
                collection_name=self.collection_name,
                vectors_config=VectorParams(
                    size=dim,
                    distance=Distance.COSINE
                )
            )

    @torch.no_grad()
    def embed_image(self, image: Image.Image) -> np.ndarray:
        inputs = self.clip_processor(
            images=image, return_tensors='pt'
        ).to('cuda')
        emb = self.clip_model.get_image_features(**inputs)
        return F.normalize(emb, dim=-1).cpu().numpy().squeeze()

    @torch.no_grad()
    def embed_text(self, text: str) -> np.ndarray:
        inputs = self.clip_processor(
            text=[text], return_tensors='pt', padding=True
        ).to('cuda')
        emb = self.clip_model.get_text_features(**inputs)
        return F.normalize(emb, dim=-1).cpu().numpy().squeeze()

    def index_product(
        self,
        product_id: str,
        product_image: Image.Image,
        metadata: dict
    ) -> None:
        embedding = self.embed_image(product_image)
        self.client.upsert(
            collection_name=self.collection_name,
            points=[PointStruct(
                id=hash(product_id) % (2**63),
                vector=embedding.tolist(),
                payload={
                    'product_id': product_id,
                    'category': metadata.get('category'),
                    'price': metadata.get('price'),
                    'brand': metadata.get('brand'),
                    **metadata
                }
            )]
        )

    def search_by_image(
        self,
        query_image: Image.Image,
        top_k: int = 20,
        filters: dict = None,       # {'category': 'shoes', 'max_price': 5000}
        score_threshold: float = 0.65
    ) -> list[dict]:
        query_embedding = self.embed_image(query_image)

        # Build Qdrant filters
        qdrant_filter = None
        if filters:
            from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
            conditions = []
            for key, value in filters.items():
                if key == 'max_price':
                    conditions.append(
                        FieldCondition(key='price', range=Range(lte=value))
                    )
                elif key == 'min_price':
                    conditions.append(
                        FieldCondition(key='price', range=Range(gte=value))
                    )
                else:
                    conditions.append(
                        FieldCondition(key=key, match=MatchValue(value=value))
                    )
            qdrant_filter = Filter(must=conditions)

        results = self.client.search(
            collection_name=self.collection_name,
            query_vector=query_embedding.tolist(),
            limit=top_k,
            query_filter=qdrant_filter,
            score_threshold=score_threshold,
            with_payload=True
        )

        return [
            {
                'product_id': r.payload['product_id'],
                'score': round(r.score, 4),
                'metadata': {k: v for k, v in r.payload.items()
                             if k != 'product_id'}
            }
            for r in results
        ]

    def search_by_text(
        self, query_text: str, top_k: int = 20
    ) -> list[dict]:
        """Text-to-image search: 'red Nike sneakers' → results"""
        query_embedding = self.embed_text(query_text)
        results = self.client.search(
            collection_name=self.collection_name,
            query_vector=query_embedding.tolist(),
            limit=top_k,
            with_payload=True
        )
        return [{'product_id': r.payload['product_id'],
                 'score': r.score} for r in results]

Cropping the Object Before Search

from rembg import remove as rembg_remove

def prepare_search_query(
    user_image: Image.Image,
    remove_background: bool = True,
    crop_to_object: bool = True
) -> Image.Image:
    if remove_background:
        # rembg → RGBA
        rgba = rembg_remove(user_image)
        if crop_to_object:
            # Auto-crop to bounding box of opaque pixels
            bbox = rgba.getbbox()   # (left, upper, right, lower)
            if bbox:
                rgba = rgba.crop(bbox)
        # White background behind the object
        background = Image.new('RGB', rgba.size, (255, 255, 255))
        background.paste(rgba, mask=rgba.split()[3])
        return background
    return user_image

Fine-tuning CLIP on Fashion Domain

from transformers import CLIPModel, CLIPProcessor
import torch
from torch.optim import AdamW

def finetune_clip_for_domain(
    model: CLIPModel,
    train_loader,          # (image_tensor, text_tensor) pairs
    num_epochs: int = 10,
    learning_rate: float = 1e-6   # very small LR — CLIP is already well trained
) -> CLIPModel:
    """
    Fine-tuning only the visual encoder.
    Text encoder is frozen — we need it for text→image search.
    """
    for param in model.text_model.parameters():
        param.requires_grad = False

    optimizer = AdamW(
        filter(lambda p: p.requires_grad, model.parameters()),
        lr=learning_rate, weight_decay=0.01
    )

    for epoch in range(num_epochs):
        model.train()
        for batch_images, batch_texts in train_loader:
            outputs = model(
                input_ids=batch_texts['input_ids'].cuda(),
                attention_mask=batch_texts['attention_mask'].cuda(),
                pixel_values=batch_images.cuda()
            )
            # InfoNCE loss
            logits_per_image = outputs.logits_per_image
            labels = torch.arange(
                logits_per_image.shape[0], device='cuda'
            )
            loss = (F.cross_entropy(logits_per_image, labels) +
                    F.cross_entropy(logits_per_image.T, labels)) / 2

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    return model

Performance

What's Included

We deliver:

• Indexing pipeline code — in Python, using PyTorch and Qdrant.
• ML model — fine-tuned CLIP (if required) with model card and metrics.
• API service — REST/gRPC endpoints with filter and similarity threshold support.
• Documentation — deployment, configuration, and monitoring guides.
• Team training — 2–3 workshops on pipeline operation and retraining.
• Support — one month post-launch, including bug fixes and latency optimization.

Process: From Request to Production

1. Analytics — we study your catalog: image quality, class distribution, need for fine-tuning.
2. Prototype — in 2–3 weeks we build an MVP on 1,000 items and measure accuracy.
3. Development — implement the production pipeline with Qdrant or pgvector.
4. Testing — load testing (1,000 RPS), validation on nonstandard photos (blur, background).
5. Deployment — deploy in your cloud or on-premises.

Timeline

Budget is determined after an initial assessment of catalog size and customization needs. We evaluate your project in 2 days — contact us for a consultation. With over 5 years of experience in computer vision and more than 10 deployed systems, we guarantee results.