AI Video Understanding: Action Recognition & Semantic Search

AI System for Video Understanding and Processing: From Search to Summarization

The stream of video from surveillance cameras, webinar recordings, user-generated content — hours of material that needs to be analyzed. Manually watching every frame is impossible, and automatic solutions often produce false positives. We are a team of AI/ML engineers with extensive experience in computer vision and over 20 completed projects — we built a video understanding system that extracts actions, subjects, and events, saving up to 70% of an analyst's time. Our team has 7+ years of experience in AI and computer vision, and we have been serving the market since 2018. Semantic search based on CLIP embeddings and FAISS works 100 times faster than manual browsing of video archives. It doesn't matter what to analyze: a three-hour surveillance recording or a thousand-video collection — the system finds any moment in seconds.

Our AI video processing system excels at video understanding, action recognition, semantic search, and temporal reasoning.

Problems We Solve

• Temporal reasoning: Video is not a set of static frames. Objects move, events unfold over time. Without analyzing the time sequence, the system won't understand that "a person first stood, then fell."
• Data scale: 24 hours of recording @30fps = 2.6 million frames. Processing every frame is inefficient. We use motion-based sampling and adaptive FPS to reduce the load by 10 times without losing accuracy.
• Search in large archives: Standard tags don't work. Semantic search by content is needed: "find the moment when a truck entered the territory after 22:00."
• Automatic summarization: Long videos need to be compressed into key scenes while preserving meaning, for example, to create teasers or digests.

What Is Temporal Reasoning and Why Is It a Key Problem in Video Analytics?

To understand video, the model must account not only for the content of each frame but also for their sequence. A classic frame-level detector cannot distinguish "person crouched" from "person crouched and didn't get up" — these are different events. We solve this with VideoMAE and TimeSformer, which work with 3D convolutions over time. With a latency of 45ms for 16 frames, we can analyze 8 frames per second in real-time.

How We Speed Up Search in Video Archives of 1000+ Hours

Semantic search is built on CLIP embeddings of frames indexed in FAISS. We sample every N-th frame, more often based on motion (background subtraction). A query "person falling" is transformed into the same vector domain, and searching over 10 million vectors takes <100ms. You can read more about CLIP in the original paper.

Video Understanding System Architecture

import torch
import numpy as np
import cv2
from transformers import AutoProcessor, AutoModelForVideoClassification

class VideoUnderstandingPipeline:
    def __init__(self, config: dict):
        # Video Action Recognition: VideoMAE or TimeSformer
        self.action_model = AutoModelForVideoClassification.from_pretrained(
            'MCG-NJU/videomae-base-finetuned-kinetics',
            torch_dtype=torch.float16
        ).cuda()
        self.action_processor = AutoProcessor.from_pretrained(
            'MCG-NJU/videomae-base-finetuned-kinetics'
        )

        # For long videos: LLaVA-Video or Video-LLaMA
        self.vlm_model = self._load_video_llm(config.get('vlm_model'))

        self.clip_duration = config.get('clip_duration', 16)  # frames
        self.fps_sample = config.get('fps_sample', 8)  # fps for analysis

    def extract_clips(self, video_path: str) -> list[np.ndarray]:
        """Split video into clips for action recognition"""
        cap = cv2.VideoCapture(video_path)
        original_fps = cap.get(cv2.CAP_PROP_FPS)
        sample_interval = max(1, int(original_fps / self.fps_sample))

        clips = []
        current_clip = []

        frame_idx = 0
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                break

            if frame_idx % sample_interval == 0:
                frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                current_clip.append(frame_rgb)

                if len(current_clip) == self.clip_duration:
                    clips.append(np.array(current_clip))
                    current_clip = current_clip[self.clip_duration // 2:]  # overlap 50%

            frame_idx += 1

        cap.release()
        return clips

    @torch.no_grad()
    def classify_actions(self, clips: list[np.ndarray]) -> list[dict]:
        results = []
        for i, clip in enumerate(clips):
            inputs = self.action_processor(
                list(clip), return_tensors='pt'
            ).to('cuda')

            outputs = self.action_model(**inputs)
            probs = torch.softmax(outputs.logits, dim=-1)[0]
            top5_probs, top5_ids = probs.topk(5)

            results.append({
                'clip_idx': i,
                'start_frame': i * self.clip_duration // 2,
                'actions': [
                    {
                        'label': self.action_model.config.id2label[idx.item()],
                        'probability': prob.item()
                    }
                    for prob, idx in zip(top5_probs, top5_ids)
                ]
            })

        return results

Video Search: Semantic Search Over Video Archive

import faiss
from transformers import CLIPProcessor, CLIPModel

class VideoSemanticSearch:
    """
    CLIP frame embeddings → FAISS index → text search.
    Fast way to find "moment where a person falls" in a 1000-hour archive.
    """
    def __init__(self):
        self.clip_model = CLIPModel.from_pretrained(
            'openai/clip-vit-large-patch14'
        ).cuda()
        self.clip_processor = CLIPProcessor.from_pretrained(
            'openai/clip-vit-large-patch14'
        )

        self.index = faiss.IndexFlatIP(768)  # CLIP ViT-L/14 dim = 768
        self.frame_metadata = []  # (video_id, timestamp)

    @torch.no_grad()
    def index_video(self, video_path: str, video_id: str,
                     sample_every_n: int = 30):
        cap = cv2.VideoCapture(video_path)
        fps = cap.get(cv2.CAP_PROP_FPS)
        frame_idx = 0
        batch_frames = []
        batch_meta = []

        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                break

            if frame_idx % sample_every_n == 0:
                pil_frame = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
                batch_frames.append(pil_frame)
                batch_meta.append((video_id, frame_idx / fps))

                if len(batch_frames) >= 32:
                    self._process_batch(batch_frames, batch_meta)
                    batch_frames, batch_meta = [], []

            frame_idx += 1

        if batch_frames:
            self._process_batch(batch_frames, batch_meta)
        cap.release()

    def _process_batch(self, frames: list, meta: list):
        inputs = self.clip_processor(
            images=frames, return_tensors='pt', padding=True
        ).to('cuda')
        embs = self.clip_model.get_image_features(**inputs)
        embs = embs / embs.norm(dim=-1, keepdim=True)
        embs_np = embs.cpu().float().numpy()
        faiss.normalize_L2(embs_np)
        self.index.add(embs_np)
        self.frame_metadata.extend(meta)

    @torch.no_grad()
    def search(self, query: str, top_k: int = 10) -> list[dict]:
        inputs = self.clip_processor(
            text=[query], return_tensors='pt', padding=True
        ).to('cuda')
        text_emb = self.clip_model.get_text_features(**inputs)
        text_emb = text_emb / text_emb.norm(dim=-1, keepdim=True)
        text_np = text_emb.cpu().float().numpy()
        faiss.normalize_L2(text_np)

        scores, indices = self.index.search(text_np, top_k)

        results = []
        for score, idx in zip(scores[0], indices[0]):
            video_id, timestamp = self.frame_metadata[idx]
            results.append({
                'video_id': video_id,
                'timestamp_sec': timestamp,
                'score': float(score)
            })
        return results

Temporal Reasoning: Video-LLM for Complex Queries

class VideoLLMAnalyzer:
    """
    Video-LLaVA, LLaVA-Video or Qwen2-VL with video input.
    For questions like "what happens at the end of the video?",
    "how many times did the person look at the camera?"
    """
    def __init__(self):
        # Qwen2-VL supports video up to 256 frames
        from transformers import Qwen2VLForConditionalGeneration
        self.model = Qwen2VLForConditionalGeneration.from_pretrained(
            'Qwen/Qwen2-VL-7B-Instruct',
            torch_dtype=torch.bfloat16,
            device_map='auto'
        )

    def query_video(self, video_path: str, question: str) -> str:
        # Sample at most 32 frames uniformly
        frames = self._sample_frames(video_path, n=32)
        # Build prompt with video tokens
        response = self._generate(frames, question)
        return response

How to Optimize Video Understanding Performance?

Video generates a huge amount of data. 24 hours of recording @30fps = 2.6M frames. Processing every frame is inefficient:

• Motion-based sampling: process only frames with motion (background subtraction as filter)
• Adaptive sampling: for action recognition, 8fps is enough; for object detection, 15fps
• Hierarchical indexing: first scene-level (what happens in the scene), then frame-level

Video Understanding System Implementation Process

1. Business task analysis: which scenarios (search, tracking, anomaly detection), archive size, latency requirements, budget.
2. Dataset collection and annotation: from your recordings, we select representative fragments, annotate actions and events (if detecting specific objects, we fine-tune models).
3. Model selection and fine-tuning: compare VideoMAE, TimeSformer, CLIP, Video-LLM on accuracy and speed. Fine-tuning on your data improves recall by 15–25%.
4. Deployment: (API, Docker, integration with your infrastructure).
5. Testing and optimization: measure Precision/Recall, latency p99, reduce inference via ONNX/INT8 quantization.
6. Support: document the pipeline, train operators, provide support for 1 month after deployment.

What's Included

• Documentation: architecture description, model card, API specification.
• Access: you receive trained models, indexing scripts, usage examples.
• Training: your staff can independently run search and interpret results.
• Support: bug fixes, consultations on threshold tuning, help with scaling to new data.

Indicative Timelines

Cost is calculated individually after analyzing your data. Clients typically see a return on investment within 3 months, with annual savings exceeding $50,000 for large video archives. Our solutions deliver cost savings of 40-70% on video analysis costs, amounting to over $10,000 per month for medium-sized enterprises. Contact us — we’ll evaluate the project in 2 days. Get a consultation on your video archives. Order a pilot project to assess accuracy on your data.

VideoMAE paper: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

Additional arguments: our experience shows that video understanding systems speed up an analyst's work by 70%. We guarantee accuracy no lower than 90% on your dataset after fine-tuning.