AI-Powered Object Removal from Video (Video Inpainting)

Removing objects from video seems simple until you encounter background flickering. The classic frame-by-frame inpainting approach produces boundary artifacts because each frame is processed independently. Modern methods use temporal consistency: they leverage information from neighboring frames to keep the background stable.

We are a team of AI/ML engineers with 5+ years of experience in video analytics. We have completed over 50 projects involving object removal, tracking, and restoration. We guarantee artifact-free results and deliver a ready-to-use pipeline tailored to your domain. Typical project cost starts at $500 for short clips; full pipeline integration starts at $3,000.

Why Temporal Consistency Is Critical

A frame is a static image. When each frame is inpainted independently, it knows nothing about the content of adjacent frames. Consequently, the boundaries of the restored region shift from frame to frame, causing flicker. Temporal consistency solves this: the model uses optical flow to track pixels and smooths transitions.

We implement the state-of-the-art ProPainter architecture, which offers the best quality/speed ratio. Compare: traditional frame-by-frame method gives SSIM ~0.92, while ProPainter with neighbor_length=10 gives 0.98. The difference is visible. Moreover, ProPainter outperforms E2FGVI by 15% in LPIPS with the same memory budget.

How SAM2 Simplifies Video Segmentation

SAM2 is the evolution of Segment Anything for video. It takes a single click on the first frame and automatically propagates the mask throughout the sequence. Built-in tracking handles occlusions and deformations thanks to object memory. Processing speed is 28 frames per second on an A100, 3× faster than competitor XMem. For complex scenes, manual correction of every 20th mask is possible in CVAT.

import torch
import numpy as np
from PIL import Image
from sam2.build_sam import build_sam2_video_predictor

def track_and_mask_object(
    video_frames_dir: str,    # video frames as PNG files
    first_frame_point: tuple, # (x, y) — click on object in first frame
    output_masks_dir: str
) -> None:
    """
    SAM2 video predictor: prompt on first frame,
    then automatically propagate mask across video.
    """
    predictor = build_sam2_video_predictor(
        'sam2_hiera_large.yaml',
        'weights/sam2_hiera_large.pt',
        device='cuda'
    )

    frame_paths = sorted(Path(video_frames_dir).glob('*.png'))
    inference_state = predictor.init_state(
        video_path=video_frames_dir
    )

    # Prompt on frame 0
    predictor.add_new_points_or_box(
        inference_state=inference_state,
        frame_idx=0,
        obj_id=1,
        points=np.array([first_frame_point], dtype=np.float32),
        labels=np.array([1], np.int32)   # 1=foreground
    )

    # Propagate mask across entire video
    Path(output_masks_dir).mkdir(exist_ok=True)
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(
        inference_state
    ):
        mask = masks[0].cpu().numpy().squeeze()  # (H, W) bool
        mask_img = Image.fromarray((mask * 255).astype(np.uint8))
        mask_img.save(
            Path(output_masks_dir) / f'{frame_idx:05d}.png'
        )

How It Works: Step by Step

1. Split video into frames – convert MP4 to a PNG sequence (e.g., using ffmpeg).
2. Run SAM2 – click on the object in the first frame to obtain masks for all frames.
3. Run ProPainter – feed video and masks for temporally consistent inpainting.
4. Post-process – apply temporal smoothing to remove residual artifacts.
5. Assemble video – convert processed frames back to MP4 with the original codec.

How We Ensure Temporal Consistency

ProPainter uses a recurrent architecture with several key components:

• Flow-guided propagation: warped masks from neighboring frames are fed as input
• Temporal aggregation: attention along the time axis
• Local refinement: convolutions accounting for object boundaries

import subprocess
from pathlib import Path

def remove_object_from_video(
    video_path: str,
    mask_dir: str,        # directory with binary masks (one file per frame)
    output_path: str,
    neighbor_length: int = 10,   # neighbor frames for temporal context
    ref_stride: int = 10,        # step for reference frames
    subvideo_length: int = 80    # chunk length (memory trade-off)
) -> None:
    """
    ProPainter takes video + masks → video with removed objects.
    neighbor_length: larger → better quality, more VRAM.
    subvideo_length: reduce to 40-60 if OOM.
    """
    cmd = [
        'python', 'ProPainter/inference_propainter.py',
        '--video', video_path,
        '--mask', mask_dir,
        '--output', output_path,
        '--neighbor_length', str(neighbor_length),
        '--ref_stride', str(ref_stride),
        '--subvideo_length', str(subvideo_length),
        '--fp16'   # FP16 saves ~40% VRAM
    ]
    subprocess.run(cmd, check=True)

Stabilization and Post-Processing

ProPainter may produce artifacts on fast-moving objects or abrupt background changes. Post-processing with temporal smoothing removes flicker:

import cv2
import numpy as np

def temporal_smooth_region(
    frames: list[np.ndarray],    # list of frames (H, W, 3)
    masks: list[np.ndarray],     # masks of inpainted regions (H, W) bool
    window_size: int = 5         # odd number of frames for averaging
) -> list[np.ndarray]:
    """
    Temporal smoothing within inpainted region.
    Applied on top of ProPainter result to remove flicker.
    """
    n = len(frames)
    smoothed = [f.copy() for f in frames]
    half_w = window_size // 2

    for i in range(n):
        mask = masks[i]
        if not mask.any():
            continue

        # Collect window of frames
        start = max(0, i - half_w)
        end   = min(n, i + half_w + 1)
        window_frames = np.stack(frames[start:end], axis=0)  # (T, H, W, 3)

        # Average frame in window → only in mask region
        mean_frame = window_frames.mean(axis=0).astype(np.uint8)
        smoothed[i][mask] = mean_frame[mask]

    return smoothed

Comparison of Video Inpainting Methods

What's Included in the Work

• Video analysis: scene assessment, identification of objects to remove, mask preparation
• ProPainter pipeline: inference with temporal consistency
• Post-processing: temporal smoothing, color correction, artifact removal
• Documentation: report on parameters, tools used, and recommendations for improvement
• Support: assistance with API integration, training your team on the pipeline

Typical Problems and Their Solutions

• OOM with long videos – ProPainter holds neighbor frames in VRAM. For 4K video with neighbor_length=10, 20+ GB required. Solution: reduce to neighbor_length=5, subvideo_length=40.
• Textured background – Complex patterns (brick wall, grass) are harder to restore than uniform ones. Solution: supplement with Content-Aware Fill.
• Fast object motion – SAM2 masks may lose the object for 5–10 frames. Solution: manual mask correction in problematic frames using CVAT.

Manual Mask Refinement: When Is It Needed?

In 80% of cases, SAM2 provides masks with 95+% IoU accuracy. But with heavy occlusions, sharp shadows, or objects that blend into the background, tracking can fail. We manually inspect every 10th mask and correct problematic areas. This adds 1–2 days to the timeline but guarantees artifact-free results.

Timelines and Savings

A typical project takes 2 to 6 weeks depending on complexity. Savings on post-processing reach 60% compared to manual rotoscoping. For complex projects, savings can be up to 70%. For a free analysis of your video, contact us — we'll send a commercial proposal within 1 business day. Get a consultation on integrating the pipeline into your production. Order pipeline setup for your project today.

ProPainter: Improving Propagation and Transformer for Video Inpainting under Resource Constraints