AI Computer Vision Systems for Industrial Robots

Without vision, a robot is a blind automaton strictly following a program. CV adds adaptability: grasping arbitrarily oriented parts, in-process inspection, navigation in dynamic environments, and safe human collaboration per ISO 15066. We integrate computer vision systems into industrial manipulators and mobile robots — this combination delivers: 6DoF pose estimation for bin picking, depth-guided grasping, and semantic mapping for AMRs. Savings on a single bin picking station are substantial due to reduced manual labor and defects.

How CV Solves the Bin Picking Problem

Bin picking is one of the most demanded and challenging tasks. Parts in a container overlap each other, are chaotically oriented, and often have reflective surfaces. The primary method is 6DoF pose estimation: determining position (x,y,z) and rotation (roll,pitch,yaw) of each part. We use RGB-D cameras (RealSense, Azure Kinect) and one of the following models:

• FoundationPose — state-of-the-art for parts with a known CAD model. Achieves ADD-0.1d 78–89%.
• GDR-Net — geometrically discretized rendering, works without CAD but with lower accuracy.
• PVPN (Point Voting) — robust to heavy noise and partial occlusions.

Example implementation in PyTorch (abbreviated):

import numpy as np
import cv2
import torch
from dataclasses import dataclass
from typing import Optional

@dataclass
class ObjectPose:
    object_class: str
    position_xyz: tuple[float, float, float]    # mm in camera coordinate system
    rotation_matrix: np.ndarray                  # 3×3
    euler_angles: tuple[float, float, float]     # roll, pitch, yaw in degrees
    confidence: float
    grasp_point: tuple[float, float, float]      # recommended grasp point
    grasp_approach: np.ndarray                   # approach vector

class BinPickingSystem:
    """
    Bin picking system: detection and pose estimation of parts in a container.
    Methods:
    1. FoundationPose / DenseFusion: based on RGB-D
    2. GDR-Net: geometrically discretized rendering
    3. PVPN: point-wise voting
    Camera: Intel RealSense D435i or Azure Kinect.
    CAD model of the part is required for FoundationPose.
    """
    def __init__(self, pose_model_path: str,
                  cad_model_path: str,
                  object_classes: list[str],
                  camera_intrinsics: dict,
                  device: str = 'cuda'):
        self.device = device
        self.object_classes = object_classes
        self.camera_intrinsics = camera_intrinsics  # fx, fy, cx, cy

        # Load pose estimation model
        self.pose_model = torch.load(pose_model_path,
                                      map_location=device).eval()

        # CAD model for rendering (used by FoundationPose)
        self.cad_model = self._load_cad_model(cad_model_path)

        # YOLO for initial object detection
        from ultralytics import YOLO
        self.detector = YOLO(pose_model_path.replace('pose', 'det'))

    def _load_cad_model(self, cad_path: str):
        """Load .ply or .obj CAD model"""
        try:
            import open3d as o3d
            return o3d.io.read_triangle_mesh(cad_path)
        except ImportError:
            return None

    def estimate_poses(self, rgb: np.ndarray,
                        depth: np.ndarray) -> list[ObjectPose]:
        """
        Estimate object poses.
        rgb: (H, W, 3) uint8
        depth: (H, W) float32 in millimeters
        """
        # 1. Detect objects for ROI
        detections = self.detector(rgb, conf=0.4, verbose=False)

        poses = []
        for box in detections[0].boxes:
            cls_id = int(box.cls.item())
            if cls_id >= len(self.object_classes):
                continue

            x1, y1, x2, y2 = map(int, box.xyxy[0])

            # Crop RGB and depth patches
            rgb_crop = rgb[y1:y2, x1:x2]
            depth_crop = depth[y1:y2, x1:x2]

            if rgb_crop.size == 0:
                continue

            # 2. Pose estimation on the patch
            pose = self._estimate_single_pose(
                rgb_crop, depth_crop, cls_id, (x1, y1)
            )
            if pose:
                poses.append(pose)

        # Sort by Z height (top parts first)
        poses.sort(key=lambda p: p.position_xyz[2])
        return poses

    @torch.no_grad()
    def _estimate_single_pose(self, rgb_crop: np.ndarray,
                               depth_crop: np.ndarray,
                               cls_id: int,
                               offset: tuple) -> Optional[ObjectPose]:
        """Pose estimation for a single object"""
        from torchvision import transforms
        transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406],
                                  [0.229, 0.224, 0.225])
        ])
        from PIL import Image
        pil = Image.fromarray(cv2.cvtColor(rgb_crop, cv2.COLOR_BGR2RGB))
        rgb_tensor = transform(pil).unsqueeze(0).to(self.device)
        depth_tensor = torch.from_numpy(depth_crop).unsqueeze(0).unsqueeze(0).float().to(self.device)

        # Concatenate RGB + depth
        depth_norm = depth_tensor / 1000.0  # mm → meters
        # Simplified model takes 4-channel input
        input_tensor = torch.cat([
            rgb_tensor,
            torch.nn.functional.interpolate(
                depth_norm, size=rgb_tensor.shape[-2:], mode='bilinear'
            )
        ], dim=1)

        output = self.pose_model(input_tensor)
        # output: (1, 6) — translation(3) + rotation_euler(3)
        if output is None or output.shape[-1] < 6:
            return None

        out_np = output.squeeze().cpu().numpy()
        tx, ty, tz = out_np[:3] * 1000  # meters → mm
        rx, ry, rz = np.degrees(out_np[3:6])

        # Rotation matrix from Euler angles
        R, _ = cv2.Rodrigues(np.array([np.radians(rx),
                                        np.radians(ry),
                                        np.radians(rz)]))

        # Grasp point: object center + offset upward along normal
        grasp_z = tz - 30  # 30mm above center
        grasp_point = (tx, ty, grasp_z)
        approach_vec = R @ np.array([0, 0, -1])  # approach direction

        conf = float(torch.sigmoid(
            self.pose_model.confidence_head(output) if hasattr(
                self.pose_model, 'confidence_head') else torch.tensor(0.0)
        ).item()) if hasattr(self.pose_model, 'confidence_head') else 0.8

        return ObjectPose(
            object_class=self.object_classes[cls_id],
            position_xyz=(round(tx, 1), round(ty, 1), round(tz, 1)),
            rotation_matrix=R,
            euler_angles=(round(rx, 1), round(ry, 1), round(rz, 1)),
            confidence=round(conf, 3),
            grasp_point=grasp_point,
            grasp_approach=approach_vec
        )

Why 6DoF Pose Estimation Is Critical for Collaborative Robots

Collaborative robots work in the same space as humans. An error in determining the part pose leads to collision or damage. For cobot applications per ISO 15066, grasp repeatability of ±1 mm and latency under 50 ms are required. Only 6DoF pose estimation provides the accuracy needed for safe approach and grasp.

Method comparison metrics:

FoundationPose outperforms GDR-Net by 1.2× in ADD when CAD is available, but without CAD GDR-Net wins due to not requiring a model. PVPN is more robust to occlusions but slower (15 FPS vs 30 FPS for FoundationPose).

How We Design a Computer Vision System for Robots

The process starts with an audit of your production: what operations are performed, what parts, what current issues. Then we select hardware (cameras, lighting, controllers) and develop the CV algorithm. Steps:

1. Data collection: capturing scenes at your facility, labeling poses (6DoF) using our tools.
2. Model selection: FoundationPose, GDR-Net, or custom Transformer depending on CAD availability and acceptable latency.
3. Training and validation: on synthetic and real data. We aim for ADD-0.1d > 85%.
4. Integration: into a ROS2 node for the manipulator or OPC-UA for PLC. We ensure real-time performance.
5. Testing: on the production line for two weeks. We record KPIs (grasp cycle time, success rate).

Vision for AMR Navigation

Mobile robots (AMR/AGV) use CV for obstacle detection, people detection, and map building. Typical architecture: YOLOv8 on RGB-D, depth segmentation, sector division for trajectory planning. Example code snippet:

class AMRNavigationVision:
    """
    Computer vision for autonomous mobile robots (AMR).
    Tasks: obstacle detection, semantic mapping, human recognition
    for cobot safety (ISO/TS 15066 protected/restricted speed zones).
    """
    def __init__(self, obstacle_model_path: str,
                  device: str = 'cuda'):
        from ultralytics import YOLO
        self.obstacle_model = YOLO(obstacle_model_path)
        self.device = device
        # Semantic map: {cell_id: label}
        self.semantic_map: dict = {}

    def process_navigation_frame(self, rgb: np.ndarray,
                                   depth: np.ndarray) -> dict:
        """
        Analyze a frame for AMR navigation.
        Returns: obstacles, nearest_human_dist_m, clear_path_sectors.
        """
        results = self.obstacle_model(rgb, conf=0.4, verbose=False)
        obstacles = []
        nearest_human_dist = float('inf')

        h, w = depth.shape[:2]
        sector_width = w // 5  # 5 sectors: LL/L/C/R/RR

        for box in results[0].boxes:
            x1, y1, x2, y2 = map(int, box.xyxy[0])
            cls_name = results[0].names[int(box.cls.item())]
            cx = (x1 + x2) // 2
            cy = (y1 + y2) // 2

            # Median depth in bbox
            depth_crop = depth[y1:y2, x1:x2]
            valid_depths = depth_crop[depth_crop > 0]
            dist_m = float(np.median(valid_depths)) / 1000.0 if len(valid_depths) > 0 else 0

            obstacles.append({
                'class': cls_name,
                'bbox': [x1, y1, x2, y2],
                'distance_m': round(dist_m, 2),
                'sector': min(cx // sector_width, 4)
            })

            if cls_name == 'person' and dist_m < nearest_human_dist:
                nearest_human_dist = dist_m

        # Determine clear sectors
        blocked_sectors = {o['sector'] for o in obstacles if o['distance_m'] < 1.5}
        clear_sectors = [s for s in range(5) if s not in blocked_sectors]

        # ISO/TS 15066: if person < 0.5m → STOP; 0.5–1.5m → reduced speed
        safety_mode = ('STOP' if nearest_human_dist < 0.5
                       else 'REDUCED_SPEED' if nearest_human_dist < 1.5
                       else 'NORMAL')

        return {
            'obstacles': obstacles,
            'nearest_human_m': round(nearest_human_dist, 2),
            'clear_sectors': clear_sectors,
            'safety_mode': safety_mode
        }

What Is Included in the CV Project Work

We provide the full cycle: requirements analysis, equipment selection, camera-robot calibration, model training on your data, integration with the controller (ROS2/OPC-UA), testing under production conditions. Deliverables:

• Pose estimation model (ONNX/TensorRT)
• Integration module for PLC
• Safety operation documentation
• Operator training
• 6-month warranty support

Timeline and Cost

Timeline — from 8 to 20 weeks depending on complexity. Cost is calculated individually after an audit of your production. We guarantee stage transparency and fix KPIs in the contract.

Our Experience and Guarantees

Years of experience in industrial CV, dozens of projects from bin picking to inspection. Certified engineers in PyTorch, ROS2, OpenCV. We use official libraries: OpenCV and ISO 15066. We guarantee model accuracy (ADD and recall are specified in the contract).

Get a consultation for your project — contact us to discuss the task and create a prototype. Request a preliminary audit of your production — it is free.