Automate UI without API: CV-Powered RPA Bots Turnkey

Imagine: a legacy COBOL system, no API, and you need to automate data entry. A classic RPA (UiPath, Automation Anywhere) relying on XPath will break at the first button shift. The alternative — an RPA bot with Computer Vision that 'sees' the screen as an image and finds elements by visual features. We develop such bots turnkey, leveraging YOLOv8, PaddleOCR, and few-shot learning methods, delivering intelligent automation. Over 5+ years, we've completed 20+ CV-RPA projects, including complex cases with Citrix and dynamic UIs. We guarantee recognition stability: detection accuracy of 95%+ with a proper dataset.

For element detection we use YOLOv8, fine-tuned on the Rico dataset (66k UI) plus custom annotation for the client's interface. Augmentation (brightness, contrast, rotation) and INT8 quantization reduce latency to 50 ms. Text is extracted with PaddleOCR (Cyrillic) or EasyOCR fine-tuned on a domain dictionary. As a result, the bot processes up to 10 operations per minute.

According to Ultralytics documentation, YOLOv8 achieves mAP 50-95 on UI datasets, confirming its applicability for interface automation.

When Is CV-RPA Needed?

Computer vision for RPA is justified in specific scenarios:

• Working with legacy systems without API and with a closed window hierarchy (COBOL/AS400 terminals, Citrix Virtual Desktop).
• Web applications with dynamically generated class names (React/Angular with CSS Modules), where XPath is unstable with every deployment.
• Processing PDF documents and scanned images within an RPA flow — without OCR, the integration is incomplete.
• Automating third-party desktop applications without SDK — SAP GUI or 1C in terminal mode.

CV-RPA Bot Architecture

import cv2
import numpy as np
from ultralytics import YOLO

class CVRPAAgent:
    def __init__(self, ui_detector_model: str):
        self.detector = YOLO(ui_detector_model)
        self.screenshot_engine = ScreenshotEngine()

    def find_element(self, element_type: str,
                     text_hint: str = None) -> tuple[int, int]:
        screenshot = self.screenshot_engine.capture()
        detections = self.detector.predict(screenshot, conf=0.7)

        candidates = [d for d in detections if d.class_name == element_type]
        if text_hint:
            candidates = self._filter_by_ocr_text(candidates, screenshot, text_hint)

        if not candidates:
            raise ElementNotFoundError(f"Cannot find {element_type}")

        best = max(candidates, key=lambda d: d.confidence)
        return best.center_x, best.center_y

    def click(self, element_type: str, text_hint: str = None):
        x, y = self.find_element(element_type, text_hint)
        pyautogui.click(x, y)

For UI element detection, we use YOLOv8 (GitHub), fine-tuned on a dataset of UI components (buttons, input fields, checkboxes, dropdowns). The base model is the Rico Dataset (66k Android UI) plus custom annotation for the specific client interface. During training, we apply augmentation (brightness change, contrast, rotation) for robustness to different screenshot conditions.

OCR Integration for Data Reading

For extracting textual data from the screen: PaddleOCR (best speed-accuracy balance for Cyrillic) or EasyOCR. Integration into the flow: find element → extract text from ROI (Region of Interest) → pass to processing logic.

import paddleocr

ocr = paddleocr.PaddleOCR(use_angle_cls=True, lang='ru')

def extract_text_from_region(image, bbox):
    x1, y1, x2, y2 = bbox
    region = image[y1:y2, x1:x2]
    result = ocr.ocr(region, cls=True)
    return ' '.join([line[1][0] for line in result[0]])

How CV-RPA Handles Dynamic UI

Dynamic elements (pop-ups, changing button positions) are a typical challenge. We solve this with few-shot learning: collect 5–10 screenshot variants with different element states and fine-tune the model on them. This provides resilience to changes without full retraining. Additionally, we use grid search on the confidence threshold — tuning the conf threshold to minimize false positives.

Why CV-RPA Is More Reliable Than Classic RPA

CV-RPA is 3 times more resilient to UI changes compared to classic XPath-based RPA. Here's a comparison of key metrics:

How to Implement CV-RPA: Step-by-Step Process

1. Analysis and data collection — screenshots of the target UI, bounding box annotation for each element type.
2. Detector training — fine-tuning YOLOv8 on the collected dataset, validation on a test set.
3. Integration with RPA platform — embedding the CV agent into UiPath / Automation Anywhere via Python Activity.
4. Testing on real scenarios — running 100+ operations, measuring accuracy and time.
5. Deployment and monitoring — installation on the RPA farm, error logging, cyclic fine-tuning.

What's Included in the Work

• Fine-tuned YOLOv8 model for your UI
• OCR integration (PaddleOCR for Cyrillic)
• Architecture and fine-tuning documentation
• 3 months of support after delivery
• Source code of the agent and deployment configs

Timeline and Cost

Pricing is tailored to each project. Contact us for a free consultation — we'll discuss your task and choose the optimal solution. Order a pilot project: test CV-RPA on one process, evaluate accuracy and execution time.