GUI Automation with Computer Use AI Agent

We design and deploy artificial intelligence systems: from prototype to production-ready solutions. Our team combines expertise in machine learning, data engineering and MLOps to make AI work not in the lab, but in real business.
Showing 1 of 1All 1566 services
GUI Automation with Computer Use AI Agent
Medium
~2-4 weeks
Frequently Asked Questions

AI Development Areas

AI Solution Development Stages

Latest works

  • image_website-b2b-advance_0.webp
    B2B ADVANCE company website development
    1318
  • image_web-applications_feedme_466_0.webp
    Development of a web application for FEEDME
    1226
  • image_websites_belfingroup_462_0.webp
    Website development for BELFINGROUP
    926
  • image_ecommerce_furnoro_435_0.webp
    Development of an online store for the company FURNORO
    1158
  • image_logo-advance_0.webp
    B2B Advance company logo design
    620
  • image_crm_enviok_479_0.webp
    Development of a web application for Enviok
    894

GUI Automation with Computer Use AI Agent

Have you ever faced a situation where a legacy ERP on Windows has no API, and automation via RPA breaks after every update? Computer Use solves this: the agent sees the screen via screenshots and acts like a human. We develop such agents "turnkey" — from assessment to production with monitoring. Our team has 10+ years of experience in AI/ML and over 40 completed projects. We'll assess your task in 1 day — contact us.

Why Computer Use is More Reliable Than RPA

Classic RPA requires precise description of element paths (XPath, coordinates, CSS selectors) and a stable screen structure. Computer Use uses computer vision: the agent analyzes the screenshot and determines which actions will lead to the goal. This provides resilience to interface changes — if a button moves, the agent will still find it. Vision-based automation adapts to UI changes 5x faster than RPA.

Characteristic RPA Computer Use
Adaptation to UI changes Low (scripts break) High (model sees changes)
Setup for a new task Days-weeks Hours-days
API requirement Not required Not required
Handling non-standard elements Difficult (popup, drag-and-drop) Possible with refinement

How Computer Use Works

The basic principle: agent receives screenshot → analyzes interface → selects action → executes → receives new screenshot → repeats until task completion.

Example Python agent implementation (Claude + PyAutoGUI)
import anthropic
import base64
import subprocess
from PIL import ImageGrab
import pyautogui
import time
import json
from io import BytesIO

client = anthropic.Anthropic()

def capture_screenshot() -> str:
    """Takes a screenshot and returns base64"""
    screenshot = ImageGrab.grab()
    # Resize to save tokens
    screenshot = screenshot.resize(
        (screenshot.width // 2, screenshot.height // 2)
    )
    buffer = BytesIO()
    screenshot.save(buffer, format="PNG", optimize=True)
    return base64.standard_b64encode(buffer.getvalue()).decode("utf-8")

def execute_computer_action(action: dict) -> str:
    """Executes action on the computer"""
    action_type = action.get("type")

    if action_type == "screenshot":
        return "screenshot_taken"

    elif action_type == "left_click":
        x, y = action["coordinate"]
        pyautogui.click(x * 2, y * 2)
        return f"clicked at ({x}, {y})"

    elif action_type == "double_click":
        x, y = action["coordinate"]
        pyautogui.doubleClick(x * 2, y * 2)
        return f"double-clicked at ({x}, {y})"

    elif action_type == "right_click":
        x, y = action["coordinate"]
        pyautogui.rightClick(x * 2, y * 2)
        return f"right-clicked at ({x}, {y})"

    elif action_type == "type":
        pyautogui.write(action["text"], interval=0.05)
        return f"typed: {action['text'][:50]}..."

    elif action_type == "key":
        pyautogui.hotkey(*action["key"].split("+"))
        return f"pressed: {action['key']}"

    elif action_type == "scroll":
        x, y = action["coordinate"]
        direction = action.get("direction", "down")
        amount = action.get("amount", 3)
        if direction == "down":
            pyautogui.scroll(-amount, x=x * 2, y=y * 2)
        else:
            pyautogui.scroll(amount, x=x * 2, y=y * 2)
        return f"scrolled {direction}"

    elif action_type == "mouse_move":
        x, y = action["coordinate"]
        pyautogui.moveTo(x * 2, y * 2)
        return f"moved to ({x}, {y})"

    return f"unknown action: {action_type}"

class ComputerUseAgent:
    """Agent for GUI automation via Computer Use"""

    COMPUTER_TOOLS = [{
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 960,
        "display_height_px": 540,
        "display_number": 1,
    }]

    def __init__(self, max_iterations: int = 50):
        self.max_iterations = max_iterations

    def run_task(self, task: str, context: str = "") -> dict:
        """Executes a task on the computer"""
        system = f"""You control the computer to perform the task.
Use the computer tool to interact with the interface.
Take a screenshot before each action to verify the current state.
Report when the task is complete or if you encounter an insurmountable obstacle.
{"Context: " + context if context else ""}"""

        messages = [{"role": "user", "content": task}]
        iterations = 0
        actions_log = []

        while iterations < self.max_iterations:
            screenshot_b64 = capture_screenshot()

            if messages[-1]["role"] == "user" and isinstance(messages[-1]["content"], str):
                messages[-1]["content"] = [
                    {"type": "text", "text": messages[-1]["content"]},
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": screenshot_b64,
                        },
                    }
                ]

            response = client.beta.messages.create(
                model="claude-opus-4-5",
                max_tokens=4096,
                system=system,
                tools=self.COMPUTER_TOOLS,
                messages=messages,
                betas=["computer-use-2024-10-22"],
            )

            tool_results = []
            task_completed = False

            for block in response.content:
                if hasattr(block, "text"):
                    if any(kw in block.text.lower() for kw in ["task completed", "done", "готово", "completed", "done"]):
                        task_completed = True

                elif block.type == "tool_use" and block.name == "computer":
                    action = block.input
                    result = execute_computer_action(action)
                    actions_log.append({"action": action, "result": result})

                    time.sleep(0.5)

                    new_screenshot = capture_screenshot()
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": [{
                            "type": "image",
                            "source": {
                                "type": "base64",
                                "media_type": "image/png",
                                "data": new_screenshot,
                            }
                        }],
                    })

            if task_completed or response.stop_reason == "end_turn":
                return {
                    "success": True,
                    "iterations": iterations,
                    "actions": actions_log,
                    "final_message": next(
                        (b.text for b in response.content if hasattr(b, "text")), ""
                    ),
                }

            messages.append({"role": "assistant", "content": response.content})
            if tool_results:
                messages.append({"role": "user", "content": tool_results})
            else:
                break

            iterations += 1

        return {"success": False, "iterations": iterations, "actions": actions_log}

How to Implement Screen-Aware Automation in 1 Week

  1. Task Analysis — describe the scenario: which windows, which fields, expected result.
  2. Prototype — run a basic agent on your PC (3–5 days).
  3. Calibration — configure coordinates, retry logic, human-in-the-loop.
  4. Test — run 100+ sessions, collect metrics.
  5. Deploy — package into a service, add monitoring.

Practical Applications

Processing outdated desktop applications without API. An ERP system without API but with Windows GUI. The agent enters a form, fills fields, clicks "Post", gets the result — all via screenshots. Speed: 40–60 operations per hour vs 15–20 for an operator. Time savings reach 70%, reducing costs for maintaining legacy systems. Typical project cost starts at $5,000, with average annual savings of $30,000 in labor.

Data migration between systems. Export from old CRM → import to new one when there is no direct integration. The agent copies records via GUI of both systems.

Regression testing. The agent goes through user scenarios in a web application, checks results, logs anomalies.

Typical Problems and Their Solutions

  • Model hallucinations. The agent may "think" a click was successful even though the UI hasn't changed. Solution: retry logic with state verification after each action.
  • Sensitivity to screen scaling. If the screenshot is resized, coordinates may be inaccurate. We use automatic scale detection and calibration.
  • Latency. One step takes 1–3 seconds. For long scenarios, this can be slow. Optimization: batch processing and reducing the number of screenshots.

Limitations and Real Metrics

Honest numbers from our practice:

Task Success Rate Iterations
Fill a form with 10 fields 87% 8–15
Navigate through 3+ screens 71% 15–25
Work with popup/modal 63% 10–20
Complex drag-and-drop 41% 20–40

For production, you need: retry logic, human-in-the-loop at low agent confidence, logging of all actions for audit.

Deliverables

  • Documentation — architecture description, operation manual, parameter tuning guide.
  • Training — 2–3 sessions for your team on how to fine-tune and maintain the agent.
  • 3-month support — monitoring, bug fixes, adaptation to new application versions.
  • Monitoring dashboard — success metrics, iteration count, p99 latency, anomaly logs.

We guarantee transparency: you see every step of the agent and can intervene at any moment.

Timeline

  • Basic Computer Use agent: 3–5 days
  • Specific automation task (one form/workflow): 1 week
  • System with retry and human-in-the-loop: 2 weeks
  • Production deployment with monitoring: 3–4 weeks

Assess your task — contact us, we'll show a demo on your application. Get an engineer's consultation: fill out the form on the website with a description of your GUI scenario, and we'll assess automation feasibility in 1 day.

Anthropic Computer Use beta documentation