GUI Automation with Computer Use AI Agent
Have you ever faced a situation where a legacy ERP on Windows has no API, and automation via RPA breaks after every update? Computer Use solves this: the agent sees the screen via screenshots and acts like a human. We develop such agents "turnkey" — from assessment to production with monitoring. Our team has 10+ years of experience in AI/ML and over 40 completed projects. We'll assess your task in 1 day — contact us.
Why Computer Use is More Reliable Than RPA
Classic RPA requires precise description of element paths (XPath, coordinates, CSS selectors) and a stable screen structure. Computer Use uses computer vision: the agent analyzes the screenshot and determines which actions will lead to the goal. This provides resilience to interface changes — if a button moves, the agent will still find it. Vision-based automation adapts to UI changes 5x faster than RPA.
| Characteristic | RPA | Computer Use |
|---|---|---|
| Adaptation to UI changes | Low (scripts break) | High (model sees changes) |
| Setup for a new task | Days-weeks | Hours-days |
| API requirement | Not required | Not required |
| Handling non-standard elements | Difficult (popup, drag-and-drop) | Possible with refinement |
How Computer Use Works
The basic principle: agent receives screenshot → analyzes interface → selects action → executes → receives new screenshot → repeats until task completion.
Example Python agent implementation (Claude + PyAutoGUI)
import anthropic
import base64
import subprocess
from PIL import ImageGrab
import pyautogui
import time
import json
from io import BytesIO
client = anthropic.Anthropic()
def capture_screenshot() -> str:
"""Takes a screenshot and returns base64"""
screenshot = ImageGrab.grab()
# Resize to save tokens
screenshot = screenshot.resize(
(screenshot.width // 2, screenshot.height // 2)
)
buffer = BytesIO()
screenshot.save(buffer, format="PNG", optimize=True)
return base64.standard_b64encode(buffer.getvalue()).decode("utf-8")
def execute_computer_action(action: dict) -> str:
"""Executes action on the computer"""
action_type = action.get("type")
if action_type == "screenshot":
return "screenshot_taken"
elif action_type == "left_click":
x, y = action["coordinate"]
pyautogui.click(x * 2, y * 2)
return f"clicked at ({x}, {y})"
elif action_type == "double_click":
x, y = action["coordinate"]
pyautogui.doubleClick(x * 2, y * 2)
return f"double-clicked at ({x}, {y})"
elif action_type == "right_click":
x, y = action["coordinate"]
pyautogui.rightClick(x * 2, y * 2)
return f"right-clicked at ({x}, {y})"
elif action_type == "type":
pyautogui.write(action["text"], interval=0.05)
return f"typed: {action['text'][:50]}..."
elif action_type == "key":
pyautogui.hotkey(*action["key"].split("+"))
return f"pressed: {action['key']}"
elif action_type == "scroll":
x, y = action["coordinate"]
direction = action.get("direction", "down")
amount = action.get("amount", 3)
if direction == "down":
pyautogui.scroll(-amount, x=x * 2, y=y * 2)
else:
pyautogui.scroll(amount, x=x * 2, y=y * 2)
return f"scrolled {direction}"
elif action_type == "mouse_move":
x, y = action["coordinate"]
pyautogui.moveTo(x * 2, y * 2)
return f"moved to ({x}, {y})"
return f"unknown action: {action_type}"
class ComputerUseAgent:
"""Agent for GUI automation via Computer Use"""
COMPUTER_TOOLS = [{
"type": "computer_20241022",
"name": "computer",
"display_width_px": 960,
"display_height_px": 540,
"display_number": 1,
}]
def __init__(self, max_iterations: int = 50):
self.max_iterations = max_iterations
def run_task(self, task: str, context: str = "") -> dict:
"""Executes a task on the computer"""
system = f"""You control the computer to perform the task.
Use the computer tool to interact with the interface.
Take a screenshot before each action to verify the current state.
Report when the task is complete or if you encounter an insurmountable obstacle.
{"Context: " + context if context else ""}"""
messages = [{"role": "user", "content": task}]
iterations = 0
actions_log = []
while iterations < self.max_iterations:
screenshot_b64 = capture_screenshot()
if messages[-1]["role"] == "user" and isinstance(messages[-1]["content"], str):
messages[-1]["content"] = [
{"type": "text", "text": messages[-1]["content"]},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": screenshot_b64,
},
}
]
response = client.beta.messages.create(
model="claude-opus-4-5",
max_tokens=4096,
system=system,
tools=self.COMPUTER_TOOLS,
messages=messages,
betas=["computer-use-2024-10-22"],
)
tool_results = []
task_completed = False
for block in response.content:
if hasattr(block, "text"):
if any(kw in block.text.lower() for kw in ["task completed", "done", "готово", "completed", "done"]):
task_completed = True
elif block.type == "tool_use" and block.name == "computer":
action = block.input
result = execute_computer_action(action)
actions_log.append({"action": action, "result": result})
time.sleep(0.5)
new_screenshot = capture_screenshot()
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": [{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": new_screenshot,
}
}],
})
if task_completed or response.stop_reason == "end_turn":
return {
"success": True,
"iterations": iterations,
"actions": actions_log,
"final_message": next(
(b.text for b in response.content if hasattr(b, "text")), ""
),
}
messages.append({"role": "assistant", "content": response.content})
if tool_results:
messages.append({"role": "user", "content": tool_results})
else:
break
iterations += 1
return {"success": False, "iterations": iterations, "actions": actions_log}
How to Implement Screen-Aware Automation in 1 Week
- Task Analysis — describe the scenario: which windows, which fields, expected result.
- Prototype — run a basic agent on your PC (3–5 days).
- Calibration — configure coordinates, retry logic, human-in-the-loop.
- Test — run 100+ sessions, collect metrics.
- Deploy — package into a service, add monitoring.
Practical Applications
Processing outdated desktop applications without API. An ERP system without API but with Windows GUI. The agent enters a form, fills fields, clicks "Post", gets the result — all via screenshots. Speed: 40–60 operations per hour vs 15–20 for an operator. Time savings reach 70%, reducing costs for maintaining legacy systems. Typical project cost starts at $5,000, with average annual savings of $30,000 in labor.
Data migration between systems. Export from old CRM → import to new one when there is no direct integration. The agent copies records via GUI of both systems.
Regression testing. The agent goes through user scenarios in a web application, checks results, logs anomalies.
Typical Problems and Their Solutions
- Model hallucinations. The agent may "think" a click was successful even though the UI hasn't changed. Solution: retry logic with state verification after each action.
- Sensitivity to screen scaling. If the screenshot is resized, coordinates may be inaccurate. We use automatic scale detection and calibration.
- Latency. One step takes 1–3 seconds. For long scenarios, this can be slow. Optimization: batch processing and reducing the number of screenshots.
Limitations and Real Metrics
Honest numbers from our practice:
| Task | Success Rate | Iterations |
|---|---|---|
| Fill a form with 10 fields | 87% | 8–15 |
| Navigate through 3+ screens | 71% | 15–25 |
| Work with popup/modal | 63% | 10–20 |
| Complex drag-and-drop | 41% | 20–40 |
For production, you need: retry logic, human-in-the-loop at low agent confidence, logging of all actions for audit.
Deliverables
- Documentation — architecture description, operation manual, parameter tuning guide.
- Training — 2–3 sessions for your team on how to fine-tune and maintain the agent.
- 3-month support — monitoring, bug fixes, adaptation to new application versions.
- Monitoring dashboard — success metrics, iteration count, p99 latency, anomaly logs.
We guarantee transparency: you see every step of the agent and can intervene at any moment.
Timeline
- Basic Computer Use agent: 3–5 days
- Specific automation task (one form/workflow): 1 week
- System with retry and human-in-the-loop: 2 weeks
- Production deployment with monitoring: 3–4 weeks
Assess your task — contact us, we'll show a demo on your application. Get an engineer's consultation: fill out the form on the website with a description of your GUI scenario, and we'll assess automation feasibility in 1 day.
Anthropic Computer Use beta documentation







