AI Agent for Desktop Application Automation

Desktop applications without an API are every automation engineer's classic headache. Legacy ERP, CAD programs, bank clients, 1C in thick client mode. Selenium can't see them, there's no REST API. We solve this with an AI agent that either uses Computer Use (screenshot + mouse/keyboard control) or accesses the accessibility API (pywinauto, UI Automation) — a more reliable option for Windows applications. In this article, we'll break down how to build such an agent, the pitfalls, and how much it saves.

Why an AI Agent Instead of RPA?

Classical RPA systems (UiPath, Blue Prism) require rigid UI markup and break on any update. An LLM-based AI agent understands the UI at the semantic level: it doesn't need a predefined selector — it analyzes the element tree and decides on an action. This reduces maintenance costs by 60% over the long term — 3x more resilient to UI changes than RPA. Typical projects start at $5,000 and pay back within 3 months. We rely on the Windows UI Automation API and pywinauto documentation.

How We Build Such an Agent?

At the core is a pywinauto + Anthropic API bundle. pywinauto accesses the Windows UI Automation API — the same technology used by screen readers. Elements are located by accessibility attributes (AutomationId, Name, ControlType), not by pixels. Significantly more reliable than screenshots.

from anthropic import Anthropic
import pywinauto
from pywinauto.application import Application
from pywinauto.findwindows import ElementNotFoundError
import json
import subprocess
import time

client = Anthropic()


class DesktopAppAgent:
    """AI agent for Windows desktop app automation"""

    def __init__(self, app_path: str = None, app_title: str = None):
        self.app_path = app_path
        self.app_title = app_title
        self.app = None
        self.main_window = None

    def launch_or_connect(self):
        """Launches app or connects to a running instance"""
        try:
            if self.app_title:
                self.app = Application(backend="uia").connect(title_re=f".*{self.app_title}.*")
            elif self.app_path:
                self.app = Application(backend="uia").start(self.app_path)
                time.sleep(2)  # wait for init
        except pywinauto.findwindows.ElementNotFoundError:
            if self.app_path:
                self.app = Application(backend="uia").start(self.app_path)
                time.sleep(2)

        self.main_window = self.app.top_window()

    def get_ui_tree(self, max_depth: int = 4) -> dict:
        """Gets UI element tree"""
        def extract_element(element, depth=0):
            if depth > max_depth:
                return None
            try:
                info = {
                    "name": element.window_text()[:100] if element.window_text() else "",
                    "control_type": element.element_info.control_type,
                    "automation_id": element.element_info.automation_id or "",
                    "enabled": element.is_enabled(),
                    "visible": element.is_visible(),
                    "rect": str(element.rectangle()),
                }
                children = []
                for child in element.children():
                    child_info = extract_element(child, depth + 1)
                    if child_info and (child_info["visible"] or child_info["enabled"]):
                        children.append(child_info)
                if children:
                    info["children"] = children[:20]  # max 20 children
                return info
            except Exception:
                return None

        return extract_element(self.main_window)

    def find_and_interact(self, instruction: str) -> str:
        """LLM determines which element is needed and what to do"""
        ui_tree = self.get_ui_tree()

        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=512,
            messages=[{
                "role": "user",
                "content": f"""Analyze the UI tree and return a JSON with the action:
{{
  "action": "click|type|select|get_value|find",
  "automation_id": "element ID if any",
  "name": "element name",
  "control_type": "element type",
  "value": "value for type/select"
}}

Instruction: {instruction}

UI tree:
{json.dumps(ui_tree, ensure_ascii=False)[:4000]}

Only JSON."""
            }],
        )

        try:
            text = response.content[0].text
            action = json.loads(text[text.find("{"):text.rfind("}") + 1])
            return self._execute_ui_action(action)
        except Exception as e:
            return f"Parsing error: {e}"

    def _execute_ui_action(self, action: dict) -> str:
        """Executes action on UI element"""
        try:
            # Search by automation_id or name
            element = None

            if action.get("automation_id"):
                element = self.main_window.child_window(
                    auto_id=action["automation_id"]
                )
            elif action.get("name"):
                element = self.main_window.child_window(
                    title=action["name"],
                    control_type=action.get("control_type"),
                )

            if not element:
                return "Element not found"

            act = action.get("action", "click")

            if act == "click":
                element.click_input()
                return f"Clicked on {action.get('name', action.get('automation_id'))}"

            elif act == "type":
                element.set_edit_text(action.get("value", ""))
                return f"Typed: {action.get('value', '')}"

            elif act == "select":
                element.select(action.get("value", ""))
                return f"Selected: {action.get('value', '')}"

            elif act == "get_value":
                return element.window_text() or element.get_value()

        except ElementNotFoundError:
            return f"Element not found: {action}"
        except Exception as e:
            return f"Error: {type(e).__name__}: {e}"

        return "Action executed"


class DesktopWorkflowAgent:
    """High-level agent for executing tasks in a desktop app"""

    TOOLS = [
        {
            "name": "get_ui_state",
            "description": "Gets the current UI tree state of the application",
            "input_schema": {"type": "object", "properties": {}},
        },
        {
            "name": "interact_with_element",
            "description": "Interacts with a UI element (click, type, select)",
            "input_schema": {
                "type": "object",
                "properties": {
                    "automation_id": {"type": "string"},
                    "action": {"type": "string", "enum": ["click", "type", "select", "get_value"]},
                    "value": {"type": "string"},
                },
                "required": ["action"],
            },
        },
        {
            "name": "wait",
            "description": "Waits for a state change in the application",
            "input_schema": {
                "type": "object",
                "properties": {
                    "seconds": {"type": "number", "default": 1.0},
                    "wait_for_element": {"type": "string"},
                },
            },
        },
        {
            "name": "keyboard_shortcut",
            "description": "Presses a key combination (Ctrl+S, Alt+F4, etc.)",
            "input_schema": {
                "type": "object",
                "properties": {
                    "shortcut": {"type": "string", "description": "E.g., Ctrl+S, Alt+Tab, F2"},
                },
                "required": ["shortcut"],
            },
        },
    ]

    def __init__(self, desktop_agent: DesktopAppAgent):
        self.agent = desktop_agent

    async def run(self, task: str) -> dict:
        messages = [{"role": "user", "content": task}]
        steps = 0

        while steps < 40:
            response = client.messages.create(
                model="claude-sonnet-4-5",
                max_tokens=1024,
                system="You are a desktop app automation agent. Use tools sequentially.",
                tools=self.TOOLS,
                messages=messages,
            )

            tool_results = []
            done = False

            for block in response.content:
                if hasattr(block, "text") and block.text:
                    if "done" in block.text.lower() or "completed" in block.text.lower():
                        done = True

                elif block.type == "tool_use":
                    result = ""
                    inp = block.input

                    if block.name == "get_ui_state":
                        result = json.dumps(self.agent.get_ui_tree(max_depth=3), ensure_ascii=False)[:3000]

                    elif block.name == "interact_with_element":
                        result = self.agent._execute_ui_action(inp)

                    elif block.name == "wait":
                        time.sleep(inp.get("seconds", 1.0))
                        result = "Waited"

                    elif block.name == "keyboard_shortcut":
                        import pyautogui
                        keys = inp["shortcut"].replace("+", " ").split()
                        pyautogui.hotkey(*[k.lower() for k in keys])
                        result = f"Pressed {inp['shortcut']}"

                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result,
                    })

            if done or response.stop_reason == "end_turn":
                return {"success": True, "steps": steps}

            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})
            steps += 1

        return {"success": False, "steps": steps}

Which Approach to Choose: pywinauto vs Computer Use?

We use both approaches: pywinauto as primary, Computer Use as fallback. If the app has an accessible UI model, we prefer pywinauto; otherwise, Computer Use.

When is Computer Use Needed as a Fallback?

Computer Use helps when the app uses custom graphics or doesn't provide an accessibility tree — for example, old CAD systems or data visualizers. In such cases, the agent takes a screenshot of the area and passes it to the LLM for analysis. However, it's 2x slower and less reliable: p99 latency is 3x higher and the model may misjudge coordinates. Therefore, we use Computer Use only where pywinauto is powerless.

Practical Case: Automating 1C:Accounting from Our Practice

Task: Monthly generation of 40 reports in 1C for 12 legal entities. The process took 3 working days for two accountants.

Approach: pywinauto for 1C desktop client (version 8.3). UI Automation works with 1C through COM objects and accessibility API.

Results:

• 40 reports × 12 entities: 3 working days → 4 hours (nightly run) — 80% reduction in processing time
• Manual input errors: reduced to 0
• Challenge: 1C periodically changes automation_id on updates — we added fallback search by element name

Key point when working with 1C: the app uses a custom engine; not all elements are visible through standard UI Automation. Part of the automation is implemented directly through 1C's COM interface. Our experience shows that a hybrid scheme (pywinauto + COM) yields the best result.

What's Included in the Work?

1. UI tree analysis — identifying accessible elements, creating an automation map.
2. Agent development for a specific workflow — implementing action sequences with fallback logic.
3. LLM integration — selecting the model (Claude or GPT), setting up prompts for stable JSON command generation.
4. Batch processing + monitoring — scheduled runs, error alerting.
5. Documentation — architecture description, run instructions, maintenance.

Estimated Timelines

We guarantee agent stability throughout the entire operational period — if the application updates, we adjust selectors and prompts. We've been working with desktop automation for over 5 years, completed more than 20 projects for banks, retail, and logistics.

Get a consultation for your scenario — we'll estimate the project in 1–2 days. Contact us, and we'll suggest the optimal turnkey solution.