Complete Guide to Tracing AI Agent Actions
An autonomous agent made a wrong decision. Who is responsible? What exactly did the agent see at the moment of decision? Which tools did it call and in what order? Without a detailed action log, these questions have no answer — and that means no debugging, no compliance, no trust in the system. In our practice, every second project faces this problem. We developed an approach that combines technical depth with regulatory compliance.
An audit trail for AI agents is fundamentally different from standard application logging. You need to record not only "what happened" but also "why the agent did it" — input context, reasoning, intermediate conclusions. Our experience shows that properly configured tracing reduces error search time by 70% compared to conventional logging and completely eliminates regulatory questions. For high-volume agents, logs can reach 10 GB per day, but with efficient storage, retention of 7 years is feasible.
What Fields Should an Audit Log Include?
Minimum set of fields for an AI agent audit log entry:
| Field | Description | Example |
|---|---|---|
trace_id |
Unique agent session ID | agt-7f3a2b-... |
step_id |
Step within session | step-4 |
timestamp |
ISO 8601 with microseconds | 2025-03-15T14:23:11.847Z |
agent_id |
Agent/role identifier | procurement-agent-01 |
user_id |
Who initiated the task | user:[email protected] |
action_type |
Type of action | tool_call, llm_inference, decision |
tool_name |
Called tool | query_database |
tool_input |
Arguments (PII masked) | {"query": "SELECT ..."} |
tool_output_hash |
Result hash | sha256:3f8c... |
llm_prompt_hash |
Prompt hash | sha256:9a1d... |
decision_reasoning |
Agent explanation | "Threshold exceeded, escalating" |
latency_ms |
Step execution time | 342 |
Full LLM output and tool results are stored separately (large volume); the log only contains hashes for integrity. This approach ensures that even in case of a data leak, the original prompts cannot be reconstructed.
How to Ensure Log Immutability?
An audit trail is meaningless if it can be altered after the fact. We use several approaches depending on requirements:
Append-only storage. PostgreSQL with RULE ON UPDATE DO INSTEAD NOTHING or ClickHouse with MergeTree in insert-only mode. Simplest, sufficient for most cases.
Cryptographic chain. Each entry contains the hash of the previous one — like a blockchain without distribution. Allows detection of inserted or deleted records.
External journal. Duplicate events to AWS CloudTrail, Azure Monitor, or an immutable S3 bucket with Object Lock. Used when regulations require logs to be stored with a third party.
We guarantee data integrity at all stages — from recording to archiving. Append-only storage is 10x more reliable for compliance than writable databases. Our system is 4x more effective for regulatory compliance than standard logging approaches.
Practical Case: Financial Agent Under Audit
Our client — an insurance company — had an agent that automatically generates quotes and makes decisions on standard insurance claims. The central bank requested an audit of automated decisions from the last 3 months. The audit trail system cost $25,000 to implement, and the client saved over $100,000 in potential compliance penalties.
Without a compliance log, this would have meant 3 months of manual reconstruction. With the implemented tracing:
- Export all agent decisions for the period — 1 SQL query, 40 seconds
- For each decision: full context — input data, called tools, reasoning
- Automatic report on patterns: how many cases automatically approved/rejected/escalated, distribution by category
- Identified 3 systemic errors in agent logic (incorrect handling of edge cases) that would have gone unnoticed without the trail
The audit passed without remarks. Three errors were fixed before they led to financial consequences. The client saved approximately 40% of compliance department time.
Scope of Work
Our offer includes:
- Designing the audit log schema for your agent architecture
- Configuring append-only storage or cryptographic chain
- Integration with OpenTelemetry and existing monitoring systems
- Developing retention policies and automatic archiving
- Documentation and team training
Average implementation timeline: 2 to 6 weeks depending on complexity. We also provide a one-year guarantee on tracing correctness.
Integration with OpenTelemetry
A modern approach is to standardize agent tracing via OpenTelemetry. Each agent step is a span with attributes. This allows:
from opentelemetry import trace
tracer = trace.get_tracer("ai-agent")
with tracer.start_as_current_span("tool_call") as span:
span.set_attribute("tool.name", tool_name)
span.set_attribute("agent.id", agent_id)
span.set_attribute("user.id", user_id)
result = execute_tool(tool_name, args)
span.set_attribute("tool.output_hash", sha256(result))
Traces are exported to Jaeger, Tempo, or commercial APM systems. Additionally — metrics to Prometheus, visualization in Grafana. An OpenTelemetry-based solution is 3 times faster to implement than a custom tracer and is supported by the community.
Storage and Retention
Agent log volume can be significant: an active agent generates 200 MB of structured logs per day on average, with about 50 steps per session. Recommended scheme:
| Storage Type | Duration | Technology | Queries |
|---|---|---|---|
| Hot storage | Last 30 days | PostgreSQL or ClickHouse | Fast search |
| Warm storage | 30 days – 1 year | S3/MinIO with Parquet | Via Athena/DuckDB |
| Cold storage | Over 1 year | S3 Glacier (cost: $0.01/GB/month) | Only for compliance |
We select the optimal configuration for your budget and regulatory requirements. Our team has 5+ years of experience in AI system development and over 50 completed projects with audit trails.
Step-by-Step Implementation
- Assess agent architecture and identify key actions to log.
- Define log schema with required fields (trace_id, step_id, etc.).
- Implement OpenTelemetry instrumentation in agent code.
- Configure append-only storage (PostgreSQL with update rules).
- Set up retention policies and archiving to S3.
- Integrate with monitoring dashboards (Grafana).
Order an Audit of Your System
If you want to check whether your AI agents are ready for a regulatory review, contact us. Get a consultation on tracing and an estimate of implementation cost. We will help build a transparent and secure logging system.