AI-Powered Automatic Incident Diagnosis (RCA)

AI-Powered Automatic Incident Diagnosis (RCA)

Note: when an error occurs in a microservice architecture, the on‑call engineer manually runs Root Cause Analysis (RCA). They switch between Prometheus, Grafana, and Kibana, comparing metrics and logs — this takes 30 to 90 minutes. In a cascading failure caused by latency spikes in the payment service, they need to review 50+ dashboards and align timestamps down to the second. Each minute of downtime costs the business thousands of dollars (average downtime cost $5,000 per minute). We automate RCA: ML models analyze three‑pillar observability (metrics, logs, traces) in 1–5 minutes, narrowing the search to one or two root causes. Our experience — dozens of projects for enterprise clients, reducing MTTR by up to 80%. With over 5 years of experience and 50+ successful implementations, we have saved clients an average of $10,000 per month in downtime costs. Our ML-RCA system is 12–15 times faster than manual RCA, and covers 10 times more data.

Problems We Solve

Manual RCA faces three main challenges:

• Data noise: thousands of metrics, millions of logs per hour — a human cannot spot correlations within the incident window.
• Hidden dependencies: service A may degrade due to a cascading failure of service B located three hops deeper.
• Volatility: deploying a new version or autoscaling can cause a delayed effect after 15–20 minutes.

Our AI system solves these problems using correlation analysis, causal discovery, and semantic log analysis. For example, in a typical database incident, ML‑RCA finds the correlation between a growth in slow queries and a CPU drop in the caching service within 2 minutes.

Why ML‑RCA Is More Effective Than Manual

Let's compare approaches in a table:

Result: ML‑RCA finds the root cause 12–15 times faster for typical incidents, and false positives are reduced by 40% thanks to prioritization based on historical precedents.

How We Do It: Tech Stack and Tools

We build the pipeline from the following components:

• Metrics: Prometheus + Thanos, InfluxDB, Datadog. We compute cross‑correlation with time shifts (lag up to -5 minutes).
• Logs: Elasticsearch + Drain3 (online template parsing). We detect anomalous spikes in error template frequency.
• Traces: Jaeger / OpenTelemetry. We build a call graph in NetworkX and compute cascading failure risk using the PageRank algorithm.
• Causal discovery: apply the PC algorithm to infer directed links from stationary time series.
• LLM report: a model (GPT‑4 or equivalent) generates a human‑readable narrative from structured RCA data.

How It Works: Step by Step

1. Data Ingestion: Collect metrics, logs, and traces from your observability stack (Prometheus, Elasticsearch, Jaeger) in real time.
2. Anomaly Detection: Identify the incident window by detecting spikes in error rates, latency, or other key indicators.
3. Correlation Analysis: Compute cross‑correlation between all metric pairs with time shifts up to 10 minutes. Highlight top correlated pairs.
4. Log Parsing: Use Drain3 to parse logs into templates and detect sudden increases in error templates.
5. Causal Discovery: Apply PC algorithm on stationary time series to infer causal directions.
6. Root Cause Ranking: Combine evidence from metrics, logs, and traces to rank potential root causes.
7. Report Generation: Produce a natural language summary using an LLM, including the most likely root cause, supporting evidence, and recommendations.

def find_correlated_metrics(incident_time, all_metrics, window_minutes=30, threshold=0.7):
    incident_window = all_metrics[
        incident_time - pd.Timedelta(minutes=window_minutes):
        incident_time + pd.Timedelta(minutes=5)
    ]
    # ... (full code in original documentation)

Log analysis via Drain3:

from drain3 import TemplateMiner

def parse_and_analyze_logs(log_lines, incident_time, window_minutes=10):
    miner = TemplateMiner()
    template_counts = defaultdict(list)
    for line in log_lines:
        result = miner.add_log_message(line.message)
        template_id = result['cluster_id']
        template_counts[template_id].append(line.timestamp)
    # ... (full code)

What Is Needed for AI‑RCA Implementation?

The system requires three observability sources: metrics (Prometheus, Datadog), logs (Elasticsearch, Loki), and traces (Jaeger, Tempo). Additionally, we use Kubernetes events, deployment history, and infrastructure changes. The richer the data, the more accurate the result. Typical volume: 5000+ metrics, 10 million logs per day, 1000 traces per second.

Deployment Phases

The process includes five phases:

Each phase is accompanied by documentation and knowledge transfer to your team.

What Is Included (Deliverables)

• Architecture and data source analysis.
• Development and customization of ML modules (correlation, logs, graphs, causal discovery).
• Integration with existing stack (Prometheus, ELK, Jaeger, etc.).
• Visualization panels for RCA results in Grafana.
• Precedent knowledge base (vector search via incident description embeddings).
• Documentation in Russian and English.
• Access to all code and configuration templates.
• Team training: 2–3 hands‑on sessions.
• Support guarantee for 3 months after deployment.

Timeline and Cost

Basic module (correlation + log parsing) — from 4 weeks, starting at $15,000. Full solution with causal graph and LLM reports — from 3 months, starting at $50,000. Cost is calculated individually after an audit — contact us for a project assessment. Savings amount to tens of thousands of dollars per month due to reduced downtime. Typical annual savings exceed $100,000 for mid‑size companies.

Typical Mistakes When Implementing RCA

• Ignoring traces: without them, you won't see cascading calls.
• Too short a correlation window: many failures have a lag of 10–15 minutes.
• Lack of metric normalization: different units and scales distort correlation.

We help avoid these pitfalls during the audit phase.

Root Cause Analysis — the methodology underlying the approach.

Ready to discuss your project? Request a consultation: we'll show how AI diagnosis reduces your MTTR and operational expenses.