Monitor GPU Utilization and VRAM with DCGM Exporter

Problem: how not to lose money on GPUs?

Picture this: you start fine-tuning an LLM on 8 A100s. After 6 hours, OOM strikes—2000 iterations lost. Or your GPUs are at 2% utilization while you pay full price. Without monitoring, it's a gamble. We've seen it on 30+ projects: proper GPU monitoring with DCGM Exporter, Prometheus, and Grafana cuts downtime by 70% and saves up to 40% on infra costs. Control VRAM, utilization, and Tensor Cores—key for stable workflow.

We offer a turnkey GPU monitoring setup in 2–3 days. Contact us for a free assessment.

Why DCGM Exporter is the best choice for NVIDIA GPUs?

DCGM (Data Center GPU Manager) is an official NVIDIA tool. Unlike nvidia-smi, it exposes profile metrics like Tensor Cores, NVLink, and accurate SM load. Comparison:

DCGM gives 10x more metrics and natively exports to Prometheus, as confirmed by NVIDIA documentation.

Which metrics are critical for AI workloads?

For AI engineers, not only basic utilization and VRAM matter. Tensor Cores are key for training and inference performance. The metric DCGM_FI_PROF_PIPE_TENSOR_ACTIVE shows how efficiently these blocks are used. If it's low while GPU load is high, you're likely hitting memory or bus bottlenecks. Also critical is NVLink usage—in distributed training, the bottleneck is often interconnects.

Typical problems and solutions

How we set up monitoring turnkey

Step 1. Deploy DCGM Exporter

We install DCGM Exporter via Docker on each GPU node:

docker run -d --gpus all --cap-add SYS_ADMIN -p 9400:9400 --name dcgm-exporter nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04

For clusters, we use docker-compose with additional collector settings. We match the version to your OS and NVIDIA drivers.

Step 2. Configure Prometheus

We create a scrape target for DCGM metrics and alerting rules:

# prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: dcgm
    static_configs:
      - targets:
          - gpu-server-1:9400
          - gpu-server-2:9400
rule_files:
  - "gpu_alerts.yml"

Step 3. Set up alerts

We define key alerts: OOM (VRAM >95%), overheating (>85°C), low utilization (<20%), and service unavailability.

# gpu_alerts.yml
groups:
  - name: gpu_alerts
    rules:
      - alert: GPUMemoryNearFull
        expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 0.95
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.gpu }} on {{ $labels.instance }}: VRAM > 95%"
      - alert: GPUUtilizationLow
        expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) < 20
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "Low GPU utilization on {{ $labels.instance }}"

How to set up an alert for OOM due to VRAM?

Add a rule that checks if VRAM exceeds 95% for 5 minutes. On trigger, you'll get a notification via Telegram or Slack. Additionally, you can set an alert on memory growth rate: if it increases by 10% in 2 minutes, it's a sign of impending OOM. We include such rules in the base configuration.

Step 4. Grafana dashboard

We build panels for utilization, VRAM, Tensor Cores, and temperature. Example VRAM panel:

{
  "title": "VRAM Usage %",
  "type": "gauge",
  "targets": [{
    "expr": "DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL * 100",
    "legendFormat": "{{instance}} GPU{{gpu}}"
  }],
  "fieldConfig": {
    "thresholds": {
      "steps": [
        {"color": "green", "value": 0},
        {"color": "yellow", "value": 80},
        {"color": "red", "value": 95}
      ]
    }
  }
}

We create separate dashboards for training (LLM, CV) and inference, adding Tensor Core utilization, NVLink throughput, and p99 latency for inference. All dashboards are adapted to your models.

Step 5. Documentation and training

After deployment, we hand over: operation manual, alert scheme, dashboard access, and a short training for DevOps/ML engineers.

What's included

• Installation and configuration of DCGM Exporter on each GPU node
• Prometheus and alerting rules setup (Telegram/Slack)
• Grafana dashboards tailored to your tasks (training, inference)
• Integration with existing monitoring stack
• Documentation and team training
• 1 month of support after launch

Our experience and guarantees

Years of experience in MLOps. 30+ GPU infrastructure projects for AI startups and enterprises. Certified NVIDIA engineers. We guarantee 24/7 monitoring stability. Reach out for a tailored setup—we'll evaluate your infrastructure and propose the best solution.