GPU Autoscaling for AI Workloads: HPA, KEDA, Pre-Warming

GPU instances for AI are expensive: an A100 hour costs a lot, and idle time eats your budget. Cold start of a new pod takes 3–10 minutes (model loading, CUDA initialization), and standard CPU autoscaling does not account for GPU specifics. Without proper autoscaling, you pay for idle time or lose requests during peaks. We solve this: we configure scaling that actually meets SLA and reduces costs by 30–40%.

Problems We Solve

Cold start: GPU pod startup time 3–10 minutes. During this, the request queue overflows. Solutions: keepalive pods (minimum 1), pre-warming (start at 70% queue load), and buffering via request queue. Using spot instances worsens the problem — they can be interrupted anytime, so we combine spot with on-demand for critical services.

GPU utilization vs queue depth: GPU load during a long request is 100%, but new requests wait. Relying on utilization is a mistake. The correct metric is vllm_num_requests_waiting or queue_depth. We use a combination: scale-up by queue depth, scale-down by utilization.

Thrashing pods: Frequent spin-ups and downs due to sharp spikes. Configuring stabilization windows prevents this: scale-down after 10 minutes, scale-up within 30 seconds.

Why Queue Depth Is the Main Metric for LLM?

GPU utilization does not reflect queue latency. When a model processes a long request, GPU is 100% busy, but new requests are queued. Queue depth shows the real resource need. It should be the primary trigger for scale-up.

How We Configure GPU Autoscaling

Kubernetes HPA with Custom Metrics

# Prometheus Adapter for custom metrics
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-autoscaler
  namespace: ai-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3
  minReplicas: 1
  maxReplicas: 8
  metrics:
    # Main metric: queue of waiting requests
    - type: Pods
      pods:
        metric:
          name: vllm_pending_requests
        target:
          type: AverageValue
          averageValue: "5"          # scale when > 5 requests in queue per pod

    # Additional: GPU utilization (for scale-down)
    - type: Pods
      pods:
        metric:
          name: nvidia_gpu_duty_cycle
        target:
          type: AverageValue
          averageValue: "70"         # scale-down when < 70% utilization

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30     # fast scale-up
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60              # +2 pods every minute
    scaleDown:
      stabilizationWindowSeconds: 600    # slow scale-down (10 minutes)
      policies:
        - type: Pods
          value: 1
          periodSeconds: 300             # -1 pod every 5 minutes

KEDA for Event-Driven Autoscaling

KEDA is more flexible than HPA: scaling by Prometheus, Kafka, RabbitMQ, SQS. Below is an example configuration.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-keda-scaler
  namespace: ai-serving
spec:
  scaleTargetRef:
    name: vllm-llama3
  minReplicaCount: 1
  maxReplicaCount: 10
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        metricName: vllm_queue_size
        query: sum(vllm_num_requests_waiting{namespace="ai-serving"})
        threshold: "10"           # 1 replica per 10 waiting requests

    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
        metricName: request_rate
        query: rate(http_requests_total{job="vllm"}[2m])
        threshold: "20"           # additional trigger by RPS

Cloud-Native Autoscaling

For cloud GPU instances, we use Auto Scaling Groups with custom metrics:

import boto3

autoscaling = boto3.client('autoscaling', region_name='us-east-1')

autoscaling.put_scaling_policy(
    AutoScalingGroupName='llm-gpu-asg',
    PolicyName='scale-on-queue-depth',
    PolicyType='TargetTrackingScaling',
    TargetTrackingConfiguration={
        'CustomizedMetricSpecification': {
            'MetricName': 'LLMQueueDepth',
            'Namespace': 'Custom/LLMMetrics',
            'Statistic': 'Average',
        },
        'TargetValue': 5.0,
        'ScaleInCooldown': 300,
        'ScaleOutCooldown': 60,
        'DisableScaleIn': False,
    }
)

Publishing Custom Metrics from vLLM

We collect metrics directly from the inference server:

from prometheus_client import Gauge, start_http_server
import requests
import time

QUEUE_SIZE = Gauge('llm_queue_depth', 'Number of pending requests')
GPU_MEMORY = Gauge('llm_gpu_memory_used_gb', 'GPU memory usage in GB', ['gpu_id'])

def collect_metrics():
    response = requests.get("http://localhost:8000/metrics").text
    for line in response.split('\n'):
        if 'vllm:num_requests_waiting' in line and not line.startswith('#'):
            queue_size = float(line.split()[-1])
            QUEUE_SIZE.set(queue_size)

    import subprocess
    result = subprocess.run(
        ['nvidia-smi', '--query-gpu=memory.used', '--format=csv,noheader,nounits'],
        capture_output=True, text=True
    )
    for i, mem_mb in enumerate(result.stdout.strip().split('\n')):
        GPU_MEMORY.labels(gpu_id=str(i)).set(float(mem_mb) / 1024)

start_http_server(9091)
while True:
    collect_metrics()
    time.sleep(15)

How Pre-Warming Prevents Cold Start?

We prevent cold start by forecasting load. If queue depth exceeds 70% of the maximum, we launch an additional pod in advance. Strategy code:

class PreWarmingStrategy:
    def __init__(self, warmup_threshold: float = 0.7, warmup_lead_time: int = 180):
        self.warmup_threshold = warmup_threshold
        self.warmup_lead_time = warmup_lead_time

    def should_scale_up(self, current_queue: int, max_queue: int, forecast: list) -> bool:
        if current_queue / max_queue >= self.warmup_threshold:
            return True
        future_queue = forecast[self.warmup_lead_time // 15]
        return future_queue / max_queue >= self.warmup_threshold

Why GPU Autoscaling Is Harder Than CPU?

Key differences: GPU instances cannot be "split" between services, cold start is much longer (model loading ~5 GB), and load metrics are misleading. On CPU you rely on CPU utilization, on GPU — only on queue depth and request arrival rate. Also, the cost of error is higher: an extra GPU pod means extra monthly expenses.

How to Choose a Metric for Scaling?

The best solution is a combination: queue depth for scale-up and GPU utilization for scale-down.

What Our Work Includes

We offer turnkey implementation:

1. Infrastructure audit — load analysis, bottleneck identification, instance selection.
2. Scaling scheme design — metric selection, HPA/KEDA configuration, behavior optimization (stabilization windows, policies).
3. Implementation — Prometheus stack deployment, vLLM/TGI integration, cloud autoscaling group setup.
4. Testing — load testing with peak simulation, threshold calibration.
5. Documentation and training — runbook for on-call, metric and alert description.
6. Post-release support — monitoring for first weeks, policy adjustments based on actuals.

Timeline and Savings

Estimated timeline:

• Basic setup (metrics + HPA) — from 5 days.
• Extended configuration (KEDA + pre-warming) — from 10 days.
• Full cycle with load testing — from 3 weeks.

Cost is calculated individually based on stack complexity and workload. Our clients save 30–40% monthly GPU costs after implementation.

Order GPU autoscaling implementation and start saving within a week. We have been working with AI infrastructure for 5+ years, completed 20+ projects. We guarantee stable operation — we include SLA on scaling response time in the contract. Kubernetes HPA and KEDA are proven tools. Contact us for a consultation on your project — we will analyze your load and offer the optimal scheme.