Configuring Load Balancing Between GPU Instances

Imagine: you launch four GPU instances with vLLM, but 80% of requests go to the first server. The rest sit idle while users complain about timeouts. The reason — load balancing isn't configured. For LLMs this is critical: one long request of 4000 tokens can block a server for a minute, while the others remain idle. As a result, p99 latency skyrockets to 30 seconds, and GPU utilization drops to 25%. A typical cluster of 4 GPUs without balancing loses up to 50% throughput.

Proper load balancing reduces GPU infrastructure costs by up to 40% through uniform utilization. Average GPU-hour savings after implementation — 30% under the same load. P99 latency drops 1.7x compared to Round Robin. If you face similar issues, contact us — we will select the optimal configuration for your scenario.

Comparison of Balancing Algorithms for LLMs

Least Pending Tokens is the optimal choice for services with heterogeneous load. It uses Prometheus metrics from vLLM (vllm:num_requests_waiting) to select the least loaded instance. Our experience shows that Least Pending Tokens outperforms Round Robin by 1.7x in p99 latency.

Example: Nginx with Health Checks and Custom Balancer

Below is a basic Nginx configuration for an upstream of four vLLM servers, with active health checks and timeouts for streaming.

upstream vllm_cluster {
    least_conn;

    server 10.0.1.10:8000 max_fails=3 fail_timeout=30s weight=1;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=30s weight=1;
    server 10.0.1.12:8000 max_fails=3 fail_timeout=30s weight=1;
    server 10.0.1.13:8000 max_fails=3 fail_timeout=30s weight=1;

    keepalive 100;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}

server {
    listen 443 ssl http2;
    server_name llm-api.internal;

    location /v1/ {
        proxy_pass http://vllm_cluster;
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Timeout для длинных streaming ответов
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
        proxy_connect_timeout 5s;

        # Streaming: отключаем буферизацию
        proxy_buffering off;
        proxy_cache off;
        chunked_transfer_encoding on;

        # Circuit breaker
        proxy_next_upstream error timeout http_500 http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 10s;
    }

    location /health {
        proxy_pass http://vllm_cluster/health;
    }
}

If a more intelligent backend selection is needed, we write a custom balancer in FastAPI that polls metrics in real time.

from fastapi import FastAPI, Request
import httpx
import asyncio

class LLMLeastPendingBalancer:
    def __init__(self, backends: list[str]):
        self.backends = {url: {"pending": 0, "healthy": True} for url in backends}
        self.client = httpx.AsyncClient(timeout=300)

    async def get_backend(self) -> str:
        """Выбираем backend с наименьшим числом pending токенов."""
        healthy = {url: info for url, info in self.backends.items() if info["healthy"]}
        if not healthy:
            raise RuntimeError("No healthy backends")

        metrics = await self._fetch_metrics(list(healthy.keys()))
        best = min(metrics.items(), key=lambda x: x[1].get("vllm_num_requests_waiting", 0))
        return best[0]

    async def _fetch_metrics(self, backends: list[str]) -> dict:
        tasks = [self._get_backend_queue(url) for url in backends]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        return {url: result for url, result in zip(backends, results)
                if not isinstance(result, Exception)}

    async def _get_backend_queue(self, url: str) -> dict:
        response = await self.client.get(f"{url}/metrics")
        for line in response.text.split('\n'):
            if line.startswith('vllm:num_requests_waiting'):
                return {"vllm_num_requests_waiting": float(line.split()[-1])}
        return {"vllm_num_requests_waiting": 0}

    async def forward(self, request: Request) -> httpx.Response:
        backend = await self.get_backend()
        url = f"{backend}{request.url.path}"
        self.backends[backend]["pending"] += 1
        try:
            return await self.client.request(
                method=request.method,
                url=url,
                content=await request.body(),
                headers=dict(request.headers)
            )
        finally:
            self.backends[backend]["pending"] -= 1

app = FastAPI()
balancer = LLMLeastPendingBalancer(["http://gpu1:8000", "http://gpu2:8000", "http://gpu3:8000"])

@app.api_route("/v1/{path:path}", methods=["GET", "POST"])
async def proxy(path: str, request: Request):
    return await balancer.forward(request)

Why Sticky Sessions Are Critical for LLMs?

If your LLM uses KV cache prefix reuse (e.g., a common system prompt in a chatbot), without sticky sessions each request may land on a different server — the cache becomes useless. The solution — consistent hashing by prefix and sticky sessions.

def get_backend_by_prefix(prompt: str, backends: list[str]) -> str:
    prefix_hash = hashlib.md5(prompt[:256].encode()).hexdigest()
    idx = int(prefix_hash, 16) % len(backends)
    return backends[idx]

Applying sticky sessions increases cache hit ratio by 30-50%, reducing latency by 20%. Without them, a typical service with a shared system prompt loses up to 60% of cache efficiency.

Typical Mistakes in GPU Balancing

• Using Round Robin for LLMs — leads to uneven load.
• Lack of health checks — traffic goes to a failed server.
• Ignoring streaming timeouts — clients get 502 errors during long generations.
• Incorrect proxy_buffering configuration — increases latency.
• No GPU failover — all traffic is lost when one instance fails.

How to Set Up Health Checks for GPU Instances?

What's Included in Turnkey Load Balancing Configuration

1. Analysis of load scenarios (number of requests, token length, latency requirements).
2. Selection of algorithm and stack (Nginx, custom balancer, Envoy).
3. Configuration of health checks, circuit breaker, timeouts.
4. Implementation of sticky sessions (if KV cache is needed).
5. Integration with monitoring (Prometheus + Grafana dashboards).
6. Operational documentation and Incident Playbook.

Process of Work

• Analytics — collection of current infrastructure metrics, request profiling.
• Design — balancing architecture, algorithm selection, failover scheme.
• Implementation — deploying configs or writing custom module.
• Testing — load testing with p50/p99/p999 latency measurements.
• Deployment — phased rollout with canary release.

Timelines and Cost

Basic configuration on Nginx — from 1 day. Custom balancer with Least Pending Tokens support — from 3 to 5 days. Cost is calculated individually, based on infrastructure complexity and fault tolerance requirements. Guaranteed service stability after implementation — our engineers with 10+ years of experience in ML infrastructure deliver turnkey work. Typical ROI after implementation — 6 months.

Load Distribution Monitoring

After implementation, track: RPS distribution (should be uniform ±20%), queue depth on each backend, error rate, p99 latency. Set an alert: "one backend receives >80% traffic" — a sign of failure. With proper configuration, p99 latency drops to 5 seconds, and GPU utilization increases to 95%. Cache hit ratio reaches 70% with sticky sessions. We also train your team to work with dashboards.

Contact us for a preliminary audit — we will assess your current configuration and propose the optimal solution. Order a consultation — we will help you choose a balancing strategy for your GPU cluster.