Optimizing Dynamic Batching for LLMs: Boost GPU Throughput

If your LLM service experiences high load with many concurrent users, each request is processed sequentially—without batching, throughput drops significantly and latency becomes unacceptable. We configure dynamic batching to drive GPU utilization to 80%+ instead of 5%. Our engineers have 4+ years of LLM production experience and have delivered over 20 projects on vLLM, TensorRT-LLM, and custom solutions. Dynamic batching merges multiple parallel requests into a single forward pass through the GPU. This is a key mechanism for high LLM throughput: the GPU is parallel and handles matrix multiplications more efficiently with larger batches. Proper batching tuning can reduce the number of required GPUs by a factor of 3–5, saving between 150,000 and 500,000 rubles per month on infrastructure.

Why Batching Is Critical for LLMs

Without batching, even a powerful A100 80GB GPU delivers only 30 tokens/sec for a Llama-3-8B model. With batch=16, it jumps to 300 tokens/sec; with batch=64, it reaches 900 tokens/sec—a 30x improvement. However, p99 latency rises from 200 ms to 400 ms, still acceptable for most real-time scenarios. If you have 100 concurrent users, without batching each waits in queue—total response time can exceed a minute. With continuous batching, all requests are processed in parallel, reducing response time to seconds.

Why Continuous Batching Outperforms Static Batching

Static batching fixes the batch size and waits for it to fill, increasing latency under low load. Continuous batching (in-flight batching) dynamically adds requests as soon as the GPU is free, reducing wait time and improving utilization.

Continuous (In-Flight) Batching in vLLM

According to the official vLLM documentation, continuous batching is implemented automatically. Key parameters: max-num-seqs—maximum number of requests per batch, max-num-batched-tokens—total tokens per batch, scheduler-delay-factor—delay before forming a batch. Example configuration:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8b-instruct \
  --max-num-seqs 256 \
  --max-num-batched-tokens 32768 \
  --scheduler-delay-factor 0.5 \
  --use-v2-block-manager \
  --enable-chunked-prefill

Chunked prefill splits long prefill into chunks to avoid blocking decode of other requests:

--enable-chunked-prefill
--max-num-batched-tokens 8192

How to Tune Dynamic Batching for a Specific GPU

Follow these steps:

1. Determine model and GPU. For example, Llama-3-8B on A100-80GB.
2. Choose a framework. vLLM for a quick start, TensorRT-LLM for maximum performance.
3. Run benchmarks. Load test with varying numbers of concurrent users.
4. Tune parameters. max-num-seqs, max-num-batched-tokens, scheduler-delay-factor.
5. Monitor. Track metrics like num_requests_running, avg_prompt_throughput_toks_per_s.

Which Monitoring Metrics Matter for Batching?

vLLM exports metrics via Prometheus: num_requests_running (requests in active batch), num_requests_waiting (queued), avg_prompt_throughput_toks_per_s, avg_generation_throughput_toks_per_s. Use these to balance throughput and latency. For comprehensive monitoring, use Grafana.

Common batching configuration mistakes:

• Too large max-num-seqs: increases p99 latency due to KV cache memory contention.
• Ignoring chunked prefill: long prompts block decode, lowering utilization.
• No real-load benchmarking: parameters tuned on synthetic data often fail in production.

Configuring Dynamic Batching in TensorRT-LLM / Triton

# tensorrt_llm/config.pbtxt
parameters {
  key: "max_tokens_in_paged_kv_cache"
  value: { string_value: "40000" }
}
parameters {
  key: "batch_scheduler_policy"
  value: { string_value: "guaranteed_no_evict" }
}
parameters {
  key: "executor_static_batch_size"
  value: { string_value: "-1" }
}

Manual Batching Implementation (Example: DynamicBatchInferenceServer)

If using a custom inference server:

import asyncio
from dataclasses import dataclass
from collections import deque
import time

@dataclass
class PendingRequest:
    id: str
    prompt: str
    max_tokens: int
    future: asyncio.Future
    enqueued_at: float

class DynamicBatchInferenceServer:
    def __init__(
        self,
        model,
        max_batch_size: int = 64,
        max_wait_ms: float = 20.0,
        max_tokens_per_batch: int = 16384
    ):
        self.model = model
        self.max_batch_size = max_batch_size
        self.max_wait_ms = max_wait_ms
        self.max_tokens_per_batch = max_tokens_per_batch
        self.queue: deque[PendingRequest] = deque()
        self.lock = asyncio.Lock()
        self._batch_worker_task = None

    async def start(self):
        self._batch_worker_task = asyncio.create_task(self._batch_worker())

    async def predict(self, prompt: str, max_tokens: int = 512) -> str:
        future = asyncio.get_event_loop().create_future()
        request = PendingRequest(
            id=str(time.time()),
            prompt=prompt,
            max_tokens=max_tokens,
            future=future,
            enqueued_at=time.time()
        )
        async with self.lock:
            self.queue.append(request)
        return await future

    async def _batch_worker(self):
        while True:
            await asyncio.sleep(self.max_wait_ms / 1000)
            async with self.lock:
                if not self.queue:
                    continue
                batch: list[PendingRequest] = []
                total_tokens = 0
                while (self.queue
                       and len(batch) < self.max_batch_size
                       and total_tokens + self.queue[0].max_tokens <= self.max_tokens_per_batch):
                    req = self.queue.popleft()
                    batch.append(req)
                    total_tokens += len(req.prompt.split()) + req.max_tokens
            if not batch:
                continue
            prompts = [req.prompt for req in batch]
            max_tokens_list = [req.max_tokens for req in batch]
            try:
                outputs = self.model.generate_batch(prompts, max(max_tokens_list))
                for req, output in zip(batch, outputs):
                    if not req.future.done():
                        req.future.set_result(output)
            except Exception as e:
                for req in batch:
                    if not req.future.done():
                        req.future.set_exception(e)

Thanks to dynamic batching tuning, our clients reduce GPU infrastructure costs by a factor of 3–10, achieving project payback within 2–3 months. Savings start at 150,000 rubles per month.

What's Included in the Configuration

• Inference server configuration (vLLM, TensorRT-LLM, or custom)
• Benchmarking and batching parameter tuning
• Integration of batching monitoring metrics
• Deployment and maintenance documentation
• Team training (optional)

Estimated timeline: 2 to 10 working days, depending on complexity. Pricing is individual, determined after analysis.

Get a consultation on optimizing your LLM throughput. Contact us — we'll evaluate your project in 1 day. Order an audit of your current batching configuration — we'll identify bottlenecks and suggest improvements with cost savings estimates.