DeepSpeed for LLMs: ZeRO, Offloading, Multi-GPU

When training an LLM with over 10 billion parameters, you quickly run out of VRAM. Even with 8× A100 80GB GPUs, a 30B model won't fit without sharding. That's why we use DeepSpeed from Microsoft—it's the industry standard for distributed training. With ZeRO stages, we cut memory usage by 3–8×, and by offloading to CPU or NVMe, we can train models up to 175B parameters on a modest cluster. Here's how we set it up for your specific hardware and model.

Problems We Solve

VRAM Overflow. An LLM with 30B+ parameters doesn't fit even on 8× A100 80GB. ZeRO Stage 3 shards not only the optimizer state but also parameters and gradients—saving up to 3× memory.

Communication Bottlenecks. All-reduce in DDP blocks GPUs. DeepSpeed overlaps communication with computation and uses reduce-scatter instead, cutting p99 latency by 40%.

Configuration Complexity. Choosing the wrong stage or batch size leads to OOM or low GPU utilization. We tune parameters to your hardware and model.

How We Do It

We build production-ready configs tested on A100/H100 clusters. Here's a typical ZeRO Stage 2 config using BF16:

{
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": 2e8,
    "overlap_comm": true,
    "reduce_scatter": true,
    "reduce_bucket_size": 2e8,
    "contiguous_gradients": true
  },
  "fp16": {
    "enabled": true,
    "loss_scale": 0,
    "loss_scale_window": 1000,
    "initial_scale_power": 16,
    "hysteresis": 2,
    "min_loss_scale": 1
  },
  "bf16": {
    "enabled": false
  },
  "gradient_accumulation_steps": 4,
  "gradient_clipping": 1.0,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": 4,
  "wall_clock_breakdown": false
}

Integration with Hugging Face Transformers is straightforward—just add the config path to the Trainer:

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./results",
    deepspeed="ds_config_zero2.json",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    fp16=True,
    num_train_epochs=3,
    logging_steps=100,
    save_steps=1000,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)

trainer.train()

Launch with:

deeepspeed --num_gpus=8 train.py --deepspeed ds_config.json
# Or via torchrun:
torchrun --nproc_per_node=8 train.py --deepspeed ds_config.json

Real-World Performance

On a recent project, we trained LLaMA 7B using ZeRO-2 on 8× A100 80GB and achieved ~7000 tokens/s—4× faster than standard DDP. The table below summarizes throughput for various models and configs:

When Offloading Makes Sense

CPU offloading (or NVMe) adds PCIe latency, so we avoid it when VRAM is sufficient. For example, 8× A100 40GB for a 30B model—Stage 3 with CPU offload gives stable 1200 tokens/s. Without offload, the model simply won't fit. Our rule of thumb: use offload only when necessary.

Common Mistakes

• Wrong Stage Choice: Stage 3 on small models incurs all-gather overhead.
• Overly Large Micro Batch: Start with 1 and increase gradually to avoid OOM.
• Ignoring Gradient Accumulation: For batch size 256 on 8 GPUs, 32 accumulation steps are sufficient.
• No Communication Overlap: Disabling overlap_comm hurts GPU utilization. We always enable it.

Our Evaluation and Implementation Process

1. Analysis: We assess your model, hardware, and performance requirements.
2. Design: We select the ZeRO stage, batch size, gradient accumulation steps, and offloading strategy.
3. Implementation: We write the config, integrate it with your code (Hugging Face, PyTorch Lightning, etc.), and test on a small dataset.
4. Testing: We measure throughput and memory usage, then fine-tune parameters.
5. Deployment: We deploy on your cluster and provide documentation for monitoring and scaling.

For a preliminary memory estimate, we use DeepSpeed's built-in tool:

from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-13b-hf")
estimate_zero3_model_states_mem_needs_all_live(
    model,
    num_gpus_per_node=8,
    num_nodes=1
)

What's Included and Timeline

Our configuration service includes:

• A production-ready DeepSpeed config tailored to your model and hardware
• Integration code with Hugging Face Transformers or your chosen framework
• A report with throughput, memory metrics, and scaling recommendations
• 30-day post-deployment consultation
• Runbook for launching and monitoring

Timeline: 3 to 10 business days depending on complexity. Cost is determined after analysis—our projects typically pay for themselves by reducing training time by 30-50%.

We have completed over 50 LLM training and fine-tuning projects, from GPT-2 to LLaMA 65B, on clusters ranging from 4 to 64 GPUs. Contact us for a free evaluation of your project—we'll help select the optimal configuration and accelerate your training without unnecessary expense.