NVIDIA NCCL Configuration for Multi-GPU Training

NVIDIA NCCL Configuration for Multi-GPU Training

You launched distributed training on eight A100s, but scaling efficiency barely reaches 30%. Sound familiar? Incorrect NCCL configuration is the main cause of GPU idle time and prolonged experiments. The NVIDIA NCCL library handles gradient synchronization. We have configured NCCL for hundreds of servers, from DGX to Ethernet-based clusters. Here’s our experience extracting 85%+ efficiency.

The problem often lies in the wrong transport choice: NVLink, PCIe, InfiniBand, or Ethernet. Each has its limitations. Let's examine typical mistakes and their solutions.

What Problems Occur in Distributed Training?

Transport not selected — NCCL silently falls back to PCIe instead of NVLink. Check nvidia-smi topo -m and force NCCL_P2P_DISABLE=0. Wrong buffers — the default buffer is too small for large models. Increase NCCL_BUFFSIZE to 8MB. Multi-node without RDMA — all-reduce over Ethernet is 2-3 times slower than over InfiniBand. Solution: migrate to InfiniBand or enable RoCE. Typical symptom: 30-50% GPU utilization when training a model with billions of parameters.

How We Configure NCCL

Typical stack: PyTorch DDP + NCCL 2.19, Ubuntu 22.04, CUDA 12.1. For benchmarks we use nccl-tests from the NVIDIA repository. Example configuration for DGX A100 with 8 GPUs and InfiniBand:

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH
export NCCL_P2P_DISABLE=0
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0
export NCCL_IB_GID_INDEX=3
export NCCL_BUFFSIZE=8388608
export NCCL_NTHREADS=512

After this configuration, all-reduce throughput for 1G grew from 35 GB/s (PCIe only) to 280 GB/s (NVLink+InfiniBand). An 8x difference — that sets apart a project with idle GPUs from fully utilized ones. Such acceleration recovers InfiniBand investment within 6–12 months, reducing GPU-hour costs by 50–70%.

Transport Comparison

Recommended NCCL Variables

How to Diagnose NCCL Issues?

First, check topology: nvidia-smi topo -m. If you see NV12 — good; if PIX — NVLink is not used. Then enable NCCL_DEBUG=INFO and run all_reduce_perf -b 1G -e 4G -f 2 -g 8. Look at algbw — it should match the theoretical transport limit. If lower, there’s a configuration issue.

Why InfiniBand Is Better Than Ethernet?

For multi-node training, all-reduce is the bottleneck. InfiniBand HDR provides 200 Gbit/s throughput and low latency. Ethernet 100GbE only 100 Gbit/s, and real performance is even lower due to TCP protocol. The cost difference of a server with InfiniBand is offset by 2-3x faster training. Contact us for an infrastructure assessment.

Work Process

1. Topology analysis — determine GPU connections, NUMA nodes, network interfaces.
2. Design — select transport, buffer sizes, threads.
3. Implementation — tweak environment variables, install drivers (gdrcopy if needed).
4. Testing — run nccl-tests on all modes, compare against baseline.
5. Deployment — freeze configuration in Ansible or Dockerfile, document.

Estimated Timeline

Turnkey project — from 3 to 7 days depending on topology complexity. Cost is calculated individually after auditing your infrastructure. Request an audit of your NCCL configuration today.

What's Included

• Automatic NCCL configuration scripts for your specific setup.
• Report with benchmarks for all transport combinations.
• Documentation for diagnostics and optimization for your team.
• Support during deployment — we debug issues in real experiments.

Typical Mistakes

• Forgetting to check NUMA affinity — bind processes to the GPU’s NUMA node.
• Using torchrun without --master_port on all nodes — ports conflict.
• Not setting NCCL_IB_DISABLE=0 on nodes without InfiniBand — NCCL hangs training.
• Too small NCCL_BUFFSIZE for large gradients — forces multiple all-reduce calls.

Properly configured NCCL is the difference between 30% and 85% scaling efficiency. Our experience: over 10 years in distributed training, dozens of deployments from 4 to 256 GPUs. We guarantee performance improvement or refund for consultation.

Official NCCL Documentation

Get a consultation on configuring your infrastructure.