Optimizing AI/ML Containers with Docker and GPU Acceleration

GPU Containers: Only Half the Work — Why Your Docker Images Are Bloated and How to Fix It

Your team uses Docker for model training. Images weigh 10+ GB, builds take 20 minutes, and when you deploy to production you get a cryptic "CUDA driver version is insufficient". These are not inevitable problems — they stem from poor base image selection, missing multi-stage builds, and the lack of NVIDIA Container Toolkit. Below is a battle-tested approach used by our engineers with 8+ years of production experience.

Docker for AI/ML is not just code packaging. It guarantees that a model runs identically on any infrastructure: from a developer's RTX 4090 to an A100 in the cloud. Multi-stage builds are three times more efficient than single-stage builds in terms of image size and build time. In 8+ years we have deployed over 50 production pipelines and developed patterns that save hours of debugging. NVIDIA officially recommends using Container Toolkit for GPU acceleration in containers.

NVIDIA Container Toolkit: Purpose and Installation

NVIDIA Container Toolkit is a layer that passes GPU devices from the host into the container. Without it, Docker cannot see the GPUs. Installation is standard:

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list \
  | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt install nvidia-container-toolkit
sudo systemctl restart docker

docker run --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

After this, nvidia-smi inside the container shows all GPUs. It is critical to check host driver version compatibility with the CUDA version in the image.

Configuring GPU in Docker: Step-by-Step

1. Install NVIDIA driver on host (version 525+).
2. Install NVIDIA Container Toolkit as shown above.
3. Verify GPU availability: docker run --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi.
4. Use a base image with CUDA: for example nvidia/cuda:12.2.0-cudnn8-runtime-ubuntu22.04.
5. Run container with --gpus all or through docker-compose with resource reservations.

Critical Role of Multi-Stage Builds for AI

PyTorch images with the CUDA toolkit can easily weigh 10 GB. If you do not separate stages, the final image copies all header files and compilers that are not needed in production. Our engineers guarantee that using the runtime tag reduces image size by half, and multi-stage builds give up to 80% savings, cutting cloud costs by approximately $200 per month for a team of 5 developers.

Dockerfile for an ML Project: Multi-Stage Build

We use multi-stage builds so the final image is minimal. First builder installs all dependencies, then runtime copies only the result.

FROM nvidia/cuda:12.2.0-cudnn8-devel-ubuntu22.04 AS builder

RUN apt-get update && apt-get install -y python3.11 python3-pip git \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

FROM nvidia/cuda:12.2.0-cudnn8-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3.11 \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH

WORKDIR /app
COPY src/ ./src/

RUN useradd -m -u 1000 mluser
USER mluser

CMD ["python", "src/train.py"]

Docker Compose for Local Development

For development we add services: MLflow for tracking, Postgres for metadata. Example docker-compose.yml:

version: '3.8'
services:
  training:
    build: .
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ./data:/app/data:ro
      - ./src:/app/src
      - ./outputs:/app/outputs
    environment:
      - MLFLOW_TRACKING_URI=http://mlflow:5000
    depends_on:
      - mlflow

  mlflow:
    image: python:3.11-slim
    command: mlflow server --host 0.0.0.0 --port 5000
    ports:
      - "5000:5000"
    volumes:
      - mlflow-data:/mlflow

volumes:
  mlflow-data:

Troubleshooting GPU Visibility in Containers

Typical causes: missing NVIDIA Container Toolkit or driver incompatibility. Check: does nvidia-smi work on host? Is toolkit installed? Is the --gpus all flag correct? If the problem persists, check the CUDA version in the image (running nvidia-smi inside the container shows compatibility). In 90% of cases reinstalling the driver or choosing a different CUDA tag fixes it.

Optimizing Image Size for Inference

Common mistakes and fixes:

• Using devel instead of runtime — switch to runtime.
• Missing --no-cache-dir in pip — always add it.
• apt cache after installation — remove it with rm -rf /var/lib/apt/lists/*.
• Mounting the entire project instead of copying only src/ — structure properly.

Final inference image: PyTorch, ONNX Runtime, application — 2-3 GB. Compare to 8-10 GB with a naive approach — a 3-4x reduction directly reflecting storage costs and deployment speed.

What Is Included in the Work?

The delivery package includes:

• Ready-made Dockerfile and docker-compose for ML pipelines with GPU support.
• Integration of monitoring (Prometheus + Grafana) for GPU metrics collection.
• CI pipeline with layer caching for faster builds.
• Documentation (model card, run instructions) and team training session.
• 30 days of support after deployment.
• Guarantee of environment reproducibility at any stage.

Get in touch to evaluate your project — you will receive a consultation and a turnkey proposal within 3-5 days. Over 5 years on the market, 8+ years of engineer experience, 50+ implemented projects — your success is in reliable hands. Order Docker setup for your ML pipeline and let your team focus on features, not environment.