ML Model Conversion to ONNX for Deployment Optimization

Problem: Model works only in the framework environment

You trained a BERT model on PyTorch with excellent accuracy. But production requires TensorRT on GPU, mobile app — CoreML, and inference on CPU without Python. Developers ask for a universal build. This situation is familiar to many engineers. We solve it by converting to ONNX (Open Neural Network Exchange) — an open format that runs on any hardware via ONNX Runtime, TensorRT, OpenVINO. One conversion — deploy on CPU, GPU, NPU, mobile devices, web. No framework lock-in.

Our experience from over 50 projects confirms: ONNX reduces deployment time by 40%, and GPU costs decrease by 30% through graph optimization. Comparison: ONNX Runtime inference on GPU is 2–3x faster than original PyTorch, and INT8 quantization gives another 2x without accuracy loss.

Contact our engineers for a consultation to analyze your model — we will select the optimal work plan.

How to convert a model from HuggingFace to ONNX?

Step-by-step instructions for quick conversion using the Optimum library:

1. Install packages and export:

pip install optimum[onnxruntime]
optimum-cli export onnx \
  --model bert-base-uncased \
  --task text-classification \
  --opset 17 \
  --device cuda \
  --fp16 \
  ./bert-onnx/

The --fp16 flag enables half-precision to speed up inference.

2. Or use Python API for automatic conversion:

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model = ORTModelForSequenceClassification.from_pretrained(
    "cardiffnlp/twitter-roberta-base-sentiment",
    export=True,
    provider="CUDAExecutionProvider"
)
tokenizer = AutoTokenizer.from_pretrained("cardiffnlp/twitter-roberta-base-sentiment")

inputs = tokenizer("Great product!", return_tensors="pt")
outputs = model(**inputs)

Converting PyTorch model directly

Note: when you need full control over the graph, export via torch.onnx.export. Specify dynamic_axes for variable batch size.

import torch

class TextClassifier(torch.nn.Module):
    def __init__(self, vocab_size, num_classes):
        super().__init__()
        self.embedding = torch.nn.EmbeddingBag(vocab_size, 128)
        self.fc = torch.nn.Linear(128, num_classes)

    def forward(self, input_ids, offsets):
        x = self.embedding(input_ids, offsets)
        return self.fc(x)

model = TextClassifier(10000, 3)
model.eval()

dummy_input = (
    torch.randint(0, 10000, (32,)),
    torch.tensor([0, 16])
)

torch.onnx.export(
    model,
    dummy_input,
    "text_classifier.onnx",
    export_params=True,
    opset_version=17,
    do_constant_folding=True,
    input_names=["input_ids", "offsets"],
    output_names=["logits"],
    dynamic_axes={
        "input_ids": {0: "num_tokens"},
    }
)

How to speed up inference with optimization and quantization?

After conversion, the graph can be optimized: fuse operations (GELU, LayerNorm, Attention) and apply quantization. Example for BERT:

from onnxruntime.transformers import optimizer
from onnxruntime.transformers.fusion_options import FusionOptions

opt_options = FusionOptions("bert")
opt_options.enable_gelu = True
opt_options.enable_layer_norm = True
opt_options.enable_attention = True
opt_options.enable_skip_layer_norm = True

optimized_model = optimizer.optimize_model(
    "bert.onnx",
    model_type="bert",
    num_heads=12,
    hidden_size=768,
    optimization_options=opt_options,
    opt_level=2,
    use_gpu=True,
    only_onnxruntime=False
)
optimized_model.save_model_to_file("bert_optimized.onnx")

For maximum performance, use INT8 quantization. It reduces model size by 4x and speeds up inference on CPU. ONNX Runtime supports dynamic quantization without additional data, but for best accuracy it's recommended to calibrate on a representative dataset.

ONNX Runtime provides cross-platform inference with support for CUDA, DirectML, OpenVINO, and other providers. (source: ONNX Runtime GitHub)

How to verify conversion correctness?

Check numerical match with the original. Tolerance — 1e-3, otherwise look for errors.

import onnx
import onnxruntime as ort
import numpy as np

model = onnx.load("bert_optimized.onnx")
onnx.checker.check_model(model)

pt_model.eval()
with torch.no_grad():
    pt_output = pt_model(**inputs).logits.numpy()

ort_session = ort.InferenceSession("bert_optimized.onnx")
ort_output = ort_session.run(None, {k: v.numpy() for k, v in inputs.items()})[0]

np.testing.assert_allclose(pt_output, ort_output, rtol=1e-3, atol=1e-4)
print("✓ ONNX output matches PyTorch output")

Conversion challenges and solutions

• Dynamic control flow (e.g., if len(x) > 0 inside forward): replace with masking or use TorchScript.
• Custom operators without ONNX equivalent: register a custom op via torch.onnx.register_custom_op_symbolic or refactor.
• Dynamic shapes: set correct dynamic_axes or padding to fixed size.
• FP16 accuracy loss: convert to FP32, then apply INT8 quantization separately with calibration.

Handling complex cases: non-convertible models

In rare cases, a model contains operations with no direct ONNX equivalent. We use several approaches:

• Replace incompatible operations with supported alternatives.
• Export via TorchScript then convert.
• Register a custom operator with C++/CUDA implementation.

After solving the issue, we always perform full validation.

What's included in model conversion work

• Model analysis: identify incompatible operations, profiling.
• Conversion with opset, dynamic axes, optimizations configuration.
• Validation: numerical comparison, graph check, latency tests.
• Prepare scripts for reproducible build.
• Documentation: configuration, environment requirements, deployment instructions.
• Support for converted model: 30 days after handover.

When to contact us

• Model cannot be exported directly — need to bypass non-standard operators.
• Maximum inference speed is required: optimization for TensorRT or OpenVINO.
• Need CI/CD integration: automate conversion and testing.

Over 7 years of experience, 50+ successful conversions. Request an analysis — we will select the optimal work plan. Contact us for a consultation and project evaluation.