Optimizing ML Models for Mobile: Quantization, Delegates, Deployment
Picture this: your server-side model achieves 0.99 F1, but on Android you get 200 ms latency and 500 MB RAM usage. Users complain, uninstalls rise. We see such projects every week. Our team — 5 years of experience, 70+ converted models — helps convert your model to TFLite format with tailored quantization and delegate selection. Result: inference under 10 ms, footprint under 10 MB. Cloud infrastructure costs drop by up to 60%. In a typical project, clients save $5,000–$15,000 annually. Inference cost per request reduces by up to 80%. We guarantee a measurable speedup of at least 3x on target devices. Get your project assessed in 2 days — just reach out to us.
Why Quantization Is Critical for Mobile Apps
On a server, you can afford FP32, but on device every megabyte and millisecond counts. Quantization reduces model size by 4x (INT8) and accelerates inference 2–4x even without specialized hardware. With delegates, you get another 3–10x. Without it, your app lags behind competitors in speed and power efficiency. Learn more about quantization of neural networks.
When TFLite Conversion Is Required
If your app runs on Android, iOS, or embedded Linux (Raspberry Pi, Coral), TFLite is the de facto standard. It supports hardware acceleration via NNAPI, GPU delegate, Hexagon DSP, and Core ML. Without conversion, you're stuck with CPU — losing 5–20x performance. TFLite outperforms ONNX Runtime on mobile platforms thanks to ARM-specific optimizations and Edge TPU support. According to TensorFlow Lite documentation, the GPU delegate can speed up inference up to 10x compared to CPU.
Choosing a Quantization Method
The choice depends on accuracy and speed requirements:
| Method | Weights | Activations | Accuracy Loss | Speedup (vs FP32) |
|---|---|---|---|---|
| Dynamic range | INT8 | float | Minimal | 2–3x |
| Full integer | INT8 | INT8 | 0.5–2% | 3–4x |
| Float16 | float16 | float | ~0% | ~2x (on GPU) |
| QAT | INT8 | INT8 | <0.5% | 3–4x |
QAT (Quantization-Aware Training) is the best choice for accuracy-critical tasks. We frequently use it on BERT-family models. For example, fine-tuning with QAT yields <0.3% accuracy drop at 4x speedup.
More on the representative dataset
A representative dataset must contain at least 200–500 samples that mirror production data. A poorly chosen dataset is a common cause of accuracy drops of up to 5%.GPU Delegate Benefits
The GPU delegate executes tensor operations on the GPU — delivering 3–10x speedup over CPU. On Qualcomm Snapdragon with Adreno GPU, we achieved 50 FPS on MobileNet v2. For iOS, the Core ML delegate can give up to 15x. The GPU delegate outperforms CPU by 3–10x in throughput but consumes more power — a trade-off. Inference cost per request drops 4x compared to CPU.
Step-by-Step Conversion Plan
- Profiling — measure size, latency, and power on target devices.
- Strategy selection — post-training or QAT, INT8 or float16.
- Calibration — prepare a representative dataset (at least 200–500 samples).
-
Conversion — use
tf.lite.TFLiteConverterwith optimizations. - Validation — compare accuracy and performance on real devices.
- Documentation — produce integration code and deployment recommendations.
# TF/Keras → TFLite
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT] # post-training quantization
tflite_model = converter.convert()
PyTorch → ONNX → TFLite: PyTorch has no direct path. We use torch.onnx.export → onnx-tf → TFLite. Thorough testing is critical — double conversion can introduce artifacts. For PyTorch models where accuracy is paramount, we recommend ONNX Runtime for mobile platforms.
Delegate Comparison
| Platform | Delegate | Speedup (vs CPU) | Hardware |
|---|---|---|---|
| Android GPU | GPU Delegate | 3–10x | Adreno, Mali |
| Qualcomm | NNAPI / Hexagon | 5–20x | Snapdragon (DSP) |
| iOS | Core ML Delegate | 5–15x | Apple Neural Engine |
| Edge TPU | EdgeTPU Delegate | 100x | Coral accelerator |
Typical Conversion Pitfalls
- Mismatched representative dataset — leads to accuracy drop. For instance, if the dataset uses high-quality JPEGs but production images are compressed, accuracy can drop by 3–5%.
- Skipping on-device validation: an emulator doesn't reflect real performance. CPU on an emulator is 2–3x faster than a real device.
- Ignoring power consumption when using GPU — on weak batteries, throttling reduces FPS.
Case Study: Porting YOLOv8 to Android
A client needed real-time detection on Snapdragon 8 Gen 2. We chose full integer quantization with a representative dataset of 500 images. After calibration, accuracy dropped by 1.2% — compensated with QAT. Final result: 30 FPS at 8 MB model size. The entire pipeline took 5 days.
What's Included
- Model architecture analysis and profiling (size, latency, power).
- Quantization strategy selection (post-training / QAT, INT8 / float16).
- Conversion with calibration on a representative dataset.
- Accuracy and performance testing on target devices.
- Integration documentation and deployment support.
Our team: 5 years of experience, 70+ converted models — from YOLOv8 to transformers. We hold Google ML certifications and have hands-on experience with Qualcomm NPU.
Contact us for a project assessment — we'll estimate timelines (typically 1–2 weeks) and cost based on your requirements. Request a free consultation.