You developed a RAG application based on LLaMA-3-8B — now you need to serve it to hundreds of users. A local RTX 4090 handles debugging, but production requires a scalable endpoint with latency p99 <500 ms and autoscaling. Azure Machine Learning Managed Online Endpoints provide this capability — but proper configuration includes VNet integration, monitoring, and asynchronous deployment. We have deployed LLMs for 20+ companies, including a large fintech with strict data privacy requirements. A typical project: choosing between Azure OpenAI and Azure ML, configuring vLLM with PagedAttention, setting up RBAC and Private Endpoints. Infrastructure cost savings with irregular loads reach 50% compared to PAYG schemes.
Azure ML endpoint documentation
Problems we solve
Cold start and autoscaling. Without scale_settings configuration, the endpoint does not scale under sudden spikes. We set TargetUtilization, polling interval, and cooldown so that scaling from 1 to 8 instances takes <2 minutes without losing requests.
GPU memory management. OOM errors are a common issue when deploying LLaMA-3-70B. We use vLLM with PagedAttention and gpu_memory_utilization=0.90, as well as Tensor Parallelism across multiple GPUs.
Monitoring and alerting. Without collecting metrics (RequestsPerMinute, Latency P50/P99, GPU Utilization), you learn about problems only from users. We configure Azure Monitor + Application Insights with alert thresholds.
How to reduce latency p99?
For latency p99 <200 ms, we use vLLM with optimizations: max_num_batched_tokens=8192, --tensor-parallel-size 4 on A100. This yields throughput of 1500 tokens/sec for LLaMA-3-8B. In Azure OpenAI with PTU, latency p99 stays around 150 ms at a fixed TPM.
Why is autoscaling important?
Without autoscaling, you overpay for idle resources or lose users during spikes. We configure scale_settings: min_instances=1, max_instances=10 with target_utilization_percentage=70. The cost of these settings is zero — savings with irregular load reach up to 50%.
What is included in the work
- Requirements audit: load, latency SLA, compliance.
- Architecture design: service selection, region, GPU type (A100, V100), network isolation.
- Implementation: writing scoring script (vLLM or custom), configuring deployment configurations, CI/CD scripts.
- Load testing: measuring latency, throughput, identifying bottlenecks.
- Documentation: architecture description, operational instructions.
- Team training: workshop on monitoring and scaling.
- Support: one month after deployment.
How to choose: Azure OpenAI vs Azure ML Endpoints?
| Criterion | Azure OpenAI Service | Azure ML Managed Endpoints |
|---|---|---|
| Available models | GPT-4, GPT-4o, GPT-3.5-turbo, Embeddings | Any open-source models (LLaMA, Mistral, Qwen) |
| Management | Fully managed — only API key | Custom scoring script, environment configuration |
| Performance | PTU for fixed TPM without throttling | vLLM + autoscaling; latency p99 <300 ms |
| Security | Azure RBAC, Private Endpoints | VNet Integration, Managed Identity, Key Vault |
| Cost | PAYG or PTU — more expensive at high volumes | Only GPU VM + storage — cheaper for batch |
For production with GPT-4, we choose Azure OpenAI (SLA, PTU). For customization and open-source — Azure ML with vLLM.
Example GPU configurations for popular models
| Model | GPU | vLLM Parameters | Expected latency p99 |
|---|---|---|---|
| LLaMA-3-8B | 1x A100 (80GB) | tensor-parallel-size=1, gpu-memory-utilization=0.90 | <200 ms |
| LLaMA-3-70B | 4x A100 (80GB) | tensor-parallel-size=4, gpu-memory-utilization=0.85 | <500 ms |
| Mistral-7B | 1x A100 (80GB) | tensor-parallel-size=1, gpu-memory-utilization=0.90 | <150 ms |
Process of work
- Requirements analysis — load, latency SLA, budget, privacy requirements.
- Infrastructure design — region selection, GPU type (A100, V100), network isolation.
- Implementation — writing scoring script (vLLM or custom), configuring deployment configurations.
- Load testing — measuring latency, throughput, identifying bottlenecks.
- Deployment and monitoring — endpoint deployment, dashboard and alert configuration.
Timeline: from 2 to 4 weeks depending on complexity. Cost is calculated individually.
Example vLLM configuration for LLaMA-3-8B
model: meta-llama/Meta-Llama-3-8B-Instruct
tensor-parallel-size: 4
gpu-memory-utilization: 0.90
max-num-batched-tokens: 8192
Results and guarantees
- latency p99 <300 ms at batch size 1 for LLaMA-3-8B on A100.
- Autoscaling from 1 to 8 instances with custom rules.
- Savings up to 35% compared to Azure OpenAI PTU for high-load scenarios.
- 99.9% endpoint availability guarantee with proper configuration.
We guarantee delivery of all configurations, documentation, and training for your team. Support — one month after deployment.
How to order deployment?
Get a consultation: our engineers analyze your task and propose an architecture within one day. Contact us — we will deploy your LLM on Azure from scratch to production in 2–4 weeks. We hold Azure Solutions Architect certification and have 5+ years of MLOps experience.