Deploying LLMs on AWS: SageMaker, Bedrock, EC2
A client comes with a task: "We need a chatbot based on Llama 3—fast, cheap, and scalable." But production-grade LLM deployment is not just spinning up a Docker container. It involves IAM policies, GPU instances, auto scaling based on p99 latency, and tensor parallelism for large models. We've learned across 30+ projects and know how to make it reliable.
AWS offers three main paths: Amazon SageMaker (managed ML service), EC2 G/P instances (GPU VMs), and Amazon Bedrock (managed LLM API). The choice depends on control vs. maintenance. SageMaker is recommended for production workloads with variable traffic. Our engineers help avoid mistakes.
Why SageMaker over EC2 for production?
EC2 gives full control but requires driver updates, GPU monitoring, and manual auto scaling. SageMaker provides built-in auto scaling, A/B testing, and CloudWatch metrics—reducing deployment time by 60% compared to EC2. We recommend SageMaker for most production scenarios unless custom software is needed.
| Criteria | SageMaker | EC2 (GPU) | Bedrock |
|---|---|---|---|
| Management | Fully managed | Manual | Managed API |
| Scaling | Auto scaling out of the box | Requires manual setup | Seamless (pay per token) |
| Model control | Full (any open source) | Full | Provider models only |
| Cost | Pay per compute (~$1.50/hr for g5.xlarge) | Pay per EC2 + GPU | Pay per token (~$0.01/1k tokens) |
| Best for | Production API with variable load | Custom pipelines, batch | Prototypes, irregular load |
When should you choose Bedrock?
If you want zero maintenance and pay-per-token pricing, Bedrock eliminates 90% of operational overhead. However, models are limited to provider sets (LLaMA, Mistral, Claude). For prototypes or irregular loads, Bedrock saves time. For production with custom settings, SageMaker or EC2 is better.
How does the LLM deployment process on AWS work?
We follow a structured process from audit to handover:
- Analysis and service selection. Determine load (RPS, context window, latency SLA). Choose instance type and model. If a custom model is needed, discuss fine-tuning.
- Design architecture: VPC, IAM roles, S3 for model weights, encryption. Prepare model quantization (INT4/INT8) via bitsandbytes or AWQ.
- Implementation. Deploy through SageMaker LMI (Large Model Inference) or TGI. Configure auto scaling based on InvocationsPerInstance. Set CloudWatch alerts on p99 latency, GPU utilization, 4xx/5xx errors.
- Testing with Locust or Artillery. Verify performance under peak load, measure latency and throughput.
- Deployment and documentation. Deploy to production, write runbook (restart, update model). Train your team on basic operations.
Timelines range from 2 to 8 weeks depending on complexity (number of models, fine-tuning, API integrations). Pricing is calculated individually.
Quantization methods comparison
| Method | Compression | VRAM for 70B | Latency impact |
|---|---|---|---|
| FP16 | 1x | 140 GB | Baseline |
| INT8 (GPTQ) | 2x | ~70 GB | +5-10% |
| INT4 (AWQ) | 4x | ~35 GB | +10-20% |
Quantization saves GPU memory. Llama 3 70B without quantization cannot fit into a G5.2xlarge instance (24 GB). We always check memory_usage.
Common mistakes when deploying LLMs on AWS:
- Ignoring quantization—model doesn't fit VRAM.
- Incorrect IAM setup—SageMaker role lacks access to the model in S3.
- Missing health checks—endpoint doesn't report status via custom health_check_path.
- Mixing batch and real-time processing—use SageMaker Batch Transform for background jobs, it's cheaper.
What deliverables do you get?
After completion, you receive:
- A working SageMaker Endpoint (or EC2/Bedrock) with auto scaling.
- Operational documentation (IAM policies, endpoint parameters, update commands).
- CloudWatch dashboard with key metrics.
- Load test report (latency p50/p95/p99, throughput, GPU utilization).
- Team training (1-2 sessions of 2 hours each).
- 90-day warranty: if something breaks, we fix it for free.
Our team brings over 10 years of AWS experience, certified AI/ML engineers, and 30+ successful projects. Request a consultation for a budget estimate.