Problem: LLM not fitting your task?
You fine-tuned Llama 3.1 8B on general data, but it gives poor answers to questions about your legal documentation. Full fine-tuning requires 80 GB VRAM and several A100s—not everyone has those resources. Parameter-Efficient Fine-Tuning (PEFT) solves this: only 0.1–5% of parameters are updated, while the base model stays frozen. We use PEFT to adapt any LLM (from LLaMA to Mistral) to your task—classification, generation, RAG—without extra GPUs.
Our experience in LLM fine-tuning spans 5+ years, with hundreds of projects for clients in FinTech and LegalTech. We guarantee stable accuracy and optimized inference.
What problems does PEFT solve?
- Lack of GPU memory. Full fine-tuning of a 7B model requires ~56 GB VRAM with Adam. LoRA with rank 16 reduces it to ~18 GB, QLoRA (4-bit) to ~9 GB. One A100 instead of a cluster.
- Long training times. Full fine-tuning of Llama 3.1 8B takes 8 hours on 4×A100. LoRA r=16 takes 55 minutes on 1×A100—an 8–10x difference.
- Catastrophic forgetting. Full fine-tuning often degrades generalization. PEFT preserves base knowledge by only training the adapter.
Which PEFT method to choose for your task?
Selection depends on data volume, resources, and latency requirements. Here's a summary:
| Method | Trainable params | Inference overhead | When to use |
|---|---|---|---|
| LoRA | 0.1–5% | None (after merge) | Generation, classification, any data size ≥500 examples |
| QLoRA | 0.1–5% | None (after merge) | Same, but with VRAM constraints (4-bit base) |
| DoRA | 0.1–5% | None (after merge) | Improved LoRA with weight decomposition |
| AdaLoRA | 0.1–3% | None (after merge) | Automatic rank allocation, unknown layer importance |
| Prefix Tuning | <0.1% | Yes (virtual tokens) | Small data (50–200 examples), NLU tasks |
| Prompt Tuning | <0.01% | Yes | Minimal data, prompt engineering |
| IA³ | <0.01% | None (scaling) | Few-shot adaptation with extreme data scarcity |
As noted in the Hugging Face PEFT documentation, PEFT can reduce resource requirements by 90%.
LoRA: the golden standard of PEFT
LoRA (Low-Rank Adaptation) adds low-rank matrices to attention layers (r=8–16). After training, they are merged with base weights—no inference latency increase. Example configuration:
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
On a financial news sentiment classification task (1200 examples, Llama 3.1 8B), LoRA r=16 achieved accuracy 0.91 vs. 0.74 for 5-shot without fine-tuning. Full fine-tuning gave 0.93 but took 8x longer.
AdaLoRA: adaptive rank for complex cases
AdaLoRA automatically distributes the parameter budget across layers, assigning higher rank to more important ones. Useful when layer criticality is unknown.
from peft import AdaLoraConfig, get_peft_model
config = AdaLoraConfig(
init_r=12,
target_r=8,
beta1=0.85,
beta2=0.85,
deltaT=10,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
Prefix Tuning and IA³: when data is very scarce
Prefix Tuning adds learnable virtual tokens (20–100), IA³ uses scaling vectors. Both require <0.1% parameters and work with 50–200 examples. LoRA outperforms them 2–3x in accuracy when 500+ examples are available.
Example code for Prefix Tuning
from peft import PrefixTuningConfig
config = PrefixTuningConfig(
task_type="CAUSAL_LM",
num_virtual_tokens=20,
prefix_projection=True,
)
model = get_peft_model(model, config)
How we fine-tuned Llama 3.1 for financial news sentiment analysis
Task: Classify sentiment (Positive/Negative/Neutral) from 1200 texts. Base model: Llama 3.1 8B Instruct.
| Method | Parameters | VRAM (A100) | Accuracy | Training time |
|---|---|---|---|---|
| 5-shot (no FT) | 0 | 16 GB | 0.74 | — |
| IA³ | ~0.01% | 16 GB | 0.81 | 15 min |
| Prefix Tuning (20 tokens) | ~0.05% | 16 GB | 0.83 | 25 min |
| LoRA r=8 | ~0.2% | 18 GB | 0.89 | 45 min |
| LoRA r=16 | ~0.4% | 19 GB | 0.91 | 55 min |
| QLoRA r=16 (4-bit base) | ~0.4% | 9 GB | 0.90 | 70 min |
| Full FT | 100% | 4×A100 | 0.93 | 8 h |
Verdict: LoRA r=16 is the best accuracy/resource trade-off. QLoRA gives comparable accuracy with half the VRAM.
Managing multiple adapters with PEFT
The PEFT library allows loading and switching adapters in a single base model:
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
model = PeftModel.from_pretrained(base_model, "./adapter-legal", adapter_name="legal")
model.load_adapter("./adapter-finance", adapter_name="finance")
model.load_adapter("./adapter-medical", adapter_name="medical")
model.set_adapter("legal")
output_legal = model.generate(...)
This architectural pattern—one base instance, multiple specializations—reduces memory consumption by 5x.
What's included in the work
- Analysis of your task and selection of PEFT method.
- Dataset preparation and labeling (if needed).
- Training with metric monitoring (loss, accuracy, F1).
- Testing on a holdout set, comparison with baseline.
- Integration of the adapter into your pipeline (Hugging Face, SageMaker, Triton).
- Documentation and team training.
- Post-training support and fine-tuning adjustments as data evolves.
Timelines
- PEFT method selection and experiments: 3–7 days.
- Data preparation: 2–4 weeks.
- Training and method comparison: 1–2 weeks.
- Total: 3–6 weeks. We'll refine the timeline for your project during a consultation.
Contact us for a free analysis of your task—we'll select the optimal PEFT method and provide a detailed fine-tuning plan. Request a consultation to get an individualized plan.