Fine-Tuning Llama: On-Premise, LoRA, QLoRA

Fine-Tuning Llama (Meta)

Imagine your company processes thousands of legal documents. You want to automate data extraction with 95% accuracy. GPT-4o does the job, but API costs grow with token volume. The solution: fine-tuning Llama 3.1 on-premise. You get weight files, deploy the model on your own infrastructure, and fine-tune without API limitations.

We are a team of AI/ML engineers with 5+ years of experience in fine-tuning open models (Transformer, GPT, LLaMA). We provide a turnkey service: from analyzing your data to deploying the model in production. Estimate the savings — self-hosted inference on Llama 3.1 8B costs 10–15 times less than an equivalent quality OpenAI API call at high loads.

Llama 3.x Model Family

For most fine-tuning tasks, Llama 3.1 8B or 70B is optimal. The former trains on a single A100 80GB, the latter requires 2–4 GPUs.

Why Choose Llama for On-Premise Deployment?

Unlike GPT-4o or Claude, you get the weight files. You can deploy the model on your own infrastructure and fine-tune without API limitations. This gives you full control over your data. According to our estimates, self-hosted inference on Llama 3.1 8B costs 10–15 times less. Compare that to an equivalent quality OpenAI API call. The difference is especially noticeable at high loads. Additionally, we have certified specialists and experience implementing in industries with strict security requirements.

Fine-Tuning Methods

Full Fine-Tuning updates all weights — maximum quality, but requires significant resources. LoRA (Low-Rank Adaptation) (Hu et al., 2021) updates only low-rank adapters on top of frozen weights. QLoRA additionally quantizes the base model to 4-bit. For 95% of tasks, LoRA or QLoRA is sufficient: they deliver quality close to full training at 5–15% of the cost.

Which target_modules to Choose for LoRA?

The target_modules parameter determines which layers receive LoRA adapters. Llama 3 architecture is a transformer with GQA (Grouped Query Attention). Typical targets:

• q_proj, k_proj, v_proj, o_proj — attention layers (minimum set)
• gate_proj, up_proj, down_proj — MLP layers (adds expressiveness)
• All 6 together — maximum quality, more adapter parameters

LoRA rank r determines the adapter size: r=8 gives ~0.1% additional parameters, r=64 gives ~0.8%. For style specialization, r=8–16 is enough; for complex knowledge extraction tasks, r=32–64.

Tech Stack: TRL + PEFT + Hugging Face

The main toolkit is the trl library paired with peft. Example QLoRA configuration:

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# QLoRA configuration
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto"
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="./llama3-finetuned",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
    ),
    train_dataset=dataset["train"],
)

trainer.train()

Practical Example: Legal Assistant

Task: fine-tune Llama 3.1 8B to analyze Russian arbitration decisions and extract structured data (parties, subject of dispute, court decision, amount).

Dataset: 3200 pairs (decision text → JSON). Data obtained from the public database kad.arbitr.ru with 20% manual annotation and synthetic labeling with GPT-4o for the rest (with manual verification of a sample).

Infrastructure: one A100 80GB, training 4 hours (3 epochs).

Results:

• F1 for claim amount extraction: 0.58 → 0.91
• Accuracy of determining claimant/defendant: 82% → 97%
• Token generation speed: 47 tok/s (vLLM, A100)
• Inference cost vs GPT-4o API: 12x lower when self-hosted

Inference of the Fine-Tuned Model

After training, the LoRA adapter can be:

1. Used separately (PEFT inference): load base model + adapter
2. Merged into one model (merge_and_unload()): simplifies deployment, removes PEFT overhead
3. Quantized after merge: GGUF via llama.cpp, AWQ via autoawq, GPTQ — to reduce VRAM requirements

# Merge adapter into base model
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./llama3-merged")
tokenizer.save_pretrained("./llama3-merged")

For production deployment, we use vLLM — it provides PagedAttention and continuous batching, increasing throughput by 2–5× compared to naive inference via transformers.

What's Included in the Work

• Data preparation (collection, cleaning, annotation, augmentation)
• Model and fine-tuning method selection
• Training and validation (metrics, tests, baseline)
• Adapter merging, quantization, and inference optimization
• Deployment on your infrastructure (vLLM, TGI, llama.cpp)
• Documentation and training for your team
• 1-month support guarantee after delivery

Timeline and Infrastructure

• Data preparation and annotation: 2–6 weeks
• Training (8B, LoRA, A100): 2–8 hours
• Training (70B, QLoRA, 2×A100): 12–48 hours
• Evaluation and iterations: 1–2 weeks
• Deployment with vLLM/TGI: 3–5 days

Total from start to production: 4–10 weeks

How to Evaluate the Fine-Tuned Model's Effectiveness?

Use metrics depending on the task: for generation — ROUGE, BLEU, F1; for QA — precision, recall; for instructions — human eval or LLM-as-judge. We include an A/B testing phase: we compare the fine-tuned model with the baseline (API or base Llama) on a representative sample. Request a consultation — we will help determine metrics for your case.

Why Choose Us

5+ years of commercial experience in NLP and Computer Vision. 30+ successful fine-tuning projects for clients in fintech, legal, and healthcare. Certified specialists in Hugging Face, PyTorch, Triton Inference Server. We work with local installations, guarantee data confidentiality, and provide full documentation.

Interested? Contact us to discuss your project and get a preliminary estimate. We will help implement Llama into your infrastructure with maximum impact.