Data Augmentation Methods for LLM Fine-Tuning

A client brought a dataset of 500 instruction–response pairs. After three epochs of fine-tuning, accuracy on the test set was 92%, but on real data it dropped to 78%. The cause: a small and homogeneous sample. The model learned patterns, not the task's essence. Data augmentation increases the number of examples while preserving labels, boosting accuracy to 90%+. In this article, we cover three augmentation methods we use in production: backtranslation, LLM-generated paraphrases, and instruction diversity expansion. We also show how to control quality using semantic similarity.

Why Standard Augmentation Fails for LLMs

In computer vision, simple transformations work: rotation, brightness change, noise. For text, such methods are useless — they break grammar or alter meaning. LLMs need semantic equivalence paired with lexical diversity. We use three approaches that preserve meaning and increase variety.

Backtranslation: Simple and Reliable

Translate to an intermediate language and back. It creates paraphrases with minimal cost.

from deep_translator import GoogleTranslator

def backtranslate(text: str, pivot_language: str = 'de') -> str:
    intermediate = GoogleTranslator(source='en', target=pivot_language).translate(text)
    back = GoogleTranslator(source=pivot_language, target='en').translate(intermediate)
    return back

# Apply to instructions only, not output
original = "How do I cancel my subscription?"
augmented = backtranslate(original)  # "How can I terminate my subscription?"

Important: we apply it only to instructions, not responses — otherwise the model learns to output paraphrases instead of precise answers. Backtranslation yields about 80% useful paraphrases, but is inferior to LLM generation (95% useful). Backtranslation saves up to 40% of budget compared to LLM generation.

LLM-Generated Paraphrases: Maximum Diversity

The most quality method — generating variants through a strong LLM (Claude, GPT-4). We specify the number of variants and ask to change wording, style, sentence structure.

from anthropic import Anthropic

client = Anthropic()

def generate_paraphrases(instruction: str, n: int = 5) -> list[str]:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"""Generate {n} diverse paraphrases of this instruction.
Keep the same meaning but vary the wording, formality level, and sentence structure.

Instruction: {instruction}

Return as JSON array of strings."""
        }]
    )
    return json.loads(response.content[0].text)

This approach yields up to 10 different phrasings for one instruction — from formal to colloquial. LLM generation outperforms backtranslation by 1.2 times in useful paraphrase share, but requires more resources and tokens.

Instruction Diversity Expansion: Different Query Types

Users formulate the same task differently. We automatically generate instruction variations: request, command, question, demand.

def expand_instruction_types(task_description: str,
                               example_output: str) -> list[dict]:
    variations = [
        f"Please {task_description.lower()}",
        f"Can you {task_description.lower()}?",
        f"I need you to {task_description.lower()}",
        f"{task_description}:",
        task_description.upper()
    ]
    return [{"instruction": var, "output": example_output}
            for var in variations]

Negation Augmentation: Safety Without Quality Loss

For borderline queries, we add examples of proper refusal. The model learns to politely decline inappropriate requests, offering an alternative.

refusal_examples = []
for ex in harmful_edge_cases:
    refusal_examples.append({
        "instruction": ex.instruction,
        "output": f"I can't help with that request as it {reason}. "
                  f"I'd be happy to help with {alternative_suggestion} instead."
    })

Step-by-Step Augmentation Process

1. Dataset analysis: assess size, diversity, instruction types.
2. Method selection: combine backtranslation and LLM generation depending on the task.
3. Generation: create paraphrases with chosen methods.
4. Filtering: check semantic similarity and discard duplicates.
5. Integration: add augmented examples to the training set.

How to Control Augmented Data Quality?

We check each augmented pair for semantic similarity to the original. We use SentenceTransformer to obtain embeddings.

from sentence_transformers import SentenceTransformer
import numpy as np

def measure_augmentation_quality(original: str, augmented: str) -> dict:
    model = SentenceTransformer('all-MiniLM-L6-v2')
    orig_emb = model.encode(original)
    aug_emb = model.encode(augmented)

    similarity = float(np.dot(orig_emb, aug_emb) /
                       (np.linalg.norm(orig_emb) * np.linalg.norm(aug_emb)))

    return {
        'semantic_similarity': similarity,
        'is_valid': 0.7 < similarity < 0.98,
        'length_ratio': len(augmented) / len(original),
        'unique_words': len(set(augmented.split()) - set(original.split()))
    }

The optimal range for similarity is 0.75–0.95. If value > 0.98 — near duplicate; if < 0.7 — meaning distorted. Such examples are discarded. Quality augmentation reduces retraining costs by 50%.

Comparison of Augmentation Methods

What Is the Optimal Augmentation Volume?

We recommend expanding the dataset 2–3 times, maintaining an original/augmented ratio of 1:2. The share of augmented examples should not exceed 70% — otherwise the model overfits to artificial patterns. In our projects, accuracy on production queries improves by 8–15% after adding 1000–3000 augmented pairs.

What's Included in Our Work

We have been performing data augmentation for over 5 years and have completed more than 20 projects for NLP tasks. With our service you receive:

• Augmentation pipeline code in Python (ready for integration).
• Documentation of methods and configurations.
• Labeled dataset in the required format (JSON, Parquet).
• Quality metrics report: similarity distribution, rejection rate.
• Consultation on choosing an augmentation strategy for your task.

Timelines — from 3 to 10 business days. Cost is calculated individually. To get started, just send a sample dataset and task description — we will evaluate the project and propose a solution.

Contact us to discuss augmentation for your fine-tuning. Order end-to-end data augmentation — get a ready pipeline and an improved dataset.