Synthetic Data Generation for LLM Fine-Tuning Turnkey

You are launching fine-tuning of an LLM, but you have only a few tens of thousands of examples. Manual labeling costs hundreds of thousands of dollars and takes months. We solve this problem by generating synthetic data: a teacher model (GPT-4, Claude) creates thousands of diverse instructions and responses, and then you fine-tune your model. This approach reduces labeling costs by up to 90% and allows you to obtain a dataset with the desired distribution in 2–4 weeks. Synthetic augmentation (LLM augmentation) is not only budget savings, but also quality control: you manage the style of responses, complexity, and domain. Unlike crowdsourcing, where annotation quality is unstable, here each example passes through an LLM-judge and selective human evaluation. The result is a dataset that improves model metrics by 15–20% without additional costs.

How Self-Instruct Scales the Dataset

The Self-Instruct method, proposed by researchers at the University of Washington, requires only 20–200 seed examples. From them, the LLM generates new instructions, then responses, and repeats the process iteratively. Over 3–5 iterations, 100 seed examples yield 2,000–5,000 pairs. We have adapted the process for Russian-language data and added a quality filter (LLM-judge) that eliminates duplicates and irrelevant examples.

from anthropic import Anthropic
import json

client = Anthropic()

SEED_EXAMPLES = [
    {"instruction": "Explain a term from ML", "output": "..."},
    {"instruction": "Write an SQL query for...", "output": "..."},
    # 20-200 seed examples
]

def generate_new_instructions(seed_examples: list, n: int = 20) -> list[str]:
    """Generate new instructions based on seed examples"""
    examples_str = "\n".join([f"- {ex['instruction']}" for ex in seed_examples[:10]])

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2000,
        messages=[{
            "role": "user",
            "content": f"""Here are some example instructions for an AI assistant:
{examples_str}

Generate {n} NEW diverse instructions in the same domain.
Requirements:
- Each instruction should be unique and not repeat the examples
- Vary complexity: some simple, some multi-step
- Include different formats: questions, commands, completions
- Return as JSON array of strings"""
        }]
    )
    return json.loads(response.content[0].text)

def generate_response(instruction: str, context: str = None) -> str:
    """Generate ideal response for instruction"""
    prompt = f"Instruction: {instruction}"
    if context:
        prompt = f"Context: {context}\n\n{prompt}"

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1000,
        system="You are an expert assistant. Provide accurate, helpful, and complete responses.",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

How Evol-Instruct Complexifies Instructions

Evol-Instruct, implemented in WizardLM, takes existing examples and applies complexity methods: adding constraints, deepening, concretizing, increasing the number of reasoning steps. For example, the simple question 'Tell me about React' transforms into 'Compare React and Vue for a large enterprise application considering SSR and code-splitting.' This improves the quality of fine-tuning, especially for reasoning tasks.

EVOLUTION_METHODS = [
    "Add constraints: add a specific constraint or requirement to the instruction",
    "Deepening: ask for more depth or detail in the response",
    "Concretizing: replace general concepts with specific examples",
    "Increased reasoning steps: require multi-step reasoning",
    "Complicate input: add more complex or ambiguous input",
]

def evolve_instruction(original: str) -> str:
    """Complexify instruction using one method"""
    method = random.choice(EVOLUTION_METHODS)

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": f"""Rewrite this instruction using this method: {method}

Original instruction: {original}

Return only the rewritten instruction, nothing else."""
        }]
    )
    return response.content[0].text.strip()

How We Evaluate Synthetic Data Quality

We have implemented two-level validation: an automatic LLM-judge (based on GPT-4) rates each example on a scale of 0–1, then randomly 10% of the sample is checked by a human expert. If the approval rate for human evaluation is below 85%, we adjust the generation. Additionally, we measure diversity (distinct n-grams) and token length to avoid monotony.

How We Generate Data for Your Domain

For domain-specific tasks, we add context: knowledge base, API specifications, corporate guidelines. For example, for fine-tuning a tech support chatbot: we upload FAQ and call logs, the LLM generates 'customer question — expert answer' pairs. The pipeline includes three steps:

1. Instruction generation — modeling possible user queries.
2. Response generation — using relevant context from your base.
3. Filtering — removing duplicates, checking length (tokens), quality assessment by LLM-judge.

In one project for a legal tech client, we generated 15,000 synthetic question-answer pairs from only 50 seed examples. The fine-tuned model improved accuracy on legal QA by 18%, reducing labeling costs by 85%.

def generate_domain_dataset(domain: str, n_examples: int,
                             output_path: str):
    """Generate dataset for a specific domain"""
    examples = []

    for i in range(n_examples):
        # Step 1: Generate diverse instruction
        instruction = generate_instruction_for_domain(domain)

        # Step 2: Generate response
        response = generate_response(instruction)

        # Step 3: Quality filter (LLM-judge)
        quality_score = judge_quality(instruction, response)

        if quality_score >= 0.7:
            examples.append({
                "instruction": instruction,
                "output": response,
                "quality_score": quality_score,
                "generated_by": "claude-3-5-sonnet-20241022"
            })

        if (i + 1) % 100 == 0:
            print(f"Generated {i+1}/{n_examples}, kept {len(examples)}")

    with open(output_path, 'w') as f:
        for ex in examples:
            f.write(json.dumps(ex, ensure_ascii=False) + '\n')

Comparison of Generation Methods

Work Process

1. Task analysis — determining the domain, dataset requirements, quality metrics.
2. Seed example preparation — collecting 50–200 representative pairs (your data can be used).
3. Synthetic dataset generation — Self-Instruct + Evol-Instruct, 5,000–50,000 pairs.
4. Filtering and validation — LLM-judge + human evaluation of 10% of the sample.
5. Documentation — model card, diversity measurements, approval rate.
6. Support — prompt adjustment if needed, one month of support.

Timelines and Cost

Timelines depend on dataset volume:

• 1,000–10,000 examples: 1 to 3 weeks
• 10,000–100,000 examples: 3 to 6 weeks

Cost is calculated individually — it includes API costs, engineer effort, and validation. On average, synthetic data costs 10–20% of manual labeling for a similar volume. We guarantee quality: approval rate at least 85% based on human evaluation.

Typical Mistakes in Synthetic Data Generation

• Overfitting to the teacher's style — the model copies GPT-4's tone instead of the target. Solution: mix with real data, add examples from your domain.
• Insufficient diversity — all instructions are similar. Solution: control topic distribution, use Evol-Instruct.
• Factual hallucinations — especially in domain-specific data. Solution: introduce context filters and manual verification.

We have 5+ years of experience in NLP and MLOps, completed over 50 projects on synthetic data generation for LLMs. Contact us for a project evaluation and an optimal strategy. Get a free consultation.