LLM Fine-Tuning Dataset Preparation: Formats, Dedup, Checklist

How to Prepare a Dataset for Fine-Tuning LLMs: Formats, Deduplication, and Checklist

Suppose you decided to fine-tune LLaMA 3 for SQL generation. You collected 50,000 examples, but the model outputs gibberish. What's wrong? Probably the dataset: duplicates, wrong format, prompt leakage. At TrueTech, we've been preparing datasets for years and know that without a systematic approach, 80% of data is garbage. According to Hugging Face Datasets documentation, dataset quality is the main factor for fine-tuning success. Garbage in, garbage out works double for LLMs: poorly structured or irrelevant examples don't just fail to help — they actively degrade the model. We guarantee data purity — every record undergoes validation for duplicates, PII, and prompt leakage.

Problems We Solve

Typical dataset preparation errors: wrong format choice, class imbalance, duplicates (exact and near-duplicate), prompt leakage into the answer, presence of PII (names, emails, phone numbers). Uneven distribution of answer lengths leads to model bias. We see this on every second project. Our engineers use automated validators and stratified splitting to eliminate these issues. For example, excessive weight of long answers can distort the model's probability distribution.

How We Do It: Stack and a Case Study

For a fintech client, we prepared a dataset for fine-tuning LLaMA 3 to generate SQL queries. We used the ChatML format with a system prompt, added 15,000 examples with varying complexity (from SELECT to JOIN and subqueries). Each example was checked with semantic deduplication at a threshold of 0.93 — we removed 12% near-duplicates. The client's budget savings were 30% (about $4,500).

Dataset Formats for Fine-Tuning

Instruction following (Alpaca format):

{"instruction": "Translate to English", "input": "Hello world", "output": "Hello world"}
{"instruction": "Write SQL query", "input": "Select all users older than 30", "output": "SELECT * FROM users WHERE age > 30;"}

Chat format (ShareGPT/ChatML):

{
  "conversations": [
    {"from": "system", "value": "You are an SQL assistant"},
    {"from": "human", "value": "How to select unique values?"},
    {"from": "gpt", "value": "Use SELECT DISTINCT: `SELECT DISTINCT column FROM table;`"}
  ]
}

Dataset format comparison

How to Properly Structure a Dataset for Fine-Tuning?

Each example must contain a clear instruction, context (if needed), and an ideal answer. Output length should be between 10 and 2000 tokens. The answer must not contain fragments of the instruction or outdated data. We use the FineTuningExample class with field validation:

class FineTuningExample:
    instruction: str    # Clear task without ambiguity
    input: str          # Specific context/data (optional)
    output: str         # Ideal model answer

    def validate(self) -> list[str]:
        issues = []
        if len(self.output) < 10:
            issues.append("Output too short")
        if len(self.output) > 2000:
            issues.append("Output may be too long for this task")
        if self.output in ["I don't know", "N/A", ""]:
            issues.append("Uninformative output")
        if self.instruction.lower()[:20] in self.output.lower():
            issues.append("Output contains instruction text")
        return issues

Dataset Volume Requirements

Why Deduplication Matters?

Duplicates (exact and near-duplicate) distort the data distribution, forcing the model to "memorize" the same patterns. This reduces generalization ability and increases overfitting risk. We apply two levels of deduplication:

import hashlib
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

def deduplicate_exact(examples: list) -> list:
    seen = set()
    unique = []
    for ex in examples:
        h = hashlib.md5(f"{ex.instruction}{ex.input}".encode()).hexdigest()
        if h not in seen:
            seen.add(h)
            unique.append(ex)
    return unique

def deduplicate_semantic(examples: list, threshold: float = 0.95) -> list:
    model = SentenceTransformer('all-MiniLM-L6-v2')
    texts = [f"{e.instruction} {e.input}" for e in examples]
    embeddings = model.encode(texts, batch_size=512, show_progress_bar=True)
    keep = [True] * len(examples)
    for i in range(len(examples)):
        if not keep[i]:
            continue
        for j in range(i+1, len(examples)):
            sim = cosine_similarity([embeddings[i]], [embeddings[j]])[0][0]
            if sim > threshold:
                keep[j] = False
    return [ex for ex, k in zip(examples, keep) if k]

Train/Eval Split

Stratified splitting by output length is mandatory. Group examples into short (<200 tokens), medium (200–500), and long (>500). Within each group, random split 90/10. The eval set must not overlap with train either exactly or semantically.

Final Checklist Before Training

• No duplicates (exact and near-duplicate)
• No PII in the dataset (names, emails, phones)
• Output does not contain dates/version references
• Balanced distribution across task types
• Eval set does not overlap with train
• Tokenized examples do not exceed model's max_length

What's Included (Deliverables)

• Analysis of source data and problem identification
• Structuring into selected format (Alpaca/ChatML)
• Validation of each example (PII, leakage, length)
• Exact and semantic deduplication
• Train/eval split preserving distribution
• Dataset documentation (statistics, field descriptions)
• Support during model training

Process Overview

1. Analytics — study the task and source data.
2. Design — choose format and validation schema.
3. Implementation — write scripts for structuring and cleaning.
4. Testing — check quality on a pilot dataset.
5. Delivery — hand over the final dataset with documentation.

Timelines and Pricing

Timelines: from 3 to 14 working days depending on volume and complexity. Pricing is calculated individually — we'll evaluate your project within 2 days. Contact us to get a consultation on your dataset. Order dataset preparation today. Our engineers have 5+ years of experience in LLM fine-tuning; we guarantee clean, balanced data.