Data Cleaning for Fine-Tuning LLMs: Pipeline and Metrics

Data Cleaning Pipeline for Fine-Tuning LLMs

Imagine you've collected 100,000 examples for fine-tuning LLaMA 3, but the model produces incoherent responses and hallucinates on every third query. The cause is dirty data: 40% duplicates, 15% contain personal data, another 10% toxic content. Without quality cleaning, fine-tuning won't yield the desired result.

We've developed a pipeline that transforms a raw dataset into clean, training-ready data in 10–14 days. MinHash LSH for deduplication works 10 times faster than pairwise comparison when searching for near-duplicates on datasets of 50,000 examples. And toxicity filtering via Detoxify reduces the probability of undesirable model responses by 25% compared to simple regex.

Why Standard Cleaning Doesn't Work for LLMs?

Texts for fine-tuning contain specific artifacts: HTML tags (if scraped from the web), Unicode variations, meta-comments like "As an AI language model...". Simply removing punctuation doesn't solve the problem. Multi-layer filtering with context awareness is needed. For example, PII detection requires not only regex but also NER models (spaCy) to find "John Doe, Lenin St." — as important as card numbers. Before running the pipeline, it's recommended to review best practices from Hugging Face Datasets documentation.

How We Build the Cleaning Pipeline

The pipeline consists of sequential stages, each checking and transforming the sample. Critical not to overdo: excessive cleaning reduces data diversity.

import re
import unicodedata
from dataclasses import dataclass

@dataclass
class CleaningResult:
    original: str
    cleaned: str
    removed: bool
    removal_reason: str = None

class TextCleaner:
    def clean(self, text: str) -> CleaningResult:
        cleaned = text

        # 1. Unicode normalization
        cleaned = unicodedata.normalize('NFKC', cleaned)

        # 2. Remove HTML/XML tags
        cleaned = re.sub(r'<[^>]+>', ' ', cleaned)

        # 3. Clean URLs (optional — replace with placeholder)
        cleaned = re.sub(
            r'https?://[^\s]+', '[URL]', cleaned
        )

        # 4. Normalize whitespace
        cleaned = re.sub(r'\s+', ' ', cleaned).strip()

        # 5. Remove repeated characters (ааааааа → а)
        cleaned = re.sub(r'(.)\1{4,}', r'\1\1', cleaned)

        # Check minimum length
        if len(cleaned.split()) < 3:
            return CleaningResult(text, cleaned, True, "too_short")

        return CleaningResult(text, cleaned, False)

class DataFilter:
    def __init__(self):
        # Toxicity (can use detoxify or fasttext)
        from detoxify import Detoxify
        self.toxicity_model = Detoxify('multilingual')

    def is_toxic(self, text: str, threshold: float = 0.7) -> bool:
        result = self.toxicity_model.predict(text)
        return result['toxicity'] > threshold

    def has_pii(self, text: str) -> bool:
        """Simple heuristic for PII detection"""
        patterns = [
            r'\b\d{3}-\d{2}-\d{4}\b',           # SSN
            r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',  # Email
            r'\b(?:\+7|8)?[\s-]?\(?\d{3}\)?[\s-]?\d{3}[\s-]?\d{2}[\s-]?\d{2}\b',  # RU phone
            r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b',  # Credit card
        ]
        for pattern in patterns:
            if re.search(pattern, text):
                return True
        return False

Step-by-Step Pipeline Configuration

1. Define filtering thresholds. For toxicity, use threshold 0.7 — balances removing bad content while preserving useful. For duplicates, set similarity 0.8.
2. Choose deduplication algorithm. Exact duplicates: exact matching; near-duplicates: MinHash LSH. SimHash suits streaming but gives more false positives.
3. Run a test on 1000 samples. Check metrics: number removed, type-token ratio, residual toxicity. If OK, run full dataset.

Cleaning Output Fields

Assistant model responses often contain unwanted introductions: "Certainly! Here is my response". The algorithm detects these patterns and trims them, leaving only useful content.

class OutputCleaner:
    def clean_output(self, output: str, task_type: str) -> tuple[str, bool]:
        cleaned = output.strip()

        # Remove unwanted model phrases
        unwanted_starts = [
            "As an AI language model",
            "As a helpful assistant",
            "I don't have access to real-time",
            "I cannot browse the internet",
            "Certainly! Here",
            "Of course! I'd be happy to",
        ]

        for phrase in unwanted_starts:
            if cleaned.lower().startswith(phrase.lower()):
                # Remove introductory phrase
                cleaned = cleaned[len(phrase):].lstrip('.,! ')

        # Check: output should not contain meta-comments
        meta_indicators = [
            "Note: This is a fictional",
            "[This response was",
            "Disclaimer:",
        ]
        for indicator in meta_indicators:
            if indicator in cleaned:
                idx = cleaned.find(indicator)
                cleaned = cleaned[:idx].strip()

        # Minimum length
        if len(cleaned.split()) < 5:
            return cleaned, True  # Mark for removal

        return cleaned, False

Duplicate Detection at Different Levels

For exact duplicates, we use hashing; for near-duplicates — MinHash LSH. A similarity threshold of 0.8 removes almost identical examples while preserving variability.

from datasketch import MinHash, MinHashLSH

def find_near_duplicates(texts: list[str],
                          threshold: float = 0.8) -> list[tuple]:
    """MinHash LSH for efficient near-duplicate search O(n log n)"""
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    minhashes = {}

    for i, text in enumerate(texts):
        m = MinHash(num_perm=128)
        for word in text.lower().split():
            m.update(word.encode('utf8'))
        lsh.insert(f"doc_{i}", m)
        minhashes[f"doc_{i}"] = m

    duplicates = []
    for i, text in enumerate(texts):
        key = f"doc_{i}"
        result = lsh.query(minhashes[key])
        result.remove(key)
        if result:
            duplicates.append((i, [int(r.split('_')[1]) for r in result]))

    return duplicates

Comparison of Deduplication Methods

Post-Cleaning Statistics

After the pipeline, we always check metrics:

Typical result: from 50,000 raw examples, 35,000–42,000 high-quality ones remain. A 15–30% volume reduction is normal, and the final model quality only improves. Compared to rough cleaning (only regex), fine-tuning accuracy increases by 15–20%. A common issue is class imbalance: if 90% of examples are positive, the model won't learn to handle negative queries. We apply stratified sampling and augmentation of rare classes. Also important to remove LLM-specific stop words: 'As a language model', 'I cannot', 'I think'. This reduces noise by 5–10%.

What's Included in the Work

We prepare a full cleaning pipeline for your dataset:

• Raw data analysis (length distribution, language, toxicity)
• Filter configuration tailored to your task (RAG, generation, classification)
• Deduplication and PII removal
• Normalization and tokenization
• Report with metrics and visualizations
• Pipeline documentation and configuration
• Training for your team

Timeline — from 10 to 14 business days depending on volume. Contact us to evaluate your project — we guarantee confidentiality and result quality. Our experience: over 5 years in NLP, over 20 projects in fine-tuning models of various sizes. Get a consultation on dataset cleaning — we will prepare a custom pipeline.