How Data Annotation Affects LLM Fine-Tuning

You launched fine-tuning for your LLM, but the quality of responses is below expectations. Most often the cause is not the architecture, but the data. 70% of fine-tuning problems stem from annotation quality, not model choice or hyperparameters. A poorly constructed dataset can cause metrics to drop by 20–30% compared to a well-annotated one. Annotation for fine-tuning is fundamentally different from classic ML annotation: here you don't assign a label, but create an ideal response for the model. The quality of annotations directly determines the quality of the trained model. Over 5 years, we have completed over 50 projects—from customer support to multi-step agents—and we guarantee a professional approach to every task.

Types of Annotation

The choice of method depends on volume, budget, and required quality. Below is a comparison of the main approaches.

Direct annotation—the annotator creates a pair (instruction, ideal response) from scratch. Highest quality, highest cost. We use it for domains where quality is critical (law, medicine).

Edit-based annotation—the annotator improves the base model's response. 2–3 times faster than from scratch. Risk: the annotator may accept a poor response with minor edits. To avoid this, we include a mandatory "rationale for changes" field.

Ranking/Preference annotation—the annotator ranks several model responses (used for RLHF and DPO). Simpler than creating from scratch, but requires a clear understanding of quality criteria. Without them, kappa between annotators rarely exceeds 0.4.

AI-assisted annotation—a strong model (GPT‑4) generates responses, humans review and correct. Optimal balance of quality and speed for most tasks. In practice, it delivers 90% of direct annotation quality at 30% of the cost.

How to Choose the Annotation Type?

If your task is to train a model to answer typical questions (support, FAQ), choose AI-assisted with final proofreading. For creative tasks (code generation, creative writing)—direct annotation. For ranking tasks (recommender systems)—ranking/preference. If budget is limited, you can combine: 20% direct for training annotators and 80% edit-based for the main volume. Get a consultation on selecting the annotation type for your project—we'll help you choose.

Annotation Guidelines

This is the key document; without it, annotators will produce inconsistent results. We include:

• quality criteria (accuracy, completeness, tone, length, structure)
• examples of good responses with analysis
• examples of poor responses with explanations of errors
• list of forbidden phrases ("I'm just a language model…", "I can't…")
• rules for handling ambiguous queries

We develop guidelines for each project and run calibration sessions before starting. Example guidelines for a Customer Support Assistant:

## Annotation Guidelines: Customer Support Assistant

### Quality criteria for a good response:
1. Accuracy: Response aligns with company policies and is factually correct
2. Completeness: Solves the user's problem without leaving open questions
3. Tone: Professional, empathetic, without apologies for non-existent issues
4. Length: Sufficient but not excessive (100–300 words optimal)
5. Structure: Paragraphs, no lists for simple answers

### What should NOT be in the response:
- "I'm just a language model..."
- "I can't..."
- Repeating the user's question
- Unnecessary apologies
- Outdated product information

### Examples of GOOD responses: [examples]
### Examples of BAD responses: [examples with explanations]

Platforms for Annotation

Label Studio (open-source):

from label_studio_sdk import Client

ls = Client(url='http://localhost:8080', api_key='...')

# Create a project for LLM annotation
project = ls.start_project(
    title='Customer Support Fine-tuning',
    label_config='''
<View>
    <Text name="instruction" value="$instruction"/>
    <TextArea name="response" toName="instruction"
              placeholder="Write ideal response..."
              rows="10" maxSubmissions="1"/>
    <Rating name="quality" toName="instruction"
            maxRating="5" icon="star" size="medium"/>
</View>
    '''
)

# Upload tasks
tasks = [{"instruction": ex.instruction, "input": ex.input}
         for ex in unannotated_examples]
project.import_tasks(tasks)

Scale AI / Appen—for large volumes with professional annotators. Significantly more expensive, but Quality Control is included. We use them for projects with volumes over 50,000 pairs.

How to Control Annotation Quality?

The primary metric is inter-annotator agreement (IAA)—the consistency between annotators. The higher it is, the more reliable the data. For quality control, we build in 10–20% overlap of tasks that are annotated independently by two specialists. We calculate Cohen's Kappa—a measure of agreement used in NLP.

from sklearn.metrics import cohen_kappa_score

def compute_iaa(annotations_a: list, annotations_b: list) -> float:
    """Cohen's Kappa for annotator consistency"""
    # For ranking tasks (1-5 rating)
    kappa = cohen_kappa_score(annotations_a, annotations_b)
    print(f"Cohen's Kappa: {kappa:.3f}")
    # < 0.4: low agreement, revise guidelines
    # 0.6-0.8: good agreement
    # > 0.8: excellent agreement
    return kappa

Low agreement (< 0.4) is a signal that the guidelines are ambiguous. We revise the document, run an additional calibration session, and re-measure IAA.

Calibration sessions before full-scale annotation: the entire team annotates 20–50 examples together, discusses discrepancies, and refines the guidelines. This critically reduces variance between annotators. Without calibration, even experienced annotators achieve kappa < 0.5 on complex tasks. After calibration—consistently 0.7+.

Why Are Calibration Sessions Important?

Because without them, the spread among annotators can nullify all data collection efforts. One calibration session reduces rework by 40% and increases final model accuracy by 15–20%. This saves up to 30% of the project budget. Optimization of annotation reduces costs by 35–40% compared to traditional methods. Order a pilot calibration—we'll show the effect on your data.

What's Included in Our Data Annotation Work?

Below are the main stages and their duration.

Timelines depend on volume and complexity: from 2 weeks for 1,000 pairs to 2–3 months for 50,000+. Cost is calculated individually. We work with both one-off projects and long-term annotation outsourcing. Get a consultation to evaluate your project—we'll assess the scope, choose the optimal annotation method, and calculate deadlines.