Fine-Tuning GPT-4 and GPT-4o for Business Tasks

Fine-Tuning GPT-4 and GPT-4o: Customizing LLMs for Business Tasks

We often encounter a situation where even the longest system prompt fails to produce a stable output format. For example, when extracting details from contracts, the base GPT-4o made errors in 29% of cases. After fine-tuning on 800 labeled examples, accuracy rose to 94%, and the prompt was reduced by a third. In this article, we break down our approach to fine-tuning GPT-4 and GPT-4o — from dataset to deployment — and when it's justified over prompt engineering. It's important to understand that fine-tuning is not the only adaptation method; sometimes a well-crafted prompt with examples is sufficient. We prepare the dataset iteratively, tune hyperparameters, and test results on a hold-out set.

When to Use Fine-Tuning vs. Prompt Engineering

Fine-tuning is justified when the output format must be rigid (JSON, table) or the domain is narrow. If a few-shot prompt works, stick with it.

How to Prepare a Quality Dataset

The key is data quality, not quantity. Common pitfalls when preparing a dataset:

• Duplicates and contradictions: The same question with different answers confuses the model. Deduplication is mandatory.
• Imbalanced answer classes: If 90% of examples are one type, the model biases toward it.
• Lack of variety: All examples written by one author in one style lead to poor generalization.

For cleaning and analysis, we use datasets (Hugging Face), pandas, and the OpenAI CLI for format validation:

openai tools fine_tunes.prepare_data -f dataset.jsonl

The number of examples depends on the task. Minimum 50–100, but for stable results, 500–2000 is better. Fewer examples cause poor generalization; more increase cost without guaranteed improvement. Maintain class balance and phrasing diversity.

Which Hyperparameters to Choose

from openai import OpenAI

client = OpenAI(api_key="...")

file = client.files.create(
    file=open("train.jsonl", "rb"),
    purpose="fine-tune"
)

job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-2024-08-06",
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 4,
        "learning_rate_multiplier": 1.8
    }
)

We recommend starting with n_epochs=3, then iteratively increasing while monitoring metrics. If overfitting occurs (train loss decreases but validation loss increases), reduce the number of epochs.

How to Evaluate Fine-Tuning Results

After the job completes, the model is available at an ID like ft:gpt-4o-2024-08-06:org-name::abc123. Evaluate using:

• Training loss / Validation loss: OpenAI provides metrics via job events. A good signal is decreasing train loss with stable val loss.
• Manual testing on a hold-out set: at least 50 examples not seen during training.
• Comparison to baseline: A/B test the base GPT-4o vs. the fine-tuned model on real queries.

A real improvement example: After fine-tuning GPT-4o on 800 legal documents (lease agreements, deeds), extraction accuracy of details into structured JSON rose from 71% to 94%, and prompt tokens were reduced by 60%.

What Tasks We Solve with Fine-Tuning

• Ticket classification (e.g., support tickets by category): 2–3 weeks from data collection to deployment. Requires 300–500 labeled examples.
• Generation in corporate style: tone, response structure, forbidden phrases. 1–2 weeks, 200–400 examples.
• Structured data extraction (NER via LLM): 3–4 weeks, 500–1500 annotated examples.
• Specialized domain (medicine, law, finance): 6–12 weeks including data collection and labeling.

For each task, we prepare a custom dataset, iteratively fine-tune, and perform A/B testing. More details can be found in OpenAI's fine-tuning documentation.

What Are the Limitations of GPT-4o Fine-Tuning?

GPT-4o fine-tuning does not provide access to model weights — you only receive a hosted endpoint. If you need on-premise or weight control, consider Llama 3, Mistral, or other open models with LoRA/QLoRA.

Also note: the fine-tuned model is more expensive to infer than the base model. At high volume, this matters, but prompt savings can offset the cost.

What Our Work Includes

• Audit existing data, define dataset requirements
• Collect, clean, label (if needed) training examples
• Iterative training with hyperparameter tuning
• Quality evaluation: automatic metrics + manual verification
• Integrate fine-tuned model into production pipeline
• Monitor quality degradation post-deployment

We guarantee the fine-tuned model will pass validation on a hold-out set. With 5 years in the market and over 20 fine-tuning projects, we can select optimal hyperparameters from the first iteration. Book a consultation — we will evaluate your dataset and determine if fine-tuning is feasible. Contact us to discuss the details.