Synthetic Test Data Generation: Approaches and Implementation

Imagine: your ML pipeline crashes in production because synthetic test data didn't cover distribution drifts. Or your QA team spends weeks manually preparing datasets. We build generators for synthetic data that automatically cover 95% of edge cases and reduce testing time by 3-5x. In one fintech project with 500+ API endpoints, we cut regression from 3 days to 6 hours. QA resource savings reached 40%. For a fintech platform with monthly transactions worth millions of rubles, we generated a dataset that uncovered 12 hidden bugs before release. Wikipedia

Synthetic data purposefully tests boundary cases, anomalies, and rare events—what's impossible to obtain from anonymized datasets. We guarantee 95% coverage of agreed-upon scenarios, and testing time is reduced by 3–5x.

Why Synthetic Test Data Is Better Than Anonymized Data

Anonymized data contains legacy anomalies, sampling biases, and incomplete coverage. Synthetic data, on the other hand, purposefully tests conditions: boundary values, missing fields, injections, rare events.

What Generation Strategies Do We Use?

We apply three approaches: rule-based, LLM generation, and ML-based. Their comparison:

Rule-based Generation with Faker

Rule-based generation—explicit description of rules for structured data. It works fast and gives full control. Faker is a library for generating fake data.

from faker import Faker
from dataclasses import dataclass
import random
import uuid

fake = Faker('ru_RU')

@dataclass
class TestUser:
    user_id: str
    email: str
    age: int
    balance: float
    subscription_tier: str

class TestDataFactory:
    def create_valid_user(self) -> TestUser:
        return TestUser(
            user_id=str(uuid.uuid4()),
            email=fake.email(),
            age=random.randint(18, 80),
            balance=round(random.uniform(0, 100_000), 2),
            subscription_tier=random.choice(['free', 'basic', 'premium'])
        )

    def create_edge_cases(self) -> list[TestUser]:
        """Edge cases for testing"""
        return [
            # Minimal age
            TestUser(str(uuid.uuid4()), fake.email(), 18, 0.0, 'free'),
            # Maximum balance
            TestUser(str(uuid.uuid4()), fake.email(), 65, 999_999.99, 'premium'),
            # Zero balance
            TestUser(str(uuid.uuid4()), fake.email(), 30, 0.0, 'premium'),
            # Special characters in email
            TestUser(str(uuid.uuid4()), "[email protected]", 25, 100.0, 'basic'),
        ]

    def create_ml_input_variants(self, n: int = 1000) -> pd.DataFrame:
        """Cover feature space for ML model testing"""
        return pd.DataFrame({
            'age': np.linspace(18, 80, n).astype(int),
            'balance': np.logspace(0, 6, n),  # Logarithmic distribution
            'days_since_last_purchase': np.concatenate([
                np.zeros(n//4),        # 0 days (just bought)
                np.ones(n//4) * 365,   # A year ago
                np.random.randint(1, 730, n//2)  # Random
            ]),
            'subscription_tier': np.random.choice(['free', 'basic', 'premium'], n)
        })

LLM Generation for Text Scenarios

LLM generation fits text data: reviews, queries, documents. Models like Claude 3.5 Sonnet and GPT-4o create diverse scenarios including sarcasm, mixed tones, and specific formats.

from anthropic import Anthropic

class TextTestDataGenerator:
    def __init__(self):
        self.client = Anthropic()

    def generate_sentiment_test_cases(self) -> list[dict]:
        prompt = """Generate 20 test cases for sentiment analysis testing.
Include:
- 5 clearly positive reviews
- 5 clearly negative reviews
- 5 ambiguous/mixed reviews
- 5 edge cases (sarcasm, neutral, very short, all caps)

Format as JSON array with fields: text, expected_sentiment, category"""

        response = self.client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=2000,
            messages=[{"role": "user", "content": prompt}]
        )
        return json.loads(response.content[0].text)

    def generate_rag_test_queries(self, knowledge_base_summary: str) -> list[dict]:
        """Generate test queries for RAG system"""
        prompt = f"""Given this knowledge base: {knowledge_base_summary}

Generate 30 test queries including:
- Direct factual questions (should return answer from KB)
- Questions outside KB scope (should return 'not found')
- Ambiguous queries (testing retrieval quality)
- Multi-hop questions requiring synthesis

Return JSON array with: query, expected_type, expected_answer_present"""

        response = self.client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=3000,
            messages=[{"role": "user", "content": prompt}]
        )
        return json.loads(response.content[0].text)

ML-based Generation for Model Testing

ML-based generation is for testing ML models themselves: concept drift, adversarial robustness, distribution shift. We create data that purposefully breaks the model to verify monitoring and anomaly detection.

class MLModelTestDataGenerator:
    def generate_distribution_shift(self, train_data: pd.DataFrame,
                                     shift_type: str) -> pd.DataFrame:
        """Generate data with intentional drift for monitoring testing"""
        if shift_type == 'covariate':
            # Shift feature distribution
            test_data = train_data.copy()
            test_data['age'] = test_data['age'] + 15  # Age shift
            return test_data

        elif shift_type == 'concept':
            # Invert dependency (for testing concept drift detection)
            test_data = train_data.copy()
            test_data['target'] = 1 - test_data['target']
            return test_data

    def generate_adversarial_examples(self, model, X: np.ndarray,
                                       epsilon: float = 0.1) -> np.ndarray:
        """FGSM adversarial examples for stress testing"""
        import torch
        X_tensor = torch.FloatTensor(X).requires_grad_(True)
        output = model(X_tensor)
        loss = output.sum()
        loss.backward()

        adversarial = X + epsilon * X_tensor.grad.sign().numpy()
        return np.clip(adversarial, X.min(), X.max())

Choosing a Strategy

For API tests and business logic, rule-based suffices. For NLP pipelines (sentiment, RAG, NER), LLMs are needed. For model monitoring testing, ML-based is best. We combine approaches and create hybrid generators covering up to 95% of edge cases. Estimate the savings on your project—contact us for a consultation.

Generator Development Process

1. Analysis — study your test scenarios, identify equivalence classes (1–2 days).
2. Design — choose strategies (rule-based, LLM, ML), write specification (2–5 days).
3. Implementation — code generators, using Faker, LangChain, PyTorch (1–3 weeks).
4. Testing — check coverage with metrics (BERTScore, coverage) (3–5 days).
5. Deployment — package into Docker, set up CI/CD invocation (2–4 days).

What's Included in Generator Development

• Analysis of your testing scenarios and compilation of edge case map
• Development of generators in Python with documentation
• CI/CD integration via Docker/CLI
• Set of usage examples and test datasets
• QA team training and 2 weeks of post-deployment support

Quality Metrics

For rule-based—coverage of specified rules (number of edge cases). For LLM—semantic accuracy (BERTScore). For ML tests—percentage of drifts found and adversarial success rate. As a result, you get a generator that automatically covers 95%+ of agreed-upon scenarios.

Timeline and Cost

Timeline: from 2 weeks for a basic rule-based generator to 2 months for a comprehensive system including ML tests. Cost is calculated individually based on the volume of scenarios and integration complexity. Typical investment for a comprehensive generator starts at $15,000 and the average payback period is 3-6 months. QA resource savings reach 40% after implementation. For one client, we reduced annual testing costs by $120,000. Our team has 7+ years of experience in test data generation and has successfully delivered over 60 projects for fintech, e-commerce, and SaaS. We have been in the market since 2017. Average payback period for a generator is 3–6 months due to reduced manual testing. Contact us for a free assessment of your project. Order generator development and get a consultation.

A properly designed test data system automatically covers 95%+ of edge cases, speeds up testing 3-5x, and lets the QA team focus on truly complex scenarios.