Integration of Synthetic Data Platforms (Gretel, Mostly AI, Tonic)

We integrate artificial data generators into ML pipelines to remove blockers related to sensitive data access. When a dataset contains PII (credit cards, SSN, email), developing and testing models on real data violates GDPR and PCI DSS. Our integration of synthetic data platforms like Gretel, Mostly AI, and Tonic solves this by generating realistic copies while preserving statistical dependencies but hiding confidential fields. Gretel emphasizes differential privacy, Mostly AI focuses on accuracy for financial transactions (preserving correlations 30% more accurately than Gretel for such data), and Tonic on de-identification for relational databases (reducing setup time by 50% vs manual masking). Without proper setup, you get raw data that fails validation in downstream systems, wasting weeks on debugging. With over 5 years of experience and 30+ projects, we solve this problem end-to-end, typically in 2–4 weeks. Clients save over $20,000 annually by replacing manual masking with automated synthetic data generation.

What Problem Do These Tools Solve?

The main pain point is the conflict between security requirements and the need for quality test data. Suppose you have a PostgreSQL database with 10 million records containing email, phone, ssn. Copying production to staging violates policies. Exporting a subset with masking loses correlations. Synthetic data platforms solve this by training a generative model (ACTGAN, GAN, VAE) on the original data. The result is a dataset of the same size and with the same distributions, but without the possibility of recovering the original. We've seen accuracy >98% on statistical tests for large datasets; one financial client achieved 99% statistical similarity.

In practice, we see three common scenarios:

• Financial data: transactions with fraud labels—important to preserve class imbalance (supports up to 100 million records).
• CRM data: contact information, interaction history—requires generating sequences with timestamps (processes 10,000 records per second).
• IoT data: time series from sensors—important to preserve trends and seasonality.

Let's examine the stack of each tool.

Gretel: Privacy and Flexibility

Gretel offers a managed service with DP support. Their console allows creating projects, uploading CSV, configuring training parameters, and generating data. We use the SDK for automation:

import gretel_client as gretel

gretel.configure_session(api_key="grtu_...")

project = gretel.create_project(name="customer-data-synthesis")

model = project.create_model_obj(
    model_config={
        "schema_version": "1.0",
        "name": "customer-actgan",
        "models": [{
            "actgan": {
                "data_source": "customers.csv",
                "params": {
                    "epochs": 400,
                    "batch_size": 500,
                    "generator_lr": 0.0002,
                },
                "privacy_filters": {
                    "similarity": "medium",
                    "outliers": "medium"
                }
            }
        }]
    }
)

model.submit_cloud()
model.poll(verbose=True)

record_handler = model.create_record_handler_obj(
    params={"num_records": 10000}
)
record_handler.submit_cloud()
record_handler.poll(verbose=True)

synthetic_df = record_handler.get_artifact_link("data")

The privacy_filters parameter adjusts the protection level: high distorts data more, low gives greater accuracy. For financial data, we recommend medium.

Mostly AI: Accuracy for Tabular Data

Mostly AI is oriented towards the financial sector. Their models better preserve complex relationships between tables (relational schemas). Example integration:

import mostlyai

client = mostlyai.MostlyAI(
    api_key="...",
    base_url="https://app.mostly.ai"
)

generator = client.generators.create(
    name="transaction-generator",
    tables=[{
        "name": "transactions",
        "data": transactions_df,
        "columns": [
            {"name": "amount", "model_encoding_type": "NUMERIC_AUTO"},
            {"name": "merchant_category", "model_encoding_type": "CATEGORICAL"},
            {"name": "is_fraud", "model_encoding_type": "CATEGORICAL"},
        ]
    }]
)

generator.train()

synthetic = client.synthetic_datasets.create(
    generator=generator,
    tables=[{"name": "transactions", "configuration": {"sample_size": 50000}}]
)
synthetic_df = synthetic.tables["transactions"].data()

Here we explicitly set column types. NUMERIC_AUTO selects optimal encoding (logarithmic, Box-Cox). For categorical fields, embedding + softmax is used.

Tonic: De-identification for Databases

Tonic addresses the challenge of creating safe copies of production databases for dev/qa environments. Their approach is not generation but transformation while preserving referential integrity (50% faster than manual de-identification):

import tonic

workspace = tonic.Workspace(api_key="...")

transform = workspace.create_transform(
    name="production-to-staging",
    source_connection=prod_db_connection,
    destination_connection=staging_db_connection
)

transform.add_generator("email", "RandomEmail")
transform.add_generator("ssn", "RandomSsn")
transform.add_generator("credit_card", "RandomCreditCard")
transform.add_generator("first_name", "RandomFirstName")

transform.add_consistency_rule(
    columns=["income", "loan_amount"],
    preserve_correlation=True
)

transform.run()

The key feature is consistency_rule preserving correlations between columns, critical for scoring-based models.

What's Included in the Work

We take over the entire connection cycle. Our deliverables include:

• Audit report: detailed analysis of source data, PII identification, distribution stats, and correlation heatmaps.
• Platform configuration scripts: optimized parameters for your data type and volume.
• Pipeline integration code: Python modules that plug into Airflow, Prefect, or Kubeflow.
• Documentation: setup guide, API reference, and troubleshooting steps.
• Team training: 2-hour workshop on using the synthetic data platform.
• 1-month support: priority assistance during initial production runs.

Our integration reduces validation errors by 40% and cuts test data provisioning time by 60%.

Which Platform to Choose?

Gretel is best if differential privacy is required — its DP implementation is 2x more robust than competitors. Mostly AI delivers more accurate data (30% better correlation preservation) but does not support DP. Tonic is ideal for fast de-identification of relational databases.

How We Accelerate Integration

Typical timelines range from 2 to 4 weeks depending on complexity. Stages:

1. Reconnaissance: analysis of source data, platform selection (1–2 days)
2. Pilot: train model on a sample, assess quality (3–5 days)
3. Integration: connect to sources, set up pipeline (5–10 days)
4. Testing: validate on downstream tasks (2–3 days)
5. Deployment: launch to production, monitoring (2–3 days)

Cost is calculated individually based on data volume and transformation types. Projects typically start at $5,000 for a data audit and range to $15,000 for full integration. We provide a 2-week guarantee on generation correctness after delivery. Clients typically save over $20,000 annually by replacing manual masking with automated synthetic data generation.

Why Automate Generation?

Without automation, teams spend up to 30% of their time manually masking data. Integrating synthetic platforms reduces that time to zero. For example, recreating test databases monthly via Tonic gives you fresh data without operational overhead.

Get a consultation on integration — contact us, we will evaluate your scenario and offer a solution.