Anomaly Detection Without Labels: Unsupervised Models

Anomalies Without Labels: Detecting Outliers Among 10 Million Records

When supervised models with 99.9% accuracy fail to find any anomalies, the issue is class imbalance. We build unsupervised pipelines using Isolation Forest, Autoencoder, and One-Class SVM that detect outliers without manual labeling. Such models train 10x faster than semi-supervised ones and require no pre-labeled data. However, choosing the right method is critical: an unsuitable algorithm can produce up to 80% false positives. That's why we start every project with a deep analysis of data structure and business context. Our MLOps experience in anomaly detection model training for unlabeled data ensures robust ML pipeline anomalies handling.

Unsupervised approaches enable detection without labeling costs: one engineer is enough for setup. In fintech, we used Isolation Forest for primary transaction screening (latency p99 < 10 ms) and Autoencoder for final verification. The detection rate increased by 40% compared to a rule-based system, and the false positive rate dropped 2x. Since anomalies make up less than 0.1% of data, accuracy is not informative — we focus on Precision/Recall/F1 and AUCPR. Clients typically achieve 30-50% reduction in incident analysis costs within the first quarter.

Choosing an Anomaly Detection Method

Autoencoder vs Isolation Forest: Choosing the Right Method

Autoencoder is better for complex, high-dimensional data where anomalies manifest in nonlinear relationships. For example, in cybersecurity attack detection based on event logs — Isolation Forest shows low accuracy, while Autoencoder trains in 1–2 hours and achieves AUCPR > 0.9. For simple tabular data with clear outliers, Isolation Forest is faster and more interpretable. We often combine them: primary screening with Isolation Forest (low latency), then verification with Autoencoder.

How We Build a Production-Ready Pipeline

In practice, we combine multiple methods to improve robustness. For example, in a fintech project with 50 million transactions per day: Isolation Forest was used for primary filtering (latency p99 < 10 ms), Autoencoder for deep inspection of suspicious records. Result: detection rate increased 40% compared to rules, false positive rate dropped 2x.

Implementation of Key Methods

Example code for Isolation Forest and Autoencoder:

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.metrics import average_precision_score

def train_isolation_forest(X, contamination=0.01):
    """Basic Isolation Forest with automatic threshold tuning"""
    model = IsolationForest(contamination=contamination, random_state=42)
    model.fit(X)
    scores = model.score_samples(X)  # lower = more anomalous
    return model, scores

# Autoencoder in PyTorch
import torch.nn as nn

class AnomalyAutoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim)
        )
    def anomaly_score(self, x, reduction='mean'):
        with torch.no_grad():
            z = self.encoder(x)
            x_rec = self.decoder(z)
            re = torch.mean((x - x_rec)**2, dim=1)
        return re

For production, we package the model into ONNX Runtime and deploy on Triton Inference Server — this ensures minimal latency and high throughput.

Evaluation and Continuous Learning

Proper metrics for imbalanced classes:

def evaluate_aupr(y_true, y_scores):
    return average_precision_score(y_true, y_scores)

Metric comparison for a dataset with 0.1% anomalies:

We use MLflow for experiment tracking: parameters (contamination, latent_dim), metrics (AUCPR, FPR), artifacts (models, PR curves). A feedback loop retrains the model every 1000 labels from engineers.

mlflow experiments create --experiment-name anomaly-detection
mlflow run . -P model=isolation_forest -P contamination=0.01

Project Workflow

1. Analytics — study data, identify seasonality and anomaly types.
2. Design — select stack and pipeline architecture.
3. Implementation — train baseline and tune hyperparameters.
4. Testing — validate on historical data with A/B test.
5. Deployment — containerization, monitoring, CI/CD.

What's Included in the Work

• Trained model with detailed operation documentation.
• REST API for inference (FastAPI or Triton).
• MLflow dashboard with experiment history and access for your team.
• Feedback loop for retraining on operator labels.
• Engineer training: 2–3 hour workshop covering model usage and maintenance.
• Technical support for 30 days after deployment, including bug fixes and troubleshooting.

Estimated Timelines and Cost

Baseline model (Isolation Forest + evaluation) — 2 to 3 weeks. Full pipeline with Autoencoder, feedback loop, and production deployment — 6 to 8 weeks. Cost is calculated individually based on data volume and required accuracy. Typical baseline solution starts from $15,000. The solution typically pays for itself within 2–3 months by reducing incident analysis time. For a medium-scale project (100k records, full pipeline), the investment is around $40,000.

We guarantee quality: all models are validated on a hold-out set. We have over 5 years of experience in MLOps and 20+ implemented anomaly detection projects in fintech, telecom, and industry. To assess your case, contact us — we'll evaluate the task in 2 days and offer a turnkey solution. Get a consultation on anomaly detection on your data.