We develop custom recommendation models that work in real conditions, not on overfitted metrics. Our training recommendation models use Two-Tower, matrix factorization, negative sampling, and temporal split. A typical case: an e-commerce client used random split, got NDCG@10 = 0.65, but after deployment the model performed at 0.4. After switching to temporal split, offline metrics dropped, but the A/B test showed a +5% revenue lift. The client noted: "Random split gave inflated metrics — temporal split turned out to be the only honest one." Our stack: PyTorch, Hugging Face Transformers, Faiss for ANN search, ONNX Runtime for inference. The team has 5+ years in ML and 50+ deployed recommender systems in e-commerce, media, and fintech. We use Weights & Biases for experiment tracking and MLflow for model management. Order a preliminary data analysis — we will evaluate your task and propose the optimal solution.
Problems we solve
Incorrect negative example generation
If you take uniform sampling from all items, the model won't learn to distinguish popular from relevant. We use popularity-based and hard negative mining (e.g., items the model already ranks high incorrectly). Our Two-Tower model outperforms ALS by 1.2x in NDCG@10, and hard negative sampling outperforms uniform by 1.3x in Recall@20.
Cold start for new users and items
Without content features, the model outputs zero predictions. We add side information (category, text description, images) via embeddings.
Temporal bias
Real scenarios are sequences of actions. If you shuffle data randomly, the model "sees" the future during training. Temporal split with time-based validation/test sets is mandatory.
How we do it: a detailed case
Among our projects is an online store with 5M interactions and 200K items. We built a Two-Tower model with HNSW index for retrieval. The pipeline included:
- log preprocessing: deduplication, bot filtering, interaction weighting (purchase > view);
- negative sampling ratio 4:1 with popularity weighting; temporal split: 28 days train, 7 val, 7 test;
- Two-Tower: user tower — embedding layer + Dense(256), item tower — same + L2 normalization. Loss — Weighted BCE with log-weight for positive examples;
- training 20 epochs with early stopping on NDCG@10, using AdamW (LR=1e-3). On GPU A100 — 45 minutes;
- deployment via Triton Inference Server with ONNX model. p99 latency — 12 ms.
Result: NDCG@10 lift of 22% over ALS baseline, online A/B test showed +8% in cart additions.
Why proper negative sampling matters
Negative examples define the decision boundary. If all negatives are random tail items, the model easily distinguishes them. But in practice, you need to discern nearly relevant items — e.g., a dark sofa vs a black sofa. Hard negative sampling (from top-100 non-interacted items) yields more robust training. We implement this via an embedding cache and online sampling.
What temporal splitting provides
Evaluating a model on a temporal split is the only way to get honest metrics. Random split can "peek" into the future, inflating results. NDCG@10 on temporal split correlates with online A/B test results: deviation <15%.
Model architecture comparison
| Model | Quality (NDCG@10) | Training time (5M) | Inference latency | Notes |
|---|---|---|---|---|
| ALS (matrix factorization) | 0.42 | 30 min (CPU) | <1 ms | Simple baseline, no side info |
| Two-Tower (Dense 256) | 0.53 | 45 min (A100) | 3–12 ms | Flexible, cold start via features |
| BERT4Rec (transformer) | 0.58 | 4 hrs (A100) | 50 ms | Sequences only, no cold start |
Compared to ALS, Two-Tower gives a 1.2x boost in NDCG@10 with comparable inference time.
Negative sampling strategy comparison
| Strategy | Quality (Recall@20) | Training speed | Implementation complexity |
|---|---|---|---|
| Uniform | 0.48 | High | Low |
| Popularity-based | 0.55 | Medium | Medium |
| Hard negative (online) | 0.61 | Low (due to embedding recomputation) | High |
In practice, we combine popularity-based and hard negatives: 80% popular, 20% hard negatives. This balances quality and speed.
Our process
1. Analytics: study interaction logs, identify bottlenecks (sparsity, cold start, imbalance). 2. Design: choose architecture, loss function, negative sampling strategy. 3. Implementation: write pipeline in PyTorch + Hugging Face, integrate with your logging system. 4. Testing: offline metrics (NDCG, recall@k) + online A/B test for 2–4 weeks. 5. Deployment: containerization, monitoring (data drift, latency), documentation.Timelines and cost
Timelines: from 2 weeks (baseline + modifications) to 2 months (full pipeline with custom architecture). Cost is individual after data analysis. Our clients report up to 40% savings on infrastructure and 30% reduction in maintenance costs due to pipeline optimization. Typical investment ranges from $15,000 to $50,000.
What's included in the work
- Prepared dataset for retraining
- Baseline (ALS or Two-Tower) with metrics report
- Trained final model (PyTorch/ONNX)
- Documentation on pipeline reproduction
- Pipeline code (preprocessing, training, inference)
- Guide on fine-tuning with new data
- Training session for your team
- 30 days of post-deployment support
- Consultation on NDCG and Temporal Split tools
Get a specialist consultation — we will discuss your project and propose the optimal solution.