A retail chain of 500 stores was losing 12% of revenue due to the lack of personalized recommendations. Items in the basket didn't cross-sell, promotions didn't leverage synergies, and the analytics department spent weeks on Excel reports. We built an AI basket analysis system that replaced manual work and increased cross-sales by 18% in the first quarter. The key idea is a hybrid of classic association rules and neural networks, providing interpretability for core rules and depth for long-tail.
ML solutions in retail are becoming the standard. Basket segmentation by customer type is another personalization layer we implement through transaction clustering.
What problems does Market Basket Analysis solve?
Data sparsity. In a typical dataset of 1 million transactions and 10,000 SKUs, most item pairs are rarely encountered. Classic algorithms (Apriori, FP-Growth) produce a lot of noise. A neural network model captures weak signals using embeddings and self-attention. We additionally apply data augmentation techniques and weighted sampling for long-tail categories.
Seasonal trends. Rules derived from January data don't work in July. Separate models for each season improve accuracy by 30%. We account for holidays, sales, and weather factors. For each season, we build its own rules and embeddings, allowing recommendations of seasonal product bundles.
Scaling. Processing 1 million transactions with FP-Growth on a single CPU takes 3-8 minutes. For daily real-time recalculations, we use incremental updates and GPU for the neural network. The hybrid architecture allows recalculating rules daily and fine-tuning the neural network weekly — a balance between speed and quality.
Dirty data is another problem. Missing values, duplicates, and outdated prices directly affect rule quality. We automate ETL processes with consistency checks.
Why is a hybrid approach better than classic rules?
According to a McKinsey report, retailers with AI recommendations increase revenue by 10–30%.
Classic FP-Growth + neural network: rules for high-support pairs (lift > 1.2, confidence > 0.3), neural network for long-tail. This yields +15-20% Recall@10 over pure association rules.
import pandas as pd
import numpy as np
from mlxtend.frequent_patterns import fpgrowth, association_rules
from mlxtend.preprocessing import TransactionEncoder
import torch
import torch.nn as nn
class BasketAnalyzer:
"""Classic FP-Growth + neural network extension"""
def __init__(self, min_support: float = 0.01, min_confidence: float = 0.3):
self.min_support = min_support
self.min_confidence = min_confidence
self.rules = None
self.te = TransactionEncoder()
def fit(self, transactions: list[list[str]]) -> pd.DataFrame:
"""
transactions: [['milk', 'bread', 'butter'], ['milk', 'eggs'], ...]
"""
te_array = self.te.fit_transform(transactions)
df = pd.DataFrame(te_array, columns=self.te.columns_)
# FP-Growth is 10-100x faster than Apriori for large datasets
frequent_itemsets = fpgrowth(df, min_support=self.min_support, use_colnames=True)
self.rules = association_rules(
frequent_itemsets,
metric='confidence',
min_threshold=self.min_confidence
)
# Add lift and conviction for filtering
self.rules = self.rules[self.rules['lift'] > 1.2]
self.rules = self.rules.sort_values('lift', ascending=False)
return self.rules
def get_recommendations(self, basket: list[str],
top_k: int = 5) -> list[dict]:
"""Recommendations for the current basket"""
if self.rules is None:
return []
basket_set = frozenset(basket)
matching_rules = self.rules[
self.rules['antecedents'].apply(lambda x: x.issubset(basket_set))
]
# Remove items already in basket
recommendations = []
seen = set()
for _, rule in matching_rules.iterrows():
for item in rule['consequents']:
if item not in basket_set and item not in seen:
recommendations.append({
'item': item,
'confidence': rule['confidence'],
'lift': rule['lift'],
'support': rule['support']
})
seen.add(item)
if len(recommendations) >= top_k:
break
if len(recommendations) >= top_k:
break
return recommendations
def get_category_affinity(self, transactions: pd.DataFrame) -> pd.DataFrame:
"""Affinity matrix between product categories"""
# Transactions with categories instead of specific items
cat_transactions = transactions.groupby('order_id')['category'].apply(list).tolist()
te_cat = TransactionEncoder()
cat_array = te_cat.fit_transform(cat_transactions)
cat_df = pd.DataFrame(cat_array, columns=te_cat.columns_)
# Category co-occurrence
co_occurrence = cat_df.T.dot(cat_df)
np.fill_diagonal(co_occurrence.values, 0)
# Normalize by support (PMI-like measure)
totals = cat_df.sum()
n = len(cat_df)
affinity = co_occurrence / n / (totals.values[:, None] * totals.values[None, :] / n**2 + 1e-9)
return affinity
class NeuralBasketPredictor(nn.Module):
"""
Neural network model for predicting the next item to add to the basket.
Input: bag-of-items binary vector of the current basket.
"""
def __init__(self, n_items: int, embedding_dim: int = 64):
super().__init__()
self.item_embedding = nn.Embedding(n_items, embedding_dim, padding_idx=0)
self.attention = nn.MultiheadAttention(embedding_dim, num_heads=4, batch_first=True)
self.fc = nn.Sequential(
nn.Linear(embedding_dim, 256),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(256, n_items)
)
def forward(self, basket_item_ids: torch.Tensor) -> torch.Tensor:
"""
basket_item_ids: (batch, seq_len) — indices of items in the basket
Returns: (batch, n_items) — logits for each item
"""
emb = self.item_embedding(basket_item_ids) # (batch, seq, dim)
attended, _ = self.attention(emb, emb, emb) # Self-attention over basket
pooled = attended.mean(dim=1) # (batch, dim)
return self.fc(pooled)
How to account for temporal patterns and seasonality?
class TemporalBasketAnalyzer:
"""Analysis of temporal purchase patterns"""
def find_sequential_patterns(self, orders: pd.DataFrame,
days_window: int = 7) -> pd.DataFrame:
"""Find sequential purchases within an N-day window"""
orders_sorted = orders.sort_values(['user_id', 'order_date'])
sequential = []
for user_id, user_orders in orders_sorted.groupby('user_id'):
order_list = user_orders.to_dict('records')
for i, order_i in enumerate(order_list):
for order_j in order_list[i+1:]:
days_diff = (order_j['order_date'] - order_i['order_date']).days
if days_diff > days_window:
break
if days_diff > 0:
sequential.append({
'item_a': order_i['sku'],
'item_b': order_j['sku'],
'days_between': days_diff
})
df = pd.DataFrame(sequential)
if df.empty:
return df
# Top sequential pairs
return (df.groupby(['item_a', 'item_b'])
.agg(count=('days_between', 'count'),
avg_days=('days_between', 'mean'))
.reset_index()
.sort_values('count', ascending=False))
def get_seasonal_baskets(self, transactions: pd.DataFrame) -> dict:
"""Seasonal baskets: what is usually bought together in a specific period"""
transactions['month'] = transactions['order_date'].dt.month
transactions['season'] = transactions['month'].map({
12: 'winter', 1: 'winter', 2: 'winter',
3: 'spring', 4: 'spring', 5: 'spring',
6: 'summer', 7: 'summer', 8: 'summer',
9: 'autumn', 10: 'autumn', 11: 'autumn'
})
seasonal_rules = {}
for season, group in transactions.groupby('season'):
season_transactions = group.groupby('order_id')['sku'].apply(list).tolist()
if len(season_transactions) < 100:
continue
te = TransactionEncoder()
arr = te.fit_transform(season_transactions)
df_season = pd.DataFrame(arr, columns=te.columns_)
freq = fpgrowth(df_season, min_support=0.02, use_colnames=True)
if not freq.empty:
rules = association_rules(freq, metric='lift', min_threshold=1.5)
seasonal_rules[season] = rules.sort_values('lift', ascending=False).head(20)
return seasonal_rules
Temporal patterns allow recommendations not only based on a single basket but also on purchase sequences over time. For example, after buying a printer, people often buy cartridges within a week. Seasonal baskets are built separately for each season, increasing recommendation relevance. For each user, the system considers their history — providing personalized temporal links.
Implementation process: from audit to deployment
- Analytics — data audit (sources, quality, update frequency). Check for missing values, duplicates, transaction distribution.
- Design — choose architecture (hybrid / rules only / NN only), prototype on a 10k transaction sample.
- Development — train models, write REST API on FastAPI, integrate with CRM/ERP via webhook.
- Testing — A/B test on 10% of audience, measure CTR and ATC, analyze lift by category.
- Deployment — containerization with Docker, deploy in your Kubernetes or cloud, set up monitoring (Grafana, Prometheus, alerting).
What is included in the work (deliverables)
- Training 2-3 models (FP-Growth, Neural, Temporal) with hyperparameter tuning.
- REST API with documentation (OpenAPI 3.0) — endpoints: basket recommendations, seasonal rules, temporal sequences.
- Metrics dashboard (Grafana) — recall@k, precision@k, CTR, latency p99.
- Training for your engineers (2 workshops, 4 hours each) — architecture, result interpretation, fine-tuning.
- Warranty support for 12 months with response time up to 4 hours.
Approach comparison: metrics and speeds
| Approach | Recall@10 | Training time (1M transactions) | Long-tail coverage | Interpretability |
|---|---|---|---|---|
| Rules only | 0.42 | 8 min | Low | High |
| Neural only | 0.55 | 2 h (GPU) | Medium | Low |
| Hybrid (ours) | 0.61 | 2 h 8 min | High | Medium |
The hybrid approach gives 45% more recall than pure rules while maintaining interpretability for top pairs. We use it for retail chains with 50+ stores — experience with over 10 projects.
More about metrics
precision@k — proportion of relevant recommendations among the first k, recall@k — comprehensiveness. For your business, recall@10 is more important if the goal is to find as many suitable items as possible. latency p99 — API response time; we guarantee < 200ms.| Scenario | Cross-sales increase | Churn reduction | Payback period |
|---|---|---|---|
| Grocery retail | +22% | -15% | 4-6 months |
| Fashion | +18% | -10% | 6-8 months |
| DIY/home goods | +25% | -12% | 5-7 months |
Get a preliminary estimate of your data — we will analyze the structure and propose an optimal architecture in 2 days.
Contact us to discuss your task. Order a pilot project — get the first recommendations in 2 weeks. Get a consultation on assessing your data.