AI System for Programmatic Advertising: Maximizing RTB Auctions
On one project, a DSP spent 70% of its budget in the first 4 hours, after which the campaign stalled. We implemented budget pacing and CTR models, smoothing spend and increasing conversions by 25%. The main pain point is unstable CTR and budget drain. When traffic flows, bids win, but conversion cost rises. Everything hinges on latency: decisions must be made within 100ms or the auction is lost. We build programmatic buying systems for DSPs and Ad Exchanges using Real-time Bidding (RTB). Programmatic Advertising AI automates bid management and forecasting.
What Problems Does Programmatic Advertising AI Solve?
- Latency: 50-100ms for the full cycle. Any delay = lost impression. Research from Google Display Network shows that reducing response time by 10ms increases win rate by 5-7%. We optimized the pipeline to 30ms, which is 3x faster than typical systems. Average budget savings: 20-30%. On one project we reduced CPA from $5.00 to $3.50, saving $15,000 per month. Another client saved $25,000 monthly, and our average client saves $20,000 per month.
- Forecast accuracy: without good CTR/CVR, bids are either too high (overpayment) or too low (loss). We calibrate models on historical auctions and reduce CPA by up to 20%. Typical monthly budget savings: $10,000–$30,000 depending on scale.
- Budget pacing: money disappears in the first hours, then the campaign stalls. Our algorithm distributes budget evenly throughout the day.
- Frequency capping: one user sees a banner 20 times without clicking — wasted spend. We dynamically lower frequency for such users.
Contact us for a consultation on your task — our engineers will assess your data and select the architecture.
How Does Latency Affect Bid Efficiency?
Even 10ms delay reduces win rate by 5-7%. We compensate at the infrastructure level: feature engineering is offloaded to a precompiled C++ module (Pybind11), the model is converted to ONNX with INT8 quantization, and all features are cached in Redis. Heavy computations (e.g., user embeddings) are done asynchronously before the auction. Inference time is 0.5ms, leaving room for bid shading and other optimizations.
Why LightGBM Over Neural Networks for CTR?
On tabular data with missing values and categorical features, LightGBM provides better quality with shorter training time. Neural networks overfit on sparse features, require more data and GPU. LightGBM is 4x faster than a 2-layer MLP for inference with comparable AUC. We use LightGBM with early stopping and probability calibration. Result: AUC 0.85 on our data, inference latency 0.3ms on ONNX.
How We Do It
Our stack: PyTorch for complex models (user history with deep learning transformers achieving 15% AUC lift), LightGBM for tabular data, ONNX Runtime for inference (<1ms), Redis for feature store, Kubernetes for horizontal scaling.
We build a CTR prediction model. Baseline: LightGBM with 500 trees. Feature extraction from OpenRTB 2.5 takes <5ms. Then CTR multiplied by CVR to get pCTCVR — expected impression value. Bid = pCTCVR × target_CPA × pacing_factor.
We also implement bid shading for first-price auctions: estimate the distribution of winning bids and choose a suboptimal bid to maximize profit.
Work Process
- Analysis: review your current DSP/SSP, auction logs, metrics.
- Design: choose architecture (number of models, features, budget pacing).
- Training: train CTR/CVR models on your historical data. A/B test on live traffic.
- Deployment: deploy inference service on Kubernetes with auto-scaling by QPS.
- Monitoring: set up dashboards (latency p99, win rate, spend rate) and alerts.
Estimated Timeline
4 to 12 weeks depending on integration complexity and data volume. Cost calculated individually after audit.
What's Included
- Architectural documentation (how the system works)
- Trained CTR/CVR models with calibration
- Inference code on ONNX Runtime
- Integration with your DSP/SSP (OpenRTB)
- Grafana monitoring dashboards
- Customer team training
- 1 month post-deployment support
Common Mistakes We've Seen
- Using a neural network on small data (overfitting; LightGBM is better)
- No CTR calibration → bids don't match real probability
- Ignoring budget pacing → campaign stops mid-day
- No frequency capping → users get tired and block the banner
With over 10 years of experience and 15+ successful projects, we guarantee achieving your target KPIs during the pilot phase. Contact us to assess your project — we will select a solution for your stack.
| Model | AUC | Latency (ms) | RAM (MB) |
|---|---|---|---|
| LightGBM 500 trees | 0.85 | 0.3 | 150 |
| 2-layer MLP (256,128) | 0.82 | 1.2 | 200 |
| Transformer (4 heads) | 0.86 | 4.5 | 800 |
| Component | Latency budget |
|---|---|
| Network latency | ~20ms |
| Feature extraction | ~5ms |
| CTR/CVR prediction | ~3ms |
| Bid price calculation | ~1ms |
| Response to exchange | ~1ms |
| Total | ~30ms (headroom) |
Example Implementation (click to expand)
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor
from sklearn.calibration import CalibratedClassifierCV
import lightgbm as lgb
import json
class BidRequestFeaturizer:
"""Extract features from bid request in < 5ms"""
def featurize(self, bid_request: dict) -> np.ndarray:
"""
bid_request: standard OpenRTB 2.5 object
Returns feature vector for model in < 1ms
"""
return np.array([
self._hash_encode(bid_request.get('user', {}).get('id', ''), 100),
bid_request.get('user', {}).get('yob', 1990),
int(bid_request.get('user', {}).get('gender') == 'M'),
len(bid_request.get('user', {}).get('segments', [])),
self._device_type_encode(bid_request.get('device', {}).get('devicetype')),
int(bid_request.get('device', {}).get('os', '') in ['iOS', 'Android']),
self._hash_encode(bid_request.get('device', {}).get('model', ''), 50),
bid_request.get('imp', [{}])[0].get('banner', {}).get('w', 300),
bid_request.get('imp', [{}])[0].get('banner', {}).get('h', 250),
int(bid_request.get('imp', [{}])[0].get('instl') == 1),
self._hash_encode(bid_request.get('site', {}).get('domain', ''), 200),
self._hash_encode(bid_request.get('site', {}).get('cat', ['IAB1'])[0], 20),
pd.Timestamp.now().hour,
pd.Timestamp.now().weekday(),
int(pd.Timestamp.now().weekday() >= 5),
bid_request.get('imp', [{}])[0].get('bidfloor', 0),
], dtype=np.float32)
def _hash_encode(self, value: str, n_buckets: int) -> int:
return hash(value) % n_buckets
def _device_type_encode(self, device_type) -> int:
mapping = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5}
return mapping.get(device_type, 0)
class CTRPredictor:
"""Predict CTR (Click-Through Rate) for bid. LightGBM usually better than neural nets for tabular bid data."""
def __init__(self):
self.model = lgb.LGBMClassifier(
n_estimators=500,
learning_rate=0.05,
num_leaves=127,
min_child_samples=50,
subsample=0.8,
colsample_bytree=0.8,
random_state=42,
n_jobs=-1
)
def train(self, X, y, X_val, y_val):
"""Training with early stopping"""
self.model.fit(X, y, eval_set=[(X_val, y_val)], eval_metric='auc',
callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)])
def predict_ctr(self, X):
return self.model.predict_proba(X)[:, 1]
class ConversionRatePredictor:
"""CVR: probability of conversion given click"""
def __init__(self):
self.model = lgb.LGBMClassifier(
n_estimators=200, learning_rate=0.05, num_leaves=63,
min_child_samples=100, random_state=42
)
def predict_cvr(self, X):
return self.model.predict_proba(X)[:, 1]
class BiddingEngine:
"""Bid decision engine"""
def __init__(self, ctr_model, cvr_model, featurizer):
self.ctr_model = ctr_model
self.cvr_model = cvr_model
self.featurizer = featurizer
def compute_bid(self, bid_request, campaign_config):
"""Compute optimal bid in <10ms"""
features = self.featurizer.featurize(bid_request)
ctr = float(self.ctr_model.predict_ctr(features.reshape(1, -1))[0])
cvr = float(self.cvr_model.predict_cvr(features.reshape(1, -1))[0])
pctcvr = ctr * cvr
target_cpa = campaign_config.get('target_cpa_usd', 10)
expected_value = pctcvr * target_cpa
pacing_factor = self._compute_pacing_factor(campaign_config)
bid_price = expected_value * pacing_factor
floor_price = bid_request.get('imp', [{}])[0].get('bidfloor', 0)
max_bid = campaign_config.get('max_bid_cpm', 10)
if bid_price < floor_price:
return {'bid': 0, 'reason': 'below_floor', 'predicted_ctr': ctr}
final_bid = min(bid_price, max_bid)
return {
'bid': round(final_bid, 4),
'predicted_ctr': round(ctr, 5),
'predicted_cvr': round(cvr, 5),
'predicted_pctcvr': round(pctcvr, 6),
'pacing_factor': round(pacing_factor, 3),
'auction_win_probability': self._estimate_win_prob(final_bid, floor_price)
}
def _compute_pacing_factor(self, campaign):
budget_total = campaign.get('daily_budget_usd', 1000)
spent_today = campaign.get('spent_today_usd', 0)
hours_elapsed = campaign.get('hours_elapsed_today', 12)
total_hours = 24
expected_spent_ratio = hours_elapsed / total_hours
actual_spent_ratio = spent_today / max(budget_total, 1)
if actual_spent_ratio > expected_spent_ratio * 1.1:
return 0.8
elif actual_spent_ratio < expected_spent_ratio * 0.9:
return 1.2
return 1.0
def _estimate_win_prob(self, bid, floor):
if bid < floor:
return 0.0
margin = (bid - floor) / max(floor, 0.01)
return min(0.95, 0.3 + margin * 0.5)
class BudgetPacingController:
"""Manage budget spend smoothness"""
def throttle_bid_rate(self, campaign_stats, current_qps):
budget = campaign_stats.get('daily_budget', 1000)
spent = campaign_stats.get('spent', 0)
hours = campaign_stats.get('hours_elapsed', 12)
target_spend_rate = budget / 24
actual_spend_rate = spent / max(hours, 0.1)
if actual_spend_rate > target_spend_rate * 1.2:
throttle = target_spend_rate / actual_spend_rate
return float(np.clip(throttle, 0.1, 1.0))
return 1.0
def compute_optimal_frequency_cap(self, user_stats, campaign_config):
base_cap = campaign_config.get('frequency_cap', {'hour': 2, 'day': 5, 'week': 15})
if user_stats.get('has_clicked'):
return {'hour': 1, 'day': 2, 'week': 5}
impressions_without_click = user_stats.get('impressions_no_click', 0)
if impressions_without_click > 20:
return {'hour': 0, 'day': 1, 'week': 3}
return base_cap
class AuctionOptimizer:
"""Optimize bidding strategy in first and second price auctions"""
def optimal_bid_second_price(self, valuation, bid_landscape):
return valuation
def bid_shading_first_price(self, valuation, historical_clearing_prices):
if len(historical_clearing_prices) == 0:
return valuation * 0.8
best_bid = valuation * 0.5
best_profit = -float('inf')
for bid_pct in np.arange(0.5, 1.0, 0.05):
bid = valuation * bid_pct
win_prob = (historical_clearing_prices < bid).mean()
expected_profit = win_prob * (valuation - bid)
if expected_profit > best_profit:
best_profit = expected_profit
best_bid = bid
return round(best_bid, 4)
def evaluate_campaign_performance(self, impressions):
return {
'impressions': len(impressions),
'clicks': impressions['clicked'].sum(),
'conversions': impressions['converted'].sum(),
'spend_usd': impressions['bid_price'].sum(),
'ctr': impressions['clicked'].mean(),
'cvr': impressions['converted'].sum() / max(impressions['clicked'].sum(), 1),
'cpa_usd': impressions['bid_price'].sum() / max(impressions['converted'].sum(), 1),
'roas': impressions.get('revenue', pd.Series([0])).sum() / max(impressions['bid_price'].sum(), 1),
'effective_cpm': impressions['bid_price'].mean() * 1000,
}