Training an RL Agent (PPO/SAC/DQN) for a Trading Strategy
Imagine: you've spent months training a DQN agent on historical data, only to have it lose capital on the live market due to unaccounted slippage. In one of our projects, a client came with a similar problem: they trained PPO on minute candles, achieving a Sharpe ratio of 1.8 in testing, but in live trading the drawdown reached 40%. The client was losing around $15,000 monthly because of these shortcomings. We discovered that the environment did not account for fees and liquidity. After calibrating the reward and adding walk-forward validation, the Sharpe ratio climbed back to 1.5 and the drawdown dropped to 12%. This saved $4,000 per month.
Designing an RL agent for crypto trading is not just about picking an algorithm. You face market non-stationarity, hidden fees, slippage, and the risk of overfitting. We take on the full cycle—from building the data pipeline to live trading. We use proven algorithms PPO, SAC, and DQN, adapting them to your strategy. Our experience: over five years in blockchain development, 15+ projects for DeFi and CEX, including integration with Binance API. Contact us for a detailed analysis of your strategy.
Three Working Algorithms: DQN, PPO, SAC
Each algorithm has its niche. Let's look at their strengths and typical use cases.
DQN (Deep Q-Network)
Suitable for discrete actions (buy/hold/sell) and simple strategies. DQN approximates the Q-function: Q(state, action) — the expected discounted reward for taking action in state.
import torch
import torch.nn as nn
from collections import deque
import random
class DQNNetwork(nn.Module):
def __init__(self, state_dim, n_actions, hidden_dim=256):
super().__init__()
# Dueling architecture: separate Value and Advantage streams
self.shared = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU()
)
self.value_stream = nn.Linear(hidden_dim, 1)
self.advantage_stream = nn.Linear(hidden_dim, n_actions)
def forward(self, x):
shared = self.shared(x)
value = self.value_stream(shared)
advantage = self.advantage_stream(shared)
# Dueling: Q = V + (A - mean(A))
q_values = value + (advantage - advantage.mean(dim=1, keepdim=True))
return q_values
class PrioritizedReplayBuffer:
"""Prioritized Experience Replay — sample important transitions more often"""
def __init__(self, capacity=50000, alpha=0.6):
self.buffer = deque(maxlen=capacity)
self.priorities = deque(maxlen=capacity)
self.alpha = alpha
def push(self, state, action, reward, next_state, done, td_error=1.0):
priority = (abs(td_error) + 1e-5) ** self.alpha
self.buffer.append((state, action, reward, next_state, done))
self.priorities.append(priority)
def sample(self, batch_size, beta=0.4):
probs = np.array(self.priorities) / sum(self.priorities)
indices = np.random.choice(len(self.buffer), batch_size, p=probs)
# Importance sampling weights
weights = (len(self.buffer) * probs[indices]) ** (-beta)
weights /= weights.max()
batch = [self.buffer[i] for i in indices]
return batch, indices, weights
Double DQN eliminates Q-value overestimation: the online network selects the action, and the target network evaluates it.
# Double DQN target calculation
with torch.no_grad():
next_actions = online_net(next_states).argmax(dim=1) # online net selects
next_q = target_net(next_states).gather(1, next_actions.unsqueeze(1)) # target evaluates
targets = rewards + gamma * next_q * (1 - dones)
PPO (Proximal Policy Optimization)
Suitable for both discrete and continuous actions, on-policy, stable training. PPO limits the policy update size via clipping.
class PPOActor(nn.Module):
def __init__(self, state_dim, action_dim, hidden_dim=256):
super().__init__()
self.network = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.Tanh(),
nn.Linear(hidden_dim, hidden_dim),
nn.Tanh()
)
self.policy_head = nn.Linear(hidden_dim, action_dim)
self.value_head = nn.Linear(hidden_dim, 1)
def forward(self, x):
features = self.network(x)
logits = self.policy_head(features)
value = self.value_head(features)
return logits, value
def ppo_update(model, optimizer, states, actions, old_log_probs,
advantages, returns, clip_eps=0.2, n_epochs=4):
for _ in range(n_epochs):
logits, values = model(states)
dist = torch.distributions.Categorical(logits=logits)
new_log_probs = dist.log_prob(actions)
entropy = dist.entropy()
# PPO clipped objective
ratio = (new_log_probs - old_log_probs).exp()
surr1 = ratio * advantages
surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
actor_loss = -torch.min(surr1, surr2).mean()
critic_loss = (returns - values.squeeze()).pow(2).mean()
entropy_loss = -entropy.mean()
total_loss = actor_loss + 0.5 * critic_loss + 0.01 * entropy_loss
optimizer.zero_grad()
total_loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
optimizer.step()
SAC (Soft Actor-Critic)
Suitable for continuous action space (positioning 0%–100% of capital), off-policy, maximum sample efficiency. SAC maximizes: J(π) = E[Σ γ^t (r_t + α H(π(·|s_t)))]. The entropy term H encourages exploration.
class SACActorContinuous(nn.Module):
def __init__(self, state_dim, action_dim, hidden_dim=256):
super().__init__()
self.network = nn.Sequential(
nn.Linear(state_dim, hidden_dim), nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim), nn.ReLU()
)
self.mean_head = nn.Linear(hidden_dim, action_dim)
self.log_std_head = nn.Linear(hidden_dim, action_dim)
def forward(self, x):
features = self.network(x)
mean = self.mean_head(features)
log_std = self.log_std_head(features).clamp(-20, 2)
std = log_std.exp()
dist = torch.distributions.Normal(mean, std)
action = dist.rsample() # reparameterization trick
# Squash to [-1, 1]
action_tanh = torch.tanh(action)
log_prob = dist.log_prob(action) - torch.log(1 - action_tanh.pow(2) + 1e-6)
return action_tanh, log_prob.sum(-1, keepdim=True)
Which Algorithm to Choose for Your Strategy?
The choice depends on the action space and sample efficiency requirements. If your strategy uses only discrete signals (buy/sell/hold), DQN with dueling and PER will give stable results. For continuous capital management (e.g., portfolio percentage), SAC is unrivaled: it's 2–3 times more sample-efficient than PPO. PPO is a universal choice when you need reliability and ease of tuning.
We often combine algorithms in a multi-agent architecture: a macro-agent based on DQN determines the overall direction, and a micro-agent based on SAC executes trades. This reduces variance and improves the Sharpe ratio by 15–30%.
Why Is a Correct Reward Function Important?
Reward shaping is a key step that determines the agent's behavior. Typical mistakes: the agent learns to accumulate unrealized profit (without accounting for slippage) or starts trading very rarely to avoid fees. We use a multi-component reward: PnL, drawdown penalty, fees, and spread. For example, reward = ΔP&L - λ1 * fee - λ2 * max_drawdown. The λ coefficients are chosen to simulate realistic conditions.
In one DeFi project, an incorrect reward led the agent to open hundreds of micro-trades, generating a loss from fees. After redesigning the reward (penalty for the number of trades), the agent became profitable.
Algorithm Comparison for Crypto Trading
| Algorithm | Action Space | Sample Efficiency | Stability | Best Use Case |
|---|---|---|---|---|
| DQN | Discrete | Medium | Medium | Simple buy/sell strategies |
| PPO | Both | Low (on-policy) | High | General-purpose, reliable |
| SAC | Continuous | High | High | Position sizing as action |
How to Set Up RL Agent Training: Step-by-Step Plan
- Define the action space and state space. For discrete actions (buy/sell/hold), DQN is suitable; for continuous positioning, SAC. The state includes prices, volumes, indicators.
- Design the reward function. Account for PnL, fees, slippage, drawdown penalty.
- Choose the algorithm and neural network architecture. We use dueling DQN, PPO with clipping, SAC with automatic entropy tuning.
- Train with validation. We apply walk-forward validation with 36 rolling windows and early stopping.
- Test on out-of-sample data. Evaluate Sharpe ratio, max drawdown, reward stability.
Typical Challenges and How We Solve Them
Market non-stationarity — an agent trained on a calm market may fail in high volatility. As noted in reinforcement learning specification, distribution shift is a serious challenge. We use curriculum learning: gradually increase environment volatility, and in production — continuous fine-tuning with a drift detector.
Reward hacking — artificially inflated rewards. Protection via reward clipping and using a realistic simulator with market data (Level 2, historical candles).
Overfitting — agent memorization. We use walk-forward validation with 36 rolling windows and testing on fully excluded periods (out-of-sample).
Example of Hyperparameter Tuning
For PPO, we tune learning rate (3e-4), clip epsilon (0.2), entropy coefficient (0.01) via Bayesian optimization on 50 trials. The best configurations are saved in MLflow. Typical search time is 2 days on GPU.What's Included in the Work
- Strategy analysis and data pipeline preparation.
- Custom environment design (gymnasium) accounting for fees, slippage, and drawdowns.
- Algorithm and neural network architecture selection.
- Training with hyperparameter tuning (grid/random search, Bayesian optimization).
- Walk-forward validation and robustness to market regime changes.
- Integration with broker API (Binance, Bybit, KuCoin).
- Documentation, training your team, 3-month support.
Contact us so we can analyze your strategy. We guarantee result quality and support at all stages.
Estimated Timelines and Stages
| Stage | Duration | Result |
|---|---|---|
| Analysis and data pipeline | 1–2 weeks | Prepared data, environment specification |
| Environment and algorithm design | 1–2 weeks | Custom environment, baseline model |
| Training and hyperparameter tuning | 2–4 weeks | Optimal policy, metrics in MLflow |
| Walk-forward validation and testing | 1–2 weeks | Report on Sharpe, drawdown, out-of-sample |
| Integration and deployment | 1–2 weeks | Live trading agent, documentation |
Order a consultation — we will select the algorithm and architecture for your task. We will evaluate the project for free within 2 business days.