FinRL Framework Setup for Training a Trading Agent

A trading agent based on DRL will fail if the environment does not reflect real commissions and liquidity. We have encountered projects where the model showed 200% returns in simulation but lost capital in live markets due to ignoring spreads and slippage. FinRL addresses this with the customizable StockTradingEnv environment, where you can set buy/sell commissions, initial capital, and transaction limits. Many developers copy the configuration from an article without considering their own broker: for example, they use the default 0.1% commission while the real figure is 0.3% including spread. After adjusting the environment, the return dropped from 30% to 8%, but the model started earning consistently in production. We configure the environment for your specific market conditions: liquidity, spreads, position size limits. Our experience — 5+ years of DRL implementation in finance, 50+ successful projects. We guarantee a working baseline in 2–3 days. Get a consultation — we will assess your case end-to-end. Contact us for a cost and timeline estimate.

Typical Problems When Configuring FinRL

Ignoring commissions and slippage. If you do not set buy_cost_pct and sell_cost_pct, the agent will execute trades with zero cost, which is impossible in reality. In a project with a hedge fund, we added 0.05% slippage per trade — the return dropped by 12%, but the model became robust to market conditions.

Overfitting on historical data. The agent may memorize noise instead of signal. To avoid this, we use out-of-sample backtesting with a 70/30 data split and regularization via entropy bonus in PPO. Typical risk penalty savings: +15% Sharpe ratio.

Incorrect feature normalization. Stock prices are unscaled, which hinders training. FinRL automatically applies Z-score normalization to technical indicators, but if the data contains outliers (e.g., flash crash), the agent may choose a wrong strategy. We add winsorization at the 0.5th percentile.

Installation and First Run

pip install finrl
pip install stockstats wrds alpaca-trade-api  # data sources

Quick start:

import finrl
from finrl.train import train
from finrl.test import test
from finrl.config_tickers import DOW_30_TICKER
from finrl.config import INDICATORS

# train on Dow Jones 30 stocks
train(
    start_date='2010-01-01',
    end_date='2021-10-31',
    ticker_list=DOW_30_TICKER,
    data_source='yahoofinance',
    technical_indicator_list=INDICATORS,
    drl_lib='stable_baselines3',
    env='stock_trading',
    model_name='ppo',
    if_store_account_value=True,
    cwd='./trained_models/ppo_dow30'
)

test(
    start_date='2021-11-01',
    end_date='2023-12-31',
    ticker_list=DOW_30_TICKER,
    data_source='yahoofinance',
    technical_indicator_list=INDICATORS,
    drl_lib='stable_baselines3',
    env='stock_trading',
    model_name='ppo',
    cwd='./trained_models/ppo_dow30'
)

How to Configure the Reward Function?

The reward function defines the agent's behavior. In FinRL it is set via reward_scaling in env_kwargs. By default, the reward equals the change in portfolio value. We often add a penalty for excessive risk, such as the variance of daily returns. This forces the agent to seek more stable strategies, improving the Sharpe ratio by 15–20%.

from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv

env_kwargs = {
    "hmax": 100,                    # max shares per transaction
    "initial_amount": 1_000_000,    # initial capital
    "buy_cost_pct": [0.001] * n,    # buying commission
    "sell_cost_pct": [0.001] * n,   # selling commission
    "state_space": state_space,
    "stock_dim": n_tickers,
    "tech_indicator_list": INDICATORS,
    "action_space": n_tickers,
    "reward_scaling": 1e-4          # reward scaling
}

env = StockTradingEnv(df=train_df, **env_kwargs)

Why Is PPO Effective for Trading?

PPO (Proximal Policy Optimization) is one of the most popular algorithms for financial DRL. It is more stable than A2C, faster than DDPG, and requires less hyperparameter tuning. In our benchmarks, PPO outperforms A2C in Sharpe ratio by 15–20% and converges 2× faster than DDPG on equal data volume. According to Wikipedia, PPO ensures reliable convergence in continuous action space problems.

To compare models, use FinRL's built-in functions:

models = ['a2c', 'ddpg', 'ppo', 'td3', 'sac']
results = {}

for model_name in models:
    train(model_name=model_name, cwd=f'./models/{model_name}', ...)
    account_value = test(model_name=model_name, cwd=f'./models/{model_name}', ...)
    results[model_name] = account_value

# visualization
from finrl.plot import backtest_plot
backtest_plot(results, baseline_start='2022-01-01', baseline_end='2023-12-31',
             baseline_ticker='^GSPC')  # vs S&P500

How to Avoid Overfitting During Agent Training?

Overfitting is a common problem in DRL for finance. Use these techniques:

• Split data into train/validation/test (e.g., 60/20/20). Use the validation set for early stopping.
• Apply regularization: in PPO, use the ent_coef parameter (entropy bonus). Values 0.01–0.05 improve generalization.
• Add noise to the environment: random order execution delays or stochastic commissions. This simulates market noise.
• Limit episode steps to no more than 252 (trading year). Short episodes force the agent to focus on short-term signals.
• Test the model on out-of-sample periods with crises (e.g., 2020). If the return drops more than 30%, the agent is overfitted.

Which Metrics Should Be Used to Evaluate the Agent?

Don't rely solely on cumulative return. Use a comprehensive set:

• Sharpe ratio — risk-adjusted return. A good agent shows Sharpe > 1.0.
• Maximum drawdown — the largest peak-to-trough decline. Should not exceed 20%.
• Win rate — percentage of profitable trades. Above 50% is good, but depends on strategy.
• Calmar ratio — annual return divided by maximum drawdown. Ideally > 2.
• Sortino ratio — similar to Sharpe but considers only negative volatility. More strict.

In FinRL, these metrics can be obtained via backtest_plot() or calculated manually from account_value. We provide all key metrics in our report.

What's Included in Our Work

• Analysis of available market data and selection of sources.
• Configuration of the StockTradingEnv environment for your broker/market.
• Training of 5 DRL algorithms with automatic hyperparameter tuning.
• Backtesting on historical data with a metrics report (Sharpe, Sortino, Max Drawdown).
• Delivery of the trained model and documentation for deployment.
• Consultation on integration with your trading terminal.

Process

1. Analytics — review requirements, data sources, constraints.
2. Design — define state/action space, reward function.
3. Implementation — configure FinRL, train baseline models.
4. Testing — backtest on out-of-sample data, check for robustness.
5. Deployment — hand over the model, train your staff.

Estimated Timelines

Cost is calculated individually — contact us to evaluate your project. Our TensorFlow-certified engineers have experience with JAX and PyTorch. Get a consultation — we'll discuss the details. Order FinRL setup — get a working prototype in 2–3 days.