Reinforcement Learning: PPO, SAC, DQN and Industrial Applications
We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.
Why is RL harder than supervised learning?
In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.
Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.
Algorithm selection by task:
| Task | Algorithm | Reason |
|---|---|---|
| Continuous control (robotics, industrial processes) | SAC, TD3 | Sample efficiency, stability |
| Discrete actions, game-playing | PPO, DQN + Rainbow | Simplicity, industry-proven |
| Multi-agent | MAPPO, QMIX | Cooperation/competition |
| Offline RL (dataset without environment) | CQL, IQL, TD3+BC | Learning without environment |
| RLHF (LLM alignment) | PPO, GRPO | Integration with reward model |
How to tune PPO and avoid common problems?
PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.
One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.
Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.
For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.
SAC for continuous control
SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.
On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.
How to transfer a trained agent to a real device?
Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.
The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.
Comparison of popular simulators:
| Simulator | Features | Performance |
|---|---|---|
| MuJoCo | Standard for robotics, medium physics | Single robot — CPU |
| Isaac Gym / Isaac Lab (NVIDIA) | GPU-accelerated, 10,000+ parallel environments | High (up to 50,000 fps on A100) |
| PyBullet | Free, convenient for prototyping | Low, CPU |
| Gazebo | ROS integration, full cycle | Medium, CPU+GPU |
Case: manipulator for PCB component sorting
We used Isaac Gym with 4096 parallel environments on an A100, PPO with domain randomization (random mass, lighting, camera position). 500 million steps — 18 hours. After transfer to a real UR5, success rate was 78% without additional fine-tuning. After 2 hours on the real robot (10k steps) — 94%. Entire process — 3 weeks.
RLHF: training LLMs from human feedback
RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.
Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:
- DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
- GRPO — used in DeepSeek-R1, good for reasoning tasks.
- ORPO — combines SFT and alignment into one stage.
The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.
"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."
What is included in the work
- Architectural solution and justification of algorithm selection
- Development and documentation of the reward function
- Creating a simulator or configuring an existing one
- Training, hyperparameter sweep (Optuna / Ray Tune)
- Transfer to real hardware or integration into product
- Documentation, access to code and simulators
- Team training and 3-month support after deployment
Work process
- Task audit — define goals, resources, constraints.
- Reward engineering — formalize desired behavior, check for reward hacking.
- Environment and algorithm selection — baseline, first runs.
- Systematic hyperparameter sweep — use Optuna.
- Training in simulation with domain randomization.
- Testing on real equipment (if necessary).
- Deployment, monitoring, support.
Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.
Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.