Genetic Algorithms Outperform Deep RL for Trading

The author shares their experience of trying to use Deep Reinforcement Learning (DRL) for stock trading, and how they ultimately found more success with genetic algorithms (GA).

Details

The author spent six months trying to make Deep Reinforcement Learning (DRL) work for stock trading, experimenting with various algorithms like PPO, SAC, and A2C. However, they faced several challenges with DRL, including the difficulty of reward shaping, catastrophic forgetting, and the inability to explain the model's decisions. The author found that reward shaping is 'black magic', as every choice of reward function can create perverse incentives. Catastrophic forgetting was also a major issue, where the model would forget everything about crash patterns when fine-tuned on bull market data. Additionally, the author struggled to debug the DRL models, as the neural network's decisions were opaque. The author also noted the insatiable data hunger of deep learning models, which require much more data than the 5,000 daily bars typically available for financial markets. Frustrated with these challenges, the author switched to genetic algorithms (GA), which involve defining a 'strategy DNA' - a set of numerical parameters that control the trading decisions. Within two weeks, the author was able to develop a profitable trading strategy using GA, which they found to be a simpler and more effective approach compared to DRL.