Reinforcement learning has demonstrated spectacular success in complex, sequential decision-making problems, from mastering strategic games to controlling robotic systems. A natural question is whether these same principles can be applied to the noisy and dynamic world of algorithmic trading. In this project, we investigate this question by focusing on a central challenge in quantitative finance: synthesizing predictive signals into optimal trading decisions. The goal is to train a reinforcement learning agent that can ingest a stream of price-returns predictions and learn a trading policy to maximize profit.
We model this as an optimal execution problem where the agent must use a short-term returns prediction to trade towards a predefined target position that is set by a long-term returns prediction. To simplify the problem, we restrict the agent to trade exclusively towards the target position. This means that for a given symbol (stock ticker) and date, it will either only buy (positive target position) or only sell (negative target position) over the course of that date. The agent receives signals that forecast short-term price movements and must learn to optimally time its buying/selling decisions. For example, if the signals indicate that prices are expected to drop in the short-term, one might imagine that an optimal agent would choose to delay buying in order to capture better entry points in the future. On the other hand, it may choose to buy immediately if a price rise is anticipated.
To support training, the agent needs an environment suitable for large-scale experimentation and evaluation. We built a market simulator using historical market data, allowing the agent to explore how different strategies for using trading signals affect its performance. This enables the agent to learn from simulated trading days.
With the simulation environment in place, the next question is how to learn the best policy. The main challenge is that financial rewards are very noisy. This can make it harder for the agent to estimate an accurate value function (expected future rewards), explore useful policies efficiently, and converge on a good strategy. Choosing the right learning algorithm is therefore important.
To establish a performance benchmark, we first implemented a simple hand-crafted, rule-based policy to act as our baseline.
This policy specifies that if market conditions look good (Directed Return > 0), then we should make steady progress by trading a bit of our position over time (Remaining Position / Remaining Time). But if we are running out of time (abs(Remaining Position) / Remaining Time > β * Vol) we should simply trade our remaining position. If neither is true – meaning there is plenty of time to get into our position but the market conditions aren’t favorable – we should do nothing (trade 0). Β * vol is a way of encoding that a higher-volatility symbol has a higher potential of giving us favorable future trading opportunities, and that we should thus be less jumpy to fill our full target position as we near the end of the day.
We then used this baseline to guide our approach to more advanced methods. To understand practical trade-offs in our context, we explored two prominent reinforcement learning paradigms with competing strategies. Off-policy methods, such as Soft Actor-Critic (SAC), prioritize data efficiency by reusing past experiences from a replay buffer. In contrast, on-policy methods, like Proximal Policy Optimization (PPO), update the agent strictly based on freshly collected data, which can lead to more stable and reliable policy improvements.
In our experiments, SAC presented an important practical problem: it tended to learn non-smooth policies, where relatively small changes in the state could produce large changes in the chosen action. In a noisy financial environment, this is particularly undesirable. We want our agent to learn a robust relationship between market conditions and trading behavior. If our policy instead exploits idiosyncratic noise patterns in the training simulations, then its in-sample performance may not translate well to out-of-sample situations.
Notice how in several of the decision-boundary plots above (such as the case of directed return 0), the policy learned spiky decision boundaries. Without a clear story for these spikes, the most likely explanation is that the policy was overfitting to noise in our trading data.
In an effort to ameliorate the likely overfit, we tried randomly initializing the states at the beginning of each episode. The hope was that this would incentivize exploration, prevent the agent from getting stuck on a single “trading path,” and encourage it to learn a more robust and generalizable policy. We can see in the plots below that random restarts do encourage more exploration (there is more spread/variance in the agent’s trading patterns), but it did not fix the core problem.
After our investigation into off-policy algorithms highlighted issues with stability and generalization, we turned to PPO (an on-policy algorithm) to address these challenges. Instead of training with a replay buffer of actions from stale policies, the on-policy strategy collects large batches of fresh data after each policy update. By using fresh, on-policy samples, the agent reduced bias in its policy updates. Simultaneously, averaging over the large batch of new data mitigated high reward variance. This resulted in significantly more stable training dynamics and, ultimately, a more performant final policy. For our specific application, we thus observed that the enhanced training stability of PPO outweighed the superior sample efficiency of off-policy algorithms.
Notice the clearly-defined trading decision boundaries achieved by PPO.
While standard PPO provided a stable learning foundation, its inherently memoryless architecture posed a limitation. In our problem domain, the environment is partially observable, meaning the latest observation is insufficient for optimal decision-making. To overcome this, we enhanced our agent by incorporating memory, creating what we’ll call a Memory-Augmented PPO.
Our final evaluation tested the trading profit and loss (PnL) of our algorithms, benchmarked against our human-crafted baseline. To assess each policy’s robustness under imperfect information, we simulated a spectrum of accuracy levels for the short-term signal and calculated the PnL for each condition.
Across a spectrum of short-term predictiveness levels, SAC outperforms our human-crafted baseline, PPO outperforms SAC, and Memory-Augmented PPO outperforms PPO (left plot). One note for future work is that the RL agents didn’t perfectly fill the target position in all out-of-sample situations (right plot). A fill fraction of 1.0 would be perfect, but there is some variance to our RL methods, and PPO is a bit biased to under-fill. Future work could better understand this outcome and further tune our reward to encourage even more reliable position-filling.
Overall, this project showed us why reinforcement learning in financial markets is such a nuanced research problem. Due to noisy rewards, imperfect signals, and partial observability, ideas that work well in other domains do not necessarily transfer directly to trading. Furthermore, the need to generalize to unseen market conditions creates practical and intellectually-interesting challenges. It did not suffice to simply apply an existing RL algorithm: much of the work was in understanding why particular approaches failed and adapting them to the structure of the problem.
That made this a particularly rewarding internship project. We had the opportunity to work across simulation, reinforcement learning, and quantitative trading, while testing our ideas against real market data. The project gave us a much better appreciation for both the potential and the difficulty of applying modern deep learning techniques to trading, and it left us with many more questions we would be excited to explore.
