Applying reinforcement learning algorithms for profitable strategies in a stock market simulator

Abstract

Financial time series present unique characteristics that an investor, analyst or algorithmic trader has to take always into account. One of the defining features is their inherent volatility and non-linearity. Unlike many other forms of data financial markets are influenced by economic indicators, geopolitical events and investor sentiment. These factors can cause sudden and unpredictable price movements resulting in extreme volatility and this violates the assumption of linearity, making traditional statistical methods less effective. They also exhibit auto-correlation where the value of a variable at one time point is correlated with its value at a previous point. This auto-correlation can persist over multiple time lags, leading to trends in the data. Identifying and modeling these trends is crucial for making informed investment decisions. They also suffer from "fat-tailed" distributions which mean there are frequent market crashes and price swings that could not be expected in a normal distribution. Financial time series are often non-stationary, with statistical properties like mean, standard deviation, skewness, kurtosis changing over different periods. Financial markets are not only influenced by macroeconomic factors but also by their own microstructure, which includes factors like bid-ask spreads, trading volumes, and market orders. Understanding and modeling market microstructure is crucial for accurately capturing its dynamics. The main problem in financial markets is how to make profitable investment strategies with the lower risk that maximize returns. In this thesis we examine the use of Reinforcement Learning as a tool of decision making which can lead us to strategies with better performance than buy and hold the underlying asset. We create ten out-of-sample synthetic time series based on standard normal distribution and simulate a trading game where we evaluate the effectiveness of major two RL algorithms Q-learning and REINFORCE. Our trading simulations demonstrated that the performance of reinforcement learning algorithms, Q-learning and REINFORCE can be influenced by the stochastic nature of underlying data. REINFORCE showed an advantage in terms of P/L (profit or loss) for most seeds, while Q-learning displayed greater consistency in risk-adjusted returns. The unexpected success of a buy-and-hold strategy for specific seeds underscores the importance of considering diverse approaches in trading scenarios. These findings emphasize the dynamic nature of algorithmic trading, where the choice of the optimal strategy depends on the specific characteristics of the underlying data. Finally the construction of a portfolio of the ten single equity curves showed acceptable performance while minimizing the risk. The results seemed quite promising.