A Novel Anti-Risk Method for Portfolio Trading Using Deep Reinforcement Learning

Abstract

In the past decade, the application of deep reinforcement learning (DRL) in portfolio management has attracted extensive attention. However, most classical RL algorithms do not consider the exogenous and noise of financial time series data, which may lead to treacherous trading decisions. To address this issue, we propose a novel anti-risk portfolio trading method based on deep reinforcement learning (DRL). It consists of a stacked sparse denoising autoencoder (SSDAE) network and an actor–critic based reinforcement learning (RL) agent. SSDAE will carry out off-line training first, while the decoder will used for on-line feature extraction in each state. The SSDAE network is used for the noise resistance training of financial data. The actor–critic algorithm we use is advantage actor–critic (A2C) and consists of two networks: the actor network learns and implements an investment policy, which is then evaluated by the critic network to determine the best action plan by continuously redistributing various portfolio assets, taking Sharp ratio as the optimization function. Through extensive experiments, the results show that our proposed method is effective and superior to the Dow Jones Industrial Average index (DJIA), several variants of our proposed method, and a state-of-the-art (SOTA) method.

1. Introduction

Portfolio trading aims to allocate resources in a way that maximizes returns while minimizing risk. Many prior studies have employed reinforcement learning (RL) algorithms to address a variety of stock trading problems and have proved both the potential and promising results of deep reinforcement learning (DRL) in financial markets [1,2,3,4,5].

RL combines the tasks of “forecasting” and “portfolio construction” into one integration step so as to closely combine the machine learning problem with the objectives of investors. The process that traders use to invest in stocks to maximize profits is the same as how the RL agents interact with the environment to obtain rewards and maximize the cumulative reward. RL can also easily consider important constraints such as transaction cost [6], market liquidity, and investors’ risk aversion.

However, the existing methods have some common limitations. When confronted with a complex market environment, agents frequently struggle to find the optimal strategy. Li [7] has proved that using feature preprocessing in a trading algorithm can obtain higher transaction performance. More crucially, the stock market will be influenced by political, economic, and natural disasters (such as COVID-19); in other words, the stock market has externalities, and the impact of external shocks will cause the stock price to rise or fall abnormally.

In this paper, we employed the autoencoder (AE) network in the RL framework. AE has been used for noise resistance training in many fields to improve robustness. Qi et al. [8] proposed a robust stacked autoencoder (R-SAE) based on maximal correntropy criteria (MCC). The proposed method outperforms other machine learning methods on the MNIST benchmark dataset in noise-proof ability. Li et al. [9] introduced a deep sparse autoencoder with lossless and nonnegative constraints, which increases the model’s robustness and the performance of anti-interference. Therefore, we propose a novel anti-risk portfolio trading system based on DRL. The trading system consists of a stacked sparse denoising autoencoder (SSDAE) and an actor–critic based RL agent. The noise resistance training is performed on the financial data to extract the robust features, and the actor–critic algorithm is based on advantage actor–critic (A2C) and consists of two networks in which the actor network learns and implements the investment policy, which is then evaluated by the critic network to determine the best action plan by continuously redistributing various portfolio assets to maximize the expected return on investment. All the sample data for this study were collected from Yahoo Finance. We tested our method on 24 Dow Jones stocks with sufficient liquidity by comparing different state preprocessing methods. Experimental results show that the proposed method is effective and better than a SOTA algorithm. The contributions of this work are three-fold: (1). we propose a state preprocessing network that employs SSDAE to perform anti-noise training on stock data to extract the robustness features so as to improve the robustness of agent transaction decision making; (2). we have performed extensive experiments to test the performance of our proposed method and demonstrate that it outperforms other state preprocessing methods, such as PCA&DWT [7], LSTM-AE [10], RSAE, Dow Jones Industrial Average index (DJIA), and a SOTA method [11]; and (3). the experimental results show that our proposed method performs better in balancing benefits and risks and demonstrates the importance of state preprocessing in reinforcement learning trading.

2. Preliminary

2.1. Problem Setup

We formulate the portfolio trading process as a generalized Markov decision process (MDP), and the stock market state was reconstructed using the autoencoder technology. MDP is defined as the tuple $\left(\mathcal{S}, \mathcal{A}, p, r\right)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $p\left(s_{t+1}|s_t,a_t\right)$ represents the probability of selecting action $a_t\in \mathcal{A}$ from $s_t\in \mathcal{S}$ to the next state $s_{t+1}\in \mathcal{S}$, and $r\left(s_t,a_t,s_{t+1}\right)$ represents the direct reward of taking action at state s and arriving at the new state $s_{t+1}$. The goal of RL agent is to maximize the cumulative income while controlling the transaction cost and risk cost.

The goal was to find an optimal policy ${\pi }^{*}$:

$${\pi }^{*}=\underset{\pi }{\arg max}{\displaystyle \sum }_{t=0}^T{\mathrm{E}}_{\left(s_t,a_t\right)\sim {\rho }_{\pi }}\left[{\gamma }^{t}r\left(s_t,a_t\right)\right]$$

(1)

where ${\rho }_{\pi }$ indicates the distribution of state–action pairs that the RL agent will encounter under the control of policy $\pi$.

For each policy $\pi$, you can define its corresponding Q-value function:

$$V_{\pi }\left(s_t\right)={\mathrm{E}}_{a_t\sim \pi }\left[Q_{\pi }\left(s_t,a_t\right)\right]$$

(2)

2.1.1. State Space

The state space of our trading environment consisted of four parts [$b_t,h_t,p_t,X_t$]. Each letter is defined as follows:

• $b_t$ ∈ ${ℝ}_{+}$:available balance at current time step t.
• $h_t$∈ ${\mathbb{Z}}_{+}^n$: shares owned of each stock at current time step t.
• $p_t$ ∈ ${ℝ}^n$: close price of each stock at current time step t.
• $X_t$ ∈ ${ℝ}^n$: encoding features at current time step t.

2.1.2. Action Space

The agent’s job in the portfolio trading problem is to figure out how much of each stock to buy and sell. We do not enable investors to short assets in our action space; we do not allow investors to borrow assets and then return them in the future; we do not allow investors to borrow assets and then return them in the future. For a single stock, our action space was defined as $\left\{-k,-1, 0, 1,\dots , k\right\}$, where k and −k represent the number of shares we can buy and sell, and $k\le h_{\max }$.$h_{\max }$ is a predefined parameter that sets the maximum number of shares for each buying action; the action space was normalized to $\left[-1, 1\right]$, which also means that our action space was continuous, and after the action selection of each state, we executed the sell action first and then the buy and hold action.

2.1.3. Reward Function

The reward function is the change in portfolio value caused by action in states $s_t$ and $s_{t+1}$.

$$r\left(s_t,a_t,s_{t+1}\right)=R_{t+1}-R_t$$

(3)

where $R_t=b_{t+1}+p_{t+1}^T{h}_{t+1}-c_t$ denotes the portfolio values at t + 1 and t, respectively. The portfolio value is the total stock value plus the balance, and $c_t$ is the transaction cost.

The reward function can also be set directly to the trade return [12,13], as follows:

$$T{r}_t^s=ln\left(\frac{\left[R_t-R_{t-s}\right]}{R_{t-s} }\right)$$

(4)

where $T{r}_t^s$ represents the realized logarithmic rate of the trade return in a period of time t, and the length of the period is s.

Each deal entails transaction charges, which come in a variety of forms and are charged differently by different brokers. We assume our transaction cost is $c_t=p^T{{\displaystyle \sum }}_{d=1}^D\left|k_t\right| \times$ $\lambda$. We also use the existing financial technical indicators to trade off the return and risk, such as the Sharpe ratio [1,14,15].

$$SH{r}_t=\frac{T{r}_t^s-r_f}{{\left(\widehat{{\sigma }_t}\right)}^2}$$

(5)

where $T{r}_t^s-r_f$ is the excess return in a period of trading; $(\widehat{{\sigma }_t})=\sqrt{252}\times \frac{1}{n-1}{\displaystyle \sum }_{t=i}^{n+i}(T{r}_t^1-\frac{1}{s}T{r}_t^s)$ is the standard deviation of the daily trade return multiplied by 252^1/2 (252 is the approximate number of trading days in a year and is the frequently used factor in the financial field), $i=\left\{1,\dots ,T\right\}$; $r_f$ is the risk-free rate of return; and $T{r}_t^1$ is the daily return.

2.2. Trading mode

Assuming that we have $1 million and trade in D stocks, each stock in the portfolio will be bought gradually, and the remaining funds after daily trading will be the balance b.

The trading environment assumptions are:

• The trading volume of the agent is very small compared to the size of the whole market, so the agent’s trades do not affect the state transition of the market environment.
• The liquidity of the market is high enough that the orders can be rapidly executed at the close price.
• The transaction cost is a fixed proportion $\lambda$ of the trade volume of the day.

3. Methodology

In this section, we present how we depicted stock markets, an overview of SSDAE, and an optimized method of DRL.

3.1. Depiction of Stock Markets

The stock market has the characteristics of a complex description, being non-stationary, and low signal–to–noise ratio. The key issue in stock trading is to grasp the correct trading moment and perform the correct trading behavior according to the market situation. As for the stock market, the commonly employed data is regular sequences, such as open price, close price, high price, low price, and volume (OHCLV) data. In order to fully describe the market, we added financial technical indicators. The technical indicators are very close to the market, respond quickly to the short-term changes of the market, and are intuitive and clear. The technical indicators that are currently popular in the stock market can be classified into five categories: moving averages, volatility, trend, momentum, and volume. Our initial features consisted of 11 + 5 features selected from the above five categories, as well as OHCLV data.

Table 1. Summary of financial technical indicators.

Type	Financial Technical Indicators
Moving averages	Simple Moving Average (SMA)
Moving averages	The Exponential Moving Average (EMA)
Volatility	Average True Range (ATR)
Volatility	Williams Percent Range (WPR)
Trend	Moving Average Convergence Divergence (MACD)
Trend	Commodity Channel Index (CCI)
Momentum	Relative Strength Index (RSI)
Momentum	Awesome Oscillator (AO)
Momentum	TSI Indicator (TSI)
Volume	Force Index (FI)
Volume	On-Balance Volume (OBV)

shows all the technical indicators we used.

Simple moving average (SMA) is a fundamental technical indicator that is defined as

$$SM{A}_t=\left(c_{t-n}+\cdots +c_{t-1}+c_t\right)/n$$

(6)

where $c_t$ represents the close data over a given time period and n represents the time span.

The exponential moving average (EMA) is a trend-following indicator that emphasizes recent data above historical data. It is defined as an exponentially decreasing weighted moving average as

$$EM{A}_n\left(c_t\right)=\frac{2c_t+\left(n-1\right)EM{A}_n\left(c_{t-1}\right)}{n+1}$$

(7)

The moving average convergence divergence (MACD) is an indicator that indicates the difference between a long-term moving average and a short-term moving average. This indicator keeps the advantages of moving averages while also avoiding the generation of erroneous trade signals. MACD can be expressed as follows:

$$MACD=2\ast \left(DIF-DEA\right)$$

(8)

$$DI{F}_t=EMA{\left(12\right)}_t-EMA{\left(26\right)}_t$$

(9)

where DEA is the arithmetic mean of DIF over the time span. and EMA is defined in Equation (7).

The Williams percentage range (WPR) is a dynamic technical indicator to judge whether the market is overbought/oversold. WPR is very similar to the random oscillator. The only difference is that the scale of WPR is from top to bottom, while the random oscillator has internal smoothing.

$$WPR=\frac{\left(H{H}_{t-n}-c_t\right)}{\left(H{H}_{t-n}-L{L}_{t-n}\right)}\times 100$$

(10)

where $H{H}_{t-n}$ and $L{L}_{t-n}$ represent the maximum value of the highest price and the minimum value of the lowest price in the past n time intervals, respectively.

The relative intensity index (RSI) is a price-following oscillator:

$$RSI=100-100/\left(1+\frac{{{\displaystyle \sum }}_{i=0}^{n-1}{u}_{t-i}}{n}/\frac{{{\displaystyle \sum }}_{i=0}^{n-1}{d}_{t-i}}{n}\right)$$

(11)

where $u_t$ represents the upward price change at time t, and $d_t$ represents the downward price change at time t.

The commodity channel index (CCI) measures the deviation between commodity prices and their average statistical prices. The higher the index, the higher the price than the average. The lower the price, the lower the average price.

$$CCI=\left(T{P}_t-SMA \left(TP, n\right)\right)/0.015MD$$

(12)

where TP (typical price) represents the typical price, and its calculation formula is $T{P}_t=\left(h_t+l_t+c_t\right)/3$; $h_t$ represents the highest price at time t; $l_t$ represents the lowest price at time t; MD (mean deviation) is the average deviation of TPl and $v_t$ represents the trading volume at time t.

The on-balance volume (OBV) is to quantify the trading volume, make a trend line, cooperate with the stock price trend line, and speculate on the market atmosphere from the change of price and the increase or decrease of trading volume.

$$\mathcal{O}\mathcal{B}{V}_t=\{\begin{array}{l}\mathcal{O}\mathcal{B}{V}_{t-1}+v_t, if c_t>c_{t-1}\\ \mathcal{O}\mathcal{B}{V}_{t-1}-v_t , if c_t<c_{t-1}\\ \mathcal{O}\mathcal{B}{V}_{t-1} , \mathrm{otherwise}\end{array}$$

(13)

where $v_t$ represents the trading volume at time t.

The awesome oscillator (AO) calculates the difference between the simple moving average of phase 34 and phase 5. The simple moving average used is not calculated using the closing price but using the midpoint of each line. The AO is usually used to identify trends or predict possible reversals.

$$AO=SMA\left(MEP, 5\right)-SMA\left(MEP, 34\right)$$

(14)

where MEP is median price, and the formula is MEP = (high + low)/2

The force index (FI) is used to indicate the strength of an upward or downward trend. The index measures the strength of bull potential at each increase and bear potential at each decrease.

$$FI=EMA\left(\left(c_t-c_{t-n}\right)\ast v_t,n\right)$$

(15)

The true strength index (TSI) is a technical momentum oscillator used to identify trends and reversals. The indicator may be useful for determining overbought and oversold conditions, indicating potential trend direction changes via centerline or signal line crossovers and warning of trend weakness through divergence.

$$TSI\left(c_t,r,s\right)=\frac{EMA\left(EMA\left(mtm,r\right),s\right)}{EMA\left(EMA\left(\left|mtm\right|,r\right),s\right)\times 100}$$

(16)

where $mtm=c_t-c_{t-1}$, r is usually set to 25, and s is set to (12).

3.2. Overview of SSDAE

Raw financial data can only indicate patterns or characteristics of market dynamics to a limited extent, so it is not ideal for direct input to RL agents. However, putting all price data and technical indicators into the state space will lead to complex design and delay. Delay is a key factor in algorithmic trading. Most studies simplify the state space by using only OHCLV data [14]. A better solution is to reduce the dimension through the feature extraction method. Therefore, we propose using an autoencoder network to extract features from raw financial data and encoder features with robustness information. Figure 1 depicts the SSDAE autoencoder network’s structure. $x_i$ indicates the input data of the ith node, and $h_k^{\left(i\right)}$ indicates the input data of the kth node of the Nth hidden layer. The arrow in the network represents the connection weight between two adjacent layer nodes.

3.2.1. Sparse Denoising Autoencoder (SDAE)

Considering that the stock market is also influenced by external factors such as social networks, politics, and natural disasters (such as COVID-19); in order to make RL agents more robust after training, we naturally introduced denoising autoencoder (DAE)) [15] and sparse autoencoder (SAE) to reconstruct original financial data, the first to eliminate accidental events and the second to minimize external factors.

The Autoencoder (AE) [16] is a type of unsupervised machine learning; the AE can be regarded as a three-layer neural network. The feedforward functions of the AE can be defined as follows:

$$\begin{array}{c}y=f_{\theta }\left(x\right)=s\left(Wx+b\right)\\ z=g_{\theta }\left(y\right)=s\left(W^{\prime }y+b^{\prime }\right)\end{array}$$

(17)

where $s\left(x\right)$ is the activation function. In this paper, we use the sigmoid function $s\left(t\right)=\left(1+\exp \left(-t\right)\right)-1$; W is the weight matrix; b is the bias vector; and $W^{\prime }$, $b^{\prime }$, $W$, and $b$ are initialized with random values.

The AE only relies on the strategy of minimizing reconstruction errors for network training, which may cause the network to learn only the copy of the original input. Considering that the stock market is also influenced by external factors such as social networks, politics, and natural disasters (such as COVID-19), in order to make RL agents more robust after training, we naturally introduced a denoising autoencoder (DAE)) [15] and sparse autoencoder (SAE) to reconstruct original financial data; the first is to eliminate accidental events, and the second is to minimize external factors.

Based on the AE, DAE adds noise to the input layer data (randomly zeroing the input layer nodes) to make the whole encoder network more robust. It passes on the original input vector $x$ to noise processing result $\tilde{x}$. The random selection unit was forcibly set to 0 according to a certain proportion, and then used the data $\tilde{x}$, so as to train better anti-noise ability.

The SAE adds conditional restrictions on the AE, ${\widehat{\rho }}_j=\rho$. The network units were randomly activated to prevent the encoder from working with a low reconstruction error at each point, which also makes the whole encoder network more robust. $\rho$ is the sparsity parameter (generally a smaller value close to 0). In order to achieve this limitation, the objective function introduced an additional penalty factor, namely relative entropy (Kullback–Leibler divergence), which is a method to measure the difference between two distributions. The principle is shown in:

$$\mathrm{KL}\left(\rho \parallel {\widehat{\rho }}_j\right)=\rho \log \frac{\rho }{{\widehat{\rho }}_j}+\left(1-\rho \right)\log \frac{1-\rho }{1-{\widehat{\rho }}_j}$$

(18)

where KL is expressed in $\rho$, ${\widehat{\rho }}_j$ is the relative entropy between two Bernoulli random variables with mean value. When ${\widehat{\rho }}_j=\rho$, $\mathrm{KL}\left(\rho \parallel {\widehat{\rho }}_j\right)=0$, and the size of KL increases with the difference between $\rho$ and ${\widehat{\rho }}_j$ increasing monotonically; and when ${\widehat{\rho }}_j$ approaches 0 or 1, the relative entropy becomes infinite, so minimizing the penalty factor can make ${\widehat{\rho }}_j$ approach $\rho$.

Given the raw financial training data set $D=\left\{\left(x_i,y_i\right)\right\}$, where $i=1,2,\dots ,N$ and $N$ denotes the total number of training samples, the SDAE was trained by a backpropagation algorithm to minimize the following sparsity regularized reconstruction loss function:

$$\begin{gathered} J_{\mathrm{SDAE}}\left(\mathit{w},\mathit{b}\right)=\frac{1}{m}{\displaystyle \sum }_{i=1}^m\left(\frac{1}{2}\parallel h_{w,b}{\left(\tilde{x}\right)}^{\left(i\right)}-y^{\left(i\right)}{\parallel }_2^2\right)+\beta {\displaystyle \sum }_{j=1}^s\mathrm{KL}\left(\rho \parallel {\tilde{\rho }}_j\right) \\ +\frac{\lambda }{2}\left(\parallel W{\parallel }_F^2+\parallel W^{\prime }{\parallel }_F^2\right) \end{gathered}$$

(19)

where $\beta$ is the sparsity term parameter, and λ is the weight decay term parameter.

3.2.2. Stacked Sparse Denoising Autoencoders (SSDAE)

To enhance the learning ability of the autoencoder network, the method of increasing the number of network layers was generally adopted. The error back propagation algorithm was used in the majority of our existing shallow neural networks, including ordinal autoencoder networks. When the number of layers is increased, the propagation error decreases, and the gradient disappearance problem becomes more frequent. This issue was avoided by using a stacked autoencoder. Figure 1 shows the SSDAE network structure of N hidden layers. At present, the SSDAE network still adopts the greedy layer-by-layer training method [16], which mitigates the step dispersion phenomena to a certain extent. That is, the first hidden layer is pre-trained. Then, the output of the first layer is used to pre-train the second layer. This procedure is repeated for the remaining layers. After all layers have been pre-trained, the network used for encoding in each layer is erased, leaving only the decoding network of each layer. Then, using the BP algorithm, all layers are fine-tuned until the network’s ideal parameters are reached. The SSDAE network converts the original input into the nth layer’s extracted features. The SSDAE network’s loss function during the fine-tuning stage is as follows:

$$J_{\mathrm{SSDAE}}\left(\mathit{w},\mathit{b}\right)=\frac{1}{m}{\displaystyle \sum }_{i=1}^m\left(\frac{1}{2}\parallel h_{w,b}{\left(\tilde{x}\right)}^{\left(i\right)}-y^{\left(i\right)}{\parallel }_2^2\right)+\frac{\lambda }{2}{\displaystyle \sum }_{l=1}^{2l}\left(\parallel W_l{\parallel }_F^2\right)$$

(20)

where $W_l$ is the weight of the lth layer in the SSDAE. The sparsity term was omitted because the pretrained weights were employed as regularization to our network.

Considering that some studies have focused on the accuracy of AE-predicted and -reconstructed financial sequence data [17,18,19,20,21], and the effectiveness of feature extraction can only be judged according to the results of strategy performance, we ignored the accuracy verification of AE reconstruction data in this paper.

3.3. Optimization via Reinforcement Learning—Advantage Actor–Critic Learning

There are currently only a handful of quantitative studies that employ the actor–critic algorithm. The authors of [11] conducted a comparison of three actor–critical algorithms: proximal policy optimization (PPO), advantage actor critical (A2C), and deep deterministic policy gradient (DDPG). Experimental results indicated that A2C outperforms other algorithms. As a result, this paper employed A2C and implemented it using stable-baselines3 [22]. The framework of our proposed model is described in Figure 2.

The A2C algorithm [23] proposed by OpenAI is the synchronous variant of Google’s Asynchronous Advantage Actor Critic (A3C) algorithm [24]. It has been demonstrated that A2C performs similarly to A3C but with less implementation and execution complexity. The A2C is different from other actor–critic algorithms in that it improves the policy gradient updates. A2C uses the advantage function instead of the original return in the critical network, which can be used as an indicator to measure the quality of the selected action value and the average value of all actions. In addition to estimating the advantage function, the critic network also calculates the value function. This means that by evaluating how excellent an action is and how good it can be at the same time, the policy network’s large variance is decreased, and the model becomes more robust. In large batch sizes, synchronous gradient updates are more cost-effective, faster, and superior. The A2C’s stability properties make it ideal for stock trading.

We updated our policy based on the value function:

$$\nabla J_{\theta }\left(\theta \right)=\mathbb{E}[{\displaystyle \sum }_{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }\left(a_t\mid s_t\right)A\left(s_t,a_t\right)$$

(21)

where ${\pi }_{\theta }\left(a_t\mid s_t\right)$ is the policy network. $A\left(s_t,a_t\right)$ is the advantage function and can be written as

$$\begin{array}{cc}& A\left(s_t,a_t\right)=q_{\pi }\left(s_t,a_t\right)-V_{\pi }\left(s_t\right),\\ \mathrm{or}& \\ & A\left(s_t,a_t\right)=r\left(s_t,a_t,s_{t+1}\right)+\gamma V_{\pi }\left(s_{t+1}\right)-V_{\pi }\left(s_t\right)\end{array}$$

(22)

where $V_{\pi }\left(s\right)$ is the value expectation of the executor network in a certain state, $q_{\pi }\left(s,a\right)$ is the maximum value obtained by selecting action a in a certain state, $V_{\pi }\left(s\right)\le$ $q_{\pi }\left(s,a\right)$, and updated policy ${\pi }^{\prime }\left(s\right)=argma{x}_a{q}_{\pi }\left(s,a\right)$. The following is evidence of strategy improvement:

$$\begin{array}{ll}& v_{\pi }\left(s\right)\le q_{\pi }(s,a)=E[R_{t+1}+\gamma v_{\pi }(S_{t+1}\left)\right|S_t=s,A_t={\pi }^{\prime }\left(s\right)\left)\right]\\ =& E_{\pi }[R_{t+1}+\gamma v_{\pi }(S_{t+1}\left)\right|S_t=s]\le E_{\pi },[R_{t+1}+\gamma q_{\pi }(S_{t+1},{\pi }^{\prime }(S_{t+1}\left)\right)|S_t=s]\\ =& E_{\pi }[R_{t+1}+\gamma E_{\pi },(R_{t+2}+\gamma v_{\pi }\left(S_{t+2}\right)\left|S_{t+1}\right]|S_t=s]\\ =& E_{\pi }[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{v}_{\pi }(S_{t+2}\left)\right|S_t=s]\\ \le & E_{\pi }[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+{\gamma }^3{v}_{\pi }(S_{t+3}\left)\right|S_t=s]\cdots \cdots \le v_{{\pi }^{\prime }}(s)\end{array}$$

(23)

According to the greedy strategy of $a=argma{x}_a{q}_{\pi }\left(s,a\right)$, the new strategy obtained by updating all states must make each state’s value equal to or greater than its value before the update. As demonstrated above, the strategy of RL must be improved after each round of renewal.

4. Experiments

4.1. Dataset Descriptions

We provide a summary of a real-world datasets of our experiments in

Table 2. Training, validation, and test sets.

Market	Start Date	End Date	Type
The US	2014/01	2019/01	Training set
	2019/01	2020/01	Validation set
	2020/01	2022/01	Test set

. All the data used in this work is available on Yahoo Finance and Google Finance.

4.2. Evaluation Metrics

We used six performance measurement metrics to evaluate the back test results:

• Cumulative return (CR): is the total amount earned by an RL agent during the trading period, excluding transaction costs.
• Maximum drawdown (MDD) [25]: refers to the maximum loss percentage from peak to trough during the trading period.
• Sharp ratio (SR): Sharp ratio [26] is used to measure the rate of return that trading strategies can achieve when confronted with a unit of risk. It is the most commonly used mainstream standardized metric for evaluating portfolio strategies’ performance. Equation (5) has illustrated the calculating formula.
• Calmar ratio (CMR): Calmar ratio [27] is used to measure the risk by using the concept of MDD.
• Alpha: Alpha value is used to measure the excess return obtained by the model relative to the benchmark within the trading range. The greater the alpha value, the more capable it is of generating additional returns in comparison to the benchmark. The calculation process of alpha value is shown in:

  $$\mathrm{Alpha}=R_p-\left[R_f+{\beta }_p\left(R_m-R_f\right)\right]$$

  (24)

  where $R_p$ is the yield of the model, ${\beta }_p$ is the beta value of the model, and $R_m$ is the yield of the benchmark strategy.
• Beta: Beta value is an indicator used to assess the systemic risk of the model relative to the benchmark. If the beta value is greater than one, the volatility of the model is greater than the benchmark; if the beta value is less than one, the volatility of the model is less than the benchmark; and if the beta value is equal to one, the volatility of the model is the same as that of the benchmark. The computation method is illustrated below:

  $$\mathrm{Beta}=\mathrm{Cov}\left(R_p,R_m\right)/{\sigma }_m^2$$

  (25)

  where $\mathrm{Cov}$ is the covariance, and ${\sigma }_m^2$ is the variance of the benchmark strategy.

CR and alpha values are to evaluate the profitability of RL agent separately; MDD and beta values are used to measure the ability of risk control; SR is a risk adjusted return, which can consider the performance, comprehensive income, and risk of the strategy.

4.3. Experimental Details

Our stock portfolio trading environment was based on OpenAI gym [28]. In order to better test the anti-risk performance of strategies, we chose to test the time set, including the time period of the COVID-19 outbreak. Finally, the parameters that needed to be set in advance are summarized in

Table 3. Summary of RL agent’s parameters.

Parameters	Value
trading window size (s)	20
risk-free rate of return $\left(r_f\right)$ A2C_learning_rate	1% 0.0007
A2C_ n_steps	5

and

Table 4. Summary of SSDAE parameters.

Structure Parameters	Value	Learning Parameters	Value
Hidden layer 1	12	Batch size	100
Hidden layer 2	9	Epoch of pre-training	50
Hidden layer 3	6	The sparsity	10−8
Hidden layer 4	3	The regularization term	10−5

. To prevent inadequate learning such as local minimum and overfitting, dropout and gradient were applied.

4.4. Discussion of Comparison Methods

It was also difficult to compare the results of our proposed method with other studies due to the difference in transaction variety, mode, environment, and RL algorithm. All comparative methods are trained and traded under the same trading environment, trading rules, and parameters. For comparison with our proposed model, in addition to the DJIA index, we selected an advanced portfolio management strategy from the current literature to compare the performance that was an open-source method that does not use feature extraction and improves performance by integrating multiple agents [11]. We also implemented several variants or simplified versions of SSDAE to investigate the roles of several key components: we chose RSAE [29], which lacked a sparse and denoise network compared with our proposed method; LSTM-AE [10], a method of extracting time series information for financial series; and a machine learning method using PCA and wavelet denoising technology [7].

4.5. Results and Analysis

As shown in Figure 3 and

Table 5. The results in different RL methods.

	CR (%)	MDD (%)	SR	CMR	Alpha	Beta
SSDAE	88.5	−21.7	1.37	1.71	0.28	0.69
RSAE [29]	67.7	−33.9	0.99	0.87	0.16	1.01
LSTM-AE [10]	62.3	−36.3	0.91	0.75	0.14	1.06
PCA&DWT [7]	36.3	−33.1	0.71	0.05	0.05	0.94
SOTA [11]	49.1	−29.2	0.89	0.76	0.10	0.90
DJIA	26.9	−37.1	−0.17	0.35	0.00	1.00

, the SSDAE agent performs best in all measurement indicators. It was observed that our proposed method outperforms the DJIA index and the SOTA method without state preprocessing in all measurement indicators. This demonstrates the advanced nature of our proposed method in RL trading algorithm. Compared with the variants (LSTM-AE, PCA&DWT) or simplified version (RSAE), our method shows that it does better in balancing benefits and risks: In terms of CR and Alpha, the SSDAE agent outperforms other state preprocessing approaches, hitting 88.5% (CR) and 0.28 (alpha). In the beta values and MDD (%), they reached 0.69 and −21.7, respectively. Finally, from the perspective of comprehensive income and risk, there is no doubt that SSDAE agent performs best in terms of SR and CMR, which are 1.37 and 1.71 respectively.

The most significant feature of our proposed method was the risk control ability compared with other methods. This shows that extracting robust features from anti-noise training can improve the anti-risk ability of trading strategy. The same as the conclusion of the research on the prediction of financial sequences using an autoencoder method [30], our experiment also found that the use of autoencoder technology can reduce the noise of the features of stock markets. In general, noise resistance training is very important for the RL trading system.

5. Conclusions

In this paper, according to the characteristics of the exogenous nature, complexity, and noise of financial time series data, we proposed an anti-risk portfolio trading system based on DRL. It consists of an SSDAE autoencoder and an advantage actor–critic based RL agent. By comparing the trading performances of different state preprocessing methods, including two variants methods, a simplified version, a SOTA method, and the benchmark DJIA, we found that the impact of anti-noise training on the trading strategy trained by RL agent is positive and promising. Results also illustrate the importance of state preprocessing in reinforcement learning. The advantages of the proposed method are its expansibility, applicability, flexibility, and speed to train, but its interpretability needs further discussion. We will explore using better risk measurement tools such as value-at-risk [31] and conditional-value-at-risk [32] to design newly effective reward functions and, considering the excessive trading volume of the strategy, optimize the transaction cost in the trading process, which is also an issue that needs attention.