Optimization of a Long Short-Term Memory Algorithm for Predicting Stock Prices: A Case Study of the State-Owned Enterprises Banking Sector ^†

Abstract

The ease of internet access allows people to utilize investment products anywhere and anytime. Stocks are one such investment product, allowing individuals to own a portion of a company. One of the factors influencing stock price fluctuations is technical analysis, which relies on previous stock price data. The Long Short-Term Memory (LSTM) algorithm has the advantage of storing data or information over the long term by using gates within LSTM, including the forget gate, input gate, and output gate. Three datasets of stock prices from state-owned enterprise banks, namely Bank BRI, Bank BNI, and Bank Mandiri, were used, totaling 4479 rows of data across the three banks. From these datasets, the optimal values for the LSTM algorithm were obtained, with Mean Absolute Percentage Error (MAPE) values ranging from 0.01 to 0.025 and R² values ranging from 0.8 to 0.96.

1. Introduction

The digitization process has facilitated the ease of online transactions and enabled people to invest using the internet [1]. Individuals can invest in deposits, mutual funds, stocks, and other investment products online. The increasing popularity of investment products, especially stocks, has become a trend across various demographics, from students and workers to the elderly, as there is no age limit for purchasing stocks. The requirements for buying stocks are having an Identification Card (KTP) and a Taxpayer Identification Number (NPWP). A KTP can be obtained when a person is 17 years old, while an NPWP is acquired when someone starts working and is used for tax administration purposes [2].

Stocks represent ownership in a company, measured in lots, with one lot consisting of 100 shares [3]. Individuals can purchase a minimum of one lot or 100 shares, but the transaction must be conducted through a securities company or broker acting as an intermediary [2]. According to the Indonesia Stock Exchange (IDX) report in 2023, stock purchases in Indonesia, particularly in Jakarta, have increased significantly since 2020. The closing price per share at the beginning of 2020 was approximately IDR 4500 and rose to around IDR 7250 in the first quarter of 2022. However, stock prices declined at the end of 2022, with the closing price around IDR 6125 in the second quarter of 2023.

The increase and decrease in stock prices in this report indicate that stock prices exhibit a rising and falling pattern. This makes stock price data suitable for predictive research. State-Owned Enterprises (SOEs) in Indonesia have six banking companies: PT Bank Negara Indonesia Tbk, PT Bank Rakyat Indonesia Tbk, PT Bank Mandiri Tbk, PT Bank Tabungan Negara Tbk, PT Bank Syariah Indonesia Tbk, which is a merger of three banks, and PT Bank Raya Indonesia Tbk, a subsidiary of PT BRI Tbk [4]. This study focuses on three independent SOE banks: PT BNI Tbk (BBNI.JK), PT BRI Tbk (BBRI.JK), and PT Bank Mandiri Tbk (BMRI.JK).

The Long Short-Term Memory (LSTM) algorithm is a modification of the Recurrent Neural Network (RNN) that addresses the vanishing gradient problem, where weights fail to update the RNN cell without reducing the accuracy produced by the RNN. LSTM is trained using the backpropagation algorithm, managing information flow with cell states and gates in LSTM.

Research by Khalis Sofi et al. (2021) compared the Linear Regression, LSTM, and GRU algorithms in predicting KEJU stock prices in 2021 [5]. Among the three algorithms used, the GRU algorithm showed the best performance with the lowest RMSE of 0.034, MSE of 0.001, and MAE of 0.024 [5]. However, in a study by Anggi Putri Meriani and Alam Rahmatulloh (2024), which compared the Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) algorithms in predicting gold prices, it was concluded that the LSTM algorithm could predict better than the GRU algorithm, with an MAE of 0.0389, RMSE of 0.0475, and MAPE of 5.2047% [6].

Long Short-Term Memory (LSTM) has multiple cells with three gates—input, forget, and output—to manage and update long-term information. Meanwhile, the Gated Recurrent Unit (GRU) is a simpler version with two gates: reset and update. GRU is easier to implement and more efficient, but LSTM is more powerful and flexible despite being more complex and prone to overfitting. This research focuses on optimizing LSTM due to its flexibility and a higher number of parameters compared to GRU [7].

A study by Prismahardi Aji Riyantoko et al. (2020) titled “Analysis of Bank Stock Price Prediction Using Long-Short Term Memory (LSTM) Algorithm” showed good results using data from Bank BRI, Bank BNI, Bank BTN, and Bank Mandiri [8]. This study compared three LSTM optimization models: SGD, Adam, and RMSprop. SGD optimization showed a loss value with RMSE between 0.0013 and 0.0023 and an accuracy of 49% to 61%. Adam optimization had an RMSE value of 0.0004 to 0.0006 and an accuracy of 89% to 95%. RMSprop showed an accuracy of 45% to 95% and an RMSE of 0.0004 to 0.0007. The conclusion of the research was that Adam optimization had the lowest loss value and highest accuracy among the three models [8].

This study aims to optimize the Long Short-Term Memory (LSTM) algorithm in predicting the daily closing prices of State-Owned Enterprise (SOE) banking companies’ stocks using Adam optimization. The data used are the stock prices of PT BRI Tbk, PT BNI Tbk, and PT Bank Mandiri Tbk from Yahoo Finance for the period from 1 January 2018 to 29 December 2023. Evaluation is conducted using Mean Absolute Percentage Error (MAPE) and R-Squared (R²) values.

2. Theoretical Framework

2.1. Stock

Stocks represent an individual’s ownership in a company, making them shareholders. Stock transactions are conducted through the capital market, primarily the secondary market, where transactions are carried out between investors at prices determined by them through securities intermediaries [9]. Suratna et al. (2020) explain that funds from stock purchases are invested in the company’s operational activities [10]. Shareholders receive profits in the form of capital gains (the difference between the buying and selling prices) and dividends (profits based on the company’s performance). However, there are risks such as capital loss (selling price lower than the buying price) and liquidation risk (last priority in case of company bankruptcy) [10].

Mohamad Kafil (2019) explains that prediction is a systematic process of estimating future events based on past and present information [11]. The goal of prediction is to minimize the error margin of the estimate. Therefore, prediction can be concluded as forecasting based on the analysis of past data [11].

It can be inferred that stock prediction is the process of forecasting future stock prices using past stock price data. Conducting stock transactions without analysis can be equated to gambling due to the inherent uncertainty [12]. According to M. Abdul Dwiyanto Suyudi et al. (2019), although stocks are an investment that promises significant profits, they also carry substantial risks of loss [13]. Therefore, predictive analysis is necessary to minimize the risk of losses [13].

2.2. Long Short-Term Memory (LSTM)

Long Short-Term Memory (LSTM) is an algorithm modified from the Recurrent Neural Network (RNN) to address RNN’s shortcomings, such as the vanishing gradient problem that prevents RNN from retaining long-term memory. LSTM overcomes this issue with its ability to remember information or memory for a longer duration while maintaining RNN’s advantage in predictions based on recent information. LSTM is trained using the backpropagation algorithm, has a faster processing speed than RNN, and can manage information flow through cell states and gates. Additionally, LSTM has an internal cell state that stores selected information from previous units. Additionally, LSTM has an internal cell state that stores selected information from previous units, which is visualized in Figure 1.

LSTM has an architecture resembling a sequence composed of several units. The number of these units varies according to the needs of the algorithm being used. Each unit in an LSTM has a similar architecture, consisting of three gates that function to manipulate memory and process information. These three gates are the forget gate, the input gate, and the output gate.

The three gates in LSTM process information according to the activation functions present in each gate and further process it within the cell state. The cell state itself functions to remember and process memories that will be stored for the long term, known as Long Term Memory. The cell state processes memories that have been previously calculated through the three gates, which function to remember, collect, classify, and process information.

Forget gate, this is the first gate in LSTM. Some irrelevant and unnecessary information is forgotten in this gate. The information received in the forget gate, consisting of the hidden state value from the previous cell (ht − 1) and the current input value (xt), is combined and processed using the sigmoid activation function. The result of this sigmoid activation function ranges from 0 to 1.

$$ft=(Wf·[ht-1, xt]+bf)$$

(1)

where

• Wf = weight of the forget gate;
• bf = bias of the forget gate.

Input gate. This gate in LSTM is responsible for inputting information used for data accuracy. Information selected in the forget gate is added to the input gate. The value produced in the forget gate updates the cell state by multiplying the value t with Ct−1. The final result of the input gate is the product of the sigmoid function value and the tanh function value produced.

$$it=\sigma (Wi · [ht-1, xt]+bi)$$

(2)

$$Ct=tanh(Wc · [ht-1, xt]+bc)$$

(3)

where

• Wi = weight of the input gate;
• bi = bias of the input gate.

Output gate. This is the final gate in LSTM that generates the output from the processed cell.

$$ot=\sigma (Wo · [ht-1, xt]+bo)$$

(4)

where

• Wo = weight of the output gate;
• bo = bias of the output gate.

Finally, the value that will be the output of the processed cell (ht) will be processed again as ht − 1 in the next cell.

$$ht=ot \ast tanh \left(Ct\right)$$

(5)

$$Ct=(Ct-1·ft)+(it·Ct)$$

2.3. Mean Absolute Percentage Error (MAPE)

Mean Absolute Percentage Error (MAPE) is the average percentage of absolute error. The MAPE value can be found by calculating the percentage of the average error between predictions and actual data. The formula for calculating MAPE is as follows:

$$\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}={\displaystyle \frac{1}{n}}{\sum }_{t=1}^n{\displaystyle \frac{y_i-\widehat{y_i }}{\widehat{y_i}}}\times 100\%$$

(6)

where

• n = number of data points;
• t = time period;
• y = actual value;
• $\widehat{y}$ = predicted value.

There are four evaluation categories based on the calculated MAPE value:

• <10% = highly accurate;
• 10–20% = good;
• 20–50% = reasonable;
• >50% = inaccurate.

2.4. R-Squared (R²)

R-Squared (R²) is a coefficient of determination that ranges from 0 to 1 and serves as the proportion of variance in the dependent variable that is predicted from the independent variable. In this context, R-Squared can be used to evaluate stock price predictions in the future using historical stock data from the past. The coefficient of determination or accuracy of R-Squared can be calculated using the following formula:

$$k^2=1-{\displaystyle \frac{\sum {\left(y_i-\overline{y}\right)}^2}{\sum {\left(y_i-\widehat{y_i}\right)}^2}}$$

(7)

where

• y_i = actual value;
• $\widehat{y_i}$ = predicted value;
• $\overline{y_i}$ = mean of actual value.

A prediction is considered better if it has a larger R-Squared value, closer to 1, and is considered less accurate if it has a smaller R-Squared value, closer to 0. For example, a model with an R² value of 0.97 would indicate that 97% of the variance in the dependent variable can be predicted by the independent variable.

3. Research Methodology

3.1. Problem Identification

The research steps begin with problem identification, as illustrated in Figure 2. Problem identification is carried out by observing various phenomena and public views, such as the increasing trend of stock investments. As previously mentioned, stock prices are dynamic and influenced by several factors, one of which is technical factors based on past stock price data. Predicting stock prices by considering technical factors is very important for investors. This study aims to address the issue of stock price prediction by optimizing the Long Short-Term Memory (LSTM) algorithm.

After identifying the problem, the next step is a literature review. This stage begins with gathering supporting theories from books, journals, and other sources related to stock prediction, Long Short-Term Memory (LSTM), Mean Absolute Percentage Error (MAPE), and R-Squared (R²). Subsequently, an analysis of relevant previous studies is conducted to compare and identify gaps that can be updated in this research. The goal is to obtain updates and improve upon the shortcomings of previous research.

3.2. Data Collection

Data collection in this study involves time series stock price data obtained from Yahoo Finance. The data used includes stock prices of PT Bank Rakyat Indonesia Tbk (BBRI.JK), PT Bank Negara Indonesia Tbk (BBNI.JK), and PT Bank Mandiri Tbk (BMRI.JK) for the period 2018–2023. The total data collected comprises 5876 entries, consisting of 1493 daily stock price records from 1 January 2018 to 29 December 2023 for each company.

3.3. Data Pre-Processing

The data pre-processing stage is performed after data collection to prepare it for further analysis. This process begins with identifying and removing unnecessary attributes to reduce data complexity. Next, an analysis of missing values is conducted to determine their quantity. Handling missing values can be achieved by deleting rows or columns containing missing values or by replacing them with averages, frequencies, or other methods. After cleaning the data, the next step is normalization, which involves transforming the data into values between 0 and 1 to reduce complexity. The data that has undergone pre-processing is referred to as clean data, which facilitates the subsequent processing stages.

3.4. Data Splitting

The data splitting stage divides the clean data into training data and testing data. The purpose is to train the LSTM model with the training data and test its accuracy using the testing data. The testing data, which contains actual data, will be compared with the prediction results to measure the model’s performance. In this study, the data is split with a proportion of 80% for training data and 20% for testing data, following the Pareto principle (80/20) developed by Vilfredo Pareto in 1896. This principle observes that 80% of results often come from 20% of inputs, which also applies in data science.

3.5. Long Short-Term Memory Model Testing

The initial stage of testing the Long Short-Term Memory (LSTM) model involves determining the model architecture, including the parameters to be varied, such as the number of LSTM layer units and the number of epochs. The LSTM model is built with these parameters to find the optimal configuration using the Adam optimization. After building the model, it will be trained with the prepared training data. The prediction results from this training will then be compared to evaluate the performance and optimization of the LSTM model.

3.6. Performance Evaluation

The performance evaluation of the LSTM model is conducted by comparing prediction accuracy using two metrics: Mean Absolute Percentage Error (MAPE) and R-Squared Score (R² Score).

• MAPE measures the prediction error as a percentage, with the MAE value multiplied by 100%. Lower MAPE values indicate more accurate predictions.
• R-Squared (R²) measures how well the model explains the variability of the data. R² values range from 0 to 1, with values closer to 1 indicating more accurate predictions.

This research aims to find the best model with a maximum MAPE of 10% and an R² close to 1, ensuring the model has a good level of accuracy and produces the most accurate results possible.

4. Result and Discussion

The research began with collecting historical stock data from three state-owned banking companies, namely Bank BNI Tbk (BBRI.JK), Bank BRI Tbk (BBRI.JK), and Bank Mandiri Tbk (BMRI.JK), for the period from 1 January 2018, to 29 December 2023. The data was obtained from Yahoo Finance, totaling 4479 entries or 1493 entries for each bank.

4.1. Data Pre-Processing and Normalization

The first step in data pre-processing is to review missing values in the four stock datasets. Checking for missing values is crucial to minimize their negative impact on the evaluation of prediction results. Missing values can be handled in several ways, such as deleting rows or columns with missing values or replacing the missing values with the mean, frequency, or other methods, as shown in Figure 3.

To check the number of missing values, the isnull() function can be used to detect null values in the dataset and then summed up using the sum() function. The results in the figure indicate that each column of the dataset has a total of 0 missing values; thus, no deletion or replacement actions are required.

After handling missing values, the data is normalized to facilitate model processing. In the dataset of the four state-owned banks, normalization is performed using the min-max scaling technique, transforming the Close column variables into a range of 0–1. This technique reduces computational complexity, especially for data with large values such as zero. Normalization also avoids bias from large numerical values, reduces large spikes in calculations, and prevents internal covariate shifts that can cause drastic changes in neural network weights and prevent the model from converging. Data normalization is calculated using the following formula:

$$New value= {\displaystyle \frac{old value-\mathit{min}value}{\mathit{max}value-\mathit{min}value}} \times (new\mathit{max}- new min)+new min$$

(8)

In Python code, data normalization can be implemented using the MinMaxScaler library from sklearn preprocessing by setting the range value in the feature_range parameter. Data normalization was implemented in Python 3.11.5 using the MinMaxScaler from the scikit-learn library (version 1.3.0), with the feature_range parameter adjusted to define the target scale. Implementation in Python can be seen in Figure 4.

From the calculations in the Python code, the normalized values before and after normalization are obtained, as shown in the following example.

Table 1. Normalized values.

Date	BBNI	New	BBRI	New	BMRI	New
	Old (Million)		Old (Million)		Old (Million)
1 January 2018	4950	0.7849	3309.03	0.4845	3309.03	0.4845
2 January 2018	4887.5	0.7735	3299.94	0.4829	3299.94	0.4829
3 January 2018	4712.5	0.7414	3281.76	0.4796	3281.76	0.4796
…	…	…	…	…	…	…
27 December 2023	5275	0.8444	5625	0.9085	5625	0.9085
28 December 2023	5350	0.8581	5725	0.9268	5725	0.9268
29 December 2023	5375	0.8627	5725	0.9268	5725	0.9268

shows the stock prices before normalization in the “Old” column in millions of rupiahs and after normalization in the “New” column in the range of 0–1.

4.2. Data Splitting

The data used is time series data that requires a timestep for model quality and performance. Too large a timestep can lose important features, while too small a timestep wastes computational resources. The optimal timestep is 5, based on experiments with various timestep values (4, 5, 7, and 8). This timestep takes the first to the fifth data as the x variable and the sixth data as the y variable, and so on.

The data used in this study amounts to 4479 entries, with 1493 entries for each bank. Using a timestep reduces the data by 5, so the data split into training and testing is 1488 per bank. Of this amount, 80% (1190 data) is used for training, and 20% (298 data) is used for testing for each bank.

4.3. Model Training

This study uses the Long Short-Term Memory (LSTM) algorithm with implementation in Python code using the Keras library with the Sequential model. The layers used in this study consist of five layers, namely 2 LSTM layers, two Dropout layers, and one Dense layer, which functions as the output layer for predictions.

Table 2. Parameter variations for each layer.

Stage	Layer/Process	Parameter	Value
Model	LSTM	Units	10, 20, 50
	Dropout	Rate	0.2
	LSTM	Units	10, 50
	Dropout	Rate	0.2
	Dense	Units	1
Compile		Optimizer	Adam
		Loss	Mean Absolute Percentage Error
Training		Epochs	300, 400, 500
		Batch Size	32

shows the five layers compiled with parameter variations for epochs during the model fitting process. The first LSTM layer uses unit variations of 10, 20, and 50 for initial data training. Next, the Dropout layer with a rate of 0.2 reduces overfitting. The second LSTM layer uses units of 10 and 50 to increase accuracy and reduce error. The second Dropout also has a rate of 0.2 to reduce overfitting. The final layer is Dense with 1 unit for the model output. The optimizer used is Adam, and the loss function is MAPE. Model fitting parameters vary at epochs 300, 400, and 500 with a batch size of 32, based on literature studies and conducted experiments.

The LSTM model in this study is built using the Keras library with KerasRegressor as the regression estimator and the create_lstm_model function for the build_fn parameter, as shown in Figure 5. The LSTM architecture starts with the first layer using 20 units, return_sequences = True, and an input shape of xbri_train.shape [1] of 5 and xbri_train.shape [2] of 1. A Dropout layer with a rate of 0.2 is added, followed by a second LSTM layer with 50 units and return_sequences=False. Another Dropout layer with a rate of 0.2 is added, and a Dense layer with 1 unit as the output. The model is compiled using the Adam optimizer and the mean_absolute_percentage_error (MAPE) loss function.

The steps in the LSTM layer start with the data entering the forget gate to determine whether the information will be discarded using the sigmoid function. The information is discarded if the result is 0 and retained if the result is not 0. Then, the data enters the input gate to add information to the cell state through the result of the sigmoid and tanh functions. Finally, the data enters the output gate that determines the output from the cell state, which is the result of the multiplication of the sigmoid function and the tanh of the input gate value. The training process of the model is illustrated in Figure 6.

After the model is built, the model is trained using the training data. The history variable or object stores the result of the fit method on the lstm_model, with the training input data xbri_train and the training target data ybri_train. The validation_data parameter is set with xbri_test and ybri_test as validation data and validation target data. The model will evaluate performance on the validation data at each epoch and store this information in history. The modeling will use Lasso Regression and Polynomial Feature as input variables to become additional features.

The LSTM model training process is repeated with parameter variations to find the optimal result in predicting stock prices. The process starts by training the data on the LSTM layer using parameter unit variations, then regulation is performed with a dropout layer with a value of 0.2, meaning 20% of the neurons will be set to zero. After that, the second LSTM model is trained by adding an LSTM layer with unit parameter variations and a dropout rate regulation of 0.2. Finally, predictions are generated through the dense layer with 1 neuron without an activation function to produce continuous values. The overall LSTM model architecture used in this study is illustrated in Figure 7.

4.4. Model Testing and Evaluation

The final stage of this research is to predict the stock data of the four banks using 297 test data. This process tests the architecture scheme with unit variations in the LSTM layer (10, 20, and 50) and the number of epochs (300, 400, and 500). The best model is selected by comparing the MAPE (Mean Absolute Percentage Error) and R²-Squared values for each parameter and bank data.

From

Table 3. Model testing results.

Units		Epochs	BBNI		BBRI		BMRI
LSTM1	LSTM2		MAPE	R2	MAPE	R2	MAPE	R2
10	10	300	0.0101	0.9338	0.026	0.843	0.0188	0.9176
10	10	400	0.0176	0.8464	0.0171	0.9201	0.0179	0.9232
10	10	500	0.02	0.8092	0.0144	0.9427	0.0204	0.8999
20	50	300	0.0115	0.9228	0.0125	0.9551	0.0111	0.9641
20	50	400	0.0138	0.8946	0.0133	0.9498	0.0116	0.9614
20	50	500	0.0103	0.9324	0.0249	0.8448	0.0115	0.9618
50	50	300	0.0122	0.9347	0.0117	0.9605	0.017	0.9244
50	50	400	0.0148	0.8839	0.0203	0.8939	0.0204	0.9015
50	50	500	0.0102	0.933	0.0106	0.9658	0.0111	0.9636

, it can be concluded that the model for BBNI.JK data produces the lowest MAPE of 0.0101 or 1.01% and the highest R² of 0.9347 or 93.47%. Based on the MAPE value category, the model for BBNI.JK data falls into the very accurate category. Meanwhile, the lowest MAPE value for BBRI.JK data is 0.0106 or 1.06%, which means that the model for BBRI.JK data also falls into the very accurate category based on the MAPE value. The highest R² value produced by the model for BBRI.JK data is 0.9658 or 96.58%. Finally, the model using BMRI.JK data produces the smallest MAPE value of 0.0111 or 1.11% and the highest R² of 0.9641 or 96.41%.

Figure 8 is a graphical visualization to see how accurate the prediction results are compared to the actual values in the test data of the three banks. The three graphs show that the model used successfully predicts the actual values in the test data very accurately.

5. Conclusions

Effectiveness of the LSTM Algorithm: The application of the Long Short-Term Memory (LSTM) algorithm in predicting stock prices has shown excellent results. The error calculation using Mean Absolute Percentage Error (MAPE) yielded error values between 0.01 and 0.025, indicating prediction errors ranging from 1% to 2.5%. These MAPE values fall into the very accurate category, which is below 10%. Additionally, the R-Squared (R²) values of the model ranged from 0.8 to 0.96, indicating that the LSTM model can explain 80% to 96% of the variation in future stock prices based on historical data. In other words, the LSTM model used is highly effective in predicting stock prices for state-owned banks, including Bank BRI Tbk (BBRI.JK), Bank BNI Tbk (BBNI.JK), and Bank Mandiri Tbk (BMRI.JK).

Optimization of the LSTM Model: The optimization of the LSTM model in this study was achieved by exploring various values for epochs and the number of units in the LSTM layers. The analysis revealed that the number of epochs significantly impacts model accuracy and error. For Bank BNI Tbk (BBNI.JK), the lowest MAPE value was achieved using 300 epochs and 10 units in each LSTM layer. However, for Bank BRI Tbk (BBRI.JK) and Bank Mandiri Tbk (BMRI.JK), the smallest MAPE was observed with 500 epochs. Although the MAPE for Bank BNI at 500 epochs was nearly equivalent to that at 300 epochs, the highest R² value was obtained for Bank BRI Tbk with 500 epochs, 50 units in the first LSTM layer, and 10 units in the second layer. Therefore, using 500 epochs can optimize the LSTM model, with an increasing number of epochs generally improving model accuracy. Additionally, the number of units in the LSTM layers also affects accuracy; most experiments indicated that using 50 units in the LSTM layers enhances accuracy and reduces error.

6. Recommendations

• Data Quantity: Increase the amount of data either by including stock prices from non-state-owned banks or by extending the data collection period.
• Parameter Variations: Explore a wider range of parameters for each LSTM layer to enhance model performance.
• Loss Functions: Utilize various loss functions to evaluate model accuracy from different error and accuracy perspectives.
• Website Development: Develop a website to help investors view future stock price predictions and reduce the risk of losses.
• Additional Algorithms: Integrate additional algorithms such as GRU or boosting techniques to improve the accuracy of the existing LSTM model.