Comparing Machine Learning Methods—SVR, XGBoost, LSTM, and MLP— For Forecasting the Moroccan Stock Market ^†

Abstract

Forecasting and modeling time series is a crucial aspect of economic research for academics and business practitioners. The ability to predict the direction of stock prices is vital for creating an investment plan or determining the optimal time to make a trade. However, market movements can be complex to predict, non-linear, and chaotic, making it difficult to forecast their evolution. In this paper, we investigate modeling and forecasting the daily prices of the new Morocco Stock Index 20 (MSI 20). To this end, we propose a comparative study between the results obtained from the application of the various Machine Learning (ML) methods: Support Vector Regression (SVR), eXtreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Long Short-Term Memory (LSTM) models. The results show that using the Grid Search (GS) optimization algorithm, the SVR and MLP models outperform the other models and achieve high accuracy in forecasting daily prices.

1. Introduction

The 2008 recession, the stock market crash of 2015, the COVID-19 Pandemic, and the Russian Invasion of Ukraine are some of the most recent crises which have had an immense impact on the financial markets and the destruction of wealth worldwide. Modeling and forecasting the stock market is a challenge that many engineers and financial researchers face. The literature review examined studies on stock market prediction using Machine Learning (ML) models. It concluded that Deep Learning (DL) was the most commonly utilized model for forecasting stock price trends [1,2]. Traditional econometric methods might require improved performance in relevant nonlinear time series and may not be appropriate for directly forecasting stock prices because of their volatility [3]. However, for complex nonlinear financial time series, methods such as Support Vector Regression (SVR), eXtreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Long Short-Term Memory (LSTM) can detect nonlinear relationships in the forecasting of stock prices [4] and achieve better fitting results by tuning multiple parameters [5]. Hyperparameter optimization or tuning in ML refers to selecting the most appropriate parameters for a particular learning model [6]. Some studies use Grid Search (GS) optimization [7], while others use Bayesian [8,9] or pigeon-inspired optimization algorithms [10]. In this paper, the GS algorithm is used to optimize the parameters of each model, such as SVR, XGBoost, MLP, and LSTM models. Then, we compare them using seven measures: Mean Error (ME), Mean Percentage Error (MPE), Mean Square Error (MSE), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), and R${}^2$. In [11], it was discovered that using LSTM with Moving Averages (MA) yields superior results for predicting stock prices compared to SVR, as measured by different performance criteria such as MAE, MSE, MAPE, RMSE, and R${}^2$ values. In their research [12], Al-Nefaie et al. proposed using LSTM and MLP models to forecast the fluctuations in the Saudi stock market. They found that the correlation coefficient for the two models was higher than 0.995. The LSTM model proved to have the highest accuracy and best model fit. In [13], the authors compared performance measures such as MSE, MAE, RMSE, and MAPE to forecast stock market trends based on Auto-Regressive Integrated Moving Average (ARIMA), XGBoost, and LSTM. Their tests found that XGBoost performed the best. In a different context, ML methods such as ANN and MLP [14] were used to forecast solar irradiance. The results indicate that the MLP model with exogenous variables performs better than the other models. Similarly, in another study [15], the predictive performance and stability of eXtreme Gradient Boosting Deep Neural Networks (XGBF-DNN) made it an optimal and reliable model for forecasting hourly global horizontal irradiance using the GS algorithm. The main objective of this investigation is to improve the GS optimization algorithm by optimizing the hyperparameters of ML models. Furthermore, this study compares two ML methods and two DL methods. To achieve this objective, the research includes an in-depth literature review of 20 studies on ML models for stock market prediction. Overall, the paper aims to contribute to the stock market prediction field by improving the performance of ML models through hyperparameter optimization. Our contribution is highlighted in Table 1, which offers a more accurate prediction than other studies.

Table 1. Some previous work using ML models for stock market forecasting.

Reference	Location	Year	Benchmark Model	MAPE (%)	RMSE	R${}^2$
[11]	Irlande	2019	LSTM, SVR, MA	1.03	347.46	0.83
[16]	China	2020	MLP, CNN, RNN, LSTM, CNN-RNN	—	39.688	0.965
[13]	American	2022	ARIMA, XGBoost, LSTM	3.8	6.101	0.961
This study	Morocco	2023	SVR, XGBoost, MLP, LSTM	0.368	3.993	0.989

The abbreviations have the following meaning: RNN = Recurrent Neural Network, CNN = Convolutional Neural Network.

The rest of this paper is organized as follows: Section 2 outlines the suggested methodology for modeling and forecasting financial time series, specifically stock prices. Section 3 details the implementation of the study, results, and discussion. Lastly, Section 4 offers conclusions and suggests future research directions.

2. Materials and Methods

This section provides an overview of the methodology employed in this research.

2.1. Data Collection

This research focuses on modeling and forecasting a new Moroccan Stock Index 20 (MSI 20) composed of the most liquid companies listed on the Casablanca Stock Exchange (CSE) from various sectors, including Attijariwafa Bank, Itissalat Al-Maghrib, Banque Populaire, LafargeHolcim Morocco, … etc. The MSI 20 is calculated in real time during the business hours of the CSE from Monday to Friday, and from 9:00 a.m. to 3:30 p.m. local time (Limited on weekends and holidays). The research was conducted using the Python software. In addition, several libraries were used, including Matplotlib, Pandas, NumPy, Sklearn, Tensorflow, and Keras. We use daily MSI 20 data to train each model and predict closing prices. Since the launch of the index of length $N=541$, we use prices as Input = (Open, High, Low, and Closing prices), see the Table 2 below:

Table 2. Sample data (First five days) and descriptive statistics of MSI 20 index daily prices from 18 December 2020 to 9 February 2023.

Trade Date	Open (MAD)	High (MAD)	Low (MAD)	Close (MAD)
2020-12-18	946.25	950.56	943.85	944.58
2020-12-21	944.58	944.58	911.51	912.17
2020-12-22	912.17	928.51	906.77	927.79
2020-12-23	927.79	935.71	927.07	932.43
2020-12-24	932.43	932.81	926.58	927.16
Statistics	Open	High	Low	Close
Count	541	541	541	541
Mean	988.86	993.28	984.91	988.84
Std	78.01	77.05	78.35	78.15
Min	775.38	797.01	775.38	775.38
Max	1140.69	1142.56	1135.86	1140.69

Note: MAD = Moroccan Dirham.

To train model parameters, we use historical data of length N,

$$X=\{X_t^{\left(1\right)},X_t^{\left(2\right)},X_t^{\left(3\right)},t=1,2,\dots ,N\},$$

where $X^{\left(i\right)}$ with $i=1,2,3$ represents the Open, High, and Low prices, respectively. For the output with a single observation sequence, we use only closing price,

$$Y=Y_t,t=1,2,\dots ,N$$

2.2. Preprocessing

The dataset was divided into 90% of the observations used for training and 10% for model evaluation during testing. Data preprocessing is an essential step in ML that helps achieve competitive results and eliminate metric unit effects. In this case, we normalized using a min–max scale, which scales all variables to a range of $[0,1]$:

$$\begin{array}{c}\hfill {\tilde{X}}_t=\frac{X_t-X_t^{min}}{X_t^{max}-X_t^{min}}\end{array}$$

(1)

where $X_t$ represents the historical data for each feature variable in the time series (Open, High, and Low prices) and the $X_t^{max}$ and $X_t^{min}$ values are the sample’s maximum and minimum values.

2.3. SVR Model

The SVR model, a new financial time series prediction method, is used to address the challenges of nonlinear regression. We assume a linear relationship exists between $X_t$ and $Y_t$ as in the left side of the Equation (2). To perform nonlinear regression using SVR, the concept consists of creating an $x⟶\phi \left(x\right)$ transformation that maps the original feature space X, which has N dimensions, onto the new feature space $X^{\prime }$. Mathematically, this can be explained by the equation shown below:

$$\begin{array}{c}\hfill Y_t=W·X_t+b=\sum _{i=1}^N{w}_i{x}_{it}+b_i,\underset{mapping}{\overset{Non-linear}{\hookrightarrow }}{Y}_t=\sum _k{\beta }_k{y}_k\mathcal{K}(x_k,x)+b\end{array}$$

(2)

where $w_i$ is the vector of weights and $b_i$ is a bias, ${\beta }_k$ is the coefficient of the Lagrange multipliers, and $\mathcal{K}(x_k,x)=\phi \left(x_k\right)·\phi \left(x\right)$ is a kernel function. The most commonly used kernels include linear, Gaussian, and polynomial functions [17].

2.4. XGBoost Model

XGBoost is an ML model used for stock market time series forecasting that uses a set of decision trees [18]. A gradient descent algorithm guides the process of preparing subsequent trees to minimize the loss function of the last tree [6].

$${\mathcal{L}}_T\left(F\left(x_i\right)\right)=\sum _{i=1}^N\chi (y_i,F_T\left(x_i\right))+\sum _{t=1}^T\Pi \left(f_t\right),$$

(3)

where $\chi (·)$ is a specified loss function that quantifies the deviations of the predicted and actual target values, and $F_T\left(x_i\right)={\sum }_{t=1}^T{f}_t\left(x_i\right)$ denotes the forecast on the i-th sample at the T-th boost and $\Pi \left(f\right)=\alpha K+0.5\times \kappa {\parallel \varpi \parallel }^2$, where K is the number of leaves. For the regularization term, $\alpha$ is the parameter of complexity. ${\parallel \varpi \parallel }^2$ is the L2 norm of weight regularization, $\kappa$ is a constant coefficient, and $\Pi (·)$ represents the term of regularization, which penalizes the model complexity [8].

2.5. MLP Model

The MLP is a frequently used ANN consisting of three layers of neurons: an input layer, one or more hidden layers, and an output layer. The inputs ($x_i$) are multiplied by their weights ($w_i$), and the resulting products are combined. This sum and a bias term (b) are fed into an activation function to produce the neuron’s output ($Y_t$) [19]. Equation (2) can be used to express this process in mathematical words:

$$\begin{array}{cc}\hfill {\zeta }_t=\sum _{i=1}^N{w}_i{x}_{it}+b& ⟹Y_t=\sigma \left({\zeta }_t\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill \sigma \left({\zeta }_t\right)& =\frac{1}{1+e^{-{\zeta }_t}}\hfill \end{array}$$

(5)

where $\sigma (·)$, the activation function, is frequently employed as a function, either continuous or discontinuous, that maps real numbers to a specific interval. Alternatively, the sigmoidal activation function can also be utilized [12].

2.6. LSTM Model

LSTM models are RNNs that excel at learning and retaining long-term dependencies, making them successful in various applications such as financial time series forecasting. The principle of an LSTM cell consists of the following four equations [20]:

$$\begin{array}{lll}\hfill \mathrm{Forget\; gate:}& f_t=\delta (W_{ef}·(X_t,h_{t-1})+W_{cf}{C}_{t-1}+b_f),& \\ \hfill \mathrm{Input\; gate:}& \hspace{1em}{i}_t=\delta (W_{ki}·(X_t,h_{t-1},C_{t-1})+b_i),& k=x,h,c\\ \hfill \mathrm{Memory\; cell:}& \hspace{1em}\hspace{1em}{C}_t=f_t·C_{t-1}+i_t·tanh(W_{lc}(X_t,h_{t-1})+b_c),& l=x,h\hfill \\ \hfill \mathrm{Output\; gate:}& \hspace{1em}\hspace{1em}\hspace{1em}{o}_t=\delta (W_{ji}·(X_t,h_{t-1},C_{t-1})+b_o),& j=x,h,c\hfill \end{array}$$

where $W_{e*}$, $W_{c*}$, $W_{k*}$, $W_{l*}$, $W_{j*}$, and $b_f$, $b_i$, $b_c$, $b_o$ are the weight and bias of each layer respectively, and $h_t=o_t·tanh\left(C_t\right)$ represents the hidden layer output. The forget gate ($f_t$) determines if the information should be kept or discarded, while the input gate ($i_t$) integrates new data into the cell ($C_t$). Eventually, the output gate ($o_t$) governs the selection of relevant data to transmit to the succeeding cell ($C_{t-1}$).

2.7. Grid Search

In ML, GS is commonly used to fine-tune parameters such as regularization strength, learning rate, several hidden layers …, etc. By using the GS algorithm, we can identify the optimal set of hyperparameters for the models, resulting in better predictions and improved performance. In our case, we first defined the hyperparameters and their search space, as shown in the Table 3. The optimal hyperparameters for each model are shown below in bold blue. The design of the research study is illustrated in Figure 1.

Table 3. The GS hyper-parameter sets for the SVR, XGBoost, MLP, and LSTM models.

ML Models	Parameters	Type	Search Space
	Epsilon in the SVR loss function (epsilon)	Continuous	[0.0001, 0.001, 0.01]
	Regularization parameter (C)	Discrete	[100, 1000, 1100]
SVR	The kernel type (kernel)	Categorical	[’linear’, ’poly’, ’rbf’, ’sigmoid’]
	Kernel coefficient (gamma)	Continuous	[1 × 10−5, 1 × 10−4, 1 × 10−3]
	Tolerance for stopping criterion (tol)	Continuous	default = 1 × 10−3
	# of regression trees (n_estimators)	Discrete	[100, 200, 1000]
	Maximum regression tree depth (max_depth)	Discrete	[5,10, 15, 20]
	Boosting the rate of learning (learning_rate)	Continuous	[0.01, 0.06, 0.09]
XGBoost	Minimum reduction of loss (gamma)	Continuous	[0.001, 0.01, 0.1]
	Regularization term L1 on weights (reg_alpha)	Continuous	[ 0.001, 0.01, 0.1]
	Regularization term L2 on weights (reg_lambda)	Continuous	[ 0.001, 0.01, 0.1]
	Objective learning function (objective)	Categorical	[’reg:squarederror’,’reg:linear’]
	# of neurons in the ith hidden layer (hidden_layer_sizes)	Discrete	[(50,50,50), (50,100,50), (100,)]
	Activation function for hidden layer (activation)	Categorical	[’relu’,’tanh’,’logistic’,’identity’]
MLP	The solver for weight optimization (solver)	Categorical	[’sgd’,’adam’,’lbfgs’]
	Strength of the regularization term L2 (alpha)	Continuous	[0.001, 0.01, 0.1]
	Learning rate for weight update program (learning_rate)	Categorical	[’constant’,’invscaling’,’adaptive’]
	# of epochs to train the model (epochs)	Discrete	[20, 90, 100]
	# of hidden layer	Discrete	[1, 2, 3, 4]
	The function that tries to optimize (optimizer)	Categorical	[’adam’,’rmsprop’,’sgd’]
LSTM	Learning rate (learning_rate)	Continuous	[0.1, 0.01, 0.001]
	Activation functions (activation)	Categorical	[’tanh’,’sigmoid’,’relu’]
	# of hidden layer neurons (neurons_1, neurons_2)	Discrete	[150, 250, 350], [200, 200, 300],
	Loss functions to be minimized during model training (loss)	Categorical	[’mae’,’mse’,’mape’]

Figure 1. The proposed research design architecture for analyzing, modeling, and predicting MSI 20.

3. Results and Discussion

In the section on results and discussion, we report and compare the performance of the SVR, XGBoost, MLP, and LSTM models for forecasting the MSI 20 stock market. Various evaluation measures such as ME, MPE, MSE, MAE, RMSE, MAPE, and R${}^2$ scores are obtained to assess the models’ accuracy. Let ${\widehat{y}}_{it}$ be the forecast of model i in time t, while $y_t$ is the real value in time t, ${\overline{y}}_t$ is the mean value, and n is the length of the set time series (i.e., training and test sets). The error for model i at time t is defined as ${\epsilon }_{it}=y_t-{\widehat{y}}_{it}$. The general formula for evaluation measures is presented in Table 4 below:

Table 4. Formulas for model performance measures.

Metrics	ME	MAE	MSE	MPE (%)	MAPE (%)	RMSE	R${}^2$ (%)
Formulas	$\frac{1}{n}{\displaystyle \sum _{t=1}^n}{\epsilon }_{it}$	$\frac{1}{n}{\displaystyle \sum _{t=1}^n}\left|{\epsilon }_{it}\right|$	$\frac{1}{n}{\displaystyle \sum _{t=1}^n}{\epsilon }_{it}^2$	$\frac{100}{n}{\sum }_{t=1}^n\frac{{\epsilon }_{it}}{y_t}$	$\frac{100}{n}{\sum }_{t=1}^n\left|\frac{{\epsilon }_{it}}{y_t}\right|$	$\sqrt{\mathrm{MSE}}$	$1-\frac{n\mathrm{MSE}}{{\sum }_{t=1}^n(y_t-{\overline{y}}_t)}$

Table 5 shows the evaluation values of various forecasting models for the MSI 20 price. XGBoost had the highest error among the four models. LSTM had the second-worst performance. SVR and MLP models had lower performance metrics than XGBoost and LSTM models. However, the SVR model had the best evaluation values for forecasting MSI 20 prices, achieving optimal ME, MAE, RMSE, MPE, MAPE, and MSE values of 0.674, 3.092, 3.993, 0.082%, 0.368%, and 15.941, respectively. The MLP model also showed excellent performance metrics. However, the predicted results of the XGBoost model in Figure 2 on the right showed a specific error. They predicted values of MSI 20, especially during the fall period (6 January 2023–10 January 2023 ), indicating that the model needed more data to improve its performance. Based on the results presented in Table 5 and Figure 2, it can be concluded that the SVR model followed by MLP models had the best performance for forecasting MSI 20 compared to other models.

Table 5. The performance analysis measures metrics of each model.

Models	SVR		XGBoost		MLP		LSTM
Error Measure	Training	Test	Training	Test	Training	Test	Training	Test
ME	0.0899	0.674	2.277	0.558	5.406 $\times {10}^{-6}$	0.883	2.339	1.4
MAE	2.058	3.092	2.460	9.165	2.059	3.101	3.554	5.065
RMSE	2.646	3.993	3.141	13.515	2.642	4.018	4.496	6.322
MPE (%)	$9.134\times {10}^{-3}$	0.082	0.226	0.116	7.603 $\times {10}^{-4}$	0.107	0.164	0.234
MAPE (%)	0.207	0.368	0.244	1.095	0.207	0.370	0.357	0.603
MSE	7.003	15.941	9.864	182.651	6.978	16.286	20.215	39.97
${\mathrm{R}}^2$	0.998	0.989	0.997	0.882	0.998	0.989	0.995	0.974

Figure 2. MSI 20 stock price prediction: results of LSTM, SVR, MLP, and XGBoost models.

4. Conclusions and Future Work

This study applied four machine learning models—SVR, XGBoost, MLP, and LSTM—to model and forecast MSI 20 prices using multivariate time series data. The results showed that SVR outperformed the other models with lower errors and higher accuracy 98.9%. Future research could focus on improving the performance of XGBoost and exploring the potential of other models, such as CNN-LSTM, for stock market forecasting.