Comparing Machine Learning Methods - SVR, XGBoost, LSTM, and MLP - in Forecasting the Moroccan Stock Market

: Forecasting and modeling time series is a crucial aspect of economic research for academics and business practitioners. The ability to predict the direction of stock prices is vital for creating an investment plan or determining the optimal time to make a trade. However, market movements can be complex to predict, non-linear and chaotic, making it difﬁcult to forecast their evolution. In this paper, we investigate modeling and forecasting the daily prices of the new Morocco Stock Index 20 (MSI 20). To this aim, we propose a comparative study between the results obtained from the application of the various Machine Learning (ML) methods: Support Vector Regression (SVR), eXtreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Long Short-Term Memory (LSTM) models. The results show that using the Grid Search (GS) optimization algorithm, the SVR and MLP models outperform the other models and achieve high accuracy in forecasting daily prices.


Introduction
The 2008 recession, the stock market crash of 2015, the Covid 19 Pandemic, and the Russian Invasion of Ukraine are some of the most recent crises with an immense impact on the financial markets and the destruction of wealth worldwide.Modeling and forecasting the stock market is a challenge that many engineers and financial researchers face.The literature review examined studies on stock market prediction using Machine Learning (ML) models.It concluded that Deep Learning (DL) was the most commonly utilized model for forecasting stock price trends [1,2].Traditional econometric methods might require improved performance in relevant nonlinear time series and may not be appropriate for directly forecasting stock prices because of their volatility [3].However, for complex nonlinear financial time series, methods such as Support Vector Regression (SVR), eXtreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Long Short-Term Memory (LSTM) can detect nonlinear relationships in the forecasting stock prices [4], and achieve better fitting results by tuning multiple parameters [5].Hyperparameter optimization or tuning in ML refers to selecting the most appropriate parameters for a particular learning model [6].Some studies use Grid Search (GS) optimization [7], while others use Bayesian [8,9] or pigeon-inspired optimization algorithms [10].In this paper, the GS algorithm is used to optimize the parameters of each model, such as SVR, XGBoost, MLP, and LSTM models.Then we compare them using seven measures Mean Error (ME), Mean Percentage Error (MPE), Mean Square Error (MSE), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), and R 2 .In [11] discovered that using LSTM with Moving Averages (MA) yields superior results for predicting stock prices compared to SVR, as measured by different performance criteria such as MAE, MSE, MAPE, RMSE, and R 2 values.In their research [12], Al-Nefaie et al. proposed using LSTM and MLP models to forecast the fluctuations of the Saudi stock market.They found that the correlation coefficient for the two models was higher than 0.995.The LSTM model proved to have the highest accuracy and best model fit.In [13], the authors compared performance measures such as MSE, MAE, RMSE, and MAPE to forecast stock market trends based on Auto-Regressive Integrated Moving Average (ARIMA), XGBoost, and LSTM.Their tests found that XGBoost performed the best.In a different context, ML methods like ANN and MLP [14] were used to forecast solar irradiance.The results indicate that the MLP model with exogenous variables performs better than the other models.Similarly, in a study [15], the predictive performance and stability of eXtreme Gradient Boosting Deep Neural Networks (XGBF-DNN) made it an optimal and reliable model for forecasting hourly global horizontal irradiance using the GS algorithm.The main objective of this investigation is to improve the GS optimization algorithm by optimizing the hyperparameters of ML models.Furthermore, this study compares two ML methods and two DL methods.To achieve this objective, the research includes an in-depth literature review of 20 studies on ML models for stock market prediction.Overall, the paper aims to contribute to the stock market prediction field by improving the performance of ML models through hyperparameter optimization.Our contribution is highlighted in Table 1, a more accurate prediction than other studies.The rest of this paper is organized as follows: Section 2 outlines the suggested 1 methodology for modeling and forecasting financial time series, specifically stock prices.

5
The Materials and Methods section provides an overview of the methodology 6 employed in this research.To train model parameters, we use historical data of length N, where X (i) with i = 1, 2, 3 represents the Open, High, and Low prices, respectively.For the output with a single observation sequence, we use only closing price, The data set was divided into 90% of the observations used for training and 10% for model evaluation during testing.Data preprocessing is an essential step in ML that helps achieve competitive results and eliminate metric unit effects.In this case, we normalized using a min-max scale, which scales all variables to a range of [0, 1]: where X t is the historical data for each feature variable in the time series (Open, High, The SVR model, a new financial time series prediction method, is used to address the challenges of nonlinear regression.We assume a linear relationship exists between X t and Y t as in the left side of the equation 2. To perform nonlinear regression using SVR, the concept consists of creating an x −→ ϕ(x) transformation that maps the original feature space X, which has N dimensions, onto the new feature space X .Mathematically, this can be explained by the equation shown below: where w i is the vector of weights, and b i is a bias, β k is coefficient of the Lagrange XGBoost is an ML model used for stock market time series forecasting that uses a 28 set of decision trees [18].A gradient descent algorithm guides the process of preparing 29 subsequent trees to minimize the loss function of the last tree [6].
where χ(•) is a loss function specified that quantifies the deviations of the predicted  The MLP is a frequently used ANN consisting of three layers of neurons: an input layer, one or more hidden layers, and an output layer.The inputs (x i ) are multiplied by their weights (w i ), and the resulting products are combined.This sum and a bias term (b) are fed into an activation function to produce the neuron's output (Y t ) [19].Equation (2) can be used to express this process in mathematical words: where σ(•), the activation function, is frequently employed as a function, either continu-38 ous or discontinuous, that maps real numbers to a specific interval.Alternatively, the 39 sigmoidal activation function can also be utilized [12].LSTM models are RNNs that excel at learning and retaining long-term dependencies, making them successful in various applications such as financial time series forecasting.The principle of an LSTM cell consists of the following four equations [20]: Forget gate: Input gate: Memory cell: Output gate: where In ML, GS is commonly used to fine-tune parameters such as regularization strength, 48 learning rate, several hidden layers...etc.By using the GS algorithm, we can identify 49 the optimal set of hyperparameters for the models, resulting in better predictions and 50 improved performance.In our case, we first defined the hyperparameters and their 51 search space, as shown in the Table 3.The optimal hyperparameters for each model are

52
shown below in bold blue.The design of the research study is illustrated in Figure 1.

54
In the section on results and discussion, we report and compare the performance

2 Section 3
details the implementation of the study, results, and discussion.Lastly, Section 3 4 offers conclusions and suggests future research directions.

8
This research focuses on modeling and forecasting a new Moroccan Stock Index9 20 (MSI 20) is composed of the most liquid companies listed on the Casablanca Stock 10 Exchange (CSE) from various sectors, including Attijariwafa Bank, Itissalat Al-Maghrib, 11 Banque Populaire, LafargeHolcim Morocco,...etc.The MSI 20 is calculated in real-time 12 during the business hours of the CSE from Monday to Friday, and from 9:00 a.m. to 3:30 13 p.m. local time (Limited on weekends and holidays).The research was conducted using 14 the Python software.In addition, several used libraries, including Matplotlib, Pandas, 15 NumPy, Sklearn, Tensorflow, and Keras.We use daily MSI 20 data to train each model 16 and predict closing prices.Since the launch of the index of length N = 541, we use prices 17 as Input = (Open, High, Low, and Closing prices), see the

20 and
Low prices), the X max t and X min t values are the sample's maximum and minimum 21 values.22 2.3.SVR Model 23

31
and actual target values, F T (x i ) = ∑ T t=1 f t (x i ) denotes the forecast on the i-th sample at 32 the T-th boost and Π( f ) = αK + 0.5 × κ 2 , where K is the number of leaves.For the 33 regularization term, α is the parameter of complexity. 2 is the L2 norm of weight 34 regularization, κ is a constant coefficient, and Π(•) represents the term of regularization, 35 which penalizes the model complexity [8].

Figure 1 .
Figure 1.The proposed research design architecture for analyzing, modeling, and predicting MSI 20.

Table 1 .
Some previous work using ML models for stock market forecasting.

Table 2 .
Sample data (First five days) and descriptive statistics of MSI 20 index daily prices from 18 December 2020 to 09 February 2023.

Table 4 .
55of the SVR, XGBoost, MLP, and LSTM models for forecasting the MSI 20 stock market.Various evaluation measures such as ME, MPE, MSE, MAE, RMSE, MAPE, and R 2 scores 57 are obtained to assess the models' accuracy.Let ŷit be the forecast of model i in time t, y t 58 is the real value in time t, y t is the mean value, and n is the length of the set time series 59 (i.e., training & test sets).The error for model i at time t is defined as ε it = y t − ŷit .The 60 general formula for evaluation measures is presented in Table 4 below: Formulas for model performance measures. 56

Table 5
shows the evaluation values of various forecasting models for the MSI 20 62 price.XGBoost had the highest error among the four models.LSTM had the second-

Table 5 and
Figure 2, it can be concluded that the SVR model followed by MLP models 72 had the best performance for forecasting MSI 20 compared to other models.
78XGBoost and exploring the potential of other models, such as CNN-LSTM, for stock 79 market forecasting.

Table 5 .
The performance analysis measures metrics of each model.The data is available on the CSE website: https://www.casablanca- 81 Data Availability Statement: 82 bourse.com.83