Interval Type-3 Fuzzy Aggregation of Neural Networks for Multiple Time Series Prediction: The Case of Financial Forecasting

: In this work, we present an approach for fuzzy aggregation of neural networks for forecasting. The interval type-3 aggregator is used to combine the outputs of the networks to improve the quality of the prediction. This is carried out in such a way that the ﬁnal output is better than the outputs of the individual modules. In our approach, a fuzzy system is used to estimate the prediction increments that will be assigned to the output in the process of combining them with a set of fuzzy rules. The uncertainty in the process of aggregation is modeled with an interval type-3 fuzzy system, which, in theory, can outperform type-2 and type-1 fuzzy systems. Publicly available data sets of COVID-19 cases and the Dow Jones index were utilized to test the proposed approach, as it has been stated that a pandemic wave can have an effect on the economies of countries. The simulation results show that the COVID-19 data does have, in fact, an inﬂuence on the Dow Jones time series and its use in the proposed model improves the forecast of the Dow Jones future values.


Introduction
Fuzzy logic has become very important in different disciplines of study, and one of the areas is the main focus of this work, which is the time series prediction area. It has also been shown in the literature that the use of type-1 fuzzy logic helps to improve the results in many problems [1,2]. Later, type-1 evolved into type-2 fuzzy systems with the work of Mendel and others in 2001 [3,4]. Initially, interval type-2 fuzzy systems were studied and applied to several problems [5]. Later, these systems were applied to many problems in areas, such as robotics, control, diagnosis, and others [6]. Simulation and experimental results show that interval type-2 outperforms type-1 fuzzy systems in situations with higher levels of noise, dynamic environments, or in highly nonlinear problems [6][7][8]. Later, general type-2 fuzzy systems were considered to manage higher levels of uncertainty, and good results have been achieved in several application areas [9][10][11]. Recently, it has become apparent that type-3 fuzzy systems could help solve even more complex problems [12][13][14]. For this reason, in this paper, we propose the basic constructs of type-3 fuzzy systems by extending the ideas of type-2 fuzzy systems, and then applying them to a prediction problem.
It has been shown in several previous works that individual neural networks have the ability to outperform statistical methods (in particular for nonlinear problems) and ensembles (formed by sets of networks) can outperform individual neural networks. So, for this reason, we concentrated our efforts on the aggregation part of the ensemble [15][16][17].
In particular, we focused on fuzzy aggregation, showing that by utilizing type-3 fuzzy logic in the aggregation phase, we are able to outperform type-2 and type-1 fuzzy logic in time series prediction.
Recently, the very rapid propagation of COVID-19 occurred, including several waves, and spreading to all continents of the world. In particular, in the case of Europe, several countries, such as Italy, Spain, and France, were hit hard with the spread of the COVID-19 virus, with a significant number of confirmed cases and deaths [18][19][20][21][22][23]. In the case of the American continent, the United States, Canada, and Brazil have also experienced a significant number of cases due to the rapid spread of COVID-19 [24][25][26]. There are also several recent works on the prediction and modeling of COVID-19 behavior in space and time [27][28][29]. In addition, it has been recognized that the COVID-19 waves have affected the economy (such as the Dow Jones in the USA). For this reason, we consider both time series to test the proposed model.
The main goal of this paper is the proposal of a prediction method based on an ensemble of neural networks with an interval type-3 fuzzy aggregator to combine the predictions of the modules. We believe that the proposed method has the potential to work for any complex time series and the reasoning behind this statement is as follows: It has been shown that individual neural networks can outperform other methods in prediction and ensembles of neural networks can outperform individual networks by having a set of predictors (such as a group of experts) to achieve the prediction of complex time series problems. Of course, to ensure an ensemble produces the best results, an aggregator for combining the outputs of the modules is needed. In this case, a fuzzy system is used. In particular, a type-3 fuzzy system is utilized to manage the uncertainty in combining the prediction of the modules, and, in this way, the best results are obtained. In this work, we test the proposal with an ensemble relating the prediction of COVID-19 data with the Dow Jones time series, but other time series could be tested in the future with the same approach.
The key contribution is the proposal of mathematical definitions of interval type-3 fuzzy theory, which were obtained by applying the extension principle to the type-2 fuzzy theory definitions. In addition, the utilization of interval type-3 fuzzy theory in the aggregation of neural network outputs (of an ensemble) for prediction has not been previously presented in the literature. This paper shows that interval type-3 fuzzy theory has the potential to be better than other methods in the literature for this task. We consider that these are important contributions to the frontier knowledge in soft computing and its applications.
The structure of this article is defined as follows: Section 2 introduces the basic terminology of interval type-3 fuzzy sets, Section 3 describes the proposed type-3 prediction method, Section 4 summarizes the results, and Section 5 outlines the conclusions and future works.

Interval Type-3 Fuzzy Logic
Interval type-3 fuzzy logic can be viewed as an extension of the type-2 models. We describe the basic terminology of type-3 fuzzy sets to show how it differs from its type-2 counterpart. Definition 1. A type-3 fuzzy set (T3 FS) [30][31][32][33], denoted by A (3) , is represented by the plot of a trivariate function, called the membership function (MF) of for short) and it is called a type-3 membership function (T3 MF) of the T3 FS: where U is the universe for the secondary variable u and V is the universe for the tertiary variable v. If the tertiary MF is uniformily equal to 1, then we have an interval type-3 fuzzy set (IT3 FS) with interval type-3 MF (IT3MF). Figure 1 illustrates IT3 FS with IT3MF µ(x, u), where µ(x, u) is the LMF and µ(x, u) is the UMF. The embedded secondary T1 MFs in x of A and A are f x (u) and f x (u). where is the universe for the secondary variable and is the universe for the tertiary variable . If the tertiary MF is uniformily equal to 1, then we have an interval type-3 fuzzy set (IT3 FS) with interval type-3 MF (IT3MF).  In this case, we utilize interval type-3 MFs that are scaled Gaussians in the primary and secondary, respectively. This function can be represented as, where * = (ℓ) ( ) and is the machine epsilon. If ℓ = 0, then * = . Then, ( ) and ( ) are the upper and lower limits of the DOU. The range ( ) and radius of the FOU are: The apex or core, ( ), of the IT3 MF ( , ) is defined by the expression: In this case, we utilize interval type-3 MFs that are scaled Gaussians in the primary and secondary, respectively. This function can be represented as, µ A (x, u) = ScaleGaussScaleGauss IT3MF, with a Gaussian footprint of uncertainty where σ * = σ ln( ) ln(ε) and ε is the machine epsilon. If = 0, then σ * = σ. Then, u(x) and u(x) are the upper and lower limits of the DOU. The range δ(u) and radius σ u of the FOU are: The apex or core, m(x), of the IT3 MF µ(x, u) is defined by the expression: where ρ = (σ + σ * )/2. Then, the vertical cuts with IT2 MF, µ A(x) (u) = µ A(x) (u), µ A(x) (u) , are described by the equations: Axioms 2022, 11, 251 4 of 13 where σ * u = σ u ln( ) ln(ε) . If = 0, then σ * u = σ u . Then, µ A(x) (u) and µ A(x) (u) are the UMF and LMF of the IT2 FSs of the vertical cuts of the secondary IT2MF of the IT3 FS.
The interval type-3 fuzzy logic system (IT3 FLS) structure contains the same main components (fuzzifier, rule base, inference machine and, in the final stage, an output processing unit) as its analogous T2 FLSs. While in the case of T2 FLSs the final stage consists of the process of type reduction to T1 FS + defuzzification, in the case of an IT3 FLS, the output processor consists of type reduction to IT2 FS + defuzzification. The fuzzy operators of the inference machine of an IT3 FLS and the type reduction methods are equivalent to a T2 FLS, except that in the inputs and outputs, we have IT3 FSs in an IT3 FLS.

Proposed Method
The method consists of utilizing two neural networks and then combining their outputs with an interval type-3 fuzzy system to obtain a revised and improved forecast of one of the time series by taking into account the influence of the other series. Figure 2 illustrates the architecture of the proposed method, where we can appreciate that two time series enter the two modules (neural networks NN 1 and NN 2 ) and individual predictions P 1 and P 2 are obtained with corresponding increments ∆P 1 and ∆P 2 , respectively. Then, these increments are the inputs to the fuzzy system for aggregation, which are obtained after the inference process and increment for the time series number 1, and, finally, the prediction is computed.
The interval type-3 fuzzy logic system (IT3 FLS) structure contains the same main components (fuzzifier, rule base, inference machine and, in the final stage, an output processing unit) as its analogous T2 FLSs. While in the case of T2 FLSs the final stage consists of the process of type reduction to T1 FS + defuzzification, in the case of an IT3 FLS, the output processor consists of type reduction to IT2 FS + defuzzification. The fuzzy operators of the inference machine of an IT3 FLS and the type reduction methods are equivalent to a T2 FLS, except that in the inputs and outputs, we have IT3 FSs in an IT3 FLS.

Proposed Method
The method consists of utilizing two neural networks and then combining their outputs with an interval type-3 fuzzy system to obtain a revised and improved forecast of one of the time series by taking into account the influence of the other series. Figure 2 illustrates the architecture of the proposed method, where we can appreciate that two time series enter the two modules (neural networks NN1 and NN2) and individual predictions P1 and P2 are obtained with corresponding increments ΔP1 and ΔP 2, respectively. Then, these increments are the inputs to the fuzzy system for aggregation, which are obtained after the inference process and increment for the time series number 1, and, finally, the prediction is computed. The fuzzy rules for the aggregation of the results with two modules are: 1. If (ΔP1 is high) and (ΔP2 is low), then (ΔP is positive). 2. If (ΔP 1 is high) and (ΔP2 is medium), then (ΔP is negative small). 3. If (ΔP1 is high) and (ΔP2 is high), then (ΔP is negative large). 4. If (ΔP 1 is medium) and (ΔP 2 is low), then (ΔP is positive). 5. If (ΔP1 is medium) and (ΔP2 is medium), then (ΔP is negative small). 6. If (ΔP1 is medium) and (ΔP2 is high), then (ΔP is negative large). The fuzzy rules for the aggregation of the results with two modules are: 1.
If (∆P 1 is low) and (∆P 2 is high), then (∆P is negative large).
The design of the fuzzy rules is based on general knowledge of training neural networks with time series data. It is known that when two different time series are related, their prediction can be improved by taking into account their interrelations. The fuzzy rules are established based on this general knowledge, so that they reflect that when the time series number two increases, this makes the time series one decrease and the vice versa. This general knowledge was used in proposing the fuzzy rules. The interval type-3 system (seen in Figure 3) has as input the increment in the prediction values of each neural network, P 1 and P 2 , respectively. After the defuzzification, the type-3 system has as output the corresponding increment in the time series number 1. The proposed approach is tested using two time series that are supposed to be related; if we combine their information, the prediction can be improved. In particular, we consider the Dow Jones index as the first time series, and the time series of COVID-19 cases as the second one; considering their relation, the prediction of the Dow Jones can be improved.
The design of the fuzzy rules is based on general knowledge of training neural networks with time series data. It is known that when two different time series are related, their prediction can be improved by taking into account their interrelations. The fuzzy rules are established based on this general knowledge, so that they reflect that when the time series number two increases, this makes the time series one decrease and the vice versa. This general knowledge was used in proposing the fuzzy rules. The interval type-3 system (seen in Figure 3) has as input the increment in the prediction values of each neural network, P1 and P2, respectively. After the defuzzification, the type-3 system has as output the corresponding increment in the time series number 1. The proposed approach is tested using two time series that are supposed to be related; if we combine their information, the prediction can be improved. In particular, we consider the Dow Jones index as the first time series, and the time series of COVID-19 cases as the second one; considering their relation, the prediction of the Dow Jones can be improved. We note that although the previous explanation and description of the applies to two time series, this approach could be generalized or extended to any number of related time series. Of course, the number of neural modules and the number of fuzzy rules will increase accordingly. In this situation, a metaheuristic could be utilized to automate the process of generating the structure and rules for particular situations.
In Table 1, we show the specific parameters of the MFs, which were found by trial and error, and could be optimized in the future with metaheuristics to achieve even better results. Basically, Table 1 shows the centers and standard deviations of the Gaussian MFs. We note that although the previous explanation and description of the applies to two time series, this approach could be generalized or extended to any number of related time series. Of course, the number of neural modules and the number of fuzzy rules will increase accordingly. In this situation, a metaheuristic could be utilized to automate the process of generating the structure and rules for particular situations.
In Table 1, we show the specific parameters of the MFs, which were found by trial and error, and could be optimized in the future with metaheuristics to achieve even better results. Basically, Table 1 shows the centers and standard deviations of the Gaussian MFs.
Regarding the lower scale (λ) and lower lag ( ) parameters, after experimentation to achieve better results, they were found to be 0.9 and 0.2 for the inputs, respectively. On the other hand, for the outputs, they were found to be 0.8 and 0.2, respectively. These values could be optimized with a metaheuristic to further improve the results.
In Figures 4 and 5, we show the input MFs for both errors, respectively. In Figure 6, we illustrate the output MFs, respectively. The actual IT3 MFs are three dimensional, but in these figures, a view on the plane is shown for simplicity. Regarding the lower scale (λ) and lower lag (ℓ) parameters, after experimentation to achieve better results, they were found to be 0.9 and 0.2 for the inputs, respectively. On the other hand, for the outputs, they were found to be 0.8 and 0.2, respectively. These values could be optimized with a metaheuristic to further improve the results.
In Figures 4 and 5, we show the input MFs for both errors, respectively. In Figure 6, we illustrate the output MFs, respectively. The actual IT3 MFs are three dimensional, but in these figures, a view on the plane is shown for simplicity.        In Figure 8, we illustrate the inference for a particular value of one of the inputs, and in Figure 9, the type-reduction and defuzzification are shown. We implemented the operations corresponding with type-3 to achieve these results using the computer programs in Matlab. In Figure 8, we illustrate the inference for a particular value of one of the inputs, and in Figure 9, the type-reduction and defuzzification are shown. We implemented the operations corresponding with type-3 to achieve these results using the computer programs in Matlab.  In Figure 8, we illustrate the inference for a particular value of one of the inputs, and in Figure 9, the type-reduction and defuzzification are shown. We implemented the operations corresponding with type-3 to achieve these results using the computer programs in Matlab.

Simulation Results
The experiments were performed with a dataset from the Humanitarian Data Exchange (HDX) [18], which includes COVID-19 data from countries of cases that occurred from January of 2020 to May, 2021 In addition, the Dow Jones time series for the same period of time was collected for use in the experiments [34]. Three different periods of 15 days were used for the testing stage. The first period is from 4 July to 18 July of 2021, the second period is 25 September to 9 October of 2021, and the third one is from 13 January to 27 January of 2022. The main idea is that the Dow Jones forecast can be improved by taking into account the influence of the COVID-19 time series for the USA, as the pandemic affected the economies of countries. The fuzzy aggregator is used to combine the information of the prediction of both time series to obtain a revised and improved prediction of the Dow Jones.

Simulation Results
The experiments were performed with a dataset from the Humanitarian Data Exchange (HDX) [18], which includes COVID-19 data from countries of cases that occurred from January of 2020 to May, 2021 In addition, the Dow Jones time series for the same period of time was collected for use in the experiments [34]. Three different periods of 15 days were used for the testing stage. The first period is from 4 July to 18 July of 2021, the second period is 25 September to 9 October of 2021, and the third one is from 13 January to 27 January of 2022. The main idea is that the Dow Jones forecast can be improved by taking into account the influence of the COVID-19 time series for the USA, as the pandemic affected the economies of countries. The fuzzy aggregator is used to combine the information of the prediction of both time series to obtain a revised and improved prediction of the Dow Jones.
The two modules of the neural network were trained with the COVID-19 time series and Dow Jones data from January of 2020 to May, 2021, respectively. Recurrent neural networks were used, with 3 delays, 300 epochs of training, and backpropagation with momentum learning and an adaptive learning rate. Three layers were used in each of the networks. Table 2 shows the forecasts of the two neural networks (Dow Jones NN and COVID NN), and the obtained increments of both time series (Inc Dow Jones and Inc COVID), which are the inputs to the type-3 fuzzy system. Finally, the forecast obtained with the type-3 fuzzy aggregator is shown (Dow Jones IT3) and the real value of the Dow Jones for comparison. In Figure 10, we illustrate the prediction with the interval type-3 and neural networks compared with the real data of the Dow Jones for this first testing period from 4 to 18 July 2021. Table 3 shows the forecasts, in the same way, for the second period from 25 September to 9 October of 2021, and Figure 11 illustrates a comparison of the prediction compared and the real values. Finally, Table 4 shows the Dow Jones forecast for the third period from 13 January to 27 January of 2022 and Figure 12 illustrates the prediction.          Additionally, we consider another simulation set up by changing the lower scale (λ) and lower lag (ℓ) parameter values in all MFs, which are now considered to be 0.8 and 0.3, respectively. In Table 5   Additionally, we consider another simulation set up by changing the lower scale (λ) and lower lag ( ) parameter values in all MFs, which are now considered to be 0.8 and 0.3, respectively. In Table 5  From the previous tables, we can conclude that the prediction with the interval type-3 fuzzy approach for the aggregation of neural networks is a good alternative in the prediction of complex time series, such as COVID-19 and the Dow Jones. In particular, for the third period, the prediction error of the Dow Jones is less than 1% on most days of the period, with some days with errors in the range of 0.3 to 0.6%. We note that previous studies with type-2 and type-1 fuzzy or even stand-alone neural networks could not achieve these values. The proposed approach can be extended to problems with multiple time series by using more neural networks and utilizing a fuzzy aggregator with more fuzzy rules.

Conclusions
In this work a new approach for type-3 fuzzy aggregation of neural networks was outlined. The main idea is to combine the predictions of related time series to improve the final predictions. In our approach, a fuzzy system is utilized to consider the predictions of neural networks as the inputs in the combination process to obtain a revised and improved prediction. The uncertainty in the process of aggregation is modeled with the interval type-3 fuzzy system as, theoretically, it should handle this better than the type-2 and type-1 fuzzy systems. The simulation results of the Dow Jones and COVID-19 time series showed the potential of the approach to outperform other methods. As future work, we plan to use our approach in other applications, such as the ones discussed in [35,36]. Moreover, we plan to optimize the type-3 system using metaheuristics to improve the results. In addition, we will formulate a general method to consider the problems of multiple time series (more than two), where we could automate the generation of the optimal fuzzy system (rules and parameters). Finally, we plan to combine type-3 with other intelligent techniques to build strong hybrid models and consider other time series prediction problems.