Multiresolution Forecasting for Industrial Applications

Round 1

Reviewer 1 Report

It is difficult to provide any positive comments for this manuscript. The major problem is the novelty of presentation. This manuscript looks like an essay - not as a scientific research manuscript. 

What is the main contribution of this study? Industrial applications based on time series prediction? Clearly, the novelty is missing here. The authors do not propose any new approach, any new strategy for time series prediction. The claim that the algorithms are easily implemented in R or Python - and can run with low computational resources - does not sound seriously. 

First of all, the authors completely miss the classification (and the positioning) of their predictors in terms of short-term or long-term time series prediction. Short/long historical data, short/long prediction horizons require completely different techniques and algorithms. These issues are ignored in the manuscript. 

The literature overview does not represent the existing state of the art in time series forecasting. Some parts of the literature overview are completely naive. Figure 1 and relevant discussions are not appropriate for a serious technical manuscript. The authors complete the literature overview by a short discussion on ARIMA class methods. Huge areas of forecasting techniques are completely ignored. The authors shortly discuss evolutionary optimization algorithms for the parameter selection (line 8) - but these issues are ignored in the text of the manuscript. 

A separate section is devoted to multi-resolution methods (Section 2). But it is difficult to understand the role of this section because it again looks like some sort of an essay. Coarse grain time series analysis techniques is a well-developed area of research. The authors neither provide a proper overview on existing methods, nor present some sort of innovation. 

The presentation style is also inappropriate. Apparently, the authors did not proof read the manuscript before submitting to the Journal. A typical example is line 6 where the phrase "different wavelet models." is dropped at the end of the sentence by repetition. 

Said that it is sufficient to recommend a straight reject decision. 

 

Author Response

Please note that the responses are marked in red in the attached Word document.

1)
It is difficult to provide any positive comments for this manuscript. The major problem is the novelty of presentation. This manuscript looks like an essay - not as a scientific research manuscript.
We are grateful that the reviewer took the time to review this paper. The novelty of presentation is now more explicitly stated in 1). We revised the manuscript by trying to follow your specified criteria by citing significantly more literature throughout the manuscript for the argumentations taken here. Additionally, we explicitly now defined our model selection and revised our contribution in the methods part as presented in point 6 later. 
2)
What is the main contribution of this study? Industrial applications based on time series prediction? Clearly, the novelty is missing here.
The accessible open-source framework performs equally or better than state of the art on typical industrial applications which directly lies within the scope of the special calls for papers by Prof. Dr. Chien-Chih Wang. Performance of Wavelet Forecasting methods was not compared to our knowledge to seasonal time series methods systematically for industrial applications. We revised a more precise contribution in the manuscript. The contribution of the work is now stated explicitly in the introduction as follows: (Line 78-84)
“
    An open-source and application-oriented wavelet framework combined with an automatic model selection through differential evolutionary optimization with standardized access in Python and R.
    Contrary to prior works, a systematic comparison of state-of-the art and open-source accessible seasonal univariate forecasting methods to our framework.
    Wavelet forecasting performs equally well on short-term and long-term forecasting
“
3)The authors do not propose any new approach, any new strategy for time series prediction.
Our work is an applied science manuscript in line with the special call. As such we provide a framework for a priorly discussed wavelet approach in theory and resolve the practical issues of model selection and parameter setting by using evolutionary optimization as stated in the introduction Page 2: (Line 52-55)
“We propose the use of a differential evolutionary optimization strategy to solve the complex model selection problem in the wavelet forecasting concept introduced by (Renaud et al., 2005a). This strategy does not require the user to set any parameters, since such task can be performed automatically by computational intelligence. Our algorithms are open-source accessible in Python and R (Stier, 2021a) and (Stier, 2021b).”
4)
The claim that the algorithms are easily implemented in R or Python - and can run with low computational resources - does not sound seriously. 
Claims such as “easy” was not made but the naïve claim “fast” were removed, which was at Line 7 in the old manuscript and the claim “straightforward” in the same line was replaced with the more appropriate expression “swiftly accessible”.
5)
First of all, the authors completely miss the classification (and the positioning) of their predictors in terms of short-term or long-term time series prediction. Short/long historical data, short/long prediction horizons require completely different techniques and algorithms. These issues are ignored in the manuscript.
We thank the reviewer for the chance to clarify this important point. We revised the manuscript as follows: (Line 103-114 )
The focus of this work is the forecasting of seasonal univariate time series. Therefore, open access forecasting methods are selected that are specially designed for dealing with seasonality (periodic patterns) independently of the classification into short- and long-term methods. Although in the general case different forecast horizons require different methods (Athanasopoulos et al., 2017), Fourier decomposition of time series allows to use methods like Prophet in both cases (Taylor and Letham, 2018). Moreover, short- and long-term forecasting strategies usually depends strongly on the resolution of the time series (Makridakis et al., 2020) and (Weron, 2014) whereas periodic patterns do not necessarily depend on the resolution. Hence, the performance of all methods over all 14 horizons are computed and grouped by horizon one versus horizon above one (multi-step). Implicitly, the performance of the one-step forecasts represents a short-term evaluation and the multi-step forecasts represent a long-term evaluation. In both cases, the distribution is estimated visualized separately with the MD plot.

6)
The literature overview does not represent the existing state of the art in time series forecasting.
We revised the literature overview to which explicitly now states that the state of the art of open-source accessible methods in Python and R is the following: (Line 123-151)
“The current state-of-the-art according to accessible open-sources for time series forecasting in Python and R contains following techniques: ARIMA, Cubic Spline extrapolation, decomposition models, exponential smoothing, Croston, MAPA, naive/random walks, neural networks, Prophet and the Theta method. Two automatized forecasting methods are used to represent the current state-of-the-art for ARIMA models: The first one is RJDemetra which is an ARIMA model with seasonal adjustment according to the “ESS Guidelines on Seasonal Adjustment” (Eurostat, 2015) available from the National Bank of Belgium using two leading concepts TRAMO-SEATS+ and X-12ARIMA/X-13ARIMA-SEATS (Abeln et al., 2019) and referred to as SARIMA'' (or short SA'') (Quartier-la-Tente et al., 2020) and an automatized ARIMA referred to as AutoARIMA'' (or short `AA'') (Hyndman and Khandakar, 2008) and (Smith, 2021). Modelling ARIMA for time series forecasting follows an objective and thus can be completely automatized by optimizing an information criterion for which AutoARIMA and SARIMA are two different approaches  (Hyndman and Khandakar, 2008). Cubic spline extrapolation is a special case of the ARIMA(0,2,2) model (Hyndman et al., 2005) and is represented by the ARIMA models. Croston is not used since it does only provide one-step forecasts (Croston, 1972). The Multi Aggregation Prediction Algorithm (MAPA) uses exponential smoothing as forecasting technique on multiple resolution levels which are recombined into one forecast on one specified resolution level (Athanasopoulos et al., 2017) Since the combination of forecasts tend to yield better results (Timmermann, 2006) MAPA represents the exponential smoothing methodology. The naive method or random walk is incorporated in the quality measure used in the results section. In order to represent neural networks (Chollet and others, 2015), a multilayer perceptron and a long short-term memory method with one hidden layer is used, since neural networks were recommended as robust forecasting techniques if they have at least one hidden layer (Tang et al., 1991). Prophet is a decomposition model using Fourier theory specially designed for seasonal time series forecasting (Taylor and Letham, 2021) and (Taylor and Letham, 2018). Therefore, no other decomposition model is used besides Prophet. Forecasts with the Theta method ``are equivalent to simple exponential smoothing with drift'' (Hyndman and Billah, 2003) and here represented with MAPA. Furthermore, XGBoost (Chen and Guestrin, 2016) is included since it was recommended as robust algorithm for general machine learning tasks (Friedman, 2001).
”
Furthermore, we found another wavelet framework which we describe now in the methods part briefly and incorporate in the benchmark study. The conclusion remains positive and our main propositions remained unchanged: (Lines 57-67 ):
“
Currently, there are accessible methods in Python or R for computing the wavelet decomposition for different task and with different approaches, such as for signal processing with the continuous wavelet transform (Virtanen et al., 2020), signal and image processing with various wavelet transforms (Lee et al., 2019) and (Aldrich, 2020), wavelet analysis with scalogram tools (Bolos and Benitez, 2021), univariate and bivariate wavelet analysis (Gouhier et al., 2021), wavelet analysis (Paul, 2019), signal processing (Roebuck, 2014), signal processing for one to three dimensions (Whitcher, 2020), wavelet statistics (Nason, 2016) and testing white noise with wavelets (Nason, 2018). There are three packages in R dealing with time series forecasting with the wavelet decomposition. The first uses a redundant Haar wavelet decomposition in combination with an ARIMA framework (Paul, 2018). The second one uses a redundant Haar wavelet decomposition combined with an artificial neural network (Paul, 2019). The last one uses GARCH and is of no interest here (Paul et al., 2020). 
“
7)
Some parts of the literature overview are completely naive.
We kindly ask the reviewer to provide constructive criticism instead of a personal and vague rating of our manuscript. Nevertheless, we tried to speculate about the issue raised and revised the literature overview in the introductory part from
“
Seasonal time series forecasting started with seasonal decomposition methods like the X-11 [11] or various variants. Other techniques consist of decomposition methods like the Holt-Winters-method, the theory of ARIMA models for time series forecasting [12,13], regression approaches and exponential smoothing.
”
to (Line 18-22)
“
Seasonal time series forecasting with computers had early success with seasonal adjusted methods as proposed in 1978 (Box et al., 1978) or with the  the X-11 method (Shiskin and Eisenpress, 1957)(Abeln et al., 2019). Over the years, improved versions such as the X-13 (Abeln et al., 2019), various variants of such models and new techniques in the area of statistical models were developed  (De Gooijer and Hyndman, 2006) and a new field emerged in computational intelligence dealing with seasonal time series forecasting arose (Stepnicka et al., 2013).
“
and gave in point 6 are clear statement on the current state-of-the-art in forecasting according to the literature.

8)
Figure 1 and relevant discussions are not appropriate for a serious technical manuscript.
The journal “provides an advanced forum for process/system-related research in chemistry, biology, material, energy, environment, food, pharmaceutical, manufacturing and allied engineering fields”. Hence, we assume that not all readers are familiar with the topic contrary to readers of technical journals like the journal of forecasting. 
Figure 1 represents a simplified representation for a Fourier and Wavelet transform in order to ease the reader of applied science into the complex topic which is the reason we provide it in the introduction. 
In our opinion the discussions for Figure 1 are quite technical and elaborate which may seem unnecessary for a reader that has in-depth knowledge about decomposition techniques, but helps the reader who has few knowledge in this topic to grasp the importance of the idea behind. 
We encourage the reviewer to rethink the statement above.
9)
The authors complete the literature overview by a short discussion on ARIMA class methods.
We extended the literature overview to ARIMA: (Line 127-135)
“
Two automatized forecasting methods are used to represent the current state-of-the-art for ARIMA models: The first one is RJDemetra which is an ARIMA model with seasonal adjustment according to the “ESS Guidelines on Seasonal Adjustment” (Eurostat, 2015) available from the National Bank of Belgium using two leading concepts TRAMO-SEATS+ and X-12ARIMA/X-13ARIMA-SEATS (Abeln et al., 2019) and referred to as SARIMA'' (or short SA'') (Quartier-la-Tente et al., 2020) and an automatized ARIMA referred to as AutoARIMA'' (or short `AA'') (Hyndman and Khandakar, 2008) and (Smith, 2021). Modelling ARIMA for time series forecasting follows an objective and thus can be completely automatized by optimizing an information criterion for which AutoARIMA and SARIMA are two different approaches  (Hyndman and Khandakar, 2008). 
“
10)
Huge areas of forecasting techniques are completely ignored.
We specified in the introduction that the work focuses on seasonal univariate time series forecasting with accessible open-source implementations.
We added to the introduction (Line 120-123)
“As it is a common issue in data science that published techniques are not implemented (e.g. swarm based techniques [Thrun/Ultsch, 2021]), this work focuses on methods for which open-source implementations are provided through either the Comprehensive R Archive Network (CRAN) or the Python Package Index (pypi).”
An now provide an extensive overview over this subset of methods in point 7.
11)
The authors shortly discuss evolutionary optimization algorithms for the parameter selection (line 8) - but these issues are ignored in the text of the manuscript.
We included a part about evolutionary optimization in the multiresolution methods part: (Line 348-387)
“
The lagged coefficient selection described above has a complexity of O(\Pi_{i=1}^j A_j^{max}) where A_j^{max} denotes the maximum possible number of coefficients at scale j. Here, finding the best wavelet model means to find the best combination of number of decomposition levels and at the same time finding the best number of coefficients per each level, which fits historical data with the goal to forecast future time points (Eiben and Smith, 2003). There is a potentially large set of possible input parameters which define the model of our framework and the output for each input is obtained by a potentially complex computation (e.g., rolling forecasting origin (Eiben and Smith, 2003). This can be viewed as a search problem. A simple but complex solution would be the search through all possible inputs which we found not practical for such complexity. Therefore, a more sophisticated approach is required. The approach used in this work is an ``differential evolutionary optimization''. A population is randomly initialized which stands in competition forcing a selected reproduction based on a fitness function (survival of the fittest) (Eiben and Smith, 2003). The starting set of candidates can be randomly initialized (Eiben and Smith, 2003). The fitness is based on a quality measure (e.g., for measuring the forecast performance). The best candidates are chosen (survival of the fittest) (Eiben and Smith, 2003). Those candidates (parents) are used to generate the next generation (Eiben and Smith, 2003). The two operations for building the next generation are recombination and mutation (Eiben and Smith, 2003) The new set of candidates is called children (Eiben and Smith, 2003). The new selection is based on a fitness function based on the quality measure and the age of the candidates. This procedure is iterated until a stopping condition is reached. This can be for example a sufficient quality level or a maximum number of steps. In our framework, we evaluate each possible decomposition with J+1 levels separately. The vector x = \{A_1,...,A_J,A_{J+1} \} carries the number of coefficients associated with the respective wavelet level J or the last smooth approximation level. The difference between classical evolutionary optimization and differential optimization is that the candidate solutions are vectors x \in \mathbb{N}^{J+1} and the new mutant \hat{x} is produced by adding a perturbation vector p \in \mathbb{N}^{J+1} to an existing one: 
 \hat{x} = x + p 
where p is a scaled vector difference of two already existing, randomly chosen candidates, which are rounded to yield integer vectors: 
p = \lfloor F \cdot (y - z) \rfloor
and F > 0 is a real number which controls the evolution rate. In this work, we use 2 to 5 decomposition level in the model selection procedure for our framework. We allow 1 to 15 coefficients per level for the regression method and 1 to 8 coefficients per level for the neural network. Our multiresolution forecasting framework with a neural network is denoted as `MRNN'' and with a regression as `MRR''. The difference to the multiresolution method in (Anjoy and Paul, 2019) and (Aminghafari and Poggi, 2007) is the lagged coefficient selection and the computation of the forecast based on a prediction scheme using the wavelet decomposition as one unit without the reconstruction scheme. Furthermore, our proposed multiresolution framework does not require the user to set any parameters since this will be completely undertaken by the model selection based on the differential evolutionary optimization.
“
12)
A separate section is devoted to multi-resolution methods (Section 2). But it is difficult to understand the role of this section because it again looks like some sort of an essay.
We adjusted the manuscript in order to state it more precisely with the answer in point 2, 3 and 11. We also changed the notation to adapt a more scientific style as in Lines 340-344.
13)
Coarse grain time series analysis techniques are a well-developed area of research.
Ware are grateful to the reviewer for this new idea. We added to the discussion: (Line 582-586)
“As an alternative, Coarse grain time series analysis techniques could be used to model time series (Kaplan and Glass, 1993) and create short-term predictions (Sugihara and May, 1990). However, in this work, we restrict our evaluation to a linear and nonlinear strategy in order to investigate the potential of wavelets for time series forecasting, although the wavelets used could be processed in many other methods as well (Renaud et al., 2005b).
“
14)
The authors neither provide a proper overview on existing methods, nor present some sort of innovation.
We outlined the overview more clearly to show which aspects of literature are relevant for this manuscript (please see point 6 and 7). Our contribution is now explicitly stated in point 1 and point 11. As this applied science manuscript, there is no necessity to present an overview over theoretically proposed methods for which no open-source code exists because such methods would not be used in industrial praxis. 
However, a reperformed survey of literature revealed two published and open-source code accessible wavelet forecasting methods which we overlooked in the first draft of the manuscript. Although these methods are accessible on CRAN, they were not compared to other methods in their publication and are not accessible in Python. Hence, we added to the methods section a description of the two existing methods as follows in Lines 152-171:
“There are two related forecasting frameworks using wavelets: (Anjoy and Paul, 2019) and (Aminghafari and Poggi, 2007) proposed independently of each other and so far not compared to each other.  Both methods are based on the redundant Haar wavelet decomposition.  In both methods, the wavelets and the last smooth approximation levels are forecasted on each level separately. (Anjoy and Paul, 2019) incorporates artificial neural networks for that purpose, whereas (Aminghafari and Poggi, 2007) incorporates an ARIMA framework in contrast to our method which uses regression optimized with least squares (Sachs and Hedderich, 2002). In these two comparable methods, the final forecasts are obtained by exploiting the additive nature of the reconstruction of the redundant Haar wavelet decomposition. Thus, a forecast is created by forecasting each level of this decomposition separately and by reconstructing the time series value (forecast) from it. Their methodology is opposed to the approach we take in this work. However, both methods do not provide a framework for model selection. Instead, the parameters like the wavelet levels, have to be specified by the user. Hence, in this study, their parameters remain in the default setting specified by the documentation of the packages, i.e., require parameters indicating the number of decomposition levels (`Waveletlevels''), a boundary condition (boundary''), the maximum non seasonal order (nonseaslag''), and the maximum seasonal order (`seaslag''). Such parameters are not necessary in our proposed multiresolution framework. In the following sections, the wavelet method using an artificial neural network is denoted as `MRANN'', whereas the method using ARIMA is denoted as `MRA''. The abbreviation `MR'' stands for `multiresolution''.
And to the results section a new subsection in which we compare our method to these two methods: (Line 426-436) 
The MASE of Prophet, MAPA, MRANN, MRNN and MRR on dataset Electricity are the only one below or equal 1 and therefore better than the seasonal naive method, see table 1. In case of dataset Electricity, the four multiresolution methods outperforms the seasonal naive method.
On dataset Callcenter, the two multiresolution methods with neural networks (MRNN) and with an artificial neural network (MRANN) outperforms every other method. The only other method performing better than the seasonal naive method for horizons larger than 1 is Prophet, which can be seen by the explicit value of central tendency of the MASE distribution that serves as comparison (see the overview for MASE in table 1).
“
And to the discussion: (Line 506-531) 
“
The MRR as well as MRNN (proposed multiresolution framework), and the MRANN (Anjoy and Paul, 2019) method perform similar across horizons and for the overall measurements (see tables1, 2, 5, and 6). For the datasets investigated, the multiresolution method with neural networks (MRNN and MRANN) tends to have larger outliers than the MRR. The MRNN has better one-step forecast performance than the MRR, whereas it is the opposite for multi-step forecasts. Computing the mean average of the MASE it could be deduces that MRANN performs better than MRR or MRNN in tables 1 and 2. However, the estimated probability density functions in the MD plot are highly similar. Moreover, according to the Kruskal-Wallis test the null hypothesis that the location parameters of the distribution of x are the same, cannot be rejected (p-value >0.05) in case of the one-step forecasts of MRANN vs MRNN and the multi-step forecasts in case of MRANN vs MRR. The difference between the overall measurements is comparably small on the seasonal datasets Electricity and Callcenter (Callcenter - MRANN: 0.9 and 0.9, MRNN: 1.0, 1.1, MRR: 1.1, 1.1; Electricity – MRANN: 0.6, 0.7, MRNN: 0.7, 0.7, MRR: 0.7, 0.7 – see overall measurements in the last two columns of tables 1 and 2). The performance of the forecasting method combining the concepts of multiresolution and ARIMA (Aminghafari and Poggi, 2007) does not perform comparable well to the investigated methods here, although  (Aminghafari and Poggi, 2007) based their work on (Renaud et al., 2003) similar to our proposed work. However, they used ARIMA as underlying forecasting method und achieved worse results (for both seasonal datasets, MASE for MRA is above 2).
In contrast, (Anjoy and Paul, 2019)forecasted each level (wavelet and the last smooth approximation level) separately with an ANN and then reconstructed the forecast by applying the reconstruction formula on the forecasted wavelet decomposition. Even though their proposition overlaps with (Aminghafari and Poggi, 2007) and this work by also using Haar wavelet decomposition. Anjoy and Paul, 2019 forecasting framework using wavelets yielded a similar estimated probability density function of MASE in comparison to our work. It should be noted that in their work no comparison with a method comparably to the approach proposedby Renaud et al, 2003 and 2005 is made.
“

And to the conclusion: (Line 593-597 and 599-601 respectively) 
“
For one-step forecasts, the multiresolution methods MRR, MRNN and MRANN outperforms almost every other method on three seasonal time series datasets and performs as expected based on the random walk theory on the Stocks dataset. Multiresolution methods perform on the seasonal datasets even better than Prophet
In sum, we conclude based on our benchmarking to use for short-term forecasting MRNN and for long-term forecasting MRR.
“
Additionally, we stated the innovation in our main contribution in the points above.

16)
The presentation style is also inappropriate. Apparently, the authors did not proof read the manuscript before submitting to the Journal. A typical example is line 6 where the phrase "different wavelet models." is dropped at the end of the sentence by repetition.
This was corrected in the revised manuscript. Additionally, some minor typos were corrected.
17)
Said that it is sufficient to recommend a straight reject decision. 
Thank you again for the time you took to review our work and for your vague critic. It seems, that you reviewed the manuscript as a technical manuscript and not an applied science manuscript. Due to the vagueness for your comments, we could only try to derive which improvements you would find necessary which we revised extensively. We kindly ask you to reconsider your decision or at least specify explicitly which sentences require revision and why. 

 

Author Response File: Author Response.docx

Reviewer 2 Report

The paper proposes an approach of comparing different forecasting techniques and tries developing a method of model selection. The authors say (in the Abstract) that their approach “proposed here is the convenience of a straightforward and fast implementation in R and Python combined with coefficient selection through evolutionary optimization”. The effectiveness of the method is demonstrated by the forecasting using four real datasets from real-world applications: scheduling of a call center, planning electricity demand, and predicting stocks and prices.

The article contains a large number of citations of modern literature from the field to which the work is focused. As far as the Reviewer understands, the authors have applied several forecasting methods to multiple datasets in order to analyze their performance and then select an appropriate model.

At the same time, an unambiguous evidence conclusion that the proposed method allows to choose one model from the set that would most effectively solve the forecasting problem (in a specific area) is not given in the work.

The whole work consists in fact in the implementation of well-known methods and approaches to forecasting based on various time series models or machine learning methods such as neural networks. Further, the forecasts obtained by this set of methods are subjected to "analysis", which consists in calculating some "quality indicators" for a very limited sample. These quality indicators are the errors between the forecast and real data, calculated using special formulas suggested in some of the cited sources. Using an analog of the cross-validation method for time series, these errors are used to build some diagrams, which considered to be used to roughly judge the quality of the predictive model.

The work is done accurately, contains a large number of well-designed graphs and tables, and is a numerical study of a certain methodology for assessing the quality of predictive models. The article does not carry out a theoretical analysis of estimates of the quality of predictive (forecasting) models, and the properties of these estimates are not used in forecasting. This approach is widespread in the field of machine learning and data mining, however, from the point of view of mathematical forecasting theory, the results obtained cannot be considered evidential.

The work also contains a number of typos.

In this regard, my recommendations are as follows: the article can be published in the journal after correcting typos and receiving answers from the authors to my comments above.

Author Response

1)
The paper proposes an approach of comparing different forecasting techniques and tries developing a method of model selection. The authors say (in the Abstract) that their approach “proposed here is the convenience of a straightforward and fast implementation in R and Python combined with coefficient selection through evolutionary optimization”. The effectiveness of the method is demonstrated by the forecasting using four real datasets from real-world applications: scheduling of a call center, planning electricity demand, and predicting stocks and prices.
The article contains a large number of citations of modern literature from the field to which the work is focused. As far as the Reviewer understands, the authors have applied several forecasting methods to multiple datasets in order to analyze their performance and then select an appropriate model.
We are grateful for the time the reviewer took for revising our manuscript.
2)
At the same time, an unambiguous evidence conclusion that the proposed method allows to choose one model from the set that would most effectively solve the forecasting problem (in a specific area) is not given in the work.
Thank you for pointing that out this very general issue which in principle cannot be proven. We revised our manuscript and refined our statements in the discussion: (Line 452-460)
“
In general, the learnability of machine learning methods for data cannot be proven [Ben-David et al., 2019]. Hence, we follow the typical approach of supervised methods by dividing the data and estimating the learnability on test data as described in section 1.1 . The quality indicators are evaluated on a large enough sample (here: 365 steps) for distribution analysis [Thrun et al., 2020a] under the assumption that the largest seasonality lies within a year. In our evaluation of test data with MASE lower than one implicates a better Mean Average Error than the naive method. Thus, our results indicate that the multiresolution method and Prophet are able to forecast the seasonal datasets more successful than the naive forecast and other methods.
“
3)
The whole work consists in fact in the implementation of well-known methods and approaches to forecasting based on various time series models or machine learning methods such as neural networks. 
We revised the conclusion for clarity: (Line 588-590) 
“
The work brings a wavelet method to the point of automatized application for industrial tasks without the need to set any parameters and provides a comparison of performance with state-of-the art methods. 
“
4)
Further, the forecasts obtained by this set of methods are subjected to "analysis", which consists in calculating some "quality indicators" for a very limited sample. 
We revised the manuscript as specified in point 2: (Line 447-450) 
“In this work, the quality of the predictive models for practical purposes is evaluated for 365 time steps. With such sample size, the MD plot is able to discover fine details of the underlying distribution (Thrun et al., 2020) under the assumption that the largest seasonality lies within a year.”
5)
These quality indicators are the errors between the forecast and real data, calculated using special formulas suggested in some of the cited sources. Using an analog of the cross-validation method for time series, these errors are used to build some diagrams, which considered to be used to roughly judge the quality of the predictive model.
We made sure in the discussion, that our approach of evaluating the performance is in accordance with literature: (Line 442-446) 
“There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown” (Breiman, 2001). “Forecasters generally agree that forecasting methods should be assessed for accuracy using out-of-sample tests rather than goodness of fit to past data (in-sample tests)”'  (Tashman, 2000).
6)
The work is done accurately, contains a large number of well-designed graphs and tables, and is a numerical study of a certain methodology for assessing the quality of predictive models. The article does not carry out a theoretical analysis of estimates of the quality of predictive (forecasting) models, and the properties of these estimates are not used in forecasting.
A theoretical proof is, in principle, impossible for machine learning methods (see point 2).

7)
This approach is widespread in the field of machine learning and data mining, however, from the point of view of mathematical forecasting theory, the results obtained cannot be considered evidential.
We approach time series forecasting from a practical view, which we already stated in a refined version under your point 6 and 3.
We revise the manuscript in the discussion to hint the reader to the raised issue: (Line 450-452) 
“It should be noted, that restricting evaluation to specific datasets makes it challenging to provide evidential results that can be generalized which in our opinion remains a great challenge in forecasting. 
“ 
8)
The work also contains a number of typos.
We proofread the manuscript and corrected a number of typos that we could find.
9)
In this regard, my recommendations are as follows: the article can be published in the journal after correcting typos and receiving answers from the authors to my comments above.
Thank you for reviewing our manuscript.

 

Author Response File: Author Response.docx

Reviewer 3 Report

The authors researched how to implement the multiresolution analysis in forecasting for industrial applications. They used a Haar wavelet on four (4) data sets (labeled as: “Electricity,” “Stocks,” “Prices,” and “Callcenter”). Further, they adopted the so-called out-of-sample method for cross-validation forecasts. Metrics for result comparisons were MASE and SMAPE. They showed that the multiresolution method is an appropriate method for seasonal forecasting and performs equally or better in comparison to the state-of-the-art methods like Prophet or MAPA for forecasting horizons higher than one. For one-step forecasts, the multiresolution method outperforms every other method on three seasonal time-series datasets and performs as expected based on the random walk theory on the Stocks dataset.

 

In my opinion, this is a very interesting report. The research is actual and worth exploring. The manuscript is well structured, and the research is well conducted. 

 

I have only the following minor questions:

1. In Figure 1 (frequency plot), instead of the ”AMPLITUDE„ label, it should be ”MAGNITUDE„. Fourier plot always shows magnitude vs. frequency plot.
2. Please, plot the four data sets (labeled as: “Electricity,” “Stocks,” “Prices,” and “Callcenter”), so the reader can see the data (seasonality, etc.).
3. For the multiresolution method, you used a Haar wavelet (Haars scaling function and mother wavelet). My question is: How many levels (approximations) you used to decompose data and then to determine c and w?

In the end, I have a suggestion for further work it would be interesting to compare the other wavelets (orthogonal and be-orthogonal) and choose the best for forecasting.

Author Response

The authors researched how to implement the multiresolution analysis in forecasting for industrial applications. They used a Haar wavelet on four (4) data sets (labeled as: “Electricity,” “Stocks,” “Prices,” and “Callcenter”). Further, they adopted the so-called out-of-sample method for cross-validation forecasts. Metrics for result comparisons were MASE and SMAPE. They showed that the multiresolution method is an appropriate method for seasonal forecasting and performs equally or better in comparison to the state-of-the-art methods like Prophet or MAPA for forecasting horizons higher than one. For one-step forecasts, the multiresolution method outperforms every other method on three seasonal time-series datasets and performs as expected based on the random walk theory on the Stocks dataset.

In my opinion, this is a very interesting report. The research is actual and worth exploring. The manuscript is well structured, and the research is well conducted. 
Thank you very much for your feedback.

I have only the following minor questions:
1)
In Figure 1 (frequency plot), instead of the ”AMPLITUDE„ label, it should be ”MAGNITUDE„. Fourier plot always shows magnitude vs. frequency plot.Thank you for pointing that out. We adjusted Figure 1 accordingly.
2)
Please, plot the four data sets (labeled as: “Electricity,” “Stocks,” “Prices,” and “Callcenter”), so the reader can see the data (seasonality, etc.).
In supplementary A, the figures are now plottet with the caption:
The time series line charts of the four investigated data sets: a) Callcenter, b) Electricity, c) Prices and d) stocks.
We added a few important lines brining the time series Prices into context in the methods part: (Line 233-235) 
“
External influences on electricity prices cause varying “seasonality at the daily, weekly and annual levels and abrupt, short-lived and generally unanticipated price spikes” (Weron, 2014).
“
3)
For the multiresolution method, you used a Haar wavelet (Haars scaling function and mother wavelet). My question is: How many levels (approximations) you used to decompose data and then to determine c and w?
We stated an answer to your question in the methods part of the multiresolution framework: (Line 378-380) 
“In this work, we use 2 to 5 decomposition level in the model selection procedure for our framework. We allow 1 to 15 coefficients per level for the regression method and 1 to 8 coefficients per level for the neural network.” 

4)
In the end, I have a suggestion for further work it would be interesting to compare the other wavelets (orthogonal and be-orthogonal) and choose the best for forecasting.
Your suggestion is very welcome. We already considered such method and will investigate it in future work. The conclusion now states additionally: (Line 602-604)
“
In future work, it would be interesting to further investigate time series forecasting with other wavelets (orthogonal and bi-orthogonal).
“

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The manuscript is seriously improved. It is pleasant to see the efforts of the authors to perform a major revision. 

However, few additional changes are recommended. 

Firstly, Fig. 1 should be deleted. It is completely sufficient to provide a short description in the text. Showing such a figure decreases the scientific impact of the manuscript. It would be distracting to specialists working in the area of signal processing. 

Secondly, the literature overview should be expanded. The authors were asked to describe forecasting techniques which do employ evolutionary algorithms for the selection of the system parameters. A typical reference could be: Short-term time series algebraic forecasting with internal smoothing. Neurocomputing (2014) vol.127, pp.161-171. 

The manuscript could be recommended for publication in the Journal if those minor changes are implemented into the revised manuscript.   

Author Response

The responses and requested corrections are attached in word document.

Author Response File: Author Response.docx