Abstract
It is well-known that the weighted averages of two competing forecasts may reduce mean squared prediction errors (MSPE) and may also introduce certain inefficiencies. In this paper, we take an in-depth view of one particular type of inefficiency stemming from simple combination schemes: Mincer and Zarnowitz inefficiency or auto-inefficiency for short. Under mild assumptions, we show that linear convex forecast combinations are almost always auto-inefficient, and, therefore, greater reductions in MSPE are almost always possible. In particular, we show that the process of taking averages of forecasts may induce inefficiencies in the combination, even when individual forecasts are efficient. Furthermore, we show that the so-called “optimal weighted average” traditionally presented in the literature may indeed be inefficient as well. Finally, we illustrate our findings with simulations and an empirical application in the context of the combination of headline inflation forecasts for eight European economies. Overall, our results indicate that in situations in which a number of different forecasts are available, the combination of all of them should not be the last step taken in the search of forecast accuracy. Attempts to take advantage of potential inefficiencies stemming from the combination process should also be considered.
Keywords:
forecast combinations; time series; mathematics; predictability; inflation; efficiency; international inflation; core inflation MSC:
91B84
1. Introduction
Decision makers are quite often confronted with two or more forecasts for the same target variable. In this scenario, [1] identifies two possible different strategies: the search for the best possible single forecasting method and the search for the best possible combination of the available forecasts. This last strategy has received a lot of attention in the literature since the seminal work of [2]. In their article, the authors combine two sets of forecasts coming from airline passenger data. They conclude that a composite forecast may display fewer mean squared prediction errors (MSPE) than either of the single original projections.
Even when the superior predictive ability of one forecast over another is suspected, one could test for forecast encompassing. Refs. [3,4] claim that the superior accuracy of one forecast over another does not necessarily mean that the inferior forecast is useless. It could be the case that the superior forecast could benefit from using some of the information contained in the outperformed forecast. Granger and Newbold consider the possibility that an average of the superior and inferior forecasts may yield a new and more accurate forecast. When this is not possible, it is said that the superior forecast is not only more accurate than the outperformed forecast but also encompasses it (see [5,6]).
Since 1969, a number of different papers have been written on topics directly or indirectly related to the combination of forecasts. A sample of the influential work from the next two decades includes [7,8,9,10]. More recent papers are also published on the topic including, for instance, [11,12,13,14,15,16,17,18,19,20,21,22].
Despite the huge variety of combination methods available in the literature, two particular families of combination strategies have attracted special attention. Using the terms in [10], these two families are known as the variance–covariance method of [2] and the regression method, introduced by [8]. Broadly speaking, the first approach generates the combined forecast as a weighted average of the pool of single individual forecasts. Notice that this weighted average does not need to be convex. The latter approach is one in which the combining weights are obtained as the coefficient estimates of a regression between the target variable and the set of available individual forecasts. Figure 1 depicts the general situation that faces a decision maker confronted with two or more forecasts for the same target variable.
Figure 1.
Decision tree when two or more forecasts are available for the same target variable. Notes to Figure 1: When two or more forecasts are available for the same target variable, the decision maker has two options: to choose one individual forecast out of the total available pool and make a decision according to it or to build a composite forecast using all the individual available forecasts as ingredients for this construction. When building a composite forecast, there are basically two major approaches in the literature: The first of them is to construct a weighted average of the forecasts. This includes a simple arithmetic average of the individual forecasts as a particular case. This strategy is known as the [2] approach. The second major approach is to construct a new forecast as a linear combination of the available forecasts but with unconstrained weights, which means that it could take negative values or be greater than one. This latter strategy is known as the [8] approach. To give an example, in the case of two available forecasts, and , the [2] approach proposes a third forecast as follows, whereas the [8] approach proposes a third forecast as follows, , where are just real numbers. Frequently, estimates of the parameters, , , and , are obtained like ordinary least squares estimators in a regression of the available target against a constant and the two available individual forecasts, and .
In general terms, the combination of forecasts is reported as a successful strategy to improve forecast accuracy. Ref. [1] shows an interesting table in which the simple average of several methods outperforms either of the individual forecasts available for US inflation. This is just one example of a pattern that the literature has been exploring so far. More empirical examples of the good behavior of combination schemes are found, for instance, in [7,17,23,24,25], just to name a few.
Different theoretical approaches aim at explaining the success of combination strategies. Given a set of forecasts and a loss function, the optimal combination could be found as the solution of an optimization problem looking for weights to minimize the expected loss. In many applications, such an optimization problem is well-defined and leads to non-trivial optimal weights, ensuring reductions in the loss function or, in other words, ensuring combination gains. Ref. [26] provides an interesting summary of the different environments in which combination gains are possible.
Despite these theoretical efforts, some questions are still unresolved. For instance, part of the relevant literature investigates what is known as the “Combination Puzzle“, which refers to the fact that simple combination schemes, like an arithmetic average, frequently outperform more complex and theoretically optimal combination rules. See, for instance, [27], for a discussion about this puzzle. More generally, as suggested by [15], the best way to build combination weights is still a matter of open debate.
While it is clear that combination gains may exist in a number of applications, part of the literature analyzes the efficiency of some combination strategies. For instance, ref. [10] indicates that the regression approach is a combination scheme that leaves room for improvement due to the autocorrelation in the residuals that is inherent to this combination method. Refs. [10,26] also mention that the Bates and Granger approach is potentially inefficient due to the introduction of the constraint of the coefficients summing to unity. The extent to which these inefficiencies are indeed relevant requires a case-by-case analysis, yet it is our reading of the literature that the potential inefficiencies of forecast combinations are mostly overshadowed by the enthusiastic literature that mainly emphasizes the direct benefits of forecast combinations in boosting forecast accuracy.
The main objective of this paper is to show that linear convex forecast combinations are almost always auto-inefficient, and, therefore, greater reductions in MSPE are almost always possible. By auto-inefficiency, we refer to the notion of efficiency analyzed by [28]. Aside from showing that auto-inefficiency almost always emerges in forecast combinations, we prove two additional results: First, we show that the process of taking averages of forecasts may induce inefficiencies in the combination, even when the individual forecasts are efficient. Second, we show that the so-called “optimal weighted average”, which is traditionally presented in the literature, may indeed be inefficient as well.
Currently, the state of the art in forecast combinations is to explore new different weighting schemes to combine forecasts, and no attempt to take advantage of potential inefficiencies stemming from the combination process is carried out. Our contribution to the existing literature is to show that auto-inefficiencies are commonplace in forecast combinations, which opens an interesting avenue for future research: to find the best way to remove these inefficiencies to further improve forecast accuracy.
We focus on linear convex combinations because they are used by practitioners in many empirical applications. In particular, Consensus Economics reports individual forecasts and their simple averages. Furthermore, many simple linear convex combinations are considered to be very accurate, which is consistent with the aforementioned combination puzzle. In addition, these combination strategies allow for an interpretation of the combination as a consensus forecast. Finally, many of these linear convex combinations do not require previous knowledge of the target variable to construct the combined forecast, which is a clear advantage of simple methods compared to the [8] approach.
The rest of the paper is organized as follows. In Section 2, we set the econometric environment. Section 3 contains the main theoretical results and simulated examples. Section 4 illustrates our findings with an empirical application. Section 5 concludes and presents possible extensions for further research. Appendix A provides some additional theoretical and empirical results.
2. Econometric Environment
Let us consider to be a stationary and ergodic time series process. We assume that at time t we want to forecast the random variable , which is equivalent to say that we look for an h-step ahead of the forecast for our target variable. We use the following notation: represents a generic forecast for the target variable , which is built with the information available at time . We assume that we have two forecasts available for the same target variable . We denote these forecasts as and .
We further assume that the vector process
is weakly stationary and ergodic and has a positive definite variance–covariance matrix .
2.1. The Combined Forecast
Consider the following combination of forecasts
where denotes the combined forecast. The corresponding forecast errors are
where and represent the errors associated with forecasts and , respectively:
We assume, without loss of generality, that the mean squared prediction errors () of forecast 2 are as good as those of forecast 1, which is to say that
When the combined forecast displays fewer than forecasts and do, we follow [26] and say that combination gains (CG) do exist.
Proposition 1 shows the conditions for this to happen.
2.2. Combination Gains
Proposition 1.
If
then combination gains are possible for λ ∈ (0, 1).
Proof.
Notice that
Therefore,
As shown later, a direct implication of a positive definite variance–covariance matrix is that
With this, we can see that (6) is a strictly convex quadratic form with, at most, two different real roots. One of them is zero, which rules out the possibility of two complex roots. The other real root is
Univariate convex quadratic forms with a zero root must fall into one of the three cases depicted in Scheme 1. Given that , we are in a situation like that depicted with the blue line. Therefore, combination gains are achieved in the interval of
which, in particular, ensures combination gains in, at least, a subset of the open set . □
Scheme 1.
Univariate convex quadratic forms with a zero root.
Well acquainted with this analysis, [12] proposes testing the following null hypothesis of no encompassing
Under this null hypothesis, a combination of forecasts is not improving the accuracy because
and no combination gains are possible.
We are also interested in analyzing a particular property of forecasts that we call auto-efficiency.
2.3. Auto-Efficiency
Definition 1.
Consider a target variable and a forecast . We say that the forecast is auto-efficient as long as
which, in the case of being unbiased, could be expressed as
This last expression indicates that forecast errors are orthogonal to the forecast itself. If this condition does not hold true, we say that the forecast is auto-inefficient. In general, definitions in the same line are originally attributed to the early work of [28] and are called Minzer–Zarnowitz efficiency.
Notice that auto-efficiency is one of the conditions satisfied by optimal forecasts under quadratic loss. Furthermore, the notion of auto-efficiency is relevant for at least two reasons:
- (1)
- When forecasts are unbiased, auto-inefficiency compromises the inverse relationship between the MSPE and the explained variance of the forecast.
- (2)
- Violations of auto-efficiency, in theory, allow for a simple modification of the forecast to produce a new revised forecast with fewer MSPE than .
Let us briefly explain the previous two points in more detail. The first point above relies on a remark made by [29]. Notice that
where is the forecast error.
Therefore,
When forecasts are unbiased, we have that
So
Therefore,
So, if auto-efficiency holds,
and there is an inverse relationship between and the “explained” variance of model . Nevertheless, if auto-efficiency does not hold, we can have reductions in that are associated with reductions in the “explained” variance of the model as well, which is counterintuitive.
Point (2) indicates that, when auto-inefficiency holds true, the forecast itself contains information that could be used to predict its own forecast errors. We could then build a linear model for the forecast errors as follows:
which define the following coefficients:
We could build a revised forecast as follows:
When forecasts are unbiased, we have and, consequently,
Therefore, provides information regarding the need for shrinkage or an upscale adjustment in the term .
The new forecast error would be
Interestingly,
This is so, as long as . In fact,
Notice that even if the original forecast is unbiased, we may have . This is so because
So, in the unbiased case,
And may still be different from zero, due to the auto-inefficiency of , as long as
In the next subsection, we describe the assumptions that we need to obtain our main results.
2.4. Assumptions
We are interested in an environment characterized by
- One target variable .
- Two forecasts and , such that
- Combination gains do exist in a region of the open set (0, 1). In other words,We also make use of the following assumption.
- The following vector
and is weakly stationary and ergodic, with a positive definite variance–covariance matrix, V. Notice that we have explicitly removed the horizon “h” from . This is just to emphasize that the relevant sub index is t, as we are considering h as a fixed horizon relative to t.
From now on, we use a simpler notation. Let . This is our target variable to forecast. We also drop the “t” subscript from , , and to further simplify the notation. We do the same with the corresponding forecast errors, so they look as follows:
Notice that a direct implication of a positive definite variance–covariance matrix V is that
Otherwise, we would have
and, due to the assumption of unbiased forecasts, we would have
This means that
Let us consider to be defined as the variance covariance matrix of sub-vector
Then, it has to be the case that is a positive definite matrix as well. Nevertheless
which is a contradiction to the fact that is positive and definite.
Remark 1.
Expression (25) is important. It says that and are not exactly the same probabilistic forecasts. They are distinct in a meaningful sense, so we combine forecasts embedding different information.
We have displayed the basic econometric framework and simplified notation that we are using in this paper. In the next section, we show the main results of the article.
3. Main Theoretical Results
One of the main points of this paper is to show that traditional weighted averages of forecasts are almost always auto-inefficient. In fact, the next proposition shows that the majority of forecast combinations with λ ∈ (0, 1) are auto-inefficient. Previous to that though, notice that with straightforward algebra the auto-efficiency of the combined forecast could be expressed in the following way:
It would be also useful to express (27) as follows:
The proofs of these expressions are in the Appendix A.
3.1. Auto-Inefficiency of Forecast Combinations
Proposition 2.
Let denote a target variable and two forecasts for , such that assumptions 1–4 hold true. Then, there are at most two different combinations , for which the combined forecast is auto-efficient.
Proof.
Let us consider the expected value of the combined forecast times its forecast error:
This expression defines the quadratic form in (27):
Let us recall that we already showed that the following expression (25) holds true:
This means that is a strictly concave quadratic form and, of course, is different from the zero function. As a consequence, has, at most, two real roots that may or may not lie within the (0,1) interval, so it may be the case that every single combination is auto-efficient. In any case, at most, two combinations are auto-efficient. □
Corollary 1.
Under Assumptions 1–4, if we further assume that both individual forecasts are auto-efficient, then any weighted average of the forecasts display a positive auto-inefficiency.
Proof.
We already saw that
Notice that
Using expression (27),
Given that the two individual forecasts are auto-efficient,
so we have
□
Proposition 2 shows that most of the possible forecast combinations are auto-inefficient. The previous corollary shows a particular case ensuring that all possible combinations within (0, 1) are auto-inefficient, even if both individual forecasts are auto-efficient. This is important because disregarding this result may lead to incorrect conclusions when analyzing tests based upon aggregated information from surveys. The finding that a combination of forecasts is inefficient does not imply that individual forecasts share this property.
The next proposition provides more general conditions under which every single combination in (0, 1) is auto-inefficient.
Proposition 3.
For , , and , as in Proposition 2, let us assume that
Then, for every single combination , the combined forecast displays auto-inefficiency.
Proof.
Using
we obtain
Therefore,
But, because we are assuming that (31) and (32) hold true, we have that the left-hand-side in the previous inequality is greater than or equal to zero:
Therefore,
Again using expression (27),
we conclude that
□
It is important to remark that conditions (31) and (32) may be tested using simple tests or a strategy based on the “reality check” of [30], which is also used in [31].
The next proposition explores if, by any chance, the optimal forecast combination satisfies some special condition of auto-efficiency.
Proposition 4.
For , , and , as in Proposition 3, we can find a unique such that
For this , we also have
Proof.
This is a strictly convex quadratic function that admits a unique global minimum defined by the following first order conditions:
This equation is solved by
Now, let us recall that by assumption 2
Therefore,
So, dividing by > 0, we have
Or
By the assumption of combination gains, we have
Therefore,
Or
This is an interesting result. When combinations gains do exist, the optimal combination is always achieved within the (0, 1) interval, and, moreover, the optimal combination assigns a higher weight to the most accurate individual forecast.
Now, let us find an expression for :
This last expression indicates that the optimal combination may or may not be auto-efficient. Notice that if
share the same sign, then there is no way for the optimal combination to display auto-efficiency, so it is auto-inefficient as well. □
3.2. Size of the Auto-Inefficiency in the Forecast Combinations
Let us consider the following notation:
Let us also recall that
Notice that
Therefore, we obtain
or
or
which could be written as
where AEG stands for auto-efficiency gains (or losses) relative to the auto-efficient status of forecast 2. The first term in the left-hand side is a strictly convex quadratic form with a zero root. The second term in the left-hand side is a strictly concave quadratic form with a zero root. If the S term is zero (S = 0), then the two quadratic forms would be the same but with opposite signs. This implies that movements along the quadratic forms are totally compensated for. When S 0, this is not the case, and movements along the quadratic forms are not totally compensated for: they are only partially compensated for. The size of this S term is indeed key for combination gains to be totally or just partially compensated by changes in auto-efficiency.
3.3. Simulated Examples
In this subsection, we illustrate with simple simulated examples the main results shown in the previous sections. In all these cases, we impose the existence of combination gains and explore if auto-efficiency is satisfied or not with one-step-ahead forecasts. Let us consider the three-dimensional vector
We consider the following three cases for the matrix :
All these matrices are symmetric, definite, and positive, as their eigenvalues are (2.33, 0.33, 0.54); (2.28, 0.25, 0.42); and (4.13, 0.76, 1.87), respectively. We use two different forecasts for :
The forecast errors are given by
Let us analyze the case in which we have . The respective MSPE and mean squared forecasts (MSF) are
So, clearly, forecast 2 is more accurate than forecast 1 in terms of MSPE and displays higher MSF. Nevertheless, forecast 1 is not encompassed by forecast 2, so combination gains are possible. This is so because
From Scheme 2, we show that most combinations in (0, 1) display reductions in MSPE compared to the best performing individual forecast. At the same time, most of these combinations are auto-inefficient. In particular, both the individual forecasts and the optimal combination, achieved by are inefficient.
Scheme 2.
Auto-inefficiency of most forecast xombinations.
Let us analyze the case in which we have . The respective MSPE and mean squared forecasts (MSF) are
So, again, forecast 2 is more accurate than forecast 1 in terms of MSPE and displays higher MSF. As in the previous case, forecast 1 is not encompassed by forecast 2, so combination gains are possible. This is so because
Different from the case (see Scheme 2), we now have that both individual forecasts are auto-efficient:
Yet, we clearly see from Scheme 3 that all the linear convex combinations of both individual forecasts are inefficient. In particular, the optimal combination in the (0, 1) interval is also inefficient.
Scheme 3.
Auto-inefficiency of forecast combinations when individual forecasts are auto-efficient.
For (3), we have a slightly different situation, in which both individual forecasts are inefficient:
Both forecasts display the same MSPE (2.25), combinations gains are possible, and the MSPE are minimized for
The most important feature of the (3) case is that all the linear convex combinations of both individual forecasts are inefficient, with the two exceptions of both the real roots of expression (27), which, in this case, are 0.5 and 0.533. Yet the optimal combination in terms of MSPE is exactly one of the previous roots: 0.5.
So, here, we have a case with almost all the combinations being inefficient, with only two exceptions, one of them being the optimal combination. Scheme 4 shows what is occurring in this case.
Scheme 4.
The optimal combination may display auto-efficiency.
Scheme 2 and Scheme 4 correspond to virtuous examples. In Scheme 2, we can see that for low values of λ we achieve both reductions in the MSPE and reductions in auto-inefficiency (in absolute value), with respect to the more accurate individual forecast. Scheme 3 is not virtuous, in the sense that reductions in the MSPE are associated with increments in the auto-inefficiency status of the combined forecast.
4. Empirical Illustration
In this section, we illustrate the relevance of our results with two different exercises aimed at forecasting headline inflation in eight different European economies: Austria, Belgium, the Czech Republic, Denmark, Estonia, Germany, Greece, and the United Kingdom. Our first approach is inspired by the previous work in [32], which compares the predictive ability of two different forecasting models: a model built with a local core inflation measure and a model built with an international inflation measure. As reported in the vast literature, the “food” and “energy” components in CPI tend to be highly volatile; hence, removing these components may improve the predictive ability of a model. See [33,34] for supporters of the “core predicts headline” argument and [35] for a comprehensive discussion. The idea that international inflation is closely related to domestic inflation comes from a seminal paper [36], which shows that there is an important pass-through from some measures of international inflation to local inflation. See [37,38] for a comprehensive analysis of this predictive ability in OECD economies.
Our second exercise is similar to the first one, but, now, we compare the predictive ability of a model built with an international inflation measure and a model built with a measure of output gap. In what follows, our target variable is the year-on-year CPI headline inflation for our sample of countries. We download our data from the OECD database (https://stats.oecd.org/, 25 July 2023) at the monthly frequency, from January 2000 to December 2022. For the construction of our measure of the output gap, we use the “Production and Sales” seasonally adjusted series available at the monthly frequency in the OECD database.
Let be the year-on-year headline inflation rate for a given country at time t, be the corresponding core inflation, and be the arithmetic average of headline inflation of all OECD economies (as a proxy of an international factor). Then, in our first exercise, the two competing forecasting models are
where and are simply error terms. In this exercise, we only focus on one-step-ahead forecasts. To conduct our out-of-sample analysis, we consider an initial estimation window of observations to estimate the parameters of Models 1 and 2. Given that the total number of observations for each country is T = 276, we are left with observations to evaluate our forecasts (T = R + P). To update the estimates of our parameters, we consider a rolling window of size . To keep the notation simple, let be the forecast constructed with Model 2 at time “t” for and be its counterpart constructed with Model 1.
Additionally, we consider the optimal combination between both forecasts. The optimal weight assigned to each forecast is determined by expression (35), and the combined forecast is constructed as follows:
Table 1 next reports that (i) the optimal weight assigned to , in order to minimize the MSPE; (ii) the MSPE of both forecasts ( and ) and the MSPE of the optimal combination; (iii) the inefficiency of the optimal combination forecast, reported as two times the covariance between the forecast and its respective forecasting error; and (iv) the ratio between the MSPE of the optimal forecast combination and the MSPE of Model 2 (notice that Model 2 produces forecasts with fewer MSPE than Model 1 in all eight cases, with no exceptions whatsoever. For this reason, the ratio is exclusively compared with Model 2. Of course, the combination gains are higher if compared with Model 1).
Table 1.
MSPE and inefficiency of the combined forecast when predicting headline inflation with a model with core and a model with international inflation.
Two features of Table 1 are worth mentioning. First, the optimal combination forecast is statistically inefficient in all eight cases, at the 10% significance level, with no exceptions whatsoever. Moreover, the null hypothesis of efficiency is rejected at the tighter 1% significance level in five out of eight cases. Second, the optimal combination always improves the accuracy of the forecasts, yet the size of the improvement is heterogeneous. For instance, the ratio is 0.793 for the Czech Republic, indicating impressive gains by considering the combined forecast. In contrast, the average MSPE ratio for Austria, Denmark, and Estonia is just 0.98, suggesting very limited gains for those economies.
Scheme 5 shows the MSPE and auto-inefficiency curves for Belgium and the United Kingdom. This illustrates, with real data, one of the main points of our paper: the optimal combination in terms of the MSPE turns out to be highly inefficient; in fact, in both figures, when approaching the optimal from the left, we observe that the combined MSPE decreases (left axis), while the inefficiency of the combined forecast simultaneously increases in absolute terms (right axis).
Scheme 5.
MSPE of the forecast combination (black solid line; left axis) and auto-inefficiency of the forecast combination (red dotted line; right axis). Notes: the black solid line represents the MSPE for the combination of forecasts when the weight associated with model 2, (1 − λ*), is determined by the x axis. The red dotted line represents the auto-inefficiency of the forecast combinations, measured as the covariance between the forecast error and the forecast of the combination.
For our second exercise, we combine forecasts coming from a model built with an international inflation measure (our previous model 2) with forecasts coming from the following third model
where is the output, and is the potential output (or output trend); hence, is just an estimate of the output gap. We construct with the Hodrick–Prescott filter with a smoothing parameter of 14,400, as traditionally suggested for monthly data. Notice that Model 3 is similar to Model 1 but uses the output gap as a predictor instead of the core inflation. Table 2 is akin to Table 1, but it compares forecasts coming from Model 3 with forecasts from our previous Model 2.
Table 2.
MSPE and inefficiency of the combined forecast when predicting headline inflation with a model with output gap and a model with international inflation.
Just like we do in our first exercise, now we consider the optimal combination between both forecasts, where the optimal weight is determined using expression (35):
Interestingly, Table 2 shows mixed results between both individual forecasts in terms of accuracy: in five out of eight cases, Model 3 exhibits fewer MSPE than Model 2. This result is in sharp contrast with Table 1, in which Model 2 displays the least MSPE in all cases, with no exceptions whatsoever. For this reason, instead of reporting , in Table 2 we report , where the are just the MSPE of the best individual model (either Model 3 or Model 2).
Notably, the main messages of Table 1 remain the same here: i) the forecast built as the optimal combination is statistically inefficient in most cases (seven out of eight), at least at the 5% significance level, and ii) combinations always improve the MSPE of the forecasts, although these combinations gains tend to be smaller in this second exercise, with a ratio ranging from 0.01% to 2.68%.
We acknowledge that there is some debate about how to estimate the potential output . For instance, [39] criticizes the use of the Hodrick–Prescott (HP) filter and proposes a regression-based strategy. In contrast, [40] compares the HP filter with Hamilton’s approach, providing support for the former relative to the latter. See [41,42] for more discussion on this topic. An interesting approach is mentioned in [43], which proposes a more aggressive smoothing parameter for the HP filter. Table 3 is akin to Table 2, but this time using the filter suggested by Ravn and Uhlig for monthly data (a smoothing parameter of 129,600). While the optimal combinations here are different compared to those in Table 2, the main message of our paper remains the same, as seven out of eight of these optimal forecast combinations are auto-inefficient as well.
Table 3.
MSPE and inefficiency of the combined forecast when predicting headline inflation with a model with a different measure of output gap and a model with international inflation.
Finally, in Appendix A.2, we show the results of our first forecasting exercise when applied to an extended set of an additional 19 OECD economies. These results corroborate those presented in this section, as, in the great majority of the cases, combination gains are also associated with auto-inefficiency.
5. Summary and Conclusions
It is well-known that the weighted averages of two competing forecasts may reduce the mean squared prediction errors (MSPE). It is less known, however, that forecast combinations may be inefficient. In this paper, we take an in-depth view of one particular type of problem stemming from simple combination schemes. This problem is called forecast auto-inefficiency and refers to the notion of inefficiency analyzed by [28].
Under mild assumptions, we show that linear convex forecast combinations are almost always auto-inefficient, and, therefore, room for accuracy improvement is almost always possible. We also identify testable conditions under which every linear convex combination of two forecasts is auto-inefficient. In particular, we show that the process of taking averages of forecasts may induce inefficiencies in the combination, even when individual forecasts are auto-efficient. The extent to which these inefficiencies are indeed relevant requires a case-by-case analysis. Nonetheless, it is striking that, in many applications in which a number of different forecasts are available, the combination of all of them seems to be the last step in the search of forecast accuracy, and no attempt to take advantage of the potential inefficiencies stemming from the combination process is pursued. Articles exploring the benefits of forecast combinations in terms of accuracy but totally disregarding their potential inefficiencies are widespread across time and topics. Some interesting and recent examples are [44], when forecasting stock market volatility; [45], when forecasting stock market returns; [46], when forecasting oil prices; and [47], when forecasting several macroeconomic variables in eight european economies.
We illustrate our findings with an empirical application in which two different forecasts for headline CPI inflation are combined. We mainly focus on the cases of eight European economies, but, in the Appendix A, we generalize our empirical results to a broader set by adding 19 OECD economies. We show that gains from combination are achieved in all of our exercises. More interestingly, we also show that, in 89% of all our different cases, the optimal combination displays a statistically significant Mincer–Zarnowitz inefficiency, which is consistent with our simple theoretical derivations.
Regarding limitations and avenues for future research, there are three things worth mentioning. First, it would be interesting to extend our derivations and main results for the general case of competing forecasts. Second, an obvious extension of our work is to explore the implications of relaxing some of our assumptions over the forecasts and the target variable (e.g., unbiased forecasts, weak stationarity, and ergodicity). Finally, we remain silent on how to exploit these inefficiencies in a real-time exercise. Given the results reported in our paper, an exciting avenue for future research is to study how to improve the accuracy of forecast combinations by taking advantage of the already reported inefficiencies in real-time exercises, when only a low number of forecast errors might be available.
Author Contributions
Conceptualization, P.P.-B. and A.B.; Methodology, P.P.-B. and N.H.; Software, N.H.; Formal analysis, P.P.-B. and N.H.; Data curation, N.H.; Writing—original draft, P.P.-B. and A.B.; Writing—review & editing, P.P.-B. and A.B.; Visualization, N.H.; Supervision, P.P.-B.; Project administration, P.P.-B. and A.B. All authors have read and agreed to the published version of the manuscript.
Funding
This research was partially funded by FONDECYT grant number 11080109, and the APC was partially funded by Universidad Adolfo Ibáñez.
Data Availability Statement
Data can be found here https://stats.oecd.org.
Conflicts of Interest
The authors declare no conflict of interest.
Appendix A
Appendix A.1. Proof of Expressions (27) and (28) in Section 3
Proof.
Let us derive expression (27):
Notice that
Therefore,
So, finally, we have
which is the desired result.
Let us derive the second expression. From (27), we obtain
Therefore,
which is expression (28), the desired result. □
Appendix A.2. Empirical Illustration with an Extended Dataset
Table A1.
MSPE and inefficiency of the combined forecast when predicting headline inflation with a model with core and a model with international inflation. Extended results for 27 OECD economies.
Table A1.
MSPE and inefficiency of the combined forecast when predicting headline inflation with a model with core and a model with international inflation. Extended results for 27 OECD economies.
| Austria | Belgium | Canada | Colombia | Czech Republic | Denmark | Estonia | |
|---|---|---|---|---|---|---|---|
| λ* | 0.255 | 0.281 | 0.490 | 0.059 | 0.443 | 0.164 | 0.330 |
| MSPE (International) | 0.094 | 0.229 | 0.213 | 0.102 | 0.364 | 0.106 | 0.872 |
| MSPE (Core) | 0.111 | 0.276 | 0.214 | 0.271 | 0.408 | 0.139 | 0.934 |
| Ratio MSPEopt/MSPEint | 0.976 | 0.963 | 0.928 | 0.994 | 0.793 | 0.987 | 0.977 |
| MSPE Optimal Combination | 0.092 | 0.220 | 0.198 | 0.101 | 0.288 | 0.104 | 0.852 |
| Inefficiencies (Opt. Combination) | −0.145 ** | −0.633 *** | −0.134 | −0.384 *** | −0.476 * | −0.364 *** | −4.300 *** |
| Finland | France | Germany | Greece | Ireland | Israel | Korea | |
| λ* | 0.141 | 0.134 | 0.321 | 0.307 | 0.195 | 0.170 | 0.405 |
| MSPE (International) | 0.128 | 0.084 | 0.173 | 0.384 | 0.248 | 0.209 | 0.161 |
| MSPE (Core) | 0.172 | 0.204 | 0.200 | 0.533 | 0.519 | 0.725 | 0.172 |
| Ratio MSPEopt/MSPEint | 0.991 | 0.965 | 0.954 | 0.906 | 0.932 | 0.892 | 0.943 |
| MSPE Optimal Combination | 0.127 | 0.081 | 0.165 | 0.348 | 0.231 | 0.187 | 0.152 |
| Inefficiencies (Opt. Combination) | −0.293 *** | −0.157 *** | −0.164 ** | −1.286 *** | −0.931 *** | −0.246 ** | −0.102 |
| Latvia | Luxembourg | Mexico | Netherlands | Norway | Poland | Slovak Republic | |
| λ* | 0.062 | 0.368 | 0.075 | 0.086 | 0.236 | 0.221 | 0.193 |
| MSPE (International) | 0.517 | 0.257 | 0.138 | 0.312 | 0.275 | 0.233 | 0.205 |
| MSPE (Core) | 1.908 | 0.373 | 0.611 | 0.430 | 0.320 | 0.266 | 0.625 |
| Ratio MSPEopt/MSPEint | 0.988 | 0.768 | 0.977 | 0.997 | 0.983 | 0.988 | 0.876 |
| MSPE Optimal Combination | 0.511 | 0.197 | 0.135 | 0.311 | 0.270 | 0.230 | 0.180 |
| Inefficiencies (Opt. Combination) | −1.923 ** | −0.237 ** | −0.063 | −0.708 ** | −0.247 ** | −1.021 *** | -0.400 ** |
| Slovenia | Spain | Sweden | Switzerland | United Kingdom | United States | ||
| λ* | 0.210 | 0.210 | 0.085 | 0.069 | 0.310 | 0.318 | |
| MSPE (International) | 0.371 | 0.308 | 0.170 | 0.089 | 0.085 | 0.246 | |
| MSPE (Core) | 0.475 | 0.431 | 0.298 | 0.266 | 0.098 | 0.278 | |
| Ratio MSPEopt/MSPEint | 0.979 | 0.970 | 0.993 | 0.989 | 0.960 | 0.963 | |
| MSPE Optimal Combination | 0.363 | 0.299 | 0.169 | 0.088 | 0.081 | 0.237 | |
| Inefficiencies (Opt. Combination) | −0.703 *** | −0.601 *** | 0.004 | −0.150 *** | −0.338 *** | −0.292 ** | |
* p < 10%, ** p < 5%, *** p < 1%.
References
- Elliot, G.; Timmermann, A. Economic Forecasting. J. Econ. Lit. 2008, 46, 3–56. [Google Scholar] [CrossRef]
- Bates, J.; Granger, C. The combination of forecasts. Oper. Res. Q. 1969, 20, 451–468. [Google Scholar] [CrossRef]
- Granger, C.; Newbold, P. Some Comments on the Evaluation of Economic Forecasts. Appl. Econ. 1973, 5, 35–47. [Google Scholar] [CrossRef]
- Granger, C.; Newbold, P. Forecasting Economic Time Series, 2nd ed.; Academic Press: Orlando, FL, USA, 1986. [Google Scholar]
- Chong, Y.; Hendry, D. Econometric Evaluation of Linear Macroeconomic Models. Rev. Econ. Stud. 1986, 53, 671–690. [Google Scholar] [CrossRef]
- Clements, M.; Hendry, D. On the Limitations of Comparing Mean Square Forecast Errors. J. Forecast. 1993, 12, 617–637. [Google Scholar] [CrossRef]
- Newbold, P.; Granger, C. Experience With Forecasting Univariate Time Series and the Combination of Forecasts. J. R. Stat. Soc. Ser. A 1974, 137, 131–165. [Google Scholar] [CrossRef]
- Granger, C.; Ramanathan, R. Improved methods of combining forecasts. J. Forecast. 1984, 3, 197–204. [Google Scholar] [CrossRef]
- Clemen, R. Linear constraints and the efficiency of combined forecasts. J. Forecast. 1986, 5, 31–38. [Google Scholar] [CrossRef]
- Diebold, F. Serial correlation and the combination of forecasts. J. Bus. Econ. Stat. 1988, 6, 105–111. [Google Scholar]
- Batchelor, R.; Dua, P. Forecaster diversity and the benefits of combining forecasts. Manag. Sci. 1995, 41, 68–75. [Google Scholar] [CrossRef]
- Harvey, D.; Leybourne, S.; Newbold, P. Tests for Forecast Encompassing. J. Bus. Econ. Stat. 1998, 16, 54–259. [Google Scholar]
- Stock, J.; Watson, M. Combination forecasts of output growth in a seven-country data set. J. Forecast. 2004, 23, 405–430. [Google Scholar] [CrossRef]
- Aiolfi, M.; Timmermann, A. Persistence in Forecasting Performance and Conditional Combination Strategies. J. Econom. 2006, 135, 31–53. [Google Scholar] [CrossRef]
- Hansen, B. Least Squares Forecast Averaging. J. Econom. 2008, 146, 342–350. [Google Scholar] [CrossRef]
- Capistrán, C.; Timmermann, A. Forecast Combination with Entry and Exit of Experts. J. Bus. Econ. Stat. 2009, 27, 429–440. [Google Scholar] [CrossRef]
- Clements, M.; Harvey, D. Combining Probability Forecasts. Int. J. Forecast. 2011, 27, 208–223. [Google Scholar] [CrossRef]
- Poncela, P.; Rodriguez, J.; Sánchex-Mangas, R.; Senra, E. Forecast combinations through dimension reduction techniques. Int. J. Forecast. 2011, 27, 224–237. [Google Scholar] [CrossRef]
- Kolassa, S. Combining exponential smoothing forecasts using Akaike weights. Int. J. Forecast. 2011, 27, 238–251. [Google Scholar] [CrossRef]
- Costantini, M.; Kunst, R. Combining forecasts based on multiple encompassing tests in a macroeconomic core system. J. Forecast. 2011, 30, 579–596. [Google Scholar] [CrossRef]
- Cheng, X.; Hansen, B. Forecasting with factor-augmented regression: A frequentist model averaging approach. J. Econom. 2015, 186, 280–293. [Google Scholar] [CrossRef]
- Wang, X.; Hyndman, R.; Li, F.; Kang, Y. Forecast combinations: An over 50-year review. Int. J. Forecast. 2022, in press. [Google Scholar] [CrossRef]
- Wright, J. Bayesian Model Averaging and exchange rate forecasts. J. Econom. 2008, 146, 329–341. [Google Scholar] [CrossRef]
- Chowell, G.; Luo, R.; Sun, K.; Roosa, K.; Tariq, A.; Viboud, C. Real-time forecasting of epidemic trajectories using computational dynamic ensembles. Epidemics 2020, 30, 100379. [Google Scholar] [CrossRef]
- Atiya, A. Why does forecast combination work so well? Int. J. Forecast. 2020, 36, 197–200. [Google Scholar] [CrossRef]
- Timmermannn, A. Forecast Combinations. In Handbook of Economic Forecasting; Elliott, G., Granger, C., Timmermannn, A., Eds.; Elsevier: Amsterdam, The Netherlands, 2006; pp. 135–194. [Google Scholar]
- Aiolfi, M.; Capistrán, C.; Timmermann, A. Forecast Combinations. In The Oxford Handbook of Economic Forecasting; Clements, M., Hendry, D., Eds.; OUP: New York, NY, USA, 2011. [Google Scholar]
- Mincer, J.; Zarnowitz, V. The Evaluation of Economic Forecasts. In Economic Forecasts and Expectations; Mincer, J., Ed.; National Bureau of Economic Research: New York, NY, USA, 1969. [Google Scholar]
- Patton, A.; Timmermann, A. Forecast Rationality Tests Based on Multi-Horizon Bounds. J. Bus. Econ. Stat. 2012, 30, 1–17. [Google Scholar] [CrossRef]
- White, H. A Reality Check for Data Snooping. Econometrica 2000, 68, 1097–1126. [Google Scholar] [CrossRef]
- Pincheira, P. Shrinkage Based Tests of Predictability. J. Forecast. 2013, 32, 307–332. [Google Scholar]
- Pincheira, P.; Hardy, N. Correlation Based Tests of Predictability; MPRA Paper 112014; University Library of Munich: München, Germany, 2022. [Google Scholar]
- Bermingham, C. How useful is core inflation for forecasting Headline inflation? Econ. Soc. Rev. 2007, 38, 355–377. [Google Scholar]
- Song, L. Do underlying measures of inflation outperform headline rates? Evidence from Australian data. Appl. Econ. 2005, 37, 339–345. [Google Scholar] [CrossRef][Green Version]
- Pincheira, P.; Selaive, J.; Nolazco, J.L. Forecasting inflation in Latin America with core measures. Int. J. Forecast. 2019, 35, 1060–1071. [Google Scholar] [CrossRef]
- Ciccarelli, M.; Mojon, B. Global Inflation. Rev. Econ. Stat. 2010, 92, 524–535. [Google Scholar] [CrossRef]
- Pincheira, P. A power booster factor for out-of-sample tests of predictability. Economía 2022, 45, 150–183. [Google Scholar] [CrossRef]
- Medel, C.; Pedersen, M.; Pincheira, P. The Elusive Predictive Ability of Global Inflation. Int. Financ. 2016, 19, 120–146. [Google Scholar] [CrossRef]
- Hamilton, J.D. Why you should never use the Hodrick-Prescott filter. Rev. Econ. Stat. 2018, 100, 831–843. [Google Scholar] [CrossRef]
- Dritsaki, M.; Dritsaki, C. Comparison of HP Filter and the Hamilton’s Regression. Mathematics 2022, 10, 1237. [Google Scholar] [CrossRef]
- Jönsson, K. Cyclical dynamics and trend/cycle definitions: Comparing the HP and Hamilton filters. J. Bus. Cycle Res. 2020, 16, 151–162. [Google Scholar] [CrossRef]
- Jönsson, K. Real-time US GDP gap properties using Hamilton’s regression-based filter. Empir. Econ. 2020, 59, 307–314. [Google Scholar] [CrossRef]
- Ravn, M.O.; Uhlig, H. On adjusting the Hodrick-Prescott filter for the frequency of observations. Rev. Econ. Stat. 2002, 84, 371–376. [Google Scholar] [CrossRef]
- Niu, Z.; Wang, C.; Zhang, H. Forecasting stock market volatility with various geopolitical risks categories: New evidence from machine learning models. Int. Rev. Financ. Anal. 2023, 89, 102738. [Google Scholar] [CrossRef]
- Lv, W.; Qi, J. Stock market return predictability: A combination forecast perspective. Int. Rev. Financ. Anal. 2022, 84, 102376. [Google Scholar] [CrossRef]
- Safari, A.; Davallou, M. Oil price forecasting using a hybrid model. Energy 2018, 148, 49–58. [Google Scholar] [CrossRef]
- Capek, J.; Cuaresma, J.C.; Hauzemberger, N. Macroeconomic forecasting in the euro area using predictive combinations of DSGE models. Int. J. Forecast. 2022, in press. [Google Scholar] [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).