Explainability for Forecasting Task with Close-Up Option

Abstract

Although machine learning-based forecasting models often function as opaque black boxes, explainability methods provide the essential tools required to foster trust in deep learning systems, optimize computational resources, and enhance model accuracy. Motivated by this, we propose a model-agnostic, local explainability method specifically designed for forecasting tasks. Our approach employs a hierarchical structure that is generated iteratively. From the outset, a small number of “interpretable components” that decompose the input are considered. In the inductive step, the methodology enables analysts to examine the data in greater detail, allowing for a deeper exploration of specific subcomponents whenever a higher resolution is required. This framework inherently provides control over computational costs by tuning the branching factor. The resulting explainability tree shares certain, but not all, properties of the explainability values provided by the SHAP-LIME framework; in this regard, we present some theoretical results. Furthermore, numerical experiments provided evidence that explanation coherence is fundamentally maintained between both methods. We explored the utility of this methodology on a real-world short-term load forecasting LSTM encoder–decoder model.

1. Introduction

Machine learning-based forecasting models often act as black-box systems. Their inputs typically consist of tensors comprising historical features, known future covariates, and other predefined information; conversely, their outputs project multi-step-ahead sequences across specified forecasting horizons. Within this setting, we present a model-agnostic, post hoc, and inductive method inspired by the foundational principles of KernelSHAP [1,2,3]. Our approach allows analysts to consider interpretable sub-components down to the required level of precision, focusing on detailed close-ups only where the analysis deems it necessary.

A simple way to apply KernelSHAP to multivariate time-series models is to treat each temporal feature cell as an independent unit. However, this leads to a combinatorial explosion; although sampling methods avoid evaluating all coalitions, they still sacrifice precision or interpretability. Several methods have developed strategies to address the computational cost issue within the time-series framework. For classification tasks, TimeSHAP is an explainability method that extends KernelSHAP to the sequential domain [4]. A key component of this approach is a pruning method that significantly reduces the computational overhead by discarding older historical data. TimeSHAP assumes that temporally distant observations are less influential. However, this is not always the case; see a counterexample in [5] [Figure 1]. On the other hand, WindowSHAP introduces several variants, ranging from uniform time segmentation to an iterative refinement method that increases the resolution around significant attribution values [5]. Both methods rely on thresholds based on “small” Shapley values to guide subsequent operations, potentially hindering the adaptation of the algorithm from classification to forecasting, particularly when dealing with extended forecast horizons.

Finally, the SeqSHAP method incorporates subsequence-level explanations [6]—specifically through temporal partitions for each feature. This approach essentially constitutes a special case of the “close-up” concept introduced in this work. The authors did not consider the error between the sum of sequential contributions and the total contribution computed at the initial stage; thereby, a hierarchical structure is not necessarily guaranteed. Indeed, our approach resolves this inconsistency by correcting the error at each step.

In this work, we propose a methodology based on interpretable components, which can be iteratively refined through the forecaster’s knowledge using what we call “close-ups”. Each close-up assumes that some players are temporarily unavailable, which allows for a more detailed analysis. The proposed explainability model produces a tree whose nodes provide explanations at different levels of granularity. The hierarchical structure provides a clearer explainability narrative. Furthermore, we present a mathematically rigorous and empirical analysis of the relationship between our methodology and the traditional KernelSHAP approach. The advantages of our method include:

• Control of computational costs based on a maximum branching factor.
• Flexibility of application to specific contexts.
• Simple implementation, particularly in production forecasting cycles.
• Adaptability in the level of analysis.
• Explainability consistency across steps.

We present Table 1 to provide a concise comparative overview of the key characteristics of the aforementioned methods, for which we share several insights. The goal of these methods is to identify the relevant sections that determined the classification, thereby facilitating an expert analysis. Although it is possible to adapt these methods for forecasting, our approach, in contrast, integrates the expert into the system to explore and iteratively refine input variable concepts through close-ups, bypassing the simultaneous management of numerous input components lacking an interpretive narrative.

Table 1. Comparative summary of similar model-agnostic and post hoc explainability approaches for time series.

Methodology	Computational Management Strategy	Refinement Strategy	Explanation Granularity	Task
TimeSHAP	Past pruning	Feature or temporal partitioning	Features, time-events, and cells	Classification
Stationary WindowSHAP	Coarse grouping	Uniform partitioning	Time windows	Classification
Sliding WindowSHAP	Local grouping and sliding	Overlapping fixed-size windows	Time windows	Classification
Dynamic WindowSHAP	Segmentation and window contribution control	Threshold-based binary window partitioning	Time windows	Classification
SeqSHAP	Two-level grouping	Feature-level time window	Time-windows partitioned feature level	Classification
Our method	Adaptive grouping	Forecaster-driven hierarchical partitioning	Context-dependent	Forecasting

The paper is structured as follows. Section 2 presents the general methodology and establishes several key properties of the proposed explainability vectors. Section 3 details the computation of these vectors and discusses the limitations of sampling for their estimation, alongside insights into the computational cost and the relationship between the hierarchical fitting error and global explainability properties equivalent to those of Shapley values. Section 4 evaluates the practical utility of this framework through a short-term energy demand forecasting case study using real-world data and a production model. Finally, concluding remarks are provided in Section 5, and all formal proofs are collected in Appendix A.

2. Explainability Trees

For a forecasting model f, we consider any trained model that takes as the input a finite sequence of time-feature matrices, e.g., $X=(X_A,X_B,X_C)$, accepting different matrix sizes—for instance, matrices corresponding to historical variables, known future covariates, or other inputs consistent with the architecture—and outputs a time vector Y with the corresponding variable for $\tau$ future instances to be forecasted. By an interpretable decomposition of the input, we mean a partitioning of the input associated with semantic terms linked to an explanatory narrative. That is, an arbitrary split of X:

$$X=X_1\otimes X_2\otimes \cdots \otimes X_n$$

such that there are no overlaps between the parts and the concatenation of all of them, with respect to their original structure, is X itself. We refer to each $X_i$ as an interpretable component when it has a clear interpretation for forecasters, e.g., “weekends”, “the temperature in the past”, “the endogenous variable lagged by 3 days”, “holiday information given in advance”, or even “everything else”. If the structure of X is intricate, we can use “grouping” methods; for instance, we can decompose X into multivariate time-series groups where the dependence between variables within the same group is high, but very low between variables in different groups [7]. Alternatively, we can use time-series segmentation techniques to divide the time series into different semantic segments (see, e.g., [8]). Consequently, after this assisted decomposition, some degree of interpretability can be established for the resulting mutually disjoint groups.

The core idea of the proposed method is to apply an initial rule based on an interpretable decomposition of components. Upon evaluating the results, we proceed with an inductive rule that incorporates interpretable sub-components, allowing for a closer look at a specific component. This iterative process can be continued until there are enough elements to explain the model. For example, the interpretable component “weekends” can be split into subcomponents “Saturdays” and “Sundays”. The final result will be an initial bias ${\phi }_0$; the explainability vectors ${\phi }_1,{\phi }_2,\dots ,{\phi }_{n_{\mathrm{total}}}$ in ${\mathbb{R}}^{\tau }$; and the tree-based links associated with the interpretable components and subcomponents. We refer to these elements as the explainability tree.

Let X be a fixed input and $f\left(X\right)$ be the forecast over $\tau$ future time steps. We propose the following methodology:

Initial rule.

Start with a fixed initial interpretable decomposition $X=X_1\otimes X_2\otimes \cdots \otimes X_{n_{\mathrm{ini}}}$. Compute ${\psi }_0,{\psi }_1,\dots ,{\psi }_{n_{\mathrm{ini}}}$ according to the methodology described in Section 3. For every $i=0,1,2,\dots ,n_{\mathrm{ini}}$, set ${\phi }_i={\psi }_i$. We shall call ${\phi }_0$ the initial bias.

Inductive rule.

For some $X_{\mathrm{comp}}$ in the interpretable decomposition considered in the previous iteration, let $X_{\mathrm{comp}}=X_{\mathrm{comp},1}\otimes X_{\mathrm{comp},2}\otimes \cdots \otimes X_{\mathrm{comp},n}$ be a subdecomposition of $X_{\mathrm{comp}}$ such that $X=X_{\mathrm{comp}}\otimes X_{\mathrm{rest}}$, where $X_{\mathrm{rest}}$ is the remainder of X. Compute ${\psi }_{\mathrm{comp},0}$ and the columns of the matrix $\Psi$, ${\psi }_{\mathrm{comp},1},\dots ,{\psi }_{\mathrm{comp},n}$, according to the methodology given in Section 3. For each $k=1,2,\dots ,n$, we set ${\phi }_{\mathrm{comp},k}={\psi }_{\mathrm{comp},k}+{\epsilon }_{\mathrm{comp}}\in {\mathbb{R}}^{\tau }$, where ${\epsilon }_{\mathrm{comp}}={\displaystyle \frac{{\phi }_{\mathrm{comp}}-\Psi 1_n}{n}}\in {\mathbb{R}}^{\tau }$ is a vector independent of k.

In each close-up, for each $k=1,2,\dots ,n$, we suppose that the estimate of ${\psi }_{\mathrm{comp},k}$ has an uncertainty error representing a fraction of the total error: ${\epsilon }_{\mathrm{comp},k}={\alpha }_k({\phi }_{\mathrm{comp}}-\Psi 1_n)$. In order to preserve the hierarchical structure, we set ${\alpha }_k\ge 0$ and ${\sum }_{k=1}^n{\alpha }_k=1$. Consequently, the coefficients ${\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n$ form a probability distribution. The principle of maximum entropy states that, when choosing a probability distribution for a system with incomplete information, the least biased choice is the one that maximizes entropy. Under conditions of complete ignorance or in the absence of prior information, the principle of maximum entropy dictates that all components are assumed to contribute equally to the total error [9] [Example 12.2.4]. Therefore, we set ${\alpha }_k={\displaystyle \frac{1}{n}}$. The uniform distribution serves as a baseline distribution that can be adjusted when additional information about the system becomes available—for instance, if the errors are structurally related to the magnitude of the estimated individual contributions, a relation that cannot be generally assumed.

We can now formally associate a tree configuration $\mathrm{\Delta }$ with this inductive process. The initial decomposition $X=X_1\otimes X_2\otimes \cdots \otimes X_{n_{\mathrm{ini}}}$ generates a tree rooted at X with children $X_k$ for $k=1,2,\dots ,n_{\mathrm{ini}}$. In turn, each subsequent subdecomposition $X_{\mathrm{comp}}=X_{\mathrm{comp},1}\otimes X_{\mathrm{comp},2}\otimes \cdots \otimes X_{\mathrm{comp},n}$ considered in the process generates new children $X_{\mathrm{comp},1},X_{\mathrm{comp},2},\dots ,X_{\mathrm{comp},n}$ with $X_{\mathrm{comp}}$ as their parent. If ${\phi }_{\mathrm{comp},k}$ is computed for some $X_{\mathrm{comp},k}$, then ${\phi }_{\mathrm{comp},k}$ becomes a child of $X_{\mathrm{comp},k}$ for $k=1,2,\dots ,n$. Ultimately, the leaves of the tree, which can be relabeled as ${\phi }_1,{\phi }_2,\dots ,{\phi }_{n_{\mathrm{total}}}$, are the $n_{\mathrm{total}}$ final explainability vectors with respect to tree configuration $\mathrm{\Delta }$. See Algorithm 1. An example of an explainability tree is shown in Figure 1.

Algorithm 1 Explainability tree.

Require: Trained model f Require: A uniform random sample of ${\mathcal{X}}_{\mathrm{train}}$ Require: An input observation X for f $t_{\mathrm{leaf}}\leftarrow (X,)$ $T\leftarrow \left[t_{\mathrm{leaf}}\right]$ while a close-up is required for some $t_{\mathrm{leaf}}$ in T do      ▹ human in the loop     $X_{\mathrm{comp}}\leftarrow t_{\mathrm{leaf}}\left[0\right]$     Set a decomposition $X_{\mathrm{comp}}=X_{\mathrm{comp},1}\otimes X_{\mathrm{comp},2}\otimes \cdots \otimes X_{\mathrm{comp},n}$     Compute ${\psi }_{\mathrm{comp},0},{\psi }_{\mathrm{comp},1},\dots ,{\psi }_{\mathrm{comp},n}$ (Section 3)     if $X_{\mathrm{comp}}=X$ then         ${\phi }_0\leftarrow {\psi }_{\mathrm{comp},0}$         ${\epsilon }_{\mathrm{comp}}\leftarrow 0$     else         ${\phi }_{\mathrm{comp}}\leftarrow t_{\mathrm{leaf}}\left[1\right]$         ${\epsilon }_{\mathrm{comp}}\leftarrow {\displaystyle \frac{{\phi }_{\mathrm{comp}}-({\psi }_{\mathrm{comp},1}+\dots +{\psi }_{\mathrm{comp},n})}{n}}$     end if     $T\leftarrow T.\mathrm{remove}\left(t_{\mathrm{leaf}}\right)$     for $k=1,2,\dots ,n$ do         ${\phi }_{\mathrm{comp},k}\leftarrow {\psi }_{\mathrm{comp},k}+{\epsilon }_{\mathrm{comp}}$         $t_{\mathrm{link}}\leftarrow (X_{\mathrm{comp}},X_{\mathrm{comp},k})$         $t_{\mathrm{leaf}}\leftarrow (X_{\mathrm{comp},k},{\phi }_{\mathrm{comp},k})$         $T\leftarrow T.\mathrm{add}\left(t_{\mathrm{link}}\right)$         $T\leftarrow T.\mathrm{add}\left(t_{\mathrm{leaf}}\right)$     end for end while

Figure 1. Explainability tree for 36 h ahead STLF to explain the inference $f\left(X\right)$ of the load demand on a summer day, given the input X to an LSTM encoder–decoder model f. For each hour t, the positive values (green bars) of the explainability vectors increase the accumulated value starting from ${\phi }_0^t$, while the negative values (blue bars) decrease it; the final total of the sum is the forecast $f_t\left(X\right)$. Each explainability vector can be decomposed into new vectors (gray arrows) that provide more precise explainability information.

The explainability tree satisfies the following consistency properties:

1.

Breakdown contributions: ${\phi }_{\mathrm{comp},1}+{\phi }_{\mathrm{comp},2}+\dots +{\phi }_{\mathrm{comp},n}={\phi }_{\mathrm{comp}}$.

2.

Local symmetry: If the local marginal contribution of two interpretable components $X_{\mathrm{comp},i}$ and $X_{\mathrm{comp},j}$ sharing the same parent node $X_{\mathrm{comp}}$ is identical for every coalition $z\in {\{0,1\}}^n$, then ${\phi }_{\mathrm{comp},i}={\phi }_{\mathrm{comp},j}$.

3.

Global efficiency: ${\sum }_{i=1}^{n_{\mathrm{total}}}{\phi }_i=f\left(X\right)-\mathbb{E}\left(f\right)$.

4.

Global nullity with error: If the global marginal contribution of an interpretable component $X_{\mathrm{comp},k}$ is null, then ${\phi }_{\mathrm{comp},k}={\epsilon }_{\mathrm{comp}}$.

3. Estimation of Explainability Vectors

Let X be a fixed observation, and let $X_{\mathrm{comp}}=X_{\mathrm{comp},1}\otimes X_{\mathrm{comp},2}\otimes \cdots \otimes X_{\mathrm{comp},n}$ be an initial interpretable decomposition or any further interpretable subdecomposition such that:

$$X=X_{\mathrm{comp}}\otimes X_{\mathrm{rest}}$$

where $X_{\mathrm{rest}}$ is the remainder of X disjoint from $X_{\mathrm{comp}}$. For the initial decomposition, there is no $X_{\mathrm{rest}}$ and $X_{\mathrm{comp}}$ is X. We want to adjust a linear explainability model $g_{\mathrm{comp}}:{\{0,1\}}^n\to \mathbb{R}$ acting on the absence/presence of the interpretable subcomponents with the remainder $X_{\mathrm{rest}}$ fixed, namely an affine map:

$$g_{\mathrm{comp}}\left(z\right)={\psi }_{\mathrm{comp},0}+\Psi z,$$

where $\Psi$ is a $\tau \times n$ matrix whose columns correspond to a vector ${\psi }_{\mathrm{comp},i}\in {\mathbb{R}}^{\tau }$ associated with the interpretable subcomponent $i=1,2,\dots ,n$:

$$\Psi =\left(\begin{array}{cccc}|& |& & |\\ {\psi }_{\mathrm{comp},1}& {\psi }_{\mathrm{comp},2}& \dots & {\psi }_{\mathrm{comp},n}\\ |& |& & |\end{array}\right).$$

To fit $g_{\mathrm{comp}}$, we generated a weighted pairwise dataset

$$\{(z,f{|}_{X_{\mathrm{comp}}}\left(z\right);{\pi }_n\left(z\right)):z\in {\{0,1\}}^n\}$$

with the weights given by ${\pi }_n\left(z\right)={\displaystyle \frac{n-1}{\left(\genfrac{}{}{0pt}{}{n}{\left|z\right|}\right)\left|z\right|(n-\left|z\right|)}}$, where $\left|z\right|$ is the number of nonzero entries and:

$${f|}_{X_{\mathrm{comp}}}\left(z\right)=\mathbb{E}\left(f\right|\mathrm{component} \mathrm{in} X \mathrm{indicated} \mathrm{by} z \mathrm{with} X_{\mathrm{rest}} \mathrm{fixed}).$$

To estimate ${f|}_{X_{\mathrm{comp}}}\left(z\right)$, we employed Algorithm 2 using a sample from a training set ${\mathcal{X}}_{\mathrm{train}}$ such that all elements within the sample share the same structure as X—specifically, possessing the same lookback window. Furthermore, the interpretable components must remain consistent across all elements of the sample in accordance with X. Following the framework of the KernelSHAP methodology, we considered the optimization problem ${\min }_g{\sum }_{z\in Z}{\pi }_n{\left(z\right)\parallel f|}_{X_{\mathrm{comp}}}{\left(z\right)-g\left(z\right)\parallel }_2^2$. Let $0_n$ and $1_n$ be the n-dimensional zero and ones vectors, respectively. Note that ${\pi }_n\left(0_n\right)={\pi }_n\left(1_n\right)=\infty$. Consequently, ${\psi }_{\mathrm{comp},0}{=f|}_{X_{\mathrm{comp}}}\left(0_n\right)$ and ${f\left(X\right)=f|}_{X_{\mathrm{comp}}}\left(1_n\right)={\psi }_{\mathrm{comp},0}+\Psi 1_n$. The optimization problem can then be solved by embedding these constraints directly into the configuration of $g_{\mathrm{comp}}$, minimizing over $Z:={\{0,1\}}^n\setminus \{0_n,1_n\}$ using standard weighted least squares solvers.

Algorithm 2 Estimation of ${f|}_{X_{\mathrm{comp}}}\left(z\right)$.

Require: Trained model f Require: A uniform random sample of ${\mathcal{X}}_{\mathrm{train}}$   ▹ as a sample of the “domain of f” Require: An interpretable decomposition of X in the domain of f: $$X=X_{\mathrm{comp},1}\otimes X_{\mathrm{comp},2}\otimes \cdots \otimes X_{\mathrm{comp},n}\otimes X_{\mathrm{rest}}.$$ Require:  $z\in {\{0,1\}}^n$ ${\mathcal{X}}_{\mathrm{mask}}\leftarrow \left[\right]$ for  U  in  ${\mathcal{X}}_{\mathrm{train}}$  do     for i in $\{1,2,\dots n\}$ do         if $z\left[i\right]=1$ then            $U_i^{\mathrm{mask}}\leftarrow X_{\mathrm{comp},i}$         else            $U_i^{\mathrm{mask}}\leftarrow U_{\mathrm{comp},i}$         end if     end for     $U^{\mathrm{mask}}\leftarrow U_1^{\mathrm{mask}}\otimes U_2^{\mathrm{mask}}\dots \otimes U_n^{\mathrm{mask}}\otimes X_{\mathrm{rest}}$     ${\mathcal{X}}_{\mathrm{mask}}\leftarrow {\mathcal{X}}_{\mathrm{mask}}.append\left(U^{\mathrm{mask}}\right)$ end for ${f|}_{X_{\mathrm{comp}}}\left(z\right)\leftarrow \mathrm{mean}\left(f\left({\mathcal{X}}_{\mathrm{mask}}\right)\right)$

Let $X_{\mathrm{comp}}=X_{\mathrm{comp},1}\otimes X_{\mathrm{comp},2}\otimes \cdots \otimes X_{\mathrm{comp},n}$ be considered at any iteration in the inductive process such that $X=X_{\mathrm{comp}}\otimes X_{\mathrm{rest}}$, where $X_{\mathrm{rest}}$ may be nonexistent. For a fixed $t\in \{1,2,\dots ,\tau \}$, the vectors ${\psi }_{\mathrm{comp},1}^t,\dots ,{\psi }_{\mathrm{comp},n}^t$ are the Shapley values for the game $(\{1,\dots ,n\},{\nu }_{\mathrm{comp}}^t)$—see Appendix A below:

$${\nu }_{\mathrm{comp}}^t\left(z\right)=\mathbb{E}\left(f_t\right|\mathrm{component} \mathrm{in} X \mathrm{indicated} \mathrm{by} z \mathrm{with} X_{\mathrm{rest}} \mathrm{fixed})-\mathbb{E}\left(f_t\right|X_{\mathrm{rest}} \mathrm{fixed}).$$

(1)

In particular, if n is small, the values ${\psi }_{\mathrm{comp},i}$, $i=1,2,\dots ,n$ can be calculated directly from the well-known Shapley value formula [10] [eq. 13].

Remark 1.

On the definition and estimation of ${f|}_{X_{\mathrm{comp}}}\left(z\right)$: The conditional expectation ${f|}_{X_{\mathrm{comp}}}\left(z\right)$ is proposed instead of $f\left(h_X\left(z\right)\right)$, which is typically used in the LIME framework [3], where $h_X$ maps binary values into the original input space. In that context, $h_X\left(z\right)$ shares the same values as X, but contains “missing values” for the components not indicated by z. For instance, Bento et al. [4] handled missing values in their TimeSHAP methodology by replacing them with the sample mean of the corresponding features. A similar approach was considered by Jiang et al. [6] in their SeqSHAP method. In contrast, we adopted an approach analogous to that of Štrumbelj and Kononenko [1] to estimate feature contributions, aligned with the framework proposed by Lundberg and Lee for the well-known SHAP values [2]. This marginalization approach is also considered in the WindowSHAP methodology [5] within the context of time series. Notably, the estimation of ${f|}_{X_{\mathrm{comp}}}\left(z\right)$ is subject to the inherent limitations of SHAP values. Specifically, Algorithm 2 involves masked inputs that, despite being syntactically compatible with f, may lie out of distribution with respect to the training set. Actually, as Kumar et al. [11] pointed out, “f has never seen an example like this in training, and has therefore not learned much about this part of the feature space. Its predictions on this feature space are not necessarily relevant to the task of explaining an in-distribution”. Indeed, Slack et al. [12] presented a technique that effectively hides the biases of any given classifier by allowing an adversarial entity to construct arbitrary, desired LIME and SHAP explanations. Nevertheless, our approach can be used as a complementary tool in the training stage of the forecasting cycle. In particular, verifying the effectiveness of adjustments based on information provided by explainability models is a natural part of the forecaster design cycle, e.g., to corroborate whether the performance improves after making some decisions based on the proposed methodology.

Remark 2.

Global consistency. The primary objective of the proposed methodology is to construct an explainability tree: given an interpretable component, the contributions of its subcomponents are estimated in a manner that preserves hierarchical coherence. This approach enables an adaptive refinement of the explainability granularity based on the outcomes of previous iterations. Note that it is natural to consider the final decomposition $X=X_1\otimes X_2\otimes \cdots \otimes X_{n_{\mathrm{total}}}$ (relabeling the components) if we knew from the outset exactly which components to analyze. Suppose that, for every hour t, the numbers ${\xi }_1^t,{\xi }_2^t,\dots ,{\xi }_{n_{\mathrm{total}}}^t$ are the Shapley values for the game $(\{1,\dots ,n_{\mathrm{total}}\},{\nu }^t)$, where

$${\nu }^t\left(z\right)=\mathbb{E}\left(f_t\right|component in X indicated by z)-\mathbb{E}\left(f_t\right).$$

However, the leaves of the tree do not share the same axiomatic status as conventional Shapley values—unless ${\epsilon }_{\mathrm{comp}}$ is null in every close-up and the symmetry property is global, since, in this case, ${\xi }_i={\phi }_i$ for all $i=1,2,\dots ,n_{\mathrm{total}}$. On the other hand, directly calculating the Shapley values of the fully decomposed game results in a loss of hierarchical structure.

We emphasize that the fundamental premise of the method is precisely that this partition is not known a priori. Refer to Figure 2 in comparison with Figure 1.

Figure 2. Explainability obtained from the direct calculation of the Shapley values ${\xi }_1,{\xi }_2,\dots ,{\xi }_{10}$ for the same $f\left(X\right)$ in Figure 1.

Remark 3.

Computational cost comparison: The computational complexity of KernelSHAP is exponential in n, where n is the number of features—in our case, interpretable components. For practical purposes, it is more convenient to consider the computational cost, which is given by $2^n·n_{\mathrm{mask}}·2·t_{\mathrm{prediction}}$, where $t_{\mathrm{prediction}}$ represents the time required to evaluate a single prediction and $n_{\mathrm{mask}}$ is the sample size used for the masking values. To compute the exact Shapley values for an instance, the algorithm must solve a weighted linear regression over all possible feature coalitions. Because the exact method is computationally expensive, practical implementations rely on Sampled KernelSHAP. Instead of generating all $2^n$ combinations, this approach draws a fixed number of samples. Specifically, the standard SHAP library implementation sets this number of samples to $n_{\mathrm{samples}}=2^n$ for $n\le 11$, and $n_{\mathrm{samples}}=2n+2^{11}$ by default for $n>11$, where n is the number of features. The computational cost of Sampled KernelSHAP is given by $n_{\mathrm{samples}}·n_{\mathrm{mask}}·2·t_{\mathrm{prediction}}$. On the other hand, to estimate the computational cost of our method and simplify the exposition, we can assume, without the loss of generality, that the branching factor $b\ge 2$ remains constant across both the initial step and each subsequent close-up. If $n_{\mathrm{close}-\mathrm{up}}$ is the number of close-ups executed during the procedure, then the total number of coalitions is given by $(n_{\mathrm{close}-\mathrm{up}}+1)2^b$. Since the number of interpretable subcomponents estimated during the process is $n=b+n_{\mathrm{close}-\mathrm{up}}(b-1)$, the computational cost in terms of n is

$$(n-1){\displaystyle \frac{2^b}{b-1}}·n_{\mathrm{mask}}·2·t_{\mathrm{prediction}}.$$

For any n, the branching factor can be bounded to maintain a manageable computational cost while even ensuring an asymptotic linear behavior with respect to n. For small n, Sampled KernelSHAP essentially involves all the $2^n$ possible coalitions.

This latter scenario aligns naturally with our method. Indeed, as established in George A. Miller’s seminal 1956 article [13], the human information processing capacity is limited to seven items or categories, plus or minus two. This cognitive limit is managed through chunking, a technique that organizes independent units of information into meaningful blocks.

4. Case Study: Short-Term Load Forecasting

Our case study is based on a real production model for CENACE (National Energy Control Center, Mexico). The system receives calendar data (hour, day, and month), demand data from the Northwest Regional Control Zone in Mexico, temperature and apparent temperature variables from 10 load zones across different cities aggregated into a single variable, and finally, binary data corresponding to 10 holidays. Using this information, the system generates an hourly multivariate time series comprising 15 variables, scaled to the interval $[-1,1]$ to be consistent with the model’s input requirements.

The implemented model f is a sequence-to-sequence model based on an LSTM encoder–decoder neural network. Both the encoder and decoder consist of a sequence of connected LSTM units. The encoder has 100 hidden states and an input sequence window of 180 h. The decoder also features 100 hidden units and a 36 h time window. The encoder’s task is to capture the underlying patterns of the input sequence, while the decoder processes the look-ahead data to generate the output sequence.

Specifically, the inputs are of the form $X=(X_{\mathrm{encoder}},X_{\mathrm{decoder}})$; the encoder accepts matrices of the form $180\times 15$ corresponding to 180 h with information from the “past” corresponding to the 15 variables, and the decoder receives calendar, weather, and holiday information in advance. The model output is a vector with the forecast of the energy demand for the next 36 h. The parameter vector was tuned on a training set of 118,969 data pair-points (1 January 2010 00:00 to 6 August 2023 23:00).

4.1. Experimental Comparison with KernelSHAP

In an initial experiment, we evaluated the robustness of the method with respect to the sample size used for masking. We compared this stability against KernelSHAP and reported the execution times for both approaches. In this experiment, we utilized the explainability tree topology $\mathrm{\Delta }$ shown in Figure 1; consequently, the resulting explainability vectors consisted of 10 final interpretable components. Since the number of components was less than 11, KernelSHAP avoids random sampling and solves the weighted least squares problem using the exhaustive set of all possible coalitions.

To this end, we considered three distinct relevance profiles within our context by randomly selecting three inputs X corresponding to summer, winter, and holiday. Specifically, tests were conducted using sample sizes of 100, 500, and 1000. Since these samples were selected at random, the experiment was repeated using five different random sample sets for each sample size. Figure 3 shows the three considered input profiles.

Figure 3. Contributions $\left|{\displaystyle \frac{{\phi }_i^t}{f_t\left(X\right)-{\phi }_0^t}}\right|$ of the 10 final interpretable components over the 36 h forecast horizon for the $\mathrm{\Delta }$ configuration across three distinct input profiles. The profile of holiday corresponds to the Christmas season.

Our empirical findings, reported in Table 2, reveal the following: (1) The standard deviation of the estimated explainability values across the five repetitions decreased as the sample size increased, though not significantly. In fact, even with a sample size of 100, the standard deviation was on the order of $4.0\times {10}^{-3}$, which we considered acceptable within this context; indeed, this is practically the same stability achieved by the KernelSHAP method. (2) We observed that the processing times were significantly shorter when estimating the explainability tree. The experiments were executed on an Apple M4 computer with 16 GB of RAM. For instance, for 100 samples, the KernelSHAP method took approximately 112 s, whereas our proposed method took less than 4 s in all cases. This is consistent with the fact that the execution time depends on the sample size of ${\mathcal{X}}_{\mathrm{train}}$ and the number of model calls. Indeed, in all cases, the KernelSHAP algorithm called the model 2046 times, while our method only called it 70 times.

Table 2. Robustness results with respect to sample size with execution time. Three different days from the test set were evaluated using 5 random sample sets per sample size.

X ID	${\mathcal{X}}_{\mathbf{train}}$ Sample Size	std $\left({\mathit{\xi }}_{\mathit{i}}^{\mathit{t}}\right)$	std $\left({\mathit{\phi }}_{\mathit{i}}^{\mathit{t}}\right)$	Execution Time KernelSHAP (s)	Execution Time Explainiability Tree (s)
summer	100	0.0047	0.0049	112.59 ± 0.36	3.87 ± 0.07
summer	500	0.0027	0.0025	409.38 ± 0.48	14.25 ± 0.07
summer	1000	0.0015	0.0015	785.66 ± 3.04	26.94 ± 0.37
winter	100	0.0045	0.0050	111.19 ± 1.34	3.84 ± 0.02
winter	500	0.0025	0.0024	408.74 ± 0.84	14.24 ± 0.03
winter	1000	0.0014	0.0015	781.6 ± 0.82	27.15 ± 0.03
holiday	100	0.0040	0.0051	111.62 ± 1.4	3.84 ± 0.01
holiday	500	0.0022	0.0031	409.13 ± 1.36	14.29 ± 0.09
holiday	1000	0.0010	0.0013	781.31 ± 0.61	27.15 ± 0.08

When addressing the question regarding the relationship between the explainability vectors provided by the tree and the output relative to the baseline Shapley values, the experiments demonstrated that the coherence of the explainability remained essentially invariant. Indeed, for this test, we used a sample of 100 elements from ${\mathcal{X}}_{\mathrm{train}}$ for the masking process. Additionally, we conducted the exercise with alternative topologies, ${\mathrm{\Delta }}_{\mathrm{alt}}$ and ${\mathrm{\Delta }}_{\mathrm{alt}}^{\prime }$. See Figure 4. The results are shown in Table 3.

Figure 4. Distribution of explainability values given by our method for configurations $\mathrm{\Delta }$, ${\mathrm{\Delta }}_{\mathrm{alt}}$, and ${\mathrm{\Delta }}_{\mathrm{alt}}^{\prime }$, categorized by sign change with respect to the sample KernelSHAP method.

Table 3. Coherence of KernelSHAP explainability with respect to the $\mathrm{\Delta }$, ${\mathrm{\Delta }}_{\mathrm{alt}}$, and ${\mathrm{\Delta }}_{\mathrm{alt}}^{\prime }$ configurations using the same 10 final interpretable components. The experiments were based on 50 randomly selected input vectors over a one-year period.

Tree Structure	Branching Factor	Model Calls	$|{\mathit{\xi }}_{\mathit{i}}^{\mathit{t}}-{\mathit{\phi }}_{\mathit{i}}^{\mathit{t}}|$	Mean Sign Change (%)	Spearman Coefficient	Execution Time (s)
$\mathrm{\Delta }$	4	70	0.0244 ± 0.0036	5.6%	0.9784 ± 0.0083	3.8973 ± 0.0458
${\mathrm{\Delta }}_{\mathrm{alt}}$	3	62	0.0294 ± 0.0041	6.7%	0.9625 ± 0.0130	3.3762 ± 0.0361
${\mathrm{\Delta }}_{\mathrm{alt}}^{\prime }$	5	106	0.0251 ± 0.0033	9.0%	0.9658 ± 0.0097	5.8304 ± 0.0353

For the three tree configurations, the average difference between the explainability vectors obtained with Sample KernelSHAP was approximately $2.0\times {10}^{-2}$. We also measured the percentage of sign changes in the explainability values between the proposed method and KernelSHAP. This percentage increased from 5.6% to 9.0%. However, as shown in Figure 4, the distribution of the explainability values that change sign revealed that they were around ten times smaller than those that maintained their sign. A substantial portion of these consisted of values very close to zero.

We computed the Spearman rank correlation coefficient between the Shapley vectors obtained via KernelSHAP and the explainability vectors from the proposed hierarchical method to quantify differences in the importance ranking of the interpretable components. Specifically, we analyzed the ordering of the explainability values for each of the 10 interpretable components across the 36 h forecasting horizon, resulting in a comprehensive ranking of these 360 values. Alignment with the KernelSHAP ordering was remarkably high, exceeding 0.96 across all three tree configurations.

The total variance of the contributions was computed over a sample of 50 inputs throughout the year for configuration $\mathrm{\Delta }$. The results demonstrated that the variability was heavily dominated by future-horizon features, with weather-futureand calendar-future accounting for $34.0\%$ and $30.2\%$, respectively. Short-term historical features exhibited moderate contributions, where calendar-day before represented $19.9\%$ and load demand 1–12 h before captured $9.5\%$. Conversely, the remaining components—including holiday—collectively contributed just $6.4\%$ to the total variance. This does not imply that the methodology does not react to holidays, as illustrated in the third profile in Figure 3; rather, because there are very few holidays throughout the year, their overall variance contributions are generally negligible.

We repeated the exercise by omitting the error correction for configuration $\mathrm{\Delta }$. Due to the relatively small magnitude of the error, the explainability narrative expectedly remained unchanged compared to the direct calculation via KernelSHAP. For instance, the Spearman coefficient shifted on average from $0.9784$ to $0.9733$. Furthermore, the rate of sign changes was even lower without error correction, dropping from 5.6% to 4.6%, affecting—once again—values very close to zero. However, without error correction, the hierarchical structure was lost, and the average distance to the Shapley values increased slightly, moving from $2.44\times {10}^{-2}$ to $3.21\times {10}^{-2}$.

Regarding the out-of-distribution samples for masking (Remark 1), we conducted tests by restricting the background sampling strategy to “similar inputs.” Specifically, we looked at days from the same season to minimize unrealistic feature combinations, such as combining future summer weather of 26–44 °C (78.8–111.2°F) with past winter weather of 8–24 °C (46.4–75.2°F). We generated five sets of 100 random samples using historical data from previous years, corresponding to the same month as the selected summer day. To explore the sampling effect, we calculated the standard deviation across the five selective random samplings relative to the mean value of the five general random samplings. In our case, the resulting standard deviation was $0.0075\pm 0.0019$. This value is comparable to the intrinsic variability among the random samples themselves, as shown in Table 2. Although we observed no significant differences in this case, it is possible that, under another scenario, it could be an element requiring special attention, especially if the explainability is linked to sensitive matters.

4.2. Granular Analysis via Multiple Close-Ups

We considered the initial decomposition rule:

$$X=X_{\mathrm{load}\_\mathrm{demand}}\otimes X_{\mathrm{calendar}}\otimes X_{\mathrm{weather}}\otimes X_{\mathrm{holidays}}$$

To describe the interpretable components more easily, let us consider the timeline $[-180,36]$. We assume that discrete encoder events are contained within the segment $[-180,0]$ (representing the 180 past hours), while decoder events are in the segment $[0,36]$ (representing the 36 future hours). The interpretable component $X_{\mathrm{load}\_\mathrm{demand}}$ involves only values within $I_{\mathrm{load}\_\mathrm{demand}}=[-180,0]$, whereas the remaining components span the interval $I_{\mathrm{calendar}}=I_{\mathrm{weather}}=I_{\mathrm{holidays}}=[-180,36]$. We compute ${\phi }_{\mathrm{load}\_\mathrm{demand}}$, ${\phi }_{\mathrm{calendar}}$, ${\phi }_{\mathrm{weather}}$, and ${\phi }_{\mathrm{holidays}}$ to set the stage for the explanation of $f\left(X\right)$. The final explainability tree T is obtained by applying consecutive close-ups. We state that an iteration n is complete when the largest height-n subtree of T has been constructed.

For this analysis, the explainability vectors were calculated using a random sample of 100 data points from the training set. The average contribution of this component across all forecasted hours accounted for only 0.06% of the sum of the average contributions of all initial components. In contrast, $X_{\mathrm{load}\_\mathrm{demand}}$, $X_{\mathrm{weather}}$, and $X_{\mathrm{calendar}}$ accounted for 72.75%, 22.69%, and 4.56%, respectively. Therefore, we omitted the further analysis of this interpretable component.

For the remaining initial components, we constructed subcomponents by partitioning the original time interval into disjoint subintervals. For each hour $t=1,2,\dots ,36$, we considered a function ${\mathrm{Shape}}_{\mathrm{comp}}^t$ defined over the corresponding time interval $I_{\mathrm{comp}}$, which is progressively refined at each iteration. This function is piecewise constant across the subintervals $I_{\mathrm{comp},i}$, such that the signed area under the curve within each subinterval equals ${\phi }_{\mathrm{comp},i}$. By the breakdown contribution property, it holds that, regardless of the iteration,

$${\int }_{I_{\mathrm{comp}}}{\mathrm{Shape}}_{\mathrm{comp}}^t\left(s\right)ds={\phi }_{\mathrm{comp}}.$$

Consequently, each iteration provides a more detailed breakdown of the initial component’s contribution over time, allowing the resolution to be adjusted as needed and the focus to be placed exclusively on specific regions of interest.

In Figure 5, we present the resulting functions ${\mathrm{Shape}}_{\mathrm{load}\_\mathrm{demand}}^1$ and ${\mathrm{Shape}}_{\mathrm{load}\_\mathrm{demand}}^{36}$ for a nine-subcomponent leaf subtree generated after three iterations. The close-ups exhibit a consistent and coherent profile across all time points, with a substantial shift occurring within 30 h prior to the forecast—the exact point at which the importance of historical events begins to diminish significantly. We observed that the longer the forecast horizon, the less importance was attributed to past events. Indeed, activities from the distant past remained negligible when examining these close-up visualizations.

Figure 5. Explainability subtree for $X_{\mathrm{load}\_\mathrm{demand}}$ and its corresponding SHAPE functions for forecast hours 1 and 36 across tree iterations. On the right side are the results of the three iterations of the functions ${\mathrm{Shape}}_{\mathrm{load}\_\mathrm{demand}}^1$ (purple) and ${\mathrm{Shape}}_{\mathrm{load}\_\mathrm{demand}}^{36}$ (magenta).

With respect to $X_{\mathrm{weather}}$, we considered an 18-subcomponent leaf subtree generated after three iterations, where 12 of these subcomponents correspond to decoder close-ups. We observed that the greatest contribution was distributed between 72 h prior to the forecast and the future calendar data known in advance. For these future calendar features, the first 18 h of the decoder concentrated the contribution—exhibiting negative values—precisely on those same forecast hours. Furthermore, the last 18 h of the decoder captured the small remaining contribution, affecting only the final hours of the forecast horizon.

For $X_{\mathrm{calendar}}$, we considered the same tree as $X_{\mathrm{weather}}$. In the first iteration of the analysis for $X_{\mathrm{calendar}}$, we observed that the greatest contribution was distributed between 72 h prior to the forecast and the future calendar data known in advance. In the second iteration, we observed that the contribution from the past was actually focused within the last 24 h prior to the forecast. Interestingly, exploring the decoder calendar data revealed that the explainability value of ${\phi }_{[0,18]}^{36}$ corresponding to the component $X_{[0,18]}$ was nearly zero ($1.62\times {10}^{-5}$); however, close-ups from the subsequent iteration indicated that this behavior does not imply that all subcomponents within that interval possess equally negligible values, but rather that their opposing contributions (on the order of $2\times {10}^{-2}$) mutually cancel out. For $X_{\mathrm{holidays}}$, no further close-up analysis was required. In this analysis, the results remained consistent across the various close-ups along the temporal horizon.

For this exercise, it was necessary to consider only 352 coalitions, rather than the $2^{11}+2\times 46=2140$ that would have resulted from applying the sample KernelSHAP method to estimate the Shapley vectors, assuming that the configuration of $\mathrm{\Delta }$ was known in advance.

5. Conclusions

We present a model-agnostic framework for explaining the contributions of an input X to a forecast $f\left(X\right)$, building upon the principles of LIME and SHAP. The proposed methodology allows analysts to refine the explanations hierarchically to the extent required by the specific problem context, thereby generating a multi-resolution explainability narrative.

We evaluated the method on two alternative tree configurations that yielded the same final components. Furthermore, it should be noted that the initial full configuration was, in fact, an extreme scenario. We report that the coherence of the explainability was successfully maintained for these cases, as shown in Table 3. This outcome aligns with our intuitive expectations; although the explainability tree inherently depends on its topology throughout the process, numerous masked values remain coincident as long as the final interpretable components are identical. Naturally, the exact variance may change depending on the context and the complexity of the input. To assess the robustness and reliability of the method against analyst-dependent refinement in a real-world scenario, several configurations can be implemented. For instance, one can evaluate alternative tree configurations that yield the same final components in order to verify whether the methodology is reliable for that specific use case.

Finally, while this method does not preserve all axioms of Shapley values and inherits some of their weaknesses, we highlight its practical utility under certain conditions due to the following advantages:

• Computational efficiency control: The precise management of the execution time via the branching factor is enabled, even in scenarios requiring high-resolution or detailed explanations.
• Straightforward practical implementation: For every close-up, the workflow only requires a dictionary specifying the partitions of the input X and a concatenation function to reconstruct a valid input for f. Consequently, the identical core procedure can handle the initial and inductive steps for any partition dictionary or assembly function.
• Architecture-agnostic deployment: Since the framework does not depend on the internal structure of f, it can be seamlessly integrated into the production pipeline of a forecasting cycle. This setup requires minimal configuration changes for different models, utilizing a unified engine as a complement to continuous forecast monitoring.
• Flexible design-stage exploration: During the model development phase, it serves as a versatile tool for exploring the behavioral dynamics of f, providing actionable insights for model refinement and tuning.
• Automation and adaptability potential: The underlying methodology provides a foundation for developing automated pipelines for generating explainability trees, e.g., it is possible to consider trees with a fixed branching factor and a desired minimum level of granularity to obtain an “unsupervised” explainability map.