LIME$\mathbf{TREE}$: Consistent and Faithful Surrogate Explanations of Multiple Classes

Abstract

Explainable artificial intelligence provides tools to better understand predictive models and their decisions, but many such methods are limited to producing insights with respect to a single class. When generating explanations for several classes, reasoning over them to obtain a comprehensive view may be difficult since they can present competing or contradictory evidence. To address this challenge, we introduce the novel paradigm of multi-class explanations. We outline the theory behind such techniques and propose a local surrogate model based on multi-output regression trees—called LIME$\mathrm{TREE}$—that offers faithful and consistent explanations of multiple classes for individual predictions while being post-hoc, model-agnostic and data-universal. On top of strong fidelity guarantees, our implementation delivers a range of diverse explanation types, including counterfactual statements favored in the literature. We evaluate our algorithm with respect to explainability desiderata, through quantitative experiments and via a pilot user study, on image and tabular data classification tasks, comparing it with LIME, which is a state-of-the-art surrogate explainer. Our contributions demonstrate the benefits of multi-class explanations and the wide-ranging advantages of our method across a diverse set of scenarios.

1. Introduction

The explainability of predictive systems based on artificial intelligence (AI) algorithms has become one of their most desirable properties [1,2,3]. While a wide array of explanation types—supplemented by numerous techniques to generate them—is available [4], contrastive statements are dominant [5,6,7,8,9,10]. Their particular realization in the form of counterfactual examples is the most ubiquitous given its everyday usage among humans and solid foundations in social sciences [5] as well as its compliance with various legal frameworks [6]. Such insights are usually of the form: “Had certain aspects of the given case been different, the predictive model would behave like so instead.” The conditional part of this proposition usually prescribes a modification of the feature vector of a particular data point, whereas the hypothetical fragment of the statement tends to capture the resulting change in class prediction.

While offering a very appealing recipe for swaying an automated decision, these explanations are intrinsically restricted to a pair of outcomes, which may impact their utility, effectiveness and comprehensibility. They can either highlight an explicit contrast between two classes—“Why A rather than B?”—or be implicit instead—“Why A (as opposed to anything else)?” As a result, counterfactuals, but also single-class explanations more broadly, have been shown to simply justify conclusions of AI systems, which may be counterproductive as it implicitly limits the number of possibilities that the explainees consider, thus bias their perception, impede independent reasoning and yield unwarranted reliance on AI or prevent trust from developing altogether [11].

In human explainability, this limitation can be overcome with follow-up questions, progressively exploring and narrowing down the scope of the lack of understanding until finally eliminating it. One could imagine generating multiple counterfactuals across all the possible outcomes to mimic this process, e.g., “Why A (and not B or C)?”, “Why A rather than B?”, “Why A instead of C?”, “Why B (and not A or C)?”, “Why B instead of A?”, etc., for three outcomes A, B and C. Other explainability methods could also be employed in this scenario to provide a wider gamut of insights varying in scope, complexity and explanation target. Such approaches embody the recent hypothesis-driven decision support conceptualization of explainable AI (XAI), which aims to provide diverse evidence for data-driven predictions instead of offering a recommendation to simply accept or reject a preselected AI decision [12]; this process keeps the explainees engaged instead of displacing them, utilizes their expertise, and mitigates over- and under-dependence on automation.

However, implementing this paradigm with current XAI tools is likely to fall short given that they tend to generate independent insights whose one-class limitation prevents them from capturing and communicating a congruent bigger picture. The lack of a single origin and shared context may yield insights that do not overlap or are outright contradictory—e.g., different conditionals used by counterfactuals and disparate pieces of evidence output by other techniques—preventing the explainees from drawing coherent conclusions and adversely affecting their trust and decision-making capabilities [13]. While a promising research direction, to the best of our knowledge the challenge of generating inherently consistent explanations of multiple classes has neither been addressed for counterfactuals nor any other explanation type. In this paper, we fill this gap by introducing the novel concept of multi-class explanations, where individual insights pertaining to different predictions (of a selected instance) originate from a single explanatory source.

To this end, we:

(i)

define a multi-class explainability optimization objective;

(ii)

operationalize it in the form of a local surrogate;

(iii)

offer an algorithm for building multi-class explainers; and

(iv)

implement it with multi-output regression trees.

We evaluate our method—called LIME$\mathrm{TREE}$—along three dimensions: an analytical assessment of human-centered XAI desiderata; a series of quantitative experiments on tabular and image data measuring explainer fidelity; and a qualitative user study capturing explainees’ preferences. We choose to demonstrate multi-class explainability with a surrogate [14] since this design yields an explainer that is post-hoc—i.e., capable of being retrofitted to pre-existing AI systems—model-agnostic—i.e., compatible with any predictive algorithm—and data-universal—i.e., suitable for tabular, text and image domains. Additionally, by using a (binary) decision tree [15] as the surrogate, LIME$\mathrm{TREE}$ offers a broad range of explanation types such as model structure visualization, feature importance, exemplars, logical rules, what-ifs and, most importantly, counterfactuals [16]. This suite of investigative mechanisms supports diverse explanation scopes spanning model simplification, subspace approximation and prediction rationales.

LIME$\mathrm{TREE}$ offers solutions to many shortcomings of currently available surrogate explainers in addition to addressing limitations found across the social and technical dimensions of XAI [17]. Specifically, by using (shallow) binary regression trees as surrogate models, it can guarantee full fidelity of the explanations with respect to the investigated black box under certain conditions, thus addressing one of the major criticisms of post-hoc approaches [1,18]. The flexible explanation generation process additionally enables it to comply with a range of desiderata such as feasibility and actionability [7] as well as facilitate algorithmic recourse [19], to name just a few [20,21]. The availability of multiple diverse explanation types also allows it to provide explainability to a broad range of stakeholders and satisfy their diverse needs [22]. With all of these contributions, we hope to launch multi-class explainability as a novel, highly beneficial XAI research direction.

2. Related Work and Background

LIME$\mathrm{TREE}$ builds upon two prominent findings in XAI: counterfactuals [5,6] and surrogate explainers [14,16,23,24,25,26]. As noted earlier, the former are lauded for their human-centered aspects, and the latter exhibit numerous appealing technical properties, making them one of the most flexible types of explainers. In a nutshell, surrogates mimic the behavior of more complex, hence opaque, predictive systems either locally or globally with simpler, inherently interpretable models, thereby offering human-comprehensible insights into their operation [14,23]. Unlike surrogates and counterfactuals, multi-class explainability is a largely under-explored topic. While counterfactual explanations can be generated for multiple classes [27], such insights may not present a coherent perspective given that they can be conditioned on different sets of features. One of the very few pieces of work, if not the only, that directly addresses this challenge expands Generalized Additive Models (GAMs [28])—which are inherently transparent and powerful predictors popular in high stakes domains [29]—to multiple classes [30].

LIME [23] is one of the most popular surrogate approaches; it uses sparse linear regression to explain (probabilistic) black-box predictions. It augments the classic paradigm of surrogate explainers with interpretable representations (IR) of raw data, making them compatible with a variety of data domains (such as images and text) and extending their applicability beyond inherently interpretable features (of tabular data). High modularity and flexibility of these explainers [24] encouraged the research community to compose their different variants, some of which use decision trees as the (local) surrogate model [9,16,25,31]. For example, Waa et al. [9] showed how a local one-vs-rest classification tree can be used to produce contrastive explanations; and Shi et al. [31] fitted a local shallow regression tree whose structure constitutes an explanation. The interpretability of decision trees and their ensembles has also been investigated outside of the surrogate explainability context [16,32,33,34]. Sokol and Flach [33,34] demonstrated how to interactively extract personalized counterfactuals from a decision tree; and Tolomei et al. [32] introduced a method to explain predictions made by tree ensembles also with counterfactuals.

More specifically, LIME builds a local surrogate model $g\in \mathcal{G}$ to explain the prediction of an instance $\mathring{x}\in \mathcal{X}$ with respect to a selected class c for a probabilistic black box $f:\mathcal{X}\mapsto \mathcal{Y}$, where $\mathcal{G}$ is the space of (sparse linear) surrogate models, $\mathcal{X}$ is the input data domain, $\mathcal{Y}$ is the space of n-dimensional probability vectors, $n\in {\mathcal{N}}^{+}$ is the number of target classes, and $c\in [1,\dots ,n]$. To this end, it employs a user-defined interpretable representation transformation function $\mathit{IR}:\mathcal{X}\mapsto {\mathcal{X}}^{\prime }$, which encodes presence (1) and absence (0) of $d\in {\mathcal{N}}^{+}$ selected human-comprehensible concepts found in a data point $x\in \mathcal{X}$, i.e., ${\mathcal{X}}^{\prime }={\{0,1\}}^d$. Additionally, $\mathit{IR}$ is defined such that the explained instance is assumed to have all of the concepts present, i.e., $\mathit{IR}\left(\mathring{x}\right)={\mathring{x}}^{\prime }=[1,\dots ,1]$, which is an all-1 vector. This step allows us to generate “conceptual” variations of $\mathring{x}$ by drawing a collection of binary vectors $X^{\prime }=\{x^{\prime }:x^{\prime }\in {\mathcal{X}}^{\prime }\}$.

Next, $X^{\prime }$ is converted back to the original data domain $\mathcal{X}$ using the inverse of the interpretable representation transformation function ${\mathit{IR}}^{-1}:{\mathcal{X}}^{\prime }\mapsto \mathcal{X}$, i.e., $X=\{{\mathit{IR}}^{-1}\left(x^{\prime }\right):x^{\prime }\in X^{\prime }\}$, which facilitates predicting these instances with the explained black box f, focusing on the probabilities of the selected class c, i.e., $Y_c=\{f_c\left(x\right):x\in X\}$. These predictions capture the influence of (the presence of) each human-comprehensible concept on the (change in) prediction of class c. We can quantify this dependence by fitting sparse linear regression to the binary sample $X^{\prime }$ and probabilities $Y_c$. This procedure can be focused on a specific aspect of the data sample by computing its distanceℓ to the explained instance either in the original or interpretable representation—i.e., $\ell :\mathcal{X}\times \mathcal{X}\mapsto \mathbb{R}$ or $\ell :{\mathcal{X}}^{\prime }\times {\mathcal{X}}^{\prime }\mapsto \mathbb{R}$—then transformed into a similarity measure by passing it through a kernel $\kappa :\mathbb{R}\mapsto \mathbb{R}$ and used as weight factor for training the surrogate model. This step allows to prioritize smaller changes to the instance, e.g., give more significance to samples with fewer alterations in the concept space.

LIME optimizes the fidelity of the surrogate, i.e., its ability to approximate the predictive behavior of the explained black box, and complexity of the resulting explanation, i.e., its human comprehensibility; this objective $\mathcal{O}$ is formalized in Equation (1). Complexity $\mathrm{\Omega }$, in the case of linear models, is computed as the number of non-zero (or significantly larger than zero) coefficients ${\mathrm{\Theta }}_g$ of the surrogate g—see Equation (2). High fidelity entails small empirical loss $\mathcal{L}$—Equation (3)—calculated between the outputs of the black box f and the surrogate g using data sampled “around” the explained instance. Individual loss components are weighted by similarity scores—$\omega (x;\mathring{x})$ for $x\in X$ or $\omega (x^{\prime };{\mathring{x}}^{\prime })$ for $x^{\prime }\in X^{\prime }$ depending on the domain—derived by kernelising distance between the explained instance and sampled data. This loss is inspired by Weighted Least Squares, where the weights are similarity scores.

$$\mathcal{O}(\mathcal{G};f)=\underset{g\in \mathcal{G}}{arg\; min}\underset{\underset{\mathrm{fidelity}}{⏟}}{\mathcal{L}(f,g)}+\underset{\underset{\mathrm{complexity}}{⏟}}{\mathrm{\Omega }\left(g\right)}$$

(1)

$$\mathrm{\Omega }\left(g\right)=\sum _{\theta \in {\mathrm{\Theta }}_g}\mathbb{1}\left(\right|\theta |>0)/|{\mathrm{\Theta }}_g|$$

(2)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{L}(f,g;X^{\prime },\mathring{x},c)=& {\displaystyle \frac{1}{{\sum }_{x^{\prime }\in X^{\prime }}\omega \left(x^{\prime };\mathit{IR}\left(\mathring{x}\right)\right)}}\hfill \\ \hfill & \sum _{x^{\prime }\in X^{\prime }}\omega \left(x^{\prime };\mathit{IR}\left(\mathring{x}\right)\right){\left(f_c\left({\mathit{IR}}^{-1}\left(x^{\prime }\right)\right)-g\left(x^{\prime }\right)\right)}^2\hfill \\ \hfill & \mathrm{where}\omega (x^{\prime };{\mathring{x}}^{\prime })=\kappa \left(\ell \left(x^{\prime },{\mathring{x}}^{\prime }\right)\right)\hfill \end{array}\end{array}$$

(3)

The precise definitions of the interpretable representation transformation function $\mathit{IR}$ and its inverse ${\mathit{IR}}^{-1}$ depend on the data domain. For text, $\mathit{IR}$ splits it into d tokens, e.g., using the bag-of-words approach, whose presence (1) or absence (0) is encoded by ${\mathcal{X}}^{\prime }$; setting a component of this domain to 0 is thus equivalent to removing a token from a text excerpt. For images, this domain transformation relies on the super-pixel partition of a picture into d non-overlapping patches whose binary vector encoding indicates whether a particular segment is preserved (1) or discarded (0); since parts of an image cannot be removed directly, an occlusion proxy that replaces selected patches with a predetermined color is used. Figure 1 shows an interpretable representation of an image and its LIME explanations for the top three predictions. LIME explanations of text follow a similar pattern, with it being split into tokens (its interpretable representation) whose influence on a prediction—e.g., the positive or negative sentiment of a sentence—is quantified through the coefficients of the corresponding surrogate linear model.

For tabular data, the $\mathit{IR}$ function is more complex; continuous features are first discretized and then, together with any remaining categorical attributes, binarized. The latter step assigns, separately for every feature, 1 to the discrete partition where the explained instance is located, with all the other partitions merged and represented by 0. As a result, the mapping between ${\mathcal{X}}^{\prime }$ and $\mathcal{X}$ tends to be non-deterministic, unlike the corresponding ${\mathit{IR}}^{-1}$ transformation for image and text data. Further information about surrogate explainers—including their generalization and in-depth analysis of their individual building blocks in the context of text, image and tabular data domains—can be found in the literature [16,24,25,26].

3. LIME$\mathbf{TREE}$

LIME fits a separate surrogate model to the probabilities of each class of interest. This makes the process of discovering the dependencies between multiple classes challenging as each explanation needs to be interpreted in isolation. A surrogate fitted to class A is implicitly a one-vs-rest explainer since it can only answer questions about the probability of this single class, with the complementary probability $p(\neg A)=1-p\left(A\right)$ modeling the union of all the other classes $\neg A\equiv B\cup C\cup \cdots$. Interpreting the magnitude of the probability $p\left(A\right)$ output by a surrogate trained for class A can also be problematic when explaining multi-class black boxes. For example, if $p\left(A\right)\le 0.5$, we cannot be certain whether there is a single class B with $p\left(B\right)>p\left(A\right)$, or alternatively the combined probability of all the complementary classes $p(\neg A)$ is greater than or equal to $p\left(A\right)$, with no single class dominating over $p\left(A\right)$.

Moreover, linear predictors—thus such surrogates as well—are unable to model target variables that are non-linear with respect to input features [35]—a property that does not necessarily hold for high-level features such as the concepts encoded by IRs [25]. Their high inter-dependence may also have adverse effects on explanation quality. Additionally, modeling probabilities with linear regression risks confusing the explainees who expect an output bounded between 0 and 1 but may be given a numerical prediction outside of this range.

We address the challenge of simultaneously explaining multiple predicted classes of an instance output by a probabilistic model by proposing a first-of-a-kind surrogate explainer based on binary multi-output regression trees. It facilitates multi-class modeling in a regression setting, allowing the surrogate to capture the interactions between multiple classes, hence explain them coherently. Each node of such a tree approximates the probabilities of every explained class—a level of detail that is impossible to achieve with surrogate multi-class classifiers—thus reflecting how individual interventions in the interpretable domain affect the predictions. Figure 2 shows an example of a surrogate multi-output regression tree. This is a significant improvement over training a separate regression surrogate for each explained class, which may produce diverse, inconsistent, competing or contradictory explanations—thus risk confusing the explainees and put their trust at stake—whenever these models do not share a common tree structure or split on different feature subsets. Our contributions establish a new direction in XAI research—concerned with consistent and faithful explanations of multiple classes—and offer a pioneering method to address this challenge.

Moreover, employing decision trees as surrogates overcomes the shortcomings identified when linear models are used to this end [16,24,25]. Trees neither presuppose independence of features nor existence of a linear relationship between them and the target variable [35]. While surrogate regression trees that approximate the probability of a single class are guaranteed to output a number within the $[0,1]$ range—since the estimate is calculated as an average (default tree behavior)—this may not necessarily hold for multi-output trees. Approximating probabilities of multiple classes by calculating the mean of their respective predictions across a number of instances may yield averages whose sum is greater than 1; nonetheless, these values can be rescaled to avoid confusing the explainees.

While surrogates based on linear models are limited to (interpretable) feature influence explanations—see Figure 1—employing trees offers a broad selection of diverse explanation types. These include: (1) visualization of the tree structure; (2) tree-based (interpretable) feature importance (Gini importance [36]); (3) logical conditions extracted from root-to-leaf paths; (4) exemplar explanations taken from the training data assigned to the same leaf; (5) answers to what-if questions generated based on the tree structure (e.g., by querying the model); and (6) counterfactuals retrieved by comparing and applying logical reasoning to different tree paths [16]. The first two explanation types uncover the behavior of a black box in a given data subspace; the remainder targets specific predictions. Since all the six explanation types—see Section 6 for their examples—are derived from a single (surrogate) model, they are guaranteed to be coherent and their diversity should appeal to a wide range of audiences.

To ensure low complexity and high fidelity of our multi-output regression trees, we employ the optimization objective $\mathcal{O}$ from Equation (1). Since we are using surrogate trees, we modify the model complexity function $\mathrm{\Omega }$ to measure the depth or width (number of leaves) of the tree as given by Equation (4), where d is the dimensionality of the binary interpretable domain ${\mathcal{X}}^{\prime }$. This choice depends on the type of explanation that we want to extract from the surrogate tree, e.g., depth may be preferred when visualizing the tree structure or extracting decision rules. In some cases, such as unbalanced trees, optimizing for width or a mixture of the two may be more desirable. We also adapt the loss function $\mathcal{L}$ to account for the surrogate tree g outputting multiple values in a single prediction as shown in Equation (5), where $C\subseteq [1,\dots ,n]$ are the classes to be explained by g, for which the c subscript in $g_c\left(x^{\prime }\right)$ indicates the prediction of a selected class $c\in C$ for the data point $x^{\prime }$.

$$\mathrm{\Omega }(g;d)={\displaystyle \frac{\mathrm{depth}\left(g\right)}{d}}\mathrm{or}\mathrm{\Omega }(g;d)={\displaystyle \frac{\mathrm{width}\left(g\right)}{2^d}}$$

(4)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{L}(f,g;X^{\prime },\mathring{x},C)=& {\displaystyle \frac{1}{{\sum }_{x^{\prime }\in X^{\prime }}\omega (x^{\prime };\mathit{IR}\left(\mathring{x}\right))}}\hfill \\ \hfill & \sum _{x^{\prime }\in X^{\prime }}\left({\displaystyle \frac{\omega (x^{\prime };\mathit{IR}\left(\mathring{x}\right))}{1+\mathbb{1}\left(\right|C|>1)}}\sum _{c\in C}{\left(f_c\left({\mathit{IR}}^{-1}\left(x^{\prime }\right)\right)-g_c\left(x^{\prime }\right)\right)}^2\right)\hfill \end{array}\end{array}$$

(5)

Note that the inner sum over the explained classes ${\sum }_{c\in C}$ is normalized by ${(1+\mathbb{1}(\left|C\right|>1\left)\right)}^{-1}$, which is 1 when the surrogate is built for a single class and becomes ${\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$ for more classes. The loss given by Equation (5) is thus equivalent to the one in Equation (3) in the former case, and in the latter, the scaling factor ensures that the inner sum is between 0 and 1 since the biggest squared difference is 2, which happens when the predictions of f and g assign a probability of 1 to two different classes, e.g., $[1,0,0]$ and $[0,0,1]$. An additional assumption is that the sum of values predicted by each leaf of the surrogate tree is at most 1, which, as noted earlier, may in some cases require normalization. In practice, the surrogate explainer is built by iteratively adding splits to a multi-output regression tree—thus incrementally increasing its complexity $\mathrm{\Omega }(g;d)$ but also improving its predictive power—which allows it to progressively minimize the loss $\mathcal{L}$ and optimize the objective $\mathcal{O}$. This procedure—captured by Algorithm A1 given in Appendix A—terminates when the loss $\mathcal{L}$ (calculated with Equation (5)) reaches a certain, user-defined level $ϵ\in [0,1]$, which corresponds to the fidelity of the local surrogate, i.e., $\mathcal{L}\left(f,g;X^{\prime },\mathring{x},C\right)\le ϵ$. Figure 3 provides a high-level overview of LIME$\mathrm{TREE}$.

4. Fidelity Guarantees

The flexibility of surrogate explainers—they are post-hoc, model-agnostic and, often, data-universal—also contributes to the instability and occasional unreliability of their explanations [24,25,26,37,38]. Their subpar fidelity, i.e., predictive coherence with respect to the explained black box, is thus a major barrier to their uptake [1]. In addition to remedying the shortcomings of linear surrogates, LIME$\mathrm{TREE}$ comes with strong fidelity guarantees, which can be achieved in practice while preserving low explanation complexity.

To imbue LIME$\mathrm{TREE}$ with near-full or full fidelity, we identify the minimal IR set $X_{\mathit{min},T}^{\prime }\subseteq {\mathcal{X}}^{\prime }$. It is unique to a tree T and composed of binary vectors $x_{\mathit{min},t}^{\prime }$ drawn from the IR—one per leaf $t\in T$ of the surrogate tree—that have the least number of 0 components while still being assigned to the leaf t. The construction of this set is formalized in Definition 1 and can be understood as seeking instances with the highest number of human-interpretable concepts being present, e.g., minimal occlusion for images, for each leaf.

Definition 1 

(Minimal Representation). Assume a binary decision tree $g\in \mathcal{G}$ fitted to a binary d-dimensional data space ${\mathcal{X}}^{\prime }={\{0,1\}}^d$, with T denoting its set of leaves. This tree assigns a leaf $t\in T$ to a data point $x^{\prime }\in {\mathcal{X}}^{\prime }$ with the function $g_{\mathit{id}}\left(x^{\prime }\right)=t$. For a given tree leaf t, its uniqueminimal data point $x_{\mathit{min},t}^{\prime }$ is given by

$$x_{\mathit{min},t}^{\prime }=\underset{x^{\prime }\in {\mathcal{X}}^{\prime }}{arg\; max}\sum _{i=1}^d{x}_i^{\prime }s.t.g_{\mathit{id}}\left(x^{\prime }\right)=t,$$

where $x_i^{\prime }$ is the i-th component of the binary vector $x^{\prime }$. We can further define a minimal set of data points $X_{\mathit{min},T}^{\prime }\subseteq {\mathcal{X}}^{\prime }$—uniquely representing a tree g and the set of its leaves T—that is composed of all the minimal data points for this tree as

$$X_{\mathit{min},T}^{\prime }=\{x_{\mathit{min},t}^{\prime }:t\in T\}.$$

Next, we transform this minimal representation set $X_{\mathit{min},T}^{\prime }$ from the interpretable into the original domain using the inverse of the IR transformation function: $X_{\mathit{min},T}=\{{\mathit{IR}}^{-1}\left(x_{\mathit{min},t}^{\prime }\right):x_{\mathit{min},t}^{\prime }\in X_{\mathit{min},T}^{\prime }\}$. We then predict class probabilities for each instance in $X_{\mathit{min},T}$ with the black box f and replace the values estimated by the surrogate tree with these probabilities for each leaf $t\in T$, i.e., modify the surrogate tree by overriding its predictions. Doing so is only feasible for the tree leaves as the minimal data points for some of the splitting nodes are indistinguishable, e.g., all the nodes on the root-to-leaf path that decides every interpretable feature to be 1 are non-unique and all would be represented by the unmodified explained instance. We additionally assume that the explained predictive model is deterministic, therefore it always outputs the same prediction for a given instance.

This variant of LIME$\mathrm{TREE}$—called TREE and codified by Algorithm A2 provided in Appendix A—guarantees full fidelity of the surrogate tree with respect to the explanations derived from the tree structure such as counterfactuals and root-to-leaf decision rules (see Section 6 for their examples). However, for this property to hold the function $\mathit{IR}$ transforming data from their original domain into the interpretable representation has to be deterministic [25], which holds for image and text but not for tabular data (refer back to Section 2). The rationale behind this claim is outlined in Lemma 1 (proof in Appendix B), which follows from the subsequent discussion.

Lemma 1 

(Structural Fidelity). A surrogate tree can achieve full fidelity with respect to the explanations derived from its structure—i.e., model-driven explanations—if the interpretable representation transformation function $\mathit{IR}$ is deterministic. Therefore, an instance $x\in \mathcal{X}$ can be translated into a unique point $\mathit{IR}\left(x\right)=x^{\prime }\in {\mathcal{X}}^{\prime }$ and vice versa ${\mathit{IR}}^{-1}\left(x^{\prime }\right)=x$, i.e., the mapping is one-to-one.

Lemma 1 guarantees that each leaf in the surrogate tree is associated with only one data point $x_{\mathit{min},t}$ in the original representation $\mathcal{X}$. This instance is derived from the minimal interpretable data point $x_{\mathit{min},t}^{\prime }$ by applying the inverse of the interpretable representation transformation function ${\mathit{IR}}^{-1}$, i.e., $x_{\mathit{min},t}={\mathit{IR}}^{-1}\left(x_{\mathit{min},t}^{\prime }\right)$. Therefore, $x_{\mathit{min},t}$ represents the explained instance with the smallest possible number of concepts deleted from it such that $g_{\mathit{id}}\left(x_{\mathit{min},t}^{\prime }\right)=t$. By assigning the probabilities predicted by the explained black box for each data point $x_{\mathit{min},t}$ to the corresponding leaf t of the surrogate, it achieves full fidelity for the minimal representation set $X_{\mathit{min},T}$, which is the backbone of model-driven explanations.

While such an approach ensures full fidelity of model-driven explanations, the same is not guaranteed for data-driven explanations such as answers to what-if questions, e.g., “What if concept $x_i^{\prime }$ is absent?” Root-to-leaf paths that do not condition on all of the binary interpretable features allow for more than one data point to be assigned to that leaf, e.g., for three binary features $[x_1^{\prime },x_2^{\prime },x_3^{\prime }]\in {\{0,1\}}^3$, a root-to-leaf path with a $x_1^{\prime }<0.5\wedge x_3^{\prime }<0.5$ condition assigns $[0,0,0]$ and $[0,1,0]$ to this leaf. This observation motivates the minimal interpretable representation $X_{\mathit{min},t}^{\prime }$ (Definition 1), which selects a single data point to represent each leaf thereby facilitating full fidelity of model-driven explanations without additional assumptions. However, for data-driven explanations to achieve full fidelity, the surrogate tree must faithfully model the entire interpretable feature space, i.e., have one leaf for every data point in ${\mathcal{X}}^{\prime }$, which can be thought of as extreme overfitting. Since the cardinality of a binary d-dimensional space ${\mathbb{B}}^d={\{0,1\}}^d$ is given by $|{\mathbb{B}}^d|=2^d$, and a complete and balanced binary decision tree of $2^d$ width (number of leaves) is d deep, relaxing the tree complexity bound $\mathrm{\Omega }$ accordingly guarantees full fidelity of all the explanations—a property captured by Corollary 1 (proof in Appendix B).

Corollary 1 

(Full Fidelity). If the complexity bound (width) Ω of a surrogate tree g is relaxed to equal the cardinality of the binary interpretable domain ${\mathcal{X}}^{\prime }$, i.e., $\mathrm{\Omega }(g;|{\mathcal{X}}^{\prime }\left|\right)={\displaystyle \frac{\mathit{width}\left(g\right)}{2^{|{\mathcal{X}}^{\prime }|}}}={\displaystyle \frac{2^{|{\mathcal{X}}^{\prime }|}}{2^{|{\mathcal{X}}^{\prime }|}}}=1$, then the surrogate is guaranteed to achieve full fidelity. This property applies to explanations that are both data-driven—i.e., derived from any data point in the interpretable representation—and model-driven—i.e., derived from the structure of the surrogate tree.

Therefore, a surrogate tree that guarantees faithfulness of model-driven explanations (Lemma 1) can only deliver trustworthy counterfactuals and exemplar explanations sourced from the minimal representation set. This may be an attractive alternative to more complex surrogate trees that additionally guarantee faithfulness of data-driven explanations (Corollary 1). The latter surrogate type, which usually yields deeper trees, can deliver a broader spectrum of trustworthy explanations: tree structure-based explanations, feature importance, decision rules (root-to-leaf paths), answers to what-if questions and exemplar explanations based on any data point, in addition to counterfactuals.

5. Qualitative, Quantitative and User-Based Evaluation

Next, we assess the explanatory power of LIME$\mathrm{TREE}$ with a multi-tier evaluation approach that consists of an assessment guided by XAI desiderata (Section 5.1) as well as functionally grounded (Section 5.2) and human-grounded (Section 5.3) experiments [20,39]. The first judges our approach against a number of criteria important for XAI systems; the second involves a (synthetic) proxy task in which we compare the (numerical) fidelity of LIME with multiple variants of LIME$\mathrm{TREE}$ on image and tabular data; the third reports results of a pilot user study, which is based on image classification to enable straightforward qualitative evaluation of explanations by means of visual inspection, thus alleviating the need for technical expertise.

5.1. Desiderata

XAI systems can generally be decomposed into two operationally distinct parts, one responsible for explanation generation and another for its presentation; this separation allows us to better identify, evaluate and report the unique desiderata important at each stage [40]. Given that LIME$\mathrm{TREE}$ is a surrogate explainer, the insights that it generates are post-hoc, therefore they may not reflect the true behavior of the underlying black box [1]. This discrepancy—empirically measured as fidelity—is an important indicator of explanation truthfulness, which should always be communicated to the explainees, especially in high stakes applications. While LIME$\mathrm{TREE}$ can achieve full fidelity without sacrificing explanation comprehensibility, this desideratum is limited to IRs that are deterministic. To take advantage of this property it is therefore important to design an IR that addresses the explainability needs of a particular use case, which may require additional effort to build such a bespoke module despite the explainer itself being model-agnostic [25,26,41]. More broadly, truthfulness is a major advantage of our approach given that it allows retrofitting explainability into pre-existing black boxes. Whatever explanation type, presentation format and communication medium are chosen, this property guarantees that the explanatory insights are based on an accurate reflection of the black-box model’s behavior.

Before reviewing the desiderata of specific explanation types, we discuss a set of general properties that are expected of all explanatory insights [20]. LIME$\mathrm{TREE}$ excels when it comes to explanation plurality and diversity—especially so given their consistency—allowing the explainees to explore distinct aspects of the underlying black box without running into spuriously contradictory observations, further improving the trustworthiness of its explanations. While some of them are inherently static, others can be operationalized within an interactive explanatory protocol [34], enabling the explainees to customize and personalize them in a natural way—refer to Section 6 for examples. This breadth of explanatory insights and access to their source—the surrogate tree structure (see Figure 2)—enables their contextualization, which makes them particularly appealing since good explanations do not only communicate what information is used by a predictive model but also how it is used [1].

By simultaneously accounting for multiple classes, LIME$\mathrm{TREE}$ offers a more comprehensive picture of the explained model’s predictive behavior and facilitates user-driven exploration, which, as noted in Section 1, can mitigate automation bias, especially so for counterfactuals [11]. Also, recall that our method is compatible with hypothesis-driven XAI since the breadth of its insights allows the explainees to consider multiple congruent explanations for different predictions of a given instance instead of only receiving a justification of the top prediction [12]. Given that our method operates as a surrogate, we can freely tweak and tune the target, breadth and scope of its explanations by adjusting its configuration, which further adds to its flexibility [20,24,25,26,34].

While LIME$\mathrm{TREE}$ offers a broad spectrum of explanation types—whose diversity makes it appealing to a wide range of audiences—we anticipate the counterfactuals to be the most attractive given their ubiquity in XAI [5]. Notably, these insights are ante-hoc with respect to the surrogate tree, therefore their truthfulness is guaranteed in this regard [42]. Their generation procedure allows to account for plausibility and actionability of their conditional part as well as other (human-centered) properties that may be desired [20,33,34,43]. Counterfactual explanations are known to be intrinsically comprehensible given their parsimony and low complexity, making them an attractive choice across a diverse range of applications [5,20].

5.2. Synthetic Experiments

We evaluate the trustworthiness and comprehensibility of LIME$\mathrm{TREE}$ explanations using the two components of the optimization objective $\mathcal{O}$ (Equation (1))—fidelity $\mathcal{L}$ and complexity $\mathrm{\Omega }$—as computational proxies. The former measures the faithfulness of the surrogate with respect to the black box, i.e., its ability to mimic the black box, which is the only metric capable of reporting the reliability of all the diverse explanation types extracted from the surrogate. To this end, we employ the formulations of fidelity used by both LIME (Equation (3)) and LIME$\mathrm{TREE}$ (Equation (5)); we compute this property when modeling the top three classes predicted by the black box for each test instance. We additionally analyze the complexity of LIME$\mathrm{TREE}$ surrogates calculated as the tree depth normalized by the dimensionality of the IR (Equation (4)); we then compare it with the corresponding measure for LIME surrogates, which is computed by counting the number of non-zero coefficients of the underlying linear models, i.e., their size (Equation (2)).

We study three variants of LIME$\mathrm{TREE}$, all of which minimize fidelity but differ in complexity constraints and post-processing:

TREE 

optimizes a surrogate tree for complexity, i.e., it determines the shallowest tree that offers the desired level of fidelity;

TREE

is a variant of TREE whose predictions are post-processed to guarantee full fidelity of model-driven explanations; and

TREE ^† 

constructs a surrogate tree without any complexity constraints, allowing the algorithm to build a complete tree that guarantees full fidelity of both model- and data-driven explanations.

These realizations of LIME$\mathrm{TREE}$ are the most relevant given that each one offers a surrogate with distinct fidelity characteristics that lead to certain types of tree-based explanations achieving desired properties as explained earlier in Section 4. We compare the fidelity of these explainers to LIME with disabled feature selection, which allows it to achieve maximum fidelity at the expense of explanation size. Our study is limited to fidelity and complexity since XAI lacks metrics suitable for multi-class explainability or for cases when multiple explanation types are derived from a single source as well as for explanations that rely on probabilities rather than crisp predictions (to mitigate automation bias) [44]. LIME is our only baseline given the general lack of multi-class explainers or methods whose underlying surrogate model can be directly accessed.

Table 1. Fidelity loss (mean ± standard deviation, smaller is better, best results in bold) computed: (n-th top) separately for each of the top three black-box predictions with the LIME loss (Equation (3)); and (top n) collectively for the top one, two and three black-box predictions with the LIME$\mathrm{TREE}$ loss (Equation (5)). We report results for four surrogates: LIME and three variants of LIME$\mathrm{TREE}$ (TREE, TREE and TREE^†). The percentage shown after the explainer name specifies the tree complexity $\mathrm{\Omega }$—i.e., its depth divided by its maximum possible depth determined by the number of features in the interpretable representation—at which loss is computed; TREE^† is equivalent to TREE@100% (and TREE@100% for deterministic IRs). We experiment with (top table) three image data sets, using various pretrained neural networks [50]: ImageNet [51] (1659 samples, 256 × 256 pixels, 1000 classes) + Inception v3 (77% accuracy); CIFAR-10 [52] (9714 samples, 32 × 32 pixels, 10 classes) + ResNet 56 (94% accuracy); and CIFAR-100 [52] (9665 samples, 32 × 32 pixels, 100 classes) + RepVGG (77% accuracy). We use all validation set images for which an interpretable representation can be built; however, for ImageNet we first preselect images that are square and at least 256 × 256, which we resize to these dimensions. These results are scaled up by 10². We also experiment with (bottom table) two tabular data sets, training the models ourselves with scikit-learn [53]: Wine [54] (36 samples, 13 features, 3 classes) + Logistic Regression (93% balanced accuracy); and Forest Covertypes [55] (2500 samples, 54 features, 7 classes) + Multi-layer Perceptron (86% balanced accuracy). For Wine, we use all the test set samples; given their small number, we repeated the study on the entire data set (178 samples) with comparable results. The Forest Covertypes test set has 116,203 samples, from which we draw a stratified subset of size 2500. These results are scaled up by 10¹. See Figure 4 for examples of the loss behavior.

$\times {\mathbf{10}}^{-\mathbf{2}}$		ImageNet + Inception v3				CIFAR-10 + ResNet 56				CIFAR-100 + RepVGG
		LIME	TREE@66%	TREE@75%	TREE†	LIME	TREE@66%	TREE@75%	TREE†	LIME	TREE@66%	TREE@75%	TREE†
n-th top	1st	3.67 ± 2.18	0.60 ± 0.61	0.64 ± 0.73	0 ± 0	7.34 ± 2.96	2.17 ± 1.25	2.77 ± 1.66	0 ± 0	3.33 ± 1.80	0.59 ± 0.56	0.66 ± 0.63	0 ± 0
	2nd	1.14 ± 1.77	0.24 ± 0.42	0.25 ± 0.40	0 ± 0	3.91 ± 3.98	1.28 ± 1.31	1.69 ± 1.76	0 ± 0	0.97 ± 1.46	0.24 ± 0.36	0.26 ± 0.40	0 ± 0
	3rd	0.63 ± 1.36	0.13 ± 0.25	0.16 ± 0.33	0 ± 0	2.57 ± 3.37	0.89 ± 1.15	1.10 ± 1.44	0 ± 0	0.56 ± 1.13	0.14 ± 0.29	0.16 ± 0.32	0 ± 0
top n	1	3.67 ± 2.18	0.60 ± 0.61	0.64 ± 0.73	0 ± 0	7.34 ± 2.96	2.17 ± 1.25	2.77 ± 1.66	0 ± 0	3.33 ± 1.80	0.59 ± 0.56	0.66 ± 0.63	0 ± 0
	2	2.41 ± 1.40	0.42 ± 0.42	0.44 ± 0.45	0 ± 0	5.63 ± 2.69	1.73 ± 1.03	2.23 ± 1.42	0 ± 0	2.15 ± 1.15	0.41 ± 0.36	0.46 ± 0.40	0 ± 0
	3	2.72 ± 1.58	0.48 ± 0.47	0.53 ± 0.50	0 ± 0	6.91 ± 3.26	2.17 ± 1.28	2.78 ± 1.73	0 ± 0	2.42 ± 1.29	0.48 ± 0.41	0.54 ± 0.45	0 ± 0
$\times {\mathbf{10}}^{-\mathbf{1}}$		Wine + Logistic Regression				Forest Covertypes + Multi-layer Perceptron
		LIME	TREE@66%	TREE@100%	TREE†	LIME	TREE@66%	TREE@100%	TREE†
n-th top	1st	0.29 ± 0.27	0.08 ± 0.11	5.54 ± 3.43	0.07 ± 0.11	0.59 ± 0.26	0.06 ± 0.06	4.56 ± 2.12	0.06 ± 0.06
	2nd	0.14 ± 0.16	0.03 ± 0.04	2.35 ± 3.26	0.03 ± 0.04	0.51 ± 0.29	0.05 ± 0.05	1.88 ± 1.21	0.05 ± 0.05
	3rd	0.20 ± 0.28	0.07 ± 0.12	3.73 ± 4.18	0.06 ± 0.11	0.13 ± 0.21	0.02 ± 0.04	0.57 ± 0.94	0.02 ± 0.04
top n	1	0.29 ± 0.27	0.08 ± 0.11	5.54 ± 3.43	0.07 ± 0.11	0.59 ± 0.26	0.06 ± 0.06	4.56 ± 2.12	0.06 ± 0.06
	2	0.22 ± 0.19	0.06 ± 0.07	3.94 ± 2.67	0.05 ± 0.07	0.55 ± 0.26	0.06 ± 0.05	3.22 ± 1.04	0.06 ± 0.05
	3	0.32 ± 0.29	0.09 ± 0.12	5.80 ± 3.56	0.08 ± 0.12	0.62 ± 0.29	0.07 ± 0.06	3.51 ± 1.09	0.07 ± 0.06

, which reports the results of our evaluation, also summarizes our experimental setup. We use a collection of popular multi-class image and tabular data sets; with the former we rely on a selection of pretrained neural networks, and with the latter we split the data into stratified 80% training and 20% test sets, and fit the models ourselves. LIME and LIME$\mathrm{TREE}$ are implemented following best practice described in the literature [24,25,26,45,46,47]. For images, we use an IR built upon SLIC (edge-based) segmentation [48] with black color occlusion as the information removal proxy; given its deterministic transformation function, we operate directly on the binary interpretable domain and generate its full set of instances instead of their random sample to enable the surrogate to reach full fidelity. For tabular data, we sample 10,000 instances around the explained data point in the original domain—using mixup, which is an explicitly local sampler that accounts for class labels [45,49]—since the corresponding IR transformation function is non-deterministic; we use quartile-based discretization applied to the data sample followed by binarization as our interpretable domain. For images we use cosine distance measured in the IR, and for tabular data we use Euclidean distance measured in the original domain; we use the exponential kernel for both, with its optimal parameter determined experimentally for each data set. Our code is available on GitHub at https://github.com/So-Cool/bLIMEy/tree/master/ELECTRONICS_2025 (accessed on 20 February 2025).

In our experiments, LIME produces three independent linear surrogates, one per class; each LIME$\mathrm{TREE}$ variant is either built as a single surrogate that models all of the classes simultaneously (n-th top), or a separate surrogate is constructed for a one-, two- and three-class problem (top n). In deployment, however, LIME$\mathrm{TREE}$ fits only a single multi-output tree, whereas LIME requires as many models as explained classes. As a result, since both methods follow the same steps except for the surrogate model training phase, our method tends to be faster for relatively small trees given that they are fitted to binary data with feature thresholds fixed at ${\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$—up to the depth of 20 in our experiments—and becomes negligibly slower for large trees—requiring 250 milliseconds more than LIME for trees as deep as 40—but these measures will fluctuate with the number of explained classes and the IR dimensionality. Since the number of interpretable features should be kept low to improve human comprehensibility of the explanations, which directly limits the surrogate tree depth, we expect LIME$\mathrm{TREE}$ to be faster in practice [25].

To assess explanation quality we measure multi-class fidelity with the LIME$\mathrm{TREE}$ loss as well as the fidelity of each class separately with the LIME loss. The experimental results, summarized in Table 1, show that our base method—TREE—provides more faithful explanations than LIME at ${\displaystyle \raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.}$ of its complexity for tabular and image data. TREE—which post-processes the surrogate tree to facilitate full fidelity of model-driven explanations when the IR transformation function is deterministic—also surpasses LIME at ${\displaystyle \raisebox{1ex}{$3$}\!\left/ \!\raisebox{-1ex}{$4$}\right.}$ of its complexity for image data given their compliant IR, but its performance is degraded for tabular data even at full tree complexity (100%) due to the stochasticity of the underlying IR. TREE requires higher complexity, i.e., deeper trees, than TREE to achieve comparable fidelity since the post-processing step makes the surrogate faithful with respect to the minimal interpretable data points but at the same time suboptimal for the remainder of the interpretable space, which is especially detrimental for stochastic IRs where each minimal interpretable data point corresponds to multiple instances in the original data domain.

The version of LIME$\mathrm{TREE}$ without a depth bound—TREE^†, which is equivalent to TREE@100% (and TREE@100% for deterministic interpretable representations)—achieves full fidelity across the board for a deterministic IR (images), where it faithfully models the entire interpretable data space by constructing one leaf per instance, but fails to do so for a non-deterministic IR (tabular) because in this case each tree leaf has to model multiple distinct data points. By allowing deeper trees we reduce the impurity of their leaves, which improves the overall performance of the surrogates—an intuitive relation, and trade-off, between the complexity of the trees and their fidelity, two representative examples of which are shown in Figure 4. Appendix C provides the complete collection of plots depicting the behavior of the LIME and LIME$\mathrm{TREE}$ loss for all the data sets used in our experiments.

5.3. Pilot User Study

To assess the real-life usefulness of our approach, we ran a pilot user study. We recruited eight participants (six males and two females) evenly distributed across the 18–45 age range with diverse skills and backgrounds; six of them had a machine learning background and three were familiar with AI explainability. The participants were not compensated for their involvement in the user study. We exposed them to LIME (Figure 1) and LIME$\mathrm{TREE}$ (Figure 2) explanations in a random order without revealing the method’s name. The study consisted of two sections, one per explainer, displaying an image split into three segments, with each part enclosing a unique object, e.g., a cat, a dog and a ball. The two most pertinent black-box predictions for each object were then explained with both methods—e.g., tabby and tiger cat for the cat object, golden retriever and Labrador retriever for the dog object, and tennis ball and croquet ball for the ball object—yielding six LIME explanations and a single multi-output tree spanning all six predictions. The participants were offered a brief tutorial illustrating how to parse the tree structure to obtain a variety of explanations.

The participants were then asked about the expected behavior of the black box in relation to any two out of the three displayed objects for each explainer—six questions in total as the relations are assumed to be non-reflexive. For example, “How does the presence of the cat object affect the model’s confidence of a presence of the dog object?”, with three possible answers: confidence decreases, confidence not affected and confidence increases. This question formulation was chosen to avoid a bias towards either explainer since we could neither ask for the importance or influence of each object on a particular prediction (LIME’s domain), nor the relation between an object and a prediction, e.g., a what-if question (LIME$\mathrm{TREE}$’s domain). Before viewing the explanations, the participants were asked to answer a similar set of questions using only their intuition, which allowed us to assess whether the explainees still relied on their intuition when explicitly asked to work with the explanations. Figure 5 provides a high-level overview of the flow of our user study.

Our findings indicate a negligible overlap between the responses based on the participants’ intuition and both explainers; they also show that LIME$\mathrm{TREE}$ helped the participants to answer 25% more of the questions correctly as compared with LIME. All of the participants indicated that using LIME was either easy or very easy, and at the same time rated the process of manually extracting LIME$\mathrm{TREE}$ explanations as either difficult or very difficult, despite many of the explainees having AI background. This disparity in conjunction with subpar performance when using LIME suggests that the explainees misinterpreted its explanations and were overconfident [56,57]; good performance when working with LIME$\mathrm{TREE}$ despite the difficulty in using its explanations, on the other hand, is promising given that the process of extracting them can be easily automated.

6. Discussion

LIME$\mathrm{TREE}$ explanations are versatile and appealing but achieving their full fidelity presupposes a deterministic IR transformation function (Lemma 1) and a complete surrogate tree (Corollary 1). This is not a problem for image and text data since the corresponding IRs can be built to be deterministic and of low dimensionality (given by the number of desired human-comprehensible concepts). The IR of tabular data, however, is inherently non-deterministic [25]—due to the many-to-one mapping introduced by discretization and binarization (refer back to Section 2)—with its dimensionality equal to the size of the original feature space. Nonetheless, since uniquely for tabular data the surrogate tree can be trained directly on their original representation, thus implicitly constructing a locally faithful and meaningful IR instead of relying on an external one [24,25], the surrogate can be overfitted to maximize its fidelity. While LIME$\mathrm{TREE}$ offers a close approximation in both cases, full fidelity cannot be guaranteed since even a complete surrogate tree is unable to achieve full coverage for non-deterministic IRs. The consequences of this shortcoming can be seen in our experimental results (refer to Table 1), which show that a complete surrogate tree— labeled TREE^†—can reach full fidelity for images but not for tabular data.

In practice, full fidelity of surrogates based on deterministic IRs (Lemma 1) is achieved by adjusting the sample size $|X^{\prime }|$ and relaxing the tree complexity bound $\mathrm{\Omega }$. Recall that a d-dimensional binary interpretable representation ${\mathcal{X}}^{\prime }\equiv {\mathbb{B}}^d={\{0,1\}}^d$ has $|{\mathcal{X}}^{\prime }|=2^d$ unique instances, and the width, i.e., the number of leaves, of a complete, balanced binary decision tree of depth d is $2^d$ (Corollary 1). Therefore, we can use all of these data points—there is no benefit from oversampling—to easily train a local surrogate with its complexity bound $\mathrm{\Omega }$ removed to allow complete trees of depth d, i.e., with one leaf per instance, guaranteeing full fidelity and access to a diverse range of faithful and comprehensible explanations. The depth bound and the sample size can be adjusted dynamically prior to training the surrogate to ensure its optimality since the size of the interpretable domain is known beforehand.

Since for images as well as text each dimension of the IR captures a human-comprehensible concept, their number is expected to be low, especially that tokens in text excerpts and segments in images do not have to be adjacent to constitute a single concept. For every additional feature in the interpretable space, the number of sampled data points doubles and the tree depth is incremented by one in order to provide the interpretable domain and the surrogate tree with enough capacity to preserve the full fidelity guarantee. While this exponential growth in the number of interpretable data points may seem overwhelming, training decision trees on binary data spaces is fast given the predetermined ${\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$ split at every node. The exponential growth of the width of the surrogate tree that guarantees its full fidelity increases its complexity and can have adverse effects on the comprehensibility of some explanation types, however, as we show next, it does not affect the most important and versatile explanation kinds.

Guaranteeing full fidelity of a surrogate tree requires relaxing its complexity bound $\mathrm{\Omega }$, which the optimization objective $\mathcal{O}$ tries to minimize (Equation (1)). Since in this setting a moderate number of interpretable features may yield a relatively large tree, the increased complexity of the resulting explanations is concerning. While a complex surrogate tree may render the explanations based on its structure, e.g., model visualizations, incomprehensible, these are not the most appealing explanation types and their appreciation often requires AI expertise. The (interpretable) feature importance, what-if explanations, counterfactuals and exemplars are not affected by the tree size in any way and remain highly compact and comprehensible—see Figure 6 for some examples and Appendix D for a more comprehensive overview of diverse explanation types. Notably, a complete surrogate tree with full fidelity will produce more counterfactual explanations for every data point, making it more interpretable.

The decision rules—logical conditions extracted from root-to-leaf paths—may indeed become overwhelmingly long, in fact as long as the tree depth, however this does not impact all the data types equally and an appropriate presentation medium can alleviate this issue regardless of the tree complexity. For image and text data, such rules will always be comprehensible, no matter their length, since they cannot have more literals than the dimensionality of the underlying interpretable domain, i.e., the number of segments for images and word-based tokens for text. Presenting this rule in the former case corresponds to displaying an image with its various segments occluded—e.g., see Figure 6d—and in the latter producing a text excerpt with selected tokens removed. For tabular data, however, these rules may become relatively long and incomprehensible since this domain lacks a similar human-friendly representation; the exception is root-to-leaf paths that impose multiple logical conditions on a single feature (in the original domain), allowing for their compression. Regardless of the presentation medium, a general criticism of rule-based explanations is the difficulty of understanding how each logical condition affects the prediction, making them less appealing than other explanation types.

In view of these observations, if explanations based on the structure of the surrogate tree are not required for image and text data, and additionally rule-based explanations are not needed for tabular data, the model complexity $\mathrm{\Omega }$ does not have to be minimized. It can therefore be removed from the optimization objective $\mathcal{O}$ given in Equation (1), paving the way for full explanation fidelity. LIME$\mathrm{TREE}$ therefore delivers a practical surrogate explainer with strong guarantees and well-understood limitations. Since it can be used with any AI model and data type—albeit with some constraints for tabular data—it promises to become an invaluable tool for inspecting, debugging and explaining black-box predictive systems.

7. Conclusions and Future Work

In this paper, we introduced the concept of multi-class explainability and proposed a surrogate explainer—called LIME$\mathrm{TREE}$—based on multi-output regression trees that is compatible with this paradigm. We then analyzed its various properties and guarantees and showed how it can achieve full fidelity. Next, we demonstrated how LIME$\mathrm{TREE}$ improves upon LIME and discussed the benefits of using trees as surrogate models. We supported these claims with an assessment of its properties based on XAI desiderata as well as a collection of quantitative experiments and a pilot user study. At a higher level, the multi-class explainability paradigm delivers more comprehensive insights into the functioning of opaque predictive models than are otherwise available with current XAI conceptualizations. Additionally, our implementation of this paradigm in the form of a surrogate explainer enables its straightforward adoption across many application domains given our tool’s compatibility with diverse data types as well as its clear guarantees and limitations. Together, our contributions advance both the conceptual and practical frontiers of explainable artificial intelligence.

In future work, we will implement methods to algorithmically extract human-centered explanations from (surrogate) trees—looking into the interactive aspects of this process—and evaluate them with large-scale user studies. We will also investigate alternative interpretable representations of tabular data that would allow LIME$\mathrm{TREE}$ to achieve better fidelity guarantees for this data domain (e.g., by providing a deterministic IR transformation function). Finally, we plan to explore other techniques capable of realizing the multi-class explainability paradigm as well as look into expanding this concept to better account for prediction uncertainty output by probabilistic classifiers—a perspective that is largely neglected by the XAI literature.