Explainable AI for Deep Visual Recognition: Evaluation, Methods, and Open Challenges

Abstract

Deep visual recognition has achieved remarkable success across various domains, including medical imaging, autonomous vehicles, and security systems. However, the black-box nature of deep learning models poses challenges in terms of transparency and trust, especially in critical applications where human understanding is essential. Explainable AI (XAI) seeks to address these concerns by providing human-interpretable explanations for model predictions. This review explores the key techniques for explainability in deep visual recognition, including model-agnostic methods such as LIME and SHAP, model-specific approaches like saliency maps and feature visualization, and intrinsically interpretable models like decision trees and rule-based systems. We also discuss the evaluation of explainability through metrics like fidelity, consistency, and stability, and explore the challenges of balancing model performance with interpretability. Furthermore, we examine applications of XAI in medical imaging, autonomous driving, security and surveillance, agriculture, satellite imagery and remote sensing, industrial inspection, and visual forensics, highlighting how domain-specific data and operational constraints affect the required form and validation of explanations. Finally, we address current research gaps and propose future directions for enhancing the robustness and human–AI interaction in explainable visual recognition systems. As AI continues to be integrated into safety-critical domains, the development of explainable, transparent, and trustworthy AI systems will be crucial for their widespread adoption and ethical use.

1. Introduction

The rapid advancement of deep learning techniques has revolutionized the field of computer vision, enabling machines to achieve unprecedented accuracy in tasks such as object detection, image classification, and facial recognition [1]. However, the growing complexity of deep learning models, especially deep neural networks, has raised significant concerns regarding their interpretability [2]. These models, often described as “black boxes,” are capable of producing high-performance results but offer little insight into how they arrive at their decisions [3]. This lack of transparency is particularly problematic in domains where understanding the reasoning behind model predictions is crucial, such as healthcare, autonomous driving, and law enforcement. Explainable AI (XAI) aims to bridge this gap by making AI systems more interpretable and providing human-readable explanations for their decisions [3,4,5].

In deep visual recognition, XAI is not limited to general model transparency; it is concerned with identifying the visual evidence that supports a prediction. Existing XAI approaches aim to make model behavior more understandable and interpretable for users [6,7], while also supporting trust, accountability, and fairness in AI-assisted decision-making [8]. In image- and video-based tasks, this means examining the features, patterns, and regions of interest that influence the model output [9]. Such explanations are especially important in safety-critical settings, where incorrect or poorly justified predictions may affect clinical diagnosis, autonomous navigation, or other high-risk decisions [10].

Deep visual recognition models, such as convolutional neural networks (CNNs), have achieved state-of-the-art performance in many visual tasks, but their internal reasoning often remains difficult to interpret [11]. In healthcare, where AI systems are increasingly used to support diagnosis and treatment decisions, explanations are needed to help clinicians examine whether model predictions are based on clinically relevant visual evidence [12]. This interpretability is important for user trust and for the practical adoption of AI-assisted decision-support systems [13]. Explainability can also support bias detection by revealing whether models rely on inappropriate or spurious cues that may lead to unfair outcomes across different user groups or deployment settings [14].

Although several broad surveys have reviewed explainability from a general machine-learning perspective, visual recognition presents distinctive explanation requirements that cannot be fully addressed by generic XAI taxonomies alone. Throughout this review, the term visual recognition is used as the preferred scope-specific term for the computer-vision tasks considered here, including image classification, object detection, segmentation, facial recognition, medical-image analysis, surveillance, agricultural monitoring, and visual forensics. The broader term computer vision is used only when referring to the wider research field, search terminology, or source titles. In computer vision, explanations are often expected to identify image regions, object parts, pixels, patches, concepts, prototypes, or attention pathways that directly support a prediction [11,15,16,17,18,19,20,21,22,23]. Therefore, this review gives particular attention to pivotal visual-XAI methods introduced in the computer-vision and machine-learning literature, including class activation mapping [15], gradient-based localization [11,16], perturbation-based visual explanations [18,24], concept-based explanations [19,25], prototype-based interpretable networks [20], and attention-based explanations for Vision Transformers [21,22,23]. This vision-oriented perspective is important because explanation quality in visual recognition depends not only on model transparency but also on spatial localization, object-boundary ambiguity, background bias, multi-object scenes, patch-token interactions, and the faithfulness of visual heatmaps to the model’s actual decision process [22,23,26].

This review aims to provide a comprehensive overview of XAI techniques for deep visual recognition, with particular emphasis on their methods, evaluation, limitations, and open challenges. The review also examines how explanation requirements differ across medical imaging, autonomous driving, security and surveillance, agriculture, satellite imagery and remote sensing, industrial inspection, retail recognition, and visual forensics, thereby avoiding a scope limited primarily to healthcare and transportation. We will explore the various techniques that have been proposed to explain the decisions of deep learning models in visual recognition tasks, including model-agnostic approaches like LIME and SHAP, as well as model-specific methods such as saliency maps and feature visualization. Additionally, we will delve into the challenges and limitations of current explainability techniques, particularly in terms of their applicability to complex real-world datasets and their trade-offs with model performance.

Despite significant progress in XAI, several research gaps remain, particularly in understanding the relationship between model performance, explanation faithfulness, and interpretability. One key challenge in deep visual recognition is to provide explanations for complex models without reducing predictive performance or producing misleading post hoc interpretations. Another challenge is the lack of universally accepted metrics for evaluating the quality of explanations. Furthermore, the human–AI interaction aspect of explainability has yet to be fully explored, particularly in terms of how users perceive and trust AI explanations. The objectives of this review are to: (1) evaluate existing XAI techniques in deep visual recognition, (2) identify the limitations of current methods, (3) compare explanation requirements and validation practices across a broad range of high-impact visual domains, including remote sensing and industrial inspection, (4) highlight open challenges in the field and (5) propose future research directions for improving explainability in deep learning-based visual-recognition systems.

2. Review Methodology and Literature Selection

This review was conducted using a structured literature-search and selection strategy to identify relevant studies on explainable artificial intelligence for deep visual recognition. The literature search considered major scientific databases, including IEEE Xplore, ACM Digital Library, ScienceDirect, SpringerLink, Web of Science, Scopus, and Google Scholar. The main review period focused on studies published between 2016 and 2025, while earlier foundational works and relevant domain-specific studies were also included when they introduced widely used XAI methods, benchmark datasets, or important contextual evidence. The main search terms included “explainable artificial intelligence,” “XAI,” “deep visual recognition,” “computer vision explainability,” “saliency maps,” “Grad-CAM,” “LIME,” “SHAP,” “visual explanation,” “interpretable deep learning,” “Vision Transformer explainability,” “medical image explainability,” “autonomous driving explainability,” “remote-sensing explainability,” “satellite-image explainability,” “GeoAI explainability,” “industrial visual inspection explainability,” “manufacturing defect explainability,” “explainable quality inspection,” and “human-centered XAI.”

The inclusion criteria were: (1) studies focused on explainability or interpretability in machine learning, deep learning, or computer vision; (2) papers proposing, evaluating, or applying XAI methods to visual-recognition tasks such as image classification, object detection, segmentation, medical imaging, autonomous driving, surveillance, agriculture, satellite and remote-sensing analysis, industrial inspection, manufacturing quality control, retail recognition, or visual forensics; (3) peer-reviewed journal or conference papers, along with highly relevant preprints when they introduced influential methods; and (4) studies written in English with sufficient methodological detail. The exclusion criteria were: (1) papers unrelated to visual recognition or explainability; (2) studies focused only on non-visual, tabular, or text-based tasks; (3) duplicate records, editorials, or short opinion pieces without technical discussion; and (4) works with insufficient information about the model, explanation method, or evaluation approach.

After screening the literature, the selected studies were organized into five principal groups: XAI methods, evaluation metrics, benchmark datasets, application domains, and open challenges. The application-domain group was further examined across medical imaging, autonomous driving, security and surveillance, agriculture, satellite imagery and remote sensing, industrial inspection and smart manufacturing, retail recognition, and visual forensics. The review emphasizes not only descriptive coverage but also critical comparison of methods in terms of explanation type, interpretability level, computational cost, faithfulness, stability, usability, and suitability for real-world visual-recognition applications. The overall literature-search and selection process, including database sources, keyword retrieval, initial screening, full-text eligibility assessment, exclusion criteria, included studies, and thematic organization, is summarized in Figure 1.

3. Fundamental Concepts of Deep Visual Recognition

Deep visual recognition refers to the use of deep learning algorithms, particularly deep neural networks (DNNs), to interpret and classify visual data such as images and videos. At the core of deep visual recognition is the convolutional neural network (CNN), a type of deep learning architecture that has proven to be highly effective in tasks such as image classification, object detection, and segmentation [27]. CNNs are designed to automatically detect and learn hierarchical features from raw input images through layers of convolutions, pooling, and activation functions [28]. These networks excel in capturing spatial patterns in data, making them ideal for visual tasks. A CNN typically comprises several layers: an input layer for receiving image data, convolutional layers that apply filters to detect local patterns (edges, textures, etc.), pooling layers that downsample feature maps to reduce dimensionality, and fully connected layers that perform classification based on learned features [29]. The success of CNNs in visual recognition tasks has led to their widespread use in various domains, including facial recognition, medical imaging, autonomous vehicles, and security systems. More advanced models, such as ResNet, DenseNet, and EfficientNet, have further improved performance by introducing techniques like residual connections, dense connections, and efficient layer designs to reduce training time and improve accuracy [30].

4. Challenges in Deep Learning Models for Visual Tasks

While deep learning models have revolutionized visual recognition tasks, several challenges remain that hinder their practical application and reliability [31]. One major challenge is the black-box nature of deep learning models. Despite their ability to achieve impressive performance, understanding how these models arrive at specific decisions remains difficult [14]. This lack of transparency makes it challenging to diagnose errors, identify biases, and trust model predictions in critical applications. For instance, a CNN used in medical imaging might misclassify a tumor as benign due to certain image features that are not immediately obvious to the human eye [32]. Another challenge is the data dependency of deep learning models. These models require vast amounts of labeled training data to generalize well to new, unseen data. In many real-world applications, however, obtaining large, high-quality labeled datasets is difficult and costly [33]. Additionally, deep learning models are often prone to overfitting, especially when the available training data is limited or unrepresentative of real-world conditions. This overfitting can lead to poor performance when the model is applied to new data, reducing its effectiveness in practical use [34,35].

Moreover, adversarial attacks are another concern in deep learning-based visual recognition systems. These attacks involve small, imperceptible perturbations to the input data that can cause the model to make incorrect predictions [36]. For example, slight changes to an image, undetectable to the human eye, can lead a model to misclassify objects with high confidence. This vulnerability raises significant concerns about the robustness and security of deep learning systems in high-stakes applications [37,38].

5. Role of Explainability in AI Models

In visual recognition, explainability contributes not only to general model transparency but also to the operational assessment of model behavior. Explanations can be used to audit whether a prediction is supported by spatially meaningful image evidence, to diagnose shortcut learning or background bias, and to compare how different explanation families expose model behavior [11]. Beyond improving user confidence, explanations should help determine whether the highlighted evidence is relevant to the task, consistent with expert expectations, and useful for model debugging or decision review [39]. In clinically oriented visual tasks, for example, the value of an explanation depends on whether it points to meaningful anatomical or pathological evidence rather than irrelevant image artifacts [12]. Similarly, in fairness-sensitive visual-recognition settings, explanations may help reveal whether predictions rely on inappropriate or spurious cues, although such findings should be interpreted together with quantitative bias and performance evaluation [14]. Because visual-recognition models make decisions from spatial, semantic, and sometimes multimodal evidence, a generic division into model-agnostic, model-specific, and intrinsically interpretable approaches is not sufficient. Figure 2 therefore organizes visual-XAI methods according to the kind of evidence they expose, ranging from low-level attribution and localization to higher-level concepts, prototypes, transformer attention, and multimodal explanations. This organization clarifies which explanation families are most suitable for different visual-recognition tasks and evaluation needs.

Within this visual-evidence taxonomy, different XAI techniques serve different practical roles. Post hoc visual explanations include model-specific heatmaps, such as saliency maps, and model-agnostic local surrogate approaches, such as LIME, which can help identify the image regions or features that influence a prediction [40,41]. However, their usefulness in visual recognition depends on the explanation unit being used, such as pixels, superpixels, patches, regions, or extracted visual features, and on whether the explanation remains stable and meaningful for the target task [42]. The demand for trustworthy visual-recognition systems therefore requires not only understandable explanations, but also careful comparison of explanation methods in terms of localization quality, faithfulness, usability, and computational cost [11,43]. Because the field is still evolving, these techniques should be selected according to the application context and evaluated with respect to both interpretability and reliability rather than visual appeal alone [43,44].

Table 1. Critical Comparison of Model-Agnostic XAI Techniques for Deep Visual Recognition.

Technique	Visual Explanation Unit/Output	Model Access Requirement	Main Strengths in Visual Recognition	Key Limitations and Assumptions	Computational Cost	Robustness to Adversarial Perturbations	Scalability to Large Datasets	Most Suitable Use Cases	References
LIME	Superpixels or image regions with local importance weights	Prediction-output access only	Intuitive local explanation; useful when the model architecture is unavailable	Sensitive to superpixel segmentation, perturbation sampling, masking baseline, and surrogate-model choice	High: approximately Q black-box model evaluations for Q perturbations per image, followed by fitting a local surrogate model	Low: small input changes can alter superpixel boundaries, sampled neighborhoods, and surrogate coefficients	Moderate: cost rises rapidly with image resolution, number of superpixels, and dataset size	Explaining individual image-classification predictions and local error cases	[24,47,48]
SHAP	Pixels, superpixels, patches, or feature-map attributions	Prediction-output access; approximations often required	Theoretically grounded additive attribution; can support local and global interpretation	Exact computation is infeasible for high-dimensional images; sensitive to feature grouping, background baseline, and correlated visual regions	Very high: exact model-agnostic Shapley computation may require evaluation of up to 2M feature coalitions; approximate variants use Q sampled coalitions	Low–moderate: attributions can change under adversarial perturbations, baseline manipulation, or correlated-feature effects	Low–moderate: direct raw-pixel application is impractical without grouping or specialized approximations	Local/global attribution analysis of CNN-based visual-recognition models	[45,46]
PDP	Average response curves for selected image-derived or engineered features	Repeated model-query access	Simple global view of how selected features affect predictions	Can be misleading when visual features are correlated; weak for raw-pixel explanations	Moderate: approximately n × g model predictions for one examined feature, where n is the number of observations and g is the number of grid points	Low: adversarial, unrealistic, or off-distribution feature perturbations can distort the estimated partial-dependence relationships, particularly when visual features are correlated	High: for a small set of engineered features; poor for dense pixel or feature-map representations	Global analysis of interpretable image-derived features or metadata-assisted vision tasks	[49]
ICE	Individual response curves for selected image-derived or engineered features	Repeated model-query access	Shows sample-level variation hidden by PDP	Can become cluttered for many samples; affected by unrealistic feature perturbations and feature correlation	Moderate–high: approximately n × g model predictions and n individual response curves for one examined feature	Low: individual curves may be unstable under small or adversarial feature changes	Moderate: with subsampling; visualization and computation become difficult for large datasets	Studying heterogeneous model behavior across image-derived feature values	[50]
Anchors	If–then rules based on visual attributes, superpixels, or hybrid features	Prediction-output access only	Produces high-precision rule-based local explanations that are easy to inspect	Coverage may be limited; less suitable for dense raw-pixel reasoning without meaningful visual attributes	Moderate–high: requires an adaptive, data-dependent number of model queries (Q) during rule search and precision estimation	Moderate: local rules may be stable within a narrow region but can fail when adversarial changes invalidate anchor conditions	Moderate: for structured or attribute-based inputs; weak for dense pixel-level representations	Hybrid vision-plus-tabular settings or attribute-based visual-recognition tasks	[51]

Note: Computational cost is expressed as an approximate analytical estimate based primarily on the number of black-box model evaluations required by each method. Here, Q denotes the number of perturbations or adaptive queries, M the number of interpretable features or image regions, n the number of observations, and g the number of evaluated grid points. Robustness and scalability are reported comparatively because published results vary across models, datasets, perturbation settings, image resolutions, and hardware platforms.

critically compares major model-agnostic XAI techniques in terms of their visual explanation outputs, model-access requirements, strengths, limitations and assumptions, computational cost, robustness to adversarial perturbations, scalability to large datasets, and suitability for visual-recognition applications. These practical considerations are particularly important for high-resolution imagery and large-scale visual-recognition datasets, where the computational requirements of LIME and SHAP increase with the number of perturbations, model queries, or feature coalitions. Exact Shapley-value computation is combinatorial and infeasible for high-dimensional images; consequently, practical SHAP implementations depend on approximation strategies, feature grouping, and baseline selection [45,46]. LIME similarly perturbs image superpixels and fits a local surrogate model, making its computational cost and explanation variability sensitive to segmentation complexity, perturbation sampling, and dataset scale [24,47,48]. PDP and ICE are more computationally practical when applied to a limited number of engineered or semantically meaningful features, but they are poorly suited to dense pixel-level or feature-map representations [49,50]. Anchors can produce concise rule-based explanations, although adaptive sampling and rule-search procedures may still require numerous model queries [51]. Moreover, none of these techniques provides an inherent guarantee of robustness to adversarial or prediction-preserving perturbations, which may alter superpixel boundaries, sampled neighborhoods, attribution values, conditional-response curves, or extracted rules [52,53]. Therefore, selecting an appropriate method requires balancing explanation faithfulness, computational resources, perturbation sensitivity, feature dimensionality, and dataset size.

6. Methods for Explainable AI in Visual Recognition

Before discussing XAI methods under the broad categories of model-agnostic, model-specific, and intrinsically interpretable approaches, it is useful to distinguish the major families of methods that have shaped visual explainability research. Early visual explanation work focused on sensitivity maps and gradient-based attribution, where the importance of pixels is estimated from the gradient of the output score with respect to the input image [54,55]. Class Activation Mapping (CAM) introduced a spatially meaningful way of localizing discriminative image regions in CNNs by using global average pooling and class-specific weights [15]. Grad-CAM generalized this idea by using gradients flowing into the final convolutional layer to generate class-discriminative localization maps without requiring architectural changes [11]. Later extensions, including Grad-CAM++ [16], Score-CAM [17], and SmoothGrad [55], attempted to improve object localization, reduce noise, or avoid direct dependence on raw gradients.

Perturbation-based visual explanations form another important family. Instead of relying only on internal gradients, these methods modify or mask image regions and observe changes in the model output. LIME explains image predictions by perturbing superpixels and fitting a local surrogate model [24], whereas RISE estimates saliency by randomly masking the input and measuring the effect of each region on the prediction score [18]. These methods are useful for black-box models, but their explanations may depend strongly on segmentation quality, masking strategy, sampling density, and computational cost [18,24].

Beyond pixel- and region-level heatmaps, concept-based and prototype-based methods aim to produce explanations closer to human reasoning. TCAV measures the influence of user-defined concepts on model predictions by using concept activation vectors in the internal representation space [19]. Concept bottleneck models make predictions through intermediate human-interpretable concepts, enabling users to inspect and sometimes intervene on concept predictions [25]. Prototype-based networks, such as ProtoPNet, explain image classification through similarity between image parts and learned prototypical parts [20]. These approaches are especially relevant in fine-grained recognition, medical imaging, and other domains where users often reason through visual patterns, attributes, or representative examples [19,20,25].

More recently, Vision Transformers have created new challenges for visual explanation. Unlike CNNs, which preserve local spatial structure through convolutional feature maps, Vision Transformers process images as patch tokens and mix information through multi-head self-attention. As a result, directly applying CNN-based methods such as Grad-CAM may not fully capture how token interactions contribute to the final prediction [21,22,23]. Attention Rollout and Attention Flow attempt to trace how information is propagated through attention layers [21], while transformer-specific relevance propagation methods integrate attention weights, gradients, and residual connections to estimate class-specific token relevance [22]. Generic attention-model explainability further extends transformer interpretation to bi-modal and encoder–decoder transformer architectures, which is important for modern vision-language and multimodal recognition systems [23].

Unlike CNN-based explanation methods, transformer explanations require different interpretation mechanisms because the decision process is based on token-to-token interactions rather than spatial convolutional feature maps. Vision Transformers divide an image into patch tokens and model long-range dependencies through multi-head self-attention, whereas Grad-CAM and related CNN-based methods typically use the gradient of a class score with respect to convolutional feature maps to identify discriminative spatial regions [56]. Therefore, raw attention maps should not be directly interpreted as faithful explanations, because attention weights indicate token interaction strength but may not fully represent class-specific decision relevance [57]. Recent ViT-specific explanation methods address this limitation by considering information flow, patch-token relevance, causal effects of patch embeddings, and token-transformation effects rather than relying only on raw attention weights [21,22,58,59]. This distinction shows that explaining attention mechanisms in Vision Transformers is fundamentally different from applying CNN-gradient-based methods such as Grad-CAM.

7. XAI Techniques for Vision Transformers

XAI techniques for Vision Transformers can be broadly grouped into attention-propagation methods, relevance-propagation methods, perturbation-based token analysis, and causal or token-transformation-based approaches. Attention Rollout and Attention Flow aggregate attention matrices across transformer layers to estimate how information propagates from input patch tokens to the classification token [21]. Transformer-specific relevance propagation extends this idea by combining attention weights, gradients, and residual connections to compute class-specific token relevance rather than relying only on raw attention maps [22,23]. Perturbation-based ViT explanations evaluate the contribution of individual image patches by masking, removing, or modifying patch tokens and observing changes in the prediction score. More recent methods further examine causal effects of patch embeddings and token-transformation processes to improve the faithfulness of ViT explanations [57,58,59]. These approaches are particularly important because Vision Transformers do not preserve image evidence in the same way as CNN feature maps; therefore, ViT explanations should be evaluated at the patch-token level using faithfulness, stability, perturbation, and localization-based tests.

Following the vision-specific taxonomy presented in Figure 2, the subsequent discussion groups XAI methods according to their practical role in visual recognition while retaining broader categories such as model-agnostic, model-specific, and intrinsically interpretable methods for organizational clarity. In this review, these broad labels are not treated as the complete taxonomy; instead, the main emphasis is placed on the type of visual evidence each method provides, such as pixel attribution, region-level localization, perturbation-based saliency, concept-level reasoning, prototype-based comparison, transformer attention pathways, and multimodal or text-based explanations.

To illustrate why explanation evaluation is necessary, Figure 3 compares representative visual-XAI outputs for the same prediction. The comparison shows that different methods may highlight different regions or produce explanations with different levels of smoothness, localization, and spatial specificity. Therefore, visual appeal alone is insufficient; explanations should also be assessed for faithfulness, stability, and task relevance. These variations arise from the distinct mathematical mechanisms used by each method to assign importance to visual evidence. Grad-CAM computes channel weights by spatially averaging the gradients of the target-class score with respect to the final convolutional feature maps and then combines these weighted feature maps to produce a class-discriminative localization map [11]. Because the final convolutional layer usually has a relatively coarse spatial resolution, Grad-CAM often highlights broad object-level regions rather than precise object boundaries. Its dependence on gradients can also make the resulting explanation sensitive to weak, noisy, or saturated gradient signals [11,17]. Grad-CAM++ extends this approach by assigning spatially varying weights to the gradients using higher-order derivative information, which can improve localization when multiple instances of the same class are present or when the relevant evidence occupies a relatively small region. Nevertheless, it remains dependent on gradient quality [16]. Score-CAM avoids direct gradient weighting by using activation maps to mask the input image and measuring the resulting change in the target-class score. Because it does not use gradients to determine channel importance, Score-CAM avoids the gradient-saturation problem associated with gradient-based weighting; however, it requires repeated forward passes and remains dependent on the selected convolutional layer, activation-map resolution, and masking procedure [17].

In contrast, RISE and LIME estimate importance through input perturbation rather than convolutional gradients. RISE generates randomized binary masks and estimates the importance of each image location from the expected prediction score when that location remains visible. Its saliency map is therefore a Monte Carlo estimate, and an insufficient number of masks, unsuitable mask resolution, or stochastic sampling variability may produce noisy or spatially diffuse explanations. Increasing the number of masks can improve convergence but substantially increases the number of model queries [18]. LIME first divides the image into superpixels, randomly activates or deactivates these regions, and fits a sparse local surrogate model around the selected image [24]. Its explanation is consequently constrained by superpixel boundaries and may vary with the segmentation algorithm, number of superpixels, masking baseline, locality kernel, and perturbation sample [24,48]. Therefore, disagreement among the methods is not merely a visualization artifact; it reflects their different attribution mechanisms, including gradient-based weighting, activation-based contribution, perturbation-based influence, and local surrogate coefficients. These differences affect spatial resolution, stability, computational cost, and faithfulness, thereby motivating multi-method and multi-metric evaluation rather than selecting an explanation solely because it appears visually smoother or more intuitive.

Table 2. Summary of Intrinsically Interpretable Models in Visual Recognition.

Model Type	Core Idea	Interpretability	Typical Trade-Off vs. Standard Deep Models	References
Decision trees	Predict through human-readable split rules	High	Very interpretable, but usually weaker on complex image recognition tasks	[60]
Rule-based models	Use explicit if–then rules for classification	High	Easy to inspect, but limited expressiveness for high-dimensional visual data	[61]
Prototype-based networks	Classify by comparing image parts to learned prototypes	High	More interpretable than CNNs, often with some loss in flexibility or peak accuracy	[62]
Prototype trees/neural trees	Combine prototypes with tree-like decision paths	High	Better balance of interpretability and accuracy than plain trees, but still simpler than full black-box models	[63]
Self-interpretable neural networks	Build explanations directly into network structure	Medium–High	More faithful explanations than post hoc methods, but often require architectural constraints	[63]
Traditional deep CNNs	Learn hierarchical visual features end-to-end	Low	Usually strongest raw accuracy, but poor transparency without extra XAI methods	[64]

summarizes representative intrinsically interpretable models used in visual recognition and highlights their core ideas, interpretability levels, and typical trade-offs compared with standard deep learning models.

8. Model-Agnostic Techniques

Model-agnostic explanation techniques are post hoc approaches designed to explain the predictions of a trained machine learning model without depending on its internal architecture [65]. These methods generate explanations without requiring access to the model’s internal workings, making them particularly useful for black-box models like deep neural networks (DNNs) [65]. Two widely used model-agnostic methods in visual recognition are LIME and SHAP [66].

• LIME (Local Interpretable Model-agnostic Explanations)

LIME is a local post hoc explanation method that approximates the behavior of a complex prediction model around a specific input by fitting a simpler interpretable surrogate model within that input’s local neighborhood [24]. Formally, the local explanation generated by LIME can be written as Equation (1).

$$\mathsf{\xi }(\mathrm{x})={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{g}\in \mathrm{G}}\mathrm{L}(\mathrm{f},\mathrm{g},{\mathsf{\pi }}_{\mathrm{x}})+\mathsf{\Omega }\left(\mathrm{g}\right)$$

(1)

where f denotes the original black-box model, g is an interpretable surrogate model selected from the class of interpretable models G, π_x is a locality-weighting function that assigns higher weights to perturbed samples closer to the original input x, L(f, g, π_x) measures how well the surrogate model approximates the black-box model in the local neighborhood of x, and Ω(g) penalizes the complexity of the explanation.

In visual recognition, LIME is usually implemented by segmenting an image into superpixels, perturbing these regions, and fitting a local surrogate model to estimate which regions support the prediction. This makes LIME useful for explaining individual image-level decisions, especially when the original model is inaccessible or difficult to inspect [48]. However, its explanation quality depends strongly on the segmentation algorithm, perturbation strategy, masking baseline, number of sampled perturbations, and choice of local surrogate model. As a result, two visually similar images, or even two different segmentations of the same image, may produce noticeably different explanations. Therefore, LIME explanations in visual recognition should be interpreted as approximate local evidence rather than definitive proof of the model’s true reasoning [67].

• SHAP (Shapley Additive Explanations)

SHAP values, based on cooperative game theory, provide a principled attribution framework for explaining model predictions [68]. The central idea is to assign each feature a Shapley value, which represents its contribution to the prediction. Formally, for a model f and an input represented by a feature set N = {1, …, M}, the Shapley value of feature i can be written as Equation (2).

φ_i(f, x) = ∑_{S ⊆ N∖{i}} [|S|!(M − |S| − 1)!/M!] [f_x(S ∪ {i}) − f_x(S)]

(2)

where S denotes a subset of features excluding feature i, M is the total number of features, and f_x(S) represents the model output when only the feature subset S is observed or retained. In visual recognition, these features may correspond to pixels, superpixels, patches, or image regions. This formulation estimates the average marginal contribution of each visual feature over all possible feature coalitions, which provides a principled explanation framework but also makes exact SHAP computation expensive for high-dimensional image data.

In visual recognition, SHAP explanations are commonly approximated over pixels, superpixels, patches, or intermediate feature maps rather than computed exactly over all raw pixels, because exhaustive Shapley-value estimation is computationally infeasible for high-dimensional images [46]. This gives SHAP a stronger theoretical foundation than many heuristic attribution methods, but its practical reliability depends on how visual features are grouped, how missing regions are replaced, and whether feature dependence is handled appropriately [45]. For CNN-based visual recognition, SHAP-style methods can provide useful local or global attribution patterns, but they may become computationally expensive and can be sensitive to background artifacts, correlated visual regions, and the choice of image baseline [69]. Therefore, SHAP should be used as a principled attribution framework whose outputs still require task-specific validation through faithfulness, stability, and localization-based evaluation.

9. Model-Specific Techniques

Unlike model-agnostic methods, model-specific techniques are tailored to specific types of models and exploit their internal structures to generate explanations [70]. In deep visual recognition, several model-specific techniques have been developed to provide insights into how a neural network processes visual data [16].

• Feature Visualization

Feature visualization aims to understand what a neural network has learned by visualizing the features that are activated during the model’s forward pass. In deep visual recognition, feature visualization typically involves generating images that maximize the activation of a particular filter or layer in a convolutional neural network (CNN) [71]. This can reveal the kinds of patterns the network is sensitive to, such as edges, textures, or object parts. For example, by visualizing the first convolutional layer, one might observe low-level features like edges, while deeper layers may reveal more abstract patterns such as shapes or specific objects [72]. Feature visualization is useful for understanding the internal representations learned by the model and can help identify whether the model is focusing on relevant image features [73].

• Saliency Maps

Saliency maps are another widely used method for visualizing the importance of specific image regions in a model’s decision-making process [74]. These maps highlight areas of an image that most influence the model’s output, helping users understand which parts of the image are contributing to the model’s predictions [26]. The most common approach to generating saliency maps is to compute the gradient of the model’s output with respect to the input image, which indicates how sensitive the model is to changes in each pixel. A high gradient value indicates that the model’s prediction is highly sensitive to that pixel, suggesting it plays an important role in the decision [75]. While saliency maps are easy to compute and visually intuitive, they can sometimes highlight noisy or irrelevant features, making them less reliable in some cases [26].

• Activation Maximization

Activation maximization is a technique used to visualize the inputs that most activate a specific neuron or filter in a neural network [76]. The idea is to generate an image that maximally activates a given neuron by performing gradient ascent on the input image with respect to the neuron’s activation [77]. This process results in images that emphasize the features that the network has learned to detect. For visual recognition tasks, activation maximization can be used to gain insights into what specific neurons in the network are detecting [78]. While activation maximization provides highly interpretable results, the generated images may not always correspond to realistic or meaningful visual patterns, particularly for deeper layers of the network [79].

10. Intrinsically Interpretable Models

Intrinsically interpretable models are designed with interpretability built into the architecture, offering explanations without the need for additional post-processing methods like saliency maps or LIME. These models are inherently simpler than deep neural networks and can offer greater transparency in their decision-making processes [80,81].

• Decision Trees

Decision trees are a type of machine learning model that makes decisions by recursively splitting the feature space based on simple thresholds [82]. Each internal node represents a decision based on a feature, and the leaf nodes represent the final prediction. Decision trees are highly interpretable because their decision-making process can be easily visualized as a flowchart [83]. In visual recognition tasks, decision trees can be used to classify images based on high-level features. However, decision trees often struggle with complex data and may require further optimization, such as ensemble methods like random forests, to improve accuracy [84].

• Rule-Based Systems

Rule-based systems are another class of interpretable models that use a set of human-readable if-then rules to make predictions. In visual recognition, these systems could use rules like “If the object is red and round, classify it as an apple.” Rule-based systems are transparent by design, making them easy to understand and audit [85]. However, they are generally less effective for complex tasks like deep visual recognition, where the relationships between features are highly nonlinear and difficult to express with simple rules [86].

• Transparent Neural Networks

Transparent neural networks refer to simplified neural network architectures designed to be more interpretable than traditional deep networks [87]. These networks aim to balance performance with transparency by limiting the complexity of the model while maintaining sufficient accuracy for the task [88]. For instance, a transparent neural network might reduce the number of layers or neurons, use simpler activation functions, or employ attention mechanisms to highlight important features. While these models are more interpretable, they often face a trade-off with performance, as more complex models tend to yield higher accuracy for tasks like visual recognition [89].

Although intrinsically interpretable models provide built-in transparency, they often face limitations in complex visual-recognition tasks. Simple models such as decision trees and rule-based systems may not capture high-dimensional spatial patterns, fine-grained textures, occlusion, and background variation [64,83,86]. More advanced interpretable models, such as concept bottleneck and prototype-based networks, can improve transparency but may require concept annotations, prototype validation, or architectural constraints [20,25]. Therefore, intrinsically interpretable models are preferable when transparency, auditability, or expert intervention is required from the beginning, whereas post hoc methods are more practical when high-performing black-box models are already available or concept-level annotations are limited.

11. Evaluation of Explainability in Deep Visual Recognition

The evaluation of explainability in deep visual recognition models is essential for understanding how and why a model produces a particular prediction. Although many XAI techniques can generate visual explanations, their usefulness depends on whether the explanations are faithful, stable, consistent, and relevant to the task. Effective evaluation ensures that explanations are not only visually interpretable but also reliable and meaningful for model analysis [90]. In this section, we discuss key metrics for evaluating explainability in visual recognition models, commonly used benchmark datasets, and case studies demonstrating the application of explainable AI (XAI) in object detection and classification. Figure 4 summarizes the main evaluation metrics used to assess explanation quality, including fidelity, consistency, and stability.

12. Distinguishing Faithfulness and Interpretability

Although the terms faithfulness and interpretability are sometimes used interchangeably, they describe different properties of an explanation. Faithfulness refers to the extent to which an explanation accurately represents the evidence, features, or internal dependencies that the underlying model actually uses to generate a prediction. Interpretability, in contrast, refers to the ease with which an intended human user can understand and make use of the explanation. An explanation may therefore be highly interpretable yet weakly faithful if it presents a simple and plausible rationale that does not reflect the model’s true decision process. Conversely, a highly faithful explanation may remain difficult for non-experts to understand because it exposes complex gradients, feature interactions, or distributed representations. Consequently, faithfulness and interpretability should be evaluated independently and treated as complementary rather than interchangeable properties of visual-XAI methods [91,92].

Table 3. Evaluation Metrics for XAI in Visual Recognition.

Metric	What It Assesses	Why It Matters in Visual Recognition	References
Faithfulness/Fidelity-based evaluation	Whether the explanation truly reflects the model’s decision	Ensures highlighted regions are actually used by the model	[93]
Stability/Sensitivity	Whether explanations remain similar under small perturbations	Important for robustness and trust in similar images	[94]
Consistency	Whether explanations agree across similar inputs or models	Supports reliability and reproducibility	[95]
Insertion/Deletion	Effect of adding or removing salient regions on prediction score	Tests causal importance of highlighted areas	[96]
Localization/Pointing Game	Whether the explanation overlaps with the correct object region	Useful for object recognition and weakly supervised localization	[97]
Sanity Checks	Whether explanations depend on learned model parameters	Prevents misleading but visually plausible explanations	[26]

summarizes the principal evaluation metrics used to assess explainable AI methods in visual recognition, including faithfulness, stability, consistency, insertion/deletion, localization, and sanity checks.

Therefore, evaluation studies should assess both faithfulness and interpretability rather than assuming that an improvement in one property automatically implies an improvement in the other.

13. Metrics for Evaluating Explainability

Evaluating the effectiveness of explanations generated by XAI techniques is challenging. Several metrics are proposed to assess the quality of explanations, focusing on different aspects of their performance [90]. The most widely used metrics for evaluating explainability include fidelity, consistency, and stability [98]. These metrics help ensure that the explanations accurately reflect the model’s behavior, are consistent across similar predictions, and are robust to perturbations.

In addition to automatic metrics, user-centered evaluation is essential for assessing whether explanations are understandable, useful, and trustworthy for real users. Metrics such as fidelity, consistency, and stability can measure technical properties of explanations, but they do not fully capture whether clinicians, engineers, domain experts, or general users can correctly interpret and act on those explanations. User studies can evaluate explanation clarity, perceived trust, decision support, cognitive load, task performance, and agreement between model explanations and expert reasoning [42,99,100,101]. Therefore, a comprehensive evaluation of XAI in visual recognition should combine automatic explanation-quality metrics with human-centered assessment, especially in high-stakes applications such as medical imaging, autonomous driving, surveillance, and agricultural decision support.

Evaluation metrics for XAI are complementary rather than interchangeable, and each metric involves mathematical and practical trade-offs [102,103]. Faithfulness refers to whether an explanation reflects the model’s actual decision process, whereas fidelity-based tests provide operational or quantitative measures of this alignment. However, a highly faithful explanation may still be visually noisy or difficult for humans to interpret [103,104]. Stability evaluates whether explanations remain similar under small input perturbations, but excessive emphasis on stability may penalize explanations that correctly change when a semantically important image region is modified [105]. Localization-based metrics, such as pointing-game or overlap-based evaluation, are useful when object masks or bounding boxes are available, but they may reward explanations that overlap with the object region even if those regions are not truly causal for the model’s prediction [103,106]. Insertion and deletion metrics provide a more causal form of evaluation by measuring how the prediction score changes when salient regions are added or removed; however, they are sensitive to perturbation strategy, baseline choice, masking artifacts, and computational cost [104,106]. Therefore, XAI methods in visual recognition should not be ranked using a single metric alone. A more reliable evaluation should combine fidelity, localization, stability, robustness, computational efficiency, and user-centered usefulness according to the target task and deployment context [102,103].

• Fidelity as a Measure of Faithfulness

Fidelity provides an operational measure of how closely an explanation aligns with the model’s actual decision process [107]. It is operationalized through quantitative tests that examine whether the features or regions identified by the explanation correspond to evidence that actually influences the model’s output. An explanation is considered high in fidelity if it accurately reflects the regions or features in the input that are responsible for the model’s prediction [108]. For example, a saliency map would provide stronger evidence of fidelity if removing or masking its highlighted regions substantially changed the model’s target prediction; overlap with a tumor annotation alone primarily indicates localization or plausibility. Fidelity is crucial because if an explanation does not align with the model’s true behavior, it could mislead users into trusting incorrect information [109]. Techniques like LIME and SHAP are often evaluated based on their fidelity to the model’s predictions [110].

Table 4. Common procedures for evaluating faithfulness and localization in visual XAI.

Evaluation Procedure	How It Is Quantified	Interpretation	Main Limitation
Deletion test	Progressively remove or mask the most salient pixels, superpixels, patches, or regions and measure the decrease in the target-class score.	A faster decrease indicates that the explanation identified regions important to the model prediction.	Sensitive to masking strategy, baseline choice, and out-of-distribution artifacts.
Insertion test	Progressively add the most salient regions back to a blurred or baseline image and measure the increase in the target-class score.	A faster increase indicates that the highlighted regions contain strong predictive evidence.	Depends on baseline image construction and region insertion order.
Deletion/insertion curves	Plot prediction-score changes as salient regions are removed or inserted; summarize using area-under-the-curve values.	Provides a quantitative comparison between explanation methods.	Curves may vary depending on perturbation design and image preprocessing.
Pointing game	Check whether the most salient point falls inside the ground-truth object bounding box or segmentation mask.	Measures whether the explanation points to the correct object region.	Measures localization agreement, not necessarily causal importance.
Localization overlap	Compare saliency maps with bounding boxes, masks, or annotated regions using overlap-based scores.	Useful when ground-truth object or lesion annotations are available.	Requires annotations and may reward spatial overlap even when the region is not causally used by the model.

summarizes common procedures used to evaluate faithfulness and localization in visual XAI, including deletion/insertion tests, deletion/insertion curves, pointing-game evaluation, and localization-overlap measures.

• Consistency

Consistency measures the reliability of explanations when the model is exposed to similar inputs [52]. If a model makes similar predictions for similar inputs, the explanations should also be similar. For instance, if a model correctly identifies a car in an image and explains its decision by highlighting the car’s wheels, consistency would require that a similar explanation be provided when another car image is presented, assuming the same visual features are involved [111]. Consistency ensures that the explanation method is stable and not overly sensitive to slight variations in the input data. It is especially important for applications where model behavior should remain predictable and reliable, such as in autonomous vehicles or medical diagnostics [112].

• Stability

Stability evaluates how the explanations change when the input data is slightly modified or perturbed. A stable explanation should not vary drastically with minor changes in the input, as this could indicate that the explanation is unreliable or based on noise [113]. For instance, if a saliency map for an object detection task highlights specific regions of an image as important, those highlighted regions should remain largely consistent even if the image is rotated or slightly altered [75]. Stability is essential for ensuring that explanations are robust and not overly sensitive to irrelevant factors that might not significantly impact the model’s decision [114].

14. Benchmark Datasets for Visual Recognition Models

Benchmark datasets play a vital role in evaluating explainable AI techniques for deep visual recognition. These datasets provide standardized tasks for model evaluation and help compare the effectiveness of different XAI methods [115]. Common benchmark datasets in visual recognition include large-scale image classification datasets, object detection datasets, and specialized datasets for more domain-specific applications like medical imaging [116].

• ImageNet

ImageNet is one of the most widely used datasets for evaluating image classification models [117]. It consists of over 14 million labeled images across 21,000 categories, with 1000 object classes commonly used for benchmark evaluations [118]. Researchers often use ImageNet to assess the performance and explainability of models like CNNs. For example, explainable AI techniques can be applied to identify which parts of an image (e.g., texture, color, or shapes) contribute to the model’s decision, helping to understand the model’s interpretability in a complex multi-class classification setting [119].

• COCO (Common Objects in Context)

The COCO dataset is widely used for object detection, segmentation, and captioning tasks. It contains over 330,000 images with 80 object categories, each annotated with segmentation masks, keypoints, and object boundaries [120]. COCO is particularly valuable for evaluating XAI techniques in object detection, where explanations need to highlight regions or bounding boxes that the model uses for identifying objects. XAI methods can be evaluated on COCO by measuring whether generated saliency maps overlap with human-annotated object regions, for example, using localization, pointing-game, and deletion/insertion-based evaluation metrics [18].

• ADE20K

ADE20K is another popular dataset for semantic segmentation tasks, where the goal is to classify each pixel in an image into one of 150 object categories [121]. The dataset provides pixel-level annotations across a wide variety of scenes, from indoor rooms to outdoor landscapes [121]. This dataset is useful for evaluating explainability techniques that aim to identify the most important regions for pixel-wise decisions. Techniques like feature visualization and saliency maps can be tested on this dataset to highlight the relevant parts of the image that influence segmentation decisions [122].

• ISIC (International Skin Imaging Collaboration)

The ISIC dataset focuses on skin cancer classification, containing a large set of labeled dermoscopic images used for training and evaluating machine learning models in medical imaging. It is especially useful for evaluating explainable AI methods in a domain where interpretability is crucial for medical professionals [123]. Techniques such as saliency maps or heatmaps can be used to highlight the regions of a skin lesion that contribute to the model’s diagnosis, assisting in the validation of AI-driven tools for clinical use [124].

15. Illustrative Application Examples of XAI in Visual Recognition

The following examples are presented as illustrative application cases rather than a direct empirical benchmark. They show how XAI is used across representative visual-recognition settings and highlight the task type, commonly used model families, explanation methods, and evaluation concerns. In medical imaging, XAI is mainly used to verify whether diagnostic models focus on clinically meaningful regions [125,126,127]. In autonomous driving, explanations help assess whether perception models attend to relevant objects and scene cues [128,129]. In retail/product recognition, XAI supports error analysis by identifying product regions or visual attributes that influence category predictions [130,131]. In satellite imagery and remote sensing, XAI supports interpretable geospatial decision-making across spatial and spectral data [18,132]. In industrial inspection and smart manufacturing, XAI supports the transparent and accountable deployment of automated quality-control systems [133,134]. These examples also show that explanation quality should be assessed not only visually but also through fidelity, localization, robustness, and domain-expert validation [90,93,94,97,99].

• Example 1: XAI in Medical Imaging (Tumor Detection)

In a medical imaging context, XAI techniques like saliency maps and SHAP have been applied to improve the interpretability of deep learning models used in tumor detection from medical scans (e.g., X-rays and MRIs) [135]. These models can detect subtle patterns in medical images that may be invisible to the human eye [135]. However, doctors need to trust the model’s decisions and understand the reasoning behind them. Saliency maps, for example, can highlight the exact regions of a medical image (such as a tumor in an MRI scan) that the model considered when making its prediction [136]. These visual explanations allow healthcare professionals to verify the model’s predictions and ensure that the AI system does not overlook critical details.

• Example 2: XAI in Autonomous Vehicles (Pedestrian Detection)

Autonomous vehicles rely on deep visual recognition models for real-time pedestrian detection. These models must make quick, accurate decisions to avoid accidents [137]. Explainable AI techniques, such as saliency maps and LIME, have been applied to highlight the features that contribute to pedestrian detection, such as the shape, size, and location of pedestrians in the environment [138]. These explanations help engineers and safety regulators verify that the model is focusing on the right visual cues and not making decisions based on irrelevant or misleading features (e.g., the background or road markings). This transparency is essential for ensuring the safety and reliability of autonomous vehicles in diverse driving conditions [139].

• Example 3: XAI in Retail (Product Categorization)

In retail, deep visual recognition models are used for automatic product categorization in e-commerce platforms [130,131]. XAI methods like feature visualization and saliency maps can be applied to understand why the model categorizes a product as, for example, “electronics” rather than “home appliances.” Feature visualization can reveal which parts of the product image (e.g., buttons, screens, or logos) contributed most to the model’s decision [130,131]. This transparency allows retailers to identify potential errors in the model, such as misclassifications due to ambiguous product features, and improve the overall accuracy and trustworthiness of their systems [131].

• Example 4: XAI in Satellite Imagery and Remote Sensing (Land-Cover and Disaster Mapping)

Satellite and aerial-image models are used for land-cover classification, disaster mapping, deforestation monitoring, crop assessment, and urban-change detection. XAI methods can help analysts identify the spatial or spectral evidence contributing to these predictions and determine whether the model is focusing on geographically meaningful regions [18,132].

• Example 5: XAI in Industrial Inspection and Smart Manufacturing (Visual Defect Detection)

Industrial visual inspection models are used to detect defects such as scratches, cracks, missing components, soldering faults, and surface irregularities. XAI methods can help engineers localize the evidence supporting a defect prediction and compare it with expected defect patterns, thereby supporting inspection review and model debugging [133,134].

16. Open Challenges and Research Directions

Explainable AI (XAI) is a rapidly evolving field, particularly in the domain of deep visual recognition [11]. While significant advancements have been made, several open challenges remain that hinder the widespread adoption and effectiveness of XAI techniques [9]. These challenges include the trade-offs between explainability and model performance, the lack of ground truth for evaluating explanations, the role of human–AI interaction, and ensuring the robustness and trustworthiness of the explanations provided by AI systems. Addressing these issues is critical for advancing the field and making deep visual recognition models more interpretable and reliable in real-world applications [42]. Figure 5 presents a nuanced view of the relationship between predictive performance and explainability in visual recognition. Although classical discussions often describe interpretability and accuracy as competing objectives, this relationship is not always strictly inverse. Post hoc XAI methods can explain high-performing black-box models after training, while concept-based and prototype-based models aim to integrate interpretability into the model design itself. Therefore, the performance–explainability relationship should be evaluated with respect to the task, model architecture, explanation faithfulness, and end-user requirements.

This relationship can be analyzed through the specific constraints imposed by different interpretable model families. First, capacity constraints reduce the number or complexity of functions that a model can represent. Shallow decision trees, sparse rule lists, and simplified transparent networks may therefore fail to capture the high-dimensional textures, shape variations, occlusions, and long-range dependencies available to deep CNNs and Vision Transformers. Second, sparsity and monotonicity constraints restrict the number, sign, or form of feature interactions so that the decision can be inspected, but these restrictions can exclude predictive interactions that improve peak accuracy. Third, concept-bottleneck models force the prediction through a limited set of human-defined concepts. Accuracy can decrease when the concept vocabulary is incomplete, concept labels are noisy or expensive to obtain, or the concept predictor discards residual information that is useful for the final classification decision [25].

Fourth, prototype-based networks constrain decisions to similarity with a finite set of learned prototypes or prototypical image parts. This supports case-based reasoning, but accuracy may decrease when the prototype set does not represent intra-class variability, when prototypes capture non-causal textures, or when projection to real training patches limits optimization flexibility [20,140]. Fifth, spatial or part-alignment constraints require the model to place evidence in coherent or semantically meaningful regions. Such constraints improve auditability but may suppress distributed evidence or contextual cues that an unconstrained classifier can exploit. Sixth, discretization and rule-extraction constraints convert continuous representations into thresholds, symbolic rules, or short decision paths and can introduce approximation error. These constraints explain why an intrinsically interpretable architecture may not always match the peak accuracy of an unconstrained model trained with the same data and backbone capacity.

These constraints do not imply that interpretable models must always be less accurate. Concept, prototype, sparsity, or localization supervision may act as useful regularization when the imposed structure matches the target domain. The relevant comparison is therefore constrained versus unconstrained representation under the same data, optimization budget, and evaluation protocol. Accordingly, the performance–explainability landscape is better represented by overlapping regions than by a strict inverse curve: post hoc methods can preserve the performance of a black-box model but may provide explanations with uncertain faithfulness, whereas intrinsically interpretable models can provide more direct reasoning at the possible cost of representational freedom, additional annotation, or peak accuracy [80].

The major open challenges in explainable AI for deep visual recognition are summarized in

Table 5. Open challenges in explainable AI for deep visual recognition.

Open Challenge	Brief Note	References
Performance–explainability balance	Accuracy and interpretability are task-, model-, and evaluation-dependent rather than strictly inverse.	[42]
No ground truth for explanations	Most visual explanations lack a clear gold standard.	[141]
Inconsistent evaluation metrics	Different metrics often rank XAI methods differently.	[142]
Human–AI interaction complexity	Different users need different types of explanations.	[99]
Limited human-centered validation	Many studies evaluate explanations algorithmically, not with users.	[100]
Bias and spurious correlations	Models may rely on shortcuts that explanations do not fully reveal.	[143]
Lack of standard benchmarks	Results are hard to compare across datasets and tasks.	[90,141,142]
Deployment and usability issues	Explanations may be too slow or too technical for practice.	[144]
Vision- and transformer-specific explanation challenges	Visual explanations must handle localization, object boundaries, background bias, multi-object scenes, and patch-token interactions in Vision Transformers.	[21,22,23]

, including the performance–explainability balance, lack of ground truth for explanations, inconsistent evaluation metrics, limited human-centered validation, deployment issues, and vision- and transformer-specific explanation challenges.

17. Vision-Specific and Transformer-Specific Explanation Challenges

Open challenges in visual XAI differ from generic explainability challenges because image-based explanations must account for spatial localization, object boundaries, multi-object scenes, occlusion, background bias, and task-specific evidence. In classification, an explanation may highlight only the most discriminative region rather than the complete object. In detection and segmentation, explanations should ideally align with bounding boxes, masks, or object regions, but this becomes difficult when objects are small, overlapping, partially occluded, or weakly annotated. Therefore, visual explanations should be evaluated not only for visual plausibility but also for localization quality, faithfulness, robustness, and consistency across similar inputs [90,93,142].

Vision Transformers introduce additional explanation challenges because image evidence is represented as patch tokens rather than convolutional feature maps. Raw attention weights should not automatically be interpreted as faithful explanations of model decisions, since token mixing, residual connections, and multi-head attention can make the decision pathway difficult to trace. Methods such as Attention Rollout, Attention Flow, and transformer-specific relevance propagation attempt to estimate how information flows through attention layers, but these explanations still require careful validation using perturbation tests, sanity checks, and task-specific evaluation protocols [21,22,23].

18. Trade-Offs Between Explainability and Model Performance

Rather than treating predictive performance and explainability as universally opposing objectives, comparisons should account for task complexity, dataset characteristics, model architecture, explanation method, and evaluation protocol. Concept-based and prototype-based approaches demonstrate that interpretability can be incorporated into model design while maintaining competitive performance in some visual-recognition tasks [20,25]. Post hoc methods such as LIME, SHAP, Grad-CAM, Score-CAM, and RISE can explain high-performing black-box models without modifying the original architecture [11,17,18,24,46], although their faithfulness, stability, and localization accuracy may vary across datasets and tasks [90,93,142]. Conversely, intrinsically interpretable approaches can provide more transparent reasoning but may require concept annotations, prototype constraints, or architectural modifications [20,25,63]. Future studies should therefore compare predictive performance and explanation quality under matched experimental conditions rather than rank models along a single accuracy–interpretability axis.

19. Addressing the Lack of Ground Truth for Evaluating Explanations

A significant challenge in the evaluation of XAI techniques is the lack of ground truth for explanations. Unlike traditional machine learning models, where evaluation metrics like accuracy or F1-score can be directly computed based on labeled data, the evaluation of explanations is subjective and lacks a universally accepted standard [7]. The concept of “ground truth” in the context of explainability is problematic because there is no definitive or objective way to measure whether an explanation is accurate, as human interpretations of model decisions can vary [90,92]. For instance, when explaining an image classification model, saliency maps might highlight different regions of the image for different users, depending on their understanding and perspective. This subjectivity makes it difficult to benchmark explainability techniques across models and datasets [90,141,142].

To address this challenge, researchers have proposed using human evaluation in the form of user studies, where participants are asked to assess the quality and usefulness of explanations. However, these studies are often time-consuming, limited in scale, and may not provide a comprehensive evaluation [92]. Furthermore, even when human evaluations are used, they are prone to biases and inconsistencies, which can make it difficult to draw generalizable conclusions. An alternative approach is to develop automatic evaluation metrics that can quantify the quality of explanations in a more objective manner [92]. These metrics could measure aspects such as fidelity (how well the explanation matches the model’s true behavior), consistency (whether the explanation is stable across similar inputs), and stability (how much the explanation changes with minor perturbations to the input). However, these metrics also face challenges due to the lack of a clear, universally accepted definition of what constitutes a “good” explanation [90]. Developing more reliable and objective methods for evaluating explainability is an ongoing area of research in XAI.

20. Human–AI Interaction in Explainability

Human–AI interaction plays a crucial role in the success of explainable AI, as the ultimate goal of XAI is to help users understand and trust AI decisions [99,101]. In many real-world applications, such as healthcare and autonomous vehicles, the users (e.g., doctors, engineers, or drivers) need to interact with AI systems to make informed decisions based on the model’s output [99]. For AI systems to be adopted in these domains, the explanations provided by the model must be understandable, actionable, and aligned with human reasoning. A major challenge in this area is the complexity of the explanations. While many XAI techniques can generate high-fidelity explanations, they may not always be presented in a form that is easily interpretable by non-experts [145]. For example, a saliency map that highlights regions of an image may be useful for a visual-recognition expert, but it may not be as clear to a medical professional who needs to make a clinical decision based on the model’s output [146]. To improve human–AI interaction, future research should focus on creating user-centric explanations that are tailored to the needs and cognitive abilities of different user groups. This also highlights the importance of user training and educational support, since users with different technical backgrounds may face different levels of difficulty in understanding computational concepts and model-based decision support [147]. This involves not only providing clear visual explanations but also considering factors such as the user’s level of expertise, the context in which the model is being used, and the decision-making process [99]. Furthermore, human-in-the-loop systems, where users can interact with and refine the explanations provided by the model, could enhance the effectiveness of XAI techniques by allowing users to correct, modify, or validate the model’s decisions [148].

Another aspect of human–AI interaction is trust and acceptance. The effectiveness of an explanation depends not only on its clarity but also on the user’s trust in the AI system [149]. Research has shown that users are more likely to trust AI systems that provide transparent and understandable explanations [149]. Therefore, the design of XAI systems must incorporate elements that foster trust, such as providing explanations that are consistent with human expectations and addressing potential biases in the model’s predictions [7].

21. Robustness and Trustworthiness of Explanations

A critical challenge in XAI is ensuring the robustness and trustworthiness of the explanations provided by AI models [150]. Explanations must not only be consistent and interpretable but also resistant to manipulation and errors, particularly in adversarial settings. Deep learning models are known to be vulnerable to adversarial attacks, where small, imperceptible changes to the input data can lead to drastically incorrect predictions [53]. Similarly, explanations themselves can be subject to adversarial manipulation, where attackers attempt to modify the explanation to make the model appear more trustworthy than it actually is. For example, an adversarial attack might change an image slightly, causing the model to misclassify the object [151]. A saliency map for the perturbed image could still highlight the same regions of the image, even though the model’s decision is incorrect. Ensuring that explanations are resilient to such attacks is crucial for maintaining the trustworthiness of AI systems, especially in critical applications where adversarial manipulation could have severe consequences [152].

Importantly, explanation robustness should be evaluated separately from prediction robustness. In some cases, an adversarial or manipulated input may preserve the predicted class while substantially changing the saliency map or forcing the explanation to highlight irrelevant regions. Such explanation-targeted attacks are problematic because they can make an unreliable model appear trustworthy to users [53,151,152]. Therefore, visual XAI methods should be tested using perturbation analysis, insertion/deletion evaluation, sanity checks, and stability assessment under transformations such as noise, blur, compression, cropping, occlusion, and adversarial perturbations [26,90,93,96,113,142]. These tests help determine whether an explanation reflects the model’s true decision process or only provides a visually plausible but fragile justification.

In addition to adversarial robustness, the trustworthiness of explanations depends on the reliability of the XAI techniques themselves. If an explanation method produces inconsistent or misleading explanations, it could undermine the user’s confidence in the AI system [139]. For instance, explanations that only highlight irrelevant features or fail to capture the model’s true decision-making process can lead to misinterpretation and errors [153]. To address this issue, ongoing research focuses on developing robust explanation methods that are less susceptible to adversarial attacks and produce trustworthy insights into the model’s decision-making process.

22. Applications of Explainable AI in Visual Recognition

XAI requirements vary substantially across visual-recognition domains because the relevant evidence, failure modes, validation criteria, and end users differ. Medical imaging may require clinically meaningful and volumetrically consistent localization, autonomous driving requires temporal and perception-to-action explanations, remote sensing requires spatial and spectral validation, and industrial inspection requires precise defect localization under operational constraints. Security, surveillance, agriculture, biometrics, and visual forensics introduce additional concerns related to bias, shortcut learning, background artifacts, and domain-specific reliability. Accordingly, XAI methods should be evaluated not only for visual clarity but also for faithfulness, localization accuracy, robustness, domain relevance, and usefulness to the intended expert user [127,128,129,132,133,134,143,154,155].

Table 6. Use Cases for Explainable AI in Visual Recognition Applications.

Application Area	Typical Visual Task	Why XAI Is Used	References
Medical imaging	Disease classification, lesion localization, image segmentation	To highlight image regions supporting diagnosis and improve clinical trust	[125,127,156,157]
Autonomous driving	Scene understanding, perception, planning	To explain what the vehicle detected and why a driving action was taken	[128,129]
Agricultural monitoring	Plant disease recognition from leaf images	To show symptom regions and support farmer-facing decision tools	[154]
Biometrics/face recognition	Identity verification and facial analysis	To improve transparency, audit bias, and support accountable deployment	[143,155]
Deepfake/visual forensics	Fake-image or fake-face detection	To help users understand forensic cues and compare explanation quality	[158]
Satellite imagery and remote sensing	Land-cover classification, change detection, disaster mapping, crop and environmental monitoring	To verify spatial and spectral evidence and detect reliance on clouds, sensor artifacts, or spatial leakage	[18,132]
Industrial inspection and smart manufacturing	Surface-defect detection, PCB inspection, component verification, and quality control	To confirm that predictions rely on true defects and to support operator review and root-cause analysis	[133,134]

summarizes representative use cases of explainable AI in major visual-recognition application domains, including medical imaging, autonomous driving, agricultural monitoring, satellite imagery and remote sensing, industrial inspection and smart manufacturing, biometrics/face recognition, and deepfake or visual forensics.

23. Medical Imaging

In medical imaging, XAI methods can succeed when they highlight clinically meaningful regions that influence diagnostic predictions, such as lesions, tumors, or abnormal tissue patterns [125,127]. However, this domain also exposes important limitations of common visual-explanation methods. Many saliency and CAM-based approaches were originally developed for 2D natural images, where the model prediction is usually explained using evidence from a single image plane. In contrast, MRI and CT data are often volumetric, and clinically meaningful evidence may depend on inter-slice anatomical continuity, lesion volume, anatomical context, contrast variation, or multi-sequence information. A 2D saliency map applied slice by slice may therefore highlight locally discriminative pixels while missing the broader 3D structure that supports diagnosis. In addition, saliency maps may focus on scanner artifacts, image borders, acquisition markers, or dataset-specific shortcuts rather than pathology-related regions [153]. Recent medical-imaging studies have also shown that visually plausible saliency maps may fail to reliably localize clinically relevant abnormalities when compared with expert annotations [156,157]. Therefore, XAI in medical imaging should be evaluated using not only visual plausibility but also lesion-level localization, volumetric consistency, robustness across scanners and acquisition protocols, and expert validation.

Image-based explanations should also be interpreted as one component of broader clinical reasoning rather than as a complete causal account of a patient’s condition. Relevant clinical outcomes may depend on physiological, behavioral, and lifestyle variables that are not visible in medical images. For example, an observational study of patients with ischemic stroke reported an association between daily garlic intake and endothelial function [159]. Although this study is not an investigation of XAI, it illustrates why image-derived explanations should be considered alongside clinical history, lifestyle factors, laboratory findings, and other non-imaging evidence when evaluating AI-assisted medical decisions.

24. Autonomous Vehicles

In autonomous driving, XAI can succeed when explanations identify decision-relevant objects and scene cues, such as pedestrians, vehicles, traffic lights, lane markings, road boundaries, and obstacles [126,128,129]. However, autonomous-driving scenes expose limitations that are not fully captured by standard image-level heatmaps. A heatmap may highlight a pedestrian or traffic sign, but it may not explain whether the model understood object distance, motion direction, occlusion, interaction with nearby vehicles, weather-related visibility, or the reason for a braking or steering decision. This is especially important because driving decisions are temporal and action-oriented rather than based only on a single static image. Frame-level saliency maps may therefore fail when the relevant evidence depends on object trajectories, sudden scene changes, multi-object interactions, or the connection between perception and planning. Recent autonomous-driving XAI studies have therefore emphasized object-level, temporal, and multimodal explanation strategies, including planning transformers that identify decision-relevant objects and language-based driving models that connect object-level visual information with interpretable textual explanations [160,161]. A dedicated review of deep vision-based autonomous-driving systems further shows that explanation requirements extend beyond static object localization to the interpretation of learned driving policies, temporal context, scene interactions, and the relationship between perceived evidence and vehicle actions [126]. Therefore, XAI for autonomous vehicles should be evaluated not only by whether it highlights visually relevant objects, but also by temporal consistency, robustness under adverse conditions, usefulness for failure analysis, and ability to explain perception-to-action decisions.

25. Security and Surveillance

In security and surveillance, XAI can succeed when it helps identify visual cues used for face recognition, anomaly detection, violence detection, and forensic analysis. For example, explanations may show whether a face-recognition model relies on facial regions such as the eyes, nose, mouth, or periocular area, or whether a forensic model focuses on manipulation-related artifacts. However, this domain also shows why visually convincing explanations can be misleading. A saliency map may highlight plausible facial regions, but this does not prove that the model is fair, robust, or using identity-relevant evidence. The model may still rely on demographic shortcuts, illumination patterns, camera-specific artifacts, compression traces, background cues, or correlations that are not obvious from the explanation. Prior studies on facial-analysis systems have shown that demographic performance disparities and insufficient auditing can affect fairness and accountability [162,163]. In addition, explainable face-verification studies show that saliency-based explanations require careful interpretation because face-recognition decisions depend on similarity learning, identity-specific cues, and robustness across realistic conditions [164]. For example, a face-recognition model may highlight facial regions, but this does not ensure robustness across lighting, pose, occlusion, or demographic groups [155]. Therefore, XAI in surveillance should be combined with demographic bias auditing, cross-camera validation, privacy-aware deployment, and human-centered evaluation rather than being used only to increase apparent trust.

26. Agricultural and Environmental Monitoring

In agriculture and environmental monitoring, XAI can succeed when it highlights visual, spatial, or spectral evidence that corresponds to plant disease symptoms, crop stress, pest damage, or environmental change [72,122,154]. However, this domain also exposes important limitations of standard heatmap-based explanations. In plant disease detection, saliency maps may appear to highlight diseased leaf regions, but they may also focus on soil, shadows, camera angle, lighting conditions, leaf boundaries, or background texture instead of disease-specific symptoms. This problem is more serious in field images than in controlled laboratory images because field data often contain class imbalance, overlapping leaves, uncontrolled illumination, seasonal variation, and heterogeneous backgrounds. In hyperspectral and remote-sensing tasks, explanation is even more complex because model decisions may depend on spectral bands, spatial resolution, mixed pixels, sensor calibration, and landscape context rather than only visible image regions [132,165,166]. Therefore, XAI in agriculture and environmental monitoring should be evaluated using disease-region annotations where available, external field datasets, spectral or spatial relevance checks, and robustness tests against background, sensor, and acquisition changes. Recent plant-disease studies using explainable deep learning, such as potato leaf disease classification with Grad-CAM-based interpretation, further show that agricultural XAI should connect high classification performance with biologically meaningful symptom localization rather than relying only on image-level accuracy [167].

27. XAI in Satellite Imagery and Remote Sensing

Satellite imagery and remote-sensing analysis pose distinctive explainability challenges because predictions are often made over large geographic regions and may be influenced by spatial context, image tiling, and geographic leakage between training and test data. XAI can help analysts determine whether a model relies on meaningful structures—such as vegetation patterns, built-up areas, or disaster-affected regions—rather than on cloud cover, sensor artifacts, or location-specific shortcuts. For multispectral and hyperspectral imagery, spatial explanations may also be complemented by band-level attribution to identify which wavelength ranges contribute to the prediction [132].

Validation should therefore use geospatially appropriate procedures rather than rely only on visually plausible heatmaps. Explanation maps can be compared with land-cover polygons, disaster masks, or expert-annotated geographic features, while stability should be tested across seasons, atmospheric conditions, sensor platforms, and spatial resolutions. Spatially separated training and test sets are also important for assessing whether explanations remain reliable outside the locations used for model development. These requirements distinguish explainable GeoAI from conventional natural-image explanation and justify its treatment as a separate high-impact application domain [132].

28. XAI in Industrial Inspection and Smart Manufacturing

Industrial visual inspection poses distinctive explainability challenges because quality-control decisions often depend on subtle visual evidence that must be localized and interpreted reliably. In these settings, a high classification score is insufficient: engineers and operators need to verify that the model is responding to the actual defect rather than to background texture, lighting, fixture position, production-batch artifacts, or camera-specific cues. Region-based methods such as Grad-CAM and Score-CAM can support rapid operator review by localizing discriminative areas, while prototype- and concept-based methods can express decisions through representative defect patterns or predefined quality concepts. Explainability can also support model debugging, root-cause analysis, regulatory documentation, and human acceptance of automated quality-control systems [133,134].

Industrial inspection creates demanding operational constraints for XAI. Defects may be small, low-contrast, or rare; false negatives can be costly; and production systems often require real-time or edge-device inference. Explanation methods should therefore be evaluated not only for visual clarity but also for fine-grained localization, faithfulness, latency, memory use, and stability under illumination changes, vibration, camera replacement, and domain shift between production batches. Perturbation methods requiring thousands of forward passes may be unsuitable for online inspection, whereas coarse heatmaps may be too imprecise for micro-defect verification. A practical industrial-XAI system should combine efficient localization with quantitative defect overlap, perturbation-based faithfulness testing, and operator-centered validation [133,134].

29. Conclusions

This review has explored the growing field of Explainable AI (XAI) in deep visual recognition, emphasizing its importance in enhancing the transparency and trustworthiness of AI systems. We discussed various methods for achieving explainability, including model-agnostic techniques like LIME and SHAP, model-specific approaches such as saliency maps and feature visualization, and intrinsically interpretable models like decision trees and rule-based systems. As XAI is increasingly deployed in high-impact visual-recognition settings—including medical imaging, autonomous driving, security and surveillance, agriculture, satellite and remote-sensing analysis, industrial inspection and smart manufacturing, retail recognition, and visual forensics—future systems must combine predictive performance with faithful explanations, domain-appropriate validation, robustness, and effective human oversight. The evaluation of explainability through metrics such as fidelity, consistency, and stability was also covered, illustrating the need for robust standards in assessing the quality of explanations.

Despite the progress made in XAI, several challenges remain that need to be addressed. One key issue is the complex relationship between model performance and explainability, which depends on the model architecture, task complexity, explanation method, and evaluation protocol. The lack of ground truth for evaluating explanations is another significant obstacle, as traditional evaluation metrics used for model performance do not easily apply to the subjective nature of explainability. Furthermore, the role of human–AI interaction remains underexplored, particularly in tailoring explanations to meet the needs of different users and fostering trust in AI systems. Ensuring the robustness and trustworthiness of explanations is also critical, especially in environments where adversarial attacks or model misbehavior can have serious consequences.

Looking ahead, the future prospects for Explainable AI in visual recognition are promising. As AI systems become more integrated into real-world applications, the demand for interpretable models will continue to grow. Research will likely focus on developing techniques that strike a better balance between explainability and performance, ensuring that models remain both accurate and understandable. Additionally, there is potential for the creation of more standardized evaluation frameworks, allowing for the objective assessment of XAI methods across various domains. With continued advancements in human–AI interaction, XAI will increasingly focus on providing explanations that are not only accurate but also actionable and user-centric. The development of robust, transparent, and trustworthy AI systems will ultimately foster greater adoption of AI technologies in safety-critical areas, paving the way for a future where explainability is an integral part of every AI model.