A Review of Explainable Artificial Intelligence from the Perspectives of Challenges and Opportunities

Abstract

The widespread adoption of Artificial Intelligence (AI) in critical domains, such as healthcare, finance, law, and autonomous systems, has brought unprecedented societal benefits. Its black-box (sub-symbolic) nature allows AI to compute prediction without explaining the rationale to the end user, resulting in lack of transparency between human and machine. Concerns are growing over the opacity of such complex AI models, particularly deep learning architectures. To address this concern, explainability is of paramount importance, which has triggered the emergence of Explainable Artificial Intelligence (XAI) as a vital research area. XAI is aimed at enhancing transparency, trust, and accountability of AI models. This survey presents a comprehensive overview of XAI from the dual perspectives of challenges and opportunities. We analyze the foundational concepts, definitions, terminologies, and taxonomy of XAI methods. We then review several application domains of XAI. Special attention is given to various challenges of XAI, such as no universal definition, trade-off between accuracy and interpretability, and lack of standardized evaluation metrics. We conclude by outlining the future research directions of human-centric design, interactive explanation, and standardized evaluation frameworks. This survey serves as a resource for researchers, practitioners, and policymakers to navigate the evolving landscape of interpretable and responsible AI.

1. Introduction

The rapid advancement of Artificial Intelligence (AI) has led to transformative changes in a variety of fields, from healthcare and finance to autonomous systems and natural language processing. However, as AI systems become increasingly complex, the demand for transparency and interpretability in these systems has grown significantly [1]. In particular, the use of AI in safety-critical decision-making contexts has highlighted the importance of explainability [2]. AI models, particularly deep learning models, can achieve high predictive accuracy in various tasks [3]. Despite their predictive prowess, these models often function as “black boxes” due to the opacity of the internal reasoning process [4]. Such opacity has triggered the emergence of Explainable Artificial Intelligence (XAI) as a critical research area [5,6]. At its core, XAI refers to a set of techniques which provide human-understandable explanation in support of the prediction of AI models [7]. This explanation makes an AI model transparent and trustworthy. Such transparency is critical to ensure regulatory compliance in sectors directly affecting individuals, such as finance, healthcare, law, and criminal justice [8]. A major challenge of XAI is to make an AI model’s decision-making process interpretable to humans, while maintaining high predictive accuracy [9].

Opaque deep learning models achieve state-of-the-art predictive accuracy in various areas, such as image analysis, natural language processing, and disease prediction. However, such predictive results are not interpretable by humans [10]. This lack of transparency is a significant barrier to the broader adoption of AI models in various safety-critical domains. For instance, in the healthcare domain, disease prediction by an AI model has to be both accurate and transparent [11]. Without transparency, the predictive output will not be intelligible to doctors and patients, triggering a lack of trust in the AI decision [12]. Similarly, in finance, regulators may require explanation for an AI-based credit scoring model’s decision to check legal compliance and potential biases [13]. Moreover, AI explanation can also teach humans new facts and knowledge. For example, Alpha Go Zero performs better at a game of Go than its human counterpart [14]. If Alpha Go could explain its gaming strategy, human players could learn new tactics to improve their proficiency. This increasing need for transparency has driven the development of various XAI techniques to improve the interpretability of AI models. For example, Local Interpretable Model-agnostic Explanations (LIME) [4], a post hoc explanation technique, explains individual prediction of a black-box AI model by approximating the prediction with a simpler, more interpretable model. Another post hoc technique is SHapley Additive exPlanations (SHAP) [15], which provides feature importance to explain a predictive output. Two primary categories of XAI techniques are model-agnostic and model-specific approaches [16]. Model-agnostic approaches can explain any machine learning model. Model-specific methods are tailored to a specific machine learning model.

Despite making significant progress, the development of XAI techniques has to navigate through several challenges. One major challenge is the trade-off in management between model accuracy and interpretability [17]. Highly accurate deep learning models are difficult to explain due to their complexity and opacity [18]. In contrast, simpler models, such as decision trees or linear regression, are easier to interpret. However, such simple models may not provide as high predictive accuracy as a deep learning model on complex tasks [19]. This trade-off between accuracy and interpretability is a central dilemma in XAI, which causes users to accept reduced performance in exchange for more interpretable models [20]. Moreover, a good explanation depends on the context in which the AI model is deployed and on the audience receiving the explanation. For example, a financial regulator’s need for explanation may be different than that of a healthcare service provider [4]. Another significant challenge is the evaluation of the explanation [21]. Unlike conventional machine learning models’ metrics, such as accuracy and F1-score, evaluating the quality of an explanation is inherently subjective and context-dependent. Various metrics have been proposed, including fidelity (how well an explanation reflects the actual decision-making process of the model) and sufficiency (whether the explanation provides enough information for a user to make an informed decision) [22]. However, no universally accepted framework exists for evaluating explanations of an AI model [23]. Moreover, the human factor plays a crucial role in the intelligibility of explanations [24]. Hence, explanation must be tailored to the audience’s level of expertise and cognitive abilities. Otherwise, such explanation may cause misinterpretation of the model’s decision by the audience [9]. For example, the explanation required by a data scientist is different from that for a layperson [8]. Such subjectivity increases the complexity of XAI techniques. Additionally, biased or under-represented datasets pose the risk of misleading prediction when using a machine learning model [25,26]. Hence, providing explanations to highlight and address such biases is also a challenge for XAI techniques [27].

Despite these challenges, the field of XAI presents numerous opportunities for future research and innovation. One promising direction is human-centered XAI, which focuses on the human cognitive and psychological aspects to provide explanations. An explanation can be made intelligible by utilizing insights from cognitive science and understanding of human–computer interaction (HCI) [28]. For example, visual explanation, such as heatmaps and saliency maps, can explain an image classification output by demonstrating which parts of the image played the most influential role to provide this decision [29]. Another XAI approach is use of an interactive explanation interface, which allows users to actively engage with the model’s decision-making process by asking questions or modifying inputs to evaluate the change in output [30]. Such interaction enhances a user’s trust in an AI model, while providing valuable feedback to the developers to refine the model further. Moreover, XAI can promote ethical AI by addressing fairness, accountability, and transparency issues. Thus, by making an AI model interpretable, XAI can facilitate the identification and mitigation of discrimination and biases present in AI decisions [31]. Finally, due to regulatory requirements, there is an increasing demand for AI systems to comply with emerging laws and guidelines. For instance, the European Union’s General Data Protection Regulation (GDPR) grants “right to explanation” to an individual for an automated decision which affects the person significantly [32]. Such a legal framework creates an opportunity for XAI researchers to collaborate with policymakers and stakeholders to ensure legal compliance of AI models.

The field of XAI is at a critical juncture as it seeks to balance the increasing power and complexity of AI with the need for transparency, trust, and accountability. While significant progress has been made in developing methods for explaining AI models, several challenges remain, including trade-offs between interpretability and accuracy, lack of standardized evaluation metrics, and understanding the human factors involved in providing meaningful explanations. At the same time, exciting opportunities lie ahead in human-centered design, interactive explanation interfaces, regulatory compliance, and ethical AI. This paper aims to provide a comprehensive survey of these challenges and opportunities. Our goal is to bridge the gap between technical advancements and practical needs, offering valuable insights for researchers, practitioners, policymakers, and end-users. We begin by establishing the foundational concepts and definitions in Section 2, followed by a taxonomy of XAI techniques in Section 3. Section 4 highlights various application domains of XAI. Section 5 explores the practical and theoretical challenges impeding XAI implementation. Section 6 discusses the future scope of XAI. Finally, Section 7 concludes the paper.

2. Background and Definition of XAI

2.1. Background

The concept of explainability in AI has long been a topic of interest, dating back to the development of early expert systems in the 1970s [33]. However, the modern emergence of XAI as a formal research area is primarily driven by the increasing adoption of complex opaque models, particularly deep learning models, and the growing demand for transparency, accountability, and trust in automated decision-making systems.

Early Expert Systems and Symbolic AI: The roots of explainability in AI can be traced back to rule-based expert systems, such as MYCIN, which was developed to diagnose bacterial infections and recommend antibiotics [34]. MYCIN used a series of if-then rules and provided human-interpretable explanations by tracing its inference chain [35]. Thus, because of its symbolic reasoning structure, MYCIN had inherent explainability. However, symbolic systems have limitations in terms of scalability and uncertainty handling, leading to a paradigm shift towards data-driven machine learning models [36].

The Rise of Black-Box Models: The paradigm shift from symbolic AI to statistical and sub-symbolic machine learning resulted in the rise of more powerful predictive models, such as random forests, Support Vector Machines (SVMs), and ultimately Deep Neural Networks (DNNs) [37]. These models have achieved superior performance across various application domains. However, the internal logic and feature importance of these models are hidden from end-users, making them “black-boxes” [20]. The term “black-box” is emblematic of the trade-off in these models between explainability and accuracy. The increasing complexity and accuracy of these models come at the cost of reduced interpretability. Such opaqueness is a critical issue in domains where transparency is essential for safety, compliance, and user acceptance.

Deep Learning and the XAI Imperative: The widespread success of deep learning in areas such as disease prediction, computer vision, and natural language processing has intensified concerns about opacity and transparency [38]. DNNs consist of millions of parameters and non-linear transformations, rendering their internal mechanisms nearly incomprehensible to humans [18]. Deployment of such models in safety-critical areas, such as healthcare, finance, and autonomous driving, exposes critical risks. To address these risks, researchers seek to extract meaningful explanations from complex models. This type of investigation has triggered the emergence of XAI as a new research area [8].

Institutional and Regulatory Factors: As an institutional initiative to promote explainability in AI, the Defense Advanced Research Projects Agency (DARPA) of the U.S. Department of Defense launched an XAI program [5,6] in 2016 to create explainable machine learning models while maintaining a high level of prediction accuracy. This program played a foundational role in formally defining XAI as a research domain and fostering interdisciplinary collaboration. The Fairness, Accountability and Transparency (FAT) collaboration [39] is another group, which is focused on promoting explainability and reducing bias in automated decisions produced by an AI model. Moreover, the European Union (EU) has enacted a regulatory framework called the “General Data Protection Regulation (GDPR)” which emphasizes the need for explainability of AI models.This regulation introduced the “right to explanation” to individuals with respect to an automated decision which significantly concerns the person [32]. This legal mandate has enhanced the importance of transparency of AI models.

Multi-disciplinary Research Field: In addition to algorithmic methods, the human-centric dimension is taken into account by XAI. Hence, XAI is a multi-disciplinary research field encompassing machine learning, HCI, philosophy, and cognitive psychology [40]. Researchers argue that a good explanation has to be understandable, relevant, and actionable by different users [23]. Consequently, XAI intersects with broader concerns of fairness, accountability, usability, and societal impact. We show the timeline of the transition from early expert systems to the present XAI in

Table 1. Timeline of transition to XAI.

Year	Milestone	Description
1975	MYCIN expert system [35]	One of the first AI systems to provide rule-based, human-readable explanations in medicine.
1980s	Growth of symbolic AI and rule-based models [34]	Explainability was inherently part of early symbolic AI using logical inference and transparent rules.
1990s	Transition to statistical machine learning [37]	Rise of machine learning models (e.g., random forests, SVMs), with reduced inherent interpretability.
2012	Deep Learning [41]	Despite high predictive accuracy, DNN is an opaque black-box model.
2016	DARPA XAI Program [5]	U.S. government’s initiative to develop interpretable machine learning models without compromising accuracy.
2016	LIME [4]	A post hoc explanation tool which uses local surrogate model to explain individual prediction.
2017	SHAP [15]	A post hoc tool, which uses Shapley values to identify feature importance.
2017	“Right to Explanation” in GDPR [32]	The EU established legal grounds for demanding explanations of automated decisions.
2019	Human-centered XAI [23]	Highlights the importance of social science and human-centered evaluation in explanation design.
2020s	Interdisciplinary research [40]	XAI intersects with machine learning, HCI, philosophy, and ethics.

.

2.2. Definition

XAI refers to a class of methods, techniques, and models which have the aim of making the decision-making processes of AI models transparent, interpretable, and understandable to human users. The increasing deployment of machine learning models, particularly DNNs, has intensified the demand for transparency and interpretability [5,8]. XAI addresses the opacity or “black-box” nature of complex AI models. Although these black-box models, such as DNNs and ensemble methods, offer high performance, lack of intelligibility makes it difficult for end-users to understand how or why a particular decision was made [20]. XAI addresses this limitation by either employing inherently interpretable models or generating post hoc explanations which approximate the model’s decision logic without changing the model’s structure [4,15].

U.S. DARPA, one of the early proponents of the term XAI, defines XAI as “an initiative to produce more explainable models while maintaining a high level of learning performance (prediction accuracy), and to enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners” [5]. This definition emphasizes both the technical and human-centered goals of explainability, while supporting user trust and effective interaction. Other researchers have expanded this definition by distinguishing between interpretability, which refers to the degree to which a human can consistently predict the model’s output, and explainability, which encompasses the broader process of generating human-understandable reasons for the model’s decisions [9,23]. Doshi-Velez and Kim [8] argue that XAI should not be narrowly defined by the provision of explanation alone. They place emphasis on how well an explanation serves the needs of different stakeholders, such as developers (for debugging), end-users (for trust), and regulators (for compliance). Consequently, a universally accepted definition of XAI remains elusive, as explainability is often context-dependent and tailored to the goals, expertise, and concerns of the target audience. To make an explanation meaningful, different domains have different requirements. We provide a comparative overview of how XAI is interpreted across various domains in

Table 2. Comparative interpretation of XAI across various domains.

Domain	Interpretation	Primary Goals
Military and Government [5]	XAI is a framework, which produces more explainable models while maintaining performance and enhancing user trust.	Trust, control, and operational reliability.
Healthcare [12]	Explanation should provide clinically sound reasoning, which physicians and patients can understand.	Transparency, clinical trust, informed decision-making.
Finance [13,32]	Explanation should place emphasis on regulatory compliance and bias detection in decision, such as credit scoring.	Fairness, accountability, legal conformance.
Legal and Regulatory [32]	Focus is on the right to explanation for algorithmic decisions which affect individuals.	Interpretability for audits.
Developers/Engineers [4,15]	Explanations assist in model debugging and feature importance analysis.	Debugging, model improvement
End-users (Non-experts) [23,28]	Explanations must be intuitive, simple, and support trust and usability.	Understandability, user trust, adoption.
Academic Researchers [8,20]	XAI should systematize interpretability through formal definition and evaluation metrics.	Scientific rigor, reproducibility, generalization.

. In light of the diversity of interpretations, we define XAI as follows:

XAI refers to a set of methods and frameworks which make the decisions of AI models transparent, interpretable, and understandable to human stakeholders, with the goal of enhancing trust, accountability, fairness, and regulatory compliance without significantly compromising the AI models’ performance.

Based on this definition, we demonstrate how XAI methods make AI models transparent and trustworthy to human users in Figure 1. This definition forms the foundation for examining the challenges and opportunities of XAI. Moreover, several terms in XAI have ambiguous meanings, which we clarify in the next subsection.

2.3. Terminology

The field of XAI encompasses a diverse and often inconsistent set of terms used across various disciplines. Hence, to ensure clarity and facilitate interdisciplinary collaboration, it is important to define the core terminology of XAI. We provide an overview of the most commonly used terms of XAI below.

Explanation: An explanation in XAI refers to the information provided by a model which makes the model’s decision-making process comprehensible to humans. According to [23], explanations are social constructs, which should be aligned with the way humans naturally seek and evaluate explanations. In the context of machine learning, an explanation is defined as an interface between a human and AI to clarify how the inputs relate to the outputs [8].

Interpretability: The degree to which a human can understand the internal reasoning or parameters of a machine learning model is called interpretability [20]. Despite often being used interchangeably with “explainability,” the term “interpretability” is more narrowly focused on the transparency of the model’s internal structure. For example, decision trees are considered interpretable because of the transparency of the decision paths [42].

Explainability: Explainability is a broader concept which encompasses both interpretability and post hoc explanation [9]. To promote explainability of an opaque AI model, various post hoc tools, such as SHAP [15] and LIME [4], are employed.

Transparency: The openness of a model’s structure and parameters is called transparency [43]. A transparent model’s inner working logic is accessed, inspected, and understood by the relevant stakeholders [20]. Examples of transparent models include linear regression and decision trees. DNN, by contrast, is non-transparent due to its opaque internal structure [44].

Faithfulness: The extent to which an explanation properly reflects the actual reasoning of a model is called faithfulness, also known as fidelity [45]. A faithful explanation does not misrepresent the model’s actual decision-making process, even if the explanation is simplified for human intelligibility.

Simulatability: A human’s ability to simulate or replicate a model’s decision-making process is called simulatability [8]. A model’s simulatability is similar to interpretability, with additional emphasis on the cognitive load of a human.

Justifiability: Ethical and social acceptance of the explanation of a model is called justifiability [46]. Even if an explanation is technically correct, it may be unjustifiable due to bias or discrimination. Such justification is important in certain critical areas, such as criminal justice and healthcare.

Post Hoc Explanation: An explanation provided by various post hoc (after the event) tools is called a post hoc explanation [47]. Such post hoc tools, applied after a model’s predictive output, explain the model’s decision without revealing its internal logic. Examples of post hoc explanations include feature importance scores, counterfactual examples, and visualizations [4,15].

Inherently Interpretable Models: Such models are designed from the ground in a transparent manner for human interpretability [48]. Examples of inherently interpretable models include decision trees, linear models, and rule-based models [12]. The inherent interpretability of such models often comes at the cost of accuracy. However, these models still remain critical in domains where the transparency of decisions is non-negotiable.

3. Research Methodology

Several journals and databases provide a rich source of literature on XAI’s role to make AI models’ decisions transparent. We conducted a systematic search through this rich literature to identify reliable studies from credible authors published from 2022 to 2025. As a verified framework to perform this search in a scientific, reproducible, and transparent approach, we employed the Systematic Literature Review (SLR) [49] approach using the PRISMA guidelines [50]. The SLR process consists of three phases: planning the review, conducting the review, and reporting.

3.1. Planning the Review

We undertook extensive reading and consulted with peers engaged in similar studies to ensure full compliance with the steps required for SLR. This enabled us to minimize the risk of unplanned research duplication and to maximize transparency of the literature review. Special emphasis was placed on research ethics, proper citations, and the intellectual property rights of other researchers. To guide the literature review, we formulated three research questions, as shown in

Table 3. List of the research questions.

Research Question	Description
RQ1	What are the key XAI methods for making the decisions of AI models transparent?
RQ2	How does the relevance of XAI methods vary based on the application domains?
RQ3	What are the major limitations and challenges of XAI methods for enhancing the transparency of AI models?

. With these questions, we systematically identified the existing XAI methods in the literature, their application domains, and limitations. To perform the literature search, we selected two databases, Scopus and Web of Science (WOS), because of their extensive coverage of peer-reviewed XAI articles published in leading international journals and conference proceedings [51]. To collect relevant articles from these two databases, we formulated search queries by combining keywords related to XAI. We show the search strings in

Table 4. Search string.

Keywords Related to Explainability	Keywords Related to AI
(“explainable” OR “interpretable” OR “transparent” OR “trustworthy” OR “xai” OR “fair” OR “ethical” OR “robust” OR “accountable” OR “bias free” OR “discrimination free” OR “safe” OR “post-hoc explanation” OR “post-hoc interpretability”) AND	(“artificial intelligence” OR “AI” OR “machine learning” OR “ML” OR “deep learning” OR “neural networks” OR “automated decision making” OR “predictive analytics” OR “algorithm”)

.

3.2. Conducting the Review

Using the search queries, a total of 3173 papers were identified. To ensure alignment with our research objectives, each paper was assessed based on its title, abstract, and keywords. We then applied inclusion and exclusion criteria as specified in

Table 5. Inclusion and exclusion criteria.

Inclusion Criteria	Exclusion Criteria
XAI methods have been proposed to enhance transparency of AI models;	Technical reports and book chapters;
Articles in English;	Editorials, viewpoints, and opinions;
Novel contributions published in peer-reviewed journals or conference proceedings.	Duplicated articles.

to filter relevant articles further for our review. We conducted the review in four primary stages: identification, screening, eligibility determination, and sorting of research articles. We represent each of these four stages in a PRISMA diagram [50], as shown in Figure 2. In the identification stage, we searched the Scopus and WoS database for XAI articles from 2022 to 2025 using the keywords shown in Table 4. In the screening stage, we removed duplicates and unrelated articles, resulting in 2425 unique articles. Out of these articles, we determined that 122 articles were eligible based on the inclusion and exclusion criteria of Table 5. In the sorting stage, we finally selected 101 articles for review. The PRISMA diagram provides a transparent overview of the whole literature search process, which can be used for replication of the survey.

3.3. Reporting

In this stage, we present the findings of the literature review. Of the 101 published articles, 61 were published in peer-reviewed journals, and the remaining 40 in conference proceedings. In terms of year-wise distribution, 41, 29, 22, and 9 articles were published in the years 2022, 2023, 2024, and 2025, respectively. The number of articles published in 2024 and 2025 is lower due to indexing delays and use of partial-year data, respectively. Among the keywords, ‘explainable artificial intelligence’, ‘machine learning’, and ‘deep learning’ were the most prevalent. On the other hand, the least prevalent keywords were ‘transparent neural networks’, ‘trustworthy decision making’, and ‘post hoc interpretability’. In terms of the geographical distribution, the highest number of publications were from the United States (n = 26), followed by China (n = 21) and India (n = 11). Based on our findings from the published articles included in the review, we present the taxonomy, application domains, challenges, and future scope of XAI in the subsequent sections.

4. Taxonomy of XAI

A systematic taxonomy is essential to apply XAI methods across different domains in an organized manner. Several taxonomic frameworks have been proposed, which are generally categorized by explanation timing, explanation scope, model dependency, explanation form, audience, and interaction modality [8,52,53]. We mention these XAI classes below.

Intrinsic (Ante Hoc) and Post Hoc Explainability: Based on explanation timing, XAI techniques are classified into intrinsic (ante hoc) and post hoc methods [47].

• Intrinsic (ante hoc) methods are inherently interpretable models, such as decision trees, rule-based models, and linear regression. These models offer transparency by exposing the internal logic directly to the user [54]. For example, the Belief Rule-Based Expert System (BRBES), a rule-based model with intrinsic explainabilty, was employed by [25] to predict and explain the energy consumption of buildings.
• Post hoc methods, by contrast, apply interpretability techniques after a model’s predictive output. These techniques extract explanations from a complex “black-box” model, such as DNNs and ensemble models. Examples of post hoc methods include LIME [4], SHAP [15], Partial Dependence Plots (PDPs) [55], Individual Conditional Expectation (ICE) Plots, counterfactual explanation [56], and anchors [57]. PDPs and ICE plots are useful for understanding the relationship between a feature and the predicted outcome, particularly for models that have complex feature interactions. PDPs show how a predicted output changes with regard to the variation in a single feature, keeping other features constant. ICE plots show the effect of a feature on individual instances [58]. Anchors are if-then rules, which “anchor” a prediction. If the conditions in a rule are true, the AI model will make the same prediction, even when other features change. These post hoc tools provide valuable insights into an opaque model’s decision-making process. For instance, Gradient Boosting Machine (GBM) was used to predict the sepsis risk of Intensive Care Unit (ICU) patients [59]. The post hoc tool SHAP explained this prediction by showing that the features, such as serum lactate level, respiratory rate, and Sequential Organ Failure Assessment (SOFA) scores, were the significant contributors to the model’s prediction [59]. Such explanations provide clinicians with actionable insights into individual patient’s risks.

Global and Local Explanation: Based on the scope of explanation, XAI techniques are classified into global and local explanation [60].

• Global explanation describes the overall behavior of a model across the entire input space, allowing insights into feature importance and model structure [53]. For example, SHAP can be used to produce global explanations by aggregating local (per-instance) Shapley values across several predictions, typically using the mean (or mean absolute) Shapley value to estimate each feature’s overall importance [61]. This global explanation of SHAP was applied to a random forest model trained with electronic health record data [62]. After predicting unscheduled hospital readmission with this trained random forest model, the authors used SHAP values to produce a global explanation by ranking features according to their overall influence across the entire cohort. The global SHAP summary identified days of stay and age as the top two most influential features. Such global explanation reveals risk factors for the whole system, rather than individual patients [62].
• Local explanation focuses on the reasoning behind individual prediction. For example, LIME explains the individual predictions of a complex model by creating a local surrogate model (e.g., linear regression or decision tree), which mimics the behavior of the complex model near the input space [4,15]. Anchors are applied locally around an instance to capture the conditions of an if-then rule [57]. To provide local explanation of individual test cases of breast masses classification output, LIME was applied to a DNN by [63]. For a particular patient, LIME highlighted high values of texture, smoothness, and concave points as determining factors for a malignant prediction [63]. Such case-level local explanation helped clinicians to inspect the specific reasoning behind a single prediction, and increased trust in the model-assisted diagnosis.

Model-specific and Model-agnostic Techniques: Based on model dependency, XAI techniques can be classified into model-specific and model-agnostic approaches [64].

• Model-specific approaches are tailored to specific types of models. For instance, attention maps are suitable for deep learning in computer vision and Natural Language Processing (NLP) tasks [65], whereas feature importance scores are applicable to tree-based models. TreeSHAP, a model-specific XAI technique for tree-ensemble models, leverages the internal structure of decision-tree ensembles to compute exact Shapley feature attributions efficiently. In a real-world study, TreeSHAP was employed by [66] to explain a random forest model trained on a clinical metabolomics dataset. TreeSHAP produced both local and global feature importance explanations and identified testosterone metabolites as a key discriminator in a urine dataset [66].
• Model-agnostic approaches are applied universally to any AI model to extract explanations. Examples of these approaches include LIME, SHAP, and partial dependence plots [67]. KernelSHAP is a model-agnostic estimator of Shapley values [68]. It treats a trained predictor as a black-box model and approximates each feature’s contribution by querying the model on perturbed inputs. In a clinical application to intelligent fetal monitoring, ref. [68] applied KernelSHAP post hoc to a stacked-ensemble classifier to locally explain each instance of cardiotocography prediction. To facilitate clinical interpretation, the authors also aggregated local importance scores to produce global explanations.

Type of Explanation Output: The form of explanation is critical to its effectiveness. Based on format, XAI can be classified into the following three types [69].

• Feature-based Explanation: This provides explanations by highlighting the contribution of each input feature to the prediction [70]. In a real-world study, ref. [71] applied a local feature-based explainer (LIME in Study 1, and SHAP in Study 2) to predict real-estate price. The authors showed that the feature-level explanations systematically changed users’ situational information processing and mental models.
• Example-based Explanation: This uses training instances to explain a prediction by highlighting its decision boundaries. Moreover, counterfactuals are generated to inform the user of preconditions to obtain an alternative outcome [56]. Thus, it offers intuitive “what-if” information at the instance level. In a real-world study, ref. [72] introduced GANterfactual, an adversarial image-to-image translation method, which generates realistic counterfactual chest X-rays to flip a pneumonia classifier’s decision. The authors showed that these counterfactuals improved non-expert users’ mental models, satisfaction, and trust compared with saliency map explanation.
• Visual Explanation: This technique (saliency/pixel-attribution maps) produces heat-maps over images to highlight the regions the model used to form a prediction. For example, Gradient-weighted Class Activation Mapping (Grad-CAM), a saliency map technique, produces a heatmap to show which regions of the image contributed the most to the predicted class [73]. If a Convolutional Neural Network (CNN) predicts “dog” for an image, Grad-CAM may highlight the dog’s face and tail, indicating that these regions were the most influential to the classification. In a real-world study, ref. [74] systematically evaluated seven saliency methods (Grad-CAM, Grad-CAM++, Integrated Gradients, Eigen-CAM, Deep Learning Important FeaTures (DeepLIFT), Layer-wise Relevance Propagation (LRP), and occlusion) on chest X-rays. This study found better performance of Grad-CAM than other saliency methods to localize pathologies. However, all methods still performed substantially worse than a human radiologist benchmark. This demonstrates that saliency heat maps are useful but not fully reliable as a standalone clinical explanation [74].

Expert and Layperson Orientation: Based on the intended audience, XAI techniques are divided into expert-oriented and layperson-oriented explanation [75].

• Expert-oriented explanation provides detailed technical insights tailored to the developers, data scientists, or domain experts [25,26]. For example, ref. [76] developed a modified Progressive Concept Bottleneck Model, which returns both anatomical segmentations and clinically meaningful property concepts (symmetry, caliper-placement feasibility, image quality) as real-time feedback for fetal growth scans. The system was validated across hospitals, where 75% of clinicians rated the explanation as useful. Moreover, the model achieved 96.3% classification accuracy on standard-plane assessment.
• Layperson-oriented explanation is provided in non-technical language using analogies, narratives, or visuals to enhance public understanding and trust [8]. Recent advancements in XAI have increasingly focused on designing explanations which align with human cognitive processes. To present explanations to a layperson in an intelligible manner, principles for explanatory debugging were proposed by [28]. In a real-world study, ref. [71] tested feature-level, plain explanations with lay participants on a real-estate price prediction task. The study demonstrated that such explanations change lay users’ decision patterns and mental models, highlighting both the usefulness and risks of exposing non-experts to model rationales. Interactive natural-language explanatory interfaces, such as TalkToModel [77] also increase non-expert users’ performance and comprehension in applied settings.

Static and Interactive Explanation: Based on interaction modality, XAI techniques are classified into static and interactive explanations [78].

• Static explanation provides fixed reports or visualizations without user input [79]. For example, PDPs are static, one-shot visual explanations to show the average effect of a feature on model predictions. PDPs are produced once and inspected as non-interactive figures. In a real-world study, ref. [80] applied one-way and multi-way PDPs to a gradient-boosted model for satellite-based $P{M}_{2.5}$ prediction. PDPs visualized how meteorological and spatiotemporal predictors influenced predicted pollution levels across regions and seasons. Thus, the authors communicated the model behavior to domain scientists using the static PDP figures.
• Interactive explanation allows a user to interact with the model. Such explanation enables a user to investigate model behavior dynamically by changing inputs and exploring “what-if” scenarios, resulting in deeper comprehension of the model’s decision-making process [81]. For example, an AI-powered recommender system can allow users to adjust preferences and immediately assess how these changes influence the recommendations [30]. This interactivity increases user trust and improves the overall understanding of AI behavior. A prominent example of interactive explanation in practice is the InteraCtive expLainable plAtform for gRaph neUral networkS (CLARUS) [82]. This is an explainability platform for graph neural networks in clinical decision support systems. CLARUS visualizes patient-specific biological networks and model relevance scores. It also allows domain experts to manually edit graphs (nodes/edges) to ask “what-if” counterfactuals, immediately re-predict outcomes, and retrain models to observe the consequences of those edits. This interactive loop moves beyond static saliency or post hoc attributions by allowing users to probe causal hypotheses and refine model behavior. This interactive pattern is increasingly emphasized in the human-centered XAI literature [83].

The taxonomy provides a structured overview of various classes of XAI techniques, as shown in Figure 3. A comparison among different XAI techniques is presented in

Table 6. Comparison among various XAI techniques.

Technique	Type	Explanation Format	Model Compatability	Advantages	Limitations
LIME	Post hoc, Local	Feature Importance (weights)	Model-agnostic	Interprets any model; good local fidelity.	Unstable explanations; lacks global consistency.
SHAP	Post hoc, Local and Global	Shapley Values	Model-agnostic	Solid theoretical foundation; consistent.	Computationally intensive for complex models.
Saliency Maps	Post hoc, Local	Visual (Heatmap)	Deep Learning (CNNs)	Intuitive for image data; visual feedback.	Sensitive to noise; lacks standardization.
PDP	Post hoc, Global	Graphical (Feature vs. Output)	Model-agnostic	Simple visualization of feature impact.	Assumes feature independence.
ICE	Post hoc, Local	Graphical (Instance-level)	Model-agnostic	Reveals heterogeneity in predictions.	Difficult to interpret in high dimensions.
Counterfactuals	Post hoc, Local	Example-based	Model-agnostic	Offers actionable, intuitive explanations.	Hard to generate for complex models.
Anchors	Post hoc, Local	Rule-based (If-Then)	Model-agnostic	High precision; human-readable rules.	Narrow coverage; computationally expensive.
Surrogate Models	Post hoc, Global	Tree/Rule-Based	Model-agnostic	Global understanding; interpretable model.	Oversimplifies complex models.
Attention Mechanisms	Intrinsic, Local	Weighted Inputs	DNN	Built-in interpretability; aligns with human focus.	May not faithfully reflect reasoning.
Feature Importance	Post hoc, Global	Ranked Features	Model-agnostic	Quick insights on key features.	Not instance-specific; may miss feature interactions.

. Various software frameworks and libraries related to XAI techniques, along with their model compatibility, applicable AI models, key features, and application domains are presented in

Table 7. Software frameworks and libraries of XAI techniques.

Software Framework and Library	Technique	Model Compatibility	Applicable AI Models	Key Features	Application Domains
LIME [84]	Local surrogate models, Feature-importance weights.	Model-agnostic	Tabular classifiers and regressors.	Human-readable local explanation, fast, lightweight, pluggable to any predictor.	Disease prediction, financial fraud detection.
SHAP [85]	Shapley-value attributions, Global and local explanations.	Model-agnostic	Tree ensembles, DNN.	Contrastive analysis, supports batching and kernel approximations.	Fairness auditing in safety-critical domains, biomedical diagnosis.
Captum [86,87]	Integrated gradients, Saliency maps, Deep Learning Important FeaTures (DeepLIFT), SmoothGrad, VarGrad.	PyTorch models (version 2.1.0)	CNN, RNN, Transformer.	Extensible API, multi-modal support, tight PyTorch integration.	Attribution for Large Language Models (LLMs), NLP model debugging.
ELI5 (Python library) [88]	Permutation importance, Decision tree weight extraction, Text explanations.	Scikit-learn estimators	Decision trees, Linear models, Random forests.	Simple API, built-in visualization, produces human-readable explanations.	Clinical decision support systems, production machine learning pipeline debugging.
AI Explainability 360 (AIX360) [89]	Counterfactual explainers, Contrastive methods, TS-LIME, TS-SHAP for time-series.	Model-agnostic and model-specific modules	Decision trees, Random Forests, Logistic Regression, SVM, DNN, RNN.	Built-in evaluation metrics, taxonomy guidance, plug-and-play pipeline.	Industrial Internet of Things (IoT) forecasting, anomaly detection, supply chain analytics.

. To ensure maximum transparency, choice of an XAI technique must be aligned with the requirements of the stakeholders concerned and the application context. We focus on some of the application domains of XAI in the next section.

5. Application Domains of XAI

XAI has found applications across various domains, each with unique challenges and requirements for transparency and interpretability. Understanding the different contexts in which XAI is applied is crucial for identifying the specific demands of each domain and for tailoring XAI methods to ensure effective use. This section outlines several key application domains of XAI, highlighting the importance of explainability in these fields.

5.1. Healthcare

In healthcare, AI models have been increasingly utilized for various tasks, such as medical diagnosis, personalized treatment plans, and drug discovery. These models, particularly DNNs, can achieve high predictive accuracy. Disease prediction by DNNs concerns life and death questions for patients. Lack of interpretability of disease prediction raises concerns for patients’ safety [12]. Hence, the explainability of an AI model’s decision in healthcare is of paramount importance. This explainability enables medical professionals to evaluate whether the decision of an AI model is consistent with medical knowledge and ethics [90]. In this context, XAI methods are used to explain the decision of medical AI, resulting in a trustworthy predictive output [91].

For instance, cancer diagnosis from Magnetic Resonance Imaging (MRI) images using an AI model has to be both accurate and transparent [12]. For transparency, an XAI method saliency map can be employed to highlight the important features of an MRI image which contributed the most to detect cancer [15]. We focus on some key aspects of the healthcare domain from the XAI perspective below.

• Key AI models and XAI methods

  –

  Diagnostic Imaging: CNN with Grad-CAM [92].

  –

  Disease Prediction: Ensemble tree algorithm with SHAP value explanation [93].

  –

  Treatment Recommendation: Rule-based model [94].

• Domain features

  –

  Highly sensitive, and heterogeneous data, such as imaging, time-series data of Electronic Health Record (EHR), and genomics [92].

  –

  Lack of adequate quantity of labeled data. [93]

  –

  Erroneous prediction and explanation have severe consequences. Hence, the healthcare domain needs transparent prediction from AI models with high accuracy [94].

• Problem types

  –

  Classification: Binary (disease versus no disease) and multi-class classification [95].

  –

  Regression: Risk score prediction for a patient, such as readmission probability, and mortality risk [96].

  –

  Sequence-to-sequence: Clinical report summarization [94].

• Advantages of using XAI

  –

  XAI improves clinicians’ trust in AI decisions through feature importance scores or example-based explanations, such as SHAP explanation in support of a heart failure prediction model [96].

  –

  XAI facilitates regulatory compliance, such as “right to explanation” of GDPR [94].

• Disadvantages of using XAI

  –

  Explanations offered by various post hoc XAI techniques may be misleading due to local approximation of a decision boundary instead of capturing the full model logic or global behavior [97].

5.2. Finance

AI models are used extensively in finance for tasks such as credit scoring, fraud detection, algorithmic trading, and risk management [98]. However, the “black-box” nature of many AI models, especially DNNs, presents challenges in ensuring transparency and accountability of the AI decisions [13]. Hence, the explainability of such decisions is critical in the financial sector to mitigate financial and personal consequences. Regulators and institutions require clear explanations for AI-driven decisions to ensure fairness, detect biases, and to comply with financial regulations [99].

For instance, a model-agnostic approach SHAP [15] can be employed to explain the decision of a credit scoring AI model by identifying the features which contributed the most to a low credit score [100]. We focus on some key aspects of the finance domain from an XAI perspective below.

• Key AI models and XAI methods

  –

  Credit scoring: Random forest with SHAP value explanation [101].

  –

  Financial time-series forecasting: Recurrent Neural Network (RNN) with SHAP value explanation [102].

• Domain features

  –

  High compliance requirements: Financial AI models must be fully auditable and traceable to meet standards such as GDPR, Basel-III, and the Fair Credit Reporting Act [101].

  –

  Fairness and Non-Discrimination: Bias in financial data, such as in relation to gender, race, and location, poses legal and ethical risks [103].

  –

  Real-time requirements: Applications such as fraud detection and high-frequency trading need sub-second predictions and lightweight explanation methods (e.g., SHAP, surrogate decision trees) to keep pace with streaming data [103].

• Problem types

  –

  Classification: Financial fraud and anomaly detection as normal versus suspicious [104].

  –

  Regression: Stock price forecasting and credit scoring by predicting the probability of default [105,106].

  –

  Optimization: Investment portfolio optimization through asset allocation [107].

• Advantages of using XAI

  –

  Feature importance and counterfactual explanations enable financial analysts to address demographic or gender biases in credit scoring and lending models [16].

  –

  Explanations provided by XAI methods enable financial analysts to improve their decision-making speed and quality [108].

• Disadvantages of using XAI

  –

  When faced with large high-frequency datasets of trading and fraud-detection pipelines, traditional post hoc XAI techniques, such as LIME and SHAP, may produce overly generalized or even misleading explanations [109].

  –

  Explanations may leak the sensitive proprietary logic of a financial AI model, which can be manipulated to fool the AI model [109].

5.3. Criminal Justice

AI models are increasingly being used in criminal justice systems for predictive policing, sentencing decisions, and parole evaluation [110]. However, the use of opaque AI models in such critical decision-making areas raises serious concerns about fairness, bias, and accountability [100]. In this context, XAI methods are essential to ensure that the AI decisions are bias-free and justifiable to the stakeholders, including judges, lawyers, and the public.

For instance, in predictive policing, a police department may use a DNN to predict the likelihood of crime in different neighborhoods based on relevant features, such as historical crime data, time of day, location demographics, and economic indicators [110]. However, as a DNN is an opaque model, officers and policy makers will not understand why a specific area is flagged as high-risk by the DNN. In this case, the post hoc tool LIME can be employed to demonstrate which features of the locality most influence a certain prediction regarding crime risk [111]. Similarly, to explain an AI-driven legal sentence, SHAP can be employed to highlight the most influential features contributing to the sentence. Such transparency reduces discrimination and supports more equitable legal practices [27]. Several key aspects of the criminal justice domain from the XAI perspective are indicated below.

• Key AI models and XAI methods

  –

  Recidivism prediction: Gradient-boosted decision trees (XGBoost) with SHAP dependence plots to reveal how factors such as age at first offense, incarceration length, and in-prison behavior drive risk scores to re-offend [112].

  –

  Spatial crime forecasting: Random forest predicts burglary or street-crime risk patterns, which are explained by SHAP values [113].

• Domain features

  –

  Fairness-critical decisions: Decisions such as bail release, sentencing, and parole directly affect individual liberties. Hence, judicial AI models must be both accurate and transparent [114].

  –

  Severe class imbalance: Criminal events such as violent re-offense and homicide cases are rare. Hence, skewed class distributions have to be addressed properly by judicial AI models [115].

  –

  Multi-modal inputs: Criminal data can be in the format of tabular records (demographics, prior convictions), text (legal documents), biometrics (face, fingerprint), and imagery. Hence, judicial AI models have to deal with heterogeneous multi-modal input data [116].

• Problem types

  –

  Classification: Whether a criminal will recidivate or not has to be classified [117].

  –

  Regression: Predicting the continuous number of crimes over time in a hotspot [118].

  –

  Sequence-to-sequence: Converting unstructured legal texts into structured summaries [119].

• Advantages of using XAI

  –

  Post hoc XAI techniques such as SHAP and LIME reveal which features drive risk scores or case clearance predictions, enabling practitioners to detect and correct biases embedded in historical data or AI models [120].

  –

  Providing judges, attorneys, and defendants with comprehensible explanations for AI-driven decisions fosters public confidence in the criminal-justice process [120].

• Disadvantages of using XAI

  –

  Through explanations provided by XAI techniques, exploitable triggers of a judicial AI model may be revealed. Such triggers may be manipulated by malicious actors to reverse engineer the AI model and evade detection [121].

  –

  Many post hoc XAI techniques produce simplified explanations, such as single-feature attribution, which may omit critical model complexities. Such simplification may causes judges to interpret the AI model incorrectly [122].

5.4. Autonomous Systems

Autonomous systems such as self-driving cars and drones rely heavily on AI models to make real-time decisions in complex, dynamic environments [123]. Decisions of autonomous systems directly affect human safety. Hence, the transparency and interpretability of such decisions is critical to ensure accountability [124]. To promote transparency, XAI plays a crucial role by explaining these decisions made by autonomous systems, such as why a car chooses to brake or swerve in a given situation. Thus, XAI can improve trust in autonomous systems, ensure compliance with safety regulations, and aid in debugging the systems for further improvement [125].

For instance, saliency map-based explanation can provide insights into the decision-making process of an autonomous car to detect obstacles and navigate [46]. Such explanations can enable developers to verify whether the autonomous car acts rationally, especially in critical situations. Several key aspects of autonomous systems from the XAI perspective are indicated below.

• Key AI models and XAI methods

  –

  Image segmentation: CNN performs pixel-wise semantic segmentation of road scenes. Critical image regions such as pedestrians and lane markings can be highlighted by saliency maps. Such highlighted regions facilitate design-time debugging and runtime safety monitoring of self-driving cars [126].

  –

  Anomaly detection: Random forest is applied on multi-sensor streams such as cameras, Inertial Measurement Units (IMUs), and the Global Positioning System (GPS) to detect anomalies or faults in a drone. Post hoc XAI techniques such as SHAP and LIME can be applied to fuse and rank the most salient sensor-level features [127].

• Domain features

  –

  Real-time closed-loop operation: Autonomous agents deal with continuous streams of heterogeneous sensor data. AI models have to act on these sensor data within milliseconds. This “sense–think–act” loop requires an AI model to have ultra-low latency for safety-critical actions [128]. Moreover, XAI techniques such as saliency maps and confidence scores have to be generated without breaking real-time constraints [128].

  –

  Safety and reliability: In self-driving cars or industrial robots, failures can lead to injury or loss of life. Hence, such systems must conform to functional safety standards, such as ISO 26262, and provide stable explanations. Such explanations enable engineers and regulators to inspect an autonomous system before and after an incident [129]. Otherwise, without robust explanations of XAI methods, loopholes in autonomous systems will remain undetected.

  –

  Dynamic environments: Autonomous systems operate in non-stationary settings, such as changing weather and where there are novel obstacles. Hence, to cover this distribution shift, XAI methods have to convey uncertainty and adaptation over time [130].

• Problem types

  –

  Classification: Objects or events in the environment are classified by AI models [131]. For example, an AI model can classify obstacles as pedestrians, cyclists, or vehicles. Similarly, a scene can be classified as a road sign or lane marking.

  –

  Regression: AI models can regress continuous quantities for an autonomous system, such as future positions, velocities, and risk scores [132].

• Advantages of using XAI

  –

  Natural language explanations for driving commands produced by transformer-based architectures enhance the trustworthiness of AI models [129].

  –

  Post hoc XAI tools such as attention-map visualizations, surrogate models, and feature-importance scores highlight which inputs triggered a particular decision. This fine-grained insight enables developers to identify the architectural weakness of an autonomous system [129].

• Disadvantages of using XAI

  –

  Many post hoc XAI methods, such as LIME, SHAP, and saliency maps, entail high computational costs to process high-dimensional sensor inputs. Such extensive computation can cause delays in perception and planning pipelines of a real-time autonomous system [128].

  –

  XAI explanations may become unstable or misleading when the sensor quality degrades due to various reasons, such as low light, fog, rain, and motion blur. Without properly quantifying such uncertainties, XAI techniques may provide incorrect explanations in critical situations [133].

5.5. Customer Service and Human Resources

In customer service, AI-powered chatbots and recommendation systems are frequently used to interact with customers and make personalized suggestions [134]. The explainability of these systems is crucial to ensure that customers trust the recommendations they receive. Similarly, in human resources, AI is increasingly being used for resume screening, employee performance evaluation, and promotion decisions [9]. As these decisions directly impact individuals’ careers, ensuring that the AI systems are transparent and free from bias is critical [46,135]. We highlight some key aspects of customer service and human resources from the XAI perspective below.

• Key AI models and XAI methods

  –

  Chatbots and virtual assistants: In customer service, various Large Language Models (LLMs), such as Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT), are used to develop chatbots and virtual assistants [136]. As an XAI method, attention visualization is used to explain chatbot responses [136].

  –

  Resume screening: As an AI model, random forest is applied by human resources departments of organizations to screen resumes of candidates [137]. To explain the candidate selection, SHAP is applied [137].

• Domain features

  –

  Real-time high-volume engagement: To serve customers efficiently, chatbots must handle large streams of customer requests (e.g., chat, voice, and email). Hence, XAI methods have to provide on-the-fly explanations to sustain user engagement. [138].

  –

  Multi-function coverage: In human resources, AI models are employed for multiple functions, such as talent acquisition, performance appraisal, and workforce planning. Hence, based on the use case, the algorithmic requirements of an AI model can be of various types, such as descriptive, predictive, and prescriptive [139].

• Problem types

  –

  Classification: Customer service departments can employ AI models to classify whether a customer will churn [140]. Human resources can classify whether an employee will stay or leave using AI models [141].

  –

  Regression: A customer’s lifetime value can be predicted using a meta-learning-based stacked regression approach [142].

  –

  Clustering: By using AI models, human resources can cluster employees into groups based on multi-modal features, such as skills, engagement, and performance. Such clustering enables management to identify cohorts for tailored training programs [143].

• Advantages of using XAI

  –

  Transparent explanations of model outputs, such as why a recommendation was made, significantly boost end-user confidence with AI-driven customer services [136].

  –

  Explainable candidate-screening systems reduce discriminatory outcomes by making decision pathways auditable and transparent [137].

• Disadvantages of using XAI

  –

  Generating post hoc explanations for high-volume, real-time interactions can introduce computational bottlenecks, slowing response time and degrading customer experience [128].

  –

  In human resources, the same explanation may be interpreted differently by other managers, resulting in divergent decisions [71,144].

5.6. Large Language Models

With the advent of LLMs, such as GPT, the Pathways Language Model (PaLM), and the Large Language Model AI (LLaMA), the landscape of AI applications has expanded dramatically [138]. LLMs enhance human–AI interaction through natural language generation. Therefore, LLMs are being increasingly deployed in diverse domains, such as education, healthcare, customer service, legal document analysis, and software development. However, the opacity of LLMs makes it challenging to understand how specific outputs are generated [145]. To address this challenge, integration of XAI techniques with LLMs has become a crucial research focus. For this purpose, various XAI techniques, such as attention visualization [146], prompt attribution [147], explanation with exemplar-based reasoning (e.g., in-context learning) [147], chain-of-thought prompting [148], and self-explaining models [149] are being investigated to provide a transparent view of the model’s internal logic [150,151]. Integration of these XAI techniques can contribute to a safer and more trustworthy deployment of LLMs in high-stakes environments. However, more research is needed to evaluate the reliability of such XAI techniques for transparency in LLMs [152]. Several key aspects of LLMs from the XAI perspective are highlighted below.

• Key AI models and XAI methods

  –

  BERT is used as an LLM. As an XAI method, attention visualization highlights which input tokens BERT attends to while making a prediction [147]. For example, the attention visualization technique highlights keywords driving sentiment classification.

  –

  Another LLM is GPT-3. To explain the outcome of GPT-3, the XAI technique Chain-of-Thought (CoT) is used [153]. CoT makes GPT-3’s logic transparent by embedding intermediate reasoning steps directly with prompts.

• Domain features

  –

  Model scale and architecture: LLMs are built on a huge-scale parameterized transformer architecture with self-attention, positional embeddings, and multi-head attention layers [154]. This architecture enables long-range context modeling of LLMs more effectively than using RNNs or Long Short-Term Memory (LSTM) [154].

  –

  Adaptation and human feedback: LLMs can be instruction-tuned to follow diverse task descriptions. Moreover, LLMs can be further enhanced through Reinforcement Learning from Human Feedback (RLHF) to refine responses and align with human preferences [147].

• Problem types

  –

  Classification: LLMs are fine-tuned for various classification tasks, such as token-level classification, topic classification, and sentence classification [155].

  –

  Generation: Decoder-only LLMs are trained for next-token prediction and used for open-ended text generation, story generation, code generation, and other creative tasks [155].

  –

  Structured output: LLMs support tasks requiring structured outputs, such as relation extraction outputs in JSON format, particularly in medical and legal information extraction settings [155].

• Advantages of using XAI

  –

  By providing human-understandable explanations, XAI techniques enable stakeholders to audit and sanity-check outputs of LLMs [156].

  –

  LLMs can be prompted to generate their own explanations by approximating traditional feature-attribution methods. Such explanations can be looped back to fine-tune the factuality of LLM output [157].

• Disadvantages of using XAI

  –

  Post hoc XAI techniques may miss complex interactions of transformer layers, resulting in incomplete rationales. Hence, the end-user cannot understand the true decision process of LLMs [128].

  –

  Applying XAI techniques to LLMs is computationally expensive and latency-intensive, which represents a serious bottleneck to provide real-time explanations of LLM output [128].

Different AI models are used in different application domains. Some AI models are used across multiple domains. In each of these domains, explainability is a precondition to ensure trustworthy and justifiable use of AI models. In

Table 8. AI models and XAI techniques across various domains.

AI Model	Domain	XAI Technique	Strength	Weakness
Decision Tree, Random Forest [158,159]	credit scoring, consumer purchase modeling.	SHAP, surrogate tree visualization.	inherently interpretable, faster to train.	overfitting, ensembles lose transparency.
SVM [160,161]	clinical risk prediction, text classification.	global surrogate models, prototype-based kernels.	robust in high dimensions, clear margin interpretation.	does not scale to large data, kernel sensitivity.
Bayesian Networks [162,163]	predictive maintenance, ecological forecasting.	probabilistic graph explanation, counterfactual inference.	uncertainty quantification, causal reasoning.	structured learning is hard to scale, domain expertise required.
DNN [164]	computer vision, genomics.	feature visualization, activation maximization.	high representation power, end-to-end learning.	high computation cost, explanation may lack fidelity.
Graph Neural Network [165]	social recommendation, traffic flow prediction.	edge importance scoring, activation masking.	leverages relational structure, parameter-efficient.	scalability issues, explanation approximation.
Transformer [146]	NLP, code generation, multi-modal fusion.	attention-based analysis, CoT rationales.	modeling long-range dependencies, in-context control.	opaque internals, high computation cost.

, we highlight major AI models, their cross-domain applications, relevant XAI techniques, and each model’s strengths and weaknesses. The increasing reliance of these domains on AI highlights the importance of progressing XAI techniques to meet the specific requirements of each domain. In the path to this advancement of XAI techniques, various trade-offs and challenges exist, which we explain in the next section.

6. Trade-Offs and Challenges of XAI

Despite the significant progress in XAI research, numerous trade-offs and challenges persist, which complicate the practical deployment of interpretable AI models. In this section, we explore these challenges and highlight the direction for XAI to evolve.

6.1. Accuracy Versus Interpretability Trade-Off

One of the most prominent challenges in XAI is the trade-off between model performance and interpretability, often referred to as the accuracy–interpretability trade-off. Complex models, such as DNNs, provide high predictive accuracy on a wide range of tasks. However, due to their intricate and high-dimensional structure, interpretation of this prediction is difficult. On the other hand, simpler models, such as decision trees and linear regression, are easier to interpret. However, in many cases, the predictive accuracy of these simple models may be lower than for complex models [12]. To bridge this gap between accuracy and interpretability, LIME [4] approximates a complex model with a locally interpretable simple model. Similarly, SHAP [15] provides the global feature importance of a black-box model. However, these XAI techniques cannot fully explain the internal decision-making process of a complex model. The key challenge is to determine the right balance between accuracy and interpretability for each application domain. For instance, in high-risk domains, such as healthcare and finance, the interpretability of an AI model is more critical than accuracy.

6.2. Evaluation Metrics

Unlike traditional machine learning metrics, such as accuracy, precision, or recall, which are objective and well-defined, the evaluation of explanations involves inherently subjective and context-dependent criteria [166]. Therefore, evaluation of XAI methods remains a complex and multifaceted challenge. This evaluation concerns the faithfulness, comprehensibility, usefulness, trustworthiness, utility, and interpretability of an explanation. Several metrics have been proposed to assess the effectiveness of XAI methods, each focusing on different aspects of explanation quality. One common metric is fidelity [167], which measures how well an explanation approximates the decision-making process of the original model. High fidelity ensures that the explanation reflects the true reasoning of the model, making it a critical metric in the evaluation of local explanation methods, such as LIME, ref. [4] and SHAP [15]. Stability is another key metric to measure how consistent explanations are across different inputs or perturbations [168]. Human interpretability of an explanation to end-users can be assessed using qualitative survey [8]. The level of interpretability is determined by various factors, such as the explanation length, sparsity, and linguistic simplicity. However, these factors are often heuristic and may not generalize across different user groups [8]. To quantify the mental effort applied by an end-user to understand an explanation, a cognitive load metric is used [23]. Lower cognitive load results in more effective interpretability of an explanation. Trust is also an important evaluation metric in safety-critical domains, such as healthcare and finance [169]. To measure an end user’s trust in a model’s decision after receiving explanations, quantitative surveys can be conducted [8]. Finally, actionability metrics evaluate whether an explanation provides sufficient information to end-users to make informed decisions, which is critical in sensitive areas, such as medical diagnosis and judicial decisions [56].

The challenge in evaluating XAI lies in the development of a comprehensive framework to capture the multi-dimensional nature of an explanation. Future research should place emphasis on the standardization of these evaluation metrics to meet the diverse requirements of various application domains [21]. Moreover, future XAI metrics can evaluate how well explanations capture the true causal mechanisms, not just statistical associations [170,171]. The creation of large-scale, domain-diverse datasets with human-annotated ground truth explanations can significantly improve the benchmarking of XAI metrics [172].

6.3. Scalability

As AI models are being deployed in real-time and large-scale systems, scalability has become a pressing challenge for XAI. Many existing explanation techniques, such as Shapley and LIME values, can be computationally expensive, especially for models with a large number of features or parameters [173]. For example, computing SHAP values [15] for a deep learning model can involve extensive Monte Carlo simulations [174]. Such extensive simulations undermine SHAP’s practical applicability for a real-time system with large datasets [175]. Thus, increasing complexity of the underlying model results in rising cost of the explanation. Recent works on more efficient explanation techniques, such as sparse feature attribution methods and approximation techniques for Shapley values [168], have addressed this scalability issue. However, the need for faster and scalable XAI methods remains a major challenge, particularly in large-scale production environments where time-sensitive decisions are critical.

6.4. Fairness and Privacy

Another significant challenge in XAI is to ensure that the explanation is free of bias and discriminatory outcomes. AI models, trained with historical data, can inherit and even magnify societal biases related to race, gender, or socioeconomic status [27]. Consequently, XAI fairness is linked with data diversity [176], which refers to a model’s ability to represent all types of objects in its output. For example, a content recommendation AI model should recommend diverse contents to its users, rather than accurate yet similar contents [177]. To identify and mitigate this bias of AI models through explanation, the concept of fairness-aware XAI has emerged as a critical area of research [135]. Counterfactual statements, introduced by [56], can also be used to identify unfairness in the decision-making process of an AI model. Moreover, if a biased model is explained in a way which makes it appear fair, the risk of “explainability laundering” will arise [178]. Hence, it is imperative for XAI to ensure fairness of AI models through an ethical approach.

XAI should also protect privacy while dealing with people’s personal data [179]. Explanation of an AI model has to be intelligible to the audience without compromising privacy. Data governance, access protocols, and the quality and integrity of the data are covered by privacy [180]. For example, a few images may be sufficient to violate a user’s privacy even if the images are obfuscated [181]. Influential parameters of a neural network can also be detected using some awkward input queries [182,183]. Cryptographic protocols can be adopted by AI models to preserve individual and group privacy [184,185].

6.5. Human-Centric Interpretability

Human-centric interpretability is crucial to make an explanation intelligible to stakeholders. According to cognitive psychology research, humans are not always adept at interpreting the raw output and complex statistical metrics of a model [23]. Hence, an explanation has to be tailored to the cognitive abilities of both domain experts and laypersons. For instance, interactive and visualization-based explanation techniques may better fit the mental model of a human user [28]. As the mental model of a human user is taken into account to design explanations, XAI requires interdisciplinary collaboration among various research disciplines, such as social science, politics, psychology, humanities, and law [186]. There needs to be a collaborative platform for XAI, where individual users can share how data are collected, preprocessed, cleaned, modeled, and analyzed. Such collaboration will enhance the accessibility of explanations provided using XAI methods.

The trade-offs and challenges of XAI are complex and multifaceted. Future research should be directed towards balancing these trade-offs, while addressing the practical challenges of real-world deployment.

7. Future Scope of XAI

The future of XAI holds significant promise, driven by advancement in AI technologies, increasing regulatory demands, and the growing need for transparency in high-stakes decision-making. As AI systems become more integrated into our society, the demand for interpretability and accountability will continue to rise. In this section, we describe the scope of future research for XAI.

7.1. Formalization of Definition and Terminology

Future research should be focused on formalization of a systematic definition of XAI [187]. A standard definition will facilitate the transfer of results among XAI platforms. All synonymous but semantically different terms require to be standardized, so that the same notions are not referred to using different names [187]. To address this persistent lack of formalized terminology in XAI, the field should produce a compact taxonomy and glossary to clearly distinguish core concepts, such as interpretability versus explainability, local versus global explanation, and ante hoc versus post hoc explanation. Such distinction will ensure understanding the same meaning of currently ambiguous terms across technical, legal, and social science audiences [188]. Moreover, each term has to be paired with an operational definition, and a precise mathematical or procedural specification. Such specification should map each term to measurable properties, such as fidelity, stability, and plausibility, leading to reproducible evaluation [189]. To report comparable XAI results across domains, the XAI community should maintain a living ontology and benchmark registry. This practice will enable researchers to test methods against the same task-and-metric pairs [40].

7.2. Advancement in Explainability Techniques

One of the most exciting areas for the future of XAI lies in the development of more sophisticated and effective explanation methods. Although current methods, such as LIME [4] and SHAP [15], are promising, they are challenging to scale to large complex models. Moreover, most of the existing explanation techniques cannot provide explanations which are comprehensible across diverse application domains. Future research may focus on developing explainability methods tailored to specific domains, such as healthcare, finance, and criminal justice. Such customized explanation will provide context-specific insights, which are crucial for decision-making. Furthermore, as deep learning continues to evolve and becomes more complex, model-agnostic XAI approaches need to be improved to provide consistent explanations [20].

7.3. Human-Centered AI and Cognitive Alignment

Future XAI research should emphasize human cognitive factors to design explanations that are aligned with human understanding. Such alignment will reduce the cognitive load of end-users and improve the trustworthiness of AI decisions [23]. For this purpose, cognitive science and HCI will become integral parts of XAI research. Thus, explanations can become more intelligible to a wide range of users, ranging from expert practitioners to the general public.

7.4. Interactive Explanation

Another significant area for the future of XAI is the development of interactive explanation interfaces [190], where users can actively engage with the model to explore its decision-making process. Interactive interfaces include features such as allowing users to query a model, modify input, and assess the impact of different parameters on model outcomes [30]. Such dynamic and adaptive explanations will provide a deeper understanding of the model’s internal reasoning to users through feedback loops, resulting in improved trustworthiness. Affective computing [191], where affordances are varied based on human feedback, will play an important role to this effect.

7.5. Ethical XAI

As AI technologies continue to influence critical sectors, such as healthcare, finance, and criminal justice, the need for ethical and fair AI models will become increasingly important [192]. The future of XAI is likely to integrate ethical considerations with explainability techniques, focusing on fairness, accountability, and transparency. Such an ethical approach will enable XAI to identify decisions which may irrationally affect an individual or a group of people [27].

7.6. Regulatory Compliance

The future of XAI will also be influenced by regulatory frameworks [193]. Various governments and organizations are increasingly adopting laws and standards to ensure the transparency of AI decisions. As regulations become stricter, new regulatory standards will emerge for explainability in the future. Hence, XAI researchers need to collaborate with lawmakers to develop solutions which balance legal requirements with technical advancements.

7.7. Explainability for New AI Paradigms

As AI continues to evolve, new paradigms, such as federated learning [194] and reinforcement learning [195], are gaining prominence. The complexity of these models introduces new challenges for interpretability. The future of XAI will involve the development of explainability methods for these next-generation AI models [196]. For instance, federated learning involves decentralized data processing. As data are not centrally located, questions are raised regarding explanations provided of the decision-making process in federated learning [197]. Similarly, reinforcement learning models, which learn through interaction with their environment, will require new strategies for providing clear and meaningful explanations of their decision-making.

7.8. Standardization of Evaluation Metrics

An important area for future research will be the standardization of evaluation metrics for XAI. We show some key XAI evaluation metrics, along with their evaluation objectives and benchmark datasets, in

Table 9. XAI evaluation metrics, objectives, and benchmark datasets.

Evaluation Metric	Evaluation Objective	Benchmark Dataset(s)
Deletion/Insertion [198]	Faithfulness: does deleting/inserting top-ranked features change the model output?	ImageNet, CIFAR-10 used in XAI saliency/deletion evaluation [199].
Local Surrogate $R^2$ [198]	Local fidelity: how well a simple local surrogate fits the black-box near an instance x?	“UCI Adult” used in LIME/SHAP analyses and fairness/explainability studies [200].
Counterfactuals [201]	Validity/plausibility: do counterfactuals flip the model?	UCI Adult, German Credit, COMPAS [202].
Rank Correlation [190]	Stability: is the feature ranking consistent under perturbations?	CIFAR, ImageNet [199].
Top-k overlap [190]	Stability: fraction of top-k features preserved under perturbations.	UCI Adult, COMPAS, ImageNet, PASCAL [198].
Lipschitz Ratio [203]	Robustness: magnitude of explanation change per unit input change (the lower the change, the better).	ImageNet, CIFAR [203].
Task Performance [190]	Human usefulness: does “Model + XAI” provide better human decision accuracy than “Model-only”?	ERASER (NLP) and clinical datasets (MIMIC, CheXpert) used in human-centred XAI experiments [204].
Calibration (Brier) [190]	Human calibration: does users’ confidence improve after seeing explanation?	ERASER, MIMIC, CheXpert [204].
Error Detection [205]	Appropriate reliance: do explanations help users detect model errors?	MIMIC, CheXpert, ERASER, and controlled synthetic datasets [206].
Subjective Trust [190]	Self-reported human trust (Likert scale) used alongside the behavioral metrics.	Any human-study task (ERASER, MIMIC, ImageNet, UCI), along with questionnaires [206].

. Although several evaluation criteria, such as fidelity, stability, and human trust currently exist [23], there is no universally accepted framework for evaluating XAI methods. To address this, we propose a unified evaluation framework to operationalize three foundational evaluation dimensions: fidelity, stability, and human trust. Fidelity reflects how faithfully an explanation captures the underlying model’s reasoning, stability shows how consistent the explanation is under input perturbations, and human trust shows how effectively an explanation supports the user’s confidence. We aggregate these three evaluation dimensions through a mathematical equation, as shown below.

$$\begin{array}{cc}\hfill {\mathrm{E}}_{\mathrm{XAI}}\left(\mathrm{x}\right)& ={\lambda }_F\left(w_1\mathrm{norm}\left(R^2\right)+w_2\mathrm{norm}\left(\mathrm{Del}\right)+w_3\mathrm{norm}\left(\mathrm{CF}\right)\right)\hfill \\ \hfill & +{\lambda }_S\left(w_4\mathrm{norm}\left(\rho \right)+w_5\mathrm{norm}\left({\mathrm{Top}}_k\right)+w_6\left(1-\mathrm{norm}\left(L\right)\right)\right)\hfill \\ \hfill & +{\lambda }_T\left(w_7\mathrm{norm}(\Delta \mathrm{Perf})+w_8\mathrm{norm}(\Delta \mathrm{Cal})+w_9\mathrm{norm}(\Delta \mathrm{Det})+w_{10}\mathrm{norm}\left(T_{\mathrm{subj}}\right)\right)\hfill \end{array}$$

(1)

$${\lambda }_F+{\lambda }_S+{\lambda }_T=1,{\lambda }_F,{\lambda }_S,{\lambda }_T\in [0,1],$$

$$w_1+w_2+w_3=1,w_4+w_5+w_6=1,w_7+\cdots +w_{10}=1,w_i\in [0,1].$$

We clarify the meaning of the symbols of this Equation (1) in

Table 10. Meanings of the symbols of Equation (1).

Symbol	Meaning
${\mathrm{E}}_{\mathrm{XAI}}\left(\mathrm{x}\right)$	Evaluation score aggregating fidelity, stability, and human trust (range $[0,1]$ after normalization) for an instance x.
${\lambda }_F,{\lambda }_S,{\lambda }_T$	Weights of the three dimensions: fidelity, stability, and human trust, respectively; ${\lambda }_F+{\lambda }_S+{\lambda }_T=1$.
$w_1,\dots ,w_{10}$	Internal sub-metric weights for the three evaluation dimensions.
$\mathrm{norm}(·)$	Min–max normalization operator mapping a raw metric to $[0,1]$.
$R^2$	Local surrogate fidelity [198].
$\mathrm{Del}$	Deletion / insertion fidelity metric. Normalized Area Under Curve (AUC) drops when top features are removed [198].
$\mathrm{CF}$	Counterfactual validity [201].
$\rho$	Spearman rank correlation between original and perturbed explanation rankings (stability) [207].
${\mathrm{Top}}_k$	Top-k overlap between important feature sets under perturbation [190].
L	Lipschitz ratio for explanations [203] (magnitude of explanation change per unit input change).
$\Delta \mathrm{Perf}$	Change in human task performance (“Model + XAI” versus “Model-only”) [190].
$\Delta \mathrm{Cal}$	Improvement in calibration (Brier score) [190] because of explanation.
$\Delta \mathrm{Det}$	Improvement in error-detection rate (fraction of model errors users flag) [205].
$T_{\mathrm{subj}}$	Aggregated subjective trust score (normalized Likert-based scale, mapped to $[0,1]$) [190].

. By default, each evaluation dimension is equally important (${\lambda }_F={\lambda }_S={\lambda }_T={\displaystyle \frac{1}{3}}$). However, based on the priority of the stakeholder, these weights can be adjusted to reflect the relative importance of the three evaluation dimensions in a given application domain. For example, safety-critical applications may prioritize fidelity (highest weight for ${\lambda }_F$), while user-centric systems may emphasize human trust (highest weight for ${\lambda }_T$). This evaluation scoring system, as shown in Equation (1), can be applied to any XAI method to evaluate its explainability. Thus, our proposed evaluation framework enables the structured evaluation and comparison of XAI methods, leading to the development of more user-aligned solutions. Moreover, the future of XAI will encompass the development of standardized evaluation protocols [21], which can be used across different models, domains, and use cases. Such standardization will enhance the comparison of various XAI methods in a quantitative manner.

7.9. Economic and Sustainability Perspective

An economic perspective on XAI has attracted little attention to date in the literature, yet this is no less important [52]. XAI can be a driving force to increase business value in future. Explainability cost estimation, algorithmic propriety, and trade secrets are among the factors which will inform the economic dimension of XAI in the future [52]. “Structural Econometrics for Explainable AI” was proposed by [208] to highlight the connection between AI and econometrics. To ensure sustainability, XAI products and services have to be compliant with the United Nations’ Sustainable Development Goals (SDG) [209] for the greater benefit of mankind.

The future scope of XAI is expansive. As AI continues to evolve and permeate new domains, the importance of explainability will only grow. The future of XAI will be pivotal in ensuring that AI systems are transparent, trustworthy, and aligned with human values. Researchers, practitioners, and policymakers have to work collaboratively to seize the opportunities presented by XAI for the future.

8. Conclusions

Explainable Artificial Intelligence (XAI) has emerged as a critical field in response to the growing complexity and opacity of modern AI systems. As AI technologies are increasingly deployed in high-stakes domains, such as healthcare, finance, law, and autonomous systems, the demand for transparency, accountability, and trustworthiness has intensified. This survey presented a comprehensive overview of XAI, including its foundational definitions, key terminologies, taxonomy, application domains, trade-offs, and future research directions. In this paper, we observed that the definition of explainability varies significantly across stakeholders, ranging from a technical description for developers to human-centric interpretation aligned with laypersons’ cognitive needs. Although the terminologies of XAI remain somewhat fragmented, efforts to establish standardized vocabularies and taxonomies will contribute to a more structured understanding of the XAI field. The taxonomy of XAI techniques provided highlights the diversity and depth of technical strategies to make models interpretable. Moreover, our review of application domains demonstrated how explainability requirements differ based on contextual constraints, legal requirements, and the potential consequences of AI decisions. Despite substantial progress, XAI continues to face various challenges, such as the accuracy versus interpretability trade-off, lack of a universal definition, and lack of standardized evaluation metrics. With regard to the future scope of XAI, we focused on human-centered interactive explanation, integrating explainability requirements with regulatory frameworks, and explainability for novel AI paradigms, such as federated and reinforcement learning. Interdisciplinary collaboration among AI researchers, cognitive scientists, legal experts, and domain practitioners is essential to align XAI with societal values.

To conclude, XAI represents not only a technical challenge, but also a sociotechnical one. It intersects with issues of fairness, responsibility, and trust. As AI continues to shape critical decisions and societal outcomes, the importance of effective, reliable, and accessible explanations will only grow. Hence, investment in research, standards, and education around explainability is essential for ensuring AI can be used for social good.