Recent Applications of Explainable AI (XAI): A Systematic Literature Review

Abstract

This systematic literature review employs the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology to investigate recent applications of explainable AI (XAI) over the past three years. From an initial pool of 664 articles identified through the Web of Science database, 512 peer-reviewed journal articles met the inclusion criteria—namely, being recent, high-quality XAI application articles published in English—and were analyzed in detail. Both qualitative and quantitative statistical techniques were used to analyze the identified articles: qualitatively by summarizing the characteristics of the included studies based on predefined codes, and quantitatively through statistical analysis of the data. These articles were categorized according to their application domains, techniques, and evaluation methods. Health-related applications were particularly prevalent, with a strong focus on cancer diagnosis, COVID-19 management, and medical imaging. Other significant areas of application included environmental and agricultural management, industrial optimization, cybersecurity, finance, transportation, and entertainment. Additionally, emerging applications in law, education, and social care highlight XAI’s expanding impact. The review reveals a predominant use of local explanation methods, particularly SHAP and LIME, with SHAP being favored for its stability and mathematical guarantees. However, a critical gap in the evaluation of XAI results is identified, as most studies rely on anecdotal evidence or expert opinion rather than robust quantitative metrics. This underscores the urgent need for standardized evaluation frameworks to ensure the reliability and effectiveness of XAI applications. Future research should focus on developing comprehensive evaluation standards and improving the interpretability and stability of explanations. These advancements are essential for addressing the diverse demands of various application domains while ensuring trust and transparency in AI systems.

1. Introduction

In recent decades, there has been a rapid surge in the development and widespread utilization of artificial intelligence (AI) and Machine Learning (ML). The complexity and scale of these models have expanded in pursuit of improved predictive capabilities. However, there is growing scrutiny directed towards the sole emphasis on model performance. This approach often results in the creation of opaque, large-scale models, making it challenging for users to assess, comprehend, and potentially rectify the system’s decisions. Consequently, there is a pressing need for interpretable and explainable AI (XAI), which aims to enhance the comprehensibility of AI systems and their outputs for humans. The advent of deep learning over the past decade has intensified efforts to devise methodologies for elucidating and interpreting these opaque systems [1,2,3].

The literature on XAI is highly diverse, spanning multiple (sub-)disciplines [4], and has been growing at an exponential rate [5]. While numerous reviews have been published on XAI in general [1,2,5], there is a noticeable gap when it comes to in-depth analyses focused specifically on XAI applications. Existing reviews predominantly explore foundational concepts and theoretical advancements, but only a few concentrate on how XAI is being applied across different domains. Although a few reviews on XAI applications do exist [6,7,8], they have limitations in terms of the coverage period and the number of articles reviewed. For instance, Hu et al. [6] published their review in 2021, thus excluding any articles published thereafter. Additionally, they do not specify the total number of articles reviewed, and their reference list includes only 70 articles. Similarly, Islam et al. [7] and Saranya and Subhashini [8] reviewed 137 and 91 articles, respectively, but also focused on earlier periods, leaving a gap in the literature regarding the latest XAI applications.

In contrast, our review fills this gap by providing a more comprehensive and up-to-date synthesis of XAI applications, analyzing a significantly larger set of 512 recent articles. Each article was thoroughly reviewed and categorized according to predefined codes, enabling a systematic and detailed examination of current trends and developments in XAI applications. This broader scope not only captures the latest advancements but also offers a more thorough and nuanced overview than previous reviews, making it a valuable resource for understanding the current landscape of XAI applications.

Given the rapid advancements and diverse applications of XAI, our research focuses on addressing the following key questions:

• Domains: what are the most common domains of recent XAI applications, and what are emerging XAI domains?
• Techniques: Which XAI techniques are utilized? How do these techniques vary based on the type of data used, and in what forms are the explanations presented?
• Evaluation: How is explainability measured? Are specific metrics or evaluation methods employed?

The remainder of this review is structured as follows: In Section 2, we provide a brief overview of XAI taxonomies. Section 3 details the process used to identify relevant recent XAI application articles, along with our coding and review procedures. Section 4 presents the findings, highlighting the most common and emerging XAI application domains, the techniques employed based on data type, and a summary of how the different XAI explanations were evaluated. Finally, in Section 5, we discuss our findings in the context of our research questions and suggest directions for future research.

2. Background: XAI Taxonomies

The primary focus of this review is on the recent applications of XAI across various domains. However, to fully appreciate how XAI has been implemented in these areas, it is essential to provide a brief overview of the key taxonomies of XAI methods. While an exhaustive discussion of these taxonomies, along with the advantages and disadvantages of each method, lies beyond the scope of this article, a concise summary is necessary to ensure that the content and findings of this review are accessible to a broad audience. For those seeking a more comprehensive exploration of XAI taxonomies and detailed discussions on the pros and cons of various XAI methods, we recommend consulting recent reviews [5,9,10,11] and comprehensive books on the subject [12,13].

Generally, XAI methods can be categorized based on their explanation mechanisms, which may rely on examples [14,15,16], counterfactuals [17], hidden semantics [18], rules [19,20,21], or features/attributions/saliency [22,23,24,25]. Among these, feature importances are the most common explanation for classification models [26]. Feature importances leverage scoring and ranking of features to quantify and enhance the interpretability of a model, thereby explaining its behavior [27]. In cases where the model is trained on images, leading to features representing super pixels, methods such as saliency maps or pixel attribution are employed. Evaluating the saliency of features aids in ranking their explanatory power, applicable for both feature selection and post-hoc explainability [5,28,29].

Other approaches to categorizing XAI methods are related to the techniques applied, such as (i) ante-hoc versus post-hoc, (ii) global versus local, and (iii) model-specific versus model-agnostic (see Figure 1). Ante-hoc/intrinsic XAI methods encompass techniques that are inherently transparent, often due to their simplistic structures, such as linear regression models. Conversely, post-hoc methods elucidate a model’s reasoning retrospectively, following its training phase [5,26,30]. Moreover, distinctions are made between local and global explanations: while modular global explanations provide an overarching interpretation of the entire model, addressing it comprehensively, local explanations elucidate specific observations, such as individual images [31,32]. Furthermore, explanation techniques may be categorized as model-specific, relying on aspects of the particular model, or model-agnostic, applicable across diverse models [5,33]. Model-agnostic techniques can be further categorized into perturbation- or occlusion-based versus gradient-based. Techniques like occlusion- or perturbation-based methods manipulate sections of input features or images to generate explanations, while gradient-based methods compute the gradient of prediction (or classification score) concerning input features [34].

Figure 1. Overview of different XAI approaches and evaluation methods. These categories were used to classify the XAI application papers reviewed in this study.

As with machine learning models themselves, there is no universally best XAI approach; the optimal technique depends on factors such as the nature of the data, the specific application, and the characteristics of the underlying AI model. For instance, local explanations are particularly useful when seeking insights into specific instances, such as identifying the reasons behind false positives in a model’s predictions [35]. In cases where the AI model is inherently complex, post-hoc techniques may be necessary to provide explanations, with some methods, like those relying on gradients, being applicable only to specific models, such as neural networks with differentiable layers [34,36]. While a variety of XAI methods are available, evaluating their effectiveness remains a less-explored area [4,11]. As illustrated in Figure 1, XAI evaluation approaches can be categorized into consultations with human experts, anecdotal evidence, and quantitative metrics.

As explained above, our review extends existing work on XAI methods and taxonomies [5,9,10,11] by shifting the focus towards the practical applications of XAI across various domains. In the next section, we will describe how we used the categorizations in Figure 1 to classify the recent XAI application papers in our review.

3. Research Methodology

Based on the research questions posed in Section 1 and the different taxonomies of XAI described in Section 2, we initiated our systematic review on recent applications of XAI. To collect the relevant publications for this review, we followed the analytical protocol of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [37]. A systematic review “is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review” [12]. According to the PRISMA guidelines, our evaluation consisted of several stages: defining eligibility criteria, defining information sources, presenting the search strategy, specifying the selection process, data collection process, data item selection, studying the risk of bias assessment, specifying effect measures, describing the synthesis methods, reporting bias, and certainty assessment [37].

Information sources and search strategy: The search was conducted in February 2024 on Web of Science (WoS) by using the following Boolean search string on the paper topic (note that searches for topic terms in WoS search the following fields within a record: Title, Abstract, Author Keywords, Keywords Plus): TS = ((“explainable artificial intelligence” OR XAI) AND (application* OR process*)). The asterisk (*) at the end of a keyword ensures the inclusion of the term in both singular and plural forms and its derivatives. The search was limited to English-language non-review articles published between 1 January 2021 and 20 February 2024 (the search results can be found here: https://www.webofscience.com/wos/woscc/summary/495b659d-8f9e-4b77-8671-2fac26682231-cda1ce8b/relevance/1, accessed on 24 September 2024). We exclusively used WoS due to its authoritative status and comprehensive coverage. Birkle et al. (2020) [38] highlight WoS as the world’s oldest and most widely used research database, ensuring reliable and high-quality data. Its extensive discipline coverage and advanced citation indexing make it ideal for identifying influential works and mapping research trends [38].

Eligibility criteria and selection process: The literature selection process flow chart is summarized in Figure 2. The database search produced 664 papers. After removing non-English articles (n = 4), 660 were eligible for the full-text review and screening. During the full-text screening, we implemented the inclusion and exclusion criteria (Table 1) established through iterative discussions among the two authors. The reviewers assessed each article under the inclusion and exclusion criteria, with 512 research articles meeting the inclusion criteria and being incorporated into the evaluation procedure.

Figure 2. PRISMA flow chart of the study selection process.

Table 1. Inclusion and exclusion criteria for the review of recent applications of XAI.

Criterion	Included	Excluded
Language	English	Other languages, such as German, Chinese, and Spanish.
Publication type	Peer-reviewed journal articles	Book chapters, conference papers, magazine articles, reports, theses, and other gray literature.
Recentness	Recent papers published in 2021 or after	Papers published before 2021.
Study content	Application of XAI methods	Papers that generally described XAI or reviewed other works without describing any XAI applications.
Quality	Papers of sufficient quality	Papers that were exceptionally short (less than six pages) or those that did not fulfill the basic requirements for a publication channel (e.g., be peer-reviewed, have an international board [35]).

As reported in Figure 2, five articles were not retrievable from our universities’ networks, and 143 were excluded because they did not meet our inclusion criteria (primarily because they introduced general XAI taxonomies or new methods without describing specific XAI applications). Consequently, 512 articles remained for data extraction and synthesis. For reasons of reproducibility, the entire list of included articles is attached in Table A1, along with the XAI application and the reason(s) why the authors say that explainability is essential in their domain.

Data collection process, data items, study risk of bias assessment, effect measures, synthesis methods, and reporting bias assessment: To categorize and summarize the included articles in this review, the first author developed a Google Survey that was filled out for each selected article. The survey included both categorical (multiple-choice) and open-ended questions designed to systematically categorize the key aspects of the research. This approach ensured a consistent and comprehensive analysis across all articles. The survey provided an Excel file with all responses, simplifying the analysis process.

Each reviewer assessed their allocated articles using the predefined codes and survey questions created by the first author. In cases of uncertainty regarding the classification of an article, reviewers noted the ambiguity, and these articles, along with their tentative classifications, were discussed collectively among both authors to reach a consensus. This discussion was conducted in an unbiased manner to ensure accurate classifications. While no automated tools were used for the review process, Python libraries were employed for quantitative assessment.

Some of the developed codes (survey questions) were as follows:

• What was the main application domain, and what was the specific application?
• In what form (such as rules, feature importance, counterfactual) was the explanation created?
• Did the authors use intrinsically explainable models or post-hoc explainability, and did they focus on global or local explanations?
• How was the quality of the explanation(s) evaluated?
• What did the authors say about why the explainability of their specific application is important? (Open-ended question.)

After completing the coding process and filling out the survey for each included article, we synthesized the data using both qualitative and quantitative techniques to address our research questions [39]. Qualitatively, we summarized the characteristics of the included studies based on the predefined codes. Quantitatively, we performed statistical analysis of the data, utilizing Python 3.11.5 to extract statistics from the annotated Excel table. This combination of qualitative and quantitative approaches, along with collaborative efforts, ensured the reliability and accuracy of our review process.

To assess the risk of reporting bias, we examined the completeness and transparency of the data reported in each article, focusing on the availability of results related to our predefined research questions. Articles that lacked essential data or failed to report key outcomes were flagged for potential bias, and this was considered during the certainty assessment.

Certainty assessment: Regarding the quality of the articles, potential bias, and the certainty of their evidence, we followed the general recommendations [40] and included only articles for which at least seven out of the ten quality questions proposed by Kitchenham and Charters (2007) [39] could be answered affirmatively. Additionally, we ensured quality by selecting only articles published in prestigious journals that adhere to established academic standards, such as being peer-reviewed and having an international editorial board [35].

Table 2 reports the number of publications per journal for the ten journals with the highest publication counts in our sample. As shown in the table, IEEE Access has the highest number of publications, totaling 45, which represents 8.79% of our sample of articles on recent XAI applications. It is followed by this journal (Applied Sciences-Basel) with 37 publications (7.23%) and Sensors with 28 publications (5.47%).

Table 2. Number of publications for the ten journals with the highest publication counts in our sample of articles on recent XAI applications.

Journal	# of Publications
IEEE Access	45
Applied Sciences-Basel	37
Sensors	28
Scientific Reports	15
Electronics	14
Remote Sensing	8
Diagnostics	7
Information	7
Machine Learning And Knowledge Extraction	7
Sustainability	7

4. Results

In this section, we present the results of the 512 recent XAI application articles that met our inclusion and quality criteria. As detailed in Section 3, we included only those articles that satisfied our rigorous standards and were not flagged for bias. Once the articles passed our inclusion criteria and were coded and analyzed, we did not conduct further assessments of potential bias within the study results themselves. Our analysis relied on quantitative summary statistics and qualitative summaries derived from these high-quality articles. The complete list of these articles is provided in Table A1, along with their specific XAI applications and the authors’ justifications for the importance of explainability in their respective domains. Next, we provide an overview of recent XAI applications by summarizing the findings from these 512 included articles.

4.1. Application Domains

As shown in Figure 3, the absolute majority of recent XAI applications are from the health domain. For instance, several works have focused on different kinds of cancer prediction and diagnosis, such as skin cancer detection and classification [32,41,42], breast cancer prediction [43,44,45], prostate cancer management and prediction [46,47], lung cancer (relapse) prediction [48,49], and ovarian cancer classification and surgery decision-making [50,51]. In response to the COVID-19 pandemic, significant research has been directed toward using medical imaging for detecting COVID-19 [52], predicting the need for ICU admission for COVID-19 patients [53], diagnosing COVID-19 using chest X-ray images [54], predicting COVID-19 [55,56,57,58,59,60], COVID-19 data classification [61], assessment of perceived stress in healthcare professionals attending COVID-19 [62], and COVID-19 forecasting [58].

Figure 3. Main XAI application domain of the studies in our corpus (including all the main domains mentioned in at least three papers).

Medical imaging and diagnostic applications are also prominent, including detecting paratuberculosis from histopathological images [63], predicting coronary artery disease from myocardial perfusion images [64], diagnosis and surgery [65], identifying reasons for MRI scans in multiple sclerosis patients [66], detecting the health status of neonates [67], spinal postures [68], and chronic wound classification [69]. Additionally, studies have focused on age-related macular degeneration detection [70], predicting immunological age [71], cognitive health assessment [72,73], cardiovascular medicine [74,75], glaucoma prediction and diagnosis [76,77,78], as well as predicting diabetes [79,80,81,82] and classifying arrhythmia [83,84].

General management applications in healthcare include predicting patient outcomes in ICU [60], functional work ability prediction [85], a decision support system for nutrition-related geriatric syndromes [86], predicting hospital admissions for cancer patients [87], medical data management [88], medical text processing [89], ML model development in medicine [90], pain recognition [91], drug response prediction [92,93], face mask detection [94], and studying the sustainability of smart technology applications in healthcare [95]. Lastly, studies about tracing food behaviors [96], aspiration detection in flexible endoscopic evaluation of swallowing [97], human activity recognition [98], human lower limb activity recognition [99], factors influencing hearing aid use [100], predicting chronic obstructive pulmonary disease [101], and assessing developmental status in children [102] underline the diverse use of XAI in the health domain.

It is also noteworthy that brain and neuroscience studies have frequently been the main application (Figure 3), often related to health. For example, Alzheimer’s disease classification and prediction have been major areas of focus [103,104,105,106,107,108,109], and Parkinson’s disease diagnosis has been extensively studied [110,111,112,113]. There is also significant research on brain tumor diagnosis and localization [114,115,116,117,118], predicting brain hemorrhage [119], cognitive neuroscience development [120], and detecting and explaining autism spectrum disorder [121]. Other notable brain studies include the detection of epileptic seizures [122,123], predicting the risk of brain metastases in patients with lung cancer [124], and automating skull stripping from brain magnetic resonance images [125]. Similarly, three pharmacy studies are related to health, including metabolic stability and CYP inhibition prediction [126] and drug repurposing [127,128].

In the field of environmental and agricultural applications, various studies have utilized XAI techniques for a wide range of purposes. For instance, earthquake-related studies have focused on predicting an earthquake [129] and assessing the spatial probability of earthquake impacts [130]. In the area of water resources and climate analysis, research has been conducted on groundwater quality monitoring [131], predicting ocean circulation regimes [132], water resources management through snowmelt-driven streamflow prediction [133], and analyzing the impact of land cover changes on climate [134]. Additionally, studies have addressed predicting spatiotemporal distributions of lake surface temperature in the Great Lakes [135] and soil moisture prediction [136]. Environmental monitoring and resource management applications also include predicting heavy metals in groundwater [137], detection and quantification of isotopes using gamma-ray spectroscopy [138], and recognizing bark beetle-infested forest areas [139]. Agricultural applications have similarly leveraged XAI techniques for plant breeding [140], disease detection in agriculture [141], diagnosis of plant stress [142], prediction of nitrogen requirements in rice [143], grape leaf disease identification [144], and plant genomics [145].

Urban and industrial applications are also prominent, with studies on urban growth modeling and prediction [146], building energy performance benchmarking [147], and optimization of membraneless microfluidic fuel cells for energy production [148]. Furthermore, predicting product gas composition and total gas yield [149], wastewater treatment [150], and the prediction of undesirable events in oil wells [151] have been significant areas of research. Lastly, environmental studies have also focused on predicting drought conditions in the Canadian prairies [152].

In the manufacturing sector, XAI techniques have been employed for a variety of predictive and diagnostic tasks. For instance, research has focused on prognostic lifetime estimation of turbofan engines [153], fault prediction in 3D printers [154], and modeling hydrocyclone performance [155]. Moreover, the prediction and monitoring of various manufacturing processes have seen substantial research efforts. These include predictive process monitoring [156,157], average surface roughness prediction in smart grinding processes [158], and predictive maintenance in manufacturing systems [159]. Additionally, modeling refrigeration system performance [160] and thermal management in manufacturing processes [161] have been explored. Concrete-related studies include predicting the strength characteristics of concrete [162] and the identification of concrete cracks [163]. In the realm of industrial optimization and fault diagnosis, research has addressed the intelligent system fault diagnosis of the robotic strain wave gear reducer [164] and the optimization of injection molding processes [165]. The prediction of pentane content [166] and the hot rolling process in the steel industry [167] have also been areas of focus. Studies have further examined job cycle time [168] and yield prediction [169].

In the realm of security and defense, XAI techniques have been widely applied to enhance cybersecurity measures. Several studies have focused on intrusion detection systems [170,171,172], as well as trust management within these systems [173]. Research has also explored detecting vulnerabilities in source code [174]. Cybersecurity applications include general cybersecurity measures [175], the use of XAI methods in cybersecurity [176], and specific studies on malware detection [177]. In the context of facial and voice recognition and verification, XAI techniques have been employed for face verification [178] and deepfake voice detection [179]. Additionally, research has addressed attacking ML classifiers in EEG signal-based human emotion assessment systems using data poisoning attacks [180]. Emerging security concerns in smart cities have led to studies on attack detection in IoT infrastructures [181]. Furthermore, aircraft detection from synthetic aperture radar (SAR) imagery has been a significant area of research [182]. Social media monitoring for xenophobic content detection [183] and the broader applications of intrusion detection and cybersecurity [184] highlight the diverse use of XAI in this domain.

In the finance sector, XAI techniques have been employed to enhance various decision-making processes. Research has focused on decision-making in banking and finance sector applications [185], asset pricing [186], and predicting credit card fraud [187]. Studies have also aimed at predicting decisions to approve or reject loans [188] and addressing a range of credit-related problems, including fraud detection, risk assessment, investment decisions, algorithmic trading, and other financial decision-making processes [189]. Credit risk assessment has been a significant area of research, with studies on credit risk assessment [190], predicting loan defaults [191], and credit risk estimation [192,193]. The prediction and recognition of financial crisis roots have been explored [194], alongside risk management in insurance savings products [195]. Furthermore, time series forecasting and anomaly detection have been important areas of study [196].

XAI has also been used for transportation and self-driving car applications, such as the safety of self-driving cars [197], marine autonomous surface vehicle engineering [198], autonomous vehicles for object detection and networking [199,200], and the development of advanced driver-assistance systems [201]. Similarly, XAI offered support in retail and sales, such as inventory management [202], on-shelf availability monitoring [203], predicting online purchases based on information about online behavior [204], customer journey mapping automation [205], and churn prediction [206,207].

In the field of education, XAI has been applied to various areas such as the early prediction of student performance [208], predicting dropout rates in engineering faculties [209], forecasting alumni income [210], and analyzing student agency [211]. In psychology, XAI was used for classifying psychological traits from digital footprints [212]; in social care, for child welfare screening [213]; and in the laws, for detecting reasons behind a judge’s decision-making process [214], predicting withdrawal from the legal process in cases of violence towards women in intimate relationships [215], and inter partes institution outcomes predictions [216]. In natural language processing, XAI was used for explaining sentence embedding [217], question classification [218], questions answering [219], sarcasm detection in dialogues [220], identifying emotions from speech [221], assessment of familiarity ratings for domain concepts [222], and detecting AI-generated text [223].

In entertainment, XAI was used, for example, for movie recommendations [224], explaining art [225], and different gaming applications, including analyzing and optimizing the performance of agents in a game [226], deep Q-learning experience replay [227], and cheating detection and player churn prediction [228]. Furthermore, several studies concentrated on (social) media deceptive online content (such as fake news and deepfake images) detection [229,230,231,232,233,234]. In summary, the recent applications of XAI span a diverse array of domains, reflecting its evolving scope; Figure 4 illustrates eight notable application areas.

Figure 4. Saliency maps of eight diverse recent XAI applications from various domains: brain tumor classification [116], grape leaf disease identification [144], emotion detection [235], ripe status recognition [141], volcanic localizations [236], traffic sign classification [237], cell segmentation [238], and glaucoma diagnosis [77] (from top to bottom and left to right).

4.2. XAI Methods

As shown in Figure 5, the majority of recent XAI papers used local explanations (53%), or a combination of global and local explanations (29%).

Figure 5. Number of papers in our corpus that used global versus local explanations.

SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) are the most commonly used local XAI methods (Figure 6). While LIME is fully model-agnostic, meaning it is independent of the prediction model and can be used on top of any linear or non-linear model, the SHAP toolbox includes both model-agnostic XAI tools (such as the SHAP Kernel Explainer) and model-specific XAI tools (such as the TreeExplainer, which has been optimized for tree-based models [239]). However, LIME has faced criticism for its instability, meaning the same inputs do not always result in the same outputs [32], and its local approximation lacks a stable connection to the global level of the model. In contrast, SHAP boasts four desirable properties: efficiency, symmetry, dummy, and additivity [240], providing mathematical guarantees to address the local-to-global limitation. These guarantees may explain SHAP’s higher popularity in recent XAI application papers (Figure 6). Another local model-agnostic method used in recent XAI application papers is Anchors, which belongs to the same XAI group as SHAP and LIME but is much less popular in recent XAI application papers (e.g., [167,188,190,229,241]).

Figure 6. Most common explanation techniques used in the papers in our corpus (only XAI techniques used in at least five papers are shown).

While perturbation-based techniques, such as LIME (e.g., [65,175,187,242,243]) and SHAP (e.g., [65,175,186,244,245]), are often the choices in recent XAI studies for tabular data, studies involving images or other more complex data frequently use gradient-based techniques such as Grad-CAM (e.g., [89,94,164,178,243]), Grad-CAM++ (e.g., [41,94,246,247,248]), SmoothGrad (e.g., [246,249,250,251,252]), Integrated Gradients (e.g, [50,179,182,253,254]), or Layer-Wise Relevance Propagation (LRP), such as those in [175,179,241,255,256]. Figure 4 shows eight examples of saliency maps from image data of diverse recent XAI applications from various domains.

The most commonly used global model-agnostic techniques are Partial Dependence Plots (PDP), such as those in [65,74,102,257,258], Accumulated Local Effects (ALE), as seen in [136,157,258,259,260], and Permutation Importance (e.g., [74,136,137,156,180]). Conversely, the most commonly used global intrinsically explainable methods are decision trees (e.g., [88,91,183,191,261]) and logistic regression (e.g., [50,53,61,191,211]). It should be noted that the latter two are used in countless other papers, but, given their inherent interpretability, they are often not explicitly listed as XAI methods [31].

4.3. ML Models and Tasks

Figure 7 represents the mostly used ML models in recent XAI papers (please note that more than one ML model can be used in the same paper). Various neural network models (predominantly deep NN) are mostly used ML models (used in 59% of papers), followed by the tree-based modes (e.g., decision tree, random forest, gradient boosting, used in 37% of papers), support vector machine (11%), linear or logistic regression (9%), K nearest neighbor (4%), Bayesian-based models (3%), and Gaussian models (e.g., Gaussian process regression and Gaussian mixture model, used in 2% of papers). The distribution of the ML models used in the reviewed articles is comparable to what is generally used.

Figure 7. Mostly used ML models in the papers in our corpus (only ML models used at least five times are shown).

Besides the most common ML models, there are some others that are less used and could therefore provide interesting alternative views on XAI. These include methods based on fuzzy logic (e.g., fuzzy rule-based classification [262], rule-based fuzzy inference [226,263], fuzzy decision tree [264], fuzzy nonlinear programming [95]), graph-based models (e.g., graph-deep NN [265,266], knowledge graph [267]), or some sort of optimization with computational intelligence (e.g., particle swarm optimization [148,160], clairvoyance optimization [268]).

The ML models have been used mainly for classification purposes (70%), followed by regression (21%), clustering (4%) and reinforcement learning (1%), as can be seen in Figure 8. Other tasks, which occurred in only one or at most two articles, include segmentation [97,269], optimization [270,271], semi-supervised [272,273] or self-supervised tasks [274], object detection [275], and novelty search [276].

Figure 8. The main ML tasks in the papers in our corpus (all other ML tasks are used in only one or at most two papers).

There is no substantial difference between the major ML models with regard to the ML task of their target application. The distributions of ML tasks for specific ML models (NN, DT, LR, kNN, etc.) are all very similar to the overall one represented in Figure 8. Among all major ML models, SVM stands out the most, which is used for classification somewhat more often than the others (in 80% of cases).

With regard to the application domain, health, environment, industry, and security and defense are among the top five domains for all the major ML models, with the only exception being linear or logistic regression. In the case when linear or logistic regression was used as an ML model, finance is among the top three application domains, which is never the case for other major ML models. As finance is the second most used with the tree-based ML models, which, similarly to linear and logistic regression, can be characterized as the most transparent and inherently interpretable models, it suggests that the users in the financial domain are especially keen on getting insights and explanations on how the ML models operate on their data.

4.4. Intrinsically Explainable Models

As shown in Figure 9, the majority of recent XAI papers used post-hoc explainability approaches on ML models, which are not naturally easily interpretable (79%), as opposed to the intrinsically explainable models (12%); other papers (9%) reported a combination of both. Figure 10 presents the distribution of intrinsically explainable ML models. From all the reviewed XAI papers that reported their used method as intrinsically explainable, the majority were tree-based (41%), followed by deep NN (19%), linear or logistic regression (5%), and some Bayesian models (3%). The predominance of tree-based ML models could have been expected, as well as a relatively high number of linear and logistic regression models, which both are considered naturally transparent and simpler to understand, given their inherent interpretability. On the other hand, the relatively high number of deep neural networks that have been represented as intrinsically explainable is somewhat surprising.

Figure 9. Number of papers in our corpus that used a post-hoc approach versus intrinsically explainable ML model.

Figure 10. Number of papers that used a specific ML model, which is presented as intrinsically explainable.

There are significant differences between different ML models represented as intrinsically explainable with regard to the form of explanation they use. While the intrinsically explainable tree-based ML models use a variety of forms of explanation, including feature importance (in 50% of all cases), rules (38%), and visualization (31%), the deep NN models being reported as intrinsically explainable rely mainly on visualization (in more than 67% of all cases). The intrinsically explainable linear and logistic regression ML models, however, use predominantly feature importance as their form of explanation (in 75% of all cases).

In the most frequent XAI application domain, namely health, the use of tree-based ML models is predominant, as the tree-based models are used in 28% of all health applications, followed by (deep) neural networks (22%), and interestingly fuzzy logic (11%), while all other models were used only once in health. Given the known history of the development of ML methods in the field of medicine and healthcare, where the ability to validate predictions is as important as the prediction itself, and consequently the key role of decision trees [277], this result does not even surprise us.

With regard to other application domains, we can see that intrinsically explainable ML models, like tree-based models and linear or logistic regression models, are used for finance and education applications much more often than other ML models. While the financial domain represents only 1% of (deep) neural network applications, it represents 6% of all tree-based ML model applications (used for credit risk estimation [192,193], risk management in insurance [195], financial crisis prediction [194], investment decisions and algorithmic trading [189], and asset pricing [186]) and even 9% of linear or logistic regression applications (used primarily for credit risk assessment [190] and prediction [193], as well as financial decision-making processes [189]). While post-hoc explainability methods, primarily SHAP and LIME, are the most favored in the financial sector [189], intrinsically explainable modes are gaining popularity for revealing the insights and are being used for stock market analysis [278] and forecasting [279], profit optimization, and predicting loan defaults [191]. Education represents 2% of all applications of tree-based ML models (including early prediction of student performance [208], predicting student dropout [209], and advanced learning analytics [211]) and 4% of linear or logistic regression models (such as pedagogical decision-making [211] and prediction of post-graduate success and alumni income [210]), while (deep) neural networks are used for comparison with other methods in only two of all the reviewed XAI papers concerning education [209,211].

4.5. Evaluating XAI

The use of well-defined and validated metrics for evaluating the quality of XAI results (i.e., explanations) is of great importance for widespread adoption and further development of XAI. However, a significant number of authors still use XAI methods as a sort of add-on to their ML models and results without properly addressing the quality aspects of provided explanations, and only a few articles in our corpus use metrics to quantitatively measure the quality of their XAI results (Figure 11). More than 58% of the reviewed articles applied XAI but did not provide any evaluation of their XAI results (e.g., [65,121,170,175,186]). Among those that evaluated their XAI results, most relied on anecdotal evidence (20% of the reviewed articles, e.g., [185,245,249,272,280]). In approximately 8% of papers, the authors evaluated their XAI results by asking domain experts to evaluate the explanations (e.g., [66,70,89,114,167]). In approximately 19% of papers, however, some sort of quantitative metrics are used to provide the quality assessment (e.g., [94,179,187,242,244,281]).

Figure 11. Evaluation of the explanations in recent XAI application papers.

These numbers are in line with a recent review article about XAI evaluation methods that also highlighted the lack of reporting metrics to measure explanation quality, according to Nauta et al. [4], only one in three studies that developed XAI algorithms evaluates explanations with anecdotal evidence, and only one in five studies evaluates explanations with users. Also, Leite et al. state that “evaluation measures for the interpretability of a computational model is an open issue” [282]. To address this issue, they introduced an interpretability index to quantify how a granular rule-based model is interpretable during online operation. In fact, the gap of “no agreed approach on evaluating produced explanations” [283] is often mentioned as future work. Having such a metric would solve several XAI issues, such as decreasing the risk of confirmation bias [283,284].

For this purpose, we further analyzed the articles that used metrics to measure explanation quality, primarily to see what the authors reported about the explainability of their results. Since different ML tasks and/or ML models may focus on different aspects, we divided the analysis according to the main task of the ML model.

In the case when metrics have been used for evaluating the quality of clustering, segmentation, and other unsupervised ML methods’ explanations, the findings highlight that the evaluated XAI approaches provided accurate, transparent, and robust explanations, aiding in the interpretation of the ML models and results (e.g., [285,286]). Human and quantitative evaluations confirmed the methods’ superiority in generating reliable, interpretable, and meaningful explanations [287], despite occasional contradictory insights that proved useful for identifying anomalies [269].

For the reinforcement learning applications, the findings of papers assessing their XAI results by evaluation metrics demonstrate that the proposed methods effectively explained complex models and highlighted the potential of Shapley values for explainable reinforcement learning [288]. Additionally, participants using the AAR/AI approach identified more bugs with greater precision [289], and while explanations improved factory layout efficiency, their interpretability remains an area for improvement [290].

The findings of the papers using regression as their main task, which used some metric to evaluate the explanations, underscore the critical role of explainability techniques like Shapley and Grad-CAM in enhancing model interpretability and accuracy (e.g., [157,291]) across various domains, from wind turbine anomaly detection [244] to credit card fraud prediction [187]. While global scores aid in feature selection, semi-local analyses offer more meaningful insights [292]. XAI methods revealed system-level insights and emergent properties [293], though challenges like inconsistency, instability, and complexity persist [157,294]. User studies and model retraining confirmed the practical benefits of improved explanations [213,295]. However, the authors mentioned that the explainability of their results was limited by the lack of suitable metrics for evaluating the explainability of algorithms [294].

Finally, for the most frequent ML task of classification, the analysis of the papers, which used some metrics to evaluate their explainability results, emphasizes the importance of explainability in enhancing model transparency, robustness, and decision-making accuracy across various applications, from object detection from SAR images [182] and hate speech detection [296] to classification of skin cancer [32] and cyber threats [297]. Techniques like SHAP, LIME, and Grad-CAM provided insights into feature importance and model behavior (e.g., [124,298,299]). In some situations, the adopted XAI methods showed improved performance and more meaningful explanations, aiding in tasks like malware detection [177], diabetes prediction [82], extracting concepts [298], and remote sensing [300]. Evaluations confirmed that aligning explanations with human expectations and ensuring local and global consistency are key to improving the effectiveness and trustworthiness of AI systems [235]. The authors concluded that while explanation techniques showed promise, there is still a long way to go before automatic systems can be reliably used in practice [32], and widely adopted XAI metrics can help here a lot.

In summary, the results reveal distinct preferences and practices in using XAI. Tree-based models, commonly used in health applications, employ various explanation forms like feature importance, rules, and visualization, while deep neural networks primarily utilize visualization. Linear and logistic regression models favor feature importance. In finance and education, tree-based and regression models are more prevalent than deep neural networks. However, despite the widespread application of XAI methods, evaluation practices remain underdeveloped. Over half of the studies did not assess the quality of their explanations, with only a minority using quantitative metrics. There is a need for standardized evaluation metrics to improve the reliability and effectiveness of XAI systems.

5. Discussion and Conclusions

This systematic literature review explored recent applications of Explainable AI (XAI) over the last three years, identifying 664 relevant articles from the Web of Science (WoS). After applying exclusion criteria, 512 articles were categorized based on their application domains, utilized techniques, and evaluation methods. The findings indicate a dominant trend in health-related applications, particularly in cancer prediction and diagnosis, COVID-19 management, and various other medical imaging and diagnostic uses. Other significant domains include environmental and agricultural applications, urban and industrial optimization, manufacturing, security and defense, finance, transportation, education, psychology, social care, law, natural language processing, and entertainment.

In health, XAI has been extensively applied to areas such as cancer detection, brain and neuroscience studies, and general healthcare management. Environmental applications span earthquake prediction, water resources management, and climate analysis. Urban and industrial applications focus on energy performance, waste treatment, and manufacturing processes. In security, XAI techniques enhance cybersecurity and intrusion detection. Financial applications improve decision-making processes in banking and asset management. Transportation studies leverage XAI for autonomous vehicles and marine navigation. The review also highlights emerging XAI applications in education for predicting student performance and in social care for child welfare screening.

In categorizing recent XAI applications, we aimed to identify and highlight the most significant overarching themes within the literature. While some categories, such as “health”, are clearly defined and widely recognized within the research community, others, like “industry” and “technology”, are broader and less distinct. The latter categories encompass a diverse range of applications, reflecting the varied contexts in which XAI methods are employed across different sectors. This categorization approach, though occasionally less precise, captures the most critical global trends in XAI research. It acknowledges the interdisciplinary nature of the field, where specific categories may overlap or lack the specificity found in others. Despite these challenges, our goal was to provide a comprehensive overview that highlights the most prominent domains where XAI is being applied while recognizing that some categories, by their nature, are more general and encompass a wider array of subfields.

By far the most frequent ML task among the reviewed XAI papers is classification, followed by regression and clustering. Among the used ML models, deep neural networks are predominant, especially convolutional neural networks. The second most used group of ML models are tree-based models (decision and regression trees, random forest, and other types of tree ensembles). Interestingly, there is no substantial difference between the major ML models with regard to the ML task of their target application.

Feature importance, referring to techniques that assign a score to input features based on how useful they are at predicting a target variable [26], is the most common form of explanation among the reviewed XAI papers. Some sort of visualization, trying to visually represent the (hidden) knowledge of a ML model [301], is used very often as well. Other commonly used forms of explanation include the use of saliency maps, rules, and counterfactuals.

Regarding methods, local explanations are predominant, with SHAP and LIME being the most commonly used techniques. SHAP is preferred for its stability and mathematical guarantees [240], while LIME is noted for its model-agnostic nature but criticized for its instability [32]. Gradient-based techniques such as Grad-CAM, Grad-CAM++, SmoothGrad, LRP, and Integrated Gradients are frequently used for image and complex data [179,182]. In general, post-hoc explainability is much more frequent than the use of some intrinsically explainable ML model. However, only a few studies quantitatively measure the quality of XAI results, with most relying on anecdotal evidence or expert evaluation [4].

In conclusion, the recent surge in XAI applications across diverse domains underscores its growing importance in providing transparency and interpretability to AI models [4,5]. Health-related applications, particularly in oncology and medical diagnostics, dominate the landscape, reflecting the critical need for explainable and trustworthy AI in sensitive and high-stakes areas. The review also reveals significant research efforts in environmental management, industrial optimization, cybersecurity, and finance, demonstrating the versatile utility of XAI techniques.

Despite the widespread adoption of XAI, there is a notable gap in the evaluation of explanation quality. The analysis of how the authors evaluate the quality of their XAI approaches and results revealed that in the majority of studies, the authors still do not evaluate the quality of their explanations or simply rely on subjective or anecdotal methods, with only a few employing rigorous quantitative metrics [284]. Cooperation with domain experts and including users can greatly contribute to the practical usefulness of the results, but above all, more attention needs to be paid to the development and use of well-defined and generally adopted metrics for evaluating the quality of explanations. It turns out that in such a case, we can expect reliable, interpretable, and meaningful explanations with a significantly higher degree of confidence. There is an urgent need for standardized evaluation frameworks to ensure the reliability and effectiveness of XAI methods, as well as to improve the interpretability and stability of explanations. The development of such metrics could mitigate risks like confirmation bias and enhance the overall robustness of XAI applications.

Limitations and Future Work

This systematic literature review has several limitations that should be acknowledged. Firstly, the review relied exclusively on the WoS database to identify and retrieve relevant studies. While WoS is recognized as one of the most prestigious and widely utilized research databases globally, known for its rigorous indexing standards and the high quality of its data sources [38], the reliance on a single database may introduce a potential bias by omitting relevant literature indexed in other databases such as Scopus, IEEE Xplore, or Google Scholar. However, it is important to note that the comprehensive nature of WoS mitigates this limitation to some extent. WoS encompasses a vast array of high-impact journals across various disciplines, ensuring that the most significant and influential works in the field of XAI are likely to be included. Moreover, the substantial volume of results yielded from WoS alone necessitated a practical constraint on the scope of the review. Including additional databases would have exponentially increased the literature volume, rendering the review process unmanageable within the given resources and timeframe.

Secondly, the exclusion criteria applied in this review present additional limitations. Only studies published in English were included, which could potentially skew the findings by overlooking valuable contributions from non-English-speaking researchers and regions. Furthermore, the review was limited to studies published after 2021 to ensure the “recentness” of the applications of XAI. While this criterion was essential to focus on the latest advancements and trends, it may have excluded foundational studies that, although older, remain highly relevant to the current state of the field. Additionally, the review was restricted to journal articles, excluding conference papers that often publish seminal work, particularly in the fast-evolving domain of XAI. Given the considerable volume of literature, including conference papers would have extended the scope beyond what was feasible within the current study.

Moreover, the review process involved manually reading and categorizing each paper to develop detailed codes, allowing for a nuanced analysis of the literature. While more automated approaches to systematic reviews could have incorporated a broader range of sources, such methods may lack the precision and depth achieved through manual categorization. Future research could explore the use of automated methods to include key conference papers and older foundational studies, providing a more comprehensive understanding of the field’s development over time. However, for this review, our focus on recent journal publications, combined with an in-depth manual analysis, was necessary to provide a manageable and focused examination of the most current trends in XAI.

In summary, while these limitations—namely, the reliance on a single database, language restrictions, the specific timeframe, and the focus on journal articles excluding conference papers—are noteworthy, they were necessary to manage the scope and ensure a focused and feasible review process. Future research could address these limitations by incorporating multiple databases, including non-English studies, expanding the temporal range to include older foundational work, and considering a broader set of sources, such as conference papers. This approach would provide a more comprehensive overview of the literature on XAI and its development over time. Finally, it is important to highlight that the field of XAI is rapidly evolving. During the course of conducting and writing this review, numerous additional relevant articles emerged that could not be incorporated due to time constraints. This underscores the dynamic and ongoing nature of research in this area.