Human-Centered AI for Decision Support Systems: A Systematic Review of Application Domains, Architecture Designs, Current Trends and Future Directions

Abstract

Artificial Intelligence is increasingly used to support decision-making across many domains. However, concerns related to transparency, reliability and human oversight indicate the need for improved human-centered AI (HCAI) approaches in decision support systems (DSSs). In this paper, a systematic review was conducted in accordance with PRISMA 2020 in the Web of Science database: ninety research articles published between 2015 and 2025 were analyzed to investigate how HCAI is applied within DSSs in multiple application domains. HCAI + DSS research outcomes were analyzed and explored, first identifying the main architectural designs and discussing the involved components integrating human interaction, generative AI models, data and knowledge management, decision logic, and orchestration mechanisms, then focusing on specific domains and highlighting impact achieved, technologies used, and validation strategies employed. In addition, alignment with United Nations’ Sustainable Development Goals (SDG) was considered, and the temporal evolution of the most relevant topics was studied to identify more interesting trends and less investigated areas. Finally, findings were summarized, current limitations were discussed, and future research directions for helping researchers and practitioners in developing more reliable, explainable, and human-aware decision support systems were outlined.

1. Introduction

Use of artificial intelligence (AI) is increasing in many application domains. Global investment in AI reached $252.3 billion in 2024, with private investments up 44.5% from the previous year. The AI sector has seen strong growth over the past decade, with total investment increasing more than thirteenfold since 2014, when investment amounted to $14.57 billion. A survey conducted in 2024 by McKinsey & Company showed an increase in the global use of AI by organizations in all regions compared to previous years: North America remains the leader in organizations’ use of AI (82%), Europe stands at 80%, recording an increase of 27 points, and China ranks third (75%), recording an increase of 27 points [1]. These numbers reveal that AI is increasingly present in today’s societies and companies; administrations and private citizens have shown growing interest in taking advantage of AI. Various AI algorithms are used to support human decisions in various tasks in different fields, such as healthcare and clinical decision-making [2], industry [3,4], environment [5], education [6], mobility [7], security [8], law enforcement [9], and commerce and finance [10].

AI offers significant advantages in decision-making processes: in addition to identifying patterns and correlations that are not immediately visible, it can manage large amounts of data, explore broader solution spaces, and increase efficiency and automation [11], with economic benefits as well, reducing dependence on human experts for repetitive or standardizable tasks. However, it has significant limitations, including inaccuracies and possible hallucinations [12], bias and discrimination [13], risks of amplifying inequalities [14], social distrust of fully automated systems [15], and legal issues related to liability in the event of errors [16]. Therefore, the growing presence of AI in decision-making processes raises social challenges such as excessive reliance on AI-based automated processes, limited transparency of predictive models, and concerns about the impartiality of decisions.

These challenges motivate a shift toward human-centered AI (HCAI), an emerging paradigm that considers AI as a partner to humans, designed to support and improve human capabilities rather than replace them. In this perspective, HCAI puts people at the center of AI development, prioritizing needs, values, and user experience throughout the system’s lifecycle [17]. HCAI systems are designed to collaborate with users (human in the loop) by supporting instead of replacing them to improve the quality and reduce the efforts of human decision-making activities [18]. This approach naturally ties in with the concept of decision support systems (DSS), namely interactive information systems designed to assist human decision-makers through data-driven suggestions grounded on concrete evidence, models, and analysis, promoting more informed and motivated decisions [19].

Building on this motivation, the following research questions have been defined:

(RQ1) In which domains have HCAI approaches been applied to DSS? (RQ2) What architectural designs, layers, and core functional components characterize HCAI + DSS systems? (RQ3) Which AI methods, human-centered interaction mechanisms, and validation strategies have been most frequently adopted in HCAI + DSS studies?

In order to answer the research questions, a review of the state of the art was conducted in accordance with the PRISMA 2020 guidelines [20]. The completed PRISMA 2020 checklist is provided in the Supplementary Materials as Table S1. (The review was not registered, and no formal review protocol was prepared. This absence of prior registration and of a formal written protocol is acknowledged as a methodological limitation. To address this limitation, the review process was based on predefined research questions, explicit search terms, eligibility criteria (reported in accordance with PRISMA 2020), independent screening by two reviewers, and an assessment of the full text of the selected studies.) The literature search was conducted on the Web of Science [21] on 20 November 2025 and covered publications dated from 1 January 2015 to 30 October 2025. The search strategy was based on two groups of keywords combined with the Boolean operator AND, one related to HCAI (“human-centered AI”, “human-centred AI”, “human-computer interaction”, “human-AI interaction”, “human in the loop”, “human-in-the-loop”, “human on the loop”, “human-on-the-loop”) and one related to DSS (“decision support system”, “decision-making”, “expert system”, “recommendation system”, “collaborative decision”, “AI-assisted decision”, “decision aid” and “DSS”). The study selection process followed a structured manual screening procedure and is summarized in the PRISMA 2020 flow diagram shown in Figure 1. The initial search returned 1979 results; after limiting the results to research and review articles and removing 1 duplicate, 1115 records were retained for screening. Articles were considered eligible if they addressed the intersection between HCAI and DSS and were relevant to the research questions. Because the search query returned a large number of relevant studies, the selection was refined through manual relevance screening by two independent reviewers, giving priority to papers published in Q1 and Q2 journals according to the SCImago Journal Rank index [22], especially when multiple studies addressed closely related topics. Citation visibility in Web of Science was considered only as a supporting element and not as a standalone inclusion criterion. After title and abstract screening, 999 records were excluded. The remaining 116 reports were assessed in full text, and 26 were excluded because they were outside the scope of the review, reported information already covered by other included studies, or had been retracted. A final set of 90 studies was selected for the survey.

For each of the included articles, information regarding the application domain was extracted. Articles were qualitatively summarized and organized in nine application domains (healthcare and clinical decision-making; mobility and transportation; industry and production; environment, climate and agriculture; energy management; governance and public administration; economy; smart living and infrastructures; and safety, security, defense and space), highlighting their impacts, used technologies, and the practices used for validation. Finally, a cross-domain analysis was conducted to identify architectural patterns, the societal challenges addressed, and the benefits enabled by these systems in relation to the United Nations Sustainable Development Goals (SDGs) within the 2030 Agenda [23]. The complete list of the 90 studies included in the review, together with their domain classifications, SDG associations, tasks, methods, reported impacts and validation strategies, is provided in Appendix A.

In order to ensure traceability of the research questions throughout the review process, they were operationalized as coding fields and synthesis outputs, as reported in

Table 1. Traceability between research questions, extracted variables, and manuscript outputs.

Research Question	Information Analyzed	Outputs	Main Evidence
RQ1	Application domain, specific use context, impact, and connection with the SDGs.	Section 3, Figure 4, Figure 5, Section 4, Table 4	Identification of 9 main application domains, distribution across the reviewed literature, reported impacts, and connection with the SDGs.
RQ2	Dominant architecture level (a–e); presence of Human Interaction, X-Generative, Data & Knowledge, Decision Logic and Orchestrator layers; feedback-loop type.	Section 2, Figure 2, Figure 3, Table 2	Definition of the layer architectural taxonomy and mapping of reviewed systems to architecture layers.
RQ3	The AI methods adopted, the human-centered and explainability approaches used, and the way each system was validated.	Section 3, Table 3	Cross-domain synthesis of AI methods, explainability and validation practices.

.

This paper focuses on the following main contributions:

• Provide an updated and comprehensive (although not exhaustive) overview of the research literature on the exploitation of HCAI in DSS (HCAI+DSS) on multiple domains, highlighting the impact of proposed solutions, the employed techniques, and the practices used to validate them.
• Provide a layer-based architectural overview of HCAI systems: a multi-layer interactive architecture with loop-feedback has been identified, summarizing key functionalities, enabling technologies, and typical implementations of HCAI + DSS systems.
• Provide a mapping of the main HCAI + DSS literature contributions with the SDGs to highlight which societal priorities have been addressed and where further studies are required to fill research gaps. Additionally, the temporal evolution of the most relevant topics has been analyzed, identifying more studied subjects and less investigated areas.
• Present a comprehensive discussion on current limits and future directions of HCAI to provide researchers with useful insights and opportunities to guide forthcoming works.

The rest of the paper is organized as follows. In Section 2, a classification of possible HCAI + DSS architectures is provided, describing the different layers and their characteristics and responsibilities. Section 3 presents state-of-the-art HCAI + DSS solutions, organizing them into nine application domains. In Section 4, the impact of HCAI + DSS with respect to the SDGs is discussed. Section 5 discusses the evolution of selected keywords over time. Current limits and future directions are presented in Section 6. Finally, Section 7 provides the conclusions of the paper.

2. Architecture

HCAI + DSS systems rely on a multi-level architecture integrating human and computational intelligence. Unlike fully autonomous AI systems, they keep humans in the loop to ensure controllable interaction by offering comprehensible, reliable, safe, and trustworthy operations grounded in auditability and oversight [24]. In Figure 2, a graphical overview illustrates the evolution/classification from simple AI-based DSS to more complex HCAI + DSS architectures.

The simplest AI-based DSS is shown in Figure 2a. In this case, a Human Interface (HI) layer is used to query and provide input to a Generative layer to retrieve models and other possible data from a Data layer. Here, the term generative refers to an epistemological classification encompassing all the systems able to produce detections, predictions, reconstructions, or more generally, results that are not guaranteed to be true a priori. At this level, the Generative layer is usually based on black-box technologies like neural networks. The Generative layer executes the tasks and returns a response to the user through the Human Interface (HI). An example could be the analysis of medical images in which the system is asked to detect the presence of a disease and respond with a binary output (yes/no).

A slightly improved architecture introduces explanatory capabilities extending the Generative layer into a new layer, renamed X-Generative (see Figure 2b). This can be achieved by implementing XAI (eXplainable AI) techniques to explain the results of the neural architecture or including symbolic solutions like numerical [3] or agent-based models [25], even in a hybrid framework [26], starting to transition toward neuro-symbolic AI. With such architecture, the output is enriched with additional information, like localization of relevant features, description of causes leading to outcomes, confidence scores, etc., and provided to the user through explanation interfaces and ways to communicate uncertainty [27,28]. The user is still out of the loop; he/she only receives a wider range of information on which more informed decisions outside the system can be taken.

The user begins to be included in the system/loop with the architecture represented in Figure 2c: a new interaction between the HI and the Data and Knowledge layer is added to model the possibility for the user to provide feedback on system outputs or to augment the system knowledge dynamically. Referring to the example provided in case (a), the physicians, after obtaining the output, can manually check the results and provide feedback indicating whether the output is correct, i.e., confirming or correcting the prediction/detection. Such information/feedback is collected in the dataspace, creating a de facto new example that may be used for retraining or fine-tuning the AI model in a continuous learning approach often implemented via refinement with human-in-the-loop as active learning and continuous learning [29].

Note that the above-presented cases (a), (b), and (c) can provide solutions to relatively simple tasks such as single iterations of what-if analysis or answers to questions like: “Does this image show a disease?”, “How many cells are visible in this image?”, “Do these two proteins interact?”, “How will the traffic congestion change on this new road graph?”, “How are the prices of these stocks expected to change?”. On the other hand, the described systems cannot address complex tasks like: “I have this problem, with these constraints. Find me possible solutions.”, which include actual suggestions of prescriptions.

To allow the system to be able to work on more complex tasks, new layers and interactions are required. Figure 2d shows an additionally improved architecture including a Decision Logic layer that extends the decision pipeline beyond pure explanation (description). This new layer is responsible for scoring the results of the X-Generative layer, providing feedback to improve the models and producing prescriptions. It can be implemented by exploiting symbolic constraints, rules, optimization-based checks, or structured decision processes, keeping the decision logic justifiable [30], thus implementing a full neuro-symbolic AI system. Moreover, the Decision layer can also introduce changes in the problem scenario by varying constraints, objective weights (reward/utility), or environment parameters, thus creating alternative versions as inputs for the X-Generative layer. The Decision layer can be instantiated as a critic evaluating candidate actions with the aim of identifying the most valid solutions/prescriptions to the user’s problem [31]. Explanations are still produced by the X-Generative layer (e.g., saliency, confidence, counterfactual candidates) and exposed through the Human Interface to make the Decision Logic’s trade-offs, active constraints, and why/why-not choices inspectable [27]. Note that the user is kept in the loop: once the iterations among Decision Logic, Data and Knowledge, and X-Generative layers reach a stable set of solutions, the user is still able to assess the output and provide feedback.

The final stage of HCAI-DSSs is illustrated in Figure 2e, in which the system is evolving toward an agentic paradigm to achieve improved robustness across a broader range of problems. Multiple X-Generative and Decision Logic layers are made available as exploitable tools, each one tailored for addressing a specific task, while several Data and Knowledge layers are at disposal as resources. This approach requires the introduction of an Orchestrator layer in charge of understanding the user requests, selecting the right tool or resource to call at the right time, enabling the execution of a chain of simple tasks to solve complex problems via modular workflows [32], adapting its behavior according to the received outputs. Note that the user can be interviewed by the Orchestrator to collect additional input to dynamically drive the production for the best solution. For example, suppose a city officer must design new public transport lines and scheduling since in the near future, part of the city will be blocked due to some road work. After defining the area of interest and specifying the roads that will be closed, it asks the system to provide novel public transport plans. The Orchestrator decomposes the request into subtasks (e.g., demand forecasting or scenario simulation), calls the appropriate generative/decision layer, and iteratively queries the user to refine objectives. It then returns a ranked set of alternative plans with estimated Key Performance Indicators (KPIs), together with explanations of the trade-offs and constraints supporting the selected solution. Where task allocation is relevant, the orchestration also enforces role assignments and calibrates autonomy levels based on interaction outcomes [33].

After presenting this final and more complete architecture, in the next section, the five introduced layers will be discussed in more detail in terms of functionalities, technologies involved, and how they are typically implemented in practice, with explicit reference to state-of-the-art literature.

To provide empirical support for the proposed five-layer taxonomy in Figure 3, the distribution of the considered works classified in one of the presented architectures is reported. Note that 42 out of 90 papers do not provide information on a specific architecture, being, for example, surveys, behavioral studies, or legal and ethical analysis. The remaining 48 papers were classified as: 7 (14.58%) in level 1 (Figure 2a), 7 (14.58%) in level 2 (Figure 2b), 7 (14.58%) in level 3 (Figure 2c), 18 (37.50%) in level 4 (Figure 2d), and 9 (18.75%) in level 5 (Figure 2e). The distribution shows that most implemented architectures are classified as level 4, corresponding to systems in which AI/model outputs are transformed into recommendations, rankings, prescriptions, plans, or selected alternatives through explicit decision logic. Conversely, each architecture of levels 1 to 3 was found in a smaller number of papers, demonstrating that simple and straightforward solutions are less used in recent HCAI + DSS systems. On the other hand, some papers implement a more advanced architecture with a specific orchestrator; however, it is important to notice that, among the 18 papers, only 2 [32,33] propose systems that exploit large language models (LLMs) following more recent developments of agentic AI.

2.1. The Five Layers of HCAI Systems

2.1.1. Human Interaction Layer

This layer represents the contact between the user and the system. It enables users to provide input, receive recommendations, and interpret outputs. Typical technologies include human–computer interaction (HCI) interfaces and dashboards [34,35], allowing interaction with natural language [36] and including solutions to present XAI insights [37]. Implementations of such technologies encompass the design of interactive visual workspaces to explore model outputs and decision variables [38,39], as well as the development of dedicated explanation interfaces that expose “why/why-not” through interactive explanations (e.g., counterfactual “what-if”, saliency-map) [27,40]. In [28], the use of uncertainty communication patterns, specifically, how uncertainty is measured and presented to support calibrated reliance, has been addressed. In contrast, a decision interaction pattern, such as requesting additional advice and comparing recommendations to reduce over-reliance, has been proposed in [41]. In [42], a solution to include human interaction by monitoring the operator state and adapting prompts or interventions has been presented. Note that explanations are treated as a cross-layer capability: they may be generated by the X-Generative or Decision Logic layers and exposed to the user in this layer [43,44].

2.1.2. X-Generative Layer

This layer represents computational intelligence and implements models that generate predictions, diagnoses, detections, simulations, and reconstructions. This layer is typically built by exploiting machine learning (ML), deep learning (DL), reinforcement learning (RL), reasoning systems or Bayesian networks, agent-based models (ABM), and numerical models. For example, in [45], the authors have proposed DL predictive models to map inputs to outputs used in decisions. Graph-based or structured models paired with XAI techniques to make them explainable have been proposed in [27], while in [42] and in [46], policy-learning and reinforcement-learning solutions have been used to adapt recommendations or autonomy levels based on interaction outcomes. Multi-stage model stacks, in which different AI modules are leveraged to produce intermediate outputs, successively combined to provide a final result, have been instead proposed in [25,32].

2.1.3. Data and Knowledge Layer

This layer manages structured and unstructured data and semantic knowledge to support decision-making. It is usually implemented with technologies such as data warehousing, knowledge graphs, ontologies, and data lakes. Works in the literature have proposed heterogeneous data ingestion pipelines by combining different input modalities (e.g., images, logs, sensor streams) into a single data workflow [45,47], and solutions capable of handling streaming and real-time data to keep system state updated as new observations arrive [47]. In this context, Internet of Things (IoT) data acquisition plays a crucial role in continuously collecting and making available to the DSS real-time signals produced by users and external systems [48]. However, data availability alone is not sufficient for trustworthy AI: integrity, provenance, and secure sharing of the information used by downstream AI and decision-logic components are also required in the Data and Knowledge layer. Blockchain and distributed ledger technologies address these requirements through tamper-resistant records, decentralized data validation and smart contracts, reducing the risk of data manipulation in HCAI + DSS workflows [49,50]. Moreover, the digital twin paradigm has been exploited [51,52] in order to keep a “live” entity representation, created on historical data and continuously updated through real-time input [31]. Efforts focusing on dataset curation and labeling workflows have appeared, such as, for example, label acquisition and iterative dataset refinement based on human input [29], or inclusion of uncertainty metadata to enable downstream components to work reliably [28]. In [53], a knowledge-based retrieval solution combined with updating mechanisms has been used to refine stored information, like cases and features, via user’s feedback.

2.1.4. Decision Logic Layer

This layer combines rules, decision-making logic, and multi-criteria analysis to evaluate results and produce recommendations, prescriptions or even complex new scenarios. Decision trees, multi-criteria decision-making, optimization models, collaborative scenario construction [54], domain-related constraints [55], reward/utility approaches [44], and environment parameters [43] are all technologies that can be used to implement the Decision Logic layer. For example, the optimization of constrained objective functions to translate model outputs into recommendations has been proposed in [30,56]. In contrast, in [57,58], the authors have proposed automated planning solutions to generate alternative action sequences and scenarios. The use of interpretable decision structures to keep decision logic inspectable and justifiable (e.g., rule-like or structured decision models) has been discussed in [59], while in [32], decision frameworks that explicitly structure decision-making into stages have been introduced.

2.1.5. Orchestrator Layer

This layer orchestrates the system and is the heart of the agentic architecture (see Figure 2e). It is in charge of interpreting and decomposing the user’s request into simpler subtasks to achieve the final goal. On the other hand, it interacts with all the other layers, exploiting them via application programming interfaces (APIs) and communication protocols. In this way, the Orchestrator is able to reach tools and resources locally or remotely, including humans, thus improving the feedback loop. The Orchestrator invokes the right tool/resource (among those at its disposal) to complete each subtask, possibly in several steps. According to the implemented policy, this layer can move over a fixed and well-defined order of steps or dynamically and independently decide which step to execute depending on the outputs progressively received by the tools.

According to the identified literature, the agentic architecture has not yet been fully implemented into HCAI+DSS; nevertheless, solutions moving toward such a paradigm have been proposed. For example, a modular orchestration via workflows connecting a user interface (UI), data, models and decision logic is proposed in [32]; a task assignment coordinating execution and responsibilities for autonomous robots mission planning is described in [58]; context-aware module selection to choose explanation methods based on context/constraints is discussed in [60]; and a multi-agent aggregation for group decisions, in which preferences and criteria are negotiated to reach a consistent decision, is presented in [61]. Similar orchestrator-like patterns are emerging in industry, for example, in agent-based digital payment systems, where AI agents can carry out transactions on behalf of users, demonstrating the practical feasibility of the coordinated use of these tools in regulated environments [62]. More recent evolutions of agentic AI systems leverage the emergent capabilities of LLMs to decompose tasks into subtasks, coordinate workflows, and decide when external tools are needed for specific activities; they also have a strong capability to understand natural language and user intention, which is fundamental for human–computer interaction. Nevertheless, the exploitation of LLMs is still challenging, since such models lack grounded knowledge, can produce hallucinations, and have limited long-term memory. Moreover, by not relying on explicit symbolic reasoning, they offer limited interpretability. In addition, technical and economic barriers can impose limits on the adoption of LLM-based agentic AI. While foundation models avoid the need for training, they require expensive hardware for on-premises deployment. On the other hand, relying on LLM-as-a-service via paid APIs can still be financially prohibitive and can raise data privacy concerns. To address these problems, further research is required. On the one hand, there is a need for studies aimed at reducing the size of LLMs without producing excessive degradation of performance. On the other hand, neuro-symbolic approaches can be further investigated to ground the LLM knowledge in accurate and factual information, like, for example, graphs describing decision-logic schemas to guide task resolution, making LLM-based agentic systems more reliable and explainable.

Other solutions, not classifiable as agentic architectures, are more focused on coordinated interaction for continuous learning. Active learning and bias-aware adjustments have been implemented, exploiting human feedback loops to update models or sampling strategies: the model selects high-value examples, asks humans to create new annotations, and then uses the newly labeled data to retrain itself, progressively improving training data quality over time [29,63]. Moreover, by leveraging online and continuous adaptation, policies or allocations can be updated as new outcomes and feedback arrive [56,64], also implementing closed-loop system updates where a “live” system representation is continuously updated and used for decision/control [31,47]. In [46], fine-tuning loops for shared autonomy or automatic calibration have been proposed, exploiting functional allocation between humans and robots (who controls which part of the task), while the usage of deployment iteration practices, performance monitoring, and data/model choice revision over time based on operational experience has been proposed in [65].

Table 2. HCAI + DSS: Five-layer architecture summary. For each layer (Human Interaction, X-Generative, Data and Knowledge, Decision Logic, and Orchestrator), the table reports the main role, typical inputs, typical outputs, and enabling technologies.

Layer	Main Role	Typical Inputs	Typical Outputs	Enabling Technologies
1. Human Interaction	Interface between users and the system; captures input, presents recommendations and explanations, and enables structured feedback.	Natural-language queries, parameters, constraints, preferences, corrections/labels, confirmations.	Recommendations, dashboards, “why/why-not” explanations, clarification questions, captured feedback.	HCI/UI dashboards, natural-language interfaces, explanation UIs, uncertainty communication patterns.
2. X-Generative	Computational intelligence that produces estimates and candidates (predictions, diagnoses, simulations, reconstructions, recommendations).	Data/features, environment state, task specification/prompts, (soft) constraints, context.	Predictions, candidate plans/actions, scenarios/simulations, uncertainty estimates, explanatory artifacts.	ML/DL models, RL/policies, Bayesian, planning engines, agent-based models, numerical models, hybrid neuro-symbolic methods.
3. Data and Knowledge	Manages structured/unstructured data and semantic knowledge; supports continuous updates and quality/uncertainty metadata.	Sensor/log streams, datasets, documents, knowledge graphs/ontologies, human labels, realized outcomes.	Curated datasets, feature stores, updated knowledge graphs, system state, quality and uncertainty metadata.	Data lake/warehouse knowledge graphs/ontologies, streaming platforms, IoT ingestion, digital-twin synchronization.
4. Decision Logic	Evaluates, filters, and ranks candidates; enforces constraints; optimizes trade-offs; produces justifiable recommendations and alternatives.	Candidate outputs from X-Generative, hard/soft constraints, objective weights, policies, rules.	Scores/rankings, selected solutions, alternatives, scenario/constraint adjustments, decision rationale.	Optimization, multi-criteria decision-making, rule/constraint systems, constraint solving, automated planning, interpretable decision structures.
5. Orchestrator	Decomposes complex tasks, selects tools/resources, manages iterative workflows, calibrates autonomy/roles, and closes feedback loops.	User request, system state, intermediate outputs, orchestration/assignment policies.	Executed workflows, tool calls, user queries for refinement, final ranked solutions with KPIs.	Workflow engines, tool/API calling, policy graphs, monitoring and continuous-learning hooks, deployment pipelines.

shows the main roles, typical inputs and outputs, and enabling technologies for each layer that has been discussed.

3. Review of HCAI + DSS Solutions

There is a wide range of research literature regarding the application of HCI and HCAI in DSS contexts. Following the queries defined in the Introduction, a total of 1979 articles, published from 2015 to 2025, were initially found. In order to provide an overview of the growing interest around these topics, the temporal evolution of the above-mentioned papers (grouped by year of publication) is reported in Figure 4. It is worth noting that the count for 2025 refers to publications indexed up to 30 October 2025, since the search was performed on that date, in advance of the actual analysis, which took time to complete.

On the other hand, many different approaches have been proposed in the literature for classifying HCI frameworks and DSS solutions across a variety of application domains. For this reason, we focused the presented study on the main application domains emerging from the literature. After careful screening, a total of 90 research and review articles on HCAI + DSS have been considered as a reference basis for the survey presented in this paper. For each paper, the following aspects have been identified: the application domain, the reference subdomain, the claimed impact, the techniques adopted for the implementation of the HCAI approach, and the validation strategy used to assess the proposed system (e.g., user studies/experiments, real-world case studies, simulation-based evaluations, or offline benchmarking). In Figure 5, the identified HCAI domains with their subdomains and application areas are graphically presented. On the other hand, Figure 6 shows the number of papers identified in the literature for each application domain. In the following subsection, each domain is separately discussed.

Explainability was considered as a cross-domain evaluation criterion. In this review, technical explainability refers to model- or algorithm-level mechanisms, such as saliency maps, counterfactuals, uncertainty estimates, SHAP explanations, etc. Functional explainability refers to the extent to which explanations are understandable and actionable for the intended decision-maker in the workflow. This distinction follows recent work stressing that explainability should be assessed both as a system property and in terms of human understanding and decision usefulness [66,67,68,69].

Table 3. Domain-level explainability criterion used to interpret the reviewed HCAI + DSS systems.

Domain	Technical Explainability	Functional Explainability
Healthcare and clinical decision-making	XAI interfaces, counterfactuals, uncertainty communication and interpretable clinical visualizations.	Evaluated through trust, reliance, cognitive load, acceptance and clinician-facing workflow integration.
Mobility and transportation	Collision-prediction logic, cockpit decision support, shared-control rationale and simulation traces.	Assessed through simulated understandability, workload and pilot/operator feedback; field evidence remains limited.
Smart industry and production	Fault rationale, interpretable resource planning, operator-state monitoring and visual analytics.	Related to control-room workload, situational awareness and operator intervention.
Smart environment, climate and agriculture	Visual thinning simulations, predictive irrigation models, scenario analysis and human-model interaction.	Often domain-expert-facing; transfer to non-expert stakeholders is less systematically evaluated.
Smart energy management	Causal models, what-if simulations, digital twins and immersive emergency visualization.	Relevant in high-risk settings, but usually validated in proof-of-concept or simulated settings.
Smart governance and public administration	Algorithmic transparency, oversight mechanisms and accountability-oriented explanations.	Mainly institutional rather than only individual; human oversight alone may not be sufficient.
Smart economy	Context-aware XAI selection, business-oriented explainability and trust/responsibility studies.	Evaluated through managerial trust, delegation, perceived responsibility and business decision usefulness.
Smart living and infrastructures	Personalized interaction models, learning-style classification, UI optimization and workload-management feedback.	Related to personalization, feedback and user self-regulation; evidence remains heterogeneous.
Safety, security, defense and space	Relevance feedback, forensic decision support, mission constraints and Cognitive Systems Engineering.	Related to accountability and safety under high uncertainty; empirical validation is often scenario-based.

shows the domain-level explainability criterion used to interpret the reviewed HCAI + DSS systems.

3.1. Healthcare and Clinical Decision-Making

From the analysis of the works in this domain, a predominant focus in the use of HCAI + DSS techniques in clinical decision support systems (CDSSs) is diagnostic support, with systems supporting diagnostic decisions in various clinical settings through the analysis of images [48,69] or physiological signals [70,71]. Additional aspects identified include prognostic support for assessing severity and clinical outcomes [56,72], decision support for triage and patient management [73], systems supporting treatment- and medication-related decisions [45,69], and DSS for promoting healthy lifestyles and nutrition (e.g., healthy food recommendations in the workplace) [74].

The impact of these systems consists of improving the accuracy and timeliness of diagnosis [70,71] and triage or timeliness in urgent decision support [56] while reducing potential clinical errors, supporting complex clinical decision-making [46,69], and improving patient safety and overall quality of care, even in highly critical contexts [73]. In addition, AI systems are made explainable to increase user trust in their use [75,76,77,78], and heterogeneous biomedical data and multiple signals are integrated and visualized to support clinical interpretation [70,72,79] while enabling the development of high-performance AI tools that can be effectively integrated into clinical workflows [39,74].

Regarding the techniques used, the main approaches include ML and DL on images [48], physiological signals [45], Electronic Health Records (EHRs), high-volume clinical data with XAI techniques (trust/reliance [75], uncertainty [80], pragmatic embedding [28], XAI interfaces [71]), Knowledge Discovery in Databases [81] or HCI interfaces guided by usability principles (ISO 9241-11 [82], user-centered design, usability studies) [39,70], exploiting augmented reality, natural user interfaces, mobile devices [39], and telehealth [71].

Regarding system validation, studies have been conducted with doctors, nurses, pharmacists, therapists, and sometimes non-expert people, using trust/reliance measures in real cases (trust in CDSS and AI) [28,75] and in explainable clinical support settings [71], as well as cognitive load and workload measures (mental workload, interaction fluency) [39,80], and measures of satisfaction and acceptance of the system (perceived trust, usefulness, intention to use) [69,74].

3.2. Mobility and Transportation

In this domain, the reviewed studies show that HCAI + DSS techniques are used in several mobility and transportation contexts. These include logistics and urban transport such as fleet composition problems [30], the routing and scheduling of intelligent autonomous vehicles in industrial logistics systems [30], and shared control and driver–automation cooperation in lane-keeping assistance systems [83], as well as remotely piloted aircraft systems (RPAS) or unmanned aerial vehicles (UAVs) for operator interaction and supervisory control [84], and for decision support in mission planning and operator interaction [58]. In the RPAS domain, Automatic Dependent Surveillance–Broadcast (ADS-B)–based separation assurance and collision avoidance systems are validated through real-time hardware- and human-in-the-loop simulations to support both fully automatic operations and remote pilot decision-making [85], aviation applications including pilot decision support for diversions and continuous versus recommendation-centric support [86], and cockpit layout evaluation [87]. Note that keywords such as cooperative, connected and automated mobility (CCAM) and connected autonomous vehicles (CAVs), even if able to exploit AI techniques, are not usually related to DSS; thus, they did not emerge from the literature selection.

For the impact, the proposed systems aim to improve safety in RPAS operations [85], support pilot decision-making in aviation contexts [86], and improve operational efficiency in urban freight transport [30] while supporting pilots in complex decision-making situations [87] and reducing cognitive load through driver–automation cooperation [83], enhancing human supervision and decision-making over UAV mission plans [58] and supporting automatic separation assurance and collision avoidance in RPAS operations [85].

Technologies adopted by these systems are based on the following: for fleet composition and routing, mathematical optimization algorithms and stochastic models [30]; for planning and simulation, interactive interfaces [58]; and for “forward” vs. “backward” AI support in cockpits [87], hybrid human–automation control algorithms (shared control, driver–automation cooperation) [83] and the integration of ADS-B data with collision-prediction logic for RPAS [85].

Regarding validation, studies have been conducted with professional pilots in simulated flight scenarios [86], or evaluations have been performed with users/operators on UAV scenarios in a simulated environment using QGroundControl and DSS [58], by measuring performance, or through subjective feedback (understandability, load) [83,87] and through real-time hardware- and human-in-the-loop simulations for RPAS separation assurance and collision avoidance [85]. However, simulation-based validation remains insufficient, as real-world factors such as sensor noise, communication delays, environmental variability, and operator stress can still affect system performance. Future studies on RPAS/UAV HCAI + DSS should therefore supplement simulations with controlled real-world testing and operational monitoring.

3.3. Smart Industry and Production

In this domain, a prevalent feature in the use of HCAI + DSS techniques is the development of industrial control rooms—for chemical processes [42] and industrial cyber-physical systems (ICPS) [88]—to support operators under multiple alarms and critical scenarios. They include decision-making frameworks integrated into the plant operational system (e.g., container terminals) [47], DSS for industrial processes [38], and interpretable systems for resource-allocation planning (e.g., crane planning) on construction sites [59], as well as shared-autonomy systems for industrial robots and high-level tasks [36].

In terms of impact, the solutions reviewed primarily aim to reduce the risk of human errors and accidents in complex industrial plants [42]. XAI is used for describing the rationale of the detected faults [89] while improving efficiency, productivity, and quality in industrial processes [47]. They also make AI models interpretable so that their results can be directly accessed by decision-makers, such as engineers and managers [59], enabling shared autonomy and adaptive human–robot policies under human supervision [46].

Regarding the techniques used, the main approaches include DL models [59], Hidden Markov Models, dynamic influence diagrams, and RL to monitor plant status [42], together with advanced human–machine interface (HMI) and visual analytics [47] and a hierarchical Markov decision process for adaptive human–robot policies [46].

For system validation, experiments have been conducted with process operators (including professionals), often in realistic control-room simulators [42]. Pilot studies were also conducted with users on robotic tasks [90], measuring performance and control preferences [42], alongside measurements of workload, situational awareness, and interaction styles (eye-tracking, questionnaires, interaction logs) [42]. In addition, performance and cognitive metrics were evaluated in operator studies involving AI-supported control room environments [42], while decision-support performance was assessed in industrial planning applications such as crane planning [59].

3.4. Smart Environment, Climate and Agriculture

In this domain, HCAI + DSS have been used for smart farming/Agriculture 5.0, sustainable crop, and soil management to support end-to-end agricultural processes [91]; predictive/adaptive irrigation integrating models, weather data, humidity sensors, and human preferences [43]; visual decision support for quantitative and visual forest thinning (2D/3D before/after simulation) [92]; human–model interaction frameworks for food safety and risk assessment (e.g., infant food) [54]; hybrid ML–geostatistical models for territorial/environmental applications like mineral grade estimation in mining, where human experts guide or validate hybrid ML–geostatistical ensemble models to improve prediction accuracy [93]; user experience (UX)-oriented demonstration prototypes for reducing the risk of disasters affecting communities through digital services [40] and DSS and agents for multi-stakeholder decisions in disaster management (e.g., strategic roads in floods) [61].

The impact of the reviewed solutions has been focused on increasing productivity and reducing waste of resources (water, fertilizers, energy) in agriculture [43,77], as well as improving food safety risk assessment by integrating models and expert knowledge [54] and simulating the effects of forestry decisions (e.g., thinning) through visual analysis [91]. In addition, the analyzed solutions have supported multi-stakeholder [94] decisions in environmental disasters [40], critical infrastructure planning [61], and decision-making in agriculture under environmental variability and complex farming conditions (irrigation, forest management, agricultural risk) [91].

The main adopted technologies have been knowledge representation and reasoning (KRR), including ontologies and knowledge graphs, and ML predictive modeling for agricultural processes and smart farming systems [91]. In addition, these systems adopt data-driven scenario analysis and stochastic/Monte Carlo simulation for food safety risk assessment [54] or control-oriented modeling and forecasting, up to model-based control/Model Predictive Control approaches [43]. Finally, they include multi-agent systems/agent-based modeling and negotiation-based decision support for multi-stakeholder decision-making [61].

Regarding system validation, the reviewed systems were evaluated with domain experts through analysis of indicators such as water savings in irrigation management [43] and forest stand structure in forestry decision support systems [92]. Additional validation was performed through real-world case studies, including applications to a sweet corn field in Florida [43], a Chinese fir plantation [92], and disaster management scenarios involving multiple stakeholders [61], while some works provided conceptual frameworks and research perspectives rather than empirical system validation [91].

3.5. Smart Energy Management

Based on the reviewed studies in this field, HCAI + DSS techniques are frequently applied to control rooms for electrical networks (critical infrastructure) [88] digital twins and integrated frameworks for photovoltaic and smart energy infrastructures [31]; and decision support for nuclear emergencies using Virtual Reality (VR) and causal models [95]. In terms of impact, these systems reduce errors and reaction times for operators in energy control rooms [88], while supporting complex decisions in high-risk (nuclear) emergencies through human-centered decision support tools [95] and increasing the resilience and security of critical energy systems [88], as well as for facilitating the control and understanding of complex photovoltaic systems through interactive digital twins [31].

The technologies used by these systems depend on the operating environment. The proposed systems rely on causal models, what-if simulations in a VR environment to evaluate emergency strategies [95], or the integration of simulators, Industrial Control Systems (ICS) testbeds, cyber-physical systems and cognitive metrics (eye-tracking) [88], and digital twins, physical models, and immersive visualization for control and training [31].

Regarding validation, studies were conducted with professional electricity grid operators or students, measuring performance, accuracy, and cognitive measures [88], or controlled experiments were conducted in simulated nuclear emergency scenarios with participants making decisions through VR [95], and proof-of-concept and initial usability studies for interaction with the digital twin [31].

3.6. Smart Governance and Public Administration

Based on the reviewed studies in this field, HCAI + DSS techniques are applied to several public-sector decision settings, including systems supporting decisions in criminal justice [96] and the assessment of defendants [97], urban management (e.g., identifying vacant/abandoned properties) [65], analyses of human oversight in welfare administration systems and automated decisions [98,99], and investigations of the role of plagiarism-screening tools in editorial decision-making [100], as well as policies on algorithms [101,102] in public administration and trust in public-sector algorithms [103]. Here, algorithmic governance, accountability, and human oversight are widely debated [98,99,101]. In particular, human-in-the-loop approaches are critically examined as a regulatory safeguard, especially with regard to their practical effectiveness [99,101].

For the impact, the proposed systems support complex decision-making in urban [65] and crisis management by balancing efficiency, fairness, and the protection of the most vulnerable [94]; help understand how AI errors influence human judgments and decision-making performance [96], including judgments about defendants [97]; how algorithms shape the decision-making space [100]; propose a shift from individual human oversight to institutional forms of oversight [101], and seek to ensure accountability, transparency, and democratic control over governmental algorithms [98,99].

Technically, these systems are based on ML classification methods [65], screening software [100], or trust in public-sector algorithms [103], human oversight of government algorithms [101], transparency and human oversight in algorithmic systems [102], and human oversight in automated decision-making [99].

Regarding validation, these systems have been evaluated by quantitatively comparing the performance of the model with and without human intervention and analyzing the differences in results [65], or through studies on oversight and policy (mainly conceptual and qualitative analysis, without numerical experimental validation [101,102]), and studies focusing on trust in public-sector algorithms [103] and human oversight in automated decision-making [98].

3.7. Smart Economy

In this domain, HCAI + DSS techniques are commonly applied to supply chain, demand forecasting, and business analytics. These include selecting XAI methods in business AI contexts [60,104]; tools supporting business innovation and human–machine collaborative decision-making [105]; decision support for IT leaders in project selection [106]; tax-advisory decision support using generative models [107]; and sentiment analysis systems across reviews, films, marketing contexts [108], and AI-supported decision-making in hospitality and tourism settings [109].

In terms of impact, these systems increase the effectiveness of data-driven strategies in business contexts (e.g., supply chain and marketing) [108] while maintaining interpretability and comprehensibility of decision-making models [60], supporting innovation (exploration/exploitation) through human–AI collaboration [105]; align the use of XAI and DSS with business objectives and constraints [60,104]; address trust in AI-supported decision-making [109] and questions of responsibility in AI-assisted decisions [110,111]; and examine when, and for which decisions, managers are willing to delegate to AI [107,109].

Technically, these systems are based on ML and XAI techniques with DSS supporting the selection of XAI techniques based on the context of use [60,104], as well as qualitative analysis of case studies [112], and discussions of practical business cases [113], and even experimental methods with hypothetical scenarios (surveys, simulated decision-making tasks) to analyze how people trust AI [109], accept its use in decisions [110] and perceive who is responsible for the decision (locus of causality/responsibility) [111].

Regarding validation, these systems have been evaluated by managers in simulated decision-making scenarios [105], IT leaders [106], hospitality professionals in decision-making scenarios [109] or by evaluating metrics such as decision-making performance measures [60] and trust, acceptance, and perceptions of responsibility in DSS [110,111], or even by analyzing case studies and practical business problems [112,113].

3.8. Smart Living and Infrastructures

HCAI + DSS systems are applied to a range of interactive systems. These include generative and interactive tools for urban design/urban aesthetics, with human-in-the-loop aesthetic evaluation [114]; monitoring of historic villages [115]; e-learning systems that classify learning styles via chatbots [116]; facial emotion recognition for human behavior analysis [108]; architectures for the metaverse/Web3.0 and human-in-the-loop RL for intelligent smart contracts [44]; frameworks for adaptive task assignment [64]; and DL systems for UI and personalized interaction design [117].

In terms of impact, these systems support complex urban design decisions (configurations, aesthetics) while keeping the designer in the loop [114], improving the management and use of historical/urban infrastructure through human-in-the-loop approaches [115], increasing the efficiency of distributed urban tasks [64], customizing UIs [117], and training courses [116] based on behavioral and cognitive styles.

The main technical approaches include generative algorithms and interactive evolutionary design with hybrid aesthetic evaluation [114]; RL (Deep Q-Networks (DQN) with attention) with human supervision to partition task streams [64]; deep learning for recommendations, layout optimization, and user interaction data analysis [117], traditional ML for learning-style classification [116]; and CNN + Hierarchical Convolutional Recurrent Neural Network (HCRNN) + Random Forest pipelines for facial emotion recognition [108].

Validation is reported via urban designers exploring urban design spaces [114]; experimental studies with students/learners [116]; tests on standard datasets with comparisons against other ML algorithms [108]; and usability and performance evaluations for UI recommendation and layout optimization [117].

3.9. Safety, Security, Defense and Space

In this context, HCAI + DSS techniques are applied to several mission- and safety-critical settings. These include space situational awareness (SSA) and decision support in space operations [55]; extravehicular activities (EVA) under strong safety constraints [118]; knowledge-based and relevance-feedback systems for fingerprint identification and other forensic applications [53]; and forensic and regulatory analyses of AI use for DeepFake detection and media forensics [119]. Related work also addresses DSS and simulation-based [61] support for disaster management scenarios [94]. These systems primarily ensure security and reliability in high-risk missions, such as space operations [55], extravehicular activities under strict safety constraints [118], and critical cyber-physical infrastructures [88], while reducing human error and increasing situational awareness in highly complex environments [88,118]. In addition, these systems aim to improve media forensics tools to address the risks of disinformation associated with DeepFakes and similar technologies [119] and aim to increase the accuracy and transparency of forensic tools (fingerprints [53], DeepFakes [119]) integrated into decision-making processes.

Regarding the techniques used, these systems are based on Cognitive Systems Engineering techniques for designing DSS in safety-critical environments [118] and on modeling of operational scenarios in space environments [55], as well as on data-driven relevance feedback approaches with visual analytics and adaptive knowledge bases for biometric systems (iterative customization and model updating with user input) [53] and frameworks for forensic analytics and regulatory/ethical assessments of AI systems in security (risk analysis and governance) [119], together with the integration of ICPS testbeds and simulator environments for monitoring and decision support [88].

Regarding validation of these systems, laboratory studies were carried out with operators in simulated EVA scenarios [118] or experimental evaluations of the accuracy of DSS for fingerprint identification [53]. Furthermore, in the case of SSA, additional conceptual validation was conducted through use cases rather than detailed numerical trials [55], while experiments were carried out with operators and measured performance metrics for cyber-power systems [88].

4. Contribution to the United Nations Sustainable Development Goals

In order to relate the literature reviewed on HCAI-DSS to broader social priorities, the United Nations Sustainable Development Goals (SDGs) of the 2030 Agenda were used as a reference framework, relying on the SDG classification provided by Web of Science (WoS) [120]. According to this classification, publications are assigned to one or more SDGs based on WoS internal SDG mapping criteria. For each SDG, we report both the total number of records returned by the initial WoS query before the study selection process described in Section 3 and the corresponding number of filtered studies. These results help identify research areas where more substantial contributions from HCAI + DSS may be needed. Out of 1979 total records, only 678 appeared to be related to at least one SDG. As a consequence, after applying these review eligibility criteria, out of the 90 selected papers, only 22 were found to be related to at least one SDG.

In

Table 4. HCAI + DSS works classified according to the United Nations SDGs. The first column reports the SDGs, the second column, the work found on WoS before filtering the results as discussed in Section 3, while in the third column, the number of papers after the selection process. The last column shows the references to the counted papers.

SDG	#WoS	#Selected	SDG Density (%)	Coverage Ratio (%)	References
SDG 1-No Poverty	7	1	4.55	14.29	[121]
SDG 2-Zero Hunger	17	1	4.55	5.88	[93]
SDG 3-Good Health and Well Being	308	8	36.36	2.60	[33,46,48,64,69,70,79,89]
SDG 4-Quality Education	64	1	4.55	1.56	[26]
SDG 5-Gender Equality	13	1	4.55	7.69	[122]
SDG 6-Clean Water and Sanitation	8	1	4.55	12.50	[43]
SDG 7-Affordable and Clean Energy	20	1	4.55	5.00	[26]
SDG 8-Decent Work and Economic Growth	4	1	4.55	25.00	[123]
SDG 9-Industry, Innovation and Infrastructure	75	2	9.09	2.67	[47,59]
SDG 10-Reduced Inequality	8	1	4.55	12.50	[121]
SDG 11-Sustainable Cities and Communities	144	4	18.18	2.8	[30,48,65,95]
SDG 12-Responsible Consumption and Production	66	2	9.09	3.03	[47,91]
SDG 13-Climate Action	30	1	4.55	3.33	[43]
SDG 14-Life Below Water	13	1	4.55	7.69	[43]
SDG 15-Life on Land	16	1	4.55	6.25	[40]
SDG 16-Peace, Justice and Strong Institutions	4	1	4.55	25.00	[124]
SDG 17-Partnerships for the Goals	13	1	4.55	14.29	[59]

, the works related to SDGs are mapped: for each SDG, the total number of papers found in the WoS records is reported in the second column, while in the third column, the number of papers remaining after applying the review eligibility criteria is shown. In the fourth column, the $SDG density$ among the selected studies is reported. This was computed as the percentage of selected papers associated with a given SDG relative to the total number of selected papers with at least one SDG assignment, i.e., 22 papers:

$${SDG density}_i={\displaystyle \frac{{\#}_{SD{G}_i}^{selected}}{{\#}_{SDG}^{selected}}}\times 100$$

where ${\#}_{SD{G}_i}^{selected}$ is the number of selected papers associated with $SD{G}_i$ and $N_{SDG}^{selected}=22$ is the total number of selected papers associated with at least one SDG.

To account for the information loss between the initial WoS records and the final selected corpus, we also considered the coverage ratio after filtering:

$${CoverageRatio}_i={\displaystyle \frac{{\#}_{SD{G}_i}^{selected}}{{\#}_{SD{G}_i}^{WoS}}}\times 100$$

where ${\#}_{SD{G}_i}^{WoS}$ is the number of initial WoS records associated with $SD{G}_i$. This second measure highlights the extent to which each SDG remained represented after applying the review eligibility criteria. Indeed, the transition from the initial WoS records to the selected papers shows a significant shift in SDG distribution. This skewness and the resulting reduction in paper counts are due to an unavoidable loss of information caused by the eligibility criteria used in this review. This analysis shows a strong filtering effect for high-volume SDGs, such as SDG 3 (8/308), SDG 11 (4/144), SDG 9 (2/75), and SDG 12 (2/66).

As can be observed, SDG 3 was the most recurrent (Good Health and Well Being), with 308 papers found on WoS, among which 8 were selected. For SDG 11 (Sustainable Cities and Communities), 144 papers were found on WoS, and 4 were selected; for SDG 9 (Industry, Innovation and Infrastructure), there were 75 WoS papers, and 2 were selected; and for SDG 12 (Responsible Consumption and Production), 66 papers were found on WoS, of which 2 were selected after filtering. Conversely, all the remaining SDGs exhibited extremely limited and flat coverage, being represented by only a single paper each after eligibility selection. In particular, the selected corpus was sparse regarding crucial equity-oriented goals, such as SDG 5 (Gender Equality) and SDG 10 (Reduced Inequalities). Therefore, the findings regarding the SDGs highlight two complementary aspects: the thematic concentration of current research on HCAI + DSS in the sectors of health, cities, industry, and manufacturing and the limited representation of social equity dimensions within the selected literature.

As expected, the SDGs with the most contributions are those related to the previously identified domains, for which a greater number of HCAI + DSS solutions have been proposed over the years. For example, health-related HCAI + DSS (SDG 3) often targets clinical decision support and monitoring, whereas smart-city, mobility, and planning systems (SDG 11) address urban services, transport, and infrastructure decision-making. Similarly, industrial and infrastructure settings (SDG 9) reflect optimization and operational decision support and resource and production-oriented applications, while SDG 12 emphasizes efficiency and responsible decisions.

It is also worth noticing that most of the works are classified with a single SDG, with only a minority among the selected papers having multiple SDGs: [43] associated with SDGs 6, 13, and 14; [26] with SDGs 4 and 7; [59] with SDGs 9 and 17; and [47] with SDGs 9 and 12.

5. Evolution of Keywords over Time

In this section, the temporal evolution of keywords has been analyzed. The most relevant technical keywords and key phrases have been extracted from the articles under consideration during the analysis. We decided not to limit the analysis to those provided by the authors in the papers, since they are usually focused on matching the location of the paper rather than the actual global trend and emerging keys. Figure 7 shows the temporal evolution: bubble size represents the number of papers in which the keyword is present, year by year, and color represents the proportion of papers mentioning the keyword, considering all the papers in the year. Such a distinction has been made to highlight both the absolute quantity of papers addressing specific topics and how the topic is relevant in different years. For example, decision-making was addressed by a greater number of papers in 2024 than in 2023, yet in 2023, the keyword was more relevant. Note that two derived keywords have been defined due to their importance: Human * groups the keywords human-in-the-loop, human-centered, and human interaction, while Decision * considers decision-making and decision support systems (DSS). This choice has been made to highlight concepts of human inclusion and DSS applications exploiting artificial intelligence processes.

Some terms show a stable pattern of occurrence: they appeared in both the early and late stages of the period covered by this review, or they remained present over time since their first appearance. These terms include AI and XAI, human-in-the-loop and human interaction, DSS and decision-making, as well as optimization, uncertainty, and reliance. Starting in 2020, the consistent occurrence of transparency and reliance highlights the importance of these topics in the field of AI systems. Over the years, other terms have emerged in HCAI systems, as they show a growing trend but account for a small percentage of the papers considered. These include RL, counterfactual explanations, digital twins, active and continuous learning, and, finally, hallucinations. Furthermore, topics such as neuro-symbolic AI, scenario generation, and agentic AI emerged only in the last year. A particular case is represented by the terms trust, human-centered and generative, which appeared as early as the first few years of the period considered and have been growing significantly in recent years. In contrast, the terms ontology/knowledge graph and multi-criteria have appeared only sporadically. Finally, automated planning, which was present in the early years, is missing in the later ones.

6. Discussion on Current Limitations and Future Directions

Current HCAI + DSS solutions increasingly combine user-centered interactions and data-driven methods to support complex decisions in real-world cases and are already applied across numerous domains. However, based on the analyses and reviews carried out in the previous sections, open challenges remain to be addressed in future work.

Many HCAI + DSS systems introduce explanations (e.g., textual or counterfactual explanations) to make recommendations more understandable, but this does not guarantee appropriate reliance: users may still rely too heavily on AI (over-reliance) or ignore useful suggestions (under-reliance), with potentially significant costs, as shown by [125]. The effect of explanations depends greatly on how the user interacts with the system: for example, counterfactual explanations can increase reliance on AI recommendations, but they do not necessarily prevent over-reliance [40]. Offering second opinions is not a unique solution either since, depending on the context, it can lead to both over-reliance and under-reliance [41]. Furthermore, the risk of over-reliance changes with the socio-organizational context (e.g., supervision in public administration) [98]. The reliance problem is even more critical when black-box advice may be underutilized or used in an uncalibrated manner in technical fields [126].

Another limitation that emerged from the studies analyzed is that HCAI + DSS solutions often present explanations as a standard system component, without considering the user, context, or risk, even though empirical studies. In [127], the authors show that these factors significantly influence human–computer collaboration. This motivates future work on adaptive and context-sensitive explanation policies, where explanations are customized to the user’s experience and the context of use. Furthermore, if the interface and workflow do not adequately make the user aware of the error, the user may not notice the error and therefore follow incorrect recommendations [96]. Many systems do not offer sufficient diagnostic and correction tools, and, as a result, having humans in the loop may not be sufficient to automatically guarantee safe systems.

In most HCAI + DSS systems, the role of humans is to accept or modify an AI suggestion only at the end, without promoting active collaboration or co-construction of the decision (human in the loop) [128,129]. Therefore, in domains such as healthcare, even with high-performance models, the decisions can remain difficult to understand. Interaction such as the manual counterfactuals implemented in [27] is promising; however, explaining is not enough if the explanation does not support the clinician’s (or more generally, the user’s) reasoning. A related direction is to define the concept of clear roles and rules for the human-in-the-loop in a more structured way, and therefore not based exclusively on approval. Current works fail to adopt recovery mechanisms in the design workflow (error detection, diagnosis, etc.) and to evaluate them on real-world case studies.

In summary, current research indicates that explainability alone does not ensure calibrated reliance, and although several insights have emerged in the literature, the field has not yet translated these insights into established design requirements for HCAI + DSS systems. Consequently, one future direction may lead to a shift from explanations as a characteristic of reliability engineering to the actual design of interaction policies that calibrate reliability in various decision-making contexts.

With regard to data, the limitations that emerge from the analyzed works concern cost and scalability of data enrichment processes [54,63], the variability introduced by the human factor in dynamic scenarios [43], dependence on experts and feedback mechanisms that are not always mature [53], and the gap in empirical validation of the real effectiveness of solutions [104]. This requires scalable methods for human enrichment of data and knowledge, with quality controls, bias mitigation, and more robust empirical evaluations in real-world case studies.

A further limitation relates to the training of AI models: many papers have not adopted versioning and machine learning operation solutions (MLOps) [130], tools (monitoring, controlled updates, etc.), which allow feedback from human-in-the-loop components to be integrated, with extended coverage of the MLOps process in the production and control phases [131,132].

Finally, HCAI + DSS systems face limitations that are not only technical: the transition from prototype to operational use raises challenges in workflow integration, roles and responsibilities, user training, and organizational acceptance, and scientific literature often underestimates deployment accountability issues in its assessments [80]. Future research should also consider HCAI + DSS applications for personalized management of employee workload and well-being. Meduri et al. [133] demonstrate that HCAI can contribute to burnout prevention when AI tools are combined with employee training, personalization, and regular feedback. This suggests that future DSS should not be limited to optimizing tasks or interfaces, but should also adapt recommendations to users’ workloads, stress levels, and working conditions.

Table 5. Mapping of HCAI-DSS architecture with the main limitations identified in the literature.

Layer	Criticalities
1. Human Interaction	Reliance Black box use Adaptive explanations Error awareness Diagnostic and correction tools Active collaboration or co-construction Clear roles and rules for the human-in-the-loop Real-world case studies
2. X-Generative	Reliance Black box use Clear roles and rules for the human-in-the-loop Exploitation of MLOPS Deployment accountability
3. Data and Knowledge	Real-world case studies Quality controls & bias mitigation
4. Decision Logic	Active collaboration or co-construction Clear roles and rules for the human-in-the-loop MLOPS
5. Orchestrator	Diagnostic and correction tools MLOPS Agentic AI

maps the HCAI + DSS architecture with the main limitations identified in the literature.

An emerging direction is to leverage LLMs and agentic AI to orchestrate multi-step decision-making workflows. This integration promises not only improved reasoning and contextual understanding but also more autonomous decision-making, adaptive collaboration, and proactive learning behaviors [134]. Although a growing body of literature already analyzes LLM-based agents and multi-agent systems, they are still rarely operationalized in HCAI + DSS contexts with human oversight and systematic evaluation of reliance calibration and accountability. In this sense, orchestration frameworks such as LangGraph [135], a low-level framework for building stateful and long-running agents, explicitly support human-in-the-loop supervision and allow users to inspect and modify the agent’s state during execution in order to approve or correct critical steps, making these frameworks particularly suitable for decision support settings in which human oversight is required.

The use of RL and DeepRL in DSS [42,136] is also linked to this trend, as it enables the learning of adaptive and sequential decision-making policies and strategies, consistent with multi-step agentic workflows and human intervention throughout execution. In particular, the evolution towards human feedback-guided learning brings RL from human feedback (RLHF) into focus. RLHF [137] provides a basis for human-centered AI in HCAI + DSS contexts, making agents more reliable partners in decision-making, capable of adapting, explaining and reasoning in accordance with human values, rather than acting as automatic tools [138]. In RLHF, static rewards defined by the environment are replaced by human feedback; in the context of DSS, this capability allows models to realign recommendations with human judgment. RLHF can therefore support a new generation of more accountable, interpretable, and adaptive systems, which are fundamental for reliable decision-making in different domains.

Future research in the field of HCAI + DSS may further explore hybrid models that integrate deep learning with swarm intelligence or optimization algorithms, as these approaches can improve decision-making accuracy in complex environments [139]. Recent studies have shown that hybrid AI–optimization frameworks outperform conventional models in solving multi-objective decision-making problems by combining global search capabilities with adaptive learning mechanisms [140].

At the same time, to make autonomy more reliable, composite architectures are emerging that combine learning and explicit knowledge. Recent perspectives suggest a shift from all-in-one models to modular systems [141] (including neuro-symbolic ones), which integrate domain knowledge and expert feedback to increase reliability, control and accountability in many domains. In this vein, neuro-symbolic AI makes systems more interpretable and aligned with humans [142]. In HCAI+DSS, it may help build solutions that are smart and understandable, capable of enriching the context with ethical aspects through scalable, ontology-based architectures.

7. Conclusions

In this paper, a review of the recent research literature on HCAI approaches in DSS frameworks has been carried out. First, architectural components were presented and discussed, highlighting the evolution from simple AI systems to full agentic solutions, progressively improving the inclusion of human input and feedback inside the AI systems for decision support. Then, recent works in HCAI + DSS were discussed according to nine different application domains. For each domain, practical applications, observed impacts, employed technologies, and validation modalities were examined in order to provide a comprehensive overview of HCAI + DSS solutions, including their alignment with the UN SDGs. In addition, the temporal evolution analysis of keywords and key phrases identified in the considered publications has shown how specific topics have received more or less attention during the last ten years. From the analysis, it emerged that in recent years, increasing attention has been paid to human-centered, explainable, and interactive AI-based DSS. Nevertheless, further attention is still required for surpassing simple explanations toward reliability engineering and better human–machine interaction policies. In addition, the limited adoption of MLOps and insufficient consideration of deployment accountability hinder the concrete deployment of HCAI + DSS solutions in real scenarios. In this context, the uptake of agentic AI, exploiting LLMs, RLHF, and neuro-symbolic methods, can constitute a relevant improvement aimed at providing more interpretable and human-aligned systems to support decision-making.