Smart In-Process Inspection in Human–Cyber–Physical Manufacturing Systems: A Research Proposal on Human–Automation Symbiosis and Its Prospects

Abstract

This positioning paper explores integrating smart in-process inspection and human–automation symbiosis within human–cyber–physical manufacturing systems. As manufacturing environments evolve with increased automation and digitalization, the synergy between human operators and intelligent systems becomes vital for optimizing production performance. Human–automation symbiosis, a vision widely endorsed as the future of human–automation research, emphasizes closer partnership and mutually beneficial collaboration between human and automation agents. In addition, to maintain high product quality and enable the in-time feedback of process issues for advanced manufacturing, in-process inspection is an efficient strategy that manufacturers adopt. In this regard, this paper outlines a research framework combining smart in-process inspection and human–automation symbiosis, enabling real-time defect identification and process optimization with cognitive intelligence. Smart in-process inspection studies the effective automation of real-time inspection and defect mitigation using data-driven technologies and intelligent agents to foster adaptability in complex production environments. Concurrently, human–automation symbiosis focuses on achieving a symbiotic human–automation relationship through cognitive task allocation and behavioral nudges to enhance human–automation collaboration. It promotes a human-centered manufacturing paradigm by integrating the studies in advanced manufacturing systems, cognitive engineering, and human–automation interaction. This paper examines critical technical challenges, including defect inspection and mitigation, human cognition modeling for adaptive task allocation, and manufacturing nudging design and personalization. A research roadmap detailing the technical solutions to these challenges is proposed.

1. Introduction

In response to the revolution of Industry 4.0, manufacturing industries have continuously integrated innovative, cutting-edge technologies into various manufacturing system applications to enhance production performance [1]. Significant changes are occurring not only at individual workstations but also at the system-wide level, driven by factors such as increased data availability, real-time data analysis and decision-making, and advancements in robotics and automation [2].

In this evolving landscape, the manufacturing environment is becoming increasingly dynamic and complex, introducing new features such as high-precision manufacturing, self-optimization and self-configuration, digitalized operations management, high-mix low-volume production, and human–automation interaction [3]. To thrive in this interconnected and rapidly changing environment, manufacturers must reassess and adapt their operations and strategies, with a strong focus on the role of human operators. The integration of smart sensors and networked manufacturing systems has fostered the development of a human-centric cyber–physical manufacturing environment [4].

This paper envisions an emerging research paradigm of smart in-process inspection (s-IPI) with human–automation symbiosis (HAS) within human–cyber–physical manufacturing systems (HCPMS). It seeks to explore the integration of s-IPI into advanced manufacturing systems (AMS), as well as the implementation of HAS from both human–automation task planning and execution in HCPMS. Specifically, s-IPI involves smart visual analytics and intelligent reasoning for part inspection, along with cognitive-driven defect mitigation. Meanwhile, HAS is realized through cognitive intelligent task allocation, behavioral economics modeling to support decision-making under conflicting goals, and smart manufacturing nudging design and personalization.

(1) Human–Cyber–Physical Manufacturing Systems: The advent of Industry 5.0 marks a transformative shift toward a human-centric approach in manufacturing systems [5]. It emphasizes reintroducing humans into automated processes in Industry 4.0, highlighting the synergy between advanced technologies and human capabilities, with the potential to revolutionize AMS by enhancing safety, flexibility, and responsiveness through human-in-the-loop integration. This paradigm shift merges human adaptability with the precision and reliability of machines, fostering a more inclusive and efficient manufacturing environment. At the heart of Industry 5.0 is the concept of human–cyber–physical systems (HCPS), where human workers and cyber–physical systems collaborate seamlessly. This integration demands robust frameworks for human–automation interaction, ensuring that humans remain central to the manufacturing process.

(2) In-process Inspection: In-process inspection (IPI) offers an effective strategy within modern manufacturing systems. Unlike traditional inspections conducted at separate workstations, IPI is performed at the same workstation where the part is processed, taking place between consecutive operations as an additional step in the original process. There are several key advantages to implementing IPI. First, it allows for the timely detection of defective parts, preventing them from progressing through additional processes and reducing the costs associated with poor quality. Second, IPI enables real-time process control and optimization by providing immediate feedback on manufacturing processes, allowing operators to address issues promptly and adjust process configurations through inspection-based closed-loop control. Third, by embedding quality control within each workstation, IPI ensures a continuous workflow, boosting productivity by minimizing disruptions caused by defects. In summary, IPI significantly enhances manufacturing systems by not only maintaining production speed with automation technologies but also enabling real-time process optimization and control.

(3) Human–Automation Interaction and Cognitive Engineering: The study of human–automation interaction (HAI) from a cognitive engineering perspective is becoming increasingly crucial. As automation technologies grow more advanced, the interaction between human operators and automated systems becomes more intricate and essential to overall system performance. Studies have found that a higher level of automation does not necessarily mean higher returns, due to the out-of-the-loop problem [6,7], and humans play an important role as to monitor the automation agents within the process, so called as human-in-the-loop [8]. Effective HAI harnesses human strengths such as adaptability while benefiting from the precision and consistency of automated systems. This synergy is particularly vital in environments that demand real-time decision-making and heightened situational awareness. Cognitive engineering, which focuses on understanding human cognition, helps translate subjective metrics like human trust and cognitive load into quantifiable measures. These metrics guide the design of interactions that align with human cognitive processes and the state of automated systems, ultimately enhancing performance and reducing human errors. Additionally, cognitive engineering can be used to design behavioral interventions, such as nudges, that support operators’ cognitive capacities, reducing cognitive load and improving collaboration efficiency. By incorporating cognitive engineering principles into HAI design, systems can better support human operators, improving resilience and overall system efficiency.

In this regard, this paper outlines the research scope and technical positioning of integrating s-IPI into HCPMS for HAS. Figure 1 illustrates the research scope. This research is driven by the integration challenges, from which two primary objectives are proposed: s-IPI and HAS. Within this framework, HCPMS emerges at the intersection of AMS, cognitive engineering, and HAI, focusing on human-centered manufacturing. It evolves from cyber–physical systems in Industry 4.0 by integrating human factors into the operations, facilitated by cognitive engineering, to achieve optimal HAI.

The s-IPI framework addresses two key questions: how to implement IPI effectively and how to achieve defect mitigation for in-process quality enhancement. The smartness of s-IPI is characterized by three main factors: data-driven technologies enabling rapid and precise inspections, intelligent agents that enhance human capabilities, and process self-configuration through inspection-based feedback.

On the other hand, HAS delves into how to incorporate human cognition into human–automation task allocation and enhance the collaborative efficiency during task execution. The successful integration of IPI into manufacturing systems demands coordinated efforts across various system components. HAS refers to the symbiotic relationship between humans and automation, emphasizing a partnership that brings mutual benefits [9]. Achieving efficient collaboration requires two key strategies: cognitive intelligent task allocation that promotes mutual adaptation during planning, and guided behavioral interventions, or nudges, to influence human actions during task execution.

Furthermore, these goals intersect across three fields: AMS, cognitive engineering, and HAI. Their convergence forms the foundation for research in HCPMS. AMS serves as the problem domain, addressing challenges such as defect inspection, process control through defect mitigation, human–automation task allocation, and manufacturing nudging. Cognitive engineering focuses on two aspects: automating human decision-making and developing models to understand human factors in manufacturing. HAI examines the interactions between human operators and automation agents, with the aim of designing systems that enhance overall performance.

The paper is structured as follows. The next section provides a literature review from both the problem and solution perspectives. It begins by reviewing Industry 5.0 manufacturing systems, inspection in manufacturing, and human–automation interaction to establish the research context. This is followed by an exploration of artificial intelligence and quantitative decision-making as key technologies to address the research questions. Section 3 presents a holistic framework for s-IPI within HAS, addressing three levels of manufacturing systems: workstation, process, and system, along with associated research challenges. Section 4 offers a case study to illustrate the practical implications of s-IPI and HAS. Section 5 examines the fundamental issues related to s-IPI and HAS in HCPMS. Finally, Section 6 outlines a technical roadmap to address the challenges identified in Section 5, for which potential solutions and future research directions are discussed.

2. Background Review and Emerging Consensus

2.1. From Industry 4.0 to Industry 5.0

Industry 4.0 focuses on the digitalization of manufacturing, where smart factories leverage advanced data analytics and automation to boost efficiency, flexibility, and productivity. Core technologies driving Industry 4.0 include the Internet of Things (IoT), big data, artificial intelligence (AI), and machine learning, which enable real-time monitoring and decision-making within manufacturing environments [10]. The emergence of Industry 4.0 has brought about significant changes in production methods, supply chain management, and product lifecycle management, fostering a more interconnected and intelligent manufacturing landscape [11]. Additionally, the role of the human workforce is evolving, requiring new skills to interact with advanced technologies and effectively manage automated systems [12].

Industry 5.0, by contrast, adopts a human-centric approach to industrial development, focusing on collaboration between humans and machines. While Industry 4.0 emphasizes automation and data exchange, Industry 5.0 enhances the synergy between human intelligence and cognitive computing, aiming to create a more sustainable, resilient, and socially responsible industrial ecosystem [13]. It highlights personalized production, where human creativity and customization play a central role, supported by collaborative robots and AI [14]. Unlike the era where humans and machines operated independently, Industry 5.0 fosters a partnership where humans and automation agents work together as a synergistic team [15]. The performance of this system depends on both the quality of automation support and how effectively humans utilize it [16]. Ultimately, Industry 5.0 represents a paradigm shift toward a more inclusive and human-friendly production environment.

(1) Human–Cyber–Physical Systems: The study of HCPS is driven by the need for human-in-the-loop systems, where human input and decision-making play a critical role in monitoring, controlling, and optimizing automation agents [8]. HCPS extends cyber–physical systems (CPS) by incorporating human systems alongside cyber and physical systems [17]. CPS, a system of systems, involves interactions between heterogeneous subsystems, integrating disciplines such as computing, control, and communication [18]. As a digitalized and automated system, CPS performance is influenced by interactions with human agents, leading to the development of HCPS to explore human–machine interaction in greater depth.

There are three primary perspectives on the human system within HCPS [17]: the first views human agents as physical entities whose interactions are coordinated and controlled by the cyber system; the second conceptualizes HCPS as the integration of CPS with cyber–social systems, emphasizing the social aspects of human interaction [19]; and the third focuses on integrating human psychology and cognitive behavior with CPS to enhance cognitive intelligence and improve system performance [4,20]. HCPS has a variety of applications aimed at improving human–automation interaction. For example, it supports dynamic task allocation in human–robot collaboration within complex manufacturing environments [21] and provides smart healthcare solutions by delivering real-time commands to mobile robots for patient care [22].

(2) Advanced Manufacturing Systems: AMS are characterized by the integration of sophisticated technologies aimed at enhancing production efficiency, flexibility, and customization. These systems incorporate a range of innovations such as automation, robotics, computer-integrated manufacturing, and cyber–physical systems, all designed to optimize the production process [2]. AMS are built to respond swiftly to market demands, reduce production costs, and improve product quality through precise control and monitoring.

A critical component of AMS is the use of AI and machine learning algorithms. These technologies facilitate predictive maintenance, real-time decision-making, and adaptive control of manufacturing processes, thereby reducing downtime and maximizing productivity [23]. The integration of the IoT further enhances AMS by enabling seamless communication between machines, devices, and systems, leading to more efficient and transparent manufacturing operations [14].

Additive manufacturing, or 3D printing, represents another significant advancement within AMS, enabling the production of complex geometries and customized products with minimal waste. This technology is particularly advantageous for creating prototypes and small-batch productions, reducing lead times and material costs [24]. Additionally, information and communication technologies, such as IoT and blockchain, enable efficient information sharing within and between enterprises, industries, and regions [25]. With increased data availability, big data analytics and AI technologies offer more responsive control of manufacturing systems through real-time operations management. These data include environmental metrics from sensors, machine process data, production operation data, and quality inspection information [2].

2.2. Inspection in Manufacturing

In manufacturing industries, inspection plays a crucial role in ensuring product quality. It involves measuring, examining, testing, or gauging one or more characteristics of a product or service and comparing the results with specifications to determine if each characteristic meets the required standards [26]. If a defect is identified, several actions can be taken before proceeding to the next stage: the product may be sent back for rework, replaced with a defect-free part, or scrapped.

In multi-stage production systems, inspection can occur at any point in the process and within any process structure. There are three main types of process structures based on item flow: serial, convergent, and non-serial [27]. In a serial structure, products move through successive manufacturing processes sequentially. In a convergent structure, different products follow their own sequential processes but may converge at certain stages. In a non-serial structure, products pass through various processes with multiple potential successors and predecessors. Inspection can be carried out at different production stages, including raw materials, intermediate work, and final products. The part quality inspection planning problem typically involves determining which processes and strategies to use for inspection to minimize the total expected cost of quality control [28].

Inspection can be classified into two types based on timing: online and offline. Online inspection involves real-time monitoring and examination of a manufacturing process while the product is being produced, whereas offline inspection occurs after the process is completed. While online inspection can be more effective, it presents challenges in data acquisition and control algorithms [29].

There are also destructive and non-destructive methods of inspection. Non-destructive methods include conventional approaches such as vision-based techniques, radiography, acoustic or ultrasonic testing, and others. Visual inspection is a key method in manufacturing, and its common approaches include manual inspection, machine vision, statistical methods, and deep learning [30]. Manual inspection relies on operators’ observations and experience, making it susceptible to human fatigue and subjectivity. To achieve faster and more reliable inspection for improved productivity, advanced techniques like machine vision and deep learning algorithms are essential.

2.3. Human–Automation Interaction

HAI research is inherently interdisciplinary, drawing from fields such as cognitive engineering, adaptive automation, behavioral economics, and more. The primary goal of HAI research is to optimize interactions between humans and automated systems, enhancing performance while maintaining a human-centered approach [31]. This interdisciplinary nature aligns with the National Science Foundation’s 10 Big Ideas, specifically the “Future of Work at the Human-Technology Frontier” initiative [32].

(1) Cognitive Engineering: Cognitive engineering plays a crucial role in HAI by providing theoretical foundations in human perception, cognition, and decision-making processes. Understanding these cognitive functions is essential for designing systems that enhance human capabilities and enable mutual adaptation between humans and automated systems [20]. Key topics in cognitive engineering for HAI include human trust and cognitive state prediction.

Cognition is defined as the process of acquiring knowledge or understanding, primarily through reasoning rather than feeling or willing [33]. In the context of HAI, studying cognition helps estimate operators’ cognitive states, which can be used to inform task allocation [34]. Cognitive state measurement can be conducted through behavioral observation, operator feedback, or physiological data. Team cognition, which measures collective performance, goes beyond the sum or average of individual cognition [35]. Statistical learning methods, such as Bayesian networks, Markov decision processes, and non-parametric Gaussian processes, have proven effective in predicting team-level cognition [20]. Cognition modeling requires interdisciplinary knowledge spanning cognitive science, psychology, human factors, and computer science.

Human trust is another vital aspect of human-centered automation. It is defined as the belief that an agent will assist in achieving an operator’s goals in uncertain and vulnerable situations [36]. Early research on human trust in automation showed that excessive trust leads to misuse—over-reliance on automation resulting in poor monitoring and decision biases—while insufficient trust leads to disuse, where automation is underutilized [37]. Trust is dynamic, influenced by the gap between expectation and observation [38]. Therefore, selecting the appropriate level of reliance on automation based on individual and situational factors is critical for maintaining system performance.

Research on human trust in automation encompasses three key areas: theoretical foundations that explore its relationship with automation use and influencing factors, trust measurement, and mathematical modeling. These studies draw on various disciplines, including psychology, cognitive science, socio-technical systems, economics, risk and uncertainty theory, and engineering [39,40,41,42]. Further details on trust measurement and modeling are discussed in the subsequent sections.

The modeling of cognitive states and human trust facilitates human-centered automation by managing operators’ workload and attention. As mentioned earlier, human operators can experience fatigue, boredom, and distraction during interactions with machines, affecting their satisfaction, system performance, and safety. The goal is to enable automation agents to dynamically adapt to the physical and cognitive demands of human operators, addressing issues such as human out-of-the-loop syndrome [43].

(2) Adaptive Automation: Adaptive automation allows for dynamic changes in the control and allocation of tasks between humans and machines [44]. In contrast, static automation systems have fixed task allocations, regardless of who operates the machine or when it is used, which can create challenges in maintaining optimal system performance. To address these issues, several approaches have been studied to enhance the adaptive capabilities of human–automation systems.

Task allocation, which assigns tasks to humans and machines, is a direct method of managing human workload and attention while enabling mutual adaptation between humans and automation. Early task allocation approaches were static, based primarily on comparing human and machine capabilities during the design process [45]. To introduce dynamic adaptability, task allocation must be framed as a real-time optimization problem. During allocation, task analysis specifies both the physical and cognitive requirements for each task [46], while considering factors like human trust, team cognition, and team performance [20]. By dynamically adjusting workload based on operators’ cognitive states, their situation awareness can be maintained at an optimal level.

Another key factor in task allocation is the level of automation, which refers to the proportion of physical and cognitive tasks managed by automation. It indirectly regulates operators’ workload and attention and can be determined based on task demands. Setting the appropriate level of automation involves balancing performance gains from automation against the potential risks associated with automation failures [43]. This process includes modeling various factors, such as the risk of human out-of-the-loop (OOTL) issues, task requirements, team cognition, human trust, and situation awareness [37]. Identifying the appropriate level of automation can be approached as a reasoning or optimization problem [47,48].

The out-of-the-loop (OOTL) problem occurs when operators fail to detect system errors and are unable to regain manual control in the event of automation failures [6]. This issue is well documented, particularly in highly automated environments like air traffic control [49]. Research has shown that as the level of automation increases, the operator’s role shifts to monitoring and intervening when necessary. While this improves system performance, it reduces operator involvement, leading to decreased attention and loss of situation awareness.

Situation awareness, as introduced by [6], is divided into three levels: perception, comprehension, and projection. Perception involves detecting environmental events or information, comprehension refers to understanding that information, and projection involves predicting its future impact. To address the OOTL issue and maintain operator situation awareness, three strategies have been identified [8]: improving information presentation in automated systems, monitoring operator vigilance and trust, and increasing operator engagement. These strategies highlight that OOTL can be mitigated through both design and operational approaches. From a design perspective, prior research has focused on measuring and assessing situation awareness [50,51] and on situation awareness-oriented design [52]. Operationally, situation awareness and attention can be sustained using operators’ mental states to dynamically adjust task allocation and automation levels through adaptive automation [53,54].

(3) Human–Automation Symbiosis: In recent years, many new concepts have emerged in the manufacturing industry, introducing advanced technologies that have significantly improved system performance. These innovations have gradually reshaped the role of operators in manufacturing activities [55]. At the same time, the relationship between humans and technology has been continuously evolving as integration progresses [13,56].

HAS has gained widespread recognition as a future vision for human–automation collaboration in both industry and academia. HAS builds upon the concept of “man-computer symbiosis”, which emphasizes the coexistence and mutual benefits of human and machine interactions [57]. What sets HAS apart from other human–automation relationships is its focus on partnership and mutual benefit [58]. HAS aims to develop systems that augment human capabilities, coordinate human and automation agents to work toward common goals, optimize resource use dynamically, and enable human-like communication [9]. In this dynamic system, human and automation agents are interdependent, and their collaborative performance exceeds the sum of their individual contributions.

There are several reasons why HAS is identified as a key area of study following HAI. First, the combined strength of human and automation agents is expected to go beyond physical power enhancement and intelligence amplification [59]. Second, viewing automation as a collaborative partner, rather than creating fully automated systems, is seen as a more effective approach [60]. This is due not only to the potential loss of situation awareness in highly automated systems but also to the inevitability of human errors, which can be carried from the design stage to the operational phase. Third, fully automated systems, where humans merely monitor and intervene when necessary, pose significant risks. In such systems, operators may struggle to regain manual control during failures, making it crucial to maintain operator engagement and situation awareness [8]. As a result, establishing a symbiotic relationship between humans and automation is both beneficial and effective for enhancing system performance.

2.4. Artificial Intelligence

(1) Deep Learning Neural Networks: In recent years, deep learning neural networks have been extensively studied in the manufacturing industry, with two primary applications: convolutional neural networks (CNNs) for image data analysis and neural networks for numerical data prediction.

CNN-based visual data analysis has seen widespread use, particularly in two key tasks: image classification and object detection. (1) Image classification involves labeling an image with specific categories. The use of CNNs for image classification gained prominence with the introduction of AlexNet in the ILSVRC 2012 competition [61], which employed five convolutional layers and demonstrated the potential of CNNs for multi-class classification. Subsequent models, such as VGGNet and GoogLeNet, incorporated more convolutional layers to deepen the networks, improving learning performance and prediction accuracy [62,63]. However, deeper networks introduced issues like gradient explosion and vanishing gradients, which were addressed by the residual neural network (ResNet) through residual learning between layers [64]. (2) Object detection, in contrast, identifies and locates various objects within an image, providing both object classes and their positions. CNN-based object detection began with Regions with CNN features (R-CNN), which used selective search for region proposals followed by CNN-based classification and support vector machines (SVM). Later improvements, such as YOLO, eliminated region proposals, using a convolutional network for both region and class predictions, dividing images into grids for bounding box predictions [65]. Faster R-CNN introduced a region proposal network to replace selective search, forming a unified network [66], while Mask R-CNN built on Faster R-CNN by adding RoI Align and a branch for segmentation mask prediction [67].

Neural networks are also highly effective for numeric data prediction, thanks to their ability to model complex, non-linear relationships. This capability is grounded in the backpropagation algorithm [68], which enables neural networks to learn by adjusting weights to minimize errors. These networks typically consist of multiple layers: an input layer, one or more hidden layers, and an output layer. Each layer comprises neurons that process input data through weighted connections and activation functions. Common activation functions include the sigmoid function, rectified linear unit (ReLU), and Softmax function.

(2) Knowledge Representation and Reasoning: In the field of artificial intelligence, knowledge representation focuses on describing real-world information using symbolic representations, while reasoning involves the formal manipulation of these symbols to derive new propositions or generate explicit or implicit conclusions from available knowledge [69]. For example, knowledge graphs are a common way to represent knowledge bases in a graph structure, where entities are represented by nodes and relationships by edges. These graphs enable the construction and description of large entity-relation networks [70]. Typical reasoning tasks for knowledge graphs include node classification, link prediction, graph classification, clustering, and predictive queries.

Case-based reasoning (CBR) and fuzzy logic are two widely used reasoning methodologies. CBR originated from the study of scripts that represent past situations as knowledge and plans to interpret new situations. Further research explored how previous situations and patterns can be applied to problem-solving and learning [71], aligning with the classic definition of CBR: a case-based reasoner solves problems using or adapting solutions from past problems [72]. CBR requires little explicit knowledge, as systems continuously learn new knowledge from retained cases [71].

Fuzzy logic and rough set theory, on the other hand, are mathematical approaches to dealing with imperfect knowledge and uncertainty [73]. Fuzzy logic is based on the observation that much of human reasoning is approximate rather than precise [74]. It introduces the concept of fuzzy sets, which contain objects with varying degrees of membership [75]. Fuzzy truth-values allow for nuanced descriptions during approximate reasoning, using terms like “true”, “very true”, or “more or less true”. Fuzzy IF-THEN rules incorporate fuzzy antecedents or consequences instead of crisp ones, making the reasoning process more flexible and representative of real-world ambiguity [76].

(3) Large Language Models and Generative AI: Large language models (LLMs) and generative AI have made significant strides in advancing natural language processing (NLP). With their vast scale and capabilities, LLMs have revolutionized tasks such as question answering, document summarization, and language translation. Models like Generative Pre-trained Transformer-3 (GPT-3) and GPT-4 have demonstrated transformative potential across various domains [77].

A key innovation behind LLMs is the Transformer architecture [78], which utilizes self-attention mechanisms to efficiently capture dependencies across sequences. This architecture has become the backbone of many subsequent LLMs, greatly enhancing scalability. Among these models, GPT-3 represents a significant milestone, showcasing its ability to generate human-like text, perform a wide range of tasks with minimal fine-tuning, and understand context at a deeper level [79].

Recent research is also exploring the intersection of LLMs and SQL model-based databases, investigating how LLMs can enhance database functionalities. This includes domain knowledge reasoning through retrieval-augmented generation (RAG) applications, where LLMs integrate retrieval techniques to improve task performance [80].

2.5. Quantitative Decision-Making

(1) Decision-based Design: Decision-based design is a methodological framework in engineering that emphasizes the central role of decision-making in design and other engineering activities, especially those characterized by ambiguity, uncertainty, risk, and trade-offs [81]. This approach combines principles from economics, engineering, and decision theory to develop methodologies that ensure designs are both technically feasible and economically viable.

A key component of decision-based design is the application of utility theory, which quantifies and compares stakeholder preferences. Utility functions are used to convert subjective preferences into numerical values, enabling more objective decision-making processes. These functions are widely employed in engineering design, providing a structured method to evaluate different design alternatives [82]. Multi-criteria decision-making, such as multi-attribute utility theory [83], is commonly used to address the inherent trade-offs in complex design problems. This approach allows designers to systematically evaluate and prioritize different design options based on multiple criteria, facilitating more informed and balanced decision-making.

(2) Behavioral Economics and Prospect Theory: Behavioral economics is an approach to economic analysis that incorporates psychological insights into human behavior to better understand decision-making processes [84]. Unlike traditional economic theory, which assumes individuals make rational decisions based on complete information, behavioral economics explores how cognitive biases and emotional influences lead people to make decisions that may appear irrational.

Prospect theory, a cornerstone of behavioral economics, is particularly valuable for modeling the irrationality of human decision-making in decision-based design [85]. It differs from expected utility theory, where utility is assumed to be linearly related to the risk of a probabilistic outcome [86]. Instead, prospect theory offers a more psychologically accurate model by accounting for the decision-maker’s risk attitude. It suggests that losses have a greater emotional impact than equivalent gains, a concept known as loss aversion. Key aspects of prospect theory include reference dependence, loss aversion, the affective impacts on decision-making (such as risk aversion), and probability weighting. The theory was later expanded into cumulative prospect theory, which applies a cumulative probability distribution to account for probability weighting [87].

(3) Operations Research and Optimization: Operations research is a multidisciplinary field that applies mathematical models, statistical analysis, and optimization techniques to improve decision-making processes in complex operational systems. Its origins date back to logistics and strategy studies during World War II. The primary goal of operations research is to mathematically formulate problems and use optimization algorithms to find solutions, thereby enhancing system efficiency and productivity.

Optimization algorithms can generally be classified into two categories: exact algorithms and heuristic algorithms. Exact algorithms are designed to find the optimal solution to an optimization problem. For large-scale mixed integer programming problems, the branch-and-bound method is widely used. Originally developed to solve the traveling salesman problem [88], branch-and-bound uses branching operations to ensure solution integrality and bounding operations to discard unpromising solutions. Variations of this method include the cutting-plane method, which refines the feasible space using linear inequalities [89,90], and branch-and-cut, which combines branch-and-bound with cutting planes to tighten linear programming relaxations [91]. Another variation, branch-and-price, integrates branch-and-bound with column generation to handle large-scale problems more efficiently by generating columns that improve the objective function [92], thus accelerating the search process.

Heuristic algorithms, in contrast, are computational methods designed to find near-optimal solutions by iteratively improving a candidate solution based on a quality measure [93]. While they do not guarantee optimal solutions, heuristic algorithms offer acceptable solutions within a reasonable timeframe [94]. Heuristics can be divided into local search and global search strategies. Local search heuristics focus on refining solutions by exploring neighboring configurations, often taking a greedier approach. Examples include A* search [95], tabu search [96], variable neighborhood search [97], and simulated annealing [98]. Global search heuristics, on the other hand, are population-based and designed to escape local optima. Popular algorithms in this category include ant colony optimization [99], particle swarm optimization [100], and genetic algorithms [101].

(4) Game Theoretic Optimization: Game theoretic optimization has been extensively studied to address complex multi-agent decision-making problems, particularly in scenarios where agents have conflicting objectives. Game theory can be broadly divided into two categories: cooperative and non-cooperative. Cooperative game theory focuses on how groups of agents or coalitions collaborate to achieve outcomes beneficial to all members. The main challenge lies in determining fair and stable payoff distributions, ensuring that no subgroup has an incentive to deviate [102]. Solutions like the Shapley value offer methods for equitable distribution based on each member’s contribution [102], while the core value ensures stability by preventing coalition deviations [103]. These frameworks highlight the difficulty of balancing individual incentives with collective objectives in cooperative scenarios.

In contrast, non-cooperative game theory examines how rational players independently make decisions to maximize their own objectives without collaboration [104]. It is particularly useful for understanding symbiosis in human–automation teams, where different agents optimize distinct performance measures. The Nash equilibrium is the key concept in non-cooperative game theory, representing a stable state where no agent can improve their outcome by unilaterally changing their strategy [105]. Bi-level optimization, exemplified by the Stackelberg game [106], extends non-cooperative game theory by modeling hierarchical decision-making processes. In these scenarios, a leader makes the first move, followed by a follower who responds based on the leader’s decision. This approach is widely applied in operational systems across economics and engineering.

2.6. Research Opportunities

This review of existing work and the emerging consensus on s-IPI with HAS in HCPMS highlights two key pillars upon which new research opportunities are built: the problem context and supporting technologies. Overall, this review establishes a solid foundation for the methods proposed in subsequent sections by providing a detailed understanding of the problem domain and the available technologies to address the research questions.

(1) Domain Problem Context: This review explores HCP manufacturing systems, tracing their evolution from Industry 4.0 to Industry 5.0, and emphasizing the shift toward a human-centered approach that fosters collaboration between humans and machines. A significant opportunity lies in integrating humans into the CPS of manufacturing to optimize human-machine interactions. This review of AMS highlights critical technologies that enhance production efficiency. Additionally, it investigates manufacturing inspection processes, focusing on methodologies that maintain product quality. The current state of HAI underscores the importance of cognitive engineering, adaptive automation, and HAS in fostering a symbiotic relationship between humans and automation.

(2) Supporting Technologies: This review identifies promising opportunities in supporting technologies, with a focus on AI and quantitative decision-making. Notable AI prospects include applications of deep learning neural networks, knowledge representation and reasoning, and LLMs, alongside generative AI. In the realm of quantitative decision-making, the review highlights the potential of decision-based design, behavioral economics, operations research and optimization, and game-theoretic optimization. These technologies provide a robust toolkit for addressing the challenges and opportunities in HCP manufacturing systems.

3. A Holistic Framework of Smart In-Process Inspection with Human–Automation Symbiosis

Advanced manufacturing is expected to be driven by a range of cutting-edge technologies, creating a more complex and dynamic production environment. To ensure consistent product quality throughout manufacturing processes and provide real-time feedback on process issues, in-process inspection has emerged as an efficient strategy widely adopted by manufacturers. s-IPI is envisioned as a key research paradigm for Industry 5.0, focusing on its implementation and integration within Industry 5.0 manufacturing systems. By analyzing the issue across three levels of a manufacturing system—the workstation, process, and system—this study identifies four fundamental challenges related to s-IPI and human–automation symbiosis in HCPMS. A holistic framework is presented to address these fundamental issues. Finally, a technical roadmap is proposed to summarize the key challenges and their corresponding technical solutions, offering a path for future research directions.

3.1. In-Process Inspection from the Manufacturing System Hierarchy Perspective

IPI serves as a quality control strategy that performs inspection operations during the manufacturing stage, identifying defective parts immediately as they are processed. This approach is designed to enhance overall product quality and yield by enabling prompt defect handling and real-time process control. In advanced manufacturing environments, which are dynamic, complex, and fast-paced, integrating IPI requires seamless collaboration between human operators and automation systems. This section explores how s-IPI impacts manufacturing from a system hierarchy perspective and introduces a holistic framework for implementing s-IPI within HCPMS. As depicted in Figure 2, the responsibilities at different levels of the manufacturing system hierarchy are outlined, consisting of three distinct levels. The proposed hierarchical structure of manufacturing systems operates at three interconnected levels to ensure continuous improvement and collaboration between human operators and automation systems:

(1) Workstation level: At this level, individual or collaborative manufacturing tasks are executed by human operators and machines. Inspection operations occur within the workstation, where defect analysis, such as scrap or rework decisions, is performed to support defect disposition.

(2) Process level: This level focuses on controlling and adjusting production line configurations through defect mitigation planning. Defect information collected from the workstation level is used to adjust the process configurations to eliminate or reduce defects.

(3) System level: Managing and planning production operations for the human–automation team, the system level coordinates collaboration between human operators and automation agents. It aims to create a symbiotic relationship between humans and automation through dynamic task allocation and personalized manufacturing nudges. These nudges, informed by behavioral science [85], influence operators’ behavior during interactions with automation via tools like augmented reality or auditory prompts.

Under this hierarchy, defect information flows from the workstation level to the process level for mitigation planning, while data on machine and operator status are fed to the system level for dynamic task assignment and configuration of nudges. This feedback loop enhances HAS and improves overall production efficiency.

3.2. A Holistic Framework

To better understand the key requirements, the fundamentals of s-IPI in HCPMS and the related technical challenges have been analyzed and summarized into a holistic framework, as depicted in Figure 3. This framework focuses on two main objectives, each spanning across three hierarchical levels: smart in-process inspection and human–automation symbiosis. Smart in-process inspection addresses two core issues: in-process inspection for quality monitoring and defect mitigation planning for real-time process control. Human–automation symbiosis focuses on (i) dynamic and adaptive task allocation to optimize human–automation collaboration and (ii) behavioral intervention design to enhance human–automation interaction. These four key issues are elaborated within the framework, offering a comprehensive approach to achieving s-IPI with HAS in HCPMS.

Q1. In-process Inspection: Implementing inspection within the same process as the manufacturing operation is a crucial and fundamental issue. Integrating inspection into the manufacturing process requires modifying existing procedures to accommodate an additional inspection task either during or immediately after the manufacturing operation. For instance, in 3D printing, infrared cameras are employed to monitor part quality during the printing process [107].

In-process inspection involves two primary technical challenges: (i) Defect identification focuses on detecting and classifying defects using appropriate vision equipment and inspection algorithms. Key challenges include ensuring the hardware and software’s capability to perform accurate and reliable defect inspection in an Industry 5.0 manufacturing environment. (ii) Decision support for part inspection aims to automate decision-making during the inspection process. This includes activities such as defect disposition (determining whether to scrap or rework a part) and assigning rework operations. Automating these decisions helps streamline and enhance the efficiency of the inspection process.

Q2. Defect mitigation planning for real-time process control: The objective of s-IPI extends beyond merely identifying defective parts; it also seeks to mitigate defect generation from a process control perspective. Achieving this requires addressing two key technical issues: (1) Process modeling of the manufacturing–inspection–mitigation control loop. This loop functions as a self-configuration process, utilizing real-time inspection results to adjust process configurations that contribute to defect generation. The control loop continuously refines the manufacturing process, ensuring that it is dynamically adjusted to minimize or eliminate defects, thereby improving overall product quality. (2) Defect mitigation planning: This involves the cognitive processes necessary for determining how to adjust process configurations to prevent future defects. Effective mitigation planning relies on real-time inspection data to identify issues early, allowing for prompt and efficient adjustments that prevent defect recurrence. By integrating real-time data, the system can continuously improve manufacturing outcomes and optimize process performance.

Q3. Dynamic and Adaptive Task Allocation: Given the cognitive complexity of inspection and other manufacturing tasks, collaboration between human operators and automation systems is essential. This study proposes two key approaches to enhance HAI and facilitate HAS: task allocation during the planning stage and behavioral interventions during the execution stage.

A central focus of this approach is dynamic and adaptive task allocation, which emphasizes mutual adaptation between human operators and automation agents. Unlike conventional task allocation methods that focus only on production performance through static allocation, the proposed method introduces human cognition as a critical factor. It leverages real-time data on agent status, allowing automation systems to adjust and better align with the cognitive and operational states of human team members.

To achieve dynamic and adaptive task allocation, three key technical challenges are addressed: (i) Modeling human cognition during tasks, involving the identification and mathematical formulation of factors that represent cognitive states such as attention, workload, and trust; (ii) team cognitive performance evaluation, enabling the assessment of team-level performance based on cognitive factors, creating a new metric that reflects overall team synergy; and (iii) balancing production and cognitive performance, which optimizes both production efficiency and team cognitive health for more effective task allocation. By resolving these issues, the method aims to create a more adaptive and efficient human–automation collaboration, ultimately enhancing HAS and overall production outcomes.

Q4. Behavioral Intervention Design for Human–automation Collaboration: Once tasks are allocated to operators and automation agents, manufacturing procedures begin. However, training operators to interact effectively with automation agents takes time, and deviations from standard procedures are common. Such deviations can hinder collaboration efficiency and reduce production performance.

To address this, the study proposes the use of behavioral interventions, or manufacturing nudges, to influence operator behavior toward desired standards. Three key technical challenges are central to this approach: (i) Modeling human decision-making behavior: This involves developing behavioral economic models to quantify the utility of different behaviors. Understanding how operators make decisions in the context of losses and gains during manufacturing is critical for designing effective nudges that align operator behavior with production goals. (ii) Designing nudges for improved collaboration: This focuses on creating models for designing nudges, including how they are introduced and how their effectiveness is evaluated. The goal is to mathematically model the perception of operators toward nudges under various work scenarios to ensure the nudges enhance collaboration. (iii) Personalizing nudges: This aspect tailors nudge configurations to fit individual operators, increasing their effectiveness. Personalized nudges accommodate the diverse backgrounds and preferences of operators, leading to improved team synergy and more efficient human–automation collaboration. By addressing these challenges, this approach aims to improve human–automation interaction and overall production efficiency through targeted behavioral interventions.

4. A Case Example of Medical Tube Assembly In-Process Inspection

Integrating IPI into HCPMS requires coordinated efforts across different levels within the system hierarchy. A case example of medical tube assembly illustrates how IPI can be incorporated to enhance operational efficiency and quality.

Figure 4 shows the medical tube assembly process, which spans across various levels of the manufacturing hierarchy. Medical tubes, critical for open-heart surgeries, require intricate assembly steps, including subpart heating and braid cutting. These operations have a high potential for defects, which are costly due to the expensive raw materials and complexity of the assembly process.

In this example, IPI is implemented to detect defects early and prevent their propagation throughout the assembly line. Through defect mitigation and reconfiguring operations that cause defects, IPI helps minimize waste of materials and labor. This proactive approach ensures higher-quality production and more efficient use of resources, ultimately reducing overall production costs.

In this example, the elements and operations in a manufacturing system integrated with IPI are analyzed. The plant has three production lines, $P_1,P_2,P_3$, each dedicated to assembly of a different product. Each production line comprises multiple workstations, $W_1^P,W_2^P,\dots ,W_n^P$, where $n$ is the number of workstations in the production line $P$. These workstations execute various manufacturing tasks, ${\gamma }_1,{\gamma }_2,\dots ,{\gamma }_{N^P}$. The operant resources, including human operators $A^H$ and automation agents $A^M$, collaborate to complete the tasks. Meanwhile, each workstation involves a manufacturing operation $O^P$ and an inspection operation $I^P$. The inspection process is coupled with cognitive reasoning activities, such as defect disposition, which determines the handling of defective parts.

The workflow starts from production operations planning at the system level, which targets at achieving HAS through properly designed interactions. This study considers two types of operations: task allocation and manufacturing nudging. Task allocation involves dynamically assigning tasks $\mathsf{\Gamma }=\left\{{\gamma }_1,{\gamma }_2,\dots ,{\gamma }_{N^P}\right\}$ within a production line between human agents $A^H$ and machine agents $A^M$ based on agent status. In this process, real-time human cognition will be considered in the optimization objective alongside conventional production performance metrics, enabling machine agents to adapt dynamically to human status. This approach creates an ergonomic working environment while maintaining high production efficiency. The results include the allocation of a task $\gamma$ to an agent $a^H$ or $a^M$ and determining the level of automation for an automation agent. The other operation that can contribute to HAS is manufacturing nudging. Manufacturing nudging aims to enhance individual production performance by influencing operators’ behaviors and decision-making in a predictable manner without restricting their options through persuasive design [108]. A manufacturing nudge can be represented as $D\left(\gamma ,a^H\right)$, referring to a nudge $D$ for task $\gamma$ to a human operator $a^H$. Like other types of nudging applications, manufacturing nudging design integrates principles from behavioral economics and psychology in the manufacturing environment. For instance, using Apple Vision Pro to reduce assembly errors by providing operators with appropriate operation instructions is a practical application.

With tasks allocated to agents and nudges configured, manufacturing processes can start. In each workstation with s-IPI, a manufacturing operation $O$ is followed by an inspection operation $I$. The inspection task can involve dimensional inspection or cosmetic inspection. Figure 5 shows an example of tube cosmetic inspection, where Figure 5a is the vision system setup and Figure 5b shows the inspection results. When a defective part is identified, a decision needs to be made regarding whether the part should be reworked, scrapped, or replaced before advancing to the next manufacturing stage. If rework is necessary, the specific rework operation should be determined by analyzing the defect information. Simultaneously, defect data serve as feedback for defect mitigation planning. By analyzing the cause of the defect, a mitigation strategy should be selected from the knowledge base, and detailed process configuration adjustments should be identified to correct the operation that generates the defects. This process is part of a manufacturing–inspection–mitigation loop aimed at controlling manufacturing processes to ensure product quality. The result of mitigation planning is the reconfigurations of a workstation $W$ based on the analysis of the defect identified during inspection $I$. For instance, if the identified defect is the part outer diameter being oversized, and the root cause is identified as an issue in the subpart heating process at workstation $W_3^{P_3}$, then the mitigation planning should adjust the heating temperature and the duration accordingly. By implementing s-IPI with HAS, the system enhances product quality and efficiency, benefiting from real-time process control and human–automation collaboration in HCPMS.

5. Fundamental Issues

With reference to the illustrative example of medical tube assembly, this section outlines the key technical issues that facilitate the understanding of implementation:

(i)

In-Process Inspection: At the workstation level, the primary challenge is integrating inspection into the existing manufacturing process. Two key technical issues are (a) defect identification for detecting and classifying defects using appropriate vision equipment and inspection algorithms and (b) decision support for part inspection for automating decision-making activities during inspection, such as defect disposition and rework assignment.

(ii)

Defect Mitigation Planning for Real-Time Process Control: At the process level, the focus is on improving in-process quality through real-time adjustments. This involves (a) process modeling of the manufacturing–inspection–mitigation control loop by establishing a feedback loop to adjust configurations based on inspection results and (b) defect mitigation planning using real-time inspection data to plan process configuration adjustments to reduce defect generation.

(iii)

Dynamic and Adaptive Task Allocation: This aims to optimize collaboration between human operators and automation by considering human cognition in real-time. Key technical issues include: (a) modeling human cognition by quantifying human cognitive states during tasks; (b) team cognitive performance evaluation by developing new performance metrics that reflect the cognitive state of the team; and (c) trade-off between production and team cognitive performance by balancing production efficiency with cognitive performance.

(iv)

Behavioral Intervention Design for Human–Automation Collaboration: This issue focuses on influencing operator behavior for better collaboration. It includes: (a) human decision-making behavior modeling by quantifying decision-making behaviors in manufacturing environments; (b) nudging design for collaboration efficiency by designing interventions (nudges) to influence operator behavior; and (c) nudging personalization by tailoring nudges to fit individual operators for enhanced team synergy. These technical issues collectively support the effective integration of IPI and human–automation symbiosis in complex manufacturing environments.

5.1. In-Process Inspection

IPI significantly enhances manufacturing system performance by ensuring consistent quality at the workstation level while simultaneously reducing resource waste and operational disruptions. The integration of inspection within the manufacturing process, however, poses fundamental challenges, as existing procedures must be modified to incorporate real-time product inspection.

Figure 6 outlines the steps involved in operations with IPI. When a part or material reaches the i-th stage, the manufacturing operation is conducted, followed by an immediate or concurrent inspection. If the part passes inspection, it moves to the next stage; if not, a defect disposition analysis determines whether it should be reworked, replaced, or scrapped, with the appropriate rework operation assigned.

This integration affects multiple aspects of production systems. At the workstation level, additional inspection and rework operations alter the manufacturing procedures, and the process cycle time must now account for these steps. Moreover, IPI changes the interactions between workstations and the overall production line. Since inspection traditionally occurs as a separate process, integrating it into the main production sequence alters the system’s workflow and can change the structure of production, influencing item flow. There are four types of item flow: same product in a single item, same product in a batch, mixed products in a single item, and mixed products in a batch [27]. Decision-making regarding rework operations, such as whether to batch defective parts or rework them immediately, can be analyzed using discrete event simulation or queuing theory [109].

To ensure smooth integration, the deployment of the inspection system must be aligned with the manufacturing context. Figure 7 present different inspection categories based on deployment [110]. IPI encompasses both in-line and in-situ inspection. In-situ inspection takes place on the production machine during the operation, commonly used in continuous processes like cable extrusion [111], whereas in-line inspection occurs off-machine but still within the production process. In these systems, inspection cycle time needs to be minimized, especially for vision systems, to prevent creating bottlenecks.

In additive manufacturing, in-situ inspection is a typical application of IPI. Given the slow nature and material consumption of 3D printing, defects that occur during printing can be catastrophic. Reworking is challenging, and post-process inspection often results in significant waste of time and resources. This issue is magnified in large-scale additive manufacturing [112]. To mitigate such losses, research has focused on integrating in-situ inspection during the printing process. For example, combining infrared and visual cameras allows for real-time monitoring, enabling users to halt printing when defects are detected, preventing further resource waste [107]. Other solutions use thermal cameras and laser profilometers to achieve similar outcomes [112].

Overall, IPI offers a proactive approach to improving product quality by enabling real-time defect identification and mitigation during manufacturing, leading to more efficient resource use and higher production quality.

(1) Defect Identification: Defect inspection is a crucial method for quality control in production and has been widely researched across various fields. It is not limited to traditional industries like wood, textiles, steel, and ceramic tiles [113,114,115,116] but also plays a key role in modern manufacturing sectors, such as semiconductors, 3D printing, and automotive [117,118,119]. As manufacturing processes grow more complex, the challenges in defect inspection also increase, necessitating more advanced solutions.

Defect inspection typically involves several interrelated activities. Figure 8 illustrates the information hierarchy of inspection systems, based on the data–information-knowledge pyramid, where inspection is broken down into three key tasks: data collection, defect identification, and inspection decision support.

(i)

Data Collection (Data Layer): This involves observing and saving product data using sensing equipment, such as vision systems for visual inspection. The collected data can take the form of images or numerical values. At this level, the focus is on capturing relevant data for the subsequent stages of the inspection process.

(ii)

Defect Identification (Information Layer): This stage involves detecting and classifying defects using the collected product data. Several terms are associated with this process: (a) Defect detection: A binary decision, determining whether a product has a defect or not. This is often the initial step in the inspection process. (b) Defect identification: A more detailed analysis that determines the type of defect. This step can be performed manually by human operators or automatically via classification algorithms. (c) Defect recognition: An in-depth analysis that not only identifies the type of defect but also seeks to understand its potential causes. This step requires a deeper understanding of the defect. Defect identification, the core task at the information layer, uses feature extraction and classification algorithms to understand the nature and type of defects present in the product.

(iii)

Inspection Decision Support (Knowledge Layer): The final stage involves supporting inspection-related decision-making activities, which may include symbolic reasoning, knowledge inference, or other analytical techniques. This stage focuses on applications like defect disposition (deciding whether to rework, scrap, or continue processing a product) and root cause analysis, both of which rely on information about the defect features gathered in the previous stage.

By integrating these three tasks, defect inspection systems aim to enhance product quality, improve decision-making processes, and reduce production waste.

In the IPI context, defect identification and decision support are two critical technical issues. Defect identification primarily involves vision-based data analysis, focusing on the detection, classification, measurement, and localization of specific features of interest. Industrial applications typically encompass two main types of inspections: dimensional defect inspection and cosmetic defect inspection. Dimensional defect inspection measures specific features using calibrated systems to determine whether the measurements fall within acceptable tolerances. Cosmetic defect inspection identifies surface defects, such as texture anomalies and misalignment of components [120,121].

In advanced manufacturing environments, the challenges associated with defect identification are amplified by the trend toward smaller defects with complex geometric features. Inspections must also be performed within shorter cycle times, which adds pressure on both the vision equipment and inspection algorithms. Vision equipment means that, to meet stringent precision requirements for defect detection and measurement, the optimal combination of camera, lens, and lighting is essential to capture high-quality image data [122]. Inspection algorithms must achieve a delicate balance among robustness, speed, and accuracy, adapting to complex and dynamic environments while maintaining rapid takt times.

A pertinent industrial example of defect identification is printed circuit board (PCB) inspection, a critical process in the electronics and semiconductor industries. PCB manufacturing involves sophisticated machinery, surface mount technologies, and automation, all of which contribute to the complexity of inspections. One common inspection task is to identify scratches and etching on PCB surfaces [123]. This task can be effectively addressed by integrating machine vision with deep learning approaches. In this approach, machine vision algorithms, such as speeded-up robust features (SURF), are utilized to extract image features, while machine learning algorithms predict fault patterns from these features. By combining the extracted features with the predicted patterns, a feature map can be generated to indicate areas of high defect density, allowing for more targeted inspections and enhancing overall product quality.

(2) Decision Support for Part Inspection: In the development and deployment of an inspection system, decision-making is critical throughout the defect identification and management process. Decisions range from selecting appropriate vision equipment and inspection algorithms to interpreting inspection results for part judgment and defect disposition. Automating or supporting these decision-making activities enhances efficiency, reduces human error, and improves overall speed in manufacturing systems.

Decision support is to apply information technology to support complex decision-making and problem-solving, with the focus on improving the efficiency and effectiveness of user decision [124]. Developing a decision support solution typically involves three key questions: knowledge modeling, reasoning, and human-computer interface [125]. Because knowledge can be comprehended in different forms and natures, knowledge modeling studies how to organize, store, and maintain knowledge so that it can be used for reasoning and inference [126]. Reasoning, the core of decision-making, uses available knowledge and information and can be implemented through various technologies like decision trees, case-based reasoning, statistical inference, and generative artificial intelligence (GenAI). The human–computer interface studies how to enable interactive queries between human and the decision support system, utilizing technologies such as natural language processing to translate between human and computer languages.

In Industry 5.0, decision-making is becoming increasingly automated. Machines, with their superior computational power, are now making more complex, high-level decisions. Several factors are driving this trend: (i) Real-time decision-making: Advanced manufacturing requires quick, often instantaneous, decisions. Machines can handle this due to their processing speed. (ii) Data volume: Modern manufacturing generates vast amounts of data. Machines excel at analyzing large datasets more quickly and accurately than humans. (iii) Reduction of human error: Humans are prone to mistakes, particularly when performing repetitive or trivial tasks. Automation eliminates these errors, improving consistency and precision.

One common decision-making task in IPI is defect disposition, which determines the next step when a defect is detected. This is a complex, knowledge-intensive task that involves analyzing the defect’s nature and severity, as well as deciding whether to scrap, rework, or repair the part. Defect disposition requires a deep understanding of the production process, defect types, and mitigation strategies, making it an ideal area for decision support systems.

A real-world example is reworking underfilled electronic components in PCB manufacturing [127]. The rework process for underfilled components is complicated by factors such as component configuration and the viscosity of the underfill material. The methodology proposed in [127] systematically analyzes underfill features and uses hierarchical clustering to generate inspection indicators. These indicators help trigger case-based reasoning, pulling from a knowledge database to recommend solutions and fine-tune engineering parameters. This system highlights how decision support can improve defect disposition in complex, knowledge-driven tasks like PCB manufacturing. By developing robust decision support systems, manufacturers can better manage defect inspection, minimize human error, and increase the overall effectiveness of production processes.

5.2. Defect Mitigation Planning for Real-Time Process Control

In addition to reducing costs associated with poor-quality products by promptly identifying defective parts, a key objective of IPI is defect mitigation. This process involves identifying the root cause of defects and adjusting process configurations to prevent reoccurrence. Defect mitigation aims to minimize, control, and eliminate defects in products, systems, and processes, aligning with the broader concept of in-process quality improvement—a new branch of quality science focusing on process monitoring, root cause diagnosis, and feedback/feed-forward control [29]. Similar strategies have been applied across various industries, such as using sensors for stamping process control and measuring up to 130 dimensions for car assembly process control [128,129]. Research has also underscored the importance of robust knowledge management systems for effective defect mitigation [130]. Two technical issues central to defect mitigation are process modeling of the manufacturing–inspection–mitigation control loop and defect mitigation planning for adjusting process configurations.

(1) Process Modeling of Manufacturing–Inspection–Mitigation Control Loop: This study introduces a manufacturing–inspection–mitigation (MIM) control loop, with mitigation planning as a key process-level element, aimed at enhancing quality improvement during production. The MIM control loop enables real-time process feedback and configuration adjustments, integrating inspection-based process control into the production system. Figure 9 and Figure 10 illustrate the distinction between traditional process control and control systems enhanced with s-IPI. Figure 8 shows a conventional control loop using process signal feedback, such as in-situ sensor data for operations like stamping, to adjust inputs. In contrast, Figure 9 incorporates an additional control loop with s-IPI, enabling defect diagnosis and mitigation planning to guide process configuration adjustments for improved control.

Different from the conventional control loop based on control theory, the manufacturing–inspection–mitigation control loop relies on decision-making using predefined knowledge of defects and mitigation strategies. This process involves selecting an appropriate mitigation strategy from a correlated knowledge database and determining specific settings to correct the process based on inspection results. Therefore, this study decomposes breaks down the process into three phases: defect analysis, mitigation strategy selection, and process configuration adjustment. After a part undergoes its manufacturing operation, an inspection is conducted to determine if this is a defective part. If a defect is identified, the inspection results are analyzed to determine the defect features. The next steps involve identifying the cause of the defect and selecting strategies to eliminate or reduce its occurrence. Based on the chosen mitigation strategy and defect features, corresponding adjustments to the manufacturing process configurations—whether for the current process or a previous one—are identified and implemented. By structuring the manufacturing–inspection–mitigation control loop, the system can effectively respond to defects in real time, continuously improving process quality.

(2) Defect Mitigation Planning for Process Configuration Adjustments: The implementation of the proposed control loop requires the development of a smart decision-making system to solve a series of complex cognitive tasks. Figure 11 presents the functional analysis of mitigation planning. The process begins with defect diagnosis, which analyzes the cause of the defect for mitigation strategy selection. This analysis can be achieved through data analysis tools such as decision trees, neural networks, and symbolic reasoning, utilizing both knowledge and experience. Once the defect analysis is complete, the information is used to assist in selecting an appropriate mitigation strategy to correct the process. This decision-making can be managed by the expert system that uses knowledge inference from a knowledge base containing various mitigation strategies and their application scenarios. Case-based reasoning is particularly effective in this context, as it leverages past experiences to solve new problems. In the manufacturing context, relationships between defect features and underlying cases. By searching historical cases with similar defect features, the strategies of the past instances can be reused. After identifying a suitable mitigation strategy, specific configuration adjustments need to be determined by adapting the strategy to the current defect scenario. These adjustments are then applied to correct the process that causes the defect.

An industrial example illustrating the concept of manufacturing–inspection–mitigation closed-loop control is welding inspection. Welding defects, such as porosity, slag inclusion, incomplete fusion and penetration, cracks, and undercutting, arise from various causes [131]. For instance, porosity can be affected by welding current and deposition speed; incomplete fusion and penetration are influenced by the temperature of the base metal; cracks are caused by thermal stresses and uneven cooling rates; undercutting is due to the melting of base metal. By inspecting the weld during or after the welding process, operators can halt the operation to avoid wastage of time and material. Additionally, timely identification of defect causes allows for the adjustment of parameters to prevent further deviation from the desired welding mode.

5.3. Human–Automation Symbiosis Through Task Allocation and Manufacturing Nudging

In HCPMS, interactions between humans and technology are inevitable. With the integration of s-IPI integrated into the manufacturing process, numerous knowledge-intensive cognitive tasks need to be performed by either humans or machines. Given the distinct strengths of humans and technology, the collaboration between the two significantly impacts system performance. Meanwhile, in Industry 5.0, manufacturing industries are striving to establish a symbiotic relationship between humans and automation agents within the working environment [4,9]. The goal is to leverage the cognitive and adaptive capabilities of humans alongside the precision and efficiency of automation to facilitate a closer, mutual beneficial, and more effective collaboration in the working environment.

As shown in Figure 12, this study proposes a conceptual model to elucidate HAS in a human–automation team. This model comprises three components: human agents, automation agents, and collaborative intelligence. The overlap between human agents and automation agents indicates HAI, while HAS can be further achieved with the integration of collaborative intelligence, which aims to build a human-centered working environment characterized by mutual benefit and interdependence. Several critical factors are essential for modeling HAI, including human trust, human cognitive states, machine intelligence, and the level of automation. Human trust and level of automation are crucial factors related to automation use and production performance. High trust in automation agents can lead to the overuse of automation, while low trust can result in underuse. Finding the appropriate level of automation is key to effective HAI. Cognitive states can represent the operator’s cognitive load and can be used for attention management. Human trust and cognitive states can reflect human cognition during human–automation collaboration. Machine intelligence refers to the computational capabilities for data analysis and automatic decision-making, which represents machine cognition.

With these factors, HAI aims to optimally allocate physical and cognitive resources in a human–automation team to complete production tasks. However, conventional task allocation is static and fails to capture the dynamic status of human operators in collaborative tasks. To facilitate HAS, collaborative intelligence is proposed, which is characterized by team cognition and manufacturing nudges for collaboration. Team cognition aims to model cognitive status of human operators and evaluate cognitive performance at the team level. Such shared cognition in a team allows automation agents to better adapt to human agents and facilitates human-centered allocation by considering cognitive load and well-being. On the other hand, manufacturing nudges aims to guide or influence operators’ behavior toward what is desired during their interactions with automation agents. Operators do not completely follow manufacturing instructions in practice, and nudges take effect during the execution of a task to enable more standard collaboration. In this regard, this study further identifies dynamic and adaptive task allocation and behavioral intervention design as two fundamental issues to achieve HAS.

5.3.1. Dynamic and Adaptive Task Allocation

In manufacturing systems, tasks on a production line are fulfilled by the collaboration of human agents and automation agents. Even in a highly automated system, it still requires humans to monitor the system and intervene when a failure happens in case of an out-of-the-loop performance issue [6]. As an inspection is conducted in each manufacturing process, it is imperative for humans and machines to collaborate to both conduct inspection and analyze its results when facing the high complexity in inspection criteria. Therefore, human–automation task allocation is an important and fundamental issue.

Meanwhile, the objectives of task allocation have been changing with the evolution of manufacturing technologies, and human cognition or human factors are an emergent concern in Industry 5.0. While earlier task allocation was one stage in the design process of an automated system and based on the comparison of capabilities between humans and machines [45,132], task allocation in recent years has been studied as an optimization problem, with the objective being system performance. In such a problem context, different evaluation criteria for the team performance result in different assignment results. The common performance measures include time and costs for conducting the tasks. However, to pursue HAS in the next industrial revolution, task allocation should gear toward human-centered performance evaluation and human-machine mutual adaptation while being conducted dynamically based on the real-time information. To achieve this, it is imperative to model human cognition during human–automation collaboration, and an evaluation method for team cognitive performance is needed. To maintain high production efficiency while achieving HAS, a trade-off also needs to be made between production performance and team cognitive performance.

(1) Modeling Human Cognition: In a human–automation team, task allocation is commonly formulated as an optimization problem, where the objective function is the evaluation of team performance. Unlike conventional task allocation where the objective is to optimize cost and time, the context of HAS requires dynamic allocation considering the human cognition to enable human–automation mutual adaptation in a dynamic, team-based, and distributed manufacturing operational environment [133].

To achieve this, modeling human cognition is crucial. This study identifies human cognitive states and human trust as two key perspectives of human cognition. Human cognition is an important research topic in HAI, which reflects operator well-being and cognitive load [134]. Human trust, on the other hand, reflects the operator’s confidence in automation. There are several considerations to select these two factors. Firstly, adaptive task allocation implies workload and attention management, which requires evaluating human cognitive states in real-time. Secondly, human trust reflects the confidence level of operators toward automation agents, and it is essential during task allocation to determine the level of automation for proper automation uses. Thirdly, evaluating cognitive performance at the team level requires team members to possess a shared understanding of their equipment and their coworkers [133], which can be represented by human cognitive states and human trust. To study cognitive state prediction, the relationship between physiological measurement and cognitive states needs to be modeled [135]. The modeling can be based on physiological theory or statistical learning using machine learning techniques. Human trust, a behavior mixed with confidence and attitudes, can change according to the perceived performance of automation agent compared to operator performance. It can be modeled with behavioral economics theory. Therefore, modeling cognitive states and human trust together provides a comprehensive understanding of human cognition.

(2) Team Cognitive Performance Evaluation: With an increase in the number and difficulty of cognitive tasks in the production environment, apart from maximizing production performance, the other objective of task allocation is to dynamically assign tasks by considering operators’ cognitive states, fostering human-centered collaboration while maintaining production efficiency. In this context, optimizing cognitive performance at the team level is crucial. Effective team performance in complex environments requires that team members possess a shared understanding of the tasks, their equipment, and their coworkers [133]. However, challenges remain in how to evaluate team cognitive performance and predicting it based on human cognition. Unlike production performance measures, which are outcome-oriented, team cognitive performance measures should be process-oriented and focus on how the team approaches problems. Processes reveal the cognitions and behaviors employed by team members to accomplish tasks, whereas outcomes only reflect the results of these processes and do not provide insights into how a particular outcome was reached [133]. Additionally, team cognition emerges from individual cognitions and team interactions and processes, making it possible to measure its performance at the holistic team level through observations.

(3) Trade-off between Production Performance and Team Cognitive Performance: Human–automation task allocation requires evaluating team performance from different perspectives, because evaluation influences how tasks are allocated. To achieve adaptive task allocation for HAS, team cognitive performance should be integrated as one aspect of the team performance, with manufacturing performance being the primary concern. Therefore, task allocation is essentially a multi-objective optimization problem. How to formulate the problem to accommodate different or even conflicting objectives is a complex problem that needs to be studied. It can be formulated as multi-objective optimization, where each objective is aggregated with weighted sum, or a Pareto optimal solution. Another solution to accommodate the evaluation of team cognitive performance is through the non-cooperative game, such as the Stackelberg game, which models task allocation as a bi-level optimization problem. The non-cooperative game suggests how symbiosis is achieved in a human–automation team by optimizing different team performance measures. This approach can prioritize the production performance while maintaining operator cognitive states at a high level for operator well-being.

One example is assembly line task allocation [136]. With inspection involved, different types of manufacturing operations are required to fulfill an assembly process. Tasks require either physical or cognitive workload and can be fulfilled manually or automatically with different levels of automation. In this case, it uses the weighted sum method to aggregate different performance measures, and real-time human cognitive status is considered as an input to predict the team cognitive performance. Such formulation models the operator cognitive workload and reflects cognitive performance under different task assignment, thus facilitating human-centered task allocation and avoiding potential errors brought by high cognitive workload.

5.3.2. Behavioral Intervention Design for Human–Automation Collaboration

To facilitate the efficiency of HAS, the industry calls for the consideration of human factors in manufacturing operations to build HCPMSs [4]. Nudging, as one approach of human–computer interaction, aims to alter people’s behaviors in a predictable way without forbidding any option or significantly changing their economic incentive [137]. Nudge theory is an interdisciplinary subject concerned with psychology, cognitive science, and behavioral economics. In the context of manufacturing, nudging aims to improve manufacturing performance by indirectly or directly influencing human to conduct desired behaviors, such as using HoloLens’s augmented reality to assist operators with machine maintenance. Manufacturing nudging can take different forms, such as visual nudges, auditory nudges, and somatic nudges [138]. These nudges have direct or indirect effects on human operations and transmit information from cyber systems. With manufacturing operations becoming more complex, there are growing demands for designing high-performance manufacturing nudges to improve the efficiency of human–automation collaboration. To address this issue, it is imperative to investigate the modeling of human decision-making behavior in manufacturing systems, nudging design for improving collaboration efficiency, and nudging personalization.

(1) Human Decision-making Behavioral Modeling: In the production environment, many human–automation interactions involve human decision-making, such as choosing not to work with automation agents when you have low confidence in it. Different from task allocation where tasks are directly assigned to operators, operators can make decisions independently during their collaboration with automation systems. It is therefore imperative to model human decision-making behaviors as the human dimension in HCPMS.

In the manufacturing context, operators and managers are two groups of people with conflicting interests: one group naturally wants to work less and earn more, while the other group hopes to spend less and perform more work. This shows the conflicting goals inherent in their behavioral economic concerns towards payoff and cost. In this regard, modeling human behavioral economics with conflicting goals can be helpful to understand operator decision-making during production. To solve the problem, modeling two-fold decision-making between customers and producers with behavioral economics theory is the key. Meanwhile, the influence of affective and cognitive factors on decision-making should also be modeled to characterize differences between individuals.

(2) Nudging Design for Collaboration Efficiency: This is information to present so that it can positively influence human behaviors for better performance. The design process involves three typical steps, including concept generation, evaluation, and selection. While a lot of studies have focused on analyzing the effects of nudging on human behaviors, there are insufficient studies on nudging design. Analyzing it from an engineering design perspective using mathematical modeling is meaningful for both industrial applications and academic studies. Axiomatic design can be applied to studying manufacturing nudging design, which includes four activities: customer needs, functional requirements, design parameters, and process variables [139]. Based on this system design methodology, the elements of manufacturing nudges can be formulated and are shown in Figure 13.

It is posited that manufacturing nudges $D$ are realized through two coupled aspects, which are nudging instruments $D^I$ and nudging operations $D^O$, where $D=D^I\bigcup D^O$. A nudging instrument can be understood as a platform to implement and execute nudges. In the manufacturing context, based on the form of the nudge (i.e., visual or auditory nudges, etc.), there can be different options for nudging instruments, such as smart glasses, smart wristbands, or earphones. The instrument can be virtual, too, such as an application that records operator behavior details and advocates certain code of conduct by rewarding certain behaviors, which is normally seen in service systems. Meanwhile, nudging operations are the contents of a nudge. They can be characterized by a set of nudging features $d_i$ and enabled with the instantiation of these features $d_{ik}^{*}$.

Let $d_i$ be one nudging feature, $D^O=\{d_i,i=\mathrm{1,2},\dots ,I\}$ where $I$ is the total number of nudging features. A nudging feature $d_i$ (can be referred to as design parameters) has various levels of nudging $d_{ik}^{*}$ (can be referred to as process variables), which can be represented as $d_i=\{d_{ik}^{*},k=\mathrm{1,2},\dots ,K\}$, where $k$ corresponds to the $k$-th level of the nudging feature $d_i$, and $K$ is the total number of levels for the nudging feature $d_i$. For instance, one nudging feature $d_i$ is the nudging information density for an assembly operation using augmented reality smart glasses, and a high-level feature instance and a low-level one may differ in the number and label of highlighted assembly subparts presented to the operator. With the above formulation, nudging design can be treated as a multi-criteria design evaluation problem: given a manufacturing nudge $D^O=\left\{d_{1k_1}^{*},d_{2k_2}^{*},\dots ,d_{I{k}_I}^{*}\right\}$, how to develop a mathematical approach to evaluate its utility.

(3) Nudging Personalization: Nudging personalization can be seen as an optimal configuration problem, which targets at a group of operators in one production line and aims to assign nudges according to the uniqueness of each individual, so the optimality can be achieved between HAS and manufacturing system performance.

Manufacturing nudging personalization aims to allocate proper nudges to each operator in a human–automation team based on their unique information (e.g., working experience, age, etc.) to facilitate HAS. There are two main aspects that should be considered during personalization decision-making. For one thing, manufacturing nudging aims to achieve a high symbiotic level characterized by more standard and accurate interactions between humans and technology to reduce task complexity and improve work quality. For another, applying manufacturing nudges to production implies certain engineering costs, such as additional time for production operations or implementation costs. Therefore, the manufacturing nudging personalization problem can be characterized by two metrics (one that reflects the symbiotic level and one that represents the engineering cost) and formulated as a multi-objective optimization problem.

An industrial example of manufacturing nudging is the use of Augmented Reality Smart Glasses (ARSG) for assembly tasks [140]. In advanced manufacturing, operators are increasingly allocated with tasks that require interpreting complex information and making informed decisions rather than merely following rote processes. In this context, nudging with ARSG can help interpret vast information flow in the cyber system, analyze the current situation, compare the assembly part with the CAD file, and generate assembly instructions by overlaying important information onto the operator’s field of view at proper time, thus simplifying the assembly processes and reducing errors.

6. A Research Roadmap and Prospects

The review above provides a solid understanding of the problem context from the application domain, setting the stage for its technical implementation. This section introduces a technical roadmap in Figure 13, which aims to bridge the gap between the application domain and the research domain. The roadmap highlights the identified technical solutions to address four fundamental issues, serving as an inspiration for further exploration and development in this area.

As shown in Figure 14, the technical roadmap is centered on two pillars: smart in-process inspection and human–automation symbiosis. The former addresses two fundamental issues: in-process inspection and defect mitigation planning. The goal is to ensure an uninterrupted process flow within the manufacturing system by promptly identifying and addressing defects, thereby maintaining high product quality and process efficiency. The latter is a critical aspect in Industry 5.0 that focuses on managing production operations and interactions within a human–automation team. It is concerned with task allocation and behavioral intervention design. These two enable mutual adaptation and maintain collaboration efficiency between human and automation systems.

Figure 14 also illustrates the coherence among these research focuses and connections between different research components. The core research methods are geared toward addressing several key areas: visual analytics and intelligent reasoning, Generative Pre-trained Transformer (GPT)-powered case-based knowledge modeling and reasoning, human cognition modeling and non-cooperative game theoretic optimization for task allocation, conjoint prospect theoretic modeling of human behavioral economics, and nudging behavioral modeling and multi-objective configuration optimization for nudging design and personalization. Visual analytics and intelligent reasoning provide information, which is taken as the input for defect mitigation planning through case-based reasoning. With the information of tasks in a production line, task allocation is conducted by non-cooperative game theoretic optimization, where human cognition is modeled and integrated into the optimization objective. After tasks are allocated, manufacturing nudging design and personalization are conducted to guide operator behavior during their interactions with automation systems for improving collaboration efficiency. This process involves two research questions; one is the modeling of human behavioral economics, which provides the basis for nudging design evaluation. The second is nudging personalization, which is essentially a multi-objective configuration optimization problem.

6.1. Visual Analytics and Intelligent Reasoning

This topic touches on a critical area in smart manufacturing systems, where Visual Analytics (VA) and Intelligent Reasoning (IR) are employed to enhance in-process inspection and autonomous decision-making, primarily aimed at reducing operator cognitive load. Key challenges and research issues firstly involve data collection and integration: (a) Multimodal data: Smart manufacturing generates vast amounts of data from sensors, cameras, and IoT devices. One challenge is integrating these diverse data sources (e.g., images, videos, sensor outputs) into a coherent system for VA and IR. (b) Real-time processing: In-process inspection systems need to process these data in real-time, which is computationally demanding. The challenge is to balance latency with accuracy.

In addition, visual data interpretation is fundamental, involving (a) High dimensionality: Visual data from cameras and sensors are high-dimensional and complex. Traditional inspection systems struggle with managing such data, and the development of efficient algorithms for pattern recognition and anomaly detection remains an ongoing challenge. (b) Noise and uncertainty: Visual data are often noisy, and understanding how to deal with this uncertainty while maintaining accurate reasoning is critical. Also related is cognitive load and human–machine interaction, specifically focused on (a) Cognitive overload: Traditional systems overload operators with too much information, making it difficult to process effectively. A research issue is developing VA and IR systems that simplify information delivery without sacrificing insight.(b) Adaptive interfaces: Human operators still play a role in decision-making. Designing adaptive interfaces that adjust to operator skill level and cognitive state in real-time is an unresolved challenge.

Moreover, intelligent reasoning and decision-making hinge upon (a) Autonomous decision-making: Creating systems that can reason about inspection data autonomously requires advances in AI models capable of complex decision-making without human intervention. (b) Explainability: Trusting autonomous systems remains difficult without explainable AI (XAI). Developing models that can not only make decisions but also explain why they made those decisions to human operators is a significant challenge. (c) Risk mitigation: Identifying when the system should intervene or when human intervention is required is a balancing act that involves risk analysis and understanding thresholds for autonomous action. Furthermore, scalability and deployment are critical for practical applications. For scalability, applying intelligent inspection systems across varying production environments, each with different operational constraints, is another challenge. These systems must be scalable and adaptable to different sizes and complexities of production lines. Given that inspection often occurs in real-time and with large amounts of data, the role of edge computing versus cloud-based processing needs more exploration.

To address these challenging issues, a visual analytics and intelligent reasoning system needs to be developed. There are several directions for research. Firstly, AI and machine learning integration are important and timely area of visual analytics. Advances in deep learning have shown promise in improving defect detection and anomaly identification. Neural networks can now be trained to detect even minor variations in product quality, which is a major prospect for smart inspection. Using reinforcement learning to optimize in-process decision-making can enable systems to learn and adapt autonomously over time, improving the decision-making process in dynamic manufacturing environments.

Secondly, human-in-the-loop systems suggest rich research opportunities. For example, collaborative autonomy lends itself to be an important research topic. Rather than fully autonomous systems, research is moving toward human-in-the-loop models where humans and machines collaborate in the decision-making process. These systems can take over routine decisions while deferring complex cases to human operators, reducing cognitive load while maintaining safety. Moreover, personalized assistance for systems that adapt to individual operator preferences and capabilities is on the horizon. By understanding the cognitive state of the operator, VA systems can modify their interaction style and feedback mechanism, providing personalized decision support.

Thirdly, edge and cloud computing synergies are necessary enabling technologies. For example, edge AI, aiming at the integration of AI capabilities at the edge, will improve real-time processing capabilities for in-process inspection. This mitigates latency issues and allows for quicker decision-making on the shop floor. Also, hybrid cloud architectures underscore a promising area for development by combining cloud resources for historical data analysis and edge systems for real-time inspection. This would allow manufacturers to balance processing power and latency.

Fourthly, interpretable AI models will be a worthwhile area of research. As AI models become more integrated into autonomous decision-making, XAI models will be critical for justifying decisions to human operators and mitigating potential risks from incorrect decisions. These models will improve trust in autonomous systems by offering more transparent decision pathways, leading to increased adoption in industrial settings.

Fifthly, predictive maintenance and self-healing systems are another emerging area attracting much attention from both academia and industrial practitioners. Predictive analytics by moving from reactive to predictive inspection systems is another area of prospect. Leveraging machine learning models for predictive maintenance can prevent downtime by identifying issues before they become critical. Self-healing systems are a potential future direction, in which autonomous systems that not only detect but also correct issues (e.g., through machine learning-driven adjustments in machine parameters).

Moreover, enhanced visualization technologies will pave the way for practical applications. For example, augmented and virtual reality (AR/VR) by integration of AR/VR into visual analytics for operators can provide immersive insights, enabling more intuitive decision-making and minimizing the need for extensive training or deep technical expertise. Also, moving beyond 2D visual analytics, future systems might employ 3D visualizations for more nuanced inspection and analysis of complex manufacturing processes, for which 3D data representation lends itself to be a fundamental research issue.

The integration of Visual Analytics and Intelligent Reasoning in smart manufacturing systems holds immense potential to transform in-process inspection and decision-making. While many challenges remain, especially in handling complex visual data, autonomous reasoning, and maintaining human trust, emerging technologies such as deep learning, reinforcement learning, edge computing, and XAI show promise for mitigating these challenges. As these technologies evolve, the cognitive load on human operators will significantly decrease, leading to more efficient and safer manufacturing processes.

6.2. GPT-Powered Case-Based Knowledge Modeling and Reasoning

Manufacturing defect mitigation involves several symbolic reasoning tasks that rely on domain knowledge. Enabling cognitive intelligence for domain knowledge-intensive decision-making is crucial for improving the effectiveness of defect mitigation. Cognitive intelligence allows systems to analyze complex problems, understand context, and generate solutions that are nuanced and well informed by domain-specific knowledge. Such systems should adapt to new information and changing circumstances, offering more flexible and resilient decision-making capabilities.

There are several challenges to implement cognitive intelligence in manufacturing defect mitigation. Firstly, domain knowledge acquisition is a manual and time-consuming process. This process involves gathering information from documented details about manufacturing processes. Secondly, efficient domain knowledge modeling is essential to ensure the knowledge base is easily accessible for further applications. This requires developing a structured and standardized format for capturing and organizing information. Efficient knowledge modeling can facilitate knowledge retrieval and inference. Thirdly, selecting mitigation strategies and adjusting configurations are complex symbolic reasoning tasks that demand a deep understanding of domain knowledge and past mitigation cases. These tasks involve analyzing defect features, understanding the underlying causes, and identifying appropriate actions to prevent recurrence. This process requires integrating historical data with current scenarios to determine the most effective interventions.

To solve these challenges, a cognitive intelligent case-based reasoning approach can be developed using LLMs and knowledge graphs. Defect mitigation is a good application of case-based reasoning, which excels in leveraging past experiences to solve new problems. In manufacturing processes, relationships exist between defect features and underlying causes. Case-based reasoning allows for efficient reuse of previous successful mitigation strategies by matching current defect scenarios with historical cases. As cases accumulate, the knowledge base also improves with these new cases, enhancing the system’s ability to defect mitigation. Meanwhile, its adaptability also makes it suitable for the dynamic and complex manufacturing environment. Under the original case-based reasoning framework, this study further proposes the following changes: knowledge graphs are employed for knowledge representation, GenAI is utilized for knowledge acquisition in the form of head-relation-tail and for case adaptation by developing a retrieval-augmented generation (RAG) application. These changes facilitate the usability and flexibility of case-based reasoning for defect mitigation.

6.3. Human Cognition Modeling and Non-Cooperative Game Theoretic Optimization for Task Allocation

Cognitive states and human trust are two critical factors that add the human cognition to a cyber–physical system; thus, how to model them should be studied. Accurately sensing and assessing cognitive states are essential for dynamically identifying changes in human operators, allowing automation agents to adapt accordingly. However, unless using self-report data for subjective measurement, the objective acquisition of cognitive states requires the comprehension and integration of multimodal physiological data. Similarly, human trust is a critical indicator in HAI, reflecting the perceived risk and uncertainty associated with various level of automation. High trust leads to increased reliance on automation, while low trust results in reduced usage. Thus, it is necessary to develop a value-based model to quantify trust as predictable behavior under uncertainty. To solve the two problems, this study proposes a Transformer-based model for predicting human cognitive states. Meanwhile, prospect theory is employed to assess human trust.

On the other hand, human–automation task allocation requires evaluating team performance from different perspectives, because evaluation influences how tasks are allocated. To achieve cognitive intelligent task allocation, team cognitive performance should be modeled and integrated as an aspect of the team performance, with manufacturing performance being the primary concern. Therefore, human–automation task allocation is essentially a multi-objective optimization problem. How to formulate the problem to accommodate different or even conflicting objectives is a complex problem that needs to be studied. To formulate the task allocation problem, this study uses the Stackelberg game model with a bi-level structure. The leader model optimizes production performance, and the follower model optimizes team cognitive performance. Additionally, unlike the completion cost and time of a task, cognitive performance lacks quantifiable measures. As an important evaluation metric that influences task allocation to achieve HAS, developing a mathematical model to predict cognitive performance using available measures is essential. To achieve this goal, team cognitive performance is evaluated from three aspects: task characteristics, human cognitive states, and human trust. Task characteristics define the complexity and workload of a task. Human cognitive states describe the individual differences in terms of cognitive load. Human trust can be used as a good indicator of shared cognition. It is possible to define team cognition by the trust value function such that it becomes correlated with each operator’s perceived trustworthiness of different settings of level of automation in the automation agents [20]. By analyzing task characteristics, current cognitive states, and human trust, multinomial logistic regression is used to predict team cognitive performance. With task allocation formulated as a non-cooperative game, solving the optimization problem also requires an optimization algorithm for game theoretic optimization.

6.4. Conjoint Prospect Theoretic Modeling of Human Behavioral Economics

People make decisions every day for various activities, and there are more scenarios where advanced techniques like AI and big data are used for decision support. Meanwhile, in the manufacturing context, more interactions are needed between human and cyber–physical systems with complex activities and decision-making allocated to machines and algorithms, making the role of humans more important [4]. This requires human perspectives to be considered during the engineering design process. In this regard, behavioral design has emerged as a new and important domain of design research.

At the same time, it is common that economic decision-making activities involve multiple groups of people with different interests, and a customer–producer relationship is one typical instance where two groups have conflicting interests. This happens not only in operational systems but also in manufacturing systems, such as with operators and plant managers. Therefore, to design product or service systems in these scenarios, it is important to model human economic decision-making behaviors with a two-fold perspective—an interplay of customers and producers—due to the conflicting goals inherent in their behavioral economic concerns toward payoff and cost.

In this regard, modeling human behavioral economics with conflicting goals is an important issue. There are two challenges for the implementation. The first challenge is formulating the economic utility of a choice to characterize human behavioral patterns during decision-making. This involves selecting a proper model that can capture the irrationality or cognitive biases and aggregating the perceptions of a choice from both customers’ and producers’ perspectives. The second challenge is the modeling of affective cognition to reflect emotional influences in decision-making. Psychological factors play an important role when people make choices, and the developed behavioral model should consider how affective cognition will fit into the utility function, thus changing the perception of a choice. To address the two challenges, a novel conjoint prospect theoretic modeling approach can be used for human behavioral economics toward designing products or services in manufacturing [141].

6.5. Nudging Behavioral Modeling and Optimization for Nudging Design and Personalization

To evaluate different manufacturing nudges, nudging behavior modeling is one necessary research area that provides a quantifiable method to represent the nudge utility toward its stakeholders and users. There are several requirements for the modeling. Firstly, the model is expected to characterize the conflicting goals between stakeholders and users. Opposite perceptions inherently exist in the manufacturing context between plant managers and manufacturing operators. From the manager’s perspective, manufacturing nudges help improve system performance. From the operator’s perspective, they bring extra workload and cognitive burdens to operators apart from their given workload. Secondly, the model should be able to characterize the varying perceptions of different user segments in different nudging scenarios. This enables the evaluation of the same nudge on different individuals for customization and personalization purposes. Thirdly, the model should reflect the influence of human affective cognition on nudge perception. HAS advocates a human-centered design philosophy, enabling the selection of nudging operations to fit different operator cognitive states.

On the other hand, as manufacturing nudging helps realize a symbiotic relation between humans and automation systems, it inevitably brings down certain aspects of manufacturing system performance (e.g., operation costs and time). The objective of nudging personalization lies in the achievement of HAS while maintaining system performance for a group of people at the same time, which suggests the formulation of a multi-objective optimization problem. While HAS can be quantified by the result of nudging behavior modeling of selected nudges, there a corresponding engineering cost function that reflects the loss of system performance is needed to execute these nudges. Meanwhile, how the objective function should be formulated to reflect the relationship between manufacturing nudging and its engineering costs needs to be studied. To address the three challenges, cumulative prospect theory with weighted choice probability through the conjoint prospect value can be used to model nudging behaviors. Furthermore, problem formulation of nudging personalization can be set as the maximum of the HAS objective function divided by the engineering cost for the user group.

7. Concluding Remarks

This paper identified four fundamental issues of s-IPI with HAS in HCPMS and proposed a technical roadmap with five technical approaches to address the proposed fundamental issues. A comprehensive review revealed that s-IPI is pivotal in maintaining high product quality and enabling real-time process control in the dynamic and complex environments characteristic of Industry 5.0. The study has been conducted across three hierarchical levels of manufacturing systems—the workstation, process, and system levels—each presenting distinct challenges and solutions. At the workstation level, the focus is on the implementation of vision-based defect inspection, highlighting the importance of accurate and reliable defect identification and the necessity of automated decision-making for defect disposition. The integration of in-process inspection within the manufacturing process not only enhances defect detection capabilities but also streamlines the workflow by reducing resource wastage and operational disruptions. At the process level, defect mitigation planning through real-time process control was examined. The establishment of a manufacturing-inspection-mitigation control loop is essential for continuous quality improvement. By leveraging inspection feedback, manufacturers can dynamically adjust process configurations to mitigate defects, thereby enhancing overall process efficiency and product quality. At the system level, the focus is put on achieving HAS through dynamic and adaptive task allocation and behavioral intervention design. A conceptual model that integrates human cognition and trust for HAS is proposed, facilitating mutual adaptation between human operators and automation agents. This approach ensures optimal collaboration efficiency and maintains high production performance. Additionally, the design and personalization of manufacturing nudges aim to influence operator behavior, further enhancing human–automation collaboration.

The paper outlined a holistic framework and a technical roadmap for integrating s-IPI with HAS in HCPMS. Five research topics are envisioned as technical solutions to the identified issues. By addressing the four fundamental issues—defect inspection, defect mitigation planning, dynamic and adaptive task allocation, and behavioral intervention design—this research paves the way for future advancements in smart manufacturing. The adoption of these strategies will not only improve operational efficiency and product quality but also foster a symbiotic relationship between human operators and automation systems, ultimately driving the evolution of manufacturing systems in the era of Industry 5.0.

Some limitations exist for this research proposal. Firstly, Industry 5.0 is a broad concept that contains a lot of topics, and this proposal aims to study how to achieve HAS in the context of s-IPI. To be specific, it considers operations management for a manufacturing system with s-IPI from the task planning and execution perspectives. Secondly, this proposal does not discuss topics related to manufacturing system responsiveness. While s-IPI provides more real-time feedback about the manufacturing process, it also requires the system to be more responsive toward disruptions brought by inspection results, such as through dynamic production scheduling. Thirdly, compared to the original use case of the prospect theory, the evaluation of prospect values change from economic gains and losses to product likes and dislikes, meaning the evaluation is conducted on subjective metrics instead of objective metrics. More research can be done to validate the application of prospect theory in such a problem context.