Human-in-the-Loop Digital Twin Modeling for Smart Civil Infrastructure Operation and Maintenance

Abstract

Traditional inspection and diagnosis methods for civil infrastructure operation and maintenance (CI O&M) rely heavily on human efforts. Such efforts are always affected by subjective judgment and human errors due to engineering knowledge and prior experiences of field engineers. On the other hand, recent development of AI-driven tools could achieve effective information acquisition but lacks interpretability and engineering credibility. How to integrate human knowledge with AI capacity for safe and effective CI O&M is thus necessary in this new era. This paper presents a human-in-the-loop digital twin (HITL-DT) framework that enables safety risk sensing, prediction and control for smart CI O&M. The proposed framework fuses human cognition (i.e., individual perception and team situation awareness), AI and engineering knowledge for 1) risk sensing and diagnosis based on spatiotemporal changes and 2) risk prediction and control for smart CI O&M. Qualitative analysis indicates that the HITL-DT approach produces more explainable, trustworthy, and actionable diagnostic outputs, which enhance the reliability and proactivity of CI O&M.

1. Introduction

With the continuous expansion of global infrastructure networks and the growing number of structures that are reaching or exceeding their design service life, the operation and maintenance (O&M) of civil infrastructure (CI) is facing increasing complexity. Data from the New Climate Economy and McKinsey show that global infrastructure assets now exceed USD 100 trillion. Around 60% of bridges and roads are already in mid- or late-life stages and are entering periods of concentrated aging and performance decline [1]. According to the Federal Highway Administration (FHWA) and the ASCE report based on the National Bridge Inventory (NBI), the United States has more than 619,000 bridges. Over 42% of these bridges have been in service for more than 50 years, and about 7.5% are classified as structurally deficient [2]. Studies by the OECD and the European Union further show that many key bridges and tunnels in the core Trans-European Transport Network (TEN-T) were built in the 1960s and 1970s and now face pressure from outdated functions and delayed rehabilitation [3].

At the same time, the digitalization level of infrastructure O&M is rising worldwide. Smart monitoring devices, large-scale sensor networks, and automated inspection systems are driving exponential growth in data volume. In intelligent transportation systems and infrastructure IoT deployments, city- and nation-level networks rely on millions of sensor nodes, and a single large monitoring system can generate gigabytes to terabytes of high-frequency data each day [4,5]. According to the Ministry of Transport of China, the total length of rural roads exceeded 4.5 × 10⁶ km by the end of 2023. In several provinces and cities, automated pavement condition surveys using inspection vehicles and digital platforms have achieved or nearly achieved a 70% automation rate, which demonstrates a clear improvement in the digitalization level of rural road O&M [6]. These trends indicate that modern CI has become a large-scale and highly complex system with multi-scale characteristics, wide spatial coverage, diverse structural forms, and heterogeneous data sources.

Although many countries are promoting smart monitoring, health diagnosis, and digital management, current CI O&M systems are approaching the practical limits of purely data-driven automation when applied to infrastructure networks of this scale and complexity. This challenge highlights the need for digital twin (DT) frameworks that go beyond data aggregation and visualization, and that can explicitly incorporate human expertise, engineering knowledge, and explainable reasoning mechanisms.

DTs have become an important technical pathway for CI O&M. A DT establishes a dynamic mapping among the physical structure, the virtual model, and real-time multi-source data. This mapping enables state visualization, behavior prediction, and quantitative decision support, which strengthens monitoring, diagnosis, and risk assessment. However, recent reviews in leading international journals show several major limitations in current DT applications for infrastructure: (1) high cost and complexity of multi-source data fusion and real-time updating; (2) weak integration between DT workflows and organizational or human decision processes; and (3) limited capability to formalize structural codes and engineering knowledge within DT systems. In parallel, deep learning and data-driven methods in Structural Health Monitoring (SHM) have achieved notable progress but still face key challenges, including incomplete data, weak explainability, poor scene generalization, and limited ability to absorb expert knowledge and engineering standards. Recent studies report that common SHM models for images and sensor data exhibit poor scalability, weak real-time performance, insufficient coupling of multi-variable features, and limited interpretability, which restrict real engineering deployment [7]. Similar conclusions have been repeatedly emphasized in multimodal SHM research.

Compared with fully automated monitoring algorithms, traditional manual inspection and expert-based judgment retain strong advantages in understanding structural codes, recognizing subtle abnormalities, and handling complex field conditions. However, this human-centered approach faces severe scalability constraints when applied to large-scale infrastructure networks. According to the White Paper Development of China’s Rural Roads in the New Era issued by the State Council Information Office, China completed the rehabilitation of 58,000 unsafe rural bridges over the past decade. The proportion of Class I, II, and III rural road bridges increased from 83.2% to 98%, and the number of unsafe bridges declined each year [8]. This achievement required extensive human and material resources within a concentrated time period, showing that large-scale O&M relies heavily on massive resource input. Under routine O&M conditions, the rapidly increasing number of structures, the accelerating aging of bridges, and the combined impact of hidden damage and extreme events make it difficult for manual inspection and human-led decisions to provide full coverage in time, space, and precision.

The analysis above reveals a critical research gap: existing CI O&M approaches remain polarized between data-driven automation with limited interpretability and human-centered assessment with limited scalability, while systematic integration mechanisms between humans, AI, and engineering knowledge remain underdeveloped.

To address this gap, this study proposes a human-AI-knowledge (HAK) integrated digital twin framework with explicit human-in-the-loop (HITL) interaction for CI O&M. The proposed framework combines human experience and cognition, structural codes, engineering knowledge, and explainable artificial intelligence within a unified DT environment. Through knowledge formalization, human experience modeling, explainable AI reasoning, and multi-source spatiotemporal data fusion, the framework supports intelligent recognition of structural changes, mechanism-based diagnosis, and risk prediction. This study does not aim to improve performance only through data-driven optimization. Instead, it focuses on improving engineering interpretability, diagnostic credibility, and practical feasibility in DT-based CI O&M systems. The HITL interaction links the device layer, the model layer, and the decision layer. In this way, the system becomes more reliable and more transparent, and it can be applied more easily in real projects. It also supports continuous development, joint participation, and long-term knowledge storage. In this study, we mainly show that the proposed framework is feasible in practice, logically consistent, and reasonable from an engineering point of view. We do not focus on quantitative performance comparison in this paper. This will be studied in future work.

2. Literature Review

2.1. Digital Twin Modeling

DTs are a key technology in CI. They link physical structures with virtual models and support real-time reflection and prediction of structural conditions. From a system engineering point of view, a DT has continuous and two-way mapping between the physical entity, the virtual model, and related data streams. Because of this mapping, the digital model can change over time and follow the operating state of the infrastructure [9].

In construction and infrastructure projects, DT modeling usually uses geometric information, material parameters, and sensor networks to build accurate models of the physical system [10]. In the virtual domain, many studies use hybrid modeling methods. These methods combine mechanism-based, data-driven, and knowledge-based approaches. Abualdenien and Borrmann show that clearly defined meta-models and multi-level model structures are important for keeping consistency and allowing models to work at different levels of detail [11]. Chiachío et al. show that virtual models should be updated and checked continuously with multi-source monitoring data. In this way, a closed loop can be built between the physical system and the digital system [12]. Jiang et al. also propose a multi-domain DT framework. This framework organizes different types of data, models, and semantic information in one unified structure, so it can support complex infrastructure systems [13].

Recent studies show that DT development is changing. In the past, it mainly focused on a “physical-virtual” two-domain framework. Now, it is moving to a more integrated “human-machine-knowledge” framework. In this new view, human interaction and knowledge are treated as core parts of the DT system. They are not seen as external inputs anymore. A recent review on human-centered DTs shows that human-machine collaboration and knowledge integration are important for reasoning and decision-making in complex and uncertain situations [14]. These studies show that DT modeling is changing from a data-centered approach to a knowledge-centered and human-centered approach. Because of this change, DTs can provide a more reliable foundation for intelligent O&M.

2.2. Human Factors in Civil Infrastructure Operation and Maintenance (CI O&M)

Human factors are very important in CI O&M. They directly affect monitoring quality, risk judgment, and decision reliability. Many studies show that structural inspection and defect detection still depend a lot on the experience and professional knowledge of field engineers. When compared with fully automated systems, humans are usually more sensitive to small abnormalities, complex environmental effects, and rare operating conditions. Reason’s classical work shows that human judgment and understanding of context are important safety barriers in complex systems [15]. Hollnagel also shows that situational awareness and adaptability are important for dealing with unexpected events in real operations [16].

In infrastructure monitoring practice, many studies show that automated sensing and robotic inspection technologies improve efficiency. But human expertise is still very important. Ahmed et al. show that non-destructive bridge evaluation now uses robotic platforms and advanced sensors more often. But experts are still needed to judge how serious the damage is and to connect inspection results with engineering knowledge [17]. Sun et al. show that structural health monitoring becomes more reliable and easier to understand when HITL mechanisms are used. This is important for damage interpretation, risk assessment, and decision checking in uncertain situations [18].

At the same time, sole reliance on manual inspection has clear limitations. Large-scale infrastructure networks and high-frequency monitoring tasks impose significant cognitive and physical burdens on inspectors. Al-Sabbag et al. show that human-machine collaboration, supported by interactive visualization and mixed reality, can enhance inspection effectiveness by enabling engineers to verify and correct automated detection observations [19]. From a broader perspective, Asad et al. argue that human-centric DTs should explicitly incorporate human perception, interaction, and decision-making capabilities to support effective collaboration between engineers and intelligent systems [20]. Recent overviews of DT applications in structural health monitoring further confirm that human expertise remains necessary for model calibration, anomaly interpretation, and maintenance planning in complex operational environments [21].

Human factors are a strength of traditional CI O&M practices. They are also an important part that future DTs and intelligent monitoring systems must consider. When human experience, situational understanding, and decision-making processes are included, the system becomes stronger. This forms the basic foundation for the proposed HAK-integrated O&M framework.

2.3. AI for Civil Infrastructure Operation and Maintenance (CI O&M)

AI is now an important tool in CI O&M. It is widely used in SHM, damage detection, and performance evaluation. Many recent reviews show that machine learning methods can automatically extract features from vibration data, strain data, and image data. These features help with damage detection and condition assessment through pattern recognition [22,23]. Azimi et al. review many studies and show that deep learning models, especially CNNs, perform well in damage detection tasks. These tasks include crack detection and structural anomaly recognition. Because of this, inspection efficiency and automation levels are improved [24].

In sensor-based monitoring, AI shows clear advantages in processing large and different types of data. Gharavi et al. show that AI-based SHM systems use piezoelectric sensor networks with different types and positions. These systems can capture complex structural responses and improve the accuracy of damage location under different operating conditions [25]. Recent studies also show that deep learning methods can learn time and space relationships from long-term monitoring data. This supports structural condition evaluation and maintenance decisions [26]. These abilities are very useful in CI O&M situations with continuous sensing and large amounts of data.

AI-based methods have many advantages. But they still face clear challenges in real engineering applications. Review studies show that data-driven models are sensitive to data quality, sensor placement, and changes in operating conditions. These factors can reduce their robustness and generalization ability in real infrastructure systems [26]. Plevris and Papazafeiropoulos also point out that AI models for SHM and infrastructure maintenance must be combined carefully with engineering knowledge and domain rules. This is necessary to ensure reliability and safety, especially in rare events or complex damage situations [27]. Because of these limitations, AI methods in CI O&M should not only use data efficiently. They should also follow engineering knowledge and constraints.

2.4. Human-AI Collaborations

Human-machine collaboration is now an important ability in intelligent CI O&M. The main idea is to combine engineers’ practical judgment with the speed and consistency of artificial intelligence. In this way, monitoring and decision-making become more reliable and easier to understand. Many recent studies show that intelligent systems should support active human involvement. This includes supervision, intervention, and correction. They should not completely replace human decision-making. Wang et al. show that human-centered and design-oriented computational thinking helps engineers better understand and control intelligent systems. This is important for complex engineering tasks that need continuous interaction between humans and algorithms [28].

In engineering and built-environment fields, many studies show that human participation can support automated reasoning. Humans provide contextual understanding and qualitative judgment. Abouelela shows that human-centered design is important for creating working environments that function well and are dependable. This means that system performance is closely related to human perception and interaction [29]. Robitzsch also shows that structured human judgment can be added to data-driven models. This can improve robustness and reduce uncertainty when the data are complex or noisy [30]. These results show that human involvement can reduce the weaknesses of fully automated systems.

Human involvement is also very important for keeping systems adaptable and trustworthy [31]. Jaryani et al. show that adding structured knowledge representations to intelligent systems makes interaction between human operators and computational modules more transparent. This helps people and systems make decisions together in engineering applications [32]. A recent review by Firmino de Souza et al. also shows that transparency, feedback, and the ability for human intervention are important factors for building trust in human-machine collaboration [33].

Although existing research has advanced DT modeling, human-centered system design, artificial intelligence, and trust-aware interaction, these efforts remain largely independent. Current studies rarely provide a unified framework that can jointly represent human experience, AI reasoning, and structured engineering knowledge within a single DT system. As a result, limitations remain in reliability, explainability, and generalization, which motivates the need for an integrated HAK framework addressed in this study.

3. A Human-in-the-Loop Digital Twin (HITL-DT) Framework for Smart Civil Infrastructure Operation and Maintenance

This section presents the proposed methodology for an HITL-DT framework designed to support smart operation and maintenance of civil infrastructure. The framework formalizes the integration of physical entities, virtual entities, and bidirectional interaction mechanisms within a digital twin environment. By integrating human experience (H), AI-based reasoning (A), and engineering knowledge (K), the proposed HITL-DT methodology enables dynamic monitoring of structural conditions, prediction of state evolution, and engineering-consistent risk assessment. As illustrated in Figure 1, these three components jointly constitute an HAK-integrated digital twin system, forming the methodological foundation for the subsequent diagnostic and decision-support processes.

Figure 1 illustrates the overall HITL-DT modeling framework and its operational workflow for smart CI O&M. The framework is organized around a multi-layer digital twin architecture, where physical entity modeling, virtual entity modeling, and physical-virtual interactions constitute the core DT layers. Human experience integration, AI interaction, and engineering knowledge integration are incorporated as complementary but essential components that interact with the DT through validation, constraint, triggering, and consistency-check mechanisms. Rather than representing simple information exchange, the directed links indicate how human judgment, AI-based inference, and engineering knowledge collaboratively constrain, activate, and verify DT-based diagnosis and decision processes. Through this closed-loop interaction, the DT functions as a coordination and decision-support hub, enabling interpretable, engineering-consistent, and operationally feasible smart O&M.

3.1. Digital Twin (DT) Modeling for Smart CI O&M

DT modeling is a core component of intelligent infrastructure O&M. By constructing the physical entity, the virtual entity, and the bidirectional interaction between them, a DT achieves real-time structural state mapping and decision feedback. The physical entity provides a computable physical foundation and supports analyses such as finite element simulation while staying synchronized with sensor data and inspection records. The virtual entity builds a computational representation of the structure through mechanism-based models, data-driven models, and knowledge-based models, which enables reliable prediction of structural responses. The physical-virtual interaction ensures synchronized updates and closed-loop reasoning between the physical and virtual domains and forms the basis for integrating human experience, AI-based reasoning, and engineering knowledge.

3.1.1. Physical Entity Modeling

The physical entity is the basic layer of a DT. Its goal is to represent the real physical properties of infrastructure in a way that can be calculated and updated. Traditional BIM focuses on static representation. But the physical entity focuses on engineering analysis and time-based updates. The model must support calculations such as finite element analysis and dynamic response simulation. It must also update continuously using data from sensors, inspections, and O&M activities. The physical entity gives a real and reliable basis for virtual model simulation, human experience integration, AI reasoning, and knowledge rules. Because of this, the three parts of HAK can work in the same physical context and cooperate well. In Formula (1), the physical entity is described as a time-changing physical state. This state is driven by sensing data, inspection data, and O&M data from different sources.

$$\begin{array}{c}{\mathrm{X}}_p\left(t\right)=\mathsf{\Phi }\left(\mathrm{S}\right(t\left)\right),\hspace{1em}\mathrm{S}\left(t\right)=\left\{{\mathrm{D}}_s\right(t),{\mathrm{D}}_i(t),{\mathrm{D}}_{O\&M}(t\left)\right\}\end{array}$$

(1)

$$Z_p\left(t\right)=\mathcal{F}\left(S\right(t),I(t),O(t\left)\right)$$

(2)

$$\begin{array}{c}{X}_p\left(t\right)=\mathcal{U}\left(X_p\right(t-\Delta t),\hspace{0.17em}{Z}_p(t\left)\right)\end{array}$$

(3)

In Formula (1), ${\mathrm{X}}_p\left(t\right)$ is the time-changing physical state of the infrastructure. It includes geometric properties, material parameters, and structural condition indicators. ${\mathrm{D}}_s\left(t\right)$ is the sensor data from the monitoring system. ${\mathrm{D}}_i\left(t\right)$ is the inspection and survey data. $D_{O\&M}\left(t\right)$ is the O&M records collected during service. The mapping function Φ describes a physics-based transformation. It combines different types of input data into an updated physical state. It focuses on continuous updates and not static representation. Because of this formulation, the physical entity gives a computable and unified physical base. This base supports cooperation between human experience, AI reasoning, and engineering knowledge in the DT.

Formulas (2) and (3) explain the internal process of the mapping function Φ. They divide it into two steps. In Formula (2), $Z_p\left(t\right)$ is a set of structural observations that follow physical rules. These observations come from combining multi-source data. The data include monitoring data $S\left(t\right)$ inspection and survey records $I\left(t\right)$, and O&M information $O\left(t\right)$. The fusion operator $\mathcal{F}$ aligns data in time and space. It also changes data meaning when needed and reduces noise. In this way, different inputs become unified observation variables in the same physical context. They are not simply put together. These observation variables show structural behavior. They include deformation features, damage-related indices, and correction terms linked to maintenance actions.

Based on these unified observations, Formula (3) shows how the physical entity changes through a physics-constrained state update process. Here, $X_p(t-\Delta t)$ is the physical state at the previous time step, and $\mathcal{U}$ is an update operator. It combines new observations $Z_p\left(t\right)$ with the past state under physical and engineering constraints. This formulation keeps the state continuous over time. It stops non-physical jumps in the state. So the physical entity can change step by step when new monitoring, inspection, and O&M data are available.

3.1.2. Virtual Entity Modeling

The virtual entity is the computing core of a digital twin. It supports structural visualization, behavior simulation, and mechanism-based reasoning. Its purpose is to copy the physical structure. It also provides a modeling space for condition understanding, change prediction, and O&M decision support. The virtual entity usually includes three types of models. The first type is mechanism-based models. These models apply physical constraints so structural responses can be calculated under loads, environmental effects, and damage conditions. The second type is data-driven models. These models use monitoring data for state identification and performance prediction. The third type is knowledge-based models. These models include engineering codes, construction logic, and past cases. They help improve clarity and engineering consistency. These models work together in a modular way. They form one reasoning space that works well with large amounts of data and with limited data. The virtual entity stays synchronized with the physical entity through sensor data, inspection results, and O&M records. In this way, the virtual entity can change over time. This idea is described in Formula (4).

$$\begin{array}{c}{\mathrm{X}}_v\left(t\right)=\mathsf{\Psi }\left({\mathrm{M}}_{phys}\right(t),{\mathrm{M}}_{data}(t),{\mathrm{M}}_{know}(t\left)\right)\end{array}$$

(4)

$$\begin{array}{c}{\mathsf{\Omega }}_{phys}(t)=\mathcal{G}(M_{phys}(t))\end{array}$$

(5)

$$\begin{array}{c}{\tilde{X}}_v(t)=\mathcal{H}(M_{data}(t){\mathsf{\Omega }}_{phys}(t))\end{array}$$

(6)

$$\begin{array}{c}{X}_v(t)=\mathcal{K}({\tilde{X}}_v(t),\hspace{0.17em}{M}_{know}(t))\end{array}$$

(7)

In Formula (4), ${\mathrm{X}}_v(t)$ is the virtual state of the DT at time t. It describes structural behavior and condition in a computational way. ${\mathrm{M}}_{phys}\left(t\right)$ is the mechanism-based model. It includes structural mechanics and physical constraints. This keeps structural responses physically clear and computable. ${\mathrm{M}}_{data}\left(t\right)$ is the data-driven model. It uses monitoring data for state identification and performance prediction. It can adapt when data availability changes. ${\mathrm{M}}_{know}\left(t\right)$ is the knowledge-based model. It includes engineering codes, construction logic, and case experience. It improves clarity and engineering consistency. The mapping function Ψ combines these different models into one reasoning space. This reasoning space stays synchronized with the physical entity over time. Because of this, the virtual entity becomes a computing base that supports later human input and AI reasoning in the DT.

Formulas (5) to (7) explain this integration process in three steps. In Formula (5), ${\mathsf{\Omega }}_{phys}(t)$ is the physics-based feasible space of the virtual entity at time $t$. It is the set of possible structural states limited by structural mechanics, material properties, and boundary conditions. The operator $\mathcal{G}$ translates mechanism-based models into physical constraints. These include equilibrium conditions, constitutive relations, and parameter limits. In Formula (6), ${\tilde{X}}_v(t)$ is an intermediate virtual state. It is inferred by data-driven models before knowledge-based checking. The operator $\mathcal{H}$ performs state identification and performance prediction using monitoring data. It works under the limits of the feasible space ${\mathsf{\Omega }}_{phys}(t)$. This keeps the data-driven result physically acceptable. In Formula (7), $X_v\left(t\right)$ is the final virtual state of the DT after engineering validation. The operator $\mathcal{K}$ combines the intermediate state ${\tilde{X}}_v(t)$ with the knowledge-based model $M_{know}\left(t\right)$. This model includes engineering codes, construction logic, and case experience. It checks and improves the virtual state. This improves engineering clarity, consistency, and readiness for decision-making.

3.1.3. Physical-Virtual Interactions

Physical-virtual interaction is the main mechanism in a digital twin. It allows real-time mapping and closed-loop reasoning. It keeps the physical structure and the virtual entity synchronized at the state, behavior, and knowledge levels. Sensor data, inspection records, and O&M information move from the physical domain to the virtual entity. They update model parameters, damage states, and behavior predictions. At the same time, diagnostic results, performance evaluations, and risk alerts are produced in the virtual domain. These results are sent back to support physical O&M decisions. In this way, a closed operational cycle is formed. Effective interaction requires not only reliable data transmission but also rapid model recalibration when new information becomes available. Under incomplete or noisy multi-source data, mechanism-based models, AI analysis, and knowledge rules jointly constrain updates to enhance robustness and credibility. Through this bidirectional interaction, the DT enables dynamic state tracking, behavior prediction, and decision support, providing the operational foundation for coordinated HAK integration.

3.2. Human Experience Integration

Human experience integration aims to incorporate the field perception of engineers, the shared cognition formed through team collaboration, and the experience-based decision logic accumulated from long-term practice into the DT in a systematic way. By converting human advantages in anomaly recognition, situational understanding, and decision-making into recordable and computable inputs, a DT gains more reliable judgment capability under complex, ambiguous, or data-limited operating conditions. As illustrated in Figure 2, these human-derived capabilities can be transformed into structured information that supports more robust interpretation of structural conditions.

Figure 2 presents the flow of human experience from field cognition, team situation awareness, and experience-based decision-making into the DT through a series of recordable and computable outputs. These include labels and annotations from individual perception, reasoning chains and consensus observations from team collaboration, and priority scores or decision rules from experiential judgment. These structured inputs are added to the virtual entity. This gives the DT better contextual understanding. It also makes later reasoning, prediction, and decision support more reliable.

3.2.1. Human Cognition

Human cognition shows the special role of engineers in field inspection, visual recognition, and situational judgment. It is the first step in adding human ability into a DT. In real O&M work, engineers can find small cracks, material problems, and geometric changes through visual checks, touch, and experience. These features are often hard for automated systems to detect. Engineers also understand environmental conditions, construction history, and structural weaknesses. This helps them make judgments in complex or unclear situations. To let a DT use human input, these thinking processes must be changed into structured information. The information must be recorded, described, and computed. For example, inspection notes and simple damage judgments can be changed into labels, rules, or semantic forms. These forms can interact with the virtual entity. Human perceptual ability works together with the continuous monitoring ability of the DT. Because of this, the system can make more reliable recognition and judgment when data are missing, conditions change quickly, or new damage patterns appear. This base supports later team decisions and AI reasoning.

3.2.2. Team Situation Awareness

Team situation awareness is the shared understanding of an O&M team during inspection, data interpretation, and risk assessment. It is an important part of the human-experience layer in a DT. In real projects, different roles take part. These roles include inspection engineers, monitoring analysts, structural experts, and managers. Each role has different information. This information includes field observations, sensor data, mechanism knowledge, and management needs. Team awareness comes from combining these different inputs. The team uses communication, documentation, and collaboration. By doing this, the team builds a consistent understanding of structural conditions and a shared view of risks. Team-level awareness reduces bias. It also improves the ability to deal with complex problems. To let a DT use collective intelligence, team discussions, expert opinions, and decision processes must be changed into structured knowledge. This knowledge includes decision reasons, agreement steps, and the basis for risk levels. After this change, the DT includes both individual views and team judgments. This gives a more complete understanding of conditions and more reliable diagnostic results. It also makes later decisions more transparent and easier to trace.

3.2.3. Experience-Based Decision-Making

Experience-based decision-making shows the practical decision logic that engineers build through long-term work. It is a main way for human knowledge to enter a DT. In real O&M work, many important decisions do not depend only on data or model results. They depend on expert understanding of damage patterns, risk change, and construction or maintenance limits. For example, when engineers see similar damage, they can quickly decide which problems need immediate repair, which can be observed for some time, and which are caused by the environment and do not need heavy treatment. These decisions include judging the current condition, predicting future risk, balancing resource limits, and using engineering codes in a flexible way. To let a DT use this experience, these decisions must be changed into clear rules, steps, or semantic forms. These forms help the system understand the logic behind expert decisions. Engineers’ working strategies can be encoded and used together with AI models and mechanism-based models. In this way, the DT can give stronger and more practical intelligent responses in complex or uncertain situations.

3.3. AI Interaction

AI interaction improves the DT. It helps the DT recognize structural states. It helps the DT predict future changes. It helps the DT give clear explanations. This part supports complex and changing O&M situations. It allows continuous monitoring. It supports data-based decisions. Figure 3 shows this module. It has three AI functions. The functions are state recognition, prediction with uncertainty, and explainable reasoning. These functions change multimodal monitoring data into useful results for action.

The DT uses data-driven state identification, prediction with uncertainty, and explainable AI. It can track changes over time. It can find anomalies. It can also assess possible future risks. These functions make up the intelligent layer of the DT. This layer brings these abilities into one system. The system can then provide reliable and clear analysis for engineering decisions. It also supports cooperation between people and models. This idea is shown in Formula (8).

$${\mathrm{X}}_t^{int}\left(t\right)=\mathcal{I}\left({\mathrm{M}}_{data}\left(t\right),{\mathrm{M}}_{pred}\left(t\right),{\mathrm{M}}_{explain}\left(t\right)\right)$$

(8)

In Formula (8), ${\mathrm{X}}_t^{int}\left(t\right)$ is the state of the intelligent layer of the DT at time t. It includes the system’s abilities for state identification, risk prediction, and decision support. ${\mathrm{M}}_{data}\left(t\right)$ is the data-driven model. It tracks structural changes and finds anomalies using real-time monitoring data. ${\mathrm{M}}_{pred}\left(t\right)$ is the predictive model with uncertainty. It helps the system assess future risks by checking possible changes and their uncertainty. ${\mathrm{M}}_{explain}\left(t\right)$ is the explainable AI model. It helps engineers understand, check, and supervise the model’s decisions. This makes the system more transparent and trustworthy. The mapping function $\mathcal{I}$ combines these three abilities into one intelligent layer. This layer supports reliable, clear, and cooperative analysis for engineering decisions. This intelligent layer improves the DT. It gives real-time and data-based results. It supports risk prediction. It keeps the system understandable. These abilities are important for sound and reliable engineering decisions.

3.3.1. Data-Driven Structural Health Assessments

Data-driven structural state recognition allows a DT to monitor and assess the condition of civil infrastructure over time. It uses monitoring data, images, and operation records. With advanced sensors and imaging tools, AI models can find damage features such as cracks, unusual vibrations, and stiffness loss. This supports frequent and large-scale condition checks. It can do more than manual inspection. Data-driven recognition alone is sensitive to data noise, environmental effects, and special features of each structure. These factors reduce reliability and physical consistency. So AI-based recognition should use mechanism-based analysis and engineering knowledge. When this approach is used in a DT, it supports continuous state tracking of the physical entity. It also provides a clear and reliable base for real-time decision support and diagnostic reasoning.

3.3.2. Predictive Modeling and Uncertainty Quantification

Predictive modeling and uncertainty quantification aim to enable a DT to generate predictions and clearly indicate the confidence level of these predictions. Structural performance in infrastructure O&M is influenced by factors such as aging, environmental changes, unexpected loads, and material degradation. AI models can learn from historical monitoring data and damage-evolution patterns to predict future trends, such as stiffness reduction rate, fatigue accumulation, or the possible time window for abnormal behavior. However, all predictive models contain uncertainty. Sources of uncertainty include data noise, model bias, changes in operating conditions, and extreme events. To prevent inaccurate predictions from misleading O&M decisions, uncertainty quantification methods need to express the reliability range of a prediction, such as through probability distributions, confidence intervals, or risk levels. Engineers can then understand the robustness of the prediction and combine it with mechanism-based analysis or expert judgment when necessary. These methods are summarized in Formula (9), which defines how uncertainty quantification integrates with predictive models.

$$\begin{array}{c}{\mathrm{X}}_t^{pred}(t)=\mathcal{P}\left({\mathrm{M}}_{data}\left(t\right),{\mathrm{M}}_{pred}\left(t\right),{\mathrm{M}}_{uncertainty}\left(t\right)\right)\end{array}$$

(9)

In Formula (9), ${\mathrm{X}}_t^{pred}(t)$ represents the predicted state of the DT at time t, indicating the forecasted structural performance based on historical data and predictive models. ${\mathrm{M}}_{data}\left(t\right)$ is the monitoring data model. It gives real-time data such as strain, displacement, and temperature. These data are important for state identification. ${\mathrm{M}}_{pred}\left(t\right)$ is the predictive model. It uses data-driven methods to predict future trends. These trends include material degradation and performance decline. ${\mathrm{M}}_{uncertainty}\left(t\right)$ is the uncertainty model. It considers variability and unpredictability in predictions. It includes data noise, model bias, and changes in operating conditions. The mapping function $\mathcal{P}$ combines these parts into one prediction model. This model includes predicted results and their uncertainty. It gives a strong and reliable base for O&M decisions.

This formulation lets engineers check the confidence level of predictions. It helps them make better and more reliable decisions. It also improves the overall performance of the DT in infrastructure management.

3.3.3. Explainable AI Mechanisms

Explainable AI helps a DT show results. It also shows the reasons for the results. This lets engineers understand why the model gives certain judgments. It also helps them keep control of the system. In infrastructure O&M, AI is used for damage detection, risk assessment, and degradation prediction. If a model gives a result and does not explain the reason, engineers may not trust it during important decisions. So explainable AI solves this problem. It can show feature importance. It can highlight the parts of the data the model uses. It can show reasoning steps. It can also give explanations in simple language. These methods make the model’s behavior clear. They help engineers supervise the system in a more reliable way during O&M work.

3.4. Knowledge Integration

Knowledge integration adds structural mechanics principles, engineering code rules, and past case experience into the DT in a clear and organized way. This part lets the DT carry out reasoning, checking, and judgment based on engineering logic. Mechanics knowledge is written in a clear form. Code rules are changed into computable rules. Case experience is stored in a searchable knowledge base. Because of this, the DT is not only a mix of data and models. The system can understand structural behavior. It can assess risks. It can also give clear decision support in a way similar to an engineer. This leads to more reliable and practical intelligent O&M.

3.4.1. Formalization of Structural Engineering Knowledge

The formalization of structural engineering knowledge changes mechanical laws, analysis methods, and component response rules into forms that a DT can compute and use for reasoning. This transformation allows a DT not only to perform numerical calculations but also to make judgments based on engineering logic. In traditional O&M work, engineers rely on experience to understand structural force patterns, stiffness degradation trends, load-displacement behavior, and damage evolution characteristics. Within a DT, these patterns need to be expressed through formulas, rules, or simplified models. For example, moment and shear transfer paths or stiffness contributions of different structural components can be encoded as inference conditions for model calibration, state assessment, or risk evaluation. The formalization of mechanics knowledge provides physical constraints for data-driven models and improves the rationality of predictions. This knowledge also enhances model robustness when data are incomplete or when operating conditions change.

3.4.2. Computable Representation of Engineering Codes

The computable representation of engineering codes aims to convert text-based requirements, diagrams, and tables in engineering standards into computational logic that a DT can automatically read, evaluate, and execute. In traditional O&M practice, engineers examine each requirement manually when assessing structural safety. When these rules exist only in text form, they cannot support automated reasoning or real-time compliance checks. By converting code contents into logical conditions, parameter limits, formula-based checks, or decision-tree structures, a DT can automatically match relevant clauses during state recognition, performance assessment, and risk diagnosis. For example, when deformation or stress problems are found, the DT can check set limits and give related suggestions. Computable code rules improve work speed. They reduce human mistakes. They also make decisions more consistent and easier to trace. When engineering standards change, the rule base can change too. This keeps the DT in line with current requirements. It also allows the DT to give reliable and automatic compliance support.

3.4.3. Case-Based Knowledge Modeling

Case-based knowledge modeling organizes past damage events, failure cases, maintenance actions, and repair results into one structured knowledge source. This lets the DT learn from past experience. It also helps the DT give suggestions that match real engineering practice. During long service life, infrastructure can show many types of damage. These include crack growth, corrosion increase, fatigue failure, and sudden problems. Many of these events are recorded in engineering documents. These cases can be organized into searchable knowledge records. When similar damage appears, the DT can find related examples and make decisions by comparison. Case knowledge is different from code rules and mechanics models. It shows the variation and uncertainty that exist in real engineering work. It supplements scenarios that analytical models cannot fully capture and provides practical grounding for AI reasoning and human decision-making. As case records increase and interact with real-time monitoring data, the knowledge base can be updated continuously. This process strengthens the self-learning capability of the DT and improves its ability to understand and respond to rare or previously unseen conditions.

4. HITL-DT-Driven Smart Sensing and Diagnosis of Civil Infrastructure Deteriorations

The HAK framework proposed in this section defines an operational methodology for smart sensing and diagnostic reasoning in civil infrastructure operation and maintenance. By linking human cognition and team situational awareness with AI-driven feature extraction and structured engineering knowledge, the proposed HITL-DT workflow enables coordinated, interpretable, and mechanism-grounded deterioration diagnosis. As illustrated in Figure 4, the workflow explicitly specifies how multi-source information is collected, aligned, analyzed, and validated through a closed-loop interaction among perception, human interpretation, and knowledge-guided verification, thereby providing a methodological basis for subsequent diagnostic and decision-support processes.

Figure 4 illustrates the operational workflow of the proposed HAK-driven framework for spatiotemporal monitoring and diagnosis of civil infrastructure deterioration. Building on the HITL-DT architecture, the framework is organized into three tightly coupled stages: multi-source spatiotemporal perception, AI-based change detection, and knowledge-guided diagnostic reasoning. These stages operate in a closed-loop manner, where human cognition, AI inference, and engineering knowledge collaboratively contribute to change interpretation, mechanism-based validation, and decision support. Rather than presenting isolated analytical steps, the workflow emphasizes engineering consistency, interpretability, and operational feasibility, enabling the digital twin to function as a reliable decision-support system for smart CI O&M under complex and uncertain field conditions.

4.1. Spatiotemporal Data Collection

Spatiotemporal data collection is the starting point of an HAK-driven monitoring system. This process integrates human perception gathered from manual inspections, consensus information formed through team collaboration, and high-frequency automated data provided by machine vision. These information sources have different roles. Machine vision helps visual sensing under human guidance and checking. Human perception gives context and experience-based understanding. Team collaboration combines and checks information through group judgment. These three types of data support each other. They form a multi-source base. This base includes detailed observations, contextual understanding, and quantitative features. Figure 5 shows how these different inputs come together into one spatiotemporal data stream for the DT.

Through this combined database, a DT can obtain comprehensive and continuous information about structural conditions within real operational contexts, which provides reliable inputs for later change detection and diagnostic analysis.

4.1.1. Human Cognition

Human cognition has an important role in collecting spatiotemporal information. It captures detailed, contextual, and unclear features. It is the main source of high-quality field information for a DT. During inspections, engineers can find early signs such as small cracks, local deformation, and material weathering through sight and touch. Engineers can also notice sounds, smells, and vibrations. This helps them find possible abnormal structural conditions. Engineers also observe environmental factors such as weather changes, construction activities, and traffic conditions. These observations give important context for later data analysis and change study. In the HAK framework, these inspection results can be changed into structured information. They can be entered into the DT through semantic labels, text notes, voice records, or mobile forms. These human inputs work together with AI-based automatic recognition and case knowledge from the knowledge base. In this way, a multi-source monitoring system is formed.

4.1.2. Team Situation Awareness

Team situation awareness focuses on the shared cognitive benefit that comes from people with different roles working together during spatiotemporal information collection. This component is an important link between human input and knowledge in the HAK framework. In real inspection and monitoring work, field inspectors, monitoring engineers, structural analysts, and managers share information. They discuss it and check it again based on their own expertise. This teamwork builds a shared understanding of structural conditions. It also makes up for the limits of individual observations. It also reduces judgment errors caused by personal bias. In the HAK framework, team conclusions are changed into structured data or semantic labels for the DT. These results join AI recognition outputs and case records from the knowledge base. Together, they form a data foundation based on multi-party agreement.

4.1.3. Machine Vision

Machine vision supports spatiotemporal data collection by extending human sensing through automated visual acquisition while remaining embedded in human interpretation and validation. Using UAVs, fixed cameras, mobile devices, or vehicle-mounted systems, visual information on cracks, spalling, corrosion, and deformation can be continuously captured as images, videos, or point clouds. Within the HAK framework, machine vision does not operate as an autonomous inspection module; instead, sensing tasks and interpretation criteria are guided by expert knowledge and inspection experience. Machine-vision results are checked against semantic records from human inspections. They are also checked against team review decisions. AI analysis can mark possible anomalies using common features stored in the knowledge base. Human judgment is then used to confirm or adjust these results. In this HITL process, machine vision gives frequent and context-related visual data. The data also follow engineering rules. This supports later diagnosis.

4.2. Spatiotemporal Change Detection

Spatiotemporal change detection finds real structural changes at different times or under different operating conditions. It uses monitoring data from many sources. The HAK framework aligns, extracts, and checks human inspection data, team-reviewed information, and machine-vision results. It changes change detection from simple image comparison or signal analysis into a broader judgment process. This process includes semantic information, engineering knowledge, and AI ability. Figure 6 shows this three-step process. It shows how multi-source alignment, AI-based extraction, and knowledge-based checking work together to support change detection.

This integrated process gives stronger and more meaningful evidence for later diagnosis. It makes sure that the detected changes show real structural behavior. They are not caused by noise, environmental effects, or observation bias. The system uses machine intelligence, human expertise, and knowledge rules together. So it produces more reliable and easier-to-understand change detection results.

4.2.1. Multi-Source Spatiotemporal Data Alignment

Multi-source spatiotemporal data alignment puts different types of data into one shared space and time system. The data come from human inspection, team review, and machine vision. This lets the DT understand structural changes in one clear context. Human inspection records have semantic descriptions. Team review gives summarized conclusions. Machine vision gives continuous and quantitative images or point clouds. These data types differ in time resolution, spatial accuracy, and expression style. The HAK framework combines the strengths of these three data sources. It uses precise location data from machine vision. It uses semantic labels from human input. It uses typical change patterns from the case knowledge base. This combination reduces errors that come from one data source. It keeps recognition stable in unclear or noisy environments.

4.2.2. AI-Driven Change Extraction

AI-driven change extraction finds structural differences over time. It uses machine-vision data, monitoring signals, and comparison from different data sources. AI models can find patterns such as crack growth, surface damage, unusual displacement, and stiffness loss. They use image comparison, point cloud matching, feature tracking, and time-series analysis. These methods support large-scale and frequent monitoring. Real detection in practice is affected by lighting, noise, environmental effects, and sensor errors. AI alone can give false alarms. It can also miss real damage. The HAK framework improves reliability. It adds semantic notes from human inspection. It adds team review results. It also uses typical change patterns from the case knowledge base. For example, AI may detect a small crack extension. Human inspection can explain the context. The team can judge if it is important for engineering. Case knowledge can check if the change matches known damage patterns.

4.2.3. Knowledge-Enhanced Change Validation

Knowledge-based change validation checks the changes found by AI or human inspection. It makes sure the changes are real. It makes sure the changes are important from an engineering view. In monitoring work, lighting changes, stains, noise, and camera angle shifts can create false changes. So a structured knowledge system is needed to check these results. The HAK framework uses structural mechanics rules, code limits, and case knowledge. It checks if the detected changes match force paths. It checks if they match common damage types. It checks if they match known cases. This process separates real risk signals from unimportant disturbances. Contextual descriptions from human inspection and shared risk assessments formed by the team also support validation, because these inputs help identify background factors such as construction activities, weather, or incidental events. AI provides preliminary change features, human inspection adds semantic interpretation, and the knowledge base offers mechanism-based justification, which together produce a more accurate and confident judgment.

4.3. Spatiotemporal Change Diagnosis

Spatiotemporal change diagnosis focuses on transforming detected changes into engineering-meaningful judgments. The HAK framework upgrades the diagnostic process from single-signal analysis to multi-source contextual reasoning by integrating AI-based recognition, human situational understanding, and mechanism knowledge, code rules, and historical cases from the knowledge layer. Changes are first explained through human–AI collaboration, then verified using structural mechanics to determine whether they represent real damage, and finally evaluated through codes and case knowledge to infer intervention levels. This process forms a scientific, explainable, and credible diagnostic chain. These methods are summarized in Formula (10), which defines how the diagnostic process integrates multiple data sources and reasoning layers.

$${\mathrm{X}}_t^{diag}(t)=\mathcal{R}\left({\mathrm{M}}_{AI}\left(t\right),{\mathrm{M}}_{human}\left(t\right),{\mathrm{M}}_{mechanism}\left(t\right),{\mathrm{M}}_{knowledge}\left(t\right)\right)$$

(10)

$$\begin{array}{c}{\tilde{X}}_t=\mathcal{I}(M_{AI}(t),\hspace{0.17em}{M}_{human}(t))\end{array}$$

(11)

$$\begin{array}{c}{\widehat{X}}_t=\mathcal{V}({\tilde{X}}_t,\hspace{0.17em}{M}_{mechanism}(t))\end{array}$$

(12)

$$\begin{array}{c}{X}_t^{diag}=\mathcal{E}({\widehat{X}}_t,\hspace{0.17em}{M}_{knowledge}(t))\end{array}$$

(13)

In Formula (10), ${\mathrm{X}}_t^{diag}(t)$ is the diagnostic state at time t. It shows the engineering judgment of detected changes. It uses data from many sources. ${\mathrm{M}}_{AI}\left(t\right)$ is the AI recognition model. It finds possible structural changes and anomalies through data analysis. ${\mathrm{M}}_{human}\left(t\right)$ is the human understanding model. It includes expert judgment, contextual knowledge, and field observations. It gives a more detailed explanation of detected changes. ${\mathrm{M}}_{mechanism}\left(t\right)$ is the mechanism-based model. It uses structural mechanics rules to check if the detected changes show real damage. ${\mathrm{M}}_{knowledge}\left(t\right)$ is the knowledge-based model. It includes engineering codes, past cases, and expert knowledge. It checks how serious the detected changes are and suggests action levels. The mapping function $\mathcal{R}$ combines these parts into one diagnostic framework. This framework supports a clear, explainable, and reliable diagnosis process for O&M decisions.

Formulas (11) to (13) explain this reasoning process in three steps. In Formula (11), ${\tilde{X}}_t$ is an intermediate change state. It comes from cooperation between AI and humans. AI results are understood together with human context. In Formula (12), ${\widehat{X}}_t$ is a checked change state after mechanism-based validation. This step makes sure the interpreted changes follow structural mechanics rules. In Formula (13), $X_t^{diag}$ is the final diagnostic state. It combines the checked change state with knowledge-based models. It judges the severity level and suggests suitable actions.

These steps form a diagnostic chain. The chain moves from interpretation to physical checking to engineering-based decisions. This formulation lets engineers use AI results, human experience, and engineering knowledge together. It forms a clear and trustworthy diagnostic process. It helps make decisions that are scientific and practical.

4.3.1. Human-AI Collaborative Interpretation

Human-AI collaborative interpretation combines human experience, AI detection capability, and knowledge-based rules to analyze the source, nature, and risk of observed changes. AI can provide an initial assessment of a change but often lacks context and engineering interpretation. The HAK framework incorporates semantic descriptions from human inspections and situational consensus from team discussions to support multi-angle interpretation with historical experience. For example, when AI detects slight crack development, human input can use construction history to judge whether the change relates to normal shrinkage, and team discussions can establish a shared view on its risk level. AI performs rapid screening, while humans provide contextual reasoning, and together they form a complementary interpretive mechanism. This process gives the DT a more credible and engineering-relevant preliminary diagnosis.

4.3.2. Mechanism-Based Damage Identification

Mechanism-based damage identification links detected changes with structural mechanics principles to determine whether a change represents real damage and to understand possible causes. In real O&M work, explanations must rely on force-transfer paths, component stiffness, load history, and boundary conditions. The HAK framework uses AI for efficient change extraction, supplements this with human experience and contextual judgment, and uses mechanics formulas, code limits, and typical deterioration patterns stored in the knowledge layer to support scientific identification. For example, when crack extension is detected, mechanics knowledge can determine whether the crack relates to tensile stress concentration or fatigue accumulation, case knowledge can compare the trend with documented deterioration patterns, and human inspection information can help judge whether environmental or construction-related factors contributed to the change.

4.3.3. Code- and Case-Guided Diagnostic Reasoning

Code- and case-guided diagnostic reasoning enables the DT to provide engineering-constrained diagnostic conclusions after identifying change features and confirming damage. The HAK framework uses engineering codes, including limit values, strength requirements, and structural provisions, to enable automatic checks of indicators such as crack width, deflection, and stress level. The case knowledge base provides degradation paths and intervention results from real projects. These records support reasoning about risk levels, future trends, and possible intervention actions. This approach makes sure that diagnostic results follow engineering standards. It also makes sure that they reflect real operational experience. This increases the reliability of the final conclusions.

5. Human-AI-Knowledge (HAK)-Driven Risk Prediction and Proactive Control

This section studies risk assessment, prediction, and proactive control of CI. It shows how the HAK framework brings together human experience, AI analysis, and engineering knowledge into one risk management process. The DT uses information from many sources. It uses clear reasoning. So it can find problems early. It can support future decisions. Figure 7 shows this HAK-based risk management process. It shows how input from humans, AI, and knowledge turns into risk assessment, prediction, and proactive control.

This integrated method makes structural condition assessment more complete. It makes risk prediction more reliable. It makes proactive control easier to use in practice. These abilities change CI maintenance from passive reaction to active prevention. They give stronger support for long-term safety. They give stronger support for efficiency.

5.1. Risk Prediction of CI O&M

Structural risk prediction tries to estimate possible future problems, degradation, or failure based on the current state of the structure and its change over time. In the HAK framework, this process uses AI, human expertise, and engineering knowledge. AI models study defect growth and predict future structural response. But they often lack clear cause explanation. They may also be weak when data are limited. Engineers use mechanism-based judgment. They consider site conditions, structural layout, material behavior, and past experience. This helps them judge if the predicted risks are reasonable. Team collaboration reduces personal bias. It improves the stability of interpretation. The knowledge layer gives physical limits and code limits. Structural mechanics explains how damage grows. It explains how loads change. Engineering codes set safety limits. Case knowledge connects observed changes with known damage patterns. These parts work together. Risk prediction follows data trends. It follows engineering logic. This idea is shown in Formula (14).

$${\mathrm{X}}_t^{diag}(t)=\mathcal{R}\left({\mathrm{M}}_{AI}\left(t\right),{\mathrm{M}}_{human}\left(t\right),{\mathrm{M}}_{mechanism}\left(t\right),{\mathrm{M}}_{knowledge}\left(t\right)\right)$$

(14)

In Formula (14), ${\mathrm{X}}_t^{diag}(t)$ is the predicted risk state of the structure at time t. It shows possible future risk situations based on the current state and its change over time. ${\mathrm{M}}_{AI}\left(t\right)$ is the AI predictive model. It estimates future structural response by learning from past data and damage growth patterns. ${\mathrm{M}}_{human}\left(t\right)$ is the human input. It includes engineering judgment, contextual understanding, and field experience. It helps judge if the prediction is reasonable. ${\mathrm{M}}_{knowledge}\left(t\right)$ is the knowledge-based model. It includes structural mechanics rules, engineering codes, and case knowledge. It gives a base for understanding the risks. ${\mathrm{M}}_{mechanism}\left(t\right)$ is the mechanism-based model. It uses physical laws to make sure the predictions follow structural mechanics rules. The mapping function $\mathcal{R}$ combines these parts into one prediction model. This model brings together data-based results, expert input, and engineering knowledge. It gives a more reliable, clear, and scientifically based risk prediction.

This formulation lets the DT predict possible risks. It also lets the DT explain where the risks come from. This helps engineers make decisions that are reliable and aware of the real situation.

5.2. Proactive Risk Control of CI O&M

Proactive risk control means taking action before small risks become real structural problems. The goal is to keep the structure safe and stable during its service life. In the HAK framework, this process uses AI prediction, human judgment, and engineering knowledge. AI studies monitoring data. It finds possible risk situations. It finds locations that may need early action. These results are based on trends. They need engineering checking. Engineers study the predicted changes. They consider site conditions, structural layout, construction effects, and environmental factors. This helps them separate real risks from normal changes. It helps them judge if the actions are practical. The knowledge layer gives rules and past experience. Engineering codes set safety limits for actions. Case knowledge gives example solutions and past results. The DT combines prediction trends. It combines human judgment. It combines knowledge reasoning. It gives ranked suggestions for intervention. This changes CI management from reacting after problems happen to acting before problems grow.

6. Case Study

This case study presents an application-oriented illustration of the proposed HITL-DT framework in a representative infrastructure operation and maintenance scenario. A typical reinforced concrete bridge deck crack deterioration case is adopted to ensure engineering relevance and generalizability. The case study shows the operational workflow and information interaction among AI-based perception, human-in-the-loop interpretation, and knowledge-guided validation in the digital twin environment. Figure 8 shows this process. It shows the step-by-step diagnostic and decision update process. A qualitative comparison with a purely AI-driven digital twin approach is further provided to highlight the advantages of the proposed framework in terms of explainability and engineering consistency. The case study serves to validate the logical coherence and practical operability of the HITL-DT framework.

Figure 8 illustrates the core operational workflow of the proposed HITL-DT framework in the case study. The process starts from multi-source inspection and monitoring inputs, followed by AI-based perception that generates preliminary anomaly detection and risk signals. These AI outputs are then examined through HITL interpretation and engineering knowledge-guided consistency verification, ensuring that diagnostic results remain explainable and engineering-consistent. Based on the validated outcomes, diagnostic decisions are accepted or rejected and used to update the digital twin state, forming a closed-loop decision and update mechanism for infrastructure O&M.

6.1. Case Study Objective and Scope

The objective of this case study is not to validate the performance of a specific bridge project, but to demonstrate the operational logic of the proposed HITL-DT framework in a representative infrastructure operation and maintenance scenario. A typical bridge crack deterioration scenario is used to illustrate how AI-based perception, human engineering cognition, and engineering knowledge are integrated within a digital twin environment to form a closed-loop collaborative mechanism, supporting interpretable and engineering-consistent diagnosis and decision support. This case study serves as an illustrative and mechanism-consistent validation, focusing on the operability and logical coherence of the proposed framework rather than quantitative performance evaluation or numerical comparison. Validation using real-world infrastructure data is beyond the scope of this illustrative case study and is considered a direction for future research.

6.2. Engineering Scenario and Basic Assumptions

6.2.1. Infrastructure Component and Deterioration Mode

In this case study, crack deterioration of reinforced concrete bridge decks is selected as the representative application scenario. As a critical structural component directly subjected to traffic loading and environmental effects, bridge decks commonly experience cracking during long-term service, making it one of the most frequently observed deterioration modes in bridge operation and maintenance. This deterioration mode is selected for two main reasons. First, bridge deck cracks are associated with well-defined engineering indicators and classification criteria, supported by mature inspection standards and technical guidelines. Second, the mechanisms, evolution characteristics, and maintenance strategies of crack deterioration have been extensively documented in the existing literature and engineering practice, providing a solid knowledge base for knowledge-guided diagnosis. It should be noted that the engineering scenario constructed in this case study represents a typical and generalizable infrastructure O&M situation. The purpose is to demonstrate the applicability of the proposed HITL-DT framework under common deterioration conditions, rather than to analyze or evaluate a specific bridge project.

6.2.2. Spatiotemporal Crack Data and Observation Assumptions

In this case study, it is assumed that spatiotemporal crack evolution information is obtained through periodic inspections, combining manual inspection and image-based perception. The observed data mainly include the temporal variation in crack width and crack length, as well as the spatial location of cracks within the structural component. Synthetic or simulated crack growth data are used to show how the framework works. The crack trends follow common crack deterioration patterns reported in past studies. This case study looks at diagnostic logic, information flow, and cooperation between different agents. It does not study sensor accuracy, data collection systems, or specific detection algorithms.

6.3. Operational Workflow of the HITL-DT Framework

6.3.1. AI-Based Perception and Temporal Change Detection

In the proposed HITL-DT framework, the AI module handles perception and early screening. It uses inspection images or past inspection records. It extracts main crack features. It studies changes over time in crack indicators such as crack width and crack length. It finds possible abnormal growth trends. The outputs of the AI module are only possible anomaly signals or risk signs. They start the next steps of human interpretation and knowledge-based checking. They are not final diagnostic results. They are not maintenance decisions.

6.3.2. Human-in-the-Loop Interpretation and Confirmation

In the HITL stage, manual inspection experience and structural engineering judgment are added to the diagnostic process. They are used to interpret and confirm the possible anomalies found by the AI perception module. In this case study, human judgment is simulated with rule-based expert knowledge. The decision rules come from bridge inspection standards and common engineering limits, such as crack width classification levels. This stage works to remove AI false alarms that are caused by data noise or environmental disturbance. It also gives clear engineering meaning to the detected anomalies. In this way, the engineering credibility and consistency of the later diagnostic results are improved.

6.3.3. Knowledge-Guided Diagnostic Consistency Verification

In the knowledge-guided stage, design codes, common deterioration mechanisms, and past case rules are added to the diagnostic process. They are extra checks for AI results and HITL judgments. The system compares information from different sources. It checks this information against engineering knowledge. It checks whether the detected anomalies match known structural behavior, damage mechanisms, and code requirements. Through this process, the HITL-DT framework establishes a traceable diagnostic reasoning chain, explicitly linking data perception, human interpretation, and knowledge-based verification. This knowledge-guided consistency verification represents a key step that distinguishes the proposed HITL-DT framework from conventional data-driven digital twin systems. To further clarify the operational logic of this knowledge-guided consistency verification process, a compact pseudo-code representation is provided in Algorithm 1.

Algorithm 1 Knowledge-guided diagnostic consistency verification
Input: AI_signals, Human_judgment, Knowledge_base
Output: Diagnostic_state, Decision_action
1	K ← load_versioned (Knowledge_base)
2	S_ai ← extract (AI_signals)
3	S_h ← interpret (Human_judgment)
4	S_f ← fuse (S_ai, S_h)
5	C_k ← match (S_f, K)
6	Conflicts ← detect_conflict (S_f, C_k)
7	if Conflicts == ∅ then
8	State ← validate (S_f)
9	else
10	State ← revise (S_f, Conflicts)
11	end if
12	ReasoningChain ← build_trace (S_f, K)
13	Decision_action ← infer (State, ReasoningChain)

Algorithm 1 provides a compact pseudo-code representation of the knowledge-guided diagnostic consistency verification process within the HITL-DT framework. It formalizes the integration of AI-based anomaly signals, HITL judgments, and engineering knowledge to perform multi-source information fusion and consistency checking. Through conflict detection and traceable reasoning chain construction, the algorithm supports validated diagnostic states and corresponding decision support outputs. This algorithmic representation serves to clarify the operational logic of the proposed framework rather than to specify implementation-level details.

6.3.4. Digital Twin-Driven Decision Support

After integrating the outputs from AI perception, human-in-the-loop interpretation, and knowledge-guided consistency verification, the DT system generates decision support observations for infrastructure operation and maintenance. Typical recommended actions include continued monitoring, local repair, and preventive strengthening at different intervention levels. In this case study, the focus is on the decision-making process is not based on a single information source or isolated judgment, but is derived from the fusion of multi-source perception data, human cognitive inputs, and engineering knowledge constraints. In this process, the digital twin functions as a decision support and information integration platform, rather than a substitute for human engineering judgment.

6.4. Qualitative Comparison and Framework Advantages

To show the features and benefits of the proposed HITL-DT framework, a qualitative comparison is made with a DT system that uses only AI. The comparison shows that AI-only systems can detect anomalies quickly and work efficiently. But their diagnostic process often does not give clear engineering meaning. They are also more likely to produce false alarms in complex situations or when data are uncertain. The HITL-DT framework includes HITL interpretation and knowledge-based checking. Because of this, it shows better reliability in diagnostic explanation, engineering consistency, and false alarm control. This qualitative comparison shows that the HITL-DT framework fits complex and diverse infrastructure O&M situations better. It is especially useful when engineering credibility is important.

6.5. Key Insights from the Case Study

This case study gives several important insights. Human and AI work together in an HITL mechanism. This helps solve the limits of purely data-driven methods. It improves engineering meaning in diagnosis. It also increases the credibility of diagnostic results in real engineering work. The structured use of engineering knowledge adds clear logical limits between data perception and decision support. This stops diagnostic results from going against design codes. It also stops them from going against established engineering experience. The case study also shows that the proposed HITL-DT framework can be used in real infrastructure O&M work. The purpose of this case study is to show that the operational mechanism works. It does not try to achieve the best numerical results. It also does not try to perform better than other systems.

7. Discussion

Unlike many human-in-the-loop DT studies, which mainly focus on manual intervention or decision checking, the proposed framework clearly defines engineering knowledge as a third core part and places it inside a traceable diagnostic reasoning loop. This study shows that combining HAK in an HITL-DT framework gives a more reliable, clear, and scalable method for smart CI O&M. Existing DT and SHM methods usually depend on AI automation or expert manual assessment. Each method has limits. AI models work well for data collection and pattern recognition. But they often lack clear explanation. They can also become unstable when data are limited or noisy. Expert-based assessment depends on deep engineering knowledge. But it is subjective. It depends on individual experience. It can lead to inconsistent results. The proposed HAK framework connects these parts. It combines human perception, AI recognition and prediction, and engineering knowledge reasoning into one decision process. From a conceptual view, this integration changes DT-based O&M. It moves from separate intelligence parts to a coordinated human-AI-knowledge system. This hybrid design improves problems seen in earlier studies. These problems include weak explanation, low robustness under mixed field conditions, and weak use of domain engineering knowledge. These problems are common in purely data-driven AI models.

The observations show the value of multi-source sensing. Human and team situational inputs add more meaning to the data. This improves the reliability of spatiotemporal change detection. The framework uses human judgment and expert knowledge. It detects anomalies. It also gives engineering-based and trustworthy explanations for the changes. Traditional black-box AI models cannot do this. From an engineering view, this rule-based interpretation keeps diagnostic results consistent with structural behavior, design codes, and inspection standards. This ability to give clear explanations makes the decision process more transparent. It increases engineers’ trust in the system. It improves reliability. It also improves proactive control in CI O&M. The system uses human situational awareness. It also uses engineering knowledge. This keeps predictions close to real conditions. This makes the system usable for different infrastructure types and environments. From an engineering view, this rule-based interpretation keeps diagnostic results consistent with structural behavior. It also keeps them consistent with design codes and inspection standards.

It should be stated that this case study gives qualitative and conceptual validation of the framework. It shows operational feasibility, logical consistency, and engineering reasonableness. It does not test numerical performance or optimization. The case study is used to check the internal logic and interaction process of the framework in typical O&M cases. The main contribution is to show how human cognition, AI perception, and engineering knowledge can work together in a closed-loop diagnostic and decision process. Large-scale data testing and comparison with other methods are important future tasks.

Some challenges still exist. One major problem is how to change expert knowledge into computable rules that can be used inside AI models. Engineering knowledge is complex. It depends on context. It is hard to measure clearly. Future research can study automatic knowledge extraction and knowledge update methods. These methods can record expert knowledge and update it over time. Another problem is the high cost of building and keeping large multi-source data systems. Infrastructure systems are becoming more complex. The framework must also work for many types of infrastructure. Different systems have different designs, materials, and environments. From a practical view, these problems show the need for flexible and scalable human-AI cooperation methods. The system should not follow a fixed design for all cases. Future research should study adaptive human-AI cooperation. The system should adjust to different infrastructure needs and conditions. Cross-domain transfer methods should also be studied. The framework should work across different infrastructure types. This will improve scalability and robustness. If these problems are solved, the HITL-DT framework can improve further. It can lead to more scalable, intelligent, and context-aware DT systems for infrastructure.

8. Conclusions

This study establishes a comprehensive HITL DT framework that integrates HAK to address the growing complexity of smart CI O&M. As highlighted in previous top-tier studies, purely data-driven or fully automated DT systems remain limited in interpretability, generalization, and reliability when confronted with large-scale, heterogeneous, and noisy infrastructure data environments. Likewise, traditional human-centered O&M, although rich in contextual cognition and engineering judgment, cannot meet the massive spatial and temporal demands of modern infrastructure networks. The proposed HAK-integrated DT framework bridges this gap by combining the complementary strengths of human cognition, AI-driven analysis, and engineering-knowledge reasoning into a unified modeling and diagnostic system.

At the modeling level, the framework enhances the physical–virtual mapping by formalizing structural mechanics knowledge, embedding computable engineering code rules, and establishing a continuously expandable case knowledge base. At the perception level, it integrates human inspection inputs, team situation awareness, and machine-vision data into a multi-source spatiotemporal sensing system, which strengthens the reliability of change detection. At the diagnostic and prediction levels, human–AI collaborative interpretation, mechanism-based validation, and knowledge-guided reasoning together enable explainable, traceable, and engineering-grounded assessment of structural deterioration. This integrated approach transforms CI O&M from fragmented workflows into a coherent, closed-loop, and evolutionary digital ecosystem.

The HAK framework moves DT research from a simple “physical-virtual” model to a system that includes human input and engineering knowledge. It shows how HITL mechanisms can improve the credibility of DT applications. It also improves robustness and practical use in complex infrastructure environments. Future work can extend this framework to autonomous learning. It can also support knowledge transfer between different infrastructure systems. It can support full-lifecycle risk management. These steps can help the long-term development of smart, resilient, and sustainable CI O&M. The framework is modular in design. Human participation and knowledge parts can be adjusted based on infrastructure size, data conditions, and organizational capacity. This allows the framework to be used in large and diverse CI networks.