A Critical Review for Trustworthy and Explainable Structural Health Monitoring and Risk Prognosis of Bridges with Human-In-The-Loop

Abstract

Trustworthy and explainable structural health monitoring (SHM) of bridges is crucial for ensuring the safe maintenance and operation of deficient structures. Unfortunately, existing SHM methods pose various challenges that interweave cognitive, technical, and decision-making processes. Recent development of emerging sensing devices and technologies enables intelligent acquisition and processing of massive spatiotemporal data. However, such processes always involve human-in-the-loop (HITL), which introduces redundancies and errors that lead to unreliable SHM and service safety diagnosis of bridges. Comprehending human-cyber (HC) reliability issues during SHM processes is necessary for ensuring the reliable SHM of bridges. This study aims at synthesizing studies related to HC reliability for supporting the trustworthy and explainable SHM of bridges. The authors use a bridge inspection case to lead a synthesis of studies that examined techniques relevant to the identified HC reliability issues. This synthesis revealed challenges that impede the industry from monitoring, predicting, and controlling HC reliability in bridges. In conclusion, a research road map was provided for addressing the identified challenges.

1. Introduction

According to the report card for America’s infrastructure, the nation’s aging civil infrastructures (CIs) (e.g., bridges) are deteriorating at an alarming rate, with overwhelming CI system failures warning of the coming national crisis. For example, 42 percent of 617,000 bridges within the United States were built more than 50 years ago, and 7.5 percent of all bridges are in poor condition. Such aging CIs pose unique challenges to the interwoven human-in-the-loop (HITL) technical processes in civil infrastructure operation and maintenance (CIS O&M). For example, bridge engineers are required to conduct routine inspections to examine bridge conditions and ensure safe bridge operations. In addition, various bridge users, stakeholders, and decision-makers could significantly influence operational risks and performance of bridges. Rigorous inspection, maintenance planning, and field considerations of bridge engineers determine the priority of maintenance options of CIs [1,2]. Engineers working on structural health monitoring (SHM) of bridges should consider coupled infrastructure failures and quantify the risks in order to prioritize maintenance and repair actions of CIs. Delayed reactions to vulnerable parts of CIs, incorrect judgment about the relative importance of maintenance options, miscommunications, and improper cooperation can cause cascading failures within connected infrastructure elements and systems.

Traditional SHM of bridges is mainly conducted through visual inspection by bridge engineers. Unfortunately, condition ratings for the same bridge structure provided by different bridge engineers vary a lot due to different engineering backgrounds and field experiences. The development and application of sensing and information technologies (e.g., GNSS, LiDAR, etc.) bring opportunities to achieve reliable real-time situation awareness and predictive data-driven systems control that could support the trustworthy and explainable SHM of bridges. Still, human involvement when performing cognitive and decision-making activities during massive spatiotemporal data processing and model simulation processes creates barriers for the reliable SHM of bridges [3]. Figure 1 illustrates human-cyber interactive processes and reliability issues during SHM processes of bridges. The human aspect involves the interactions between human, data, and digital models for monitoring, predicting, and analyzing SHM and maintenance options based on the conditions of bridges and network-level maintenance performance. This aspect refers to the (1) human reliability that quantifies the risks of the cognition and decision-making processes of workers during field inspection of bridges, and (2) data analysis reliability that captures how human decisions about the selection and settings of data analysis methods influence the accuracy, precision, and reliability of the information derived from raw data. The cyber aspect involves information technologies that collect, process, store, and transmit data and digital models for generating Digital Twins (DTs) of the physical infrastructure to support engineers in monitoring health conditions of bridges and ensure safe bridge operations. This aspect involves three reliability concerns: (1) data reliability, that quantifies the data quality issues (e.g., uncertainties caused by low-resolution images and missing data points); (2) computation reliability issues, where computational processes applied to data and digital models introduce uncertainties in the information derived from raw data and digital models; (3) data storage, exchange, and transmission reliability issues, where information losses occur due to improper data storing, format conversion, and transmission processes.

The reliability theory provides theoretical tools for comprehending the safety and efficiency of bridge inspection as a dynamic human-cyber system. Existing reliability studies scattered around multiple disciplines and domains have the potential of collectively addressing the vision shown in Figure 1. For example, due to the advantages of the application of computer vision techniques, such as non-contact, long distance, fast, low cost and labor, and low interference with the daily operation of the structure, the authors propose the use of computer vision techniques for local and global structural health monitoring. Existing studies of computer-vision-based SHM focus on the implementation and integration of two-dimensional computer vision techniques to solve SHM challenges and the conversion of three-dimensional problems into two-dimensional problems using implemented projective geometry methods [4]. The rapid development of wireless technology has led to a significant development in the integration of SHM systems with wireless sensor network technology. SHM systems based on wireless sensor networks introduce a novel technology that has the benefit of reducing the installation and maintenance compared to traditional wired systems [5]. The advances in sensing technologies and data acquisition platforms have led to a new era of big data, where a large amount of heterogeneous data is collected by a variety of sensors. The increasing accessibility and diversity of data resources provide new opportunities for SHM, while it remains challenging to aggregate information obtained from multiple sensors to make robust decisions [6]. In addition, machine learning (ML) has powerful computational and image processing capabilities in dealing with different aspects of reinforced concrete bridges. Once the ML model is trained, the efficiency of prediction is significantly improved. It surpasses the speed of traditional methods for structural damage identification and strength prediction, achieving almost real-time performance [7]. Numerical simulations of the dynamic response of structures subjected to different types of excitations are performed to assess the variability of the spectrum-driven approach in terms of the type and location of the excitation source [8].

SHM studies have examined methods for monitoring the physical infrastructure systems and assessing systems reliability [9,10]. Psychologists have explored cognitive reliability issues in various contexts [11,12]. Studies in the human systems engineering (HSE) domain have examined reliability issues related to manual data analysis [13,14,15], interpersonal communications [16,17], and the vulnerabilities of collaborative data-driven decision-making processes [18,19]. Some engineering studies examined the quality issues of data and models used in various engineering applications [20,21,22]. Additional examples that examine the reliability of cyberinfrastructure in engineered systems control are the reliability assessment of computing workflows [23,24], data storage and compression methods [25,26], and data exchange and transmission mechanisms [27,28].

Human-cyber (HC) reliability refers to the uncertainties that could arise during data analysis processes while analyzing large amounts of data related to civil infrastructure operation and decision processes. Such reliability demands more accurate and effective human-cyber interactions to ensure the safety and efficiency of the CIS O&M [29]. However, the lack of a systematic review of a wide range of HC reliability issues impedes SHM of bridges and prevents researchers and professionals from effectively using existing theoretical and practical tools in resolving their difficulties related to HC reliability problems. The overall goal of this literature review is to synthesize various literature scattered around multiple domains to form a coherent framework that can guide SHM researchers in identifying similar HC reliability issues in relevant domains. Such identification of similar problems will lead these researchers to find useful theoretical solutions and practical tools developed in those domains. The following sections will first present an SHM case to demonstrate the various HC reliability issues shown in Figure 1 using a bridge inspection case (Section 2). The following two sections will then synthesize existing studies related to the two aspects of HC reliability issues shown in Figure 1 (Section 3 and Section 4). These two sections comment on existing studies on each aspect of HC reliability issues and discuss the knowledge gaps and challenges of transferring those studies to the SHM domain (Section 5). Finally, the last section summarizes a research road map to approach these challenges and knowledge gaps based on the specific challenges and knowledge gaps identified (Section 6).

2. Motivating Case: Human-Cyber Reliability Issues in Structural Health Monitoring and Risk Prognosis of Bridges

Routine visual inspections are necessary for (1) discovering bridge defects, (2) providing bridge condition ratings, and (3) establishing guidance for necessary bridge maintenance and repairs. Trustworthy and explainable bridge inspection is crucial for extending the service life of bridges and ensure operational safety within its service life. Such visual inspections require engineers to conduct visual inspections at field, update the finite element model based on the identified bridge defects and spatiotemporal changes, and predict deterioration patterns based on the updated finite element model. Unfortunately, various HC reliability issues and challenges exist that create barriers for achieving the reliable SHM of bridges.

Figure 2 shows various human-cyber reliability issues during the SHM of bridges with HITL. For example, even the same inspector could have variations in their performance of identifying bridge defects comprehensively for producing reliable condition ratings due to the inherent diversity and complexity of cognitive interaction between bridge inspectors and field environments. Such variations usually cause conflict or the omission of bridge defect information from being documented in historical bridge inspection records, which could bring significant challenges for reliable bridge condition prediction. In addition, updating bridge digital models based on information from multiple sources (e.g., images, textural data, contact sensory data) requires (1) effective data processing workflows for processing data in many formats, (2) information fusion methods for fusing data from multiple data sources, and (3) model updating algorithms for updating the digital model to reflect the true bridge condition. Unfortunately, reliability issues are inevitable while establishing data processing workflows, selecting algorithms, and setting up parameters. All such reliability issues could generate biases and errors that prevent bridge engineers from having a comprehensive understanding of the true bridge condition.

2.1. Human Reliability Issues

The human reliability aspect refers to the reliability issues during the interactions between a human, the data, and digital models for monitoring, analyzing, and predicting bridge conditions and making recommendations for reliable bridge maintenance and repairs.

Traditional visual inspection requires bridge engineers to visually look for bridge defects during field inspections to diagnose bridge health condition. Performance issues still exist when identifying critical bridge defects and predicting the underlying reasons, even with well-designed qualitative inspection standards and procedures. For example, experienced engineers tend to perform more consistently and reliably than less experienced ones while looking for critical bridge defects and assigning condition ratings for bridge elements. The “subjective” nature of bridge condition assessment conducted by bridge engineers could hardly achieve reliable bridge condition assessment due to (1) the uniqueness of different structure types, (2) dynamic site conditions, (3) the availability of data and data quality, and (4) the knowledge level and years of inspection experience of different bridge engineers. Unfortunately, limited studies have examined the impacts of the human factors of bridge engineers on the reliability of SHM results. Even experienced engineers have the potential to fail to identify all critical defects during field inspections, which could cause biases in bridge condition ratings. Another challenge is the lack of sufficient datasets that document how the SHM of bridges is conducted by engineers with different background and experiences. All such studies and datasets are vital for comprehending the processes of field inspections, data analysis, and condition assessments conducted by experienced bridge engineers.

This case aims to illustrate the HC reliability issues that could occur when bridge engineers are conducting field inspections, data analysis, and condition assessments. For capturing all such HC reliability issues, previous studies (as shown in Figure 3) have established a game environment for simulating the process of bridge inspection with augmented finite element analysis (FEA). In this inspection game, the established finite element model (FEM) provides FEA simulation data under different loading conditions. In addition, inspection reports contain observed bridge defect information (e.g., cracks, locations, etc.) to help the inspectors examine the bridge condition. During the inspection game, multiple participants, including both experienced and inexperienced bridge engineers, are hired to examine the bridge condition through FEA and searching information from the inspection reports. All cognitive behaviors during bridge inspection processes are logged and analyzed. Using all such “virtual bridge inspection” log data can then help tracking the behavioral pattern differences between experienced and inexperienced bridge engineers when conducting bridge inspections. In addition, log mining of such inspection behavioral data could help improve the teachability of a reliable bridge inspection process. In addition, the captured inspection behavioral patterns can be used as guidance for improving the bridge inspection process. For example, Liu and Xiong examined the performance differences of the SHM of a bridge between an inexperienced bridge engineer (on the right) and an experienced bridge engineer (on the left); the experienced engineer had a much more organized pattern when searching for bridge defects when inspecting a large span continuous rigid frame bridge (e.g., more attention was been paid to the bottom slabs, especially to the mid-span of every major span) [30,31].

HC reliability issues still exist in remote sensing techniques and influence the data quality for bridge inspection and management. Less experienced bridge engineers could choose locations for imaging sensors where critical parts of the bridge are invisible to the sensor, use the wrong imaging parameters that lead to lengthy data collection, or use sensors that lack the required resolution for capturing critical bridge features. Engineers processing the imagery data could use the wrong data processing algorithms that compromise the accuracy of the information derived from images. A systematic review of HC reliability issues related to using imaging technologies in bridge inspection and maintenance can guide engineers to identify and monitor such issues. Moreover, such a review can help bridge engineers identify relevant theoretical and technical methods for reducing the impacts of many HC issues on the reliability of bridge management decision making based on remote sensory data.

2.2. Cyber Reliability Issues

Cyber reliability issues refer to the reliability of the processes that use various data and information sources to form information models, such as digital as-built models of bridge structures. In bridge inspection, numerical models (i.e., finite element models) of bridges are usually established using design drawings, and remote sensory images are useful for engineers to examine bridge deteriorations under different loading conditions [32]. Using deformations, defects, and other spatiotemporal data collected from bridge inspection reports and structural health monitoring systems could help estimate and update the parameters of finite element models based on as-built bridge conditions. Bridge engineers can then use the updated digital model to run simulations of bridges under various scenarios and discover bridge deterioration patterns and failure modes.

The algorithm plays an important role in the process of model updating. Various algorithms form data processing workflows for producing FEM models augmented by environmental conditions to support bridge maintenance planning. These algorithms could introduce various errors into the derived information and influence the accuracy, timeliness, comprehensiveness, and detail-level of bridge information models. Data storage and exchange processes could introduce additional technical problems that influence the quality of bridge inspection models. Comprehending how various parameters in data processing workflows can influence information models’ quality is challenging due to the exponentially large search space of data processing workflows. The next paragraph uses a data processing workflow for supporting the updating of an FEM model based on field data.

The FEM updating process uses observed geometric, visual, and sensory information to update the parameter of a model until the FEM simulation results finely agree with the physical conditions of the bridge (Figure 4) [33,34]. These parameters in the digital models of bridges need updates to reflect physical conditions and predict deterioration trends. Comprehending how all these data processing algorithms and their parameters influence the quality of the FEM is critical for efficient diagnosis of bridge conditions based on digital models and field data. The challenge to identifying a reliable data processing workflow for producing a reliable FEM is that the combination explosion of hundreds of data processing parameters makes the search space of data processing workflows exponentially large [35,36].

3. Human Reliability Analysis and Team Cognition for Trustworthy and Explainable Structural Health Monitoring and Risk Prognosis of Bridges

This section synthesizes studies that examine human reliability issues during the SHM of bridges. The authors found 322 related studies in the literature ranging from 2010 to 2022 on Web of Science search engines (search criteria—Topic: “human reliability” and “operation”) (see Figure 5). Among all these studies,

Table 1. Top five cited papers related to human reliability.

Title	Objectives
Modeling and Evaluating the Resilience of Critical Electrical Power Infrastructure to Extreme Weather Events [37]	This study established a framework for comprehending the impact of human responses on power systems resilience during severe weather events.
A fuzzy causal relational mapping and rough set-based model for context-specific human error rate estimation [38]	This study established a fuzzy rule-based causal relational mapping approach for deriving human error rates under different contexts.
Prediction of human error probabilities in a critical marine engineering operation on-board chemical tanker ship: The case of ship bunkering [39]	This study presents a Shipboard Operation Human Reliability Analysis (SOHRA) method for predicting human errors during bunkering operations.
A modified human reliability analysis for cargo operation in single point mooring (SPM) off-shore units [40]	This study established a framework for a human error assessment and reduction technique (HEART) with human uncertainties in decision-making.
A methodological extension to human reliability analysis for cargo tank cleaning operation on board chemical tanker ships [41]	This study developed a method for augmenting human reliability analysis in examining human reliability impacts on cargo tank cleaning operations.

lists five highly cited papers that have examined human reliability issues in various aspects.

Human reliability issues need two levels of studies: individual level and team level. Both levels have three types of reliability issues: (1) perception reliability, which is related to the data collection and sensing performance of human individuals and teams when engaging with the physical environments; (2) cognition reliability, which is related to the decision-making performances of human individuals and teams based on the available information of the workspace and people’s domain knowledge, expertise, and prior working experiences; and (3) response reliability, which is related to the specific actions of and cooperation between human individuals while working on the assigned activities. This section synthesizes studies that examine multiple interactions between human, data, and digital models for monitoring, predicting, and analyzing SHM options based on the health conditions of bridges. Two aspects include the individual-level human reliability (Section 3.1) and the team-level human reliability (Section 3.2).

3.1. Human Reliability (Individual Level)

Human reliability research examines the perception, cognition, and response reliabilities of human individuals at the individual level.

Table 2. Human reliability (individual level) studies.

Perception Reliability—Reliability of the sensed spatiotemporal information about the self and environmental objects	Visual perception [42]; Auditory sense [43]; Taste sense [44]; Sense of smell [45]; Tactile and somatosensory [46]
Cognition Reliability—Impact of the self-sensed physical conditions of human bodies and environmental conditions on the decisions of human individuals and teams	Visual information [47,48]; Auditory information [49]; Taste [50]; Smell [51]; Body motions [52]; Temperature [53]; Space size (confined space) [54]; Motion speeds [55]; Frequencies of changes [56]; Interruptions/Distractions [57,58]
Response Reliability—Impact of the individual’s capability and team’s situational awareness on the risks and efficiency of collaborative operations of a team	Reaction time [59,60]; Time limits [61,62]; Physical demand [63,64]; The impact of the environmental conditions (performance shaping factors—PSFs) on the operational performance of individual workers [65]

lists human factor studies exploring how various cognitive factors, decision contexts, and individual capability influence SHM reliability in carrying out assigned activities.

Overall, existing human factors studies conducted extensive exploration of the reliability of individual human workers. The major challenge is the lack of reliable brain signal interpretation methods for recovering the meaningful mental processes of workers in reaction to the human individual’s physical conditions and environmental conditions [66]. Most studies examined human individuals’ observed decisions and decision outcomes in different physical human body conditions and environments. As conducting experiments with human subjects in real working environments is extremely time-consuming, many studies use simulators [67,68,69] and Virtual Reality (VR) games [70,71] to collect human behavioral data in controlled environments. Such simulator experiments or VR games allow human individuals to repeat similar processes in controlled virtual or simulator environments. Limited studies were on reconstructing the contents of thinking processes based on brain signals. Brain-Computer Interface (BCI) studies aim at recovering meaningful thoughts based on brain signals. However, these BCIs could only recover simple control commands for human individuals focusing on simple tasks in controlled environments [72].

Most human reliability analysis (HRA) studies originated from other industries, and they are not specifically used in the construction industry. For tackling the above challenges, the authors find suitable HRA methods by drawing on research from other fields. Some scholars proposed a dynamic HRA approach-IDAC (cognition, decisions and executions in crew context) model. The model was developed to probabilistically predict the response of nuclear power plant (NPP) control room operators facing system anomalies for dynamic probabilistic risk assessment (DPRA). IDAC considers operator cognitive response during the process of mitigating consequences or/and bringing the system to a safe state during an accident [73]. Figure 6 shows a keywords network related to HRA.

There are also some experts who believe that the main causes of human errors are different in different operation scenarios and therefore divide the power system operating processes into three categories. Time-centered human reliability is quantified by using the proportional hazard model (PHM), process-centered human reliability is analyzed by the modified Cognitive Reliability and Error Analysis Method (CREAM), and human error probability (HEP) is quantified by the human cognitive reliability method for the emergency-centered scenario, respectively [74]. In these studies, scholars have found shortcomings in the current CREAM model, which needs to consider and examine the interdependencies among the common performance conditions simultaneously. Therefore, a hybrid HRA model combined with CREAM has been proposed to overcome these drawbacks [75].

At the individual level, the human reliability analysis (HRA) framework created for risk monitoring of nuclear power plant operations [76] covers all three aspects of human reliability assessment: (a) cognition reliability, (b) decision reliability, and (c) execution reliability. Certain HRA methods (e.g., Standardized Plant Analysis Risk Human Reliability Analysis, SPAR-H) [65,77] could help to classify field incidents or accidents into diagnosis (perception, decision) failures and action (execution) failures. SPAR-H formulates a framework about how various factors influence individual workers’ performance. All such “performance shaping factors” (i.e., PSFs) can be human physical conditions, features of the task or operational process design, or environmental conditions. The author found that these PSFs are of three categories, as shown in Figure 7 [78]. Unfortunately, SPAR-H mostly focuses on the performance of individual workers. Limited studies focused on extending the human performance assessment framework of SPAR-H to teamwork performance assessment. Additionally, SPAR-H considered human performance’s cognition aspect but did not fully consider the changing contexts of task executions [79].

Some studies examined one or more of the three aspects of individual-level human reliability in various engineering systems. Along the perception reliability direction, some researchers investigated the visual perception reliability of field workers to design better data visualization and wearable information systems for supporting filed condition diagnoses [80,81]. The Cognitive Reliability and Error Analysis Method (CREAM) has been a theoretical framework for integrating human individuals’ perception behaviors into predicting human errors [82]. Some augmented reality (AR) devices have the potential to integrate these visualization methods, designed with consideration of the visual perception performance of human individuals [81,83]. Most of these studies examine advanced data visualization software or hardware design while claiming such systems should allow human to better perceive data patterns [84,85]. The assumption is that better perceptions of visual information could help workers make appropriate decisions with improved awareness of the information underlying those data patterns. Unfortunately, few studies conducted extensive and quantitative user tests for quantifying the improvement of human perception and decision performance with the augmentation of visualized information delivered through mobile devices or immersive VR-AR environments.

Perception reliability issues of human individuals have other dimensions, such as temperature and taste senses. However, the authors found that relatively fewer studies comprehend humans’ sensing mechanisms and relevant reliability issues [86]. Some studies designed automated audio processing methods for augmenting human perception so that workers could better hear task-relevant information or commands in complex and changing environments with noisy backgrounds [87,88]. Few studies were on human individuals’ sensing reliability of smelling and tasting [89,90]. Overall, most existing studies with engineering applications were on vision and auditory senses. Integrated characterization of sensorial performances (e.g., smells of different odor, tastes of different flavors, reactions to temperature changes, etc.) in civil and infrastructure engineering workers is still in the infancy.

On the topic of cognition reliability, previous methods had limited discussion about human individuals’ analysis and decision mechanisms based on the perceived information and situation awareness. The challenge is how to measure human individuals’ mental processes to reveal their cognitive and decision processes [91]. Some recent development of EEG hardware and brain interface equipment provides some potential for monitoring human thinking processes in controlled environments. Brain interface and EEG instruments can help examine how various perceived information influences human individuals’ analytical capabilities [92]. Some researchers examined how 2D and 3D visualization information influences firefighters’ navigation decisions and field operations during emergency responses [93,94]. In the context of the SHM of bridges, this technology could help researchers analyze how various external factors influence inspectors’ judgment and decision-making processes, thereby improving the reliability of inspection results.

Reliability issues of human individuals are the focus of HRA methods. Most HRA methods predict human errors based on task analysis without considering the dynamic nature of structure engineers’ contexts and field workflows. Examples of such static HRA methods include the SPAR-H HRA method [65]. Some HRA methods consider contexts and dependency by considering the quantitative impacts of related events on human error probabilities (HEP) in various contexts. Unfortunately, dependency modeling produces an overall HEP for given tasks with known performance shaping factors (e.g., workload, training, experiences, and task complexity). It does not systematically model the variations of PSFs during complex field processes. PSFs can change when human individuals work in changing environments that might have events that interrupt their cognitive processes. In most cases, a person’s capability and physical conditions can deteriorate and cause significant PSF changes along with the SHM processes. None of the static HRA methods could reliably handle PSF variations for achieving reliable predictions of HEP with full consideration of SHM and maintenance events and environmental changes in the field [95].

Some researchers started to capture PSF evolutions along various working processes in different environments through computational simulations and simulator experiments. Simultaneously, some fundamental challenges remain to integrate the simulation and simulator data into traditional HRA methods. The main challenge is that conventional HRA methods are not using detailed step-by-step analysis. Instead, conventional HRA methods focus on general overall task analysis. Therefore, the detailed simulation models could produce data incompatible with conventional HRA approaches [95,96,97]. In many cases, the HEPs calculated based on detailed process models can overestimate the actual human error rates. The integration of conventional HRA methods with detailed computer simulations based on step-by-step human error assessment and computational simulations requires further studies [95].

3.2. Human Reliability (Team Level)

At the team level, reliability research studies aim to examine: (1) What attributes of communication and data exchange processes could help reliably quantify uncertainties in the shared mental model critical for team situation awareness? (2) What properties of changing workspaces, real-time states of workflows, and co-workers determine the team’s performance and SHM and maintenance risks? These questions are not about the analysis and communication reliability concerns mentioned above; instead, they focus on quantifying how communication, site condition and decision optimization and prioritization analysis, and information quality issues influence the efficiency and effectiveness of executing teamwork.

Table 3. Human reliability (team level) studies.

Perception Reliability—Reliability of the sensed spatiotemporal information about the self and environmental objects	Visual communication: Gestures [98]; Flag [99]; Signs [100]; Auditory communication [101]; Motion communication [102]; Somatosensory and visual and auditory [103]
Cognition Reliability—Impact of the self-sensed physical conditions of human bodies and environmental conditions on the decisions of human individuals and teams	Motions and positions [104]; Voice [105]; Impact of environmental conditions gained through team communication and collaboration on the team decisions; Relative motions [106,107]; Relative differences between workspaces [108]; Speeds of changes in remote workspaces [108]
Response Reliability—Impact of the individual’s capability and team’s situational awareness on the risks and efficiency of team operations	Team reaction time [109]; Task independence [110,111]; The impact of environmental conditions on team performance [112]

provides an overview of three categories of studies that examine the team level reliability issues in three aspects: perception, cognition, and response.

Table 3 shows that team level studies still need extensive investigations compared with many active individual-level studies of human cognitive behaviors. Team cognition reliability in dynamic environments is an emerging and challenging area [113]. The main reason is that team decision and collaboration processes involve analyzing various teams’ compositions with different team members and combinations of various teamwork parameters that capture team dynamics [114]. Such complexity limits the studies investigating team decision and execution reliability [115].

Figure 8 shows a keywords network related to team cognition. At the team level, some researchers have developed frameworks for characterizing cognition, decision, and cooperation at the team level. Some studies examined the “shared mental model” that defines the reliability of coordinating team collaborations through a shared situational awareness of the team based on shared cognition, focusing on all team members’ static knowledge structures. Such a shared situation awareness integrates multiple team members’ knowledge and memories [116]. One challenge is how to achieve a theoretically rigorous resolution of conflicting facts obtained by different team members [117]. Some scholars proposed a method for quantitively examining the neural synchronization between subjects in the collaborative process through electroencephalogram (EEG) hyperscanning. The authors assumed that the neural synchronization in EEGs changes with different team performances and communication effectiveness [118].

Cooke et al. conducted a series of studies to profile team performance [119]. They mostly examined the communication network dynamics of various collaborating teams, such as UAV controllers, aircraft/fighter pilots, and fight teams of soldiers. These studies found that research on team cognition and teamwork performance should emphasize team members’ interactions [120]. Moreover, the changing and dynamic context of these interactions is critical for comprehending team cognition as a behavior of a group of collaborating people [121]. As a result, measuring interactive team cognition (ITC) should be at the team level with full consideration of various activities (e.g., communication and decision-making) [122]. These studies pointed out that relatively little work has been published along the dimension of dynamic decision reliability at the team level with full consideration of team cognition processes and interactions between team members [120].

Some NASA-funded research projects examined long flight teams of astronauts to understand how to establish technical environments to ensure astronaut teams’ reliable long-term operation [123,124,125]. The current study was conducted to investigate the dynamics of team coordination using an extended version of the NDS method. The authors compared three team conditions and the results showed that the experimental group exhibited better team efficiency. The results indicate that future studies should explore synthetic teams and examine the coordination dynamics within teams [126]. Those studies have revealed many team parameters and factors, such as team members’ diverse training and background knowledge and the dynamic history of team coordination patterns, that need systematic characterization.

4. Cyber Reliability for Trustworthy and Explainable Structural Health Monitoring and Risk Prognosis of Bridges

This section synthesizes studies that examine cyber reliability issues during SHM and risk prognosis of bridges. The authors found 121 related studies in the literature ranging from 2010 to 2022 on Web of Science search engines (search criteria—Topic: “data quality” and “SHM”).

Table 4. Top five cited papers related to cyber reliability.

Title	Objectives
Convolutional neural network-based data anomaly detection method using multiple information for structural health monitoring [127]	This study established an anomaly detection method based on convolutional neural networks that mimic human vision and decision making.
Review of Bridge Structural Health Monitoring Aided by Big Data and Artificial Intelligence: From Condition Assessment to Damage Detection [128]	This study has established a method that uses big data (BD) and artificial intelligence (AI) techniques to solve the data interpretation problem.
Deep learning for data anomaly detection and data compression of a long-span suspension bridge [129]	This study has established a method for data compression and reconstruction based on deep learning.
Decentralized fault detection and isolation in wireless structural health monitoring systems using analytical redundancy [130]	This study has established a decentralized approach for automatic sensor fault detection and isolation for wireless SHM systems.
Reliability analysis and damage detection in high-speed naval craft based on structural health monitoring data [131]	This study has established a method for reliability analysis and damage detection of high-speed ships (HSNC) using SHM data.

lists the five most cited studies that examine cyber reliability issues and risk prognosis during SHM. Figure 9 shows the studies collected from the Web of Science database. This section synthesizes studies that examine multiple technical aspects of computer hardware and software systems that can produce uncertainties and errors while processing and managing the data and information. These aspects include the reliability of the data and information models (Section 4.1), computational processes that transform data and information (Section 4.2), and the reliability of storing and exchanging data of various formats (Section 4.3).

4.1. Data and Model Reliability

Data and model reliability refers to the quality of data and information extracted from the data to form information models, such as digital as-built models of structures and facilities. Previous studies explored metrics for measuring the quality of various data and information sources. Figure 10 shows a keywords network related to data and model reliability.

Table 5. Synthesis of example studies exploring various reliability issues of data and information models.

	Category	Example Studies of Reliability Issues
Data	Visual and Geometric Data	Accuracy and level of detail of 3D imagery data reconstructed from photos [145]; spatial resolutions of images [146]; temporal resolution of videos [147]
	Reports	Errors in field notes [148]; omitted structural defects in inspection reports [149]
	Tabular Data	Missing and anomalous values of locations, structural condition ratings in the NBI database [148]
	Relational Database	Incorrect external keys for representing the related columns in two tables and linking the information from two tables [150]; redundant information items having inconsistent values at different parts of the database [151,152]; missing relationships between two tables while the link should exist for linking common columns in two tables [153,154]
	Sensory Data	Errors or missing values in time series sensory data that measures structure vibrations [155]
	Metadata	Errors in the metadata for specifying the formats and organization of datasets, such as the meaning of columns of numbers in a data file [144]; errors in the metadata for specifying the time and data collection environments [156]; errors in the metadata for specifying the methods of processing and transforming the data, such as a transformation matrix for transforming point clouds to a global coordinate system [157]
Model	2D/3D Maps	Location errors of points [158]; length and direction errors of lines representing paths on 2D or 3D maps [159]; level of detail of maps [160]; missing values in the properties of objects on 2D or 3D maps [161]
	Semantic-Rich Digital Models	Missing and additional objects [162]; dimensional and shape deviations from actual dimensions [163,164]; wrong type information of objects [165]

indicates that data reliability issues have attracted more attention from the domain of civil engineering. Some researchers examined the quality of 3D imagery data in terms of accuracy, level of comprehensiveness, and detail [132,133]. Some researchers define data quality metrics for quantifying the information losses and uncertainties caused by missing data [134,135,136,137,138,139,140,141,142,143]. The data can be tabular data, multimedia data such as images and audios, natural language reports, and sensory time series. Time series data usually capture motions and vibrations of structures, people, vehicles, equipment, or other objects of interest. Some datasets have metadata that specify the meanings and organization of the data contents and data collection contexts. Any errors in those metadata could cause errors in data analysis, interpretation, and data use in practice [144].

These data-quality studies still have not yet addressed some challenges related to quantifying the data and model reliability in engineering application contexts. Compared with systematic quality quantification of structured data and images, one challenge is that relatively limited studies examined metrics for measuring the quality of audio, video, and natural language reports (Taleb et al. 2018 [166]). Another gap is that limited studies focused on characterizing unreliable metadata’s impacts on the data interpretation in practice. Some researchers pointed out that misleading metadata could result in propagative misunderstanding of the raw data and cause failures in data use [144].

Finally, compared with many studies on data reliability, fewer studies were on characterizing the reliability of information models derived from data. Most studies discussed sampling and statistical quality assessment and quality control (QA/QC) methods for checking the information models against reality [167,168]. Unfortunately, those studies have not yet addressed the following questions: (1) how to produce a context-related information quality index that can quantify the confidence level of using specific digital models against given engineering requirements [169]; (2) how to assess the impact of qualitative information errors, such as missing objects or wrong properties of objects, in information models on decision risks [170].

4.2. Computational Reliability

Computation reliability studies examine how computational processes of transforming data and information introduce errors or uncertainties in the information derived from raw data and digital models. Such studies are of two categories: (1) numeric error studies and (2) workflow studies. Numeric error studies examine how digital computers’ discrete nature produces numerical errors in representing numbers and how numeric errors influence data and model transformation results’ reliability. Workflow studies examine the propagation of numeric errors within deterministic or stochastic processes that transform data and information models into useful information. Figure 11 illustrates most used keywords related to computational reliability studies.

Numeric errors are at the algorithm level for examining different numeric stability problems due to cut-off errors or specific computation processes that magnify errors and residuals into biases in the results [171]. Traditional numerical analysis methods examine numeric errors and algorithm stability problems [172]. Many researchers reviewed various numeric analysis studies and how algorithms produce varying results due to numeric errors [172,173,174,175]. The major challenge lies in fundamental theories for capturing random numerical errors with unknown factors underlying hardware and software designs [176,177,178]. For example, Kendall and Gall have explored the feasibility of modeling aleatoric and epistemic uncertainties through Bayesian deep learning models for computer vision tasks [176]. Different types of algorithmic processes have their areas and related topics for comprehending individual algorithms’ reliabilities, such as optimization and inverse calculations of matrices. Some algorithmic processes pose significant challenges for researchers to fully reveal the reliability and numeric errors associated with those algorithmic processes. For example, Franco et al. claimed that some floating-point arithmetic-based algorithms of machine learning, computer vision, and computer graphics heavily rely on numerical libraries for calculations. Significant challenges exist for accurately estimating numerical errors based on the underlying numerical libraries [179]. In addition, algorithms for the camera pose estimation based on three 2D/3D point correspondences always accumulate numerical errors that jeopardize the accuracy [180].

Numeric analysis research examines numerical errors at the micro-level. In contrast, workflow level studies examine how numeric errors generated from each algorithm within a workflow influence the following data processing algorithms. Relatively fewer studies were on the characterization of data processing workflows in terms of error accumulation and transformation [181]. Previous studies focused on individual algorithms’ characterization [182] or modeling uncertainty propagation through a pipeline of algorithms connected by input-output relationships [176]. The challenge is to integrate individual algorithms’ performance models into performance models of complex workflows. Exploring the exponentially large solution spaces of all possible combinations of parameters involved in multiple algorithms in the workflow is challenging.

At the workflow level, the exponentially growing complexity of investigating various combinations of various algorithmic processes makes the tracking and characterization of error propagation and accumulation challenging. The emerging theories of process resilience and stochastic process vulnerability have produced new mathematical models for characterizing workflows. “Anti-patterns” of workflows and systems dynamics studies also produce methods for revealing process patterns that could cause unwanted accumulations of errors and result in workflow failures due to small variations of data inputs or individual data processing parameters [183,184].

In brief, the knowledge gap about computational reliability exists at multiple levels of manipulating data and information. Future studies should focus on those challenging algorithmic processes with highly uncertain numeric error generation behaviors and process patterns that pose difficult-to-quantify behaviors of error escalation and process failures. New process modeling and characterization theories, such as process pattern discovery [185], process resilience, and systems dynamics models [186,187], will bring opportunities to overcome the difficulties associated with the characterization of the reliability of various data processing workflows.

4.3. Data Storage, Exchange, and Transmission Reliability

Data storage, exchange, and transmission reliability research focuses on characterizing data and information losses during storing, converting the formats of, and transmitting data files containing various information. Such characterization investigates how data storage and exchange mechanism design and hardware factors influence the data and information losses and determine upper and lower bounds of such losses within specific hardware and software environments [168,188].

Table 6. A summary of example studies investigating data storage, exchange, and transmission reliabilities.

	Reliability Issues	Example Studies
Data Storage	Data and information losses due to compression of data for saving storage space	Point cloud compression research for reducing point cloud data sizes while keeping the geometric changes captured in the point clouds [189]
	Losses of data and information due to data saving errors and hardware defects	Corrupted files or missing parts of files due to problematic file saving processes for saving data of large sizes or unique data structures, such as Gigabytes of imagery datasets [190]
	Losses of data and information due to decaying hardware devices for storing the data files	Corrupted files or missing parts of files due to storage unit failures under unfavorable environmental conditions or decaying of storage media materials [191,192]
Data Exchange	Losses of data and information while converting files between different formats	Mapping the same objects stored in different formats based on properties of objects while accepting losses of semantics associated with specific properties uniquely stored in only one of the formats [193]
	Losses of data and information while updating the data schema	Mapping the entity definitions in different versions of a schema for automated updating of building information model files into files that use a new version of the schema [194,195]
Data Transmission	Losses of data and information due to problems in communication protocols	Losses of data packets due to problems in data and file transmission protocols, especially for transferring large images and data files [196]

below provides a synthesis of example studies that examine reliability issues related to data storage, exchange, and transmission. Figure 12 shows a keywords network related to data reliability.

Specifically, data storage reliability studies examine how to store large amounts of data of different data types without causing information loss. Some studies quantify geometric information losses due to 3D image compression using different algorithms [189]. Data exchange reliability studies examine semantic losses while converting files into different formats and reconstruct semantic relationships lost during format conversions. Such studies focus on modeling information losses due to the transformation between different data formats [197]. These studies also examined methods that identify similar objects across different formats for enabling translation between data formats [193]. Data transmission reliability studies investigate how to transmit and exchange data across different computing devices or platforms without causing information loss [198,199].

Although the current studies about data storage, exchange, and transmission reliability have significant contributions in their fields, a few gaps still exist. First, there are limited studies on how errors in data storage, exchange, and transmission propagate along the data analysis workflows and cause misleading results due to information losses introduced by interconnected data management and transformation systems [200,201]. Second, non-loss semantic data exchange needs a systematic investigation into semantic equivalence and quantification of semantic information losses due to the lack of consideration of changing contexts of the entity. In practice, many entities have changing meanings in changing contexts, while the lack of context modeling can cause information losses and misinterpretations of data [201].

5. Human-Cyber Reliability for Trustworthy and Explainable Structural Health Monitoring and Risk Prognosis of Bridges

Human-cyber (HC) reliability considers the reliability of data analysis with HITL. The data analysis reliability dimension captures how various factors influence the quality and reliability of information derived from the data to support decision-making. The data analysis usually involves three sequential stages that gradually derive various information from raw data sources (e.g., images, sensory time series, and inspection reports): (1) data pre-processing, (2) data processing and (3) data interpretation. The data pre-processing stage takes the raw datasets as inputs to transform raw datasets into structured or semi-structured data that can serve as inputs for the data processing stage. The data processing stage takes pre-processed datasets as inputs and extracts features or patterns corresponding to certain objects or events captured in the data. The data interpretation stage derives correlations between the features and data patterns extracted by data processing algorithms and derives behavior and process information that represent how objects and events evolve along the timeline so that engineers can obtain meaningful views of objects and events for diagnosing the engineered systems or workspaces.

Table 7. Three stages of data analysis that derive information from raw data sources, and their reliability issues.

Stage	Inputs/Outputs	Reliability Issues
Data Pre-Processing (Prepare the raw data in formats that are suitable for reliable feature extraction and pattern recognition)	○ Inputs: raw data (images, Excel tables, field notes, and inspection reports). ○ Outputs: cleaned and sub-sampled data, linked or combined data (e.g., cleaned and registered laser scanning point clouds).	○ Losses of object or event details due to improper data cleaning actions and sampling rates [189,205]. ○ Errors in linking or combining datasets due to improper selections of corresponding objects or properties for data linking and integration [203,206].
Data Processing (Extract features and data patterns that are corresponding to objects and changes captured in the spatiotemporal patterns of features)	○ Inputs: cleaned, sampled, and linked/combined datasets. ○ Outputs: Objects, object properties, changes of objects (location and material property changes of concrete elements) and changing rates (moving speed).	○ Missing features or feature extraction errors due to improper selection of the parameters of feature extraction algorithms [207,208]. ○ Errors in object/change detection due to mismatches between feature and pattern definitions and features/patterns extracted from the pre-processed data [209].
Data Interpretation (Analyze relationships between objects and events to interpret the correlated objects and events into meaningful change information of the facilities and workspaces)	○ Inputs: cleaned, sampled, and linked/combined datasets. ○ Outputs: Various relationships (e.g., statistical and spatiotemporal relationships) between objects and changes of objects (location and material property changes of concrete elements).	○ Missing or errors in the detection of existing relationships between objects and changes due to improper selection of statistical methods [210]. ○ Improper setting of the parameters that are not suitable for the processed data and decision contexts [211,212]. ○ Improper use of statistical metrics in identifying statistically significant relationships between objects and changes [213].

defines these three stages of data analysis and shows how these three stages have different reliability issues. This table shows that some previous studies examined how various factors involved in data pre-processing, data processing, and data interpretation influence the reliability of the processed results. The authors found that relatively more studies are on the reliability issues related to data pre-processing and data analysis, while limited studies were on the reliability of data interpretation. The reason is that data interpretation work focuses on detecting and assessing various relationships between individual features, objects, and events extracted from the data. Relationship detection relies on the detection of features, objects, and events, which are necessary bases before carrying out extensive relationship modeling and assessment [202,203]. As a result, data scientists invest more efforts into extracting features, objects, and events from data first, and then focus on relationship analysis. On the other hand, computational modeling researchers started modeling various relationships from the knowledge representation point of view without considering the challenges of extracting those relationships from various data [204].

Some recent research studies have started examining approaches for automatic generations of various relationships between objects and events from diverse datasets in the domains of big data analytics [214], geospatial analysis [215], and data-driven simulations [216]. Many semantic relationships, such as those representing safety rules between specific types of objects in workspaces [193,214], contain subtle semantic information in natural language documents or semantic rich digital models (e.g., Building Information Models). Extracting material properties and other semantic information about objects through integrated analysis of textual reports, images, and various sensory data is still challenging [217,218,219,220].

6. Conclusions: A Research Road Map for Advancing Trustworthy and Explainable Structural Health Monitoring and Risk Prognosis of Bridges

This review of HC reliability offers a comprehensive characterization of identified knowledge gaps related to HC reliability issues. This section provides a summary of all identified grand challenges and aims at helping researchers to recognize the various demands of research activities for advancing the reliable structural health monitoring of bridges. The following content shows a summary of the significant challenges and uses a three-part coding system to label these challenges. The codes for each aspect or reliability issue are also provided. Additionally, the “number” is an integer that is used to code the challenges in the category in which the aspect and reliability issues were identified. For example, challenge H.DA.1 is the challenge for the “Human” (code: H) aspect which is associated with the reliability issue related to “Data Analysis Reliability” (code: DA) and is the first challenge in the category (code: 1).

These challenges fall into the following five main areas. Data analysis reliability: H.DA.1—Lack of methods for assessing the reliability of the rules and relationships generated automatically from multiple unstructured data sources. H.DA.2—Lack of methods for reliable integrated analysis of images, audio, and unstructured documents. Operational Reliability: H.O.1—Lack of methods for handling variations of performance shaping factors (PSFs) reliably for achieving accurate predictions of HEP with full consideration of detailed working processes and environmental changes. H.O.2—Limited investigations along the dimension of dynamic team decision reliability with full consideration of team cognition processes and various interactions between team members. Data and Model Reliability: C.DM.1—Lack of methods for characterizing the impacts of errors in natural language reports and metadata on automatic document and data interpretation results and related decisions based on such interpretations. C.DM.2—Lack of methods for generating the context-aware information quality index necessary for quantifying the risks of using certain information models in given decision contexts with specific information requirements and changing environments. Computational Reliability: C.CP.1—Lack of methods for characterizing the data processing workflows composed of multiple data and information transformation algorithms with diverse numeric error generation and propagation behaviors. Data Storage, Exchange, and Transmission Reliability: C.SET.1—Lack of reliable application context and semantic mapping between different file formats to allow reliable data exchange across different software used in different domains. C.SET.2—Lack of methods for assessing both qualitative and quantitative error propagation along data processing workflows composed of algorithms that handle diverse datasets to produce decision information (e.g., deterioration status of a bridge based on reports, images, and time series collected by contact sensors).

In response to the major challenges discussed in this phase, the authors give the corresponding research line, which represents a route that encompasses various types of scientific research efforts (e.g., data collection, data processing, and theoretical modeling) aimed at addressing all the challenges discussed in the previous sections, while understanding dynamic HC reliability issues to ensure bridge safety. The research line is divided into the following three areas: data collection, data processing, and theoretical modeling. Data Collection: (1) Collect large amounts of data (e.g., bridge condition data, human behavior data, computer logs, human network interaction logs) in a controlled laboratory environment or under field conditions. (2) Use the collected data to evaluate the performance of bridges and behavior patterns of human individuals in different decision contexts and environmental conditions. This proposal is intended to address issues H.O.2 and C.DM.2 above. Data Processing: (1) Establish data processing models for representing data processing processes. (2) Develop and test data processing methods for integrated analyses of multiple heterogeneous data sources. (3) Develop scientific workflow systems that connect data processing algorithms into workflows to support the characterization of various workflows. This proposal is intended to address issues H.O.2, H.DA.1, H.DA.2, C.DM.1, C.DM.2, C.CP.1, C.SET.1, and C.SET.2 above. Theoretical Modeling: (1) Create models based on collected data for predicting the deterioration trends of bridges in various environments with various maintenance plans. (2) Create models based on collected data for predicting the dynamic reliability of given teams in changing decision contexts. (3) Create models based on collected data for predicting the impact of both qualitative and numerical data errors on the reliability of the information derived from given data processing workflows. This proposal is intended to address issues H.O.1, H.O.2, H.DA.1, H.DA.2, C.DM.1, C.DM.2, C.CP.1, C.SET.1, and C.SET.2 above. The theoretical modeling aspect of this research line illustrates the imperative and specific research activities that the authors think would resolve the identified grand challenges and bridge the knowledge gaps to ensure the security of civil infrastructures.

This paper synthesizes the literature towards a research vision of “Reliable Structural Health Monitoring and Risk Prognosis of Bridges with Human-In-The-Loop”. The major conclusion is that relevant research studies, knowledge, methods, and data from multiple domains (e.g., human systems engineering, computer science, civil engineering, infrastructure systems, etc.) can contribute to such a vision for ensuring the reliable SHM of bridges. At the same time, certain fundamental limitations still exist for future explorations. This paper presents a review of the relevant literature to promote multi-disciplinary collaborations and discussions of the limitations and challenges identified by the authors in this critical review effort.