Image-Based Structural Health Monitoring: A Systematic Review

Abstract

The early discovery of factors that compromise a civil infrastructure’s structural integrity allows for safety monitoring, timely prevention, and a prompt remedy to the discovered problem. As a result, researchers have been researching various methodologies and types of structural health monitoring (SHM). A systematic search was performed following the updated Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA 2020) in Scopus and ScienceDirect from Elsevier, Google Scholar, MDPI, Springer, Wiley Online and ASCE Library, EOP and IOP Science, IEEE, and other databases with the reliable peer review process. From 1480 identified pieces of literature, one hundred and nine (109) sources met the criteria for inclusion and exclusion and were used to produce our findings. This study presents the identified purpose and application of image-based SHM, which includes: (1) identifying and discovering; (2) measuring and monitoring; (3) automating and increasing efficiency; and (4) promoting development and creating 3D models. Furthermore, the responsibilities and relevance of components and parameters for implementing image-based SHM devices and systems, and their issues, are covered in this paper. Future research can benefit from the stated applications for innovation and the requirements of image-based SHM.

1. Introduction

Civil infrastructures are vulnerable to considerable functional loss because of structural deficiencies produced mostly by material degradation and loadings from earthquakes, wind, automobiles, or ambient vibrations [1]. Vertical structures and other infrastructures are mainly affected and are continuously being gradually damaged, as seen from the evidence of these studies [2,3,4,5]. It is alarming to know that in the status quo, there is a lack of applied implementation and appreciation of these modernized devices and technologies that allows detection and security monitoring for infrastructures that recognize spalling, delamination, and cracks, which will determine the present strength status of a building. This is caused by the fact that existing studies are overly technical and, at times, unavailable to individuals who need them. According to [6], on deep convolutional neural networks, civil infrastructures, such as bridges, dams, and skyscrapers, are more vulnerable to losing their intended functions as they deteriorate over usage.

Structural health monitoring (SHM) has the potential to become a critical instrument in civil engineering [7]. The majority of existing SHM systems include strain sensors and accelerometers [8]. SHM is a data processing approach that employs technology to offer early signals of disruption and the progression of damages to avert potentially hazardous results to a specific structure. As research has concentrated on the usefulness of artificial intelligence in supplementing the inspection process, image data has increased in importance [9]. SHM is a strong and effective method for analyzing the state of existing structures by giving useful information for enhanced decision-making [10]. The digital processing of images requires massive storage for the collection of big data, and requires advanced hardware devices to boot images at a smooth pace. This also provides an objective interpretation of the current standing of a building with the basis of quantitative data and not just by subjective construal of the human eye. Big data is a huge factor in the performance of image-based monitoring on structures, since data must be stored to have a comparative assessment over a period. Nowadays, big data are used in popular social media platforms such as Facebook, Twitter, and Instagram. These applications allow the multiple storage of information for users. In analogy to SHM, the images collected from a specific building will be stored and interpreted by variation in color, the number of line cracks, increasing voids, or imbalance geometry. These may require image segmentation, which converts image representation into something much easier to understand, and the setting of thresholds to the said discrepancies caught on images to induce the right evaluation. Some studies propose innovative ideas in the likes of [11], who idealized the system of unmanned aerial vehicles (UAVs) for real-time SHM. Vision-based algorithms, generally investigated in two main categories, object detection and damage classification, were also given the spotlight in the field of SHM [12]. In object detection and damage classification, filters are particularly important [13]. For example, on roads, the Gabor filter developed by [14] can be applied to extract crack patterns at multiple sizes and directions, comparable to human visual perception. As time and technology advanced, several methods applicable for road monitoring, inspection, and maintenance were established. The CrackTree by [15] is an automated method for identifying road cracks, even in low contrast and shadows. Large pits on asphalt roads were identified by [16]. They initially categorized their captured images either into the defect and non-defect regions using histogram shape-based thresholding. To date, image processing algorithms help quantify crack width, length, and direction [17], remove noise around cracks after region segmentation [18], and use convolutional neural networks for boosting accuracy, which imposes the use of large amounts of data. Despite all these technological evolutions, hardware configuration [12], the influence of noise caused by environmental factors [19], and alternatives to costly sensors, are the main challenges we face today [20,21,22,23]. For visual monitoring, quantitative tables and continuous graphs can be utilized to illustrate daily conditions and simple access to information. Due to computer-based assessment, local governments and structural monitoring agencies will ensure effective and reliable transmittal of information in line with structural status by applying computer-aided vision. This can be done through non-contact observation with less labor cost, transportation cost, and low interference to the daily operational work of the officials. Even international countries like Korea, Japan, and America, are investing to this kind of technology that promotes advancement to engineering projects with the help of image-based monitoring.

Integrating monitoring system concepts in structural design is essential in innovative structural engineering, paving the way to developing intelligent adaptive structural systems [24]. This paper is a systematic review of scientific works involving image-based structural health monitoring that employs the updated Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA 2020) as a technique for reporting and discussing results [25]. A list of abbreviations in this article is provided for clarity.

2. Materials and Methods

Systematic literature reviews (SLR) can reveal gaps in the literature, highlight notions that are widely accepted but lack empirical evidence, and aid in assessing the quality of research being conducted on a certain issue [26]. Hence, all relevant study questions or topic areas in the available literature must be rationally identified, examined, and interpreted [27]. This paper uses the 2020 Preferred Reporting Items for Systematic Reviews (PRISMA 2020) methodology [28].

A research question should be developed as a starting point [29], which must be theoretical, empirical, nomothetic, probabilistic, and causal [30]. Three research questions (RQs) were developed for this study and are provided as follows:

RQ1:

“What are the thematic purposes and implementation of image-based SHM and how to successfully employ them?”

RQ2

“If image-based SHM systems and devices are tailored to fit the requirements set before their implementation, what are its distinct and frontline features?”

RQ3:

“What are the innovations that solve difficulties expected in image-based monitoring?”

Popular databases and criteria for searching studies must be decided. The current methodology considered the exploration and investigation of studies in Scopus and ScienceDirect from Elsevier, Google Scholar, MDPI, Springer, Wiley Online and ASCE Library, IOP Science, IEEE, and other databases with a reliable peer review process. Articles, conference papers, reviews, book chapters, and conference reviews are used as primary sources of information. The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria were then utilized to guide the selection of keywords to be used to query all databases [31]. The keywords that were selected are listed below.

“SHM” OR “Image Monitoring” OR “Structural Monitoring” OR “Structural Health Monitoring” AND “Technology” AND “DEVICE” AND “Implementation” OR “Applications” OR “INNOVATIONS” OR “Computer Vision”

The next step is to filter the identified papers for further investigation by defining the inclusion and exclusion criteria before conducting the review to prevent bias [31]. The following are the inclusion and exclusion criteria developed and specified for this study.

• Period (Inclusion: 2001–2023; Exclusion: Documents before 2001)
• Language must be English (Inclusion: translated version is available; Exclusion: translated version is not available)
• Type of document or literature (Inclusion: Articles, Conference Papers, Reviews, Book Chapters, Conference Reviews; Exclusion: Otherwise)
• Accessibility (Inclusion: Full-text available (In cases where access to the included journal is unavailable, the copy of the manuscript is requested directly from the authors); Exclusion: Otherwise)
• Relevance to RQs (Exclusion: Not relevant to at least two RQs)

The Zotero reference management software was then used to automatically identify and reject items that appeared in more than one database. A total of one-hundred and nine (109) papers were selected for data extraction, assessment, and synthesis. Figure 1 depicts a thorough illustration of the methodology employed in this paper.

3. Results

One thousand four hundred and eighty (1480) literatures were identified as potential information sources. These are bounded from the year 2001 up until 2022 that perform SHM system on buildings, bridges, and roads. The acquired review papers are from Western, European, and Asian countries that apply CV techniques, the digital presentation of data, or deep learning with regards to structures to assess the status and provide assessment for if it requires immediate restoration or repair. After acquiring all possible review papers for the topic, fifty-five (55) published articles are tagged as duplicates. The writers of this study each made the decision to adhere to the protocol outlined by our methodology. From there, we applied our inclusion and exclusion criteria and combined our selected literatures. Upon adding them using their respective DOIs, 55 were tagged as duplicates by Zotero. These duplicated items were then merged. These were removed during screening with the other one thousand two hundred and forty seven (1247) research papers that do not complement the goal of the researchers’ current study, answer the RQs, and violate the inclusion and exclusion criteria. After the initial evaluation of papers, one hundred and seventy eight (178) potential literature reviews remained. In the screening phase, researchers have seen sixteen (16) papers having the same contents in their study. In addition, twenty one (21) research papers were focused on the application of vibration-based SHM to structures to assess strength and durability and not in terms of the image-based monitoring technique. There were also thirty two (32) review papers that are irrelevant to the research study since most of them are mainly related to human science courses. During the screening and inclusion process, the authors encountered issues regarding the full text access on some literatures. Full access to some literatures is unavailable due to limited access to the selected databases. To address this, the authors requested a copy of the manuscript directly from the lead authors, while the remaining articles were accessed using a proxy web server. Hence, the researchers have obtained one hundred and nine (109) literatures relevant to the topic.

As an overview of the succeeding discussions reported in the next sections, it was evident that SHM is a method being researched by previous researchers in the field of construction and engineering to find an objective interpretation of the actual status of infrastructures after calamities, or due to a long period of use. This is constantly changing, especially in the fourth industrial revolution, the digitalization era, with the use of computer vision (CV) to assess structural integrity via tables and graphs. Image-based SHM may focus on surface defect recognition using feature extractors [17] and optimal fiber ultrasonic sensing [32], damage detection [33], target-tracking digital image correlation [34], guided wave-based SHM [35], wavelet [36,37,38], and GPS technology [39]. In other cases, sensors [40,41,42,43,44] and IoT [45,46] applications and interventions are practiced. There are also existing studies that use convolutional neural networks, which is an advanced artificial intelligence to assess graphical data for feasible interpretation. Other research also used programming languages such as MATLAB and C++ Programming, with the application of Arduino to produce a product that will provide a sensor for a specific structural element; any member designed to resist any and a combination of forces and moments, are examples of structural elements. Furthermore, previous research storage systems require innovation in areas where big data is not supported, resulting in information becoming time limited.

4. Discussion

4.1. Purpose and Applications of Image-Based Structural Health Monitoring

A circle diagram in Figure 2 reveals the purposes and applications for image-based SHM. They are organized and discussed into themes: (1) to identify and discover; (2) to measure and monitor; (3) to automate and increase efficiency; and (4) to promote development and create 3D models.

4.1.1. To Identify and Discover

Founded from the study of [6], CV-based methods were developed to go beyond the limits of the human eye to perform assessment and to detect structural impairment in pictures. However, these techniques can only detect certain forms of damage in concrete or steel. This study proposes the application of R-CNN may be used for technological vision tasking or object detection [6]. According to the methodology of their study, multiple types of damage may be detected and localized using the Faster R-CNN method for the quasi-real-time processing of pictures and videos. Another fascinating use of image-based SHM synthesized from included literature is DIC. The finding of [47] on SHM-based damage detection, which is focused on global vibration response, has proven helpful in expressing the existence of the damage as well as occasional coarse-grained location information, but it is severely restricted in reconstructing the 3D geometry of non-external damage. They proposed rebuilding the interior damage using DIC’s full-field response data in a topology optimization framework. The optimization aims to reduce the differences between the actual full-field response obtained experimentally with DIC and the numerically estimated response using the model. Using steel coupon specimens with intentionally manufactured faults, the structure was tested in a variety of simulated and real-world studies. On simulated and real-world tests, the suggested approach could detect the location, size, and shape of the damage, with average F1 scores of 82.8% and 69.6%, respectively. This work has effectively recovered exact internal damage information that would otherwise be prohibitively expensive and difficult to get using innovative technology, and it may thus be used as a viable subsurface damage detection tool.

CV-based structural dynamics approaches have evolved and proven considerable cost, speed, and spatial resolution improvements. A procedure for this application, as illustrated by [48], utilizes motion magnification that allows the magnification of small displacements of large structures [49]. It is vital to highlight that motion magnification is a technique that reveals movements discovered from video records that are not discernible to the human eye, allowing for more effective frequency domain analysis [49].

A region of interest must be defined before video recording and motion magnification [48]. As the magnification is carried out, additional noises are expected that results in the inaccuracy of findings. It can be eliminated and processed in adherence to phase-based video motion processing techniques presented by [50]. The application of motion magnification includes protection of cultural heritages [51], ancient constructions [49], identifying the dynamic range of motion, and modal frequencies for large infrastructures [52], which is a non-invasive methodology [53].

4.1.2. To Measure and Monitor

SHM has the potential to become an important instrument in the field of civil engineering [7]. A proper monitoring approach enables the long-term assessment of structural performance, maintenance planning, and occupant safety [7]. A standard health-monitoring system is comprised of a network of sensors that monitor factors related to the structure and its environment’s condition. Many of these factors may be monitored with standard sensors based on mechanical and/or electrical transducers. Fiber-optic sensors have made a modest but substantial debut in the sensor scene during the last few years. When compared to more established conventional sensors, fiber-optic sensors provide superior performance. Higher measurement quality, increased reliability, the ability to replace manual readings and subjective judgment with automatic evaluations, faster installation and maintenance, and a lower lifetime cost, are all examples of added value. Fiber-optic sensing technology is classified according to the measurement principle [54] and among them is the SOFO interferometric sensors made up of a pair of single-mode fibers that are inserted in the structure being monitored. The measuring fiber is mechanically linked to the existing structure (connected at both ends and prestressed in between), whilst the reference fiber is placed loose in the same pipe, which was illustrated by [54]. All structural deformations will thus result in a change in the length difference between these two fibers [54].

Long-gauge fiber-optic SOFO sensors can monitor temperature and strain changes in concrete columns and steel link-bridge components. Fiberoptic SOFO sensors use low-coherence interferometry to function. As employed by [7], a monitoring strategy allowed the extraction of the local and global behavior of the Pinnacle@Duxton. All seven buildings and four of the 12 bridges have long-gauge fiberoptic strain sensors. During construction, building sensors were embedded in the first-floor concrete columns, and sensing data was accessible from the start of construction until the end of the structure’s operational life. Bridge sensors were mounted on the bridge’s surface shortly after it was built and continue to provide data throughout its operational life. This setup enabled the assessment of the long-term behavior of structures on a local and global scale. SHM has the potential to become a key instrument in civil engineering. A reliable monitoring method enables one to analyze long-term structural behavior, plan maintenance, and assure occupant safety [7]. The fiber-optic SOFO sensor and systems can be integrated to monitor dams [55], subterranean structures [56], and tunnels [57]. A schematic diagram of the SOFO system can be accessed from here [57].

In addition to measuring and monitoring purposes, a pedestrian bridge’s displacement may be tracked using 3D digital picture correlation (DIC) [58]. A pedestrian bridge subjected to static stresses is predicted to deflect, which can be observed using cameras mounted in windows of an adjacent building. CV reduces the subjectivity and error inherent in traditional SHM. SHM can be performed with minimum interruption to the environment and the structure itself thanks to CV. The detection of displacement is critical, especially for constructions that will be used for an extended period. By integrating CV and knowledge of the structure’s behavior under certain conditions, monitoring the displacement of a pedestrian bridge is a realistic undertaking, but it does not limit to bridges only, since the 3D DIC measuring approach may also be used to monitor the behavior of other structures and their parts.

4.1.3. To Automate and Increase Efficiency

According to [59], the vision-based technique is quickly gaining traction as the most effective structural assessment and monitoring tool. Noncontact bend measuring, steel corrosion identification, and spalling detection, are examples of recent developments in vision-based inspection and surveillance. The vision-based strategy, on the other hand, has significant disadvantages. In the actual world, it is difficult to create an algorithm that can protect against all unexpected events. DL has emerged as one of the most promising solutions to this challenge in recent years. Deep learning refers to machine learning approaches that use hidden layers in ANN to increase performance. It has demonstrated exceptional performance, particularly in object identification, natural language processing, advertising, and biology [59]. Deep learning has been used for the fields mentioned above and other engineering problems. For instance, ref. [60] developed a traffic surveillance system that analyses aerial data to follow cars and their movements using deep learning and a SURF-based technique. Although CV-based algorithms have been designed to beat qualified human resources in graphic assessment and to detect structural impairment in photos remotely, most systems detect only specific types of damage, such as concrete or steel fissures. As [6] suggests, the usage of R-CNN, a sort of machine learning modality utilized for technological vision tasking, is primarily object detection. In their methodology, the faster R-CNN approach is employed for the quasi-real-time image and video processing to detect and locate forms of damage. The reasons and functions of image-based SHM also cover the improvement of infrastructure maintenance practices’ efficiency and safety, by assessing the cracks of reinforced concrete structures using stereo cameras [61].

4.1.4. To Promote Development, and Create 3D Models

Safety around dams has long been a major social concern [60]. Dams that are not appropriately monitored or maintained will have a substantial impact on the security of private property and natural settings. Dam health monitoring systems are therefore essential for spotting anomalous behavior and eliminating or lessening its effects. For the non-contact distant measurement of structural reactions, camera and CV-based sensors have shown promise. Due to improvements in camera resolution and processing power, vision-based technologies have become recognized as an effective method for SHM. Crack identification on the pixel level is possible using a number of techniques. CNN-based frameworks provide quick and precise detection of infrastructure cracks. These algorithms are intended to estimate local damage; however, their small-scale features are insufficient for overall dam evaluation. In order to assess damage and undertake non-contact, optical-based measurements for disaster prevention, an image-based rendering approach has been developed for the construction of three-dimensional (3D) models of the structures. A three-dimensional (3D) reconstruction approach can be used for dam crisis monitoring and inspection based on unmanned aerial vehicle (UAV) photos [60]. By fitting the characteristics in numerous overlapping photos, the structure from the motion approach is used to build a high-precision 3D dam model with scene geometry. The bundle correction algorithm then uses the resulting model to generate 3D point positions and camera posture. Various position choices for adjusting to various conditions investigate the impact of ground control points on model accuracy. The suggested emergency monitoring model was assessed on two rock-fill dams: (1) a concrete dam inspection model; and (2) a damage detection technique on a small-scale dam model with loadable panels [60]. According to the findings, the suggested 3D dam reconstruction model based on UAV photographs may reach enough accuracy, resulting in a considerable boost in dam observation and inspection efficacy.

Fitting to the claims of [8], significant efforts have been made over the last few decades toward image-based SHM approaches. Monitoring structure displacement reactions can give quantitative data for structural safety and maintenance evaluations. They went on to say that connecting the pixel to the physical location is essential for deriving structural displacements from acquired video pictures. When the camera’s optical axis is at right angles to the surface of an object, all points on its surface have equivalent depths of field, which means they may all be equally scaled down onto the picture plane with only single matching scaling factor required. Generally speaking, there are two methods that may be used to compute the scaling factor: (1) SF1 based on the known physical dimension of the object surface and its corresponding image dimension in pixels; or (2) SF2 based on the intrinsic parameters of the camera as well as the extrinsic parameters between the camera and the object structure. On its early phase, image-based SHM that uses sensors to measure displacement is cheaper than accelerometers installed in civil infrastructures. For instance, a CV-based method for bridges which is based on the bridge’s influence line was proposed by [62] to estimate a moving vehicle’s tire loads and the displacement response of the bridge. Their novel method utilizes two cameras focused to taking a video of the entire bridge, while the other takes a photo of the vehicles entering. The cameras are then calibrated against lens distortion, and to obtain real-world reference points. From here, a 2D image coordinates are mapped using the pinhole camera model available in the OpenCV library. The full procedure in using influence lines to perform image-based SHM was documented by [62]. This non-contact technique can be applied on monitoring not just the integrity of the bridge. Since the reasons for deflection (changes in influence lines) are dictated by the moving vehicles, a threshold may be set following the design capacity of the bridge and see if vehicles moving it are overloaded. Most of the countries prohibit vehicles from overloading. An additional camera may be setup at the exit point of the bridge to capture more pictures since plate numbers may be missing on the front part of the vehicle.

A newly created, completely contactless SHM system framework, established on regular photographic camera and CV techniques, is launched by [10] for obtaining displacements and ambiances of structures. These are dangerous responses the for performance-based design and assessment of structures from their work; the current vision-based displacement measurement methods are enhanced by eliminating the bodily target attachment to provide contactless and real-world monitoring. This is achieved using imaging key points as virtual targets. Pixel-based displacements of a monitored structural site may be computed using an improved detection and match key-points approach, in which erroneous matches are recognized and dismissed outright. To convert pixel-based displacements to engineering units due to the lack of calibration standards on physical targets, a helpful camera calibration technique was developed here [10]. Furthermore, [10] devised a study method for assessing the accuracy of vision-based deformation measurements, which provides users with the most critical measurement information. The proposed framework, a conventional sensor network, and a data collection system, are employed and proved in a real-life venue during football matches for structural assessment. Utilizing information gathered by sensors like accelerometers and linear variable differential transformers, the outcomes of the unique technique have been successfully tested. The suggested method does not need sensor and target connection, therefore typical activities such as sensor setup, wiring, and the preservation of conventional information and data-collection systems, are not required. For real-world structures in particular, this benefit offers a cheap and practical structural analysis.

4.2. Implementation of Image-Based Devices and Systems

Figure 3 depicts various image-based device implementations for SHM. The discussion of these four options is centered around the complexity and efficiency of the entire configuration of image-based devices and systems for SHM.

4.2.1. 2D Vision System for DIC

A series of digital photographs taken of the surface of a structural component at various loading stages are used in the full-field, non-contact structural sensing technique known as DIC, to reveal deformation patterns between the photos [47]. This setup is a non-destructive method for SHM that utilizes image and video capturing devices.

According to the study of [58], the 2D system, shown in Figure 3a, is placed near a ground floor window of a building adjacent to the region of interest and capturing it from the side of the pedestrian bridge. The camera for the 2D vision system was located 13 m away from the monitored area to capture 140 cm of the bridge’s length. Note that the location of the camera for such a 2D vision system must be established properly, so the line of sight is not obstructed by any means, before proceeding to image acquisition. Additionally, the camera can also be adjusted with different angles, distances, and framerates, as conducted in the study of [10]. A Canon VIXIA HF R42 camcorder can be used to begin measuring the deformations of an object that was perfectly still for a series of predetermined distances. Using a 2D vision system, the result is nearly indistinguishable from using vibration devices to track the displacement of a particular structure.

4.2.2. 3D Vision System for Digital Image Correlation

DIC has gained recognition as a structural monitoring method for evaluating bridge performance [62,63,64], as well as other structures that might eventually undergo permanent displacements, or changes in the displacement behavior under loads [16].

Founded from the study of [58], using cameras positioned near windows on various levels of a neighboring structure, 3D DIC may assess the deflection brought on by a static load on a pedestrian bridge. Two Bosch DINION IP Ultra 8000 MP cameras with a 75 mm lens were used in the 3D system configuration. These cameras were mounted on tripods and placed next to their region of interest; in this case, near the windows on the third and fourth floors of the building next to the pedestrian bridge. Both the computer and the cameras are linked to a single Ethernet switch, which is used to operate the cameras and capture images. The 3D system’s cameras were placed 4 m apart and roughly 30 m from the area of interest, giving rise to a stereo angle of about 3 degrees and a field of vision of about 3.5 m along the length of the bridge. The 3D-DIC process also heavily relies on stereo calibration, which establishes the intrinsic (such as focus length) and extrinsic (such as relative position and orientation) aspects of the camera system. This calibration is presented here [47]. Figure 3b reveals a setup of an internet protocol (IP) camera for a 3D system.

4.2.3. Novel Inspection System Using Inspection Robot

The effectiveness and security of infrastructure maintenance processes may be increased by using stereo cameras to examine fractures in reinforced concrete structures [61]. On the other hand, current damage techniques for reinforced concrete structures are focused on the classification of two-dimensional surfaces without taking into account the actual level of concrete damage.

An inspection robot can acquire detailed crack images and point cloud data of 3D representation of the concrete crack [61]. The inspection robot is equipped with a stereo camera (i.e., ZED 2), developed by STEREOLABS to conduct spatial object detection, a monocular camera for object detection, and a Lidar sensor to determine the gap between the structure and the camera, with an edge computing system, and almost-real-time data transmission system. The wireless remote controller is then attached to the inspection robot to provide movement control. The robot moves using four-wheel drive differential steering, which enables visual mapping, 3D convex hull instantiation, and 3D scene reconstruction for quantification of segmental fractures.

4.2.4. UAV Photogrammetry for Displacement Monitoring

UAV-based photogrammetry is widely employed to create high-resolution topography for researching surface processes because to recent developments in the use and accessibility of unmanned aerial vehicle (UAV) platforms and the development of user-friendly picture processing tools [60]. Utilizing a drone for UAV photogrammetry is quite a complex task as there are very few practical case studies about it for SHM; however, it is a convenient strategy and much more efficient compared to conventional SHM.

Founded from the study of [11], the crucial inspection and maneuvering characteristics, such as the capacity to hover and reach higher heights of the structure, auto take-off, and landing, are taken into consideration while developing a multirotor unmanned aerial vehicle (MUAV) or multirotor drone system. A camera is attached to a drone with a carbon fiber frame and glass epoxy to sustain stable flight under harsh environments (or atmosphere). It can fly for a total of 20 min with a maximum payload of one hundred grams. The drone is then connected to a transmitter to control the movement of the drone, particularly for hovering to capture a stable image from a region of interest. The elevation of the drone concerning the region of interest may vary depending on the researchers. Additionally, the cost of drones shall be considered a key factor in this type of SHM. Two different drones [60] may be used to capture the dam images. The heavy-lift UAV has two batteries, but the low-cost UAV only has one. As a result, the heavy-lift UAV can fly for a longer period and is thus more suited for photogrammetry of huge regions of interest, such as dams.

4.3. Roles and Importance of Components and Parameters

As shown in the previous subchapters (Section 4.1 and Section 4.2), it is understood that image-based SHM systems and devices are tailored to fit the requirements set before their implementation. The succeeding subchapters encapsulate the roles and importance of components and parameters of image-based SHM together with their corresponding data treatment, collection, and storage. The arrangement below highlights its distinct and frontline features.

4.3.1. Digital Image Correlation

A structure is regularly subjected over its lifespan to recurrent loads that might result in damage and imbalances from its original conditions. The application of DIC on pertinent surfaces suggests significant benefits in SHM. It is possible to identify a reduced kinematic foundation (consisting of “modes”) and a statistical amplitude distribution for each mode by practicing with DIC since typical loads are statistically immobile. In the use of any DIC algorithm, the damage must be visible on the external structure with an ostensible crack opening [65]. A danger is not appropriate for SHM reliance if it is just weakly perceptible or underlying since it may not be discovered.

A pedestrian bridge’s movement can be monitored using 3D DIC. A bridge, particularly a pedestrian bridge, is subjected to static loads, hence, it is expected that the pedestrian bridge can have a deflection, and this can be monitored using cameras, which are installed by windows of an adjacent building. The use of CV eliminates the subjectiveness and erroneousness of conventional SHM. With the help of CV, it is possible to conduct SHM with least disruptions to the surroundings and the structure itself [58]. The detection of displacement is crucial, especially for structures that will be subjected to long-term usage. By integrating CV and knowledge of the structure’s behavior under certain conditions, the displacement monitoring of a pedestrian bridge is a feasible task; however, it is not only limited to bridges, as the 3D DIC measurement technique can also be used to monitor the behavior of other types of structures. However, as multiple sensors are installed, this entails costly implementation and maintenance.

The use of DIC is also useful and efficient for displacement monitoring of crossbeams, particularly in an airport runway extension. For instance, ref. [66] focused on monitoring the displacement of the Madeira Airport runway extension, which is composed of a slab supported by frames. The Madeira Airport runway extension is subjected to loads when the landings from aircraft flying in an east-west direction take place, hence, it is expected that displacement may occur. By using DIC, the monitoring of displacement and strain fields of crossbeams is easier and more accurate. DIC measurement techniques are used to assess the permanent displacements of structures, as well as their behavior to dynamic loads. The current monitoring system utilizing the DIC technique is not reliable when it comes to obtaining strain values, still, it is successful in monitoring displacements and their evolution over time.

In summary, the acclaimed features of DIC are to reveal deformation patterns on structures [47], evaluate the deflection caused by a static load [58], assess seismic response structures [34], and assess exterior displacement fields between damaged and not-damaged conditions [56,67,68].

4.3.2. Detection and Sizing of Deep and Multiple Damages

Through the use of a faster R-CNN-based structural visual inspection in a database with 2366 images (with 500,375 pixels) labeled for five types of damages—concrete crack, steel corrosion with two levels (medium and high), bolt corrosion, and steel delamination—it is possible to achieve the near-real-time simultaneous detection of multiple types of damages [69].

Metallic surface defect identification is a critical method for managing industrial product quality. Existing defect datasets, however, are inaccessible for the deployment of the detection algorithm due to the limited data volume and blemish classifications. In a straightforward way, with regard to defect categories, image quantity, and data scale, the new dataset GC10-DET for large-scale metallic surface defect identification faces significant difficulties. Additionally, standard detection methods are inefficient and inaccurate in the complicated real-world context. This calls for the development of a revolutionary end-to-end defect detection network (EDDN) based on the single shot multibox detector. The EDDN model can handle flaws of various scales. Additionally, a strong negative mining approach is designed to address the issue of data equity, and a few data augmentation techniques are advocated to enrich the training data for the problem of exclusive data gathering. Subsequently, extensive experiments demonstrate that combining GC10-DET with EDDN can meet accuracy requirements for metallic defect recognition [70].

The use of guided wave imaging (GWI) in SHM is an effective way to determine the size of the defect and its location inside a particular structure [71]. Machine learning is used for automatic defect sizing and localization, which is useful for non-destructive testing (NDT) methods in SHM. The convolutional neural network (CNN) is a subset of machine learning, which is primarily used for image recognition and processing, and they used it as a basis for their inversion strategy. Only simulations of guided wave signals that are subsequently subjected to the delay and sum method are employed with CNN. Although the delay-and-sum (DAS) GWI method is used to produce pictures linked to piezoelectric sensor data, it is still very difficult to automate guided waves imaging to diagnose various sorts of structures. However, GWI is useful and practical for evaluating the state of a structure, particularly for defect magnitude and localization, which are essential in SHM.

4.3.3. Pilotless Operations

The need for strong, autonomously managed SHM to improve the dependability of such buildings and their surroundings is growing as the demand to construct more skyscrapers and tall bridges in metropolitan areas increase across the world. An autonomous SHM system based on UAV has been investigated by [72]. To provide a comprehensive perspective of the building, the UAV’s photographs of the construction site were pieced together. A well-known speeded-up robust feature (SURF)-based feature detection technique was used to complete the picture stitching process. SURF produces a huge amount of features, which are initially condensed using a random sample consensus approach. The photos are then aligned for final stitching once the appropriate changes have been done. A real-world, UAV-based, image-based SHM application on the structure’s rear column may be utilized to determine displacement by comparing recent and old photographs of the structural location.

Additionally, data collecting is the most challenging and important stage in any SHM system, especially in multiple measurement locations that are often at extremely high altitudes or in entirely inaccessible places, and which are frequently connected with time-consuming, expensive, and, to some extent, dangerous sensor installation and cable wiring. Despite major drawbacks, noncontact vision-based measurement methods have recently been seen as a mostly feasible choice and to address these constraints, ref. [73] recommended and tested an improved noncontact displacement measurement procedure that employed a UAV and CV algorithms in a small steel tower. The proposed system can address several issues with current vision-based techniques for locating a stationary place for the camcorder and reducing the inaccuracy brought on by the long-distance distance seen between the video camera and the measurement location because UAVs can transport cameras into hard-to-reach areas. The UAV, which was controlled by the schematic framework of the system, had a camera connected to it to film the measurement location. Then, displacements at that position were calculated using a key-point vision-based measuring approach. Additionally, the translations of the UAV were extracted from background references. An autonomous strategy based on Canny edge detection and Hough transform may be used to identify the components between the picture and engineering unit for each individual image frame, in order to address the problem of rapidly changing values of measurements [73]. Following the removal of the UAV movements from the displacement data, the real deformations of the measurement location will be measured.

Finally, the concept of integrating unmanned aerial vehicles (UAVs) and image processing systems enables real-time monitoring of various types of [11]. UAVs such as drones are capable of monitoring structures on large scale, which allows full field mapping of large civil structures. The multi rotor UAV (MUAV) system is used for inspection because of its excellent maneuvering capabilities, and is equipped with color imaging sensors for image diagnosis algorithms. Various image-oriented inspection algorithms, such as the Hat transform approach and HSV thresholding, are also used for effective and rapid SHM. Crack detection algorithms rely on grayscale images by extracting dark objects from a light background. Hat transform is a process of extracting small elements and details from given images, and HSV thresholding uses color-based filters like HSV, which yields nearly optimal results in identifying the cracks. With that said, the integration of MUAV and image-oriented inspection algorithms is convenient and reliable for full-field SHM of various types of structures.

4.3.4. Long-Term Applications

Over the course of their lifespan, civil infrastructure systems (CIS) vary in condition. It is difficult to evaluate different real conditions and execute performance evaluations since continuous monitoring is needed to trace the reaction throughout time [74]. Supertall structures have been constructed all over the world in recent decades to suit the economic and social demands of communities, particularly in Asia. The Shanghai Tower, which stands 632 m tall in Shanghai, China, is a supertall structure. Given its supertall height and complicated structural layout, the project requires a thorough detailed analysis to determine its structural performance against dead loads, severe winds, earthquakes, and temperatures [75]. For effective monitoring of the skyscraper throughout both construction and operation, a complete building performance monitoring system with over 400 sensors was constructed. The monitoring system’s primary characteristics are as follows: the direct determination of wind loads on building facades using 27 wind pressure sensors; the measurement of structural inclination and the redefinition of structural sway at various heights using 27 wind pressure sensors; and the simultaneous placement of sensors and data measurement devices with structural construction to document input data. To capture the early structural responses, sensors and data collecting devices were placed along with the structure. All data are transmitted to the service stage following construction. The life-cycle monitoring and assessment of the structure may be carried out from the point of its “birth” thanks to the integration of the in-construction and in-service monitoring systems.

4.4. Issues on Image-Based SHM

Image-based SHM has evolved and progressed over time as researchers around the world embrace its prospective technical advancements and applications. Despite the plethora of research, there are still issues that arise in doing image-based SHM. These problems are presented to provide relevant knowledge for innovative applications.

4.4.1. Limited Range

Monitoring a building has a range limit [76]. In two-dimensional (2D) and three-dimensional (3D) structural displacement monitoring, the device used has a specific range to get a more accurate result. Regardless of the application, whether on infrastructure or element of a structure, captured images and videos must be of the best quality. It is important to acknowledge that image quality can be improved. This can be obtained by considering these factors that influence the picture quality of long-range cameras:

• Hardware components of a camera. Each lens will vary in how much it magnifies the image (video).
• Atmosphere and weather. Even under ideal conditions, the air you are viewing through contains molecules that impair the image by obstructing light transmission. Moisture, rain, snow, dirt, and fog lessen visual contrast, while air turbulence caused by thermal fluctuations throughout the scene causes image distortions.
• Pixels Per Meter (PPM). A number that indicates the number of identifiable pixels over a 1 m width at a certain distance from the camera. PPM is an objective assessment of detail that takes into consideration the camera’s lens, resolution, and sensor size.
• Angle of View (AOV). The focal length of the lens in proportion to the sensor size determines AOV. Longer lenses or smaller sensors generate narrower fields of vision, whereas shorter lenses or bigger sensors produce broader fields of view.
• Sensor resolution. The amount of detail inside a camera’s field of vision is determined by this factor.

4.4.2. Broad-Scale of Natural Frequencies to Be Utilized

In general, SHM, structural engineers, and building managers may lack knowledge in collecting, classifying, and interpreting the inherent frequency of various materials or components. This is caused by the complexity of material behavior, the connection between other materials with the same and varying compositions, and the effects of different loading combinations. Loads may be predicted, but in practice, designing is limited to the conservative measures considered. Tracking the tensions of stay cables, which transmit the primary loads acting on the girder to the pylon, is often the most important aspect of vibration monitoring and dynamic feature identification for bridges [77]. This issue to adequately capture cable vibration measurements is conceivable using digital photogrammetry approaches that establish the geometric transition between the motion trajectory in the object space and that in the picture plane [78]. Now, as records of vibration are fittingly measured in the form of videos, the image processing procedure follows as taken from the works of [77]. First, one must separate colored digital images for different time instants and convert them from RGB to greyscale images. Then, an assigned threshold is taken from the converted binary images. In this stage, a boundary target for each binary image is set [78]. In the final step, the coordinates of a border’s centroid for each binary image are calculated, and the displacement time history of cable vibration is established depending on where the target center was at various points in time. This method may be used in many situations where the materials and structural components have a history of displacement through time. Other tension, compression, and flexural-induced members may also use it; cables are not its only possible application.

4.4.3. Difficulty in Securing High-Quality Devices, Use of Advanced Types of Sensors, Determining Enough Quantities of Sensors for a Strategic Location

It is vital to implement sensors in most critical areas where damage and failure are highly expected to occur and manifest abruptly or through time. Low-cost Internet of Things platforms for SHM are the best solution for this problem. In order to detect an unusual change in the values of the monitored parameters and spikes in the values of the monitored parameters, SHM can be performed by integrating a temperature sensor, geophone sensor, pressure sensor, turbidity sensor, crack width sensor, gyroscope plus accelerometer sensor, strain sensor, temperature, and humidity sensor [79]. Technology integration [80] using the combined application of the interferometric survey of structures (IBIS-S) sensor and system and the high-frequency thermal (FLIR) camera to study the dynamic behavior can serve as references to increase reliability to estimate modal shapes that are closest to laboratory and actual field measurements useful for engineers [80,81,82].

4.4.4. Automation

Local deformations in steel using DIC are only possible in the presence of stress or strain [63]. An image-based monitoring system, for any structure, may be centered on recognizing physical changes that persist when a material is subjected to several types of loads, applied vertically/horizontally/longitudinally, acting on a structure. Numerous cracks form in different sizes and shapes due to the aging of equipment or materials used in constructing a building. However, there are at least a hundred members, if not a thousand, in any civil infrastructure. It is tedious if these deformations, which are often the reason why structural members fail, are traditionally monitored. Automated crack inspection and characterization in bolts on a steel structure are feasible [83]. This creates the opportunity for the creation of a database that records the status of structural elements, not only for steel, but also for other materials such as concrete in buildings, bridges, and roads. Automating the process is highly achievable through: (i) picture capture from several viewpoints; (ii) the recognition of object patches that may contain damage; (iii) the grouping of identical items across images; and (iv) the detection of crack damage triggered by objects.

4.4.5. Other Issues

Poor image quality is prominent in image-based SHM. Capturing pictures of weak quality results in the inaccurate detection of the number of days and total loads that can be applied before delamination [84]. To solve this, image enhancement [84,85], the process of modifying digital pictures to make them more acceptable for display or further image analysis [86,87], must be explored to carry out accurate structural behavior analysis [88], cable force monitoring [89], building inspection and monthly check-up [90], and bridge capacity monitoring [91], which are all areas where image-based SHM and other forms of SHM are useful.

4.5. Applications for Innovation

A machine learning-based SHM model, created to learn on its own, combines technical developments in sensors with regard to data collecting and battery life, high-speed internet, and cloud computing [92]. There are three types of ML-based SHM models: supervised, unsupervised, and improved learning [1]. Using a training set of data, supervised learning may be used to train an ML model. The model is categorized as classification if the algorithm’s outputs are autonomous or categorical data, and there is no way to train the dataset for unsupervised learning.

The 2D image-based crack monitoring approach was discovered to be suitable, affordable, traceable, and mindful of the building’s usage [93]. When used to correct indoor flaws in historically significant structures with minimal social projection, 2D image-based technology is precise and affordable. To provide a framework for storing and displaying information about the built environment, ref. [94] studied the potential of combining image-based documentation and augmented reality. Efficiency is therefore guaranteed both on and off-site. To determine whether the building has sensors and to locate them, a tablet or perhaps other portable electronic devices may be used. Additionally, a user will be able to interact with the AR comments via on-click circumstances to display pertinent facts and/or information. By utilizing the advantages of augmented reality, a user may easily examine and update information about a structure while on-site.

Image-based SHM employs a method for damage detection and diagnosis that includes data from a system collected in various structural states. Image-based SHM is a modern technology that has improved over the past decades; however, there is always room for improvement. As a result, delays in processing, transferring, and archiving data must be shortened. In addition, accuracy must be tested to perceive human-like judgment and acceptance. The Internet is a major concern for most of the systems mentioned above. An uninterrupted connection must be achieved by future researchers and if not prevented, one must create a built-in function that stores and process the data internally for a given time. Only when the connection is booted-up, up will the system send again all necessary information to its users, or data center. The succeeding segments adhere to the additional innovative application for image-based SHM.

○

Digital Camera Detection SHM

The DCM SHM is an affordable image-based SHM. This can be used like a normal digital camera. This uses 2D crack monitoring and big data processing for storage and higher resolution. It can detect the structural health of a building by taking a picture of the infrastructure. It has a wide lens camera for bigger buildings and with a push of a button, you can see the structural condition of the infrastructure.

○

Eyeglass Augmented Reality SHM

The EAR SHM uses an eyeglass with a microchip that contains big data for scanning processing thus, it can also take a photo for longer inspection. Using the EAR, we can see the real-time damage or condition of infrastructure. This has Bluetooth for transferring files and Wi-Fi for easy transfer.

○

Flashlight to Image SHM

The flashlight to image SHM is like a flashlight, wherein the device can scan part-by-part of the structure; this has Wi-Fi for easy transfer to a laptop. The laptop has a machine learning system that can easily read the scanned parts and turn them into an image.

○

Numerical simulations

With the aid of cutting-edge algorithms, inverse dynamics theories, and actual infrastructure observations, new advancements in sensor and information technology enable the interpretation of structural state. The main approach in conducting structural evaluation and local material sampling frameworks has been through performance evaluation methods [95]. However, not everyone can perform multiple tests and experiments due to limited access to equipment, and resources. When it comes to health monitoring, numerical investigation, and data fusion are often utilized in the creation and assessment. These approaches may include the finite element method (FEM) or discrete element method (DEM). Further, [96] implemented DEM by creating a linear and linear parallel bond model, which was used to describe the mechanical constitutive behavior of asphalt mixture through binary image processing, followed by particle formation, clump establishment, and the removal of voids. As the model achieves an accuracy of more than 85%, the difference between the modeled results and actual testing may be caused by the contact stiffness of mortar and aggregates. If structures are subject to any type of loading, they act accordingly. As a result, when evaluating the transformation of force and displacement between particles based on Newton’s second law of motion, access to limitless simulations and analyses is achievable using DEM [97]. When compared to real test results, this technique may reveal viable models with little or no inaccuracy. Understanding different relationships and underlying links amongst structures and their elements when loaded or subject to extreme excitations can be just a click away using high computing devices. Varying but suitable dynamic loadings [98] to realize performance and application of data fusion [99] to investigate harmony and dissonance of different components, several parameters, multiple purposes of the application, and various mechanical and physical properties and behaviors can be the highlight of future works in civil infrastructures using image-based SHM.

○

Big Data Analysis in Image-based SHM

Big data may be classified according to its variety, volume, velocity, and complexity [100]. Structured data has a well-defined format and is simple to store and query. On the other hand, data mining is needed for the proper organization and analysis of semi-structured and unstructured data [101]. SHM has mostly concentrated on directly instrumenting a structure utilizing sensors that provide data useful for the evaluation. There are several common sensor options, such as accelerometers, strain gauges, inclinometers, thermometers, and wind pressure sensors, which produce semi-structured data for vital monitoring applications of data matrix entries and sensor type, sensitivities, locations, and orientations [100]. Now, big data is defined by its volume. In wired and wireless applications of SHM, infrastructures are aimed to be monitored during their lifetime and by monitoring, the data collected files up. In the case of full-field imaging, large datasets are generated. For instance, common camera frame rates vary from 25 to 60 frames each second (fps), with high-speed cameras capable of reaching 240 fps [102]. In dynamic applications, this amount rises to 500 frames per second for full-field data [103]. Given that each shot is 2 MB in size, monitoring even a single structure may quickly generate terabytes of data every week [104]. One of the most challenging challenges in big data is managing and keeping up with the high velocity of data output in terms of data transmission, storage, and processing. The SHM goal is not achieved until such high-resolution datasets are processed and timely structural health decisions are made, despite continual developments in sensing technology that enable the collection of structural data with high resolution in time and space. This is a significant barrier to making timely judgments in buildings with real-time SHM in the wake of catastrophic events like natural and man-made disasters. [100]. Furthermore, another problem for SHM is the complexity of the datasets and the changing interactions between observations. Long-term health monitoring causes data source uncertainty and dataset unpredictability, which makes administering SHM applications complicated.

HD cameras [69,86] and unmanned aerial vehicles [55,65] are the most common tools to collect pictures and videos from concrete columns and slabs and other steel structures. The development of crack width relates to the evolution of temperature and humidity because they indicate stiffness loss. Using 2D image-based crack monitoring, preventative conservation of masonry structures at a cheap cost and in a non-invasive manner is achieved [93]. It is suggested that compared to other methods based on digital data collection and 3D photogrammetry is a good direction for innovation. In addition, once the database is set up, automation of crack detection as the basis for decision-making may be explored. The 2D image-based crack monitoring achieves the best resolution using the maximum telephoto lens. Meanwhile, the process of 3D displacement calculation using a motion capture system has three stages. First, structural coordinates must be defined using a wand practiced by [76]. Then, a motion capture system must be set up to be followed by calibration. From here, the structural coordinate system is transformed into x, y, and z directions for monitoring. MCS entails installing cameras, calibrating them, and attaching markers to a building [76]. Three-dimensional (3D) structural displacements are determined using a motion capture system (MCS) with high-level precision and sampling frequency. Since multiple cameras are used, capable of recording low and high-resolution images, big data is expected for the comparison of displacements which is the focus of monitoring. If a single building, of three stories, generates tons of information from recorded images and videos [76], a server to store and collect these is required. Among the potential big data management control is Hadoop- a widely used open-source software built for distributed data storage and processing [105,106]. It is also feasible to decrease the amount of data before processing or gathering it. Compressive sensing [107], a modern technology brought to the research community, can minimize the volume of data transferred, stored, and processed during sampling. Recently, this methodology was applied to SHM datasets. Crack detection and segmentation, which is common from the set of literature included in this paper, employs a region convolutional neural network [108] for object detection and extends the faster RCNN framework’s target detection capabilities by including an extra branch to achieve instance segmentation for each output proposal box via a fully connected (FC) layer [61]. The model is housed on a web server and is used to process picture data gathered and uploaded by the inspection robot [61]. The best feature for unmanned aerial vehicle developments is to replace human inspectors in conducting life-threatening onsite civil structure inspections. Since most areas may be difficult to access, a wireless connection shall be practiced. Improving the extent of wireless control is suggested. Completely contactless using cameras and computer vision [10] offers the most crucial information of measurement in SHM. Traditional data-driven SHM algorithms are less suitable for large structures and may not be ideal for vision-based SHM applications [1]. In order to address this issue, deep learning-based SHM techniques attempt to create completely automated extraction of features and hierarchical representation processes from the raw input data using stacked blocks of CNN layers with nonlinear mappings [6,102]. Deep learning is one of the many new machines learning algorithms. Adopted from the works of [1,103], Figure 4 shows a concept map for machine learning applications on big data analysis in image-based SHM applicable for the following concerns: (1) crack monitoring and inspection; (2) displacement measurement; (3) concrete damage detection and quantification; (4) condition assessment [109,110]; (5) automation; (6) contactless application; (7) defect recognition and sizing; (8) cost-effectiveness; (9) image quality enhancement; and (10) material degradation.

4.6. Guidelines and Prerequisites

One can inquire: “How should Image-based SHM be implemented and what are the prerequisites for successfully doing so?” Listed and expounded below are the criteria, prerequisites, and suggestions for future studies from all research included in this study using PRISMA 2020.

4.6.1. Creation of Threshold

Deep learning-based structural system condition assessment methods, which mostly use CNN, have significantly increased in the SHM community [155]. Convolutional neural networks are trained on a wide range of datasets for damage and anomaly assessment as well as post-disaster monitoring. Improved non-contact capability sensors are utilized in the development of self-driving SHM systems by using trained networks to review updated data to identify the kind and extent of the damage. The capabilities of non-contact sensors are then enhanced in the creation of autonomous SHM systems by analyzing more recent data using the trained networks to identify the kind and extent of the damage. In recent years, researchers have developed a broad variety of CNN designs to control the intensity of lighting and weather, picture quality, the amount of background and foreground noise, and multiclass damage in structures.

4.6.2. Mandatory Video-Image Sensor

Taken from the study of [74], over the course of their lives, civil infrastructure systems (CIS) vary in condition. The examination of various objective situations and performance is a challenge since it need some sort of monitoring to follow the reaction over time. So, in the context of SHM, the combined use of video pictures and sensor data supports the attainment of civil structure safety through efficient monitoring. To create a unit influence line (UIL), which serves as an indication for tracking bridge performance under certain loading conditions, evaluation of the synchronized image and sensor data is required. A UCF 4-span bridge is used to demonstrate the integration and operation of imaging devices and traditional sensing technologies with UIL for assessing and verifying the bridge [74]. It is shown that the normalized response of the bridge may be calculated by using video images and CV techniques to detect input and output interactions.

4.6.3. Eradication of Multiple Sensors and Expanding of Storage of Data

As established in Section 4.1, image-based SHM, like any type of SHM, is developed and applied: (1) to identify and discover; (2) to measure and monitor; (3) to automate and increase efficiency; and (4) to promote development and create 3D models. Data fusion methods bring together numerous sensor responses with information from a connected data bank to draw more assumptions about potential danger than a single sensor would and increase accuracy and reliability [156]. Any civil infrastructure may be monitored in different and many areas. In line with the study of [157], there is a need to provide multiple structural decline detection at local points. These include concrete cracks, pavement cracks, steel cracks, spalling, delamination, crack propagation, rust detection, loose bolt detection, and damage construction via 3D reconstruction. Multiple sensors for human loading and vehicular loading detection were used. In both CVs-SHM for local level CV-SHM-LL and CV-SHM for global level CV-SHM-GL, the object in the physical world is projected into an image through a camera and lens. By evaluating the changes in the images such as their motion or an irregular phenomenon, those events in the real world can be estimated. Damages, on the other hand, are measured by their influence on certain structural characteristics such as material type and connection. The best selection of one or more applicable SHM evaluation techniques and sensors involves an understanding of the structural attributes most substantially altered by the damages of interest. Future research must focus on presenting the prospective use of fewer sensors that is multi-purpose [158], highly sensitive [159], and strategically placed on critical points, usually connections, of the structures resulting in more economical conduct of SHM. Accordingly, a faster region-based convolutional neural network can deliver human-like inspection of structures [69]. This is accomplished by developing a database with 2366 pictures (500,375 pixels) identified for five categories of damages: concrete fracture; steel rust with two levels (middle or upper); bolt rusting; and metal delamination. Collectively, images require more data storage [69]. If piled up, big data is expected and so is data analysis and exploration entails. From here, we can start to consider the following questions when performing a classification task when dealing with big image files:

• How are the features generated?
• What is the best number of features to use?
• Having adopted the appropriate features, for the specific task, how does one design the classifier?
• How can one assess the performance of the designed classifier?
• What is the classification error rate? This is the task of the system evaluation stage.

4.6.4. Application of Digital Image Correlation on Moving Loads

A work that creates a new autonomous SHM technique based on DIC for structures subjected to statistically static loading during their life cycle was delivered by [65]. During the learning phase, a kinematic modal foundation is established using a collection of images obtained under various (natural) loadings on the intact structure. The observed displacement fields are mapped onto the original modal basis during the following step, which also involves the acquisition of images. Although the connection of two photographs obtained over time can successfully locate concrete fractures, it is important to remember that concrete degradation is caused by a variety of load combinations, not just fixed and recurring loads. Future study must take this into account and examine the variance in the learning stage when the original undamaged structure is taken, as well as the comprehensive relationship among all stated loadings in the structure.

4.6.5. New Algorithms for Grayscale and Colored Images

A thorough examination of image processing applications in the disciplines of material deterioration and steel corrosion is necessary [160]. Corrosion forms and, in certain situations, the goal of the monitoring is used to categorize various corrosion detection and measuring techniques. Since the conclusion must be provided in qualitative form for easier and quicker understanding, the key issue is to produce quantitative data based on the photos that were gathered. In addition, recent developments in algorithm techniques on image collection and deep learning can improve the removal of dust, decrease quality degradation of photos and 3D images, collect faster scanning acoustic microscope SAM images, process X-ray microtomography that gives out specifics on surface roughness, determine a range of rust, and analyze the deepness of corrosion pits. Deep learning has become extremely popular in scientific computing, and businesses that deal with complex problems routinely use its approaches. To execute certain tasks, some of the deep learning algorithms which utilize different kinds of neural networks include:

Convolutional Neural Networks (CNNs)

CNN employs a different approach, simulating how we view our surroundings with our eyes. Here, the principle of how we see, assemble, and process and interpret images, is converted into three phases. The first stage is to define a convolutional layer. The convolutional layer is employed for image feature extraction as it deals with spatial redundancy through weight sharing. At this stage, filters are defined to specify the size of the partial pictures to be examined, as well as a step length that determines how many pixels must be calculated between computations. The average or maximum value of results, depending on the application, is determined during the next stage which is called the pooling layer. A pooling layer has been defined by [161] as a set of hyperparameters as $f_{pool }\left(K,S\right)$ where $K$ means Kernel Size- refers to the height $k_{h }$ and width $k_{w }$ of the pooling window, and $S$ means Stride- is passed as a tuple ($S_{h }$, $S_{w }$) that signifies the number of pixels by which the pooling should slide along the height and width, respectively. It works as in the case of convolutions. Lastly in the step of CNN, a fully connected layer, is produced where individual sub-images are joined again to detect the connections and conduct the classification. Hyperparameters of the convolution operation and a sample are both illustrated in the works of [161].

Long Short-Term Memory Networks (LSTMs)

LSTMs are recurrent neural network (RNN) types that can learn and remember long-term dependencies. The default habit is to recall prior knowledge over extended periods.

Information is retained by LSTMs over time. They are valuable in time-series prediction because they can recall earlier inputs. Four interconnected layers that clearly communicate with one another make up the chain-like structure of LSTMs. Along with time-series predictions, LSTMs are frequently used in speech recognition, music production, and pharmaceutical research.

Recurrent Neural Networks (RNNs)

RNNs have directed cycles that enable the current phase to receive the LSTM outputs as inputs. The LSTM result becomes an entry to the initial stage and may recall previous inputs because of its internal memory. In image captioning, time series analysis, natural language processing, handwriting identification, and machine translation, RNNs are often used.

Generative Adversarial Networks (GANs)

Deep learning algorithms called GANs produce new data instances that closely resemble the training data. GAN consists of two components: a discriminator that learns from the false data it encounters, and a generator that learns to make false data. The discriminator develops the ability to distinguish between genuine sample data and false data produced by the generator. The generator produces fake data during the first training, and the discriminator quickly picks it up. The findings are delivered to the generator and discriminator via the GAN, which updates the model.

Radial Basis Function Networks (RBFNs)

The family of feedforward neural networks (RBFNs) known as radial basis functions is employed as an activation function. They have an input layer, a hidden layer, and an output layer, and are frequently used for classification, regression, and time-series prediction. Figure 5 shows how RFBNs work.

5. Conclusions

Image-based SHM is a technological breakthrough designed to overcome the current uncertainties in civil engineering and construction. It is a multifaceted and complicated system. Prior studies were compiled to give a more thorough knowledge, appreciation, and application of Image-based SHM technologies.

In this paper, the following was discovered:

• Devices and systems are not commercially available because there is no single absolute method for Image-based SHM. One must design their own that is tailored to their specific needs. These needs include, but are not limited to: the monitoring of displacement of structures and their elements; the automation of crack detection; the identification of medium and high levels of corrosion; the discovery of subsurface damage; the promotion of development; the measurement of strain and temperature changes; increasing efficiency; the creation of 2D and 3D models; the development of complete contactless approaches and methods through machine learning and autonomous systems; and the development of cost-competitive alternatives.
• Image acquisition is carried out using: IP cameras connected to a computer on an ethernet switch; industrial and stereo cameras equipped with a light source for better image quality; drones with cameras controlled by a transmitter; and remote-controlled robots with stereo and monocular cameras.
• In image-based SHM, the collection, treatment, and storage of data vary. DIC, a computer vision tool, is well-known for: its ability to disclose deformation patterns on structures; analyzing deflection produced by static loads; measuring the seismic response of structures; and comparing external displacement fields between damaged and non-damaged circumstances.
• Deep and multiple damages can be detected using faster R-CNN, end-to-end defect detection network, guided wave imaging, and convolutional neural network.
• Unmanned aerial vehicles can be used to perform in-construction and in-service condition monitoring, allowing for life-cycle monitoring and the assessment of the structure.
• We presented the typical challenges in image-based SHM that future researchers and implementers may face. The solutions for these issues and the concept map for machine learning applications on big data analysis in image-based SHM were highlighted in Section 4.4 and Section 4.5 of this study.
• In Section 4.6 of this work, the answer to the question: “How should image-based SHM be implemented, and what are the requirements for success?” is given.