Next Article in Journal
Attitude-Orbit Coupled Control of Gravitational Wave Detection Spacecraft with Communication Delays
Previous Article in Journal
Modern Electrochemical Biosensing Based on Nucleic Acids and Carbon Nanomaterials
Previous Article in Special Issue
Assessment of Accuracy in Unmanned Aerial Vehicle (UAV) Pose Estimation with the REAL-Time Kinematic (RTK) Method on the Example of DJI Matrice 300 RTK
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Reliability Assessment of a Vision-Based Dynamic Displacement Measurement System Using an Unmanned Aerial Vehicle

by 1 and 2,*
1
Department of Architectural Engineering, Kyungpook National University, Daegu 41566, Republic of Korea
2
Department of Architectural, Civil, Environmental, and Energy Engineering, Kyungpook National University, Daegu 41566, Republic of Korea
*
Author to whom correspondence should be addressed.
Sensors 2023, 23(6), 3232; https://doi.org/10.3390/s23063232
Received: 23 February 2023 / Revised: 15 March 2023 / Accepted: 15 March 2023 / Published: 17 March 2023
(This article belongs to the Special Issue Advanced UAV-Based Sensor Technologies)

Abstract

:
In recent years, many studies have been conducted on the vision-based displacement measurement system using an unmanned aerial vehicle, which has been used in actual structure measurements. In this study, the dynamic measurement reliability of a vision-based displacement measurement system using an unmanned aerial vehicle was examined by measuring various vibrations with a frequency of 0 to 3 Hz and a displacement of 0 to 100 mm. Furthermore, free vibration was applied to model structures with one and two stories, and the response was measured to examine the accuracy of identifying structural dynamic characteristics. The vibration measurement results demonstrated that the vision-based displacement measurement system using an unmanned aerial vehicle has an average root mean square percentage error of 0.662% compared with the laser distance sensor in all experiments. However, the errors were relatively large in the displacement measurement of 10 mm or less regardless of the frequency. In the structure measurements, all sensors demonstrated the same mode frequency based on the accelerometer, and the damping ratios were extremely similar, except for the laser distance sensor measurement value of the two-story structure. Mode shape estimation was obtained and compared using the modal assurance criterion value compared with the accelerometer, and the values for the vision-based displacement measurement system using an unmanned aerial vehicle were close to 1. According to these results, the vision-based displacement measurement system using an unmanned aerial vehicle demonstrated results similar to those of conventional displacement sensors and can thus replace conventional displacement sensors.

1. Introduction

The development of large structures, including ultra-high-rise structures and super-long spanning bridges, is becoming increasingly common. Thus, the importance of structural health monitoring (SHM) becomes more relevant. In SHM, basic responses (e.g., acceleration, displacement, deformation, and tilt) are used because they can clearly reflect the local and global behaviors of existing structures under various load conditions [1,2]. Among such basic responses, displacement measurement is an important method that directly exhibits high accuracy in low-frequency regions and the deformation geometry of a structure. However, there are limitations in measuring the displacement of an actual structure [3].
Sensors for displacement measurement can be classified into contact and noncontact sensors. Linear variable differential transducers (LVDTs) are representative contact sensors used for measuring longitudinal displacement. When a current flows in the coil inside such a sensor and the core is moved by the occurrence of displacement, an induced current is generated in the auxiliary coil, which is converted into distance. The sensor has a long service life because of slight friction, and it exhibits high accuracy because a corresponding output is generated even for the slightest movement of the core. However, applying such a sensor to an actual structure is difficult because it must be in direct contact with the structure for displacement measurement and should be installed at a fixed point independently from the structure [4].
Alternatively, noncontact sensors include laser distance sensors (LDSs) and the global positioning system (GPS). LDSs measure displacement by irradiating a laser beam at the point to be measured and determining the time required by the beam to reflect and return. Unlike LVDTs, they can easily ensure high data reliability because of performance improvements in sensors. However, applying noncontact sensors to actual structures is difficult because the effective distance is short, and they must be installed independently from structures. Furthermore, the use of the GPS for measuring dynamic displacement is challenging because it is costly, has a small amount of measurement data per unit time, and has a relatively wide error range [5,6,7,8].
Computer vision detection has been utilized in numerous studies for various purposes, including crack detection, photogrammetry, damage detection, and VDMSs [9,10,11,12,13]. By using deep learning techniques, cracks and damages can be accurately detected, and this information can be used as a maintenance technique for structures [9,10]. Moreover, a study has been conducted to estimate the deformation of a structure by generating a 3D point cloud from an image using a photogrammetry technique [11,12]. In particular, Tang et al. applied a mark-free vision measurement system to the seismic performance evaluation of a reclaimed aggregate concrete filled steel pipe column and generated a hysteresis curve [13]. The results demonstrated good agreement with the theoretical curve.
Thus, a vision-based displacement measurement system (VDMS) has been proposed for alternative sensors. A VDMS is easy to install and dismantle because the measurement equipment, which uses a hand-held camcorder, is relatively inexpensive and highly portable [14]. It has fewer restrictions under field conditions compared with conventional sensors because it requires no separate equipment other than the hand-held camcorder during measurement and performs noncontact remote measurements.
Research on vision-based displacement measurement commenced in the mid-1980s, and there have been significant advances in computer vision algorithms. For instance, Peters et al., for the first time, developed a vision software program that can measure displacement and strain in an experiment [15]. However, the initial VDMS could not avoid aliasing in actual civil engineering structures and buildings because cameras then had a small number of frames per second and would not capture small displacements of structures because of their low resolutions [16,17].
Thereafter, owing to the development of cameras and algorithms, displacement measurements at actual buildings and static displacement measurements and small-scale verification tests in laboratories have been performed. For instance, Part et al. developed a three-dimensional (3D) displacement measurement model using a motion capture system. They measured the free vibration of a small three-story structure and compared the results with those obtained using an LDS [18]. Meanwhile, Choi et al. applied an algorithm to adjust the size of the region of interest (ROI) and ROI updating to the VDMS to improve the accuracy of measurements from digital images [19]. Kim et al. performed measurements of laboratory-scale small structures and actual structures using the VDMS and verified the validity of the obtained dynamic characteristics [20]. Additionally, various studies have been conducted to develop vision systems for real-time vision-based displacement measurements and an actual structure [21,22,23]. Lydon et al. developed a displacement measurement system using a commercial action camera equipped with a low-cost telescopic lens and experimentally verified its performance in a laboratory. Based on the obtained results, they measured the displacement response on two actual bridges and verified the effectiveness of the system [24]. Given that VDMS exhibits low measurement accuracy as the measurement distance increases because of the reduction in the image resolution from the target, its application to high-rise buildings and large structures, which require long-distance measurements, is difficult.
Vision-based monitoring technologies encompass a variety of tools, including video recorders, virtual vision sensors, digital image correlation, and 3D laser Doppler vibrometry [25,26]. Scislo proposed an innovative solution to reduce the time required for quality control by utilizing a 3D laser Doppler vibration measurement system. This system allows for the measurement of natural frequencies, damping ratios, and mode shapes of civil structures or building structural elements, making it a valuable tool for modal response analysis. For example, a study conducted by used a laser Doppler vibrometer to compare the modal response of a manufactured product with an ideal template [26].
Recently, various studies have been conducted for monitoring civil engineering facilities to detect static deformation, local cracks, and corrosion using unmanned aerial vehicles (UAVs) [27,28,29,30,31]. In particular, studies have been conducted on VDMS using UAVs. The use of UAVs can increase measurement accuracy because target structures can be accessed and high-resolution images can be collected regardless of the measurement target. UAVs move and collect images using the abnormal camera shake in six degrees of freedom (DOF) in real time, unlike the existing VDMS. However, a correction method is required, and studies have been conducted to calculate the movement of UAVs by tracking fixed points inside images and measuring the final behavior of a structure [32,33]. In this regard, Yoon et al. measured the dynamic displacement of a structure by estimating the movement of a UAV using a camera calibration technique based on the optical flow [34,35,36]. Moreover, Yan et al. and Weng et al. proposed a UAV-based vibration measurement method that combines a convolutional neural network (CNN) and the Kanade–Lucas–Tomasi optical flow methods [37,38]. This method uses the CNN to extract fixed points inside UAV images and track the movement of the target and background, and it does not require fixed points (e.g., markers) outside the structure. Additionally, Wang et al. conducted a shaking table test on a full-scale structure and measured its displacement using a VDMS using a UAV and a light and ranging detection scanner [39]. Han et al. measured the displacement of a structure via continuous scaling factor updating by generating a laser spot on the same surface as the target measurement point [40].
As described earlier, various studies have been conducted on the VDMS using UAVs, but a reliable measurement range must be clearly determined for its application in the field. In previous studies, the reliability and measurable range of the VDMS using UAVs were not considered. Hence, in this study, measurements were performed while the frequency of sine waves and displacement were varied, and the results were compared with those obtained using conventional displacement sensors in an experiment examining the dynamic measurement reliability of VDMSs using UAVs. As regards the method for correcting the movement of UAVs, the camera calibration technique based on a checkerboard was applied. Additionally, free vibration was generated in a model structure with one and two stories, and the accuracy for estimating the mode frequency, damping ratio, and mode shape was examined to verify the validity of the obtained dynamic characteristics in a structure.

2. Materials and Methods

2.1. Camera Calibration

The camera calibration technique is the process of obtaining a transformation matrix where a 3D object is projected onto a 2D image plane by acquiring transformation data (e.g., size, shape, rotation, and distortion) lost in the process of projecting a 3D object and space onto a 2D image. A 3D object is affected by the factors inside a camera, such as the lens, focal distance, and image sensor, during its output as an image. Therefore, the internal factors of the camera should be determined for the 3D restoration of the object within the image. Furthermore, the 3D positional relationship between the camera and the object should be considered. Accordingly, Figure 1 shows the 2D image coordinates and 3D world coordinates.
The transformation matrix between a point in 3D space X and the corresponding point in the image plane coordinate system of the 2D camera x is as follows [41].
X = X , Y , Z T
x = x , y T
s x 1 = f x S k c x 0 f y c y 0 0 1 r 11 r 12 r 13 t 1 r 21 r 22 r 23 t 2 r 31 r 32 r 33 t 3 X 1
where s denotes the scale factor, f x and f y denote the focal distances of the camera, and c x and c y denote the principal point, which is the center of the camera lens. Furthermore, S k denotes the skewness coefficient, r i j ( i = 1,2,3, j = 1,2,3) denotes the rotational movement element, and t i ( i = 1,2,3) denotes the parallel movement element. The first 3 × 3 matrix refers to the internal parameters of the camera, including the camera focal distance and angle between the image sensor and lens, whereas the second 3 × 4 matrix refers to external parameters, including geometric factors such as the camera position and direction.
Because projecting a 2D image to 3D world coordinates using a single camera has scale ambiguity, the Z coordinates of all moving targets remain constant during tests. Z in Equation (1) can be replaced by a constant value Z ˜ to reconstruct the world coordinates of the target. Therefore, Equation (3) can be rewritten as Equation (4) [39].
s x 1 = f x S k c x 0 f y c y 0 0 1 r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 X Y Z ˜ + f x S k c x 0 f y c y 0 0 1 t 1 t 2 t 3
Equation (4) is rewritten as Equation (5) to calculate the motion of the target in global coordinates.
X Y Z ˜ = s r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 1 f x S k c x 0 f y c y 0 0 1 1 x 1 r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 1 t 1 t 2 t 3

2.2. UAV Movement Correction

Unlike the existing VDMS technique, which measures displacement by attaching a checkerboard to the target structure to be measured, the rotation angle of the camera must be extracted using an additional fixed point at which displacement does not occur in the image. One checkerboard is attached to the structure that moves, whereas the other checkerboard is attached to the fixed point with no displacement. In this case, because locating the two checkerboards on the same plane is difficult, the distance between them must be considered to remove the displacement generated by camera rotation. In this study, the position of the calibration board was obtained using MATLAB 2022b’s Camera Calibration application [42].
The movement and rotation of the UAV must be eliminated to calibrate the actual displacement of the target structure. Figure 2 shows the displacement of the target structure, including the UAV’s movement and rotation and the displacement of the background target installed to calibrate the UAV’s movement and rotation. If the total displacement of the structure captured by the moving and rotating the UAV is assumed as D 1 , then D 1 includes the actual displacement generated in the structure ( D r ), the displacement generated by the UAV’s rotation ( D 2 ), and the displacement generated by the UAV’s movement ( D 3 ). Moreover, the total displacement of the background target is D 4 , including the displacement generated by the UAV’s rotation at the background target ( D 5 ) and the displacement generated by the UAV’s movement ( D 3 ). Therefore, to measure the accurate displacement of the structure, the displacement generated by the UAV’s movement and rotation must be removed from the total displacement of the target structure in the following process.
D 5 = H 2 tan θ
D 3 = D 4 D 5
D 5 D 2 = H 2 H 1 tan θ
D 2 = D 5 H 2 H 1 tan θ
D r = D 1 D 2 D 3
where H 1 denotes the distance between the camera and structure and H 2 denotes the distance between the UAV and the background target attached to the fixed point with no displacement. Note that θ denotes the rotation angle of the UAV with respect to the Y-axis.

3. Vibration Measurement Performance Verification of VDMS Using a UAV

3.1. Test Overview

In this study, a shaking table test was performed by applying sine waves while increasing the frequency by 0.2 Hz (from 0.2 to 3 Hz) and the displacement by 5 mm (from 5 to 100 mm) to examine the vibration measurement performance of the VDMS using a UAV. The measurements of the VDMS using a UAV were compared with those of conventional displacement measurement sensors (i.e., LVDT and LDS). UAV imaging was performed at a distance of ~5 m from the target, and the altitude was set to 1 m to capture the images of the target from the front. A grid-type checkerboard was used as the target for the VDMS using a UAV. The checkerboard had 12 edges, and the gap between the edges was 70 mm. The test time was set to be more than ten times the excitation period to secure sufficient data for the estimation of dynamic characteristics. Figure 3 shows the shaking table, background target, displacement sensor, and the scene of the experiment where the UAV captured the images of the target.
The measurement accuracy was evaluated by determining the root mean square (RMS). Equation (11) shows the calculation process, where x i is i th displacement data and n is the total number of displacement data. Furthermore, the natural frequency and magnitude errors were compared in the frequency domain.
RMS = i = 1 n x i 2 n

3.2. Measurement Equipment Configuration

The experimental setup was composed of a shaking table that can apply sine waves, displacement sensors (LVDT and LDS), UAV for imaging, and a measurement target. A portable one-axis vibration device was used as the shaking table. It comprised a shaking table, a control panel, and a touch panel. When the frequency and displacement are set and an operational command is provided using the touch panel, three-phase signals for driving the motor are produced from the servo pack installed in the control panel. The shaking table was composed of Al of dimensions 550 mm × 350 mm × 75 mm. M5 tapped holes were drilled in both directions at 25 mm intervals on the upper table. KL3-W400 from KAIS, a Korean company [43], was used as the laser sensor, and DP-500E from Tokyo Sokki, a Japanese company, was used as the LVDT [44]. Measurements using the LDS and LVDT were simultaneously acquired using TMR-311 [45], a data logger from Japan that can adjust the sampling rate at regular intervals, and a sampling rate of 1000 Hz was set in the experiment for dynamic displacement measurement. Chinese company Dji’s MAVIC 3 [46] was used as a UAV, and the imaging resolution was set to 4K (3840 × 2160) to increase the accuracy of recognizing the edges of the target. The frame rate was set to 30 frames per second (fps) and the shutter speed to 1/1000 s considering the image analysis time. Table 1 shows the detailed specifications of the equipment used in the experiment.

3.3. Analysis of Measurement Results

The VDMS using UAV measurement was performed by capturing the images of the target using a UAV camera in flight and postprocessing them. The displacement was extracted by analyzing the measurement target and background target after setting the ROI in the video to reduce the analysis time, as shown in Figure 4.
The measurement results from the third cycle after the start of measurement to the last vibration were used as data for analysis. For error analysis, the data of all sensors were linearly interpolated, and the sampling rate was matched to 2000 Hz.

3.3.1. Analysis of Displacement Response Measurement Results

The calibration method presented in Section 2 was used to estimate the displacement by the movement ( D 3 ) and rotation ( D 2 ) of the UAV to determine the absolute displacement of the measurement target. Figure 5 shows the result of the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm. Figure 5 shows that the movement of the UAV generates a displacement of up to 19.823 mm, whereas the rotation of the UAV generates a displacement of up to 1.368 mm.
Figure 6a,c show displacement response graphs for the experiment with an excitation frequency of 2 Hz and an excitation displacement of 10 mm and the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm. Furthermore, peaks were magnified in Figure 6b,d to examine the accuracy at the displacement peak. The VDMS using a UAV demonstrated a tendency similar to those of the LDS and LVDT, but a difference of 1–2 mm was observed at the peak.
The accuracy of displacement measurement was examined by obtaining the percentage errors of the RMS values of displacement with the LDS. Figure 7 shows the percentage errors of the RMS values of the LVDT and UAV when compared with the LDS. The LVDT’s difference in error as per the excitation frequency was not large; however, the error increased when the maximum excitation displacements were ~10 and 100 mm. The maximum error of the RMS value of the LVDT was 2.851% under the experiment with an excitation frequency of 1.4 Hz and an excitation displacement of 15 mm, whereas the minimum error was 0.009% under the experiment with an excitation frequency of 2.8 Hz and an excitation displacement of 60 mm. The average error was 0.541%. For the VDMS using a UAV, the maximum error of 4.589% occurred under the experiment with an excitation frequency of 3 Hz and an excitation displacement of 5 mm, and the minimum error of 0.003% was observed under the experiment with an excitation frequency of 0.8 Hz and an excitation displacement of 95 mm. The average error was 0.622%, which was similar to those of the LDS and LVDT; however, the error tended to decrease as the excitation displacement increased.

3.3.2. Analysis of Measurement Results in the Frequency Domain

Figure 8 shows the responses of the experiment with an excitation frequency of 2 Hz and an excitation displacement of 10 mm and the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm in the frequency domain using the power spectral density (PSD) function. All of the displacement sensors clearly exhibited peaks in the frequency domain, and the same excitation frequency was observed in all the tests. However, the PSD amplitude showed a difference, as in the displacement response results.
The LVDT demonstrated a maximum error of 2.8929% in the experiment with an excitation frequency of 1.4 Hz and an excitation displacement of 15 mm and a minimum error of 0.0013% in the experiment with an excitation frequency of 2.2 Hz and an excitation displacement of 55 mm. The average error of the LVDT was 0.7697%, which was similar to that of the LDS, as shown in Figure 9. Furthermore, the VDMS using a UAV showed a maximum error of 4.0633% in the experiment with an excitation frequency of 0.4 Hz and an excitation displacement of 25 mm and a minimum error of 0.0082% in the experiment with an excitation frequency of 1 Hz and an excitation displacement of 65 mm. The average error of the VDMS using a UAV was 0.8164%, which was not significantly different from that of the LVDT.

4. Structure Measurement Accuracy Verification Test

4.1. Test Overview

Model structures with one and two stories were excited to verify the structure measurement accuracy of the VDMS using a UAV, and the accuracy was analyzed compared with the LDS utilized in Section 3. Accelerometers were used additionally for analyzing the accuracy of the identification of dynamic characteristics. The model structure was composed of aluminum alloy masses with dimensions of 600 mm × 600 mm × 20 mm and steel columns with a width of 50 mm. The thickness (3.0 mm) of the columns was significantly smaller compared with their width to induce the weak-axis direction behavior of the entire structure. For the accelerometers, KS48C from MMF GmbH [47] was used. The accelerometers and the LDS were installed on all floors. Figure 10 shows the front view of a two-story structure in an experiment to verify the structure measurement accuracy, where the targets were attached on the ground, first, and second floors. Similar to the vibration measurement performance verification test, UAV imaging was performed from the front to measure the measurement and background targets.
The structure was constructed in the range of one and two stories. The highest floor of the structure was excited, and free vibration was applied two times. The mode frequency, damping ratio, and mode shape, based on the displacement response of the structure, were compared with the LDS and accelerometer measurement results.

4.2. Analysis of Measurement Results

4.2.1. Mode Frequency Estimation Results

Figure 11 and Figure 12 show the free vibration measurement results for the structures with one and two stories in the frequency domain using the PSD function. Only the 0 to 6 Hz frequency band clearly indicated the mode frequency peaks of the model structure. All of the results from the accelerators, LDS, and the VDMS using a UAV were the same for the mode frequency of the one- and two-story structures. In Figure 12a,b, harmonic peaks with a frequency of an integer multiple of the first mode frequency occurred around 3.8 Hz, and these harmonic peaks were ignored when estimating mode frequencies and mode shapes. Table 2 lists the mode frequency estimation results for each number of stories.

4.2.2. Damping Ratio Estimation Results

The damping ratio was estimated using the logarithmic decrement method in the time domain. The damping ratio estimation results used the response from the highest floor of the structure, and the errors of LDS and the VDMS using a UAV were compared on the basis of the accelerometers.
The logarithmic decrement method represents the degree of reduction in amplitude at a logarithmic rate after one cycle in the time history measurement results of a free-vibrating structure. In this study, the damping ratio was estimated in 30 positive and negative peak sections in the measurement results, and the average values were derived, as listed in Table 3.
The damping ratios of the accelerometer on the highest floor of the two-story structure were determined as 0.5890% in the positive section and 0.6877% in the negative section, and the average was estimated as 0.6383%. Meanwhile, the damping ratios of the LDS were 0.6257% in the positive section and 0.6198% in the negative section, and the average was estimated as 0.5338%. For the VDMS using a UAV, the damping ratios were 0.6278% in the positive section and 0.7247% in the negative section, and the average was determined as 0.6368%.
In the one-story structure, the error of the LDS was 14.4866% with the accelerometer, whereas the error of the VDMS using a UAV was 2.0993%. In the two-story structure, the error of LDS was 2.4396%, whereas the error of the VDMS using a UAV was 0.2428%. When the estimated damping ratio of the VDMS using a UAV was compared with that of the accelerometer, there was no significant difference from the LDS results.

4.2.3. Mode Shape Estimation Results

The mode shape was expressed using the frequency domain decomposition (FDD) method [48]. Specifically, the FDD method obtains the natural frequency and mode shape by applying singular value decomposition to the result from the PSD function. It can obtain the mode shape even when the characteristics of the excitation load are unknown. Figure 13 shows the mode shapes of the first and second modes obtained from the two-story structure measurement results of the accelerometers, LDS, and the VDMS using a UAV.
The modal assurance criterion (MAC) values of all specimens for each mode were obtained from the measurements of LDS and the VDMS using a UAV, and the average of the mode MAC values was obtained for each number of stories, as listed in Table 4.
For LDS and the VDMS using a UAV, the lowest MAC values (0.9900 and 0.9972, respectively) were observed in the second mode of the two-story structure. Furthermore, LDS and the VDMS using a UAV clearly showed mode shapes and relatively high MAC values in all tests.

5. Discussion and Conclusions

In this study, vibrations were generated using a shaking table by increasing the excitation frequency by 0.2 Hz (from 0.2 to 3 Hz) and the displacement by 5 mm (from 5 to 100 mm) to verify the dynamic measurement accuracy of the VDMS using a UAV. Specifically, various experiments were conducted using the UAV, and the measurement results were compared with those of LDS and LVDT. Additionally, free vibration was applied to the multi-DOF model structure, and measurements were performed using LDS and accelerometers to verify the performance for identifying structural dynamic characteristics.
In the test for vibration measurement performance verification, the vibration measurement accuracy of the VDMS using a UAV was verified by calculating the RMS of the measurements of the LVDT and the VDMS using a UAV based on the LDS measurements. In the RMS percentage error analysis results, the LVDT showed no error tendency according to the test case and exhibited a maximum error of 2.851%. The LVDT showed such type of tendency because the peak could not be accurately achieved as the excitation displacement and excitation frequency increased because of the limit of the contact sensor. For the VDMS using a UAV, the error tended to decrease as the excitation displacement increased, and the maximum error was 4.589%. The reason why the error of the VDMS using a UAV was larger than that of the LVDT was that the movement of the UAV could not be completely eliminated because of the distortion by the camera lens and the rotation angle estimation error in the course of subtracting the displacement value of the background target from that of the measurement target. In addition, regardless of the frequency, a relatively large error appeared in the displacement measurement of 10 mm or less, and the reliability of the VDMS using the UAV was lowered in this section.
In the structure measurement accuracy verification test, the mode frequency, damping ratio, and mode shape estimation results were compared. In the mode frequency estimation results, all of the accelerators, the LDS, and the VDMS using a UAV exhibited the same results. In the estimation of the damping ratio, all measurement sensors showed similar results, excluding the damping ratio of the two-story structure of the LDS. In the estimation of the mode shape, all measurement sensors clearly showed mode shapes and high MAC values.
Given the above results, the VDMS using a UAV can replace conventional displacement sensors. Additionally, more accurate measurements would be possible if displacement is measured by applying a filter that can remove the residual tendency due to the movement of the drone. Moreover, methods for removing the movement of UAVs by finding key points that can correct such movements on the same plane as the measurement target must be studied.

Author Contributions

Writing—original draft, G.K.; Writing—review & editing, H.K.; Project administration, H.K. All authors have read and agreed to the published version of the manuscript.

Funding

This work was carried out with the support of “Cooperative Research Program for Agriculture Science & Technology Development (Project No. PJ01560502)” Rural Development Administration, Republic of Korea.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data available on request due to restrictions.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Worden, K.; Dulieu-Barton, J.M. An overview of intelligent fault detection in systems and structures. Struct. Health Monit. 2004, 3, 85–98. [Google Scholar] [CrossRef]
  2. Cho, S.; Spencer, B.F., Jr. Sensor attitude correction of wireless sensor network for acceleration-based monitoring of civil structures. Comput. Aided Civ. Infrastruct. Eng. 2015, 30, 859–871. [Google Scholar] [CrossRef]
  3. Khuc, T.; Catbas, F.N. Computer vision-based displacement and vibration monitoring without using physical target on structures. Struct. Infrastruct. Eng. 2017, 13, 505–516. [Google Scholar] [CrossRef]
  4. Gentile, C. Application of microwave remote sensing to dynamic testing of stay-cables. Remote Sens. 2010, 2, 36–51. [Google Scholar] [CrossRef]
  5. Oberholster, A.J.; Heyns, P.S. Online condition monitoring of axial-flow turbomachinery blades using rotor-axial Eulerian laser Doppler vibrometry. Mech. Syst. Signal Process. 2009, 23, 1634–1643. [Google Scholar] [CrossRef]
  6. Nassif, H.H.; Gindy, M.; Davis, J. Comparison of laser Doppler vibrometer with contact sensors for monitoring bridge deflection and vibration. Struct. Faults Repair 2005, 38, 213–218. [Google Scholar] [CrossRef]
  7. Xu, L.; Guo, J.J.; Jiang, J.J. Time–frequency analysis of a suspension bridge based on GPS. J. Sound Vib. 2002, 254, 105–116. [Google Scholar] [CrossRef]
  8. Feng, D.; Feng, M.Q. Vision-based multipoint displacement measurement for structural health monitoring. Struct. Control Health Monit. 2016, 23, 876–890. [Google Scholar] [CrossRef]
  9. Que, Y.; Dai, Y.; Ji, X.; Leung, A.K.; Chen, Z.; Tang, Y.; Jiang, Z. Automatic classification of asphalt pavement cracks using a novel integrated generative adversarial networks and improved VGG model. Eng. Struct. 2023, 277, 115406. [Google Scholar] [CrossRef]
  10. Ali, L.; Alnajjar, F.; Jassmi, H.A.; Gocho, M.; Khan, W.; Serhani, M.A. Performance Evaluation of Deep CNN-Based Crack Detection and Localization Techniques for Concrete Structures. Sensors 2021, 21, 1688. [Google Scholar] [CrossRef]
  11. Zhao, S.; Kang, F.; Li, J.; Ma, C. Structural health monitoring and inspection of dams based on UAV photogrammetry with image 3D reconstruction. Autom. Constr. 2021, 130, 103832. [Google Scholar] [CrossRef]
  12. Zhao, S.; Kang, F.; Li, J. Concrete dam damage detection and localisation based on YOLOv5s-HSC and photogrammetric 3D reconstruction. Autom. Constr. 2022, 143, 104555. [Google Scholar] [CrossRef]
  13. Tang, Y.; Zhu, M.; Chen, Z.; Wu, C.; Chen, B.; Li, C.; Li, L. Seismic performance evaluation of recycled aggregate concrete-filled steel tubular columns with field strain detected via a novel mark-free vision method. Structures 2022, 37, 426–441. [Google Scholar] [CrossRef]
  14. Zona, A. Vision-based vibration monitoring of structures and infrastructures: An overview of recent applications. Infrastructures 2021, 6, 4. [Google Scholar] [CrossRef]
  15. Peters, W.H.; Ranson, W.F. Digital imaging techniques in experimental stress analysis. Opt. Eng. 1982, 21, 427–431. [Google Scholar] [CrossRef]
  16. Schumacher, T.; Shariati, A. Monitoring of structures and mechanical systems using virtual visual sensors for video analysis: Fundamental concept and proof of feasibility. Sensors 2013, 13, 16551–16564. [Google Scholar] [CrossRef]
  17. Spencer, B.F.; Hoskere, V.; Narazaki, Y. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 2019, 5, 199–222. [Google Scholar] [CrossRef]
  18. Park, S.W.; Park, H.S.; Kim, J.H.; Adeli, H. 3D displacement measurement model for health monitoring of structures using a motion capture system. Measurement 2015, 59, 352–362. [Google Scholar] [CrossRef]
  19. Choi, H.S.; Cheung, J.H.; Kim, S.H.; Ahn, J.H. Structural dynamic displacement vision system using digital image processing. NDT E Int. 2011, 44, 597–608. [Google Scholar] [CrossRef]
  20. Kim, H.; Shin, S. Reliability verification of a vision-based dynamic displacement measurement for system identification. J. Wind Eng. Ind. Aerodyn. 2019, 191, 22–31. [Google Scholar] [CrossRef]
  21. Feng, D.; Feng, M.Q.; Ozer, E.; Fukuda, Y.A. Vision-based sensor for noncontact structural displacement measurement. Sensors 2015, 15, 16557–16575. [Google Scholar] [CrossRef] [PubMed]
  22. Lee, J.J.; Shinozuka, M. Real-time displacement measurement of a flexible bridge using digital image processing techniques. Exp. Mech. 2006, 46, 105–114. [Google Scholar] [CrossRef]
  23. Lee, J.H.; Ho, H.N.; Lee, J.J.; Shinozuka, M. An advanced vision-based system for real-time displacement measurement of high-rise buildings. Smart Mater. Struct. 2012, 21, 10. [Google Scholar] [CrossRef]
  24. Lydon, D.; Lydon, M.; Taylor, S.; Del Rincon, J.M.; Hester, D.; Brownjohn, J. Development and field testing of a vision-based displacement system using a low cost wireless action camera. Mech. Syst. Signal Process. 2019, 121, 343–358. [Google Scholar] [CrossRef]
  25. Ngeljaratan, L.; Moustafa, M.A. Structural health monitoring and seismic response assessment of bridge structures using target-tracking digital image correlation. Eng. Struct. 2020, 213, 110551. [Google Scholar] [CrossRef]
  26. Scislo, L. Single-Point and Surface Quality Assessment Algorithm in Continuous Production with the Use of 3D Laser Doppler Scanning Vibrometry System. Sensors 2023, 23, 1263. [Google Scholar] [CrossRef]
  27. Metni, N.; Hamel, T. A UAV for bridge inspection: Visual servoing control law with orientation limits. Autom. Constr. 2007, 17, 3–10. [Google Scholar] [CrossRef]
  28. Shen-En, C.; Corey, R.; Chuck, B.; Edd, H. Small-format aerial photography for highway-bridge monitoring. J. Perform. Constr. Facil. 2011, 25, 105–112. [Google Scholar] [CrossRef]
  29. Zhang, C.; Elaksher, A. An unmanned aerial vehicle-based imaging system for 3D measurement of unpaved road surface distresses. Comput. Aided Civ. Infrastruct. Eng. 2012, 27, 118–129. [Google Scholar] [CrossRef]
  30. Ellenberg, A.; Branco, L.; Krick, A.; Bartoli, I.; Kontsos, A. Use of unmanned aerial vehicle for quantitative infrastructure evaluation. J. Infrastruct. Syst. 2015, 21, 04014054. [Google Scholar] [CrossRef]
  31. Yeum, C.M.; Dyke, S.J. Vision-based automated crack detection for bridge inspection. Comput. Aided Civ. Infrastruct. Eng. 2015, 30, 759–770. [Google Scholar] [CrossRef]
  32. Perry, B.J.; Guo, Y. A portable three-component displacement measurement technique using an unmanned aerial vehicle (UAV) and computer vision: A proof of concept. Measurement 2021, 176, 109222. [Google Scholar] [CrossRef]
  33. Ribeiro, D.; Santos, R.; Cabral, R.; Saramago, G.; Montenegro, P.; Carvalho, H.; Correia, J.; Calçada, R. Non-contact structural displacement measurement using Unmanned Aerial Vehicles and video-based systems. Mech. Syst. Signal Process. 2021, 160, 107869. [Google Scholar] [CrossRef]
  34. Yoon, H.; Hoskere, V.; Park, J.W.; Spencer, B.F. Cross-correlation-based structural system identification using unmanned aerial vehicles. Sensors 2017, 17, 2075. [Google Scholar] [CrossRef] [PubMed]
  35. Yoon, H.; Shin, J.; Spencer, B.F., Jr. Structural displacement measurement using an unmanned aerial system. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 183–192. [Google Scholar] [CrossRef]
  36. Lee, Y.; Lee, G.; Moon, D.S.; Yoon, H. Vision-based displacement measurement using a camera mounted on a structure with stationary background targets outside the structure. Struct. Control Health Monit. 2022, 29, e3095. [Google Scholar] [CrossRef]
  37. Weng, Y.; Shan, J.; Lu, Z.; Lu, X.; Spencer, B.F., Jr. Homography-based structural displacement measurement for large structures using unmanned aerial vehicles. Comput. Aided Civ. Infrastruct. Eng. 2021, 36, 1114–1128. [Google Scholar] [CrossRef]
  38. Yan, Z.; Jin, Z.; Teng, S.; Chen, G.; Bassir, D. Measurement of bridge vibration by UAVs combined with CNN and KLT optical-flow method. Appl. Sci. 2022, 12, 5181. [Google Scholar] [CrossRef]
  39. Wang, X.; Lo, E.; De Vivo, L.; Hutchinson, T.C.; Kuester, F. Monitoring the earthquake response of full-scale structures using UAV vision-based techniques. Struct. Control Health Monit. 2022, 29, e2862. [Google Scholar] [CrossRef]
  40. Han, Y.; Wu, G.; Feng, D. Vision-based displacement measurement using an unmanned aerial vehicle. Struct. Control Health Monit. 2022, 29, e3025. [Google Scholar] [CrossRef]
  41. Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1330–1334. [Google Scholar] [CrossRef]
  42. Available online: https://www.mathworks.com/help/vision/camera-calibration.html (accessed on 17 February 2023).
  43. Available online: https://ekais.kr/kais/item.php?it_id=1616043998 (accessed on 17 February 2023).
  44. Available online: https://tml.jp/e/product/transducers/dp.html (accessed on 13 February 2023).
  45. Available online: https://tml.jp/e/product/instrument/tmr311.html (accessed on 13 February 2023).
  46. Available online: https://www.dji.com/kr/mavic-3 (accessed on 13 February 2023).
  47. Available online: https://www.mmf.de/empfindlich.htm#kb12-ks48 (accessed on 13 February 2023).
  48. Brincker, R.; Zhang, L.; Andersen, P. Output-Only Modal Analysis by Frequency Domain Decomposition. In Proceedings of the ISMA25: 2000 International Conference on Noise and Vibration Engineering, Leuven, Belgium, 13–15 September 2000; Sas, P., Moens, D., Eds.; Katholieke Universiteit: Leuven, Belgium, 2000; pp. 717–723. [Google Scholar]
Figure 1. Configuration of a point in a 2D image coordinate and the corresponding 3D world coordinate.
Figure 1. Configuration of a point in a 2D image coordinate and the corresponding 3D world coordinate.
Sensors 23 03232 g001
Figure 2. Overview of a UAV’s motion elimination method.
Figure 2. Overview of a UAV’s motion elimination method.
Sensors 23 03232 g002
Figure 3. Front view and sensor setting status of the test for the vibration measurement performance verification: (a) front view of the test; (b) enlarged LDS and LVDT installation view and orange and blue dashed lines indicate the measurement direction of the displacement sensor.
Figure 3. Front view and sensor setting status of the test for the vibration measurement performance verification: (a) front view of the test; (b) enlarged LDS and LVDT installation view and orange and blue dashed lines indicate the measurement direction of the displacement sensor.
Sensors 23 03232 g003
Figure 4. Experiment view showing specific ROI in UAV images (the lower left image represents the measurement target, and the lower right image represents the background target).
Figure 4. Experiment view showing specific ROI in UAV images (the lower left image represents the measurement target, and the lower right image represents the background target).
Sensors 23 03232 g004
Figure 5. VDMS using the UAV result of the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm: (a) total displacement of the measurement target captured by the moving and rotating UAV ( D 1 ); (b) displacement generated by the UAV’s rotation ( D 2 ); (c) displacement generated by the UAV’s movement ( D 3 ); and (d) actual displacement generated in the measurement target ( D r ).
Figure 5. VDMS using the UAV result of the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm: (a) total displacement of the measurement target captured by the moving and rotating UAV ( D 1 ); (b) displacement generated by the UAV’s rotation ( D 2 ); (c) displacement generated by the UAV’s movement ( D 3 ); and (d) actual displacement generated in the measurement target ( D r ).
Sensors 23 03232 g005
Figure 6. Displacement response graphs of the vibration measurement performance verification test: (a) response for the experiment with an excitation frequency of 2 Hz and an excitation displacement of 10 mm; (b) one peak in (a); (c) response for the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm; and (d) one peak in (c).
Figure 6. Displacement response graphs of the vibration measurement performance verification test: (a) response for the experiment with an excitation frequency of 2 Hz and an excitation displacement of 10 mm; (b) one peak in (a); (c) response for the experiment with an excitation frequency of 2 Hz and an excitation displacement of 100 mm; and (d) one peak in (c).
Sensors 23 03232 g006
Figure 7. Percent error of the RMS value compared with LDS: (a) LVDT error and (b) error for the VDMS using a UAV.
Figure 7. Percent error of the RMS value compared with LDS: (a) LVDT error and (b) error for the VDMS using a UAV.
Sensors 23 03232 g007
Figure 8. Frequency response graphs of the vibration measurement performance verification experiment: (a) experimental result at an excitation frequency of 2 Hz and an excitation displacement of 10 mm; (b) peak interval graph of (a); (c) experimental result at an excitation frequency of 2 Hz and an excitation displacement of 100 mm; and (d) peak interval graph of (c).
Figure 8. Frequency response graphs of the vibration measurement performance verification experiment: (a) experimental result at an excitation frequency of 2 Hz and an excitation displacement of 10 mm; (b) peak interval graph of (a); (c) experimental result at an excitation frequency of 2 Hz and an excitation displacement of 100 mm; and (d) peak interval graph of (c).
Sensors 23 03232 g008aSensors 23 03232 g008b
Figure 9. PSD amplitude error compared with LDS: (a) LVDT error; (b) error of the VDMS using a UAV.
Figure 9. PSD amplitude error compared with LDS: (a) LVDT error; (b) error of the VDMS using a UAV.
Sensors 23 03232 g009
Figure 10. Front view of a two-story structure in an experiment to verify the structure measurement accuracy: (a) front view of a two-story structure; (b) enlarged 2nd floor accelerometer and LDS installation view.
Figure 10. Front view of a two-story structure in an experiment to verify the structure measurement accuracy: (a) front view of a two-story structure; (b) enlarged 2nd floor accelerometer and LDS installation view.
Sensors 23 03232 g010
Figure 11. Frequency response graphs for the one-story model structure: (a) acceleration response; (b) LDS response; (c) response of the VDMS using a UAV.
Figure 11. Frequency response graphs for the one-story model structure: (a) acceleration response; (b) LDS response; (c) response of the VDMS using a UAV.
Sensors 23 03232 g011
Figure 12. Frequency response graphs for the two-story model structure: (a) acceleration response; (b) LDS response; (c) response of the VDMS using a UAV.
Figure 12. Frequency response graphs for the two-story model structure: (a) acceleration response; (b) LDS response; (c) response of the VDMS using a UAV.
Sensors 23 03232 g012
Figure 13. Mode shapes of the two-story model structure: (a) first mode and (b) second mode.
Figure 13. Mode shapes of the two-story model structure: (a) first mode and (b) second mode.
Sensors 23 03232 g013
Table 1. Vibration measurement performance verification test equipment.
Table 1. Vibration measurement performance verification test equipment.
ComponentModelTechnical Specifications
Shaking
table
Sensors 23 03232 i001
TE Solution’s portable 1-axis vibration device
Maximum speed: 4.8 m/s
Rated thrust: 140 N
Rated current: 2.2 Arms
Thermal resistance: 1.48 K/W
LVDTSensors 23 03232 i002
Tokyo Sokki’s DP-500E
Capacity: 500 mm
Sensibility: 20 μ strain/mm
Nonlinearity: 0.3% rated output
LDSSensors 23 03232 i003
KAIS’s KL3-W400
Measurement distance: 500 mm
Measurable range: ±200 mm
Optical mode: Diffuse reflection
Resolution: 10 μm
Data loggerSensors 23 03232 i004
Tokyo Sokki’s TMR-311
Sampling time: 1–1000 ms (set by every 1 ms)
Power supply: AC 100–240 V, 50/60 Hz, 100 VA at maximum
Measurement range: ±20,000 × 10−6 strain (bridge excitation DC 2V)
UAVSensors 23 03232 i005
DJI’s MAVIC 3
Takeoff weight: 895 g
Max flight time: 46 min
Max hovering time: 40 min
Max wind speed resistance: 12 m/s
UAV
Camera
Sensors 23 03232 i006
DJI’s Hasselblad Camera
Sensor: 4/3 CMOS, Effective pixels: 20 MP
ISO Range: 100–6400
Shutter Speed: 8–1/8000 s
FOV: 84°
Format Equivalent: 24 mm
Aperture: f/2.8 to f/11
Focus: 1 m to ∞ (with autofocus)
Video Resolution:
H.264/H.265
5.1K: 5120 × [email protected]/25/30/48/50fps
DCI 4K: 4096 × [email protected]/25/30/48/50/60/120fps
4K: 3840 × [email protected]/25/30/48/50/60/120fps
FHD: 1920 × [email protected]/25/30/48/50/60/120/200fps
Table 2. Mode frequency estimation results for one and two-story structures (Hz).
Table 2. Mode frequency estimation results for one and two-story structures (Hz).
Number of
Stories
ModeAccelerometerLDSUAV
11st3.173833.173833.17383
21st1.892091.892091.89209
2nd5.096445.096445.09644
Table 3. Result of damping ratio estimation on the top floor of one- and two-story structures.
Table 3. Result of damping ratio estimation on the top floor of one- and two-story structures.
Number of StoriesAccelerometerLDSUAV
10.447102%0.382332%
(14.48663)
0.456488%
(2.09930)
20.638349%0.622776%
(2.43957)
0.636799%
(0.24281)
() Percentage error with an accelerometer.
Table 4. Mode MAC value estimation results for one- and two-story structures.
Table 4. Mode MAC value estimation results for one- and two-story structures.
Number of StoriesNumber of ModesLDSUAV
11st11
21st0.99990.9996
2nd0.99000.9972
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kim, H.; Kim, G. Reliability Assessment of a Vision-Based Dynamic Displacement Measurement System Using an Unmanned Aerial Vehicle. Sensors 2023, 23, 3232. https://doi.org/10.3390/s23063232

AMA Style

Kim H, Kim G. Reliability Assessment of a Vision-Based Dynamic Displacement Measurement System Using an Unmanned Aerial Vehicle. Sensors. 2023; 23(6):3232. https://doi.org/10.3390/s23063232

Chicago/Turabian Style

Kim, Hongjin, and Guyeon Kim. 2023. "Reliability Assessment of a Vision-Based Dynamic Displacement Measurement System Using an Unmanned Aerial Vehicle" Sensors 23, no. 6: 3232. https://doi.org/10.3390/s23063232

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop