Three-Dimensional Reconstruction-Based Vibration Measurement of Bridge Model Using UAVs

Abstract

This paper presents a measurement method of bridge vibration based on three-dimensional (3D) reconstruction. A video of bridge model vibration is recorded by an unmanned aerial vehicle (UAV), and the displacement of target points on the bridge model is tracked by the digital image correlation (DIC) method. Due to the UAV motion, the DIC-tracked displacement of the bridge model includes the absolute displacement caused by the excitation and the false displacement induced by the UAV motion. Therefore, the UAV motion must be corrected to measure the real displacement. Using four corner points on a fixed object plane as the reference points, the projection matrix for each frame of images can be estimated by the UAV camera calibration, and then the 3D world coordinates of the target points on the bridge model can be recovered. After that, the real displacement of the target points can be obtained. To verify the correctness of the results, the operational modal analysis (OMA) method is used to extract the natural frequencies of the bridge model. The results show that the first natural frequency obtained from the proposed method is consistent with the one obtained from the homography-based method. By further comparing with the homography-based correction method, it is found that the 3D reconstruction method can effectively overcome the limitation of the homography-based method that the fixed reference points and the target points must be coplanar.

1. Introduction

During the long-term operation process of bridges, various damage may occur and lead to the reduction of their bearing capacity inevitably. To ensure the safe operation of bridges and avoid collapse accidents, it is necessary to carry out real-time health monitoring for bridges. The anomaly detection methods, such as the temperature-driven moving principal component analysis (MPCA) method [1], threshold-based, and anomaly trend detection method [2], have been used for structural health monitoring (SHM). Currently, vibration-based damage detection of bridges plays a vital role in SHM [3]. A fundamental task of vibration-based detection methods is to measure the vibration responses accurately. Vibration measurement is mainly divided into contact methods and non-contact methods. The contact measurement methods are primarily to use the sensors installed on the structural surface, such as displacement sensors [4], acceleration sensors [5,6], and strain gauges [7] to collect structural responses. Although acceleration is commonly used for measurement, it is inaccurate in low-frequency vibration; in addition, the corresponding velocity and displacement time histories obtained show irrational drift and usually need to be corrected [8]. There are some other disadvantages for the contact methods. Firstly, a large number of sensors need to be deployed on the structure for the whole field measurement, which is costly or impossible in some cases, as the sensor weight may affect the measurement results in some light structures. In some situations, sensors are difficult to be deployed. Therefore, some non-contact measurement methods, such as the global positioning system (GPS) [9], and laser doppler vibrometer (LDV) [10], have been used to measure structural dynamic displacement under these situations. As the measurement accuracy and sampling frequency of GPS is low, it is generally used for bridges with large vibration amplitude [11]. Though the LDV system can be utilized to collect the vibration response of bridges at various locations, it is costly and time-consuming to measure vibration in practical applications [12]; moreover, it is not suitable for long-term bridge monitoring because it needs to be deployed on the ground underneath the bridge and supervised at all times [13].

With the rapid development of computer technology and image processing technology, the vision-based measurement methods [14,15,16] have been widely used as they are accurate, non-contact, and full-field. Compared with contact methods, the vision-based methods provide the possibility for displacement measurement of a bridge at a distance. Through the motion magnification analyses, the vision-based methods can identify modal parameters for SHM without providing displacement measurement [17]. In addition, it is more accurate and applicable than other non-contact methods, e.g., GPS. Moreover, the emergence of low-price and high-resolution cameras makes these methods more and more popular. Among the vision-based methods, the digital image correlation (DIC) [18,19] has been commonly used to measure the deformation and displacement of engineering structures. By calculating the correlation of gray-scale values between the reference subset and the target subset of a series of images, the deformation information of the region of interest (ROI) on the measured surface can be obtained [20]. Therefore, it is extensively applied in the motion tracking of the target points on bridges to obtain their displacement [21,22,23]. To ensure the measurement accuracy and efficacy, sufficient image stability, contrast for an optimal choice and track of target points should be considered. In addition, in an outdoor environment, the weather and lighting conditions (e.g., fog interference and illumination change, etc.) may also influence the measurement results, and a subpixel level method can be utilized to handle these negative factors [24]. In the vision-based measurement, cameras need to be fixed in a position with sufficient light and keep an appropriate distance from the measured bridge. It is, however, difficult to find an ideal place to deploy cameras for some bridges across rivers and valleys.

Consequently, many scholars use UAVs carrying high-resolution cameras instead of fixed-position cameras for SHM of bridges [25,26,27]. UAVs have been widely used in structural crack detection [28], safety inspections [29], and displacement measurement [30]. Moreover, UAVs have been used for 3D geometry reconstruction to obtain accurate numerical models (e.g., FEM models) [31]. Nevertheless, due to UAV ego-motion, the tracked displacement is the combination of the absolute displacement of bridges and false displacement induced by UAV motion [30]. Hence, the methods to eliminate the false displacement induced by the UAV motion have been proposed [26,30,32,33]. The camera parameters of each UAV video frame are calculated using triangulation [34], so that the world coordinates of the structure can be restored, and the absolute displacement can be obtained [30]. The absolute displacement is estimated by subtracting the UAV movement, which derived from an embedded Inertial Measuring Unit (IMU) [33]. The false displacement is regarded as the low-frequency noise caused by the UAV hovering, which can be filtered away by the high-pass filter method [26]. The planar homography transformation between the images before and after the UAV motion can be determined by direct linear transformation based on at least four sets of 2D-to-2D (two-dimensional) point correspondences [35,36]; according to this algorithm, the false displacement induced by the UAV motion can be corrected [32]. However, this method has a restriction that the reference points on the fixed object plane must be coplanar with the target points, that is, the selected fixed reference objects on the background should be close enough to the measured bridge; otherwise, there will be a significant error. To select reference points far from the target bridge in field measurements, it is necessary to find a way to overcome the above limitation.

In this paper, an improved correction algorithm is developed for eliminating the false displacement induced by the UAV motion by using 3D reconstruction. First, a video of bridge model vibration is captured by a UAV and analyzed by the DIC. Then, the projection matrices of each UAV motion are obtained by camera calibration, and the 3D coordinates of the target points on the bridge model are derived by the projection matrices. Then, the real displacement can be obtained by subtracting the coordinates of the target points on the first frame. The displacement time-histories of the target point are analyzed by the OMA; then, the natural frequencies can be extracted [37]. Finally, the results obtained from the homography-based method and 3D reconstruction method are compared. This work integrates Zhang’s calibration method [38] with a drone to provide a measurement method for bridge vibration. The outcome from this investigation demonstrates the efficacy of the proposed approach for obtaining dynamic displacement of bridges that are inaccessible to fixed-position cameras or sensors.

2. Methods

The general flowchart of this work is shown in Figure 1. The video captured by the UAV can be converted into a sequence of continuous digital images stored in frames. The DIC principle is provided to track the displacement of the points of interest at first; the homography transformation theory is then defined, which can be used to remove the false displacement induced by the UAV motion; the 3D reconstruction method is further presented to remove the false displacement and the results obtained by these two methods are compared.

2.1. Displacement Tracked by DIC

The DIC principle is shown in Figure 2. To track the movement of the measurement point $P(x,y)$ on the image $I(x,y)$, the area of $N\times N$ pixels surrounding it is taken as the reference sub-region (Figure 2a), while the sub-region centered on $Q(x^{\prime },y^{\prime })$ (where $x^{\prime }=x+\Delta x$, $y^{\prime }=y+\Delta y$) on the deformed image $J(x,y)$ is defined as the deformed sub-region; hence, the correlation between the two sub-regions can be expressed as [39]:

$$C\left(\Delta x,\Delta y\right)=\frac{{\displaystyle {\iint }_{S}I\left(x,y\right)J\left(x+\Delta x,y+\Delta y\right)}dxdy}{\sqrt{{\displaystyle {\iint }_S{I}^2\left(x,y\right)}dxdy}\cdot \sqrt{{\displaystyle {\iint }_S{J}^2\left(x+\Delta x,y+\Delta y\right)}dxdy}}$$

(1)

where $S$ is the area of two sub-regions; the real displacement of $P(x,y)$, $(\Delta x,\Delta y)$, maximizes the correlation coefficient $C$ [32].

The pixel values of the points of interest in the process of bridge model vibration can be obtained by the DIC. The tracked displacement by the DIC is the combination of the absolute displacement of the measurement point and the false displacement induced by UAV motion. The next section introduces the homography transformation method to remove the false displacement.

2.2. Homography Transformation Method

Figure 1 shows the procedure of displacement correction based on homography transformation for UAV measurement, which consists of three steps: (1) The DIC principle is utilized to track the pixel values of the measurement points (step 1); (2) Four reference points are used to establish the homography matrix between the original frame and the frame to be corrected (step 2a); and (3) Remove the false displacement by the homography transformation to obtain the real displacement of the target points (step 3a). The above-mentioned steps can be realized by a MATLAB (MathWorks, Natick, MA, USA) program.

As shown in Figure 3, there exists a 2D projection mapping between every two frames of UAV images, which can be expressed as [32]:

$$s\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=H_i\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(2)

where $s$ is a scale factor; $H_i$ $(i=1,2,3\dots ,n-1)$ is called the homography matrix; $u^{\prime }$, $v^{\prime }$ are the pixel coordinates of the reference points on the original frame, and $u$, $v$ are the pixel coordinates of the reference points on the frame to be corrected. At least four reference points are needed to estimate the homography matrix between two frames:

$$\left[\begin{array}{cccccccc}{u}_1& v_1& 1& 0& 0& 0& -u_1{u}_1{}^{\prime }& -u_1{}^{\prime }{v}_1\\ 0& 0& 0& u_1& v_1& 1& -u_1{v}_1{}^{\prime }& -v_1{v}_1{}^{\prime }\\ u_2& v_2& 1& 0& 0& 0& -u_2{u}_2{}^{\prime }& -u_2{}^{\prime }{v}_2\\ 0& 0& 0& u_2& v_2& 1& -u_2{v}_2{}^{\prime }& -v_2{v}_2{}^{\prime }\\ u_3& v_3& 1& 0& 0& 0& -u_3{u}_3{}^{\prime }& -u_3{}^{\prime }{v}_3\\ 0& 0& 0& u_3& v_3& 1& -u_3{v}_3{}^{\prime }& -v_3{v}_3{}^{\prime }\\ u_4& v_4& 1& 0& 0& 0& -u_4{u}_4{}^{\prime }& -u_4{}^{\prime }{v}_4\\ 0& 0& 0& u_4& v_4& 1& -u_4{v}_4{}^{\prime }& -v_4{v}_4{}^{\prime }\end{array}\right]·\left[\begin{array}{c}{h}_{11}\\ h_{12}\\ h_{13}\\ h_{21}\\ h_{22}\\ h_{23}\\ h_{31}\\ h_{32}\end{array}\right]=\left[\begin{array}{c}{u}_1{}^{\prime }\\ v_1{}^{\prime }\\ u_2{}^{\prime }\\ v_2{}^{\prime }\\ u_3{}^{\prime }\\ v_3{}^{\prime }\\ u_4{}^{\prime }\\ v_4{}^{\prime }\end{array}\right]$$

(3)

If there are more than four reference points, the least square method is used to solve the homography matrix.

The first frame of video can be regarded as the original reference frame, and the later frames are used to set up the one-to-one homography relationship with the first frame (Figure 3), separately. The corrected pixel coordinates of a target point are given as:

$$\left[\begin{array}{c}{u}_j{}^{\prime }\\ v_j{}^{\prime }\\ 1\end{array}\right]=H_{j-1}\left[\begin{array}{c}{u}_j\\ v_j\\ 1\end{array}\right]$$

(4)

where $j$ is the frame number; ${\left[\begin{array}{cc}{u}_j& v_j\end{array}\right]}^T$ is the pixel coordinates of the target point on the $j^{th}$ frame, ${\left[\begin{array}{cc}{u}_j{}^{\prime }& v_j{}^{\prime }\end{array}\right]}^T$ is the pixel coordinates of the target point after homography correction.

The real displacement of the target point can be obtained by subtracting ${\left[\begin{array}{cc}{u}_1& v_1\end{array}\right]}^T$ from ${\left[\begin{array}{cc}{u}_j{}^{\prime }& v_j{}^{\prime }\end{array}\right]}^T$; here, ${\left[\begin{array}{cc}{u}_1& v_1\end{array}\right]}^T$ is the pixel coordinates of the target point on the first frame. The unit of displacement is pixels, which can be transferred to physical values in millimeters by multiplying the conversion factor $\eta =l/n$ (where $l$ is the actual length in the real world, $n$ is the pixel number in the image corresponding to the actual length). The homography transformation reflects the relationship between the image plane and the fixed object plane. Therefore, the fixed object plane must be coplanar with the bridge model plane if we want to correct the target point on the bridge model plane. However, it is hard to find a fixed object plane on the same plane as the bridge plane for practical applications.

2.3. Three-Dimensional Reconstruction Method

As shown in Figure 1, the vibration measurement based on 3D reconstruction is composed of three primary steps. Step 1 is to track displacement by DIC which is the same as Section 2.2. Step 2b is the UAV camera calibration, through which the projection matrices of the UAV camera can be estimated. Step 3b is to remove the false displacement induced by UAV motion to obtain the actual displacement by combining the output data from Step 1 and Step 2b.

2.3.1. Camera Calibration

Let ${[X_w,Y_w,Z_w]}^T$ be the coordinates of a point in the 3D world coordinate system and ${[u,v]}^T$ the 2D projection of the point on the image. The relationship between the 3D world coordinates and 2D pixel coordinates can be written as [38]:

$$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left[\begin{array}{cc}R& t\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(5)

where $s$ is an arbitrary scale factor; $R=\left[\begin{array}{ccc}{r}_1& r_2& r_3\end{array}\right]$ is the rotation matrix and $t={[\begin{array}{ccc}{t}_1& t_2& t_3\end{array}]}^{{\rm T}}$ is the translation vector, $\left[\begin{array}{cc}R& t\end{array}\right]$ represents the extrinsic matrix that converts the world coordinate system to the camera coordinate system; and $A$ is the intrinsic matrix which is given by:

$$A=\left[\begin{array}{ccc}{f}_x& \gamma & u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]$$

(6)

where $f_x$ and $f_y$ are the scale factors in the $x$ and $y$ axes of the image plane, respectively; $u_0$, $v_0$ are the pixel coordinates of the image center; $\gamma$ is the skewness parameter.

According to Zhang’s calibration method [38], the solution of the camera projection matrix (camera calibration) consists of three parts: (a) Intrinsic parameter calibration; (b) Extrinsic parameter calibration; and (c) Minimizing the reprojection error.

By taking more than three pictures of the chessboard with different views, the intrinsic matrix $A$ can be solved by Zhang’s method. The intrinsic parameters of the UAV camera are invariable, and the calibration algorithms are available in the Camera Calibrator app of MATLAB.

The extrinsic matrix represents the position and orientation of the UAV camera for different moments. As shown in Figure 4, the upper left corner of the fixed object plane can be set as the origin of the world coordinate system, the horizontal direction is the $X_w$ axis, and the vertical direction is the $Y_w$ axis. Once $A$ is known, the UAV extrinsic matrix can be computed by at least four sets of world coordinates and their corresponding image pixel coordinates.

During calibration, the extrinsic parameters are estimated numerically by minimizing the reprojection errors for all calibration images, which is:

$${\displaystyle \sum _{i=1}^n{\displaystyle \sum _{j=1}^m\parallel m_{ij}-\stackrel{⌢}{m}(A,R_i,t_i,P_j)}}{\parallel }^2$$

(7)

where $n$ is the number of frames. $m$ is the number of points. $m_{ij}$ represents the $j^{th}$ detected point in frame $i$; $\stackrel{⌢}{m}(A,R_i,t_i,P_j)$ is the projection of world point $P_j$ in frame $i$, using the estimated $R_i$ and $t_i$.

The solution of the extrinsic matrix is completed by applying the extrinsic function in the Computer Vision Toolbox of MATLAB. Once the matrices $R_i$ and $t_i$ $(i=1,2,3\dots ,n)$ for each UAV frame have been calculated, the projection matrices can be estimated by combining the intrinsic matrix $A$. The details of 3D reconstruction using a projection matrix will be introduced in the next section.

2.3.2. Recovering the 3D Coordinates

The projection matrix $M$ can be obtained by combining the intrinsic and extrinsic parameters as follows:

$$s\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)=\left[\begin{array}{cccc}{m}_{11}& m_{12}& m_{13}& m_{14}\\ m_{21}& m_{22}& m_{23}& m_{24}\\ m_{31}& m_{32}& m_{33}& m_{34}\end{array}\right]\left(\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right)$$

(8)

Equation (8) can be written in the following form:

$$\begin{array}{c}{m}_{11}{X}_w+m_{12}{Y}_w-su=-m_{14}-m_{13}{Z}_w\\ m_{21}{X}_w+m_{22}{Y}_w-sv=-m_{24}-m_{23}{Z}_w\\ m_{31}{X}_w+m_{32}{Y}_w-s=-m_{34}-m_{33}{Z}_w\end{array}$$

(9)

which leads to:

$$\left[\begin{array}{ccc}{m}_{11}& m_{12}& -u\\ m_{21}& m_{22}& -v\\ m_{31}& m_{32}& -1\end{array}\right]\left(\begin{array}{c}{X}_w\\ Y_w\\ s\end{array}\right)=-\left(\begin{array}{c}{m}_{14}\\ m_{24}\\ m_{34}\end{array}\right)-Z_w\left(\begin{array}{c}{m}_{13}\\ m_{23}\\ m_{33}\end{array}\right)$$

(10)

Then, the 3D world coordinates of the target points on the bridge can be calculated by:

$$\left(\begin{array}{c}{X}_w\\ Y_w\\ s\end{array}\right)={\left[\begin{array}{ccc}{m}_{11}& m_{12}& -u\\ m_{21}& m_{22}& -v\\ m_{31}& m_{32}& -1\end{array}\right]}^{-1}\beta$$

(11)

where $\beta =-\left(\begin{array}{c}{m}_{14}\\ m_{24}\\ m_{34}\end{array}\right)-Z_w\left(\begin{array}{c}{m}_{13}\\ m_{23}\\ m_{33}\end{array}\right)$, $Z_w$ is the vertical distance between the fixed object plane and the bridge model plane. Finally, by substituting the pixel coordinates $u$ and $v$ of the target points into Equation (11), the 3D world coordinates of the points can be recovered. The real displacement of the target points can be calculated from the subtraction between $Y_{wi}-Y_{w1}$ (where $i$ is the frame number, $Y_{w1}$ is the value of the original coordinate).

Ultimately, the approach mentioned above is compared with the homography transformation approach.

2.4. Operational Modal Analysis (OMA)

The Response Transmissibility (RT) is commonly used to identify the modal parameters of a structure in the OMA theory. The RT, $T_{io}\left(\omega \right)$, between the degrees of freedom, $i$ and $o$, can be defined as [40]:

$$T_{io}\left(\omega \right)=\frac{X_i\left(\omega \right)}{X_o\left(\omega \right)}$$

(12)

where $X_i\left(\omega \right)$ and $X_o\left(\omega \right)$ are the Fourier transforms of $x_i\left(t\right)$ and $x_o\left(t\right)$ which represent the response time-histories at the degrees of freedom $i$ and $o$, respectively.

The Power Spectral Density Transmissibility (PSDT), ${\widehat{T}}_{io}$, is commonly used for modal identification [41]:

$${\widehat{T}}_{io}\left(\omega \right)=\frac{S_{i,o}\left(\omega \right)}{S_{o,o}\left(\omega \right)}=\frac{X_i\left(\omega \right)X_o^{\ast }\left(\omega \right)}{X_o\left(\omega \right)X_o^{\ast }\left(\omega \right)}=T_{io}\left(\omega \right)$$

(13)

where $X_o^{\ast }\left(\omega \right)$ is the conjugate of $X_o\left(\omega \right)$ and $S_{o,o}\left(\omega \right)$ is the Power Spectral Density (PSD) of $x_o\left(t\right)$$x_o\left(t\right)$, while $S_{i,o}\left(\omega \right)$, which is the Cross Power Spectrum (CPS) for $x_i\left(t\right)$ and $x_o\left(t\right)$.

In summary, the natural frequency can be determined by picking the peaks in the PSD plots of the response of a degree of freedom [42].

3. Experiments

3.1. Experimental Setups

The bridge model has 28 spans, and its total length is 9.8 m (Figure 5a). Each span has dimensions of 0.35 m × 0.35 m × 0.35 m. The model components include bolted balls and rods (Figure 5b). The yellow rod has a length of 0.35 m, and the red rod has a length of 0.5 m. The truss is simply supported at both ends.

The quadrotor UAV (Spark, Da-Jiang Innovations, Shenzhen, China) used has a camera (Figure 6a) with a resolution of 1920 × 1080 pixels and a recording rate of 30 frames/second.

The fixed object is a rectangular steel frame as in Figure 6b. The four corner points at the rectangular frame plane are used as the reference points. For the homography-based correction approach, the fixed object plane should be as close to the bridge model as possible, but there is no such request for the 3D reconstruction approach.

3.2. Experimental Schemes

Figure 7 shows the four reference points (red points on the rectangular frame) and the target point (the yellow point on the bridge model). In the experiment, the bridge model is excited by an arbitrary force.

For the homography transformation approach, the four reference points can be used to establish the homography relationship between the original frame and the frame to be corrected.

For the 3D reconstruction approach, the fixed rectangular frame is the Xw–Yw plane of the world coordinate system (Figure 8). The top left corner point is its origin. The world coordinates of the four reference points are $(0.0,0.0,0.0)$, $(500.0,0.0,0.0)$, $(0.0,500.0,0.0)$, and $(500.0,500.0,0.0)$, respectively, and their corresponding pixel coordinates obtained from the first frame are $(577.1,147.0)$, $(1245.2,165.1)$, $(559.2,806.9)$, and $(1228.2,804.9)$, separately. Using the four sets of coordinate values and the intrinsic matrix $A$, R₁ and $t_1$ can be computed using the extrinsic function of MATLAB.

Similarly, the pixel coordinates of the reference points from the $2^{nd}$ to the $n^{th}$ frames can also be utilized to calculate the extrinsic matrices $R_i$ and $t_i$ for each frame of images. Ultimately, the projection matrices $M_i$ $(i=1,2,3\dots ,n)$ for each frame can be estimated.

In order to study the influence of the distance between the fixed object and the bridge model on the measurement accuracy, three working conditions are set up: (1) the reference points (on the fixed object plane) are coplanar with the target points, that is, the distance between the fixed object and bridge model is 0.0 m. In addition, the distance between the UAV and the bridge model is 3 m (Figure 9a); (2) the fixed object is 1 m in front of the bridge model, and the UAV-bridge model distance is 5 m (Figure 9b); (3) the fixed object is 1 m behind the bridge model, and the UAV-bridge model distance is also 3 m (Figure 9c).

4. Results

4.1. Correction through 3D Reconstruction

The six-degrees-of-freedom (6 DOF) UAV motion, including three DOF rotations and three DOF translations, can be obtained after the camera calibration (Figure 10). It represents the movement of the UAV camera around the three axes of the world coordinate system. During the experiment, the bridge model is excited five times; hence, there will be five peaks in the displacement curve.

The displacement time–history curves (y-direction) of a reference point (the top left corner point) before and after correction are shown in Figure 11. Since it is a fixed point, its corrected displacement curve is almost a straight line along the x-axis. The uncorrected displacement of the target point collected by the UAV is presented in Figure 12, which contains the false displacement induced by UAV motion. The real displacement of the target point (y-direction) obtained by the 3D reconstruction method is also shown in Figure 12.

The OMA is used to analyze the modal parameters of the corrected displacement curve of the UAV, and the result is shown in Figure 13. It can be seen from Figure 13 that the first natural frequencies obtained by the two methods are 3.940 Hz, which demonstrates that the OMA successfully extracts the modal parameters from the 3D reconstruction-based displacement data.

4.2. Comparison of Homography Transformation and 3D Reconstruction

Figure 14 shows the displacement obtained by the two correction methods. The displacement curves are extremely similar to each other, and their maximum amplitudes are about 9 mm. Therefore, for the case that the reference points and the target point are coplanar ($Z_w=0$ mm), the results of the homography transformation method and the 3D reconstruction method are consistent.

The following context will discuss the case in which the reference points and the target point are in different planes.

By moving the fixed objected 1 m towards the UAV camera, the reference points and the target point are non-coplanar ($Z_w=1000$ mm). Figure 15 shows the 6 DOF UAV motion in this case. As demonstrated in Figure 16, the correction result of the homography transformation is entirely unacceptable, while the 3D reconstruction method is still effective. As the displacement PSD shows in Figure 17, the first natural frequency obtained by the proposed method is 3.927 Hz, which has a relative error of 0.33% compared to 3.940 Hz, while the one obtained by the homography-based method is 0.103 Hz.

By moving the fixed object 1 m behind the bridge model, the distance between the plane of reference points and the plane of the target point is $Z_w=-1000$ mm. Figure 18 shows the 6 DOF UAV motion in this case. The correction results of the two methods are also compared (Figure 19). In this case, the homography transformation method fails again, while the 3D reconstruction method is still effective. As shown in Figure 20, the obtained first natural frequency by the proposed method is 3.966 Hz, which has a relative error of 0.66% compared to 3.940 Hz, while the one obtained by the homography-based method is 0.044 Hz. Finally,

Table 1. Frequencies obtained by two methods.

Zw (m)	Homography	3D Reconstruction	Homography	3D Reconstruction
	Frequency (Hz)	Frequency (Hz)	Relative Error (%)	Relative Error (%)
0	3.940	3.940	0	0
1	0.103	3.927	97.4	0.33
−1	0.044	3.966	98.9	0.66

shows the frequencies obtained by two methods.

5. Discussion and Conclusions

5.1. Discussion

Our experimental results suggested that the two methods work well in the situation that the reference points are on the same plane as the target points. However, for the situation that the reference points and the target points are in different planes, the homography-based correction method is invalid, while the 3D reconstruction is still feasible. As the plane homography is defined as a projection mapping from one plane to another, the homography matrix calculated from the fixed object plane and image plane cannot be used for the points on the third plane. This is why the homography transformation fails if the fixed object plane is non-coplanar with the bridge model plane.

Although the proposed method shows a promising measurement result, there are still several problems that should be addressed in the future study:

• For the monocular camera, the value of $Z_w$ is assumed to be constant, that is, the out-of-plane displacement ($Z_w$-direction) is ignored. In some structures with mainly in-plane displacement, the small change of $Z_w$ can be ignored. Nevertheless, there will be an obvious error for the structures with large out-of-plane displacement if the change of $Z_w$ is ignored. The structure-from-motion (SFM) technique can restore a bridge’s 3D model coordinates [31], including the Z_w-direction, by processing high-resolution stereo-photogrammetric photos; it has been used for slow deformation monitoring. An image splitter system, which consisted of four fixed mirrors, is used to mimic four different views by using a single camera with a 45-degree horizontal angle with respect to the target [43]. However, it needs a large splitter and mirror to measure large bridges from a sufficient distance. Using two UAVs combined with a binocular vision principle to measure three-dimensional displacement needs further investigation.
• The experiment is carried out in a laboratory environment. Its conditions, including light, weather, reference points, etc. are in the ideal state. It is, however, inevitable that some negative factors may occur in the real bridge measurement, for example, difficulty in finding fixed reference objects. An artificial fixed object needs to be deployed under this situation. It is expected that this method can be realized in the real bridge measurement in the near future.
• In the current algorithm, the theory of planar homography is used to calculate the camera extrinsic matrix R and t. Hence, the four fixed reference points must be on the same plane with fixed-Z_w. However, in some measurement circumstances, it is hard to guarantee that all four points are coplanar. Whether or not the reference points can be on different planes needs further study.

5.2. Conclusions

In this paper, we have described a 3D reconstruction correction method for bridge vibration measurement using a UAV. Through the UAV camera calibration, the projection matrix for each frame of images can be estimated, and the actual displacement of the bridge model can be obtained by recovering the 3D world coordinates using the projection matrices. With the development of computer vision theory and UAV technology in the future, the UAV-based method may play an important role in vibration measurement and damage detection of bridges. The above investigation suggested that:

The proposed method can estimate the intrinsic and extrinsic parameters (6 DOF motion) of the UAV camera using Zhang’s method, and then recover the 3D world coordinates of the target points through the projection matrices. The natural frequencies obtained by this method are consistent with the homography-based method. By using a fixed object on the background as the reference, the proposed method can be used to effectively remove the false displacement caused by the UAV motion.

To further confirm the applicability of the proposed approach for bridge vibration measurement, different positions of the fixed object are set up. The experimental results demonstrated that the proposed method is more applicable than the homography-based correction method. In practical measurement, it avoids the limitation that the reference points have to be coplanar with the bridge; they can be artificially arranged in any plane parallel to the bridge surface. Regardless of whether the fixed object is close enough to the bridge, the 3D reconstruction can be carried out as long as the distance between them is known, while the measurement results are more accurate for the short distance between the fixed object and the bridge.