Computer Vision-Based Bridge Displacement Measurements Using Rotation-Invariant Image Processing Technique

Abstract

Bridges are exposed to various kinds of external loads, including vehicle and hurricanes, during their life cycle. These loads cause structural damage, which may lead to bridge collapse. To ensure bridge safety, it is essential to periodically inspect the physical and functional conditions of bridges. The displacement responses of a bridge have significance in determining the structural behaviors and assessing their safety. In recent years, many researchers have been studying bridge displacement measurements using image processing technologies. Image-processing-based displacement measurements using a computer analysis system can quickly assess bridge conditions and, thus, can be used to enhance the reliability of bridges with high accuracy. This paper presents a method based on multiple-image processing bridge displacement measurements that includes enhanced robustness to image rotation. This study applies template matching and a homography matrix to measure the displacement that works well regardless of the angle between the smartphone camera and the target.

1. Introduction

In modern society, bridges serve as the backbone of a nation’s economy as they connect separated regions and provide effective means of transportation to accommodate the needs of populations experiencing explosive growth. However, severe weather can badly damage bridge structures, reducing their life and safety. Therefore, systematic inspections and maintenance must be performed to maintain the optimum conditions of bridges. A bridge’s displacement responses provide valuable information for structural analysis to assess bridge safety [1,2] and performance. Thus, the monitoring of displacement characteristics is a critical issue for the safety and maintenance of bridges. Displacement response measurements in bridges are normally done using high precision sensors, such as linear-variable differential transformers (LVDTs) [3], accelerometers [4], global positioning systems (GPSs). However, such conventional methods have limitations [5,6,7]. For instance, an LVDT or accelerometer must be installed with additional equipment, such as data loggers (including cables and power supply equipment), which can introduce inconvenience and extra monitoring-cost for large-scale bridges and structures. Moreover, bridge monitoring based on acceleration measurements is relatively an effortless approach, as it determines the displacement by simply double integrating the acceleration values, however, the biased initial conditions may create unstable results [8]. On the other hand, the GPS-based method has low precision of vertical displacement measurements [9]. The computer vision-based techniques offer significant advantages over conventional sensing systems for measuring structural responses such as (i) computer vision-based techniques can measure displacements of multiple points from a motion image captured by a single camera; (ii) the computer vision-based technique does not require physical access to the structures, as the vision sensors (such as cameras) can be setup at a remote location and can easily alter the measurement points; (iii) computer vision-based techniques can be placed hundreds of meters away from the bridge (when using a zoom high resolution lens) and still achieve satisfactory measurement accuracy. Thus, the computer vision method can effectively reduce time and cost compared to previous sensor-based methods. In past structural inspections, computer vision technology was often used to detect cracks [10]. Crack inspection research focuses primarily on techniques that accurately analyze the length, width, and depth of cracks [11]. Recently, computer vision technique has gradually expanded in the field of dynamic response measurements for various structures, including buildings and bridges [12]. For example, Stephen et al. [13] presented an image-tracking system to measure bridge–deck displacement. This image tracking system was successfully applied at Humber Bridge in the UK. Olaszek [14] utilized photogrammetric principle and introduced a scheme to measure displacement and dynamic characteristics of bridges. Guo [15] proposed computer vision technology including a Lucas–Kanade template tracking algorithm to measure the dynamic displacement of large-scale structures. Shan et al. [16] proposed bridge deflection monitoring technique based on Zhang’s [17] camera calibration method. Feng and Feng [18,19] proposed structural modal property identification method utilizing computer vision technique. Similarly, Yoon et al. [20] proposed a computer vision-based method using consumer-grade cameras to identify the dynamic characteristic of structures.

Computer vision-based systems for structural displacement measurements mainly rely on template matching [21]. Template matching approach is an image processing technique to determine the locations of an image matching a predefined template image. To match image, the template image is compared against the source image by moving it one pixel at a time. At each location, a correlation-coefficient is computed indicating the similarity between template image and a specific area of the source image. As all the possible locations of template image are considered and compared with source image, therefore, the area with highest correlation-coefficient becomes the target position. The displacement measurement using the digital image correlation (DIC) approach is a computer vision technology that employs a digital camera and image processing for accurate measurements of changes in images. The use of the DIC method to measure displacements of objects was initially proposed by researchers at the University of South Carolina [22,23,24]. The DIC technique has shown wide scale powerful and flexible applications in the field of experimental solid mechanics, particularly for measuring the surface deformations [25]. Moreover, it has been studied and applied by several researchers for structural displacement measurement [5,26,27]. In recent years, the displacement measurement method using DIC has been realized as a simple system because it is an image-processing technique correlating the original image and deformation image. Therefore, it has shown high applicability to the various devices. In object tracking, DIC is not an ideal technique as it can be easily affected by image scale and rotation [28]. In order to overcome the limitations of template-based matching using DIC technique, Jeong et al. [29] proposed a pan-tilt-zoom (PTZ) camera-based computer vision system using a sum of squared differences (SSD) and homography-matrix [30], this system is capable of expanded measurable range in monitoring structural displacement. Till present, several correlation-coefficient functions have been introduced including normalized cross-correlation (NCC), sum of absolute differences (SAD), sum of squared differences (SSD), zero-mean normalized cross-correlation (ZNCC), zero-mean sum of absolute differences (ZSAD), zero-mean sum of squared differences (ZSSD), optimized sum of absolute difference (OSAD), and optimized sum of squared difference (OSSD) [31,32]. Among these, NCC, SAD, SSD, and ZNCC have been widely adopted in template matching from the field of objects tracking [32]. The SSD is a popular method for many applications, including template-based matching, due to its efficient computational schemes. However, it is sensitive to illumination changes [33]. On the other hand, NCC has robustness advantages against illumination change. In addition to, NCC is a very simple approach for determining multiple patterns from a source image. Thus, the NCC function is mainly used to measure the displacement of the structure in the civil engineering [34,35].

Present study introduces a computer vision method for bridge displacement measurement, enabled by the robust template matching supported by homography-matrix and NCC function. Additionally, for motion image capture, smartphone camera was used instead of the conventional commercial-scale expensive cameras. The adoption of smartphone would allow the quick safety inspection of bridge because of being cost-effective, mobile, and ease of installation.

2. Literature Review

Jauregui et al. [36] successfully measured the vertical deflections of bridge structures by implementing close-range terrestrial photogrammetry approach. For highly accurate displacement measurements, a sub-pixel edge detection algorithm was introduced by Poudel et al. [37]. Fu and Moosa [38] utilized high resolution CCD camera to develop a cost-effective and efficient technique for the measurement of displacement in civil structures. A vision-based system for the accurate monitoring of absolute displacement was introduced by Wahbeh et al. [39]. The architecture of this system is simply comprised of a LED, a camera, and a computer. Through experiments on the Vincent Thomas Bridge in California, researchers demonstrated the feasibility of this approach for measuring bridge displacement. Kohut et al. [40] conducted a comparative research on radar interferometry and vision-based methods to measure the displacements in civil structures. They demonstrated the precision of vision-based techniques by implementing them on a steel bridge; moreover, they accurately determined the bridge displacement. Lee and Shinozuka [5,41,42] developed an image processing system which is digital enough to monitor and measure the dynamic displacements of flexible bridges in real-time. This system was also shown to have the ability to assess the load carrying capacity of a steel girder bridge located in Korea. Fukuda et al. [6] introduced a vision-based displacement measurement system that utilizes a low-cost digital camcorder and a PC. Similarly, Choi et al. [43] presented a structural dynamic displacement system that could perform multi-measurements using an economical handheld digital camcorder. Ho et al. [44] developed a synchronous vision system implementing an image-processing approach to measure the dynamic displacement of civil infrastructures. This system can simultaneously support several camcorders for real-time multipoint displacement measurements. Tian and Pan [45] proposed a vision-based displacement measurement device using LED targets for real-time measurements of bridge deflection. Yoneyama et al. [27] obtained bridge deflection measurements using a digital image correlation method for bridge load tests. Busca et al. [46] combined pattern matching, digital image correlation, and edge detection and developed a vision-based technique for the monitoring of dynamic response of bridge.

A remote vision-sensor system in combination with advance template matching, to measure the displacement in civil infrastructures was proposed by Feng et al. [47]. Ribeiro et al. [7] introduced a video system based on the RANdom SAmple Consensus (RANSAC) algorithm. This algorithm accurately determines the dynamic displacement of railroad bridges. Feng et al. [48] developed a markerless vision-based system using an advanced orientation code-matching (OCM) algorithm to measure the dynamic displacement of bridges. Jeon et al. [49] presented a paired visual servoing system to measure the six degrees of freedom (6-DOF) displacement of high-rise buildings and long-span bridges. Similarly, Park et al. [50] measured the three-dimensional displacement associated with structures with the utilization of a capture system. This system is capable of high sample rate for the accurate measurement of displacement. Chang and Xiao [51] used a single-camera based video-grammetric approach to measure both three-dimensional translation in concurrence with the rotation of the planer marker installed on the structure. Ji and Chang [52] proposed a novel vision-based method that used an optical flow technique to eliminate the use of a marker. This system used a camcorder to measure the vibrational responses of a small pedestrian bridge. In another study, Ji and Chang [53] presented a markerless stereo vision technique to monitor the dynamic responses of line-like elements, such as cables, in both spatial and temporal domains. Kim and Kim [54] proposed a markerless vision technique that utilized digital image correlation obtained from a portable and digital camcorder to measure hanger cable tensions of a suspension bridge. Kim [55] proposed a multi-template matching method that rectifies the errors occurring during marker recognition as noise in after-images. This noise is usually caused by the camera shaking and by high-speed motion of the cables. A digital photogrammetry method [56] using multiple camcorders to measure the synchronized ambient vibrations at various points of a stay cable bridge has been proposed, this approach effectively identifies the mode shapes ratios. For the quick inspection of the cable supported bridge, Zhao et al. [57] conducted an experiment on the feasibility and utilization of the computer vision-based cable force estimation method using a smartphone camera. They used a D-viewer and iPhone 6 to measure the vibration characteristics of the cable. Based on a previous study [57], Zhao et al. [58] proposed a computer vision-based approach for identifying the natural frequencies of cable and estimating cable force using a smartphone camera.

3. Computer Vision-Based Bridge Displacement Measurement

3.1. Overview

Figure 1 shows an overview of the proposed computer vision-based displacement measurement methodology. This system consists of a marker, a smartphone camera, and a computer installed with an image processing software. The connection between the smartphone and the computer is conducted by a wireless communication or cable. The proposed computer vision-based displacement measurement method is composed of three main phases: (1) motion image acquisition; (2) image processing for object tracking; and (3) calculation of bridge displacement. The smartphone camera is fixed on a tripod and is located at about 1 m away from the marker. Installation of smartphone camera and its focusing is time-effective, as it takes a few minutes. The smartphone camera gets the motion images of the marker for displacement measurement, and computer analyzes the marker’s movement and bridge displacement through image processing software. This image processing relies on the homographic matrix for the planar projections of the captured motion images, and template-based matching algorithms using on the NCC function for object tracking.

3.2. Camera Calibration

Accurate calculation of bridge displacement based on measurements from motion images requires the correction of camera distortion. Generally, camera calibration is the most crucial stage for a vision-based system. Because the proposed system relies on motion images from a camera to determine displacement, the accuracy is highly influence by the precision calibration of the camera. In this regard, to eliminate distortion from the camera lens, Zhang [17] proposed a camera calibration algorithm based on the 2D planar pattern. In the present study, the inherent and extraneous characteristics of camera, and distortion coefficient of camera lenses were estimated using Zhang’s calibration algorithm (for more details see [17]).

3.3. Template-Based Matching Method

Among the object tracking methods, the template-based matching technique is commonly applied specifically for the measurement of structures. The template-based matching technique is normally implemented by first registering a part of the search image to use as a template image (such as marker). Because the template image is registered, an object can be found in the search image using NCC [28] function. The NCC is a sub-class of DIC is one of the most promising approach to analyze the resemblance of two individual images base on their cross-correlation. The NCC coefficient is defined by Equation (1). The basic concept of the template-matching technique is shown in Figure 2.

$$\mathsf{\gamma }\left(u,v\right)=\frac{{\sum }_{x,y}\left\{f\left(x,y\right)-\overline{f}\right\}\left\{t\left(x-u,y-v\right)-\overline{t}\right\}}{\sqrt{{\sum }_{x,y}{\left\{f\left(x,y\right)-\overline{f}\right\}}^2{\sum }_{x,y}{\left\{t\left(x-u,y-v\right)-\overline{t}\right\}}^2}}$$

(1)

• $\mathsf{\gamma }\left(u,v\right)$ = correlation coefficient
• $f\left(x,y\right)$ = intensity value of search image at point $\left(x,y\right)$
• $\overline{f}$ = mean value of $f\left(x,y\right)$ within the region of the template
• $\overline{t}$ = mean value of the template

3.4. Homography-Matrix

For precise displacement measurements, an image transformation function should be applied to eliminate geometric distortions. Therefore, the present method includes the homography approach [30] to eliminate the geometric distortions of image scale and rotation. Homography simultaneously maps two planes by translating a point P from the x-y plane to a point P` in the newly-projected plane x’-y’. As shown in Figure 3, the x-y plane points P1(x1, y1), P2(x2, y2), P3(x3, y3), and P4(x4, y4) are relocated as points P1’(x1’, y1’), P2’(x2’, y2’), P3`(x3’, y3’), and P4’(x4’, y4’), respectively, on the x’-y’ plane. These newly oriented points are located at distance R (mm) from the origin. The distances of points P1’, P2’, P3’, and P4’ have the same attributes as the origin, and the intersection of each point occurs at 90°; thus, the orthogonality for each new point is maintained. Conclusively, it can be said that, in homography, the standard plane is the distortion-corrected plane in units of millimeters. The homography shows the relationship between points in the form of a 3 × 3 matrix, as illustrated by Equation (2). This relationship remains valid, not only for the direct projection of two planes, but for every plane that is directly or indirectly projected onto a flat plane.

$$\left[\begin{array}{c}{x_i}^{\prime }\\ {y_i}^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right],i=1,2,\dots$$

(2)

3.5. Scaling Factor

To obtain the accurate displacement response from the image, it is mandatory to transform the pixel coordinates and physical coordinates. Therefore, a scaling factor is required to determine the transformation coefficient. This scaling factor (mm/pixel) defines as the transformation coefficient between the units on image plane and the physical units. When the image plane is parallel to the object surface, the scaling factor can be calculated using Equation (3).

$$\mathrm{Scaling}\mathrm{factor}=\frac{\mathrm{physical}\mathrm{units}\left(\mathrm{such}\mathrm{as}\mathrm{marker}\mathrm{length}\right)}{\mathrm{pixel}\mathrm{units}\mathrm{on}\mathrm{the}\mathrm{image}\mathrm{plane}}$$

(3)

3.6. System Implementation

In this study, we developed software to accurately estimate bridge dynamic displacements based on the template-based matching technique and a homography-matrix. This software is operated on a Windows platform, and it is performed in a user-friendly graphical user interface (GUI). The programming tool for development the image processing software was used Visual Studio 2012 using C++ language. As shown in Figure 4, the developed software can import the motion images acquired from the smartphone camera, register the template image directly using the mouse on the video screen. Also, region of interest (ROI) can be set in the same way. The scaling factor is easily changed by inputting the length of the used marker. The software has been developed to fit the Galaxy S7 Samsung smartphone. However, it can be easily adapted to other smartphone models by changing the focal length through coding. In real time, the users can view the displacement curve through the chart screen. In addition, a fast Fourier transform (FFT) was applied to calculate the natural frequency of the bridge from displacement data. This software is a stand-alone executable program helpful for the measurement of bridge dynamic displacement. Through this software, the bridge dynamic displacements can be obtained easily by recognizing the movement of the pre-designed marker attached on the target using smartphone camera without expensive displacement sensors.

4. Experimental Validation

4.1. Laboratory Scale Tests

Initially, the performance of proposed computer vision-based bridge displacement measurement technique was evaluated at laboratory scale. In these experimental trials, specially design test equipment was used as shown in Figure 5. A predesigned black and white marker (150 mm × 150 mm) acting as tracking target, was fixed on computer numerical controlled (CNC) movement axis of machine. The linear repeated movement of the axis was fixed at 20 cm from origin point. To evaluate the proposed displacement measurement technique based on homographic matrix, motion images of marker attached movement axis were captured by successively placing the smartphone camera at angles of 0°, 10°, 20°, and 30° with reference to front of marker (shown in Figure 6). The marker’s motion images captured by the smartphone camera were recorded at 30 frames per second. The smartphone camera was placed at 1 m from the marker front side so that appropriate installation like field settings without any zoom can be tested. In these experiments, there was no need to install reference sensor to measure the displacement, as the marker is attached CNC movement axis. Thus, the CNC coordinates are more accurate compare to sensor readings and can easily be obtained from the control program. The marker’s motion images were recorded and transmitted to computer over wireless communication. The marker displacement readings obtained from the control program were regarded as reference-value and the errors of the proposed approached at various rotation angles were compared. We used Samsung Galaxy S7 smartphone for experimental trials.

Table 1. The hardware configuration of Computer Vision System.

Types	Models	Specifications
Marker	-	- Two white rectangles with dark background - Length: 150 mm × 150 mm
Smartphone Camera	SM-G930S	Image sensor: Sony IMX260 UHD 4K (3840 × 2160) resolution @30 fps Aperture: f/1.7 Pixel size: 1.4 µm Focal length: 4.20 mm
Computer	Samsung- NT870Z5G	Intel(R) Core(TM) i7-4720HQ CPU 2.60 GHZ 8.00 GB RAM Windows 10 (64-bit)

summarizes the hardware specifications of computer vision-based system.

Figure 7, Figure 8, Figure 9 and Figure 10 show the measurement results of the displacement at different angles. We conducted error evaluation based on the root mean squared error (RMSE; Equation (4)) to quantify the measurement errors.

Table 2. Measurement errors (root mean squared error (RMSE)).

	0°	10°	20°	30°
RMSE	0.037	0.371	0.400	0.398

provides the calculated RMSE values for the bridge displacements.

$$\mathrm{RMSE}=\sqrt{\frac{{\sum }_{i=1}^n{\left(x_{1,i}-x_{2,i}\right)}^2}{n}}$$

(4)

• $x_{1,i}$ = ith set of bridge displacement data at time $t_i$ measured using the proposed method
• $x_{2,i}$ = ith set of bridge displacement data at time $t_i$ measured using LVDT
• n = number of data

The test results show that the proposed method provides a small change according to the angle between the smartphone camera and the marker. When the angle of the smartphone camera and the marker is 0°, RMSE value is 0.037 mm that is similar to the reference value. Although the error is increased by 0.371 mm when the angle is 10°, the magnitude of the error is sufficient to replace the existing displacement sensor. Also, RMSE value is 0.400 mm and 0.398 mm for the angle of 20° and 30°, respectively, that did not show any significant different compare to the angle of 10°. The proposed method maintains a minor error even when the angle increases that can measure accurate displacement of bridge in various range of the angle.

4.2. Field Test

The further experimental verification of proposed system has been performed field test. Like laboratory experiments, the computer vision system for the field test comprised a marker, a smartphone camera (Galaxy S7), and a computer installed with image processing software. In real field experiment, to reduce data transfer time, smartphone and computer were connected using data cable. The marker was an acrylic plate glued to metallic-plate to prevent deformation (Figure 11). The field tests were performed at Seongdong Bridge located in Seoul. In this field experiment, the displacement values obtained from the proposed vision-based approach were compared with conventional displacement sensors (LVDT). The acrylic plate marker was attached to the bridge girder using adhesives. The distance between smartphone camera and marker was set at 1 m and the angle was 0°. The motion images were recorded at 30 fps and transmitted to the computer. In field experimental setup of the proposed vision-based system is shown in Figure 11.

In the field test, two factors affecting measurement errors should are considered. The first factor to consider is the shaking of the smartphone camera caused by the wind, and the second factor is the influence of vibration around the smartphone camera due to vehicle traffic. Thus, a windless sunny day was selected for the field experimentation to minimize the errors and the trials were performed at the ambient vibration conditions. Using these experimental procedures, we obtained the dynamic displacements-time curve of the Seongdong Bridge by the proposed method and the LVDT. During the two-day test, we measured four sets of displacement data at the same point. The authors compared the displacement data measured by the proposed method and the measured values by the LVDT sensor. As shown in Figure 12, the measurements obtained using the proposed displacement measurement approach agree well with those from the conventional displacement sensor (LVDT).

Similar to the laboratory experiments, error analysis was performed using the RMSE of Equation (4) to quantify the accuracy of proposed method.

Table 3. Measurement errors (RMSE).

	(a)	(b)	(c)	(d)	Max.	Min.
RMSE	0.044	0.051	0.040	0.050	0.051	0.040

provides the calculated RMSE values that the values were slightly increased compared to the laboratory test. However, the proposed method demonstrated high measurement accuracy with a maximum RMSE value of 0.051 mm in the field. The proposed approach uses motion images obtained by portable devices, such as smartphone camera, to measure the bridge displacement. This technique requires no additional equipment and allows measurements at multiple locations in shorter time than when using conventional methods. Thus, the proposed method can effectively reduce monitoring time and cost compared to the method using conventional displacement sensor.

5. Limitations and Future Research

The accuracy of computer vision-based displacement measurement technique can even compete with conventional displacement sensors. It is also more efficient in terms of sensor installation cost and working time. The accuracy and efficiency of computer vision-based method has been verified by researchers. However, computer vision-based system requires a large-capacity server for data storage and database of images for the long-term monitoring as compared to existing sensors systems. Moreover, the camera performance and image processing algorithm significantly influence the quality of displacement data. In recent years, even though, several displacement measurement algorithms have been developed, yet it is difficult to find a highly suitable algorithm for structures monitoring. This limitation bounds the utilization for entire civil infrastructure. Another, drawback of computer vision approaches is the possibility of errors due to noise caused by the vibrations around the camera. The utilization of vibration control device such as gimbal may solve this problem.

6. Conclusions

The main objective of this study is to extend the measurable range and to enhance the accuracy of displacement measurement, so that computer vision-based systems can easily be adopted for the civil engineering structures (such as bridges). In this study, we introduced a computer vision-based technique for the measurement of rotation-invariant displacement with the successful utilization of a smartphone camera. The proposed method can transmit motion images acquired by using a smartphone camera to a remote computer using wireless communication. This computer vision-based approach combines two separate image-processing techniques: template-based matching and a homography matrix. As an initial step toward the implementation of this proposed method, we also calibrated the smartphone camera using hang’s calibration algorithm to eliminate lens distortion. The next step was marker tracking in the captured motion images. We applied template-based matching using NCC function as the image-processing algorithm to track the marker. To remove the geometric distortion, a planar projection method by homography-matrix was applied. Through this research, we developed GUI-based software for measuring bridge displacements with a user-friendly interface. In addition to laboratory scale testing, field tests were conducted to evaluate the performance of the proposed method. To quantify the measurement error, an error evaluation was performed based on the RMSE. The proposed method showed the accuracy of the displacement measurement even when the angle between the smartphone camera and the marker increased.

Significant advantages of the proposed system include low cost, ease of operation, and flexibility to extract bridge displacements at various angles from a single measurement, making this system highly applicable to civil structures.