Study of Lateral Displacements and the Natural Frequency of a Pedestrian Bridge Using Low-Cost Cameras

Vision-based techniques are frequently used to compute the dynamic deflections of bridges but they are rather computationally complicated and require demanding instrumentation. In this article, we show that it is possible to reconstruct the 2-D kinematics of flexible bridges using a simplified algorithm to analyze common video imagery. The only requirements are that the movement of the control points is clearly visible on the images and that next to each control point, there exist vertical and horizontal bridge elements defining the image scale. We applied this technique during controlled, forced excitations of a timber bridge that was stiff in the vertical but very flexible in the lateral axis because of cumulated damage. We used videos from low-cost cameras, in which the changes of the pixel coordinates of several control points during excitation events and their attenuation were clear. These videos were obtained during two annual structural health monitoring surveys using numerous sensors (Global Navigation Satellite Systems (GNSS), robotic total station (RTS), accelerometers), and hence the output of the video analysis was fully controlled. Because of various errors, the transformation of the video image coordinates into bridge coordinates yielded spurious deflections along the main axis of the bridge, which were used to control the uncertainty of our results. We found that the computed lateral deflections (i) were statistically significant, (ii) satisfied structural constraints, and (iii) were consistent with structural estimates derived from other sensors. Additionally, they provided accurate estimates of the natural frequency and the damping factor of the bridge. This approach can be applied in other cases of monitoring of flexible structures if the requirements for planar deformation, pixel resolution and scale definition are satisfied.


Introduction
Vision-based techniques have been extensively used in the monitoring of static and dynamic deformations of various structures [1], especially of bridges (e.g., Xu and Brownjohn [2]). The overall approach is based on the exploitation of a linear relationship between the 3-D reference ("world") coordinate system of a control point on the bridge with its 2-D coordinates deduced from a series of images; the latter can be obtained using a wide range of image-recording hardware, from common video-recording cameras to high-rate, high-resolution cameras and CCD cameras. This relationship is in the form: x p , y p , 1 T = k A T x r , y r , z r , 1 where p and r indicate the image (photo) and reference coordinate systems, respectively; k is a scale factor; A is a 3 × 3 matrix describing the camera characteristics; and T is a 4 × 3 matrix defining the

The Kanellopoulos Timber Bridge and its Decay
The studied bridge is shown in Figure 1 and is located at the northern entrance of Patras, not far from the Patras University Campus, and was constructed in 2000. It is about 30 m long and 2.9 m wide and is made of Glulam wood and certain metallic elements. The omission of X-bracing below the deck and poor construction of the metal X-bracing at its roof ( Figure 2) rendered the structure structural elements of the bridge, as shown in the images analyzed. The ratio of the length of these elements relative to their length in pixels in the image permits deriving scales for correction of the displacement derived from the measurement of the number of pixels in the image. This is of course possible if the amplitude of displacements is small relative to the dimensions of the examined bridge elements, if the distance of the camera from these structural elements is in a range permitting the necessary pixel resolution, and visibility conditions are very good. Vertical lines are added to highlight the high amplitude of the bridge deflections. In the lower figure, the polygonal connections of steel X-braces at the roof of the bridge are seen to be displaced during a force-excitation event by jumping pedestrians (hardly shown on the bridge). The width of the polygonal connection, about 15 cm, indicates the order of magnitude of the lateral deflections of the bridge. These deflections are at a maximum at point P1, slightly smaller at point P2 at the midspan, and even smaller at point P3, the approximate position of which is shown using arrows. These photos are from a video shot from a location near to those of the analyzed videos.

Computation of Displacements
For the reasons discussed above, it is possible to compute approximate dynamic displacements with a low-cost camera using the technique explained in the three steps below. This is possible if the study bridge a priori satisfies three criteria: planar dynamic deformation, a significant amplitude of Figure 2. Two snapshots of a part of the Kanellopoulos bridge during a forced controlled excitation. Vertical lines are added to highlight the high amplitude of the bridge deflections. In the lower figure, the polygonal connections of steel X-braces at the roof of the bridge are seen to be displaced during a force-excitation event by jumping pedestrians (hardly shown on the bridge). The width of the polygonal connection, about 15 cm, indicates the order of magnitude of the lateral deflections of the bridge. These deflections are at a maximum at point P1, slightly smaller at point P2 at the midspan, and even smaller at point P3, the approximate position of which is shown using arrows. These photos are from a video shot from a location near to those of the analyzed videos.
In this article, we describe the methodology we used to estimate the dynamic horizontal deflections of specific points of this bridge using a simple video analysis technique and a low-cost camera to derive some dynamic characteristics of this structure. The basic characteristics of the proposed method were that it focused on significant deflections (amplitude of several mm) and may lead to statistically significant results without any correlation analysis [8]. Additionally, we describe the techniques we used to validate these results, as well as the perspectives and limitations of the proposed methodology.

The Kanellopoulos Timber Bridge and its Decay
The studied bridge is shown in Figure 1 and is located at the northern entrance of Patras, not far from the Patras University Campus, and was constructed in 2000. It is about 30 m long and 2.9 m wide and is made of Glulam wood and certain metallic elements. The omission of X-bracing below the deck and poor construction of the metal X-bracing at its roof ( Figure 2) rendered the structure vulnerable to events producing lateral oscillations. This, in combination with an extraordinary-for-the-area flash icing event and subsequent strong winds flowing normal to the deck direction, maximizing the lateral deflections (cf. Meng et al. [12]), led to a dramatic drop of its first lateral modal frequency in less than two years, and then led to a further, slower drop [11]. However, the vertical stiffness of the bridge was not significantly affected and the bridge remained stiff with a vertical main natural frequency >6 Hz, and hence with millimeter-level deflections at mid-deck. These results come from an analysis of annual surveys of the bridge since 2007. These surveys were based on combinations of various sensors (GNSS, RTS, accelerometers) at various parts of the bridge and were focused on different types of Sensors 2020, 20, 3217 4 of 13 forced excitations. The analysis of oscillations was especially focused on the free attenuation intervals of excitations, which provide evidence of natural frequencies. These results are summarized in various publications, especially in [9,11].

Bridge Quasi-Planar Dynamic Displacements and Impacts in Vision-Based Monitoring
Equation (1) describes the general case of photogrammetric analysis and is typically used to derive displacements from various types of imagery [3,4]. However, in specific cases, the relationship between the displacement and the imagery can be simplified. For example, in the case of a linear displacement on a plane normal to the sightline of the camera, an essentially linear relationship exists between a length in pixels read on an image describing the displacement and the "true" displacement. In this study, we focused on a special case in which displacements were two-dimensional (cf. Pan et al. [8]).
Detailed analysis of measurements of the dynamic displacements in bridges has shown that the deformation along their main (longitudinal) axis is nearly zero [13], and only vertical and lateral displacements are beyond the threshold of identification of most modern instruments. The only exceptions are extremely flexible structures and bridges swaying laterally, not only in their first mode but in higher modes as well, which are rare. For this reason, no displacements along the longitudinal axis of bridges are discussed in the literature. This means that during dynamic movements, the horizontal and vertical deflections are practically confined to a vertical plane that is moving back and forth with time. Lateral shifting of the vertical plane of dynamic deformation can be of the order of a few meters in long suspension bridges [12,14] and up to a few centimeters in common pedestrian bridges up to a few tens of meters long [9,15]. The fact that dynamic deflections are essentially planar in many bridges has important implications in vision-based analysis of their deflections.
If the recording camera is in a stable, known position and orientation, the planar character of the displacements of a monitored bridge will be preserved after any rotation and translation of the coordinate system. This is because rotations correspond to affine transformations, which preserve planarity. Consequently, Equation (1) can be simplified. However, reconstruction of the displacements requires a scale correction. The scale can be computed easily and precisely if the vertical plane containing the deformation vectors is next to, or very close to, horizontal and vertical structural elements of the bridge, as shown in the images analyzed. The ratio of the length of these elements relative to their length in pixels in the image permits deriving scales for correction of the displacement derived from the measurement of the number of pixels in the image. This is of course possible if the amplitude of displacements is small relative to the dimensions of the examined bridge elements, if the distance of the camera from these structural elements is in a range permitting the necessary pixel resolution, and visibility conditions are very good.

Computation of Displacements
For the reasons discussed above, it is possible to compute approximate dynamic displacements with a low-cost camera using the technique explained in the three steps below. This is possible if the study bridge a priori satisfies three criteria: planar dynamic deformation, a significant amplitude of some of its dynamic deflections, and target points next to structural elements permitting scale correction are monitored.

Digitization of the Motion of a Selected Point of the Deforming Bridge
A characteristic moving target point of the bridge that was visible in a series of i = 1, 2, . . . , n images (frames) was identified. Using common software, the variable position of this point relative to each frame shot from a stable camera was translated into a series of coordinates (x vi , z vi , t i ), where the index v indicates video. These coordinates refer to the plane of the image and describe the variations Sensors 2020, 20, 3217 5 of 13 of the location of the target point as a function of time relative to stable (reference) points common in the n images (frames).

Scale Correction
In the examined images, certain vertical and horizontal elements of the bridge that were near the control point were marked and their lengths in pixels were measured. The true length of these elements was also measured. A mean quotient of the true length of these elements to their length measured in pixels in the images could then be computed: The index v corresponds to the vertical axis and the index x to the horizontal axis, and and p correspond to the length and length in pixels, respectively. The quotients s v and s x define the scales used to transform the coordinates of displacements measured in pixels into scale-corrected, metric displacements (s x x vi , 0, s v z vi , t i ). Because this approach is approximate, non-linear scaling effects (lens distortion) were ignored.

Rotation of Images
Since the camera was kept fixed, the video images from which the displacements of the target point were measured were in a plane rotated relative to a bridge plane defined by the vertical and the main longitudinal axes of the bridge ( Figure 3). Furthermore, since the coordinates and orientation of the camera and the bridge were known/measured, the translation and rotation angles of the image plane relative to the bridge plane could be easily computed. Hence, an unknown planar displacement of the bridge [x, 0, z] T was mirrored into a displacement [X, 0, Z] T in the image plane. This process is described by Equation (3): Matrix T with dimensions 1 × 3 describes the translation of the origin of the two coordinate systems and matrix R z R x is a 3 × 3 matrix reflecting the superimposition of rotations around axes z and x. A detailed expression is given in the Appendix A. Matrix R z R x can be easily inverted because: Hence, in ideal conditions (i.e., ignoring errors), from the planar displacement vectors measured in the image plane [X i , 0, Z i ] T , the planar displacements of the point on the bridge [x i , 0, z i ] T can be computed through a simple linear transformation: T describes a translation similar to matrix T. However, because various measurements were contaminated by errors, Equation (5) should be replaced by Equation (6): which does not describe a planar deformation. The reason is that [s x x vi , 0, s z z vi ] T and R (as well as T ) contained errors, which were reflected in x * i , y * i , z * i T and especially in y * i 0. For this reason, the output of Equation (6) should be written as: Sensors 2020, 20, 3217 6 of 13 with u xi , u yi , u zi T describing the errors. The time series of y * i = u yi , which indicates the deviations from a planar deformation, permitted an estimate of the quality of the results: A preliminary analysis indicated that in the conditions of our experiment, the proposed technique could not provide estimates of deflections below the threshold of 15 mm. For this reason, and since the study bridge remained stiff along the vertical axis despite its damage [16], we focused only on the horizontal deflections, which were expected to be within the resolution limits of the method and the instruments used. Further analysis of the errors is given below. measured in pixels in the images could then be computed: The index v corresponds to the vertical axis and the index x to the horizontal axis, and ℓ and pℓ correspond to the length and length in pixels, respectively. The quotients s and s define the scales used to transform the coordinates of displacements measured in pixels into scale-corrected, metric displacements (s x , 0, s z , t ). Because this approach is approximate, non-linear scaling effects (lens distortion) were ignored.

Rotation of Images
Since the camera was kept fixed, the video images from which the displacements of the target point were measured were in a plane rotated relative to a bridge plane defined by the vertical and the main longitudinal axes of the bridge (Figure 3). Furthermore, since the coordinates and orientation of the camera and the bridge were known/measured, the translation and rotation angles of the image plane relative to the bridge plane could be easily computed. Hence, an unknown planar displacement of the bridge [x, 0, z] was mirrored into a displacement [X, 0, Z] in the image plane. This process is described by Equation (3):

Computation of Certain Dynamic Characteristics of the Bridge
Each of the videos examined covered a part of a controlled, forced excitation of the bridge, followed by free attenuation intervals. Analysis of the latter permitted computation of the main modal frequency in the lateral axis of the bridge, as well as the damping factor without any formal modal analysis.

Data
The analyzed videos came from the 2013 and 2015 annual field surveys when the effects of damage were very clear in the bridge, both as structural vicissitudes [11] and as high-amplitude lateral deflections during excitations by pedestrians and wind ( Figure 2); video in the Supplementary Materials of Stiros and Moschas [9]). During these field surveys, deflections at the mid-span of the deck were of the order of several centimeters, which was above the identification threshold of 15 mm estimated for the adopted methodology and the video camera used.
The videos that were selected were parts of two longer videos taken from known points, a few meters away from the SW edge of the bridge, with the camera sighting obliquely to the bridge under excellent visibility conditions. A low-cost but high-resolution pocket camera (16M pixels, 1920 × 1080 resolution) with automatic stability control and common mounting was used. No significant vibrations of the camera due to air currents generated by passing vehicles (cf. Feng and Feng [5]) occurred during the specific experiments. The two videos analyzed in this article covered parts of controlled, forced excitations of the bridge by well-trained parties of students and covered part of a transient and the following free attenuating oscillation (just after the jumping party froze). Details of the process are given in References [13,17,18]. Each of the analyzed videos had a duration of a few seconds and a rate of 30 images per second (frames per second (fps)).
For each of the two measurement years, namely 2013 and 2015, three control points were identified on the south face of the bridge, which were clearly visible in all frames. Point 1 (P1) was near the crown of the bridge, close to its midspan, next to a GPS/GNSS. Point 2 (P2) was at the midspan, on the top of the handrail of the bridge, close to a collocated GPS/GNSS, a reflector of the robotic total station, and an accelerometer. Finally, Point 3 (P3) was at about one-fourth of the span, close to an accelerometer ( Figure 2). Near each point, it was possible to identify and measure at least two vertical and two horizontal bridge elements, the edges of which were clearly identified in all video frames such that the scale corrections of Equation (2) were computed. Furthermore, stable reference points were identified in all frames.
In addition to these data, certain qualitative and quantitative constraints of the dynamic behavior of the bridge were available: (1) no significant vertical displacements were expected because the bridge remained stiff in the vertical axis (first natural frequency of the order of 6 Hz despite its decay [16] and (2) lateral deflections of several centimeters at the deck mid-span level were expected with a frequency of the order of 1 Hz [9], where these deflections were attenuated to the deck edges, while the crown of the bridge was characterized by lateral deflections of an amplitude larger than the deck [11]. These are constraints that allowed for evaluation of the performance of the video analysis.

Computation of Horizontal Deflections
Step 1: Videos were analyzed using the software "DLTdv Digitizing Tool" (biomech.web.unc.edu/dltdv/), which involved locking onto a selected point and certain non-moving reference points each time ( Figure 2). The output of each analysis was a CSV file of the type (x vi , z vi , t i ), which described the kinematics of the selected point in image pixel units as a function of the time. However, the variation of the vertical coordinate was small and within the error limits.
Step 2: The lengths of certain vertical and horizontal reference lengths, clearly visible in the images, were measured with a tape. The corresponding lengths in pixels in the images were also measured. From these measurements and using Equation (2), the mean scales s v and s x were computed for each video. Using these estimates for scale, matrices (x vi , z vi , t i ) describing the displacement in pixels were transformed into matrices [s x x vi , 0, s v z vi , t i ] describing the corrected metric displacements. The mean scale factors are shown in Table 1.
Step 3: Based on survey measurements, the coordinates of the setup point of the camera in each epoch and its pitch (along axis X) and yaw (along axis Z) angle were computed and are summarized in Table 1. These rotation angles were used to compile matrix R in Equation (6). The matrix x * i , y * i , 0 T of deflections in the coordinate system defined by the main axis of the bridge were then computed. The time series of computed deflections of the three control points for the 2013 and 2015 data are plotted in Figure 4. This figure also shows apparent deflections along the longitudinal axis of the bridge, which were negligible, as already explained. The mean square value σ 2 of these apparent deflections provided an estimate of the order of magnitude of the errors in the specific study, both concerning the method and the type of camera used.

Possible Structural Implications for the Kanellopoulos Bridge
The results of the video analysis indicate a minimal reduction of the stiffness of the bridge between 2013 and 2015 (the first modal frequency changed from 0.90 to 0.87 Hz). This difference was consistent with the overall trend of the bridge that was derived from an analysis of a survey and accelerometer data and summarized in Table 3, as well as with eye-witness evidence for a gradual structural decay [11]. However, such evidence comes from only two experiments, the results are within uncertainty limits, and their documentation requires a formal statistical analysis (cf. Stiros and Moschas [21]).

Structural Characteristics of the Bridge
Because of their design, the analyzed videos covered the transient part of the oscillations and its free attenuation. The latter could directly (i.e., without any formal structural identification analysis) lead to two important characteristics of this bridge: its main modal frequency f along the main lateral axis and the damping factor (attenuation coefficient) ζ. f was computed through spectral analysis of displacements x(t), while ζ is computed through a least-squares analysis of Equation (9): In this equation, which is valid for ζ < 0.1, ω = 2πf, ϕ is the phase characterizing the oscillation, and A is the initial amplitude just before the decay of the motion [19]. Results are summarized in Table 2, and Figures 5 and 6.     Table 3. Variations of the first lateral natural frequency f of the Kanellopoulos Bridge, known from the analysis of geodetic sensors and accelerometers (from Thalla and Stiros [11], with additions) and those derived in this study.
Year f (Known) f (Video) Figure 6. Modeling of the free attenuation intervals of the oscillations of point P2 as free attenuating oscillations based on Equation (9). The period input into the analysis was computed from the spectra of Figure 5.

Discussion
Vision-based estimation of deflections of bridges is usually based on high-quality cameras and a rather complicated analysis of digital images [2,20]. In this article, it is shown that under certain conditions, low-cost cameras and a simplified methodology can lead to a time series of displacements. These results first need to be validated and the limitations of the method must be discussed.

Data Validation
The obtained results were first tested for statistical significance, and second for their consistency with structural constraints. 7.1.1. Statistical Significance Figure 4 and Table 2 indicate that the lateral deflections were larger than the corresponding longitudinal deflections indicating noise. Furthermore, in each case, the corresponding signal-to-noise ratio (SNR, measured as the mean square value of lateral deflections to the mean square value of the apparent longitudinal deflections) indicated statistically significant results for all three points, in both the 2013 and 2015 surveys.

Consistency with Structural Constraints
From a qualitative point of view, as noticed above (Section 4), it was expected that the deflections of P1 were to be higher than those of P2, and those of P2 (mid-span) were to be higher than those of P3. These constraints were satisfied, as the results summarized in Figure 4 and Table 2 indicate.
The previous knowledge of the structural characteristics of the bridge indicated that during excitations, such as those analyzed, the lateral deflections at P2 were expected to reach and even to exceed an amplitude of several centimeters; this was consistent with the results of Table 2 and Figure 4.
The spectral analysis of the free attenuating oscillations led to estimates of the first modal lateral frequency that were very similar in all points for each epoch, and the values were consistent with values derived from independent measurements (Table 3). Damping factors computed for the three points were also consistent with each other (Table 2) and of an order of magnitude expected in a timber bridge with high deflections (cf. Bachmann et al. [19], p. 4). What is especially impressive is that the recorded deflections were characterized by a pattern that is expected in free attenuating oscillations, even for points with a low SNR (as evident from the difference between the red and black curves ( Figure 4). Table 3. Variations of the first lateral natural frequency f of the Kanellopoulos Bridge, known from the analysis of geodetic sensors and accelerometers (from Thalla and Stiros [11], with additions) and those derived in this study. These results provide evidence that this simplified video analysis led to reliable data.

Possible Structural Implications for the Kanellopoulos Bridge
The results of the video analysis indicate a minimal reduction of the stiffness of the bridge between 2013 and 2015 (the first modal frequency changed from 0.90 to 0.87 Hz). This difference was consistent with the overall trend of the bridge that was derived from an analysis of a survey and accelerometer data and summarized in Table 3, as well as with eye-witness evidence for a gradual structural decay [11].
However, such evidence comes from only two experiments, the results are within uncertainty limits, and their documentation requires a formal statistical analysis (cf. Stiros and Moschas [21]).

Limitations and Applications of the Method
The method proposed is simple and may lead to significant results, as has been shown above. However, it is an approximate method and it can only be applied under certain conditions. The first condition is that it is only efficient for slender structures if deflections are above a certain threshold. This threshold depends on the resolution of the camera and its distance from the observed structure such that displacements are reflected as a signal of several pixels. In the case of a low-cost camera, as was used during this study, and in good visibility conditions, the method is suitable for deflections above the threshold of 10-15 mm and at distances of up to a few tens of meters; these conditions are not unusual, even for hardware with high specifications [2,20]. Another requirement of the proposed method is that the morphology of the analyzed bridge and its background should permit a stable reference frame for the video and that the analyzed bridge should contain elements that are next to control points and are seen in all video frames such that scale corrections are possible.

Conclusions
It was shown that the analysis of low-cost video images using a simple approximate technique permits the reconstruction of deflections of flexible bridges and even the computation of some of their structural characteristics. This result is possible under certain conditions: the deformation is two-dimensional, deflections of the selected target points are characterized by a signal exceeding the pixel resolution, the camera is in a fixed position and orientation, and finally, the video image covers stable points defining a stable reference system and structural elements near the selected target points such that the scale of the photo in the two examined axes can be defined. This approach was developed for dynamic displacements but it can also be applied in the case of semi-static and static motions (for example, static loading tests). In this last case, the resolution and accuracy of the method are expected to be higher (and hence the method may be applied to stiffer structures) because the noise in images produced by atmospheric turbulence, a major threat for optical techniques around bridges, is randomized and minimized (cf. Stiros and Moschas [21]).
The proposed method is certainly not limited to certain types of bridges since it may also be applied to various other structures, both in close range in the laboratory and with a longer range in the field. For example, it can be applied in laboratory tests for control of the deformation of various structural elements, avoiding expensive contact methods. It can also be applied in various types of structures in the field, as long as they are subject to 2-D deformation. However, significant results are expected if the whole series of images show stable objects marking a stable reference frame. Furthermore, since a single video time series of displacement allows for more than one point to be measured, the rotation angles (e.g., pitch, yaw, etc.) of rigid bodies can also be computed.

Equation (1) in its full form is:
r 11 r 21 r 12 r 22 r 13 r 23 r 13 t x r 23 t y r 33 t z where index p indicates the image (photo) and r indicates the reference ("world" or "global") coordinates, k is a scale factor, A is a matrix expressing the deformation of the image through the camera, and R is a matrix expressing the rotation and translation of the camera [3,4]. The full form of the matrix product R z R x in Equation (3) is: cos c −sin c 0 sin c cos c 0 0 0 1 cos c −cos a sin c sin a sin c sin c cos a cos c −cos c sin a 0 sin a cos a where c is the angle of rotation along axis z and a is the angle of rotation along axis x, as shown in Table 1.