A New Slice Template Matching Method for Full-Field Temporal–Spatial Deflection Measurement of Slender Structures

Abstract

A sufficient number of sensors installed in all structural components is a prerequisite for obtaining the full-field temporal–spatial displacement and is essential for large-scale structure health monitoring. In this paper, a novel lightweight vision-based temporal–spatial deflection measurement method is proposed to measure the full-field temporal–spatial displacement of slender structures. First, the geometric and mechanical properties of slender members are introduced as the priori information to vision-based measurement. Then, a slice template matching model is proposed by deploying a one-dimensional template matching model in every pixel column of each image frame, based on traditional digital image correlation (DIC) method. An indoor experiment was carried out to verify the proposed method, and experiment results show that measurement precision of STMM agrees well with the theory and the laser ranger, with a maximum measurement error of 0.03 pixels and the root-mean-square error (RMSE) of 0.052 mm, for transient beam deflection curve; with the correlation coefficient and coefficient of determination of 0.9994 and 0.9986, for dynamic deflection–time history curves at the middle-span point. Finally, further investigation reveals that brightness inconstancy is the source of STMM measurement error.

1. Introduction

In the past 40 years, many bridges have been built to serve the public, and 1,079,300 road bridges with a total length of 95,288,200 m [1] had been constructed in China alone by the end of 2023. Meanwhile, recent years have witnessed frequent bridge collapses, overturning, and other accidents, which have attracted great attention from highway maintenance and management departments. The Chinese transportation ministry launched an ambitious plan to install structural health monitoring (SHM) systems for all long-span bridges and special bridges across the sea [2] by the end of 2025, and many SHM systems have been installed [3,4]. SHM systems are composed of a large number various types of sensors, for example, the of Sutong Bridge SHM systems are composed of 785 sensors [3], and the Stonecutters Bridge SHM systems are consist of 1500 sensors of 20 types. Large quantity and types of sensors, complex data transmission and power supply circuits pose significant challenges to the system’s stability and service life.

Research [5,6] revealed that the SHM system’s ability to identify and locate structural damage is closely related to its density of the measurement points sets, and the damage can be accurately identified only when the sensors are placed on both sides of the damage location but cannot be identified if the sensors are located on only one side of the damage. Therefore, a large number of sensors becomes a prerequisite to locate the damage in each structural component of bridges, which poses great challenge of tremendous cost to bridge owners, as well as huge workload to engineers. As a result, it is an urgent requirement to develop low-cost full-field displacement monitoring technology.

Machine vision-based motion tracking technology has the advantages of non-contactness, cost-effectiveness, convenience, and high efficiency, and can record continuous temporal and spatial motion of objects at high speed. As the technology is highly compatible with the requirements of structural monitoring and damage identification, it has been widely applied in bridge health monitoring [7,8,9,10,11,12,13,14]. Dong [8] and Ye [9] systematically investigate various motion tracking methods, including the DIC, feature points matching, optical flow, geometrical matching, etc. Vision-based motion tracking methods are applied in different aspects of bridge measurement [7,8], for example, influence line of deflection measurement [13], modal information recognition [8], damage identification, cable force estimation, and structural technical condition evaluation [5,14,15,16,17,18]. Feature extraction and matching-based motion tracking algorithms suffer from a large number of redundant traversal searches and multi-scale feature computations, and feature mismatching occurs frequently [13,14], which reduces the accuracy of structure monitoring.

In this paper, we aim to transfer the geometric properties of large slender ratio member structure and small deformation hypothesis in engineering mechanics to priori knowledge and introduce it into the machine vision-based slender structure deflection monitoring model. A lightweight vison-based model, slice template matching model (STMM), is proposed and verified to measure the full-field temporal–spatial displacement of slender ratio structure by constructing a 1-D template matching model in every pixel column of each structure image frame. This paper is organized as follows: Section 2 introduces the theory of vison-based full-field structural displacement measurement methods and problems encountered; Section 3 proposes the STMM method based on the geometric properties and mechanical priori knowledge of slender structure; Section 4 verifies the STMM method by an indoor multi-box simple support beam experiment; finally, Section 5 provides conclusions and possible perspectives.

2. Vision-Based Structural Displacement Measurement Methods

2.1. Digital Image Correlation Theory

Digital Image Correlation (DIC) is one of the most commonly used dynamic displacements measurement method for small structures or components with speckles preinstalled on specimen surface. In DIC, correlation coefficients are used to characterize the approximation of two same size images, and a variety of correlation coefficients definitions have been proposed [15]. For two same size images f and g in an image sequence, indicating different deformed states of a component, the most commonly used correlation coefficient is the Normalized Sum of Squared Differences (C_NSSD) [5,9], formulated as follows [15]:

$$C_{NSSD}={\displaystyle \frac{{\displaystyle {\sum }_{i=0}^{M-1}{\displaystyle {\sum }_{j=0}^{N-1}[f(i,j)-g}}(i,j){]}^2}{\sqrt{{\displaystyle {\sum }_{i=0}^{M-1}{\displaystyle {\sum }_{j=0}^{N-1}f{(i,j)}^2}\cdot {\displaystyle {\sum }_{i=0}^{M-1}{\displaystyle {\sum }_{j=0}^{N-1}g{(i,j)}^2}}}}}}$$

(1)

where M and N are the image height and width, and i and j are the row and column pixel coordinates, respectively, all are in terms of pixels. In general, high precision motion tracking in DIC can be divided into two stages, pixel-level displacement location and subpixel-level displacement calculation, as shown in Figure 1.

For a n-frame image sequence, the sub-image of tracked object is selected as mask and the C_NSSD matrices are calculated traversed in all frames of images. In the kth frame (k = 1, 2, 3, …, n), the pixel coordinate (x_kp, y_kp), in accordance with the smallest value in C_NSSD, is the pixel-level location of the mask in motion. Then, the sub-pixel coordinates (x_k, y_k) of tracked target can be calculated using C_NSSD surface fitting, or gray moment conservation [19] in neighborhood of the pixel-level coordinates (x_kp, y_kp). The motion of tracked object can be recorded with the initial coordinates (x₀, y₀) and a series of dynamic coordinates (x_k, y_k) (k = 1, 2, 3, …, n) at different times.

2.2. Feature-Based Motion Tracking

Feature extraction-and-matching is another popular vision-based motion tracking method, and Scale Invariant Feature Transform (SIFT) is the most frequently used multiscale feature [7,8,16,17,18], among Harris–Tomasi, SIFT, and Speeded Up Robust Features (SURF). However, there are a large amount of computation redundancies, as well as feature mismatching, in structural displacement measurement by feature-based motion tracking method, due to the non-uniformly distributed gray gradients that are caused by dust and oil stains texture on structure surfaces, as shown in Figure 2. Chu [13] proposed a method to constrain motion inconsistency to eliminate the mismatching problems, and Zheng [14] proposed a two-staged Random Sample Consensus (RANSAC) and a geometrical prior information method to remove the mismatching. Huang [20] suggested that the accuracy of the displacement measured by feature extraction of natural structure surface texture is low, and that high-reflectance artificial markers or speckles should be prefabricated or installed. Among the problems of feature-based motion tracking methods in full-field displacement measurements, the first is that the non-uniform distribution of feature points, the red points in Figure 2, do not satisfy the requirements of high-precision strain calculation through displacement interpolation, which prefer uniformly distributed feature points as shown in Figure 3.

Moreover, to achieve high precision measurements, the huge workload in prefabrication and installation of a large number of high-reflectance artificial markers or speckles on structure surface is also a great challenge for engineers in full-field temporal–spatial deformation of large-scale structures. Generally, feature mismatching, computational redundancy, and huge time and economic consumption pose great difficulties to real-time full-field displacement monitoring of large-scale structures. Therefore, it is necessary to put forward a light-weight, full-field temporal–spatial deformation measurement or displacement monitoring method for large-scale structures.

3. Full-Field Temporal–Spatial Displacement Measurement of Large Slender Ratio Structure

3.1. Priori Knowledge of Slender Structure Geometry and Deformation

Large slender ratio member is widely applied in bridge engineering, including beams, cables, bars, piers, and the scale of one dimension in the structure is much larger than the other two dimensions. In the case of slender beam members subjected to pure bending loads, for example, the length is much larger than its height and width, and the main deformation concerned in engineering mechanics is the deflection, with the axial and shear deformation being neglected. Accordingly, the deformation magnitudes and rotation angles of points within the same transverse section are considered the same, and the full-field displacement measurement of the pure bending beam in Figure 2 can be transferred to the measurement of deflection monitoring of points along the beam length in Figure 4.

For slender beams subjected to eccentric axial and transverse loads, the ideal single row of uniform feature points distribution for deflection measurement, as shown in Figure 4, is also applicable, but the axial–bending interactions should be taken into consideration in structural health diagnosis or status evaluation.

In this way, the geometrical features and deformation characteristics of slender structure could be taken into the vision-based motion tracking model as the priori information to construct a new feature paradigm, which lays a foundation for the extraction full-field temporal–spatial displacement of slender structures. Therefore, for large-span bridges, the number of measurement points deployed on the surface of the structure will be greatly reduced, but it is still difficult to manually preinstall reflective markers for DIC or feature-based motion tracking.

3.2. One-Dimension DIC Model

For each point in the ideal single row of uniform distributed monitoring points on the beam in Figure 4, a reduced one-dimension DIC model could be constructed, degrading from the traditional template matching method or DIC method. For 1-D images of any column or row in 2-D image, shown in Figure 5, the template matching method still works.

For a one-dimension template matching model, C_NSSD can be formulated as

$$C_{NSSD}={\displaystyle \frac{{\displaystyle \sum _{i=0}^{m-1}{[f(i,j)-g(i,j)]}^2}}{\sqrt{{\displaystyle {\sum }_{i=0}^{m-1}f{(i,j)}^2\cdot {\displaystyle {\sum }_{i=0}^{m-1}g{(i,j)}^2}}}}}$$

(2)

where m is the number of correlation transversal in one column or row.

3.3. Slice Template Matching Model

3.3.1. Concept of Single Slice Template Matching Model (STMM)

In vision-based monitoring, the slender structure, for example a beam, is captured and stored in a 2-D gray image as shown in Figure 6a, and there is good distinguishability of the gray-scale values in the vertical direction, but poor distinguishability horizontally. Without preinstalled markers, it is difficult to track the motion of the beam by DIC method or feature-based motion tracking method, but it is possible to track any motion of the single 1-D image in the i-th column of the image, as shown in Figure 6a.

For the 1-D image composed of N pixels, the N₀ⁱ pixels contains the beam section and is set as the mask as shown in Figure 6b, and the transversal pixels is the mask’s possible motion region in the column indicated as N_sⁱ, where i is the column number in the image. Therefore, a one-dimension template matching model could be set up, and the number of pixels within the transversal sections in each column can be determined by the priori geometrical, mechanical information of the beam as follows.

The number of pixels in mask N₀ⁱ should be slightly larger than that of beam height, and the height of transversal section H_s is formulated as follows:

$$H_S=h+kD\hspace{1em}(1.2\le k\le 1.5)$$

(3)

where h is the height of beam, and D is the deflection value calculated theoretically from material property, priori loading and geometry information, as shown in Figure 7. k is a constant empirical coefficient between 1.2 and 1.5.

In this way, the full-field deformation measurement of the beam can be transferred to a series of 1-D template matching models to trace the deflections of uniform measure points in Figure 4, and this one-dimensional template matching model is called Slice Template Matching Model (STMM). In practical application, the number of measurement points arranged on the beam surface can be determined according to engineering requirement, and the maximum number is the number of pixels columns M_l covered by the beam length in image space, i.e., one template matching model is established for each column of pixels. Then, the dynamic full-field displacement measurement of the slender structure is converted into M_l series of STMM per frame in the image sequence.

It is clear that the STMM resolves the two-direction distinguishability challenge in the beam image, by converting the measurement task into multiple one-dimensional template matching processes, which require distinguishability only in the vertical dimension. This approach can be easily achieved for bridge structure located in natural light environment while maintaining high computational efficiency. Moreover, multiple one-dimension models are computed independently each column, and it helps the realization of parallel computation. Meanwhile, this computation independence between columns helps to avoid interference in the neighborhood, as in the two-dimension template matching method model or in the feature matching method, which induces computation redundancy as well as mismatching. The simplicity and interpretability of STMM are better than feature extraction-and-matching based method.

3.3.2. Multiple Slice Template Matching Model and Implementation Process

To reduce local optical noise or similar texture on structure surface, causes by similar oil and dust stains or unexpected nonuniform illustrations, the slice template matching model can also be applied in a one-dimension image composed of n neighbor pixel columns (n = 1, 3, 5), as shown in Figure 8, and it is named the Multiple Slice Template Matching Model (MSTMM). For n-pixel width Slice Template Matching Model, the number of measurement point is M_l − n + 1, M_l is the number of pixels in the beam length in the image space, and the computation amount increase almost linearly with width of STMM.

The STMM proposed in this paper is a vision-based structure displacement measurement method, and, in practical engineering applications, the prior information of geometry, material and structure should be carefully analyzed as the basis for the design and setup of vision-based systems, for example, the selection of cameras, lens. At the same time, the selection of template, transversal region and mask width is a complex trial and error process. The workflow of the proposed slice template matching is outlined in Figure 9.

4. Experiment Verification

4.1. Experiment Design and Arrangement

An indoor simply supported the composite–box–beam experiment under moving load was designed to validate the accuracy of full-field temporal–spatial structural displacement measurement with STMM. In the experiment, the beam is 5.77 m long, with a cross section of 160 mm in width, 40 mm in height, and 2 mm in thickness, shown in Figure 10.

On the simply supported beam, the dynamic load F was applied with an 8 kg toy truck, traveling from the right support end to the left at a constant speed, as shown in Figure 11. The total image sequence was taken in 8 s, and the toy truck started 1.8 s after the shooting began. Then, the truck traveled to the middle span point at 4.2 s, arriving the left end at 6.6 s and stopped until the shot was over.

Theoretically, the beam deflection curve, under concentrated force F, can be formulated according to the differential equation in mechanics of materials.

$$w(x)=\left\{\begin{array}{c}{\displaystyle \frac{Fbx}{6E{I}_{z}l}}(l^2-x^2-b^2)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} 0\le x\le a\\ {\displaystyle \frac{Fb}{6E{I}_{z}l}}[{\displaystyle \frac{l}{b}}{(x-a)}^2+(l^2-b^2)x-x^3]\hspace{1em} a\le x\le l\end{array}\right.$$

(4)

where w is the deflection of beam, a and b are the distance between concentrated force and the two supports end, respectively, shown in Figure 11, and EI_z is the flexural rigidity of the beam.

In the experiment, the image sequence was captured orthogonally by a SONY video camera, and the dynamic displacement–time history curve in the mid-span of the beam were measured with a laser ranger, and the parameters of the camera and laser ranger are listed in

Table 1. Parameters of experiment devices.

Experiment Device	Model	Technical Parameters
Video Camera	SONY/FDR-AX40	Resolution 3840px × 2160px /Frame Rate 25fps
Laser Ranger	RIFTEK/R603	9600Baud

.

In the image, the number of pixels along test beam length direction is 3770 pixels, and the spatial measurement resolution of the system is 1.53 mm/pixel. Research showed that high reflection coefficient material [20] could help to improve the accuracy of machine vision measurement. A 1.5 cm reflective strip in height was stuck on the beam surface, shown in Figure 12, to improve the distinguishability between structural surface and the environment.

The setup of the simple supported composite box beam experiment under moving toy truck load is shown in Figure 13.

Following the procedure in Figure 9, the mask and transversed region are selected in each column of 1-D image, as shown of the ith column in Figure 14, and slice template matching model was set up column by column. Totally the entire beam region was segmented into 3770 columns, and 3770 1-D slice template matching models were setup to track the full-field temporal–spatial deflection of the simply supported beam.

The dynamic response of the beam is recorded at the speed of 25 frames per second when the toy truck travels from the right end to the left end, and a total of 94,250 displacement data can be calculated per second for all 3770 monitoring points along the beam.

4.2. Measurement Error Analysis

As the data volume is huge in full-field temporal–spatial deflection measurement of large structures, it poses great difficulties to the calibration of the dynamic displacements of such a large number of measure points at the same time. To verify the measurement precision of STMM, two specific scenes were designed. One scene is to verify the transient full-field displacements of the beam at any time, and another scene is to verify the dynamic displacement–time curve at any point on the beam. In the first scene, the full field deflection curve of simply supported beam can be calculated theoretically with Equation (4) if the loading location is known, and the deflection curve can also be measured by STMM at the same time, when the toy truck traveling at the loading location. The errors of all 3770 STMM measurement points can be obtained by comparing the theoretical and experimental deflection curves. In the second scene, the dynamic displacement–time curve of any point along the beam can be measured by both a laser ranger and a corresponding one-dimension STMM, and similarly the error between the two motion tracking methods can be calculated.

4.2.1. Transient Full-Field Displacement Error Analysis

As mentioned above, the toy truck started at 1.8 s, and finished the trip in 4.8 s at the time of 6.6 s. When the toy truck traveled at the mid-span of the beam at 4.2 s, and the theoretical deflection curve of the simply supported beam and deflection curve calculated using the STMM are shown in Figure 15. The results show that the two curves agree well with each other with a maximum measurement error of 0.72 mm.

From Equation (2), it can be seen that the deflection curve of the simply supported beam under concentrated force is a segmented cubic equation, so the STMM measurement results are fitted with a cubic equation using the least squares method. The fitted deflection curve and theoretical deflection curve are shown in Figure 16. It is clear that the fitted deflection curve exhibits better consistency with theoretical deflection curve, with a maximum measurement error of 0.17 mm.

The fitted deflection curves could be obtained when the toy truck traveled along the beam at 4.2 s, 4.6 s, 5.0 s, 5.4 s, 5.8 s, and 6.2 s, respectively, as shown in Figure 17.

At any time, subtracting theoretical deflections of 3770 measurement points from corresponding STMM measured results, the spatial measurement error distribution of the STMM can be obtained. As an example, the spatial measurement error of original STMM deflection and fitted STMM deflection along the beam can be calculated under middle span load and shown in Figure 18.

When the toy truck traveled to the mid-span, error analysis reveals that the maximum measurement error of STMM occurs near the right end support, specifically with an error of 0.47 pixels and a corresponding measurement error of 0.72 mm, and the maximum measurement error is 0.094 pixels or 0.17 mm for fitted STMM.

Measurement errors of all measure points at a given time can be quantified using the Root Mean Square Error (RMSE), formulated as follows:

$$RMSE=\sqrt{{\displaystyle \frac{1}{m}}{\displaystyle \sum _1^m{[h(x_i)-y_i]}^2}}$$

(5)

where h(x_i) and y_i are the measured and theoretical deflection value of point I, respectively. To show the error variation in column width, the measurement error of 1, 3, 5, and 11 pixels width STMM are listed in

Table 2. RMSE of STMM under mid-span load.

Template Width	Data Types	RMSE/mm
		STMM	Studies
1	Original data	0.204
3		0.191	0.1~0.2 [15]
5		0.182	0.7~2.6 [21]
11		0.169	0.1~0.3 [22]
11	Fitted data	0.052

, comparing with that of vision-based static deflection measurement in studies [15,21,22].

The results demonstrate that the maximum measurement error of STMM is comparable to that of static deflection measurements reported in the current literature, and the measurement error decreases as the width of template slice increases. It can also be seen that the fitted measurement error decreases to 0.03 pixels or 0.052 mm when the template slice width is 11 pixels. The STMM proposed in this paper is of high precision in displacement measurement and can meet the requirements of engineering measurement.

4.2.2. Single Point Dynamic Displacement Measurement Error Analysis

To reveal the dynamic displacement measurement error of a single point on the beam, the displacement–time history curves of the mid-span point by STMM and laser ranger are shown in Figure 19, and the results show that the displacement–time history curves obtained by the two methods are in good consistency.

From the two displacement–time history curves in Figure 19, it can be deduced that the dynamic displacement measurement error is not constant but varied over time. To clarify the measurement error over time, the measurement errors of two curves over time are shown in Figure 20, and results indicate that the maximum error is approximately 0.5 mm or 0.3 pixels.

The errors of two displacement–time history curves are characterized by correlation coefficient $\rho$ and determination coefficient R² [16], formulated as follows:

$$\rho ={\displaystyle \frac{\left|{\displaystyle \sum _i(r_L(i)-{\mu }_{r_L})\times {\displaystyle \sum _i(r_v(i)-{\mu }_{r_v})}}\right|}{\sqrt{{\displaystyle \sum _i{(r_L(i)-{\mu }_{r_L})}^2}}\sqrt{{\displaystyle \sum _i{(r_v(i)-{\mu }_{r_v})}^2}}}}$$

(6)

$$R^2=1-{\displaystyle \frac{{\displaystyle \sum _i{(r_v(i)-r_L(i))}^2}}{{\displaystyle \sum _i{(r_v(i)-{\mu }_{r_v})}^2}}}$$

(7)

where $r_L\left(i\right)$ and $r_v\left(i\right)$ are displacement measured by the laser ranger and the STMM of measurement point at time i, respectively, and ${\mu }_{r_L}$, ${\mu }_{r_v}$ are the mean values of each data sets. The correlation coefficient $\rho$ and determination coefficient $R^2$ of the two displacement–time history curves are listed in

Table 3. Correlation of mid-span displacement time curves of STMM and laser ranger.

Source of Data	Correlation Coefficient	Coefficient of Determination
Reference [17]	0.9689~0.9887	0.9368~0.9775
This Paper	0.9994	0.9986

.

From Table 3, the correlation coefficient and the determination coefficient of the two mid-span displacement–time history curves are 0.9994 and 0.9986, respectively; both are superior to the values reported in the literature [14].

4.3. Measurement Error Source Analysis

The transient full-field deflection error analysis indicates that the displacement measurement error of the 3761th column of 1-D STMM is the largest, and it is selected as the target model to trace the source of error by comparing the gray distribution of the transversal region. Figure 21 illustrates the gray value distribution of the 3761th column at the time of 0 s, 3.2 s, 4.0 s, 4.8 s,5.6 s, and 6.4 s, respectively, denoted as TS_i, where TS is the transversal search region, and i is the time. In Figure 21, it can be seen that the gray value distribution along the transversal search region varied slightly with the time, especially at the time of 4.8 s, 5.6 s, and 6.4 s, due to the brightness inconstancy of the light environment that is crucial in DIC and optical flow method.

5. Conclusions

In this research, a novel method of slice template matching model is proposed to measure the full-field temporal–spatial deformation of slender structures, and the indoor experiment shows that STMM is proved to be of high precision and has good application prospects in future bridge health monitoring. The following conclusions can be drawn:

(1) The slice template matching model proposed can effectively measure the full-field temporal–spatial deflection of structure with large slender ratio, and an indoor experiment was designed and carried out to validate the STMM method. Experiment results show that the displacement measurement result of the STMM agreed well with the theoretical results and laser ranger measurement results.

(2) Experiment results demonstrated that the temporal full-field displacement measurement error of the STMM is not constant and changes both spatially and temporally, and the maximum displacement measurement error is 0.47 pixels. Transiently, the RMSE of full-field simply supported beam deflection curve from the STMM measurement and theoretical results is about 0.169 mm. For single points, the correlation coefficient and the determination coefficient of displacement–time history curves of the STMM and laser ranger are 0.9994 and 0.9986, respectively.

(3) Gray distribution analysis of the transversal region reveals that the STMM is also greatly influenced by the brightness inconstancy caused by optical environment variation, and further study should be carried out to investigate the mechanism of the measurement error and control in complex weather conditions and illumination environments, or structures with complex inelastic mechanical properties and large deformations, for example, cables, bars, and piers.