Vision Measurement Method Based on Plate Glass Window Refraction Model in Tunnel Construction

Due to the harsh environment of high humidity and dust in tunnel construction, the vision measurement system needs to be equipped with an explosion-proof glass protective cover. The refractive effect of the plate glass window invalidates the pinhole model. This paper proposes a comprehensive solution for addressing the issue of plane refraction. First, the imaging model for non-parallel plane refraction is established based on dynamic virtual focal length and the Rodriguez formula. Further, due to the failure of the epipolar constraint principle in binocular vision systems caused by plane refraction, this paper proposes the epipolar constraint model for independent refractive plane imaging. Finally, an independent refraction plane triangulation model is proposed to address the issue of triangulation failure caused by plane refraction. The RMSE of the depth of field errors in the independent refraction plane triangulation model is 2.9902 mm before correction and 0.3187 mm after correction. The RMSE of the positioning errors before and after correction are 3.5661 mm and 0.3465 mm, respectively.


Introduction
A Tunnel Boring Machine (TBM) is an extensive piece of equipment utilized in tunnel construction.It combines various functions including excavation, removal of debris, and lining.TBMs are widely used in many fields such as railroads, highways, water diversion projects, and municipal pipelines.In the TBM digging process, the cutter is subjected to complex and variable loads.The cutter consumes a lot during the construction process and requires frequent replacement [1,2].The tool change area [3] is shown in Figure 1.The cutter change area is very narrow and not conducive to manual work.At present, TBM tool-changing work is dangerous and inefficient [4].This has led to an urgent demand for robotic tool-changing technology in the field of tunnel construction.Currently, manual operations are primarily relied upon for tool detection and replacement due to limitations in technological development.During the construction process of TBMs, there are considerable safety hazards related to manual operations in harsh construction environments.These environments include deep burial, high water pressure, and high levels of dust, which can potentially result in serious safety accidents such as casualties.Thus, achieving intelligent tool-changing in harsh and complex tunnel construction environments requires precise positioning and accurate measurement of the tool system.
Machine vision measurement technology has advantages over other measurement technologies such as non-contact measurement, remote manipulation, information-rich, fast response, and accuracy [5][6][7].The environment of the TBM tool change bin is often characterized by high pressure [8], high humidity, and dust.To ensure safety, the camera inside the tool changing bin needs to be equipped with an explosion-proof glass protective cover [9,10].However, the traditional pinhole camera model cannot eliminate the refractive distortion caused by glass.The existing distortion models with a single factor (only considering the lens) cannot describe the imaging patterns of cameras under planar refraction [11].
Sensors 2023, 23, x FOR PEER REVIEW 2 of 20 cover [9,10].However, the traditional pinhole camera model cannot eliminate the refrac tive distortion caused by glass.The existing distortion models with a single factor (only considering the lens) cannot describe the imaging patterns of cameras under planar re fraction [11].This paper presents a comprehensive solution for binocular measurement under the refraction of plate glass.The specific contributions are as follows: (1) The non-parallel plane refractive camera imaging model is established based on dy namic virtual focal length and the Rodriguez formula.(2) The main problems are the epipolar constraint failure and triangulation failure unde refraction images.This paper proposes an epipolar constraint model and triangula tion method based on independent refraction planes.
The remainder of this paper is structured as follows.Section 2 introduces the related work.Section 3 describes, in detail, the model for image distortion correction under non parallel plane refraction, the model for imaging epipolar constraint on independent re fractive surfaces, and the triangulation method for independent refraction planes.Section 4 validates the effectiveness of the proposed method by conducting a series of physica experiments.Section 5 discusses the relevant issues.Section 6 presents the conclusion.

Related Work
The refraction effect of glass severely affects the accuracy of visual measurements Scholars have conducted in-depth research on the characterization and elimination of re fractive distortion.In the calibration of visual measurement system parameters under pla nar refraction, Treibitz [12] studied the trajectory of camera viewpoints under planar re fraction environments.In addition, Treibitz proposed an imaging model for a camera based on dynamic virtual focal length.The measurement accuracy of this model is signif icantly greater compared to single-viewpoint models.However, this model requires prio knowledge of the depth information of the target object.The camera's optical axis mus also be parallel to the normal vector of the refractive surface.Agrawal [13] established an imaging model for multiplane refraction.The multi-plane refraction imaging model has been proven to have strong similarities to the single-plane refraction imaging model.
Chen [14] introduced a calibration method for determining the normal vector of the refractive surface as well as the thickness of the glass.The experiment demonstrates tha this method exhibits high accuracy and robustness.Shimizu [15,16] proposed a camera depth estimation method that utilizes the parallel plane refraction effect.However, this method requires a complex calibration setup.Gong [17] proposed a 3D reconstruction model based on plane refraction correction.This method applies to the scene of any num ber and direction of plate glass.He also proposed a method for flexibly determining the This paper presents a comprehensive solution for binocular measurement under the refraction of plate glass.The specific contributions are as follows: (1) The non-parallel plane refractive camera imaging model is established based on dynamic virtual focal length and the Rodriguez formula.(2) The main problems are the epipolar constraint failure and triangulation failure under refraction images.This paper proposes an epipolar constraint model and triangulation method based on independent refraction planes.
The remainder of this paper is structured as follows.Section 2 introduces the related work.Section 3 describes, in detail, the model for image distortion correction under non-parallel plane refraction, the model for imaging epipolar constraint on independent refractive surfaces, and the triangulation method for independent refraction planes.Section 4 validates the effectiveness of the proposed method by conducting a series of physical experiments.Section 5 discusses the relevant issues.Section 6 presents the conclusion.

Related Work
The refraction effect of glass severely affects the accuracy of visual measurements.Scholars have conducted in-depth research on the characterization and elimination of refractive distortion.In the calibration of visual measurement system parameters under planar refraction, Treibitz [12] studied the trajectory of camera viewpoints under planar refraction environments.In addition, Treibitz proposed an imaging model for a camera based on dynamic virtual focal length.The measurement accuracy of this model is significantly greater compared to single-viewpoint models.However, this model requires prior knowledge of the depth information of the target object.The camera's optical axis must also be parallel to the normal vector of the refractive surface.Agrawal [13] established an imaging model for multiplane refraction.The multi-plane refraction imaging model has been proven to have strong similarities to the single-plane refraction imaging model.
Chen [14] introduced a calibration method for determining the normal vector of the refractive surface as well as the thickness of the glass.The experiment demonstrates that this method exhibits high accuracy and robustness.Shimizu [15,16] proposed a camera depth estimation method that utilizes the parallel plane refraction effect.However, this method requires a complex calibration setup.Gong [17] proposed a 3D reconstruction model based on plane refraction correction.This method applies to the scene of any number and direction of plate glass.He also proposed a method for flexibly determining the normal vector of the refractive surface without any auxiliary devices.Ke [18] examined how the thickness of a glass plate affects the resulting image.He used a target extremely close to the plate glass to obtain the normal vector of the refractive surface and the distance between the plate glass and the camera's optical center.Huang S [19] proposed a multi-camera calibration method based on planar refraction.Multiple cameras can capture calibration images from different angles.This calibration method has high accuracy.The basic geometric relationships based on plane refraction can be used to eliminate refraction errors.Shortis et al. [20] employ corrective lenses or dome glass to eliminate refractive distortion, guaranteeing that the light rays from every point are perpendicular to the refractive surface.This situation can still use a single viewpoint model.However, this method requires extremely high accuracy.In practical applications, manufacturing errors and installation deviations of lenses often lead to other refractive errors.
Additionally, scholars have conducted research on stereo matching in binocular vision under plane refraction.Yamashita [21] considered the refraction effect and proposed a model for underwater 3D reconstruction.However, he did not consider the issue of polar constraint failure under refractive effects.Huang et al. [22] proposed a multi-plane refractive imaging model.He proposed the theory of dynamic epipolar constraint, which solved the problem that the epipolar of stereo matching under plane refraction is a curve.This method verifies the accuracy of matching points.However, the camera's optical axis remains parallel to the normal vector of the refractive surface.Gedge [23] studied the effect of planar refraction on underwater imaging and stereo matching.He proposed a 3D reconstruction model when the camera's optical axis is not parallel to the normal vector of the refractive surface, and the curve equations for the epipolar constraint in stereo matching are calculated.
In summary, a situation of approximate correction in the research of visual measurement methods under plane refraction remains.Many scholars have built plane refraction imaging models that are too complex, with too many introduced external parameters.The model is not concise and requires large computational efforts.
This paper investigates the problem of error correction for vision measurement systems considering plane refraction and presents a complete approach to visual measurements.This method includes image distortion correction, refraction plane normal vector solving, polar constraint modeling of refraction images, triangulation modeling of refraction imaging, and refraction error measurement.

Methodology
The traditional pinhole camera model based on perspective imaging leads to vision measurement errors due to refraction distortion.A new refractive imaging model needs to be established to eliminate refractive errors.Ideally, the camera's optical axis is parallel to the normal vector of the refractive plane.Due to installation and manufacturing errors, the camera's optical axis is not guaranteed to be parallel to the normal vector of the refraction plane.An image distortion correction model is established based on the concept of dynamic virtual focal length in a parallel plane.It complements the non-parallel plane camera rotation model to achieve correction.The main flow of this paper is shown in Figure 2.
normal vector of the refractive surface without any auxiliary devices.Ke [18] examined how the thickness of a glass plate affects the resulting image.He used a target extremely close to the plate glass to obtain the normal vector of the refractive surface and the distance between the plate glass and the camera's optical center.Huang S [19] proposed a multicamera calibration method based on planar refraction.Multiple cameras can capture calibration images from different angles.This calibration method has high accuracy.The basic geometric relationships based on plane refraction can be used to eliminate refraction errors.Shortis et al. [20] employ corrective lenses or dome glass to eliminate refractive distortion, guaranteeing that the light rays from every point are perpendicular to the refractive surface.This situation can still use a single viewpoint model.However, this method requires extremely high accuracy.In practical applications, manufacturing errors and installation deviations of lenses often lead to other refractive errors.
Additionally, scholars have conducted research on stereo matching in binocular vision under plane refraction.Yamashita [21] considered the refraction effect and proposed a model for underwater 3D reconstruction.However, he did not consider the issue of polar constraint failure under refractive effects.Huang et al. [22] proposed a multi-plane refractive imaging model.He proposed the theory of dynamic epipolar constraint, which solved the problem that the epipolar of stereo matching under plane refraction is a curve.This method verifies the accuracy of matching points.However, the camera's optical axis remains parallel to the normal vector of the refractive surface.Gedge [23] studied the effect of planar refraction on underwater imaging and stereo matching.He proposed a 3D reconstruction model when the camera's optical axis is not parallel to the normal vector of the refractive surface, and the curve equations for the epipolar constraint in stereo matching are calculated.
In summary, a situation of approximate correction in the research of visual measurement methods under plane refraction remains.Many scholars have built plane refraction imaging models that are too complex, with too many introduced external parameters.The model is not concise and requires large computational efforts.
This paper investigates the problem of error correction for vision measurement systems considering plane refraction and presents a complete approach to visual measurements.This method includes image distortion correction, refraction plane normal vector solving, polar constraint modeling of refraction images, triangulation modeling of refraction imaging, and refraction error measurement.

Methodology
The traditional pinhole camera model based on perspective imaging leads to vision measurement errors due to refraction distortion.A new refractive imaging model needs to be established to eliminate refractive errors.Ideally, the camera's optical axis is parallel to the normal vector of the refractive plane.Due to installation and manufacturing errors, the camera's optical axis is not guaranteed to be parallel to the normal vector of the refraction plane.In Figure 3, the camera's optical axis is not parallel to the normal vector n of the refractive plane.Based on the expansion offset effect of plane refraction, the normal vector of the refraction plane of the plate glass is determined.In Figure 3, the camera's optical axis is not parallel to the normal vector n of the refractive plane.Based on the expansion offset effect of plane refraction, the normal vector of the refraction plane of the plate glass is determined.The camera coordinate system is established with O as the origin.The Mg1(ug1, vg1) and Mg2(ug2, vg2) points are the refracted imaging points.M1(u1, v1) and M2(u2, v2) are the image points without refraction.Mc(uc, vc) is the distortion center point.One can maintain the calibration board posture and take two separate images, one directly imaged and the other imaged through glass.These two images can then be used to get the refraction plane normal vector.As depicted in Figure 3a, the vector McO is parallel to the normal vector n of the refraction plane in the camera coordinate system.As depicted in Figure 3b, the intersection of two straight lines in the ideal state is the distortion center point.The intersection point between the normal vector of the glass refractive plane passing through the optical center and the image plane is the distortion center.Therefore, different glass attitudes can cause significant differences in the location of distortion centers.In practical implementation, it is necessary to calculate multiple lines to ensure the accuracy of the distortion center.The point closest to all the lines is taken as the distortion center.
The equation of the line connecting the refracted image point and the non-refracted image point is: The distortion center Mc(uc, vc) is the point closest to all straight lines, which can be solved by linear least square method [23].The objective function is established as follows: The normal vector of the refraction plane of the plate glass is The camera coordinate system is established with O as the origin.The M g1 (u g1 , v g1 ) and M g2 (u g2 , v g2 ) points are the refracted imaging points.M 1 (u 1 , v 1 ) and M 2 (u 2 , v 2 ) are the image points without refraction.M c (u c , v c ) is the distortion center point.One can maintain the calibration board posture and take two separate images, one directly imaged and the other imaged through glass.These two images can then be used to get the refraction plane normal vector.As depicted in Figure 3a, the vector M c O is parallel to the normal vector n of the refraction plane in the camera coordinate system.As depicted in Figure 3b, the intersection of two straight lines in the ideal state is the distortion center point.The intersection point between the normal vector of the glass refractive plane passing through the optical center and the image plane is the distortion center.Therefore, different glass attitudes can cause significant differences in the location of distortion centers.In practical implementation, it is necessary to calculate multiple lines to ensure the accuracy of the distortion center.The point closest to all the lines is taken as the distortion center.
The equation of the line connecting the refracted image point and the non-refracted image point is: The distortion center M c (u c , v c ) is the point closest to all straight lines, which can be solved by linear least square method [23].The objective function is established as follows: arg min The normal vector of the refraction plane of the plate glass is

Modeling of Camera Rotation under Non-Parallel Plane Refraction
The image distortion correction model under parallel plane refraction is established based on the dynamic virtual focal length.After determining the refraction plane normal vector n, the position of the inclined plate glass in the camera coordinate system can be accurately described.It can establish a camera rotation model under non-parallel plane refraction.In Figure 4, the virtual camera coordinate system (blue dotted line in the figure ) is established with point O as the origin.The tilt of plate glass can be viewed as a rotation centered on the rotation axis past the origin O.

Modeling of Camera Rotation under Non-Parallel Plane Refraction
The image distortion correction model under parallel plane refraction is established based on the dynamic virtual focal length.After determining the refraction plane normal vector n, the position of the inclined plate glass in the camera coordinate system can be accurately described.It can establish a camera rotation model under non-parallel plane refraction.In Figure 4, the virtual camera coordinate system (blue dotted line in the figure) is established with point O as the origin.The tilt of plate glass can be viewed as a rotation centered on the rotation axis past the origin O.The unit vector v of the rotation axis can be found from the camera optical axis direction vector w and the refraction plane normal vector n as follows: The angle of rotation is as follows: The virtual camera coordinate system can be obtained by rotating the real camera coordinate system around the rotation axis v using λ angle.First, the Qz(xz, yz, zz) of the camera coordinate system is transformed into the virtual camera coordinate system.It follows from the Rodriguez equation: where qz = [xz, yz, zz] T , qz' = [qz1, qz2, qz3].The qz' is the coordinate of Qz in the virtual camera coordinate system.In the virtual camera coordinate system, the intersection of the line OQz with the virtual image plane is the image point Q2(x2, y2, z2).
The following is from Equation ( 7): The unit vector v of the rotation axis can be found from the camera optical axis direction vector w and the refraction plane normal vector n as follows: The angle of rotation is as follows: The virtual camera coordinate system can be obtained by rotating the real camera coordinate system around the rotation axis v using λ angle.First, the Q z (x z , y z , z z ) of the camera coordinate system is transformed into the virtual camera coordinate system.It follows from the Rodriguez equation: where q z = [x z , y z , z z ] T , q z ′ = [q z1 , q z2 , q z3 ].The q z ′ is the coordinate of Q z in the virtual camera coordinate system.In the virtual camera coordinate system, the intersection of the line OQ z with the virtual image plane is the image point Q 2 (x 2 , y 2 , z 2 ).
The following is from Equation ( 7): In conclusion, the coordinates of the image point in the virtual camera coordinate system can be obtained.The correction for non-parallelism between the camera optical axis and the normal vector of the refracting surface can be realized.

Modeling of Camera Imaging under Non-Parallel Plane Refraction
In Figure 5, the intersection point between the camera optical axis and the measurement plane in the camera coordinate system is Q 0 .Q 1 is any point on the measurement plane.The image point of Q 1 refracted by the plane on the virtual imaging plane is Q 2 .The main point of the virtual camera is Q 4 .The incident angle is θ a1 .The distance between line OQ 2 and line L is denoted as d w1 .In the camera coordinate system, the coordinates of Q 0 are (0, 0, h).In the virtual camera coordinate system, the coordinates of Sensors 2024, 24, 66 6 of 20 Q 3 , and Q 4 are Q 0 (x 0 , y 0 , z 0 ), Q 1 (x 1 , y 1 , z 1 ), Q 2 (x 2 , y 2 , z 2 ), Q 3 (x 3 , y 3 , z 3 ), and Q 4 (x 4 , y 4 , z 4 ).The coordinates of Q 0 and Q 2 in the virtual camera coordinate system can be determined based on the camera rotation model.The coordinates of Q 4 in the virtual camera coordinate system are (0, 0, −ƒ).In the camera coordinate system, the normal vector of the refractive surface is n = [n 1 , n 2 , n 3 ].The normal vector of the measurement plane is w = [0, 0, 1].The w can be corrected by the camera rotation model to obtain the normal vector of the measurement plane in the virtual camera coordinate system as In Figure 5, the intersection point between the camera optical axis and the measurement plane in the camera coordinate system is Q0.Q1 is any point on the measurement plane.The image point of Q1 refracted by the plane on the virtual imaging plane is Q2.The main point of the virtual camera is Q4.The incident angle is θa1.The distance between line OQ2 and line L is denoted as dw1.In the camera coordinate system, the coordinates of Q0 are (0, 0, h).In the virtual camera coordinate system, the coordinates of Q0, Q1, Q2, Q3, and Q4 are Q0(x0, y0, z0), Q1(x1, y1, z1), Q2(x2, y2, z2), Q3(x3, y3, z3), and Q4(x4, y4, z4).The coordinates of Q0 and Q2 in the virtual camera coordinate system can be determined based on the camera rotation model.The coordinates of Q4 in the virtual camera coordinate system are (0, 0, −ƒ).In the camera coordinate system, the normal vector of the refractive surface is n = [n1, n2, n3].The normal vector of the measurement plane is w= [0, 0, 1].The w can be corrected by the camera rotation model to obtain the normal vector of the measurement plane in the virtual camera coordinate system as w1= [−n1, −n2, n3].In the virtual camera coordinate system, the normal vector w1 of the measurement plane and a certain point Q0(x0, y0, z0) in the measurement plane are known.From any point Q(x, y, z) in the measurement plane: The point Q coordinate satisfies the following: After this, the equation of the measurement plane can be obtained as follows: where In the virtual camera coordinate system, the distances Q2Q4 and Q3Q4 on the virtual imaging surface are r2 and r3, respectively.
This can be obtained from the geometric relationship in Figure 5: In the virtual camera coordinate system, the normal vector w 1 of the measurement plane and a certain point Q 0 (x 0 , y 0 , z 0 ) in the measurement plane are known.From any point Q(x, y, z) in the measurement plane: The point Q coordinate satisfies the following: After this, the equation of the measurement plane can be obtained as follows: where In the virtual camera coordinate system, the distances Q 2 Q 4 and Q 3 Q 4 on the virtual imaging surface are r 2 and r 3 , respectively.
This can be obtained from the geometric relationship in Figure 5: Since the line Q 2 Q 4 is collinear with the line Q 3 Q 4 , it follows: Sensors 2024, 24, 66 7 of 20 By combining Equations ( 12) and (13), x 3 and y 3 can be obtained.The coordinates of Q 3 can be obtained.
In the virtual camera coordinate system, the line OQ 2 is parallel to the line L. In Figure 5, the equation for line L is as follows: The intersection point between the line L and the measurement plane is Q 1 .The coordinates of Q 1 in the virtual camera coordinate system can be obtained by combining Equations ( 11) and (14).Its converted coordinate in the camera coordinate system is as follows: Here, q 1 ′ = [x 1 , y 1 , z 1 ] T , q 1 = [q 11 , q 12 , q 13 ], (q 11 , q 12 , q 13 ) are the coordinates of Q 1 in the camera coordinate system.

Epipolar Constraint Modeling of Binocular Vision System under Nonparallel Plane Refraction
Based on the geometric relationship of plane refraction, the epipolar constraint model of the binocular vision system under non-parallel plane refraction is derived.The virtual camera coordinate system is introduced, and the conversion relationship between the camera coordinate system and the virtual camera coordinate system is shown in Figure 5.The coordinate system of the left camera is denoted as O l X l Y l Z l .The coordinate system of the left virtual camera is denoted as O l X cl Y cl Z cl .The coordinate system for the right camera is denoted as O r X r Y r Z r .The coordinate system for the right virtual camera is denoted as O r X cr Y cr Z cr .The optical centers of the two cameras are O l and O r .The base distance between the two optical centers is a.The focal length of the cameras is ƒ.The left and right virtual imaging planes are maintained parallel to the plate glass.
In Figure 6b, the two camera refraction planes are independent as shown.Based on the virtual camera coordinate system, the epipolar constraint model of independent refraction plane imaging is established.The distortion centers of the left and right virtual imaging planes are denoted as F cl and F cr , respectively.The thickness of plate glass is d.O l is the origin of the left virtual camera coordinate system O l X cl Y cl Z cl .The left virtual camera coordinate system serves as the world coordinate system.F 1 (x 1 , y 1 , z 1 ) is any point on the left virtual imaging plane.The line O l F 1 intersects the front of the glass at point F 2 , and after being refracted by the left plate glass, intersects the back of the glass at point F 3 .After this, take any point F h (x h , y h , z h ) on the refraction ray and set z h = h.The inverse extension of the line F 3 F h intersects the left virtual imaging plane at point F 4 .Set the refraction planes in the left and right cameras as Π 1 , Π 2 .As a result, the intersection of the refraction plane Π 1 with the left virtual imaging plane is the line F 1 F 4 .The distortion center F cl (x cl , y cl , z cl ) of the left virtual imaging plane is on line F 1 F 4 .The coordinates of F 4 is (x 4 ,y 4 ,z 4 ).The incident angle is θ a2 .The lengths of lines F cl F 1 and F cl F 4 are R 1 and R 2 , respectively.The distance between the incident and refraction rays is d w2 .
Π1 with the left virtual imaging plane is the line F1F4.The distortion center Fcl (xcl, ycl, zcl) of the left virtual imaging plane is on line F1F4.The coordinates of F4 is (x4,y4,z4).The incident angle is θa2.The lengths of lines FclF1 and FclF4 are R1 and R2, respectively.The distance between the incident and refraction rays is dw2.From the geometric relationship in Figure 6: where the coordinate of Fcl is (0, 0, ƒ).After calculating the coordinates of point F4, then it is obtained from the point F4 and Fh are co-linear: From Equations ( 16) and ( 17), the coordinates of points F4 and Fh can be obtained in the same way.In the right virtual camera coordinate system, the coordinate of the right camera optical center on the left virtual camera coordinate system is Or(a, 0, 0).After this, it can be obtained as F9(x9, y9, z9): From the geometric relationship in Figure 6: where the coordinate of F cl is (0, 0, ƒ).After calculating the coordinates of point F 4 , then it is obtained from the point F 4 and F h are co-linear: From Equations ( 16) and ( 17), the coordinates of points F 4 and F h can be obtained in the same way.In the right virtual camera coordinate system, the coordinate of the right camera optical center on the left virtual camera coordinate system is O r (a, 0, 0).After this, it can be obtained as F 9 (x 9 , y 9 , z 9 ): The extension of the line F h F 5 intersects the right virtual imaging surface at point F 8 .The points F 7 , F 8 , F 9 , and the center of distortion F cr are collinear, and the line of this collinearity is L r1 .Therefore, it can be obtained that: The solution for the coordinates of the distortion center F cr in Equation ( 19) can be referred to Section 3.1.1.After calculating the coordinates of F 8 (x 8 , y 8 , z 8 ), the coordinates of F 7 (x 7 , y 7 , z 7 ) can be obtained in the same way.Any point on the left virtual imaging plane can obtain the corresponding epipolar line on the right virtual imaging plane.

Triangulation Modeling of Binocular Vision System under Non-Parallel Plane Refraction
The traditional triangulation method based on a pinhole camera model is ineffective in binocular vision measurement due to the plane refraction effect.This section analyzes the binocular vision measurement system under independent refractive planes.The image points of any space point P on the left and right imaging planes are P l and P r , respectively.The refraction rays O l P l and O r P r are not necessarily in the same plane.This leads to the failure of the conventional triangulation method based on the pinhole camera model.The camera rotation model and the camera imaging model under non-parallel plane refraction are proposed in Section 3.1.2.The model corrects the refraction error under non-parallel plane refraction.On this basis, this paper establishes a binocular vision system triangulation model under independent refractive planes.
In Figure 7, when the two refractive surfaces are independent, the camera coordinate system should be first corrected with rotation.A triangulation model of a binocular vision system with an independent refraction plane is established based on the virtual camera coordinate system.The distortion centers of the left and right virtual imaging surfaces are W cl and W cr , respectively.The image points of any space point W on the left and right virtual imaging surfaces are points W 1 and W 7 , respectively.The reverse extension of line W 3 W intersects with the left virtual imaging surface at point W 4 .The reverse extension of line W 5 W intersects with the right virtual imaging surface at point W 8 .Triangulation modeling of binocular vision systems with independent refraction planes is based on a virtual camera coordinate system.From the previous analysis, points W, W 4 , and W 8 are coplanar.
The extension of the line FhF5 intersects the right virtual imaging surface at point F8.The points F7, F8, F9, and the center of distortion Fcr are collinear, and the line of this collinearity is Lr1.Therefore, it can be obtained that: The solution for the coordinates of the distortion center Fcr in Equation ( 19) can be referred to Section 3.1.1.After calculating the coordinates of F8(x8, y8, z8), the coordinates of F7(x7, y7, z7) can be obtained in the same way.Any point on the left virtual imaging plane can obtain the corresponding epipolar line on the right virtual imaging plane.

Triangulation Modeling of Binocular Vision System under Non-Parallel Plane Refraction
The traditional triangulation method based on a pinhole camera model is ineffective in binocular vision measurement due to the plane refraction effect.This section analyzes the binocular vision measurement system under independent refractive planes.The image points of any space point P on the left and right imaging planes are Pl and Pr, respectively.The refraction rays OlPl and OrPr are not necessarily in the same plane.This leads to the failure of the conventional triangulation method based on the pinhole camera model.The camera rotation model and the camera imaging model under non-parallel plane refraction are proposed in Section 3.1.2.The model corrects the refraction error under non-parallel plane refraction.On this basis, this paper establishes a binocular vision system triangulation model under independent refractive planes.
In Figure 7, when the two refractive surfaces are independent, the camera coordinate system should be first corrected with rotation.A triangulation model of a binocular vision system with an independent refraction plane is established based on the virtual camera coordinate system.The distortion centers of the left and right virtual imaging surfaces are Wcl and Wcr, respectively.The image points of any space point W on the left and right virtual imaging surfaces are points W1 and W7, respectively.The reverse extension of line W3W intersects with the left virtual imaging surface at point W4.The reverse extension of line W5W intersects with the right virtual imaging surface at point W8.Triangulation modeling of binocular vision systems with independent refraction planes is based on a virtual camera coordinate system.From the previous analysis, points W, W4, and W8 are coplanar.Triangular measurement of a binocular vision system under independent refractive surfaces can be achieved by using ray coplanarity as a constraint.The coordinates of points W 4 and W 8 can be solved from Equation (16).

Three-Dimensional Measurement Solution Process under Plane Refraction
The main flow of this paper is shown in Figure 8.The refraction plane normal vector n is solved by capturing before and after images of the refraction at the same attitude of the calibration plate.The camera rotation model is established based on the Rodriguez formula.The camera coordinate system is rotated and corrected to obtain the virtual camera coordinate system.The coordinates of Q 1 in the virtual space coordinate system can be obtained by combining the equations of the measurement plane and the line L in the virtual camera coordinate system.The coordinates in the camera coordinate system are obtained from Equation (15).Both the left and right imaging planes undergo rotation correction, and binocular measurement is accomplished by employing the epipolar constraint model and the triangulation model under refraction effects.
The corner coordinates of the checkerboard image under the refraction effect are known, and the corresponding coordinate points in the virtual coordinate system can be obtained through simultaneous Equations ( 12) and (13).In Figure 7, the W 1 and W 7 coordinate points under the virtual coordinate system can be obtained.Further, the point coordinates of W 4 and W 8 in the virtual coordinate system can be solved by Equation ( 16) in Section 3.2.Finally, binocular measurements are realized using the refraction plane imaging epipolar constraint modeling and the triangulation model.Triangular measurement of a binocular vision system under independent refractive surfaces can be achieved by using ray coplanarity as a constraint.The coordinates of points W4 and W8 can be solved from Equation (16).

Three-Dimensional Measurement Solution Process under Plane Refraction
The main flow of this paper is shown in Figure 8.The refraction plane normal vector n is solved by capturing before and after images of the refraction at the same attitude of the calibration plate.The camera rotation model is established based on the Rodriguez formula.The camera coordinate system is rotated and corrected to obtain the virtual camera coordinate system.The coordinates of Q1 in the virtual space coordinate system can be obtained by combining the equations of the measurement plane and the line L in the virtual camera coordinate system.The coordinates in the camera coordinate system are obtained from Equation (15).Both the left and right imaging planes undergo rotation correction, and binocular measurement is accomplished by employing the epipolar constraint model and the triangulation model under refraction effects.
The corner coordinates of the checkerboard image under the refraction effect are known, and the corresponding coordinate points in the virtual coordinate system can be obtained through simultaneous Equations ( 12) and (13).In Figure 7, the W1 and W7 coordinate points under the virtual coordinate system can be obtained.Further, the point coordinates of W4 and W8 in the virtual coordinate system can be solved by Equation ( 16) in Section 3.2.Finally, binocular measurements are realized using the refraction plane imaging epipolar constraint modeling and the triangulation model.

Experimentation and Analysis
A series of physical experiments were conducted to validate the effectiveness of the proposed method.Section 4.1 presented the experimental setup.Section 4.2 conducted experimental verification of camera imaging models under non-parallel plane refraction.Section 4.3 studied the 3D measurement experiment of binocular vision under plane refraction.The camera calibration was performed using the most common Zhang's calibration method [24,25].

Experimental Setups
To validate the method proposed in this paper, a set of experimental platforms for the vision scheme of the tool change robot was built [26].This is shown in Figure 9.It

Experimentation and Analysis
A series of physical experiments were conducted to validate the effectiveness of the proposed method.Section 4.1 presented the experimental setup.Section 4.2 conducted experimental verification of camera imaging models under non-parallel plane refraction.Section 4.3 studied the 3D measurement experiment of binocular vision under plane refraction.The camera calibration was performed using the most common Zhang's calibration method [24,25].

Experimental Setups
To validate the method proposed in this paper, a set of experimental platforms for the vision scheme of the tool change robot was built [26].This is shown in Figure 9.It consists of a binocular camera, plate glass, target, motion module, and robot.Details are as follows: consists of a binocular camera, plate glass, target, motion module, and robot.Details are as follows:

Experimental Validation of Camera Imaging Model under Non-Parallel Plane Refraction
This section verifies the positioning accuracy and measurement accuracy of the camera imaging model (hereinafter referred to as the refractive imaging model) under nonparallel plane refraction.Each set of experiments kept the last target attitude fixed, and images without and with plate glass were collected separately.To compare the positioning error and measurement error under different distortion center positions, the experimental section conducted two sets of experiments with different glass postures: (a) Glass attitude 1 and (b) Glass attitude 2. Figures 10-14 show experimental data under two different glass postures.
Figure 10 shows the distribution of image deviations under non-parallel plane refraction.Arrows indicate the direction of distortion.As shown in Figure 10a, the minimum deviation of the image points is 0.5 pixels, and the maximum deviation is 9.2 pixels.As shown in Figure 10b, the minimum deviation of the image points is 2.3 pixels, and the maximum deviation is 12.1 pixels.From the figure, it is evident that the distortion center is not the main point of the camera.The distortion center is shifted because the camera's optical axis is not parallel to the normal vector of the refraction plane of the plate glass.

Experimental Validation of Camera Imaging Model under Non-Parallel Plane Refraction
This section verifies the positioning accuracy and measurement accuracy of the camera imaging model (hereinafter referred to as the refractive imaging model) under non-parallel plane refraction.Each set of experiments kept the last target attitude fixed, and images without and with plate glass were collected separately.To compare the positioning error and measurement error under different distortion center positions, the experimental section conducted two sets of experiments with different glass postures: (a) Glass attitude 1 and (b) Glass attitude 2. Figures 10-14 show experimental data under two different glass postures.
Figure 10 shows the distribution of image deviations under non-parallel plane refraction.Arrows indicate the direction of distortion.As shown in Figure 10a, the minimum deviation of the image points is 0.5 pixels, and the maximum deviation is 9.2 pixels.As shown in Figure 10b, the minimum deviation of the image points is 2.3 pixels, and the maximum deviation is 12.1 pixels.From the figure, it is evident that the distortion center is not the main point of the camera.The distortion center is shifted because the camera's optical axis is not parallel to the normal vector of the refraction plane of the plate glass.
In Figure 11, the cyan point is the distortion center point.The distortion centers of the two experimental groups obtained by the least squares method are (953.07,432.1) and (92.2, 395.81), respectively.In the two experiments shown in Figure 11, due to the difference in glass placement attitude, there was a significant change in the intersection point between the normal vector passing through the optical center and the image plane.The normal vectors of the refraction plane for the two sets of experiments are n 1 = (0.1268, −0.0284, 1) and n 2 = (−0.2262,−0.0431, 1).The camera rotation model corrects for the non-parallelism of the camera optical axis and the normal vector of the refraction plane.Figure 12 shows the image points before and after rotational correction with glass refraction.In Figure 11, the cyan point is the distortion center point.The distortion centers of the two experimental groups obtained by the least squares method are (953.07,432.1) and (92.2, 395.81), respectively.In the two experiments shown in Figure 11, due to the difference in glass placement attitude, there was a significant change in the intersection point between the normal vector passing through the optical center and the image plane.The normal vectors of the refraction plane for the two sets of experiments are n1 = (0.1268, −0.0284, 1) and n2 = (−0.2262,−0.0431, 1).The camera rotation model corrects for the nonparallelism of the camera optical axis and the normal vector of the refraction plane.Figure 12 shows the image points before and after rotational correction with glass refraction.In Figure 11, the cyan point is the distortion center point.The distortion centers of the two experimental groups obtained by the least squares method are (953.07,432.1) and (92.2, 395.81), respectively.In the two experiments shown in Figure 11, due to the difference in glass placement attitude, there was a significant change in the intersection point between the normal vector passing through the optical center and the image plane.The normal vectors of the refraction plane for the two sets of experiments are n1 = (0.1268, −0.0284, 1) and n2 = (−0.2262,−0.0431, 1).The camera rotation model corrects for the nonparallelism of the camera optical axis and the normal vector of the refraction plane.Figure 12 shows the image points before and after rotational correction with glass refraction.In Figure 11, the cyan point is the distortion center point.The distortion centers of the two experimental groups obtained by the least squares method are (953.07,432.1) and (92.2, 395.81), respectively.In the two experiments shown in Figure 11, due to the difference in glass placement attitude, there was a significant change in the intersection point between the normal vector passing through the optical center and the image plane.The normal vectors of the refraction plane for the two sets of experiments are n1 = (0.1268, −0.0284, 1) and n2 = (−0.2262,−0.0431, 1).The camera rotation model corrects for the nonparallelism of the camera optical axis and the normal vector of the refraction plane.Figure 12 shows the image points before and after rotational correction with glass refraction.Then, the measurement accuracy of the refraction imaging model is verified.To comprehensively reflect the impact of plane refraction on measurement accuracy, pinhole camera models and refractive imaging models are used to measure line segments L1, L2 L3, R1, R2, and R3 on the two-dimensional measurement plane.The measured line segments are shown in Figure 14.In Table 1, the length of the line segment measured without glass is taken as the true value.The measurements of the pinhole camera model and the refraction imaging mode are compared with the true value.The measurement results of two sets of experiments are shown in Tables 1 and 2. The RMSE of the measurement errors using the pinhole camera model are 2.3821 mm and 2.4037 mm.The RMSE of the measurement errors using the refraction imaging model are 0.2130 mm and 0.2136 mm, respectively.From the above analysis, it can be concluded that the refraction imaging model can significantly improve measurement accuracy.Then, the measurement accuracy of the refraction imaging model is verified.To comprehensively reflect the impact of plane refraction on measurement accuracy, pinhole camera models and refractive imaging models are used to measure line segments L1, L2, L3, R1, R2, and R3 on the two-dimensional measurement plane.The measured line segments are shown in Figure 14.
in Figure 13.The maximum localization errors of the pinhole camera model ar mm and 3.1139 mm, respectively.The maximum localization errors of the refracte ing model are 0.2428 mm and 0.3394 mm, respectively.The RMSE of positionin using the pinhole camera model are 1.3441 mm and 1.9937 mm.The RMSE of loca errors using the refraction imaging model are 0.1135 mm and 0.1484 mm, respect Then, the measurement accuracy of the refraction imaging model is verified.prehensively reflect the impact of plane refraction on measurement accuracy, camera models and refractive imaging models are used to measure line segment L3, R1, R2, and R3 on the two-dimensional measurement plane.The measured l ments are shown in Figure 14.In Table 1, the length of the line segment measured without glass is taken as value.The measurements of the pinhole camera model and the refraction imagin are compared with the true value.The measurement results of two sets of experim shown in Tables 1 and 2. The RMSE of the measurement errors using the pinhole model are 2.3821 mm and 2.4037 mm.The RMSE of the measurement errors u refraction imaging model are 0.2130 mm and 0.2136 mm, respectively.From th analysis, it can be concluded that the refraction imaging model can significantly i measurement accuracy.In Table 1, the length of the line segment measured without glass is taken as the true value.The measurements of the pinhole camera model and the refraction imaging model are compared with the true value.The measurement results of two sets of experiments are shown in Tables 1 and 2. The RMSE of the measurement errors using the pinhole camera model are 2.3821 mm and 2.4037 mm.The RMSE of the measurement errors using the refraction imaging model are 0.2130 mm and 0.2136 mm, respectively.From the above analysis, it can be concluded that the refraction imaging model can significantly improve measurement accuracy.In Table 2, the actual dimensions between the target points of the marker are taken as the real values.It can evaluate the actual measurement accuracy of the proposed method.The RMSE of the measurement errors using the pinhole camera model were 2.2827 mm and 2.4179 mm, while the RMSE of the measurement errors using the refraction imaging model were 0.3929 mm and 0.2627 mm, respectively.From the above analysis, it can be concluded that the refraction imaging model can significantly improve measurement accuracy.

Experiments on 3D Measurement of Binocular Vision under Plane Refraction
First, the Zhang calibration method calibrates the internal and external parameters of the two cameras.The method proposed solves the refraction plane normal vector.Table 3 demonstrates the calibration parameters.Where (f x , f y ) is the camera focal length, (u 0 , v 0 ) is the coordinates of the camera main point, kc is the distortion coefficient, M R is the rotation matrix, M T is the translation matrix, (u c , v c ) is the distortion center, and n is the normal vector of the refraction plane of the plane glass.Find the center of distortion and the normal vector of the refraction plane.In Figure 15, the cyan point is the center distortion point for the left and right cameras.In the experiment, keeping the optical axis of the left and right cameras not parall to the normal vector of the refractive plane, the cameras collect one image with the pla glass and one image without the plate glass.
To utilize a binocular vision system for 3D measurement, it is essential to initial conduct feature point matching between the left and right images.However, Figure 1 demonstrates that the polar constraint principle of traditional binocular vision systems rendered ineffective due to the influence of the plane refraction effect.The green line rep resents the limit constraint.In Figure 17

Experimental Verification of Epipolar Constraint Model for Independent Refractive Plane Imaging
After calibration of the internal and external parameters of the binocular vision system is completed, the independent refraction plane imaging epipolar constraint model is experimentally validated.It has the same basic principles as the shared refraction plane imaging epipolar constraint model and has a wider range of applications.Therefore, this paper only analyzes the epipolar constraint model of independent refractive plane imaging.
In the experiment, keeping the optical axis of the left and right cameras not parallel to the normal vector of the refractive plane, the cameras collect one image with the plate glass and one image without the plate glass.
To utilize a binocular vision system for 3D measurement, it is essential to initially conduct feature point matching between the left and right images.However, Figure 16 demonstrates that the polar constraint principle of traditional binocular vision systems is rendered ineffective due to the influence of the plane refraction effect.The green line represents the limit constraint.In Figure 17, Position 1 indicates a large difference in the longitudinal tilt angle of the left and right plate glass.In Figure 17, Position 2 indicates that the left and right plate glass are parallel.
In Figure 17, when the tilt angle difference between the left and right plate glass in the longitudinal direction is large, the matching error without refraction is less than 1 pixel, and its RMSE is 0.2343 pixels.The RMSE of the matching error in the presence of refraction is 11.1107 pixels.The traditional principle of polar constraints is no longer applicable to the binocular vision systems affected by plane refraction.In Figure 17, when the tilt angle difference between the left and right plate glass in the longitudinal direction is large, the matching error without refraction is less than pixel, and its RMSE is 0.2343 pixels.The RMSE of the matching error in the presence o refraction is 11.1107 pixels.The traditional principle of polar constraints is no longer ap plicable to the binocular vision systems affected by plane refraction.But it's not that the polar constraint principle will fail as long as there is refraction When the left and right plate glass are nearly parallel in the longitudinal direction, th refraction has minimal impact on the coordinates v of the image points.The normal vecto of the refractive surface of the left plate glass is nl = (−0.1181,−0.2283, 1).The normal vecto of the refractive surface of the right plate glass is nr= (0.0063, −0.2396, 1).When the left and right plate glass are nearly parallel in the longitudinal direction, The matching error wit refraction is less than 1 pixel, and the RMSE of the matching error with refraction is 0.757 pixels.The traditional polar constraint principle remains applicable in this situation.In Figure 17, Position 1 indicates a large difference in the longitudinal tilt angle of the lef and right plate glass.In Figure 17, Position 2 indicates that the left and right plate glas are parallel.

Experimental Validation of a Triangulation Model for Binocular Vision System under Independent Refraction Planes
Finally, the proposed triangulation model of the binocular vision system under an independent refraction plane is experimentally validated.It compares depth-of-field er rors, positioning errors, and measurement errors for space points with and without con sidering refraction effects.Figure 18a shows the comparison of depth-of-field between th checkerboard corners before and after correction, where the depth-of-field calculated without refraction is taken as the true value.Figure 18b shows the comparison of th depth-of-field error before and after correction.The RMSE of the depth-of-field error be fore and after correction are 2.9902 mm and 0.3187 mm, respectively.In Figure 17, when the tilt angle difference between the left and right plate glass in the longitudinal direction is large, the matching error without refraction is less than 1 pixel, and its RMSE is 0.2343 pixels.The RMSE of the matching error in the presence o refraction is 11.1107 pixels.The traditional principle of polar constraints is no longer ap plicable to the binocular vision systems affected by plane refraction.But it's not that the polar constraint principle will fail as long as there is refraction When the left and right plate glass are nearly parallel in the longitudinal direction, the refraction has minimal impact on the coordinates v of the image points.The normal vecto of the refractive surface of the left plate glass is nl = (−0.1181,−0.2283, 1).The normal vecto of the refractive surface of the right plate glass is nr= (0.0063, −0.2396, 1).When the left and right plate glass are nearly parallel in the longitudinal direction, The matching error with refraction is less than 1 pixel, and the RMSE of the matching error with refraction is 0.7573 pixels.The traditional polar constraint principle remains applicable in this situation.In Figure 17, Position 1 indicates a large difference in the longitudinal tilt angle of the lef and right plate glass.In Figure 17, Position 2 indicates that the left and right plate glas are parallel.

Experimental Validation of a Triangulation Model for Binocular Vision System under Independent Refraction Planes
Finally, the proposed triangulation model of the binocular vision system under an independent refraction plane is experimentally validated.It compares depth-of-field er rors, positioning errors, and measurement errors for space points with and without con sidering refraction effects.Figure 18a shows the comparison of depth-of-field between the checkerboard corners before and after correction, where the depth-of-field calculated without refraction is taken as the true value.Figure 18b shows the comparison of the depth-of-field error before and after correction.The RMSE of the depth-of-field error be fore and after correction are 2.9902 mm and 0.3187 mm, respectively.But it's not that the polar constraint principle will fail as long as there is refraction.When the left and right plate glass are nearly parallel in the longitudinal direction, the refraction has minimal impact on the coordinates v of the image points.The normal vector of the refractive surface of the left plate glass is n l = (−0.1181,−0.2283, 1).The normal vector of the refractive surface of the right plate glass is n r = (0.0063, −0.2396, 1).When the left and right plate glass are nearly parallel in the longitudinal direction, The matching error with refraction is less than 1 pixel, and the RMSE of the matching error with refraction is 0.7573 pixels.The traditional polar constraint principle remains applicable in this situation.In Figure 17, Position 1 indicates a large difference in the longitudinal tilt angle of the left and right plate glass.In Figure 17, Position 2 indicates that the left and right plate glass are parallel.

Experimental Validation of a Triangulation Model for Binocular Vision System under Independent Refraction Planes
Finally, the proposed triangulation model of the binocular vision system under an independent refraction plane is experimentally validated.It compares depth-of-field errors, positioning errors, and measurement errors for space points with and without considering refraction effects.Figure 18a shows the comparison of depth-of-field between the checkerboard corners before and after correction, where the depth-of-field calculated without refraction is taken as the true value.Figure 18b shows the comparison of the depth-of-field error before and after correction.The RMSE of the depth-of-field error before and after correction are 2.9902 mm and 0.3187 mm, respectively.Figure 18b shows the comparison of positioning errors of checkerboard corner points before and after correction.The 3D coordinates calculated without refraction are used as the true spatial point coordinates.Figure 19 shows that the RMSE of the positioning errors before and after correction are 3.5661 mm and 0.3465 mm, respectively.

Discussion
This paper presents a complete vision measurement scheme applicable to the presence of plate glass refraction effects.The effectiveness of the scheme was verified through a series of experiments.

Analysis of the Proposed Method
In Section 3.2, the normal vector n of the refractive surface is used to describe the camera imaging model accurately when the cameraʹs optical axis is not parallel to the normal vector of the refractive plane.The distortion center is used to obtain the normal vector of the refraction plane of the plate glass based on the expansion offset effect of plane refraction.The established model for image distortion correction under plane refraction realizes the correction of image distortion.This model can not only eliminate image distortion but also intuitively reflect the impact of different incident angles and depth of field on measurement error.

Stability Experiment under Simulated Tunnel Construction Environments
Two types of stains shown in Figure 19 were attached to the outer surface of the plate glass to analyze how the tunnel construction environment affects the proposed plate glass calibration method.Refer to previous experiments on glass pollution.The first experiment (Non-pollution) is conducted under the ideal condition where glass is not polluted.The second experiment (Pollution A) simulated the condition of glass polluted by sewage.The third experiment (Pollution B) simulates the condition where glass is polluted by sludge and partially obstructed.According to the above experimental scheme, the calibration Figure 18b shows the comparison of positioning errors of checkerboard corner points before and after correction.The 3D coordinates calculated without refraction are used as the true spatial point coordinates.Figure 19 shows that the RMSE of the positioning errors before and after correction are 3.5661 mm and 0.3465 mm, respectively.Figure 18b shows the comparison of positioning errors of checkerboard corner points before and after correction.The 3D coordinates calculated without refraction are used as the true spatial point coordinates.Figure 19 shows that the RMSE of the positioning errors before and after correction are 3.5661 mm and 0.3465 mm, respectively.

Discussion
This paper presents a complete vision measurement scheme applicable to the presence of plate glass refraction effects.The effectiveness of the scheme was verified through a series of experiments.

Analysis of the Proposed Method
In Section 3.2, the normal vector n of the refractive surface is used to describe the camera imaging model accurately when the cameraʹs optical axis is not parallel to the normal vector of the refractive plane.The distortion center is used to obtain the normal vector of the refraction plane of the plate glass based on the expansion offset effect of plane refraction.The established model for image distortion correction under plane refraction realizes the correction of image distortion.This model can not only eliminate image distortion but also intuitively reflect the impact of different incident angles and depth of field on measurement error.

Stability Experiment under Simulated Tunnel Construction Environments
Two types of stains shown in Figure 19 were attached to the outer surface of the plate glass to analyze how the tunnel construction environment affects the proposed plate glass calibration method.Refer to previous experiments on glass pollution.The first experiment (Non-pollution) is conducted under the ideal condition where glass is not polluted.The second experiment (Pollution A) simulated the condition of glass polluted by sewage.The third experiment (Pollution B) simulates the condition where glass is polluted by sludge and partially obstructed.According to the above experimental scheme, the calibration

Discussion
This paper presents a complete vision measurement scheme applicable to the presence of plate glass refraction effects.The effectiveness of the scheme was verified through a series of experiments.

Analysis of the Proposed Method
In Section 3.2, the normal vector n of the refractive surface is used to describe the camera imaging model accurately when the camera's optical axis is not parallel to the normal vector of the refractive plane.The distortion center is used to obtain the normal vector of the refraction plane of the plate glass based on the expansion offset effect of plane refraction.The established model for image distortion correction under plane refraction realizes the correction of image distortion.This model can not only eliminate image distortion but also intuitively reflect the impact of different incident angles and depth of field on measurement error.

Stability Experiment under Simulated Tunnel Construction Environments
Two types of stains shown in Figure 19 were attached to the outer surface of the plate glass to analyze how the tunnel construction environment affects the proposed plate glass calibration method.Refer to previous experiments on glass pollution.The first experiment (Non-pollution) is conducted under the ideal condition where glass is not polluted.The second experiment (Pollution A) simulated the condition of glass polluted by sewage.The third experiment (Pollution B) simulates the condition where glass is polluted by sludge and partially obstructed.According to the above experimental scheme, the calibration experiment of the refraction plane of plate glass under the simulated tunnel construction environment is carried out.
The plate glass calibration is accomplished using target points that are unpolluted or only slightly polluted.The experimental results are shown in Table 4.According to Table 4, compared to the Non-pollution condition, the cosine similarity of the unit normal vector of the refractive surface under both polluted conditions is close to 1.This shows that pollution A and pollution B have less of an effect on the solution accuracy of the proposed method.

Future Work
Section 4.3 conducted stability experiments in simulated tunnel construction environments.However, the real tunnel construction scenes are even more harsh.Image restoration using deep learning techniques serves as a set of potential solutions.The plate glass used for the experiments in this paper is K9 optical glass.However, in practical applications, some explosion-proof tempered glass will be used, whose glass performance may not be as stable as optical glass.In the future, its practical application scenarios can be considered, and several experiments can be conducted using glass with different parameters to verify the stability of the model.

Conclusions
To overcome the failure of conventional vision measurement techniques under plate glass refraction, this paper provides an effective and stable solution for refraction imaging modelling, glass parameter calibration, and 3D coordinate solutions.It analytically realizes the correction of refractive distortion images.The glass normal vector is obtained from the calibration.When the camera's optical axis is not parallel to the normal vector of the refractive plane, a camera rotation model and a camera imaging model under non-parallel plane refraction are established to correct refractive errors.In particular, an independent refractive plane triangulation model is proposed to address the issue of triangulation failure under planar refraction.This experiment verifies the positioning accuracy and measurement accuracy of the camera imaging model under non-parallel plane refraction.These experimental results show that the RMSE of the positioning errors before correction are 1.3441 mm and 1.9937 mm.The positioning errors after correction are 0.1135 mm and 0.1484 mm, respectively.The measurement errors before correction are 2.3821 mm and 2.4037 mm.The measurement errors after correction are 0.2130 mm and 0.2136 mm, respectively.This experiment verifies the accuracy of the binocular vision system triangulation model under independent refractive planes.The RMSE of the depth of field error before and after correction are 2.9902 mm and 0.3187 mm, respectively.The RMSE of the positioning errors before and after correction are 3.5661 mm and 0.3465 mm, respectively.From the above analysis, the proposed triangulation model under independent refractive planes greatly improves the accuracy of depth-of-field and positioning.This proves the effectiveness of the proposed model.While this solution is designed for tool changer robot vision systems in harsh construction environments, it can serve as a reference for any vision measurement application under glass window protection.

Figure 1 .
Figure 1.The TBM diagram of the changing area.

Figure 1 .
Figure 1.The TBM diagram of the changing area.
An image distortion correction model is established based on the concept of dynamic virtual focal length in a parallel plane.It complements the non-parallel plane camera rotation model to achieve correction.The main flow of this paper is shown in Figure 2.

Figure 2 .
Figure 2. The basic content of this article.
Two-Dimensional Measurement with Monocular Vision under Non-Parallel Plane Refraction 3.1.1.Finding the Normal Vector of a Refraction Plane

Figure 2 .
Figure 2. The basic content of this article.

3. 1 .
Two-Dimensional Measurement with Monocular Vision under Non-Parallel Plane Refraction 3.1.1.Finding the Normal Vector of a Refraction Plane

Figure 3 .
Figure 3.An Imaging model when the camera's optical axis is not parallel to the normal vector of the refraction plane.(a) Plane refraction imaging and (b) image points on the imaging surface.

Figure 3 .
Figure 3.An Imaging model when the camera's optical axis is not parallel to the normal vector of the refraction plane.(a) Plane refraction imaging and (b) image points on the imaging surface.

Figure 7 .
Figure 7. Triangulation measurement model under an independent refraction plane.Figure 7. Triangulation measurement model under an independent refraction plane.

Figure 7 .
Figure 7. Triangulation measurement model under an independent refraction plane.Figure 7. Triangulation measurement model under an independent refraction plane.

Figure 8 .
Figure 8. Overall flow chart of binocular measurement under plate refraction.

Figure 8 .
Figure 8. Overall flow chart of binocular measurement under plate refraction.

Figure 9 .
Figure 9. Experimental platform for vision program of the tool-change robot.(1) The camera model is Cognex CIC-1300, which boasts a resolution of 1280 pixels × 1024 pixels.The focal length of the camera lens is 12.5 mm, and the field of view angle is 25°.The measurement volume is 0.08 m³.The camera baseline distance is 205 mm.The distance between the checkerboard calibration plate and the camera optical center is approximately 610 mm.(2) The factory-measured refractive index [27] of K9 optical glass is 1.5437, with a thickness of 20 mm, which is considered as the true value.(3) The number of target points is 5 × 6, with a distance of 39 mm between adjacent points.(4) The motion module in Figure 9 can move in the left and right directions.Before the experiment starts, the motion module can move the target near the working distance of 600 mm.The motion module of the platform does not require high motion accuracy.

Figure 9 .
Figure 9. Experimental platform for vision program of the tool-change robot.(1) The camera model is Cognex CIC-1300, which boasts a resolution of 1280 pixels × 1024 pixels.The focal length of the camera lens is 12.5 mm, and the field of view angle is 25 • .The measurement volume is 0.08 m 3 .The camera baseline distance is 205 mm.The distance between the checkerboard calibration plate and the camera optical center is approximately 610 mm.(2) The factory-measured refractive index [27] of K9 optical glass is 1.5437, with a thickness of 20 mm, which is considered as the true value.(3) The number of target points is 5 × 6, with a distance of 39 mm between adjacent points.(4) The motion module in Figure 9 can move in the left and right directions.Before the experiment starts, the motion module can move the target near the working distance of 600 mm.The motion module of the platform does not require high motion accuracy.

Figure 11 .
Figure 11.Determination of the distortion center.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 12 .
Figure 12.Image point comparison before and after camera rotation with glass.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 11 .
Figure 11.Determination of the distortion center.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 12 .
Figure 12.Image point comparison before and after camera rotation with glass.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 11 .
Figure 11.Determination of the distortion center.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 11 .
Figure 11.Determination of the distortion center.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 12 .
Figure 12.Image point comparison before and after camera rotation with glass.(a) Glass attitude 1 and (b) glass attitude 2.

Figure 12 .
Figure 12.Image point comparison before and after camera rotation with glass.(a) Glass attitude 1 and (b) glass attitude 2. To verify the localization accuracy of the refraction imaging model, both the pinhole camera model and the refraction imaging model are utilized to measure and compare the points on the 2D measurement plane.As shown in Figure 13, P1 represents the pinhole camera model under glass attitude 1. R1 represents the refraction imaging model under glass attitude 1. P2 represents the pinhole camera model under glass attitude 2. R2 represents the refraction imaging model under glass attitude 2. The two experiments are shown in Figure 13.The maximum localization errors of the pinhole camera model are 2.2707 mm and 3.1139 mm, respectively.The maximum localization errors of the refracted imaging model are 0.2428 mm and 0.3394 mm, respectively.The RMSE of positioning errors using

Figure 13 .
Figure 13.Pinhole camera model versus refractive imaging model positioning error.

Figure 13 .
Figure 13.Pinhole camera model versus refractive imaging model positioning error.

Figure 13 .
Figure 13.Pinhole camera model versus refractive imaging model positioning error.

Figure 15 .
Figure 15.Determination of the distortion center.(a) Left camera and (b) right camera.4.3.1.Experimental Verification of Epipolar Constraint Model for Independent Refractiv Plane ImagingAfter calibration of the internal and external parameters of the binocular vision sy tem is completed, the independent refraction plane imaging epipolar constraint model experimentally validated.It has the same basic principles as the shared refraction plan imaging epipolar constraint model and has a wider range of applications.Therefore, th paper only analyzes the epipolar constraint model of independent refractive plane imag ing.In the experiment, keeping the optical axis of the left and right cameras not parall to the normal vector of the refractive plane, the cameras collect one image with the pla glass and one image without the plate glass.To utilize a binocular vision system for 3D measurement, it is essential to initial conduct feature point matching between the left and right images.However, Figure1demonstrates that the polar constraint principle of traditional binocular vision systems rendered ineffective due to the influence of the plane refraction effect.The green line rep resents the limit constraint.In Figure17, Position 1 indicates a large difference in the lon gitudinal tilt angle of the left and right plate glass.In Figure 17, Position 2 indicates th the left and right plate glass are parallel.
, Position 1 indicates a large difference in the lon gitudinal tilt angle of the left and right plate glass.In Figure 17, Position 2 indicates th the left and right plate glass are parallel.

Figure 15 .
Figure 15.Determination of the distortion center.(a) Left camera and (b) right camera.

Figure 17 .
Figure 17.Matching error with and without refraction at different glass positions.

Figure 17 .
Figure 17.Matching error with and without refraction at different glass positions.

Figure 17 .
Figure 17.Matching error with and without refraction at different glass positions.

Figure 18 .
Figure 18.Comparison before and after calibration.(a) Depth of field and (b) depth of field error and positioning error.

Figure 19 .
Figure 19.Pollution type diagram.(a) Glass is polluted by sewage.(b) Glass is polluted by sludge.

Figure 18 .
Figure 18.Comparison before and after calibration.(a) Depth of field and (b) depth of field error and positioning error.

Sensors 2023 , 20 Figure 18 .
Figure 18.Comparison before and after calibration.(a) Depth of field and (b) depth of field error and positioning error.

Figure 19 .
Figure 19.Pollution type diagram.(a) Glass is polluted by sewage.(b) Glass is polluted by sludge.

Figure 19 .
Figure 19.Pollution type diagram.(a) Glass is polluted by sewage.(b) Glass is polluted by sludge.

Table 4 .
Unit normal vectors and refractive indices.