An Accurate Linear Method for 3D Line Reconstruction for Binocular or Multiple View Stereo Vision

For the problem of 3D line reconstruction in binocular or multiple view stereo vision, when there are no corresponding points on the line, the method called Direction-then-Point (DtP) can be used, and if there are two pairs of corresponding points on the line, the method called Two Points 3D coordinates (TPS) can be used. However, when there is only one pair of corresponding points on the line, can we get the better accuracy than DtP for 3D line reconstruction? In this paper, a linear and more accurate method called Point-then-Direction (PtD) is proposed. First, we used the intersection method to obtain the 3D point’s coordinate from its corresponding image points. Then, we used this point as a position on the line to calculate the direction of the line by minimizing the image angle residual. PtD is also suitable for multiple camera systems. The simulation results demonstrate that PtD increases the accuracy of both the direction and the position of the 3D line compared to DtP. At the same time, PtD achieves a better result in direction of the 3D line than TPS, but has a lower accuracy in the position of 3D lines than TPS.


Introduction
The 3D line reconstruction is a basic problem in computer vision and optical measurements. It is widely used in computer vision problem, such as 3D scene reconstruction, non-cooperative target reconstruction and pose estimation for symmetric targets. Line features in the image are robust, stable, and easy to extract; the corresponding lines and points are often used in 3D reconstruction. In the position and attitude estimation for the non-cooperative target, the non-cooperative target is often reconstructed by points, lines, rectangles, or other shapes composed of lines, then the reconstructed model is used for tracking and relative pose estimation in subsequent frames. In pose estimation for the long symmetrical target, such as rockets and missiles, the direction of the target's central axis is often used to represent the target's attitude, which is often reconstructed by the Direction-then-Point (DtP) method. From the above analysis, we can determine that the accuracy of the 3D line is of significance for computer vision.
There are many studies on the reconstruction and measurement of non-cooperative targets [1][2][3][4][5][6][7][8][9][10][11] or attitude measurement of long symmetric targets based on binocular or multiple stereo vision. There are many kinds of non-cooperative targets to be reconstructed, especially targets in the space, such as satellites and spacecrafts. Many of them are based on feature points [2], point clouds [3,4], ellipses, circles [5,6] and line structures [7,8], and so on. However, the number of the feature points is huge, and the matches between them are more complicated and less robust than the feature lines. The ellipses and circles are more difficult to extract than the feature lines. There always are obvious line features on The simulation results demonstrated that PtD can achieve more accurate results, both on direction and position than DtP, because it combines the information of the corresponding image points. At the same time, it can achieve a more accurate direction but less accurate position than TPS. If there is only one pair of corresponding image points on the line, the linear method proposed in this paper can achieve better results.

Stereo-Vision Scenario
A pinhole camera model [12] was used in this study. As shown in Figure 1, The world coordinate system is O w − X w Y w Z w , which is a right-handed coordinate system. The image coordinate system is O i − X i Y i , O i in the top left corner of the image, X i faces right, and Y i faces down. The camera coordinate system is O c − X c Y c Z c , where O c is the camera's optical center. The z-axis is pointing in the positive direction of the optical axis; the x-axis is perpendicular to the z-axis horizontally to the right; and the y-axis is perpendicular to the x-axis and to the z-axis, whose direction is determined by the right-handed definition criterion. The simulation results demonstrated that PtD can achieve more accurate results, both on direction and position than DtP, because it combines the information of the corresponding image points. At the same time, it can achieve a more accurate direction but less accurate position than TPS. If there is only one pair of corresponding image points on the line, the linear method proposed in this paper can achieve better results.

Stereo-Vision Scenario
A pinhole camera model [12] was used in this study. As shown in Figure 1, The world coordinate system is − , which is a right-handed coordinate system. The image coordinate system is − , in the top left corner of the image, faces right, and faces down. The camera coordinate system is − , where is the camera's optical center. The z-axis is pointing in the positive direction of the optical axis; the x-axis is perpendicular to the z-axis horizontally to the right; and the y-axis is perpendicular to the x-axis and to the z-axis, whose direction is determined by the right-handed definition criterion.

Plane Representation by a Single Camera
As shown in Figure 1, AB is the 3D line , C1 and C2 are the optical centers of the two cameras, A1B1 is the projection of on camera C1, and A2B2 is the projection of on camera C2. When atmospheric refraction is not considered, we can assume that is on the plane C1A1B1 and also on the plane C2A2B2. Therefore, if we can attain both of the planes' equations in the world coordinate system, then we can obtain the direction of the 3D line and a point on the line.
Next, we derive its expression by taking a camera as an example. The equation of the line in the image coordinate system is: The image plane is the = plane in the camera coordinate system, where is the physical focal length of the camera. The z-axis of the camera coordinate system passes through the principal point of the image plane ( , ), so the transformation from the image coordinate system to the camera coordinate system is determined as: where ( , ) is in the camera coordinate system, ( , ) is in the image coordinate system, and ( , ) is the actual physical size of the camera pixel. Plane intersection for stereo vision. AB is the 3D line I, C 1 and C 2 are the optical centers of the two cameras, A 1 B 1 is the projection of I on camera C 1 , and A 2 B 2 is the projection of I on camera C 2 .

Plane Representation by a Single Camera
As shown in Figure 1, AB is the 3D line I, C 1 and C 2 are the optical centers of the two cameras, A 1 B 1 is the projection of I on camera C 1 , and A 2 B 2 is the projection of I on camera C 2 . When atmospheric refraction is not considered, we can assume that I is on the plane C 1 A 1 B 1 and also on the plane C 2 A 2 B 2 . Therefore, if we can attain both of the planes' equations in the world coordinate system, then we can obtain the direction of the 3D line and a point on the line.
Next, we derive its expression by taking a camera as an example. The equation of the line in the image coordinate system is: where A = a b c T and X = x y 1 T . The image plane is the z c = f plane in the camera coordinate system, where f is the physical focal length of the camera. The z-axis of the camera coordinate system passes through the principal point of the image plane C x , C y , so the transformation from the image coordinate system to the camera coordinate system is determined as: where (x c , y c ) is in the camera coordinate system, (x, y) is in the image coordinate system, and d x , d y is the actual physical size of the camera pixel. The line in the image can be expressed in the camera coordinate system as: The equivalent focal length of the camera is abbreviated as: The plane equation can be summarized as: Given the parameter matrix n = aF x bF y aC x + bC y + c T , the plane equation is then rewritten as: n T P c = 0 where P c = x c y c z c T is a point from the camera coordinate system, and n is the normal vector of the plane. R and t are the rotation matrix and the translation vector from the world coordinate system to the camera coordinate system, respectively. Therefore, the point in the world coordinate system can be transformed to the camera coordinate system using Equation (7).
where P w = x w y w z w T is a point in the world coordinate system. The plane equation in the world coordinate system can then be derived as:

Plane Intersection
The plane equations of the two cameras, respectively, are: The 3D line I = (l, m, n) must be located on these two planes. Therefore, the direction vector of I can be solved as: A 3D line can be uniquely determined by a point and the direction vector. Therefore, we chose the foot point P 0 (x 0 , y 0 , z 0 ) from the O w to the I; (l, m, n) * P 0 T = 0. P 0 can be solved by (l, m, n) * P 0 T = 0 and Equation (9). The point is directly solved by direction, and as a result the overall accuracy mainly depends on the accuracy of the line direction.

Stereo Vision Scenario for PtD
As shown in Figure 1, Camera C 1 and Camera C 2 observe the line I from different directions, and the projections on the image are A 1 B 1 and A 2 B 2 , respectively. In this paper, we assume that one of the two endpoints of the line segments are the corresponding image points, i.e., A 1 and A 2 are the corresponding image points, or B 1 and B 2 are the corresponding image points. If the camera parameters and the line extracted from the image are error-free, the 3D point will be on the line obtained by the DtP, and then the results obtained by the DtP and Ptd are same. However, if either the camera parameters or the line extracted from the image have an error, the 3D point may not be on the line I.
In a similar manner to the image-space residual method for camera calibration, the line I was projected to the image plane according to the theoretical model, and the angle between the projection and the 2D line extracted on the image plane was considered as the angle residual. We also minimized the square sum of all the angles. In this paper, this term is called the IAR. As shown in Figure 2, the C 2 A 2 B 2 plane is the plane composed of the optical center and the axis of the camera; A'B' is the 3D line; B 2 A 3 is the projection of A'B' on the image plane; and α is the image angle residual.
Sensors 2021, 21, x FOR PEER REVIEW the DtP and Ptd are same. However, if either the camera parameters or the line ex from the image have an error, the 3D point may not be on the line .
In a similar manner to the image-space residual method for camera calibrat line was projected to the image plane according to the theoretical model, and th between the projection and the 2D line extracted on the image plane was consid the angle residual. We also minimized the square sum of all the angles. In this pa term is called the IAR. As shown in Figure 2, the C2A2B2 plane is the plane comp the optical center and the axis of the camera; A'B' is the 3D line; B2A3 is the proje A'B' on the image plane; and α is the image angle residual. From the above definition and stereo-vision scenario, we can ascertain that the passes through the 3D point, and the intersection of the plane composed of the 3D the camera's optical center and the corresponding imaging plane is parallel to the on the image. For a multiple camera system, we can obtain the least square solution The mathematical expression is derived below.

Definition of the Problem
It is assumed that the direction of the 3D line is ( , , ), and passes th point ( , , ) in the world coordinate system. It is easy to see that also through the point ( + , + , + ). In this paper, the is obtained by tersection method of minimizing the distance from the spatial point to the back pr ray to obtain the three-dimensional coordinates from the corresponding image po and [24]. The plane of each camera passes through the point , and th sponding optical center. The plane consisting of and the optical center of the ith can be expressed as: From the above definition and stereo-vision scenario, we can ascertain that the 3D line passes through the 3D point, and the intersection of the plane composed of the 3D line and the camera's optical center and the corresponding imaging plane is parallel to the 2D line on the image. For a multiple camera system, we can obtain the least square solution.
The mathematical expression is derived below.

Definition of the Problem
It is assumed that the direction of the 3D line I is (l, m, n), and I passes through a point P 0 (x 0 , y 0 , z 0 ) in the world coordinate system. It is easy to see that I also passes through the point P 1 (x 0 + l, y 0 + m, z 0 + n). In this paper, the P 0 is obtained by the intersection method of minimizing the distance from the spatial point to the back projection ray to obtain the three-dimensional coordinates from the corresponding image points A 1 and A 2 [24]. The plane of each camera passes through the point P 0 , P 1 and the corresponding optical center. The plane consisting of I and the optical center of the ith camera can be expressed as: Or where (Ox i , Oy i , Oz i ) is the optical center of the ith camera. The plane equation after simplification is: It can be seen from Equation (12) that n wi can be represented linearly by I. Letting the relationship matrix be M: It is obvious that M is a full rank matrix. Assuming the transformation matrix from the world coordinate system to the camera coordinate system is R i t i 0 1 , the plane is expressed in the camera coordinate system as: From the image plane equation z = f , the direction of the intersection line is I i = (n ci1 , n ci2 , 0). Then, the angle between the 2D line in the image plane and the projection of I on the corresponding image is: where a i and b i are coefficients of the linear equation a i x + b i y + c i = 0 in the image plane. Therefore, the optimized objective function is: where N is the number of cameras.

Linear Method
A method of linearly solving for the minimum value is given by the derivation below. The original expression is very complicated, especially the partial derivative expression. The numerator and the denominator contain (l, m, n), which cannot be solved linearly. Therefore, the expression is deformed.
If the two 2D lines I 1 = (a 1 , b 1 , c 1 ) T , I 2 = (a 2 , b 2 , c 2 ) T are nearly parallel, the angle is: This is a small angle, and we have: For the sake of discussion, we divided a 1 , b 1 and a 2 , b 2 by the larger one, respectively, so that the maximum value was 1. Then: where |α i | is positively correlated with |a 1 b 2 − a 2 b 1 |. |α| is the absolute value of α. Then, Equation (17) can be used instead of Equation (21) as the optimization solution objective function: From the previous derivation, n ci1 , n ci2 is a linear function of I = (l, m, n) T . The equation can therefore be recorded as (18), and it can be representative of a normal vector to a plane. This can be thought of as a function of (l, m, n) T , which is a function of (l, m, n). Clearly, this function is a basic elementary function-it is continuous and differentiable in the domain of the definition, and the minimum value must exist. At the point where the minimum value is obtained, the partial derivative of (l, m, n) exists and it is 0, and the partial derivative is obtained separately.
We can solve Equation (19) by the SVD [25] decomposition of the coefficient matrix to obtain (l, m, n).

Efficiency Analysis
The linear method for minimizing the image angle residual is as follows.

1.
Solve the 3D coordinate of the corresponding image points with a time complexity of O(n); 2.
Solve the line parameters projected onto the image plane with a time complexity of O(n); 3.
Solve the coefficient matrix with a time complexity of O(n); 4.
Find the final result in fixed time.
It can be seen from the above analysis that the image angle residual method can be solved only by calculating the 3D point coordinate, the projection line parameter, the solution coefficient matrix, and the least-squares solution, so the time complexity is O(n).

Experiments
Because the ground truth was not easy to obtain in the real experiment, we used the simulation to verify the accuracy of the algorithm and then we used only the real data to verify the validity of the algorithm.

Simulation
We simulated many conditions to validate the accuracy of our method. Firstly, the variation of the intersection angle of the two cameras and the 3D line's attitude variation were simulated to verify the accuracy and robustness of the proposed algorithm. Secondly, the various error conditions for the stereo vision were simulated, including the 2D line extraction error, the external camera parameter error, and both the above errors. Thirdly, we verified the accuracy while in a multiple camera system. Finally, we assessed the time performance of our method in a multiple camera system.

Simulation Environment
The simulation platform was Windows 10 Pro, the implementation language of the algorithm was C++, the implementation environment was Microsoft Visual Studio 2013, and the processor was an Intel(R) Core TM i7-9850H 2.6GHz.
The equivalent focal length was (2181.8, 2181.8) and the image principal point was (1024, 1024). The cameras were arranged in a circle around the target, and the radius of the circle was 4.5 m. The origin of the world coordinate system was the center of the circle. The two endpoints of the 3D line were A(−0.5, −0.5, −0.5) and B(1.5, 1.5, 1.5). The top view of the simulation scenario is shown in Figure 3.  is the distance between the endpoint of the 3D line result line, and is the distance between the endpoint of the 3D line result line.
Simulations were conducted separately to determine the following co Case 1 was to verify the precision under the different angle between th Case 2 was to verify the precision under the target's different attitude. Cas verify the precision under the error of the cameras' parameters. Case 6 wa precision as the number of cameras increased. Case 7 was to test the tempor The corresponding image points used in the PtD method were obtained by adding the same error to the projection of the endpoint A in the image, which ensured that PtD method was also solved under the same error conditions. 2.
The TPS, DtP, and PtD methods were used for each simulation. We used the angle between the result and the ground truth and the distance from the two endpoints to the result line to evaluate the accuracy. All simulations were calculated 1000 times, and the root mean square (RMS) of the angular error and the average of distance were obtained. The units of angular error were degrees, and the units of distance error were meters.
In the above equation I t (l t , m t , n t ) is the ground truth, and I(l, m, n) is the algorithm's result. d AI is the distance between the endpoint A of the 3D line segment to the result line, and d BI is the distance between the endpoint B of the 3D line segment to the result line.
Simulations were conducted separately to determine the following conditions: 1.
Angle between optical axes of binocular cameras. 2.

3.
Line segments extraction error in image.

4.
Errors of all camera external parameters. 5.
All camera external parameters and extraction errors. 6.
Number of cameras. 7.
Running time.
Case 1 was to verify the precision under the different angle between the two cameras. Case 2 was to verify the precision under the target's different attitude. Cases 3-5 were to verify the precision under the error of the cameras' parameters. Case 6 was to verify the precision as the number of cameras increased. Case 7 was to test the temporal performance of the algorithm.

Simulation Results
The simulation details and results are given, respectively. Firstly, we simulated the intersection angles of two cameras, which varied to verify the accuracy of the three methods. As shown in Figure 3, C 1 was placed in the fixed point (4.5,0,0), and C 2 was moved on the circle with a radius of 4.5 m. The angle between the optical axes of C 1 and C 2 varied from 30 • to 150 • . We simulated two conditions of the 3D line, one of which was the 3D line perpendicular to the baseline of C 1 and C 2 . When the 3D line was perpendicular to the baseline, the endpoints of the 3D line segment were A(−0.5, −0.5, −0.5) and B(−0.5, 1.5, −0.5). From the definition of the simulation scenario, we knew that, as the intersection angle changed from 30 • to 150 • , the line segment AB was always perpendicular to the baseline, and the distance between AB and the baseline became smaller and smaller. When the 3D line is not perpendicular to the baseline, the endpoints of the 3D line segment were A(−0.5, −0.5, −0.5), B(1.5, 1.5, 1.5). A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 5 cm RMS value. A Gaussian error with a mean of 0 was added to the camera angle, with a 1 • RMS value. A Gaussian error with a mean of 0 was added to the xand y-directions of the two endpoints of the 2D line segments, with a 2-pixel RMS value. The results are shown in Figure 4. we knew that, as the intersection angle changed from 30° to 150°, the line segment AB was always perpendicular to the baseline, and the distance between AB and the baseline became smaller and smaller. When the 3D line is not perpendicular to the baseline, the endpoints of the 3D line segment were (−0.5, −0.5, −0.5), (1.5,1.5,1.5). A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 5 cm RMS value. A Gaussian error with a mean of 0 was added to the camera angle, with a 1° RMS value. A Gaussian error with a mean of 0 was added to the x-and y-directions of the two endpoints of the 2D line segments, with a 2-pixel RMS value. The results are shown in Figure 4.  When the 3D line remained unchanged, the error was relevant to the intersection angle. In the beginning, the three methods achieved the same accuracy. As the intersection angle varied from 30° to 150°, the distance between the baseline and AB grew smaller and smaller. In other words, they were becoming closer and closer to each other, which meant the intersection condition was getting worse; the PtD method had the same accuracy as the DtP method. TPS had the best positional accuracy but the worst angular accuracy. In the condition where the 3D line was not parallel to the imaging plane and was not perpendicular to the baseline, PtD had higher accuracy both on the position and angle than DtP; TPS achieved the worst angular accuracy, and DtP achieved the worst positional accuracy. When the 3D line remained unchanged, the error was relevant to the intersection angle. In the beginning, the three methods achieved the same accuracy. As the intersection angle varied from 30 • to 150 • , the distance between the baseline and AB grew smaller and smaller. In other words, they were becoming closer and closer to each other, which meant the intersection condition was getting worse; the PtD method had the same accuracy as the DtP method. TPS had the best positional accuracy but the worst angular accuracy. In the condition where the 3D line was not parallel to the imaging plane and was not perpendicular to the baseline, PtD had higher accuracy both on the position and angle than DtP; TPS achieved the worst angular accuracy, and DtP achieved the worst positional accuracy.
Secondly, we simulated the condition where the 3D line moved while the intersection angle remained unchanged; the C 1 and C 2 places were fixed. Endpoint A was (−0.5,−0.5,−0.5), The initial position of endpoint B was (−0.5,1.5,−0.5), and the kth position was (−0.5 − k * 0.05, 1.5, −0.5 − k * 0.05); k was the offset time. The end position of point B was (−5.5,1.5,−5.5). A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 5 cm RMS value. A Gaussian error with a mean of 0 was added to the camera angle, with a 1 • RMS value. A Gaussian error with a mean of 0 was added to the xand y-directions of the two endpoints of the 2D line, with a 2-pixel RMS value. The results of the simulation while the intersection angles are 90 • and 120 • , and shown in Figure 5a,b respectively. Secondly, we simulated the condition where the 3D line moved while the intersection angle remained unchanged; the C1 and C2 places were fixed. Endpoint A was (−0.5,−0.5,−0.5), The initial position of endpoint B was (−0.5,1.5,−0.5), and the th position was (−0.5 − * 0.05,1.5, −0.5 − * 0.05); was the offset time. The end position of point B was (−5.5,1.5,−5.5). A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 5 cm RMS value. A Gaussian error with a mean of 0 was added to the camera angle, with a 1° RMS value. A Gaussian error with a mean of 0 was added to the x-and y-directions of the two endpoints of the 2D line, with a 2-pixel RMS value. The results of the simulation while the intersection angles are 90° and 120°, and shown in Figure 5a  As shown in Figure 5, at the start, the TPS method had the best positioning accuracy and the worst angular accuracy. However, as the intersection condition became, the TPS had the best accuracy both in direction and position. The reasons are as follows.
Firstly, the 3D line became longer, and the final length was 3.68-times greater than that of the initial length. Therefore, theoretically, if only the influence of the line length was considered, the direction error of TPS would be the 1/3.68 of the original value. However, the error of the 3D point B increased because the intersection angle of 3D point B became smaller. Therefore, the error reduction was not proportional.
Secondly, the angle between the 3D line and the baseline remained unchanged-it was always 90°. However, the angle between the 3D line and the imaging plane varied from 0 to 45° and the distance between the 3D line and the baseline became smaller. At the start, when the intersection angle was 90°, the distance between the 3D line and the As shown in Figure 5, at the start, the TPS method had the best positioning accuracy and the worst angular accuracy. However, as the intersection condition became, the TPS had the best accuracy both in direction and position. The reasons are as follows.
Firstly, the 3D line became longer, and the final length was 3.68-times greater than that of the initial length. Therefore, theoretically, if only the influence of the line length was considered, the direction error of TPS would be the 1/3.68 of the original value. However, the error of the 3D point B increased because the intersection angle of 3D point B became smaller. Therefore, the error reduction was not proportional.
Secondly, the angle between the 3D line and the baseline remained unchangedit was always 90 • . However, the angle between the 3D line and the imaging plane varied from 0 to 45 • and the distance between the 3D line and the baseline became smaller. At the start, when the intersection angle was 90 • , the distance between the 3D line and the baseline was 3.89 m; when the intersection angle was 120 • , the distance between the 3D line and the baseline was 2.94 m. At the end, however, the distances were 2.50 m and 2.30 m, respectively. The lengths of the baseline were 6.36 m and 7.79 m, respectively. From the above analysis, we can determine that the smaller distance aggravated the degeneracy of the lines' reconstruction.
Thirdly, we simulated the condition where the cameras and the 3D line remained unchanged, but the errors of the camera's external parameters and 2D line were different. The intersection angle was 120 • . The endpoints of the 3D line were A(0.5, −0.5, 0.5) and B(1.5, 1.5, 1.5). We simulated for 2D line error, the error of the camera external parameters, and all above errors. The error details were as below: Case 1: A Gaussian error with a mean of 0 was added to the xand y-directions at the head and the tail of the line segment, with 0-4 pixels of RMS. The results are shown in Figure 6.
Sensors 2021, 21, x FOR PEER REVIEW 1 baseline was 3.89 m; when the intersection angle was 120°, the distance between t line and the baseline was 2.94 m. At the end, however, the distances were 2.50 m an m, respectively. The lengths of the baseline were 6.36 m and 7.79 m, respectively. the above analysis, we can determine that the smaller distance aggravated the degen of the lines' reconstruction. Thirdly, we simulated the condition where the cameras and the 3D line rem unchanged, but the errors of the camera's external parameters and 2D line were dif The intersection angle was 120°. The endpoints of the 3D line were (0.5, −0.5,0.5 (1.5,1.5,1.5). We simulated for 2D line error, the error of the camera external param and all above errors. The error details were as below: Case 1: A Gaussian error with a mean of 0 was added to the x-and y-directions head and the tail of the line segment, with 0-4 pixels of RMS. The results are sho Figure 6. From Figure 6, we can see that if only the same 2D line error existed, the 3D accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP Case 2: A Gaussian error with a mean of 0 was added to the optical center camera, with 1 mm to 50 mm of RMS, and the step size was 1 mm. A Gaussian erro a mean of 0 was added to the camera angle, with 0.05-2.5° of RMS. The simulation r are shown in Figure 7.  From Figure 6, we can see that if only the same 2D line error existed, the 3D line's accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP.
Case 2: A Gaussian error with a mean of 0 was added to the optical center of the camera, with 1 mm to 50 mm of RMS, and the step size was 1 mm. A Gaussian error with a mean of 0 was added to the camera angle, with 0.05-2.5 • of RMS. The simulation results are shown in Figure 7. baseline was 3.89 m; when the intersection angle was 120°, the distance between the 3D line and the baseline was 2.94 m. At the end, however, the distances were 2.50 m and 2.30 m, respectively. The lengths of the baseline were 6.36 m and 7.79 m, respectively. From the above analysis, we can determine that the smaller distance aggravated the degeneracy of the lines' reconstruction. Thirdly, we simulated the condition where the cameras and the 3D line remained unchanged, but the errors of the camera's external parameters and 2D line were different. The intersection angle was 120°. The endpoints of the 3D line were (0.5, −0.5,0.5) and (1.5,1.5,1.5). We simulated for 2D line error, the error of the camera external parameters, and all above errors. The error details were as below: Case 1: A Gaussian error with a mean of 0 was added to the x-and y-directions at the head and the tail of the line segment, with 0-4 pixels of RMS. The results are shown in Figure 6. From Figure 6, we can see that if only the same 2D line error existed, the 3D line's accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP.
Case 2: A Gaussian error with a mean of 0 was added to the optical center of the camera, with 1 mm to 50 mm of RMS, and the step size was 1 mm. A Gaussian error with a mean of 0 was added to the camera angle, with 0.05-2.5° of RMS. The simulation results are shown in Figure 7.  From Figure 7, we can see that under the same error of the camera's external parameters, the directional accuracy of the 3D line of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the positional accuracy of the 3D line of the three methods from most to least accurate, was TPS, PtD, then DtP. Case 3: A Gaussian error with a mean of 0 was added to the optical center of the camera, with 1 mm to 50 mm of RMS, and the step size was 1 mm. A Gaussian error with a mean of 0 was added to the camera angle, with 0.05-2.5 • of RMS. A Gaussian error with a mean of 0 was added to the x-and y-directions of the two endpoints of the 3D line, with 0.1-5 pixels of RMS. The simulation results are shown in Figure 8. From Figure 7, we can see that under the same error of the camera's external parameters, the directional accuracy of the 3D line of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the positional accuracy of the 3D line of the three methods from most to least accurate, was TPS, PtD, then DtP.
Case 3: A Gaussian error with a mean of 0 was added to the optical center of the camera, with 1 mm to 50 mm of RMS, and the step size was 1 mm. A Gaussian error with a mean of 0 was added to the camera angle, with 0.05-2.5° of RMS. A Gaussian error with a mean of 0 was added to the x-and y-directions of the two endpoints of the 3D line, with 0.1-5 pixels of RMS. The simulation results are shown in Figure 8. From Figure 8, we can see that under same error of external camera's parameters and the 2D lines, the 3D line's direction accuracy of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the 3D line's position accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP. The position accuracies of TPS and PtD are close, and the directional accuracy of PtD was far better than TPS.
Additionally, we simulated the condition where the camera number varied from 2 to 20; the cameras were evenly distributed on the circle as shown in Figure 3. The DtP method used the LS [26] directly solve this scenario. A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 10 mm root mean square (RMS) value. A Gaussian error with a mean of 0 was added to the camera angle, with a 0.5° RMS value. A Gaussian error with a mean of 0 was added to the x-y-directions of the two endpoints of the 3D line, with a 1-pixel RMS value. The endpoints of the 3D line were A(0.5, −0.5,0.5), B(1.5,1.5,1.5). The results are shown in Figure 9.  From Figure 8, we can see that under same error of external camera's parameters and the 2D lines, the 3D line's direction accuracy of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the 3D line's position accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP. The position accuracies of TPS and PtD are close, and the directional accuracy of PtD was far better than TPS.
Additionally, we simulated the condition where the camera number varied from 2 to 20; the cameras were evenly distributed on the circle as shown in Figure 3. The DtP method used the LS [26] directly solve this scenario. A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 10 mm root mean square (RMS) value. A Gaussian error with a mean of 0 was added to the camera angle, with a 0.5 • RMS value. A Gaussian error with a mean of 0 was added to the x-y-directions of the two endpoints of the 3D line, with a 1-pixel RMS value. The endpoints of the 3D line were A(0.5, −0.5, 0.5), B(1.5, 1.5, 1.5). The results are shown in Figure 9. From Figure 7, we can see that under the same error of the camera's external parameters, the directional accuracy of the 3D line of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the positional accuracy of the 3D line of the three methods from most to least accurate, was TPS, PtD, then DtP.
Case 3: A Gaussian error with a mean of 0 was added to the optical center of the camera, with 1 mm to 50 mm of RMS, and the step size was 1 mm. A Gaussian error with a mean of 0 was added to the camera angle, with 0.05-2.5° of RMS. A Gaussian error with a mean of 0 was added to the x-and y-directions of the two endpoints of the 3D line, with 0.1-5 pixels of RMS. The simulation results are shown in Figure 8. From Figure 8, we can see that under same error of external camera's parameters and the 2D lines, the 3D line's direction accuracy of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the 3D line's position accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP. The position accuracies of TPS and PtD are close, and the directional accuracy of PtD was far better than TPS.
Additionally, we simulated the condition where the camera number varied from 2 to 20; the cameras were evenly distributed on the circle as shown in Figure 3. The DtP method used the LS [26] directly solve this scenario. A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 10 mm root mean square (RMS) value. A Gaussian error with a mean of 0 was added to the camera angle, with a 0.5° RMS value. A Gaussian error with a mean of 0 was added to the x-y-directions of the two endpoints of the 3D line, with a 1-pixel RMS value. The endpoints of the 3D line were A(0.5, −0.5,0.5), B(1.5,1.5,1.5). The results are shown in Figure 9.  From Figure 9, we can see that as the number of cameras increased, the accuracy of the three methods increased gradually, indicating that these methods utilized the constraints of the multi-camera. Similar to the result in Figure 8, with the increase in the number of cameras, under the same error of the external camera's parameters and the 2D lines, the 3D line's direction accuracy of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the 3D line's positional accuracy of the three methods, from most to least accurate, was TPS, PtD, then DtP. The positional accuracies of TPS and PtD were close, and the directional accuracy of the PtD method was far better than the TPS method.
Finally, we simulated the condition where the camera number increased from 2 to 20 to verify the time performance of our method. The linear algorithm took a small amount of time; therefore, for the convenience of comparison, we counted the time it took to run 10,000 times. A Gaussian error with a mean of 0 was added to the optical center of the camera, with a 5 cm RMS value. A Gaussian error with a mean of 0 was added to the camera angle, with a 1 • RMS value. A Gaussian error with a mean of 0 was added to the x-y-directions of the two endpoints of the 2D line segments, with a 2-pixel RMS value. The simulation results are shown in Figure 10.
the three methods increased gradually, indicating that these methods utilized straints of the multi-camera. Similar to the result in Figure 8, with the increase in t ber of cameras, under the same error of the external camera's parameters and the the 3D line's direction accuracy of the three methods, from most to least accur PtD, DtP, then TPS, and the 3D line's positional accuracy of the three methods, fr to least accurate, was TPS, PtD, then DtP. The positional accuracies of TPS and P close, and the directional accuracy of the PtD method was far better than the TPS Finally, we simulated the condition where the camera number increased from to verify the time performance of our method. The linear algorithm took a small of time; therefore, for the convenience of comparison, we counted the time it too 10,000 times. A Gaussian error with a mean of 0 was added to the optical cent camera, with a 5 cm RMS value. A Gaussian error with a mean of 0 was adde camera angle, with a 1° RMS value. A Gaussian error with a mean of 0 was adde x-y-directions of the two endpoints of the 2D line segments, with a 2-pixel RM The simulation results are shown in Figure 10.

Simulation Conclusion
The simulation results are shown in Table 2. The "F" in the table means first means second, and the "T" means third and the "S->T" means from second to thi  Figure 10. Relationship between the running time (in seconds) and the number of cameras. TPS, PtD, and DtP are linear methods, but PtD takes longer than DtP. Figure 10 confirms that all three methods are linear methods; TPS took the longest, then PtD, and the fastest was DtP. DtP only needs to solve each plane equation and then solve the least-squares method; PtD needs to solve the 3D point coordinate and the line parameters in two steps; TPS needs to solve the 3D point coordinate two times, which explains the variances in running time between the methods.

Simulation Conclusion
The simulation results are shown in Table 2. The "F" in the table means first, the "S" means second, and the "T" means third and the "S->T" means from second to third.
From the above simulation results, we can come to some conclusions: 1.
Under the same error of the external camera parameters and the 2D line, the PtD method has the best directional accuracy, and the TPS method has the best positional accuracy.

2.
When the intersection condition was normal, the 3D line's direction accuracy of the three methods, from most to least accurate, was PtD, DtP, then TPS, and the 3D line's position accuracy of the three methods, from most to least accurate, was TPS, PtD, and DtP. Therefore, if the direction accuracy is more important, the PtD method should be used, and if positional accuracy is more important, the TPS method should be used-provided that there are two pairs of corresponding image points. 3.
When the intersection condition was bad and the target was relative long, TPS achieved the best accuracy, both in the direction and position of the 3D line. However, it needs two pairs of corresponding image points, which is sometimes hard to satisfy. PtD achieved better accuracy than DtP both in the direction and position of the 3D line, so PtD can be used if there is only one pair of corresponding image points.

Physical Experiment
We used physical experiment 1 to verify the precision of our method, and used physical experiment 2 (in which TPS could not be used) to verify the correctness of the algorithm.

Physical Experiment 1 Physical Environment
We used three cameras to measure the lines-the camera setup is shown in Figure 11. The physical environment is shown in Figure 12. The target was a checkerboard and was placed in front of the cameras. The precision of the checkerboard was 0.01 mm. The 3D lines to be reconstructed were the four edge lines in the checkerboard, as marked in Figure 12. The length of the lines 1, 3, 5, and 7 were 30 cm, and the lines 2, 4, 6, 8 were 21 cm. From the positional relationship between the camera and the target, we could ascertain that (a) was in a better intersection condition than (b), and the lines 1 and 3 were in a better intersection condition than the lines 2 and 4. All cameras were calibrated by the diagonal markers behind the checkerboard, and the 3D coordinates of the diagonal markers were obtained by the total station Leica TS60, with an accuracy of 0.6 mm. At the same time, we also obtained the 3D coordinates of the four corners of the checkerboard using the Leica TS60. PtD, DtP, and TPS methods were used to reconstruct the four 3D lines. The camera and lens models were Basler (acA1440-220uc) and RICOH (6 mm 1:1.4), respectively.
From the above simulation results, we can come to some conclusions: 1. Under the same error of the external camera parameters and the 2D line, the PtD has the best directional accuracy, and the TPS method has the best positional ac 2. When the intersection condition was normal, the 3D line's direction accura three methods, from most to least accurate, was PtD, DtP, then TPS, and the position accuracy of the three methods, from most to least accurate, was T and DtP. Therefore, if the direction accuracy is more important, the PtD should be used, and if positional accuracy is more important, the TPS metho be used-provided that there are two pairs of corresponding image points. 3. When the intersection condition was bad and the target was relative lo achieved the best accuracy, both in the direction and position of the 3D lin ever, it needs two pairs of corresponding image points, which is sometime satisfy. PtD achieved better accuracy than DtP both in the direction and po the 3D line, so PtD can be used if there is only one pair of corresponding imag

Physical Experiment
We used physical experiment 1 to verify the precision of our method, and used experiment 2 (in which TPS could not be used) to verify the correctness of the algori 4.2.1. Physical Experiment 1

Physical Environment
We used three cameras to measure the lines-the camera setup is shown in F The physical environment is shown in Figure 12. The target was a checkerboard placed in front of the cameras. The precision of the checkerboard was 0.01 mm. The to be reconstructed were the four edge lines in the checkerboard, as marked in F The length of the lines 1, 3, 5, and 7 were 30 cm, and the lines 2, 4, 6, 8 were 21 cm. positional relationship between the camera and the target, we could ascertain tha in a better intersection condition than (b), and the lines 1 and 3 were in a better int condition than the lines 2 and 4. All cameras were calibrated by the diagonal marker the checkerboard, and the 3D coordinates of the diagonal markers were obtained by station Leica TS60, with an accuracy of 0.6 mm. At the same time, we also obtaine coordinates of the four corners of the checkerboard using the Leica TS60. PtD, DtP, methods were used to reconstruct the four 3D lines. The camera and lens models we (acA1440-220uc) and RICOH (6 mm 1:1.4), respectively.   Experimental Procedure The procedure of the experiment was as follows: 1. The world coordinate system was established by the total station. The total stat was used to obtain the coordinates of the calibration control points. The coordin system direction was vertically upwards (Y) and horizontal (XZ), and the syst constituted a right-hand coordinate system. 2. The stereo cameras were used to take photos synchronously. The checkerboard w placed at two angles to verify the accuracy of the methods in two conditions. O was where the 3D lines were parallel to the imaging plane, and the other was n The calibration control points were extracted from the image, and the 3D coordina were used to calibrate all of the cameras in a unified world coordinate system. parameters of the camera were calibrated, including the main point, equivalent fo length, lens distortion, and external parameters of the camera. 3. The 2D lines and the corresponding image points were extracted from the two ima by extracted the two endpoints of the 2D lines. The corresponding image points w one of the endpoints of the 2D lines. 4. After the correction of lens distortion, DtP, PtD, and TPS were used separately obtain the 3D line I.
First, we estimated the precision by the distance of two points-the results are sho in Table 3. From the results, we determined that the maximum of the distance error w 0.2 mm and the corresponding angle was . = 0.038°. The theoretical accur of the total station was 0.6 mm, which meant a corresponding angle . = 0.1 Table 3. Errors of the total station (mm). Experimental Procedure

Line No
The procedure of the experiment was as follows: 1. The world coordinate system was established by the total station. The total station was used to obtain the coordinates of the calibration control points. The coordinate system direction was vertically upwards (Y) and horizontal (XZ), and the system constituted a right-hand coordinate system.

2.
The stereo cameras were used to take photos synchronously. The checkerboard was placed at two angles to verify the accuracy of the methods in two conditions. One was where the 3D lines were parallel to the imaging plane, and the other was not. The calibration control points were extracted from the image, and the 3D coordinates were used to calibrate all of the cameras in a unified world coordinate system. All parameters of the camera were calibrated, including the main point, equivalent focal length, lens distortion, and external parameters of the camera.

3.
The 2D lines and the corresponding image points were extracted from the two images by extracted the two endpoints of the 2D lines. The corresponding image points were one of the endpoints of the 2D lines.

4.
After the correction of lens distortion, DtP, PtD, and TPS were used separately to obtain the 3D line I.
First, we estimated the precision by the distance of two points-the results are shown in Table 3. From the results, we determined that the maximum of the distance error was 0.2 mm and the corresponding angle was tan −1 0.2 300 = 0.038 • . The theoretical accuracy of the total station was 0.6 mm, which meant a corresponding angle tan −1 0.6 210 = 0.16 • . The errors between the coordinates obtained by the total station and given by the intersection method [21] to demonstrate the precision of the cameras' parameters are shown in Table 4. The ith point was the top or left endpoint of the ith line. Table 4. Results of 3D points' errors (mm).
Therefore, we used the point coordinates obtained by the total station to calculate the direction of the line as the ground truth I t . We calculated the angle between I t and I as the angular error, and the sum of distances from the two endpoints to I as the distance error, to estimate the accuracy of our method.
All the results are shown in Tables 5 and 6. The second, third and fourth columns are angular errors ( • ), and the fifth, sixth and seventh columns are distance errors (mm). From the above tables, we can see that for the lines 1, 3, 5, and 7, the sum of the angular errors of the 3D line, from lowest to highest, was TPS, PtD, then DtP, and they were very close; the maximum difference was 0.02. The sum of the distance error of the 3D line, from highest to lowest, was DtP, PtD, then TPS. For the lines 2, 4, 6, and 8, the accuracy of the direction and distance, from lowest to highest, was TPS, PtD, then DtP. However, the TPS needed two pairs of corresponding image points, which was difficult to meet under certain conditions, such as in physical experiment 2.

Physical Experiment 2
Physical experiment 2 was designed to verify the correctness of our method if there were more than two cameras.

Physical Environment
The setup for the physical experiment is shown in Figure 13. The target was placed on the board in the center, and the diagonal markers on the side were the calibration control points. The 3D line to be reconstructed was the target's central axis. The resolution of the camera was 4288 × 2848 pixels, and the equivalent focal distance was approximately 5200.

Physical Experiment 2
Physical experiment 2 was designed to verify the correctness of our method if there were more than two cameras.

Physical Environment
The setup for the physical experiment is shown in Figure 13. The target was placed on the board in the center, and the diagonal markers on the side were the calibration control points. The 3D line to be reconstructed was the target's central axis. The resolution of the camera was 4288 × 2848 pixels, and the equivalent focal distance was approximately 5200. Figure 13. Physical scene for experiment 2. The target was placed on the board in the center, and the diagonal markers on the side were the calibration control points (CCPs). The 3D line to be reconstructed was the target's central axis.

Experimental Procedure
The experiment was conducted as follows: 1. The total station was used to obtain the coordinates of the calibration control points in the world coordinate system. The total station coordinate system was used as the world coordinate system. The coordinate system direction was vertically upwards (Y) and horizontal (XZ) and the system constituted a right-hand coordinate system. 2. The same camera was used to take photographs from nine different locations. The calibration control points were extracted from the image, and the three-dimensional coordinates were used to calibrate all of the cameras in a unified world coordinate system. 3. The 2D line was extracted from the image. The corresponding image point was the top point of the target. 4. DtP and PtD were used separately to obtain the results. For hard-to-find pairs of corresponding image points on the target, TPS was not used.
All the target directional results are shown in Table 7. It can be seen from the table that the results of the two methods were essentially the same.  Experimental Procedure The experiment was conducted as follows: 1.
The total station was used to obtain the coordinates of the calibration control points in the world coordinate system. The total station coordinate system was used as the world coordinate system. The coordinate system direction was vertically upwards (Y) and horizontal (XZ) and the system constituted a right-hand coordinate system. 2.
The same camera was used to take photographs from nine different locations. The calibration control points were extracted from the image, and the three-dimensional coordinates were used to calibrate all of the cameras in a unified world coordinate system. 3.
The 2D line was extracted from the image. The corresponding image point was the top point of the target.

4.
DtP and PtD were used separately to obtain the results. For hard-to-find pairs of corresponding image points on the target, TPS was not used.
All the target directional results are shown in Table 7. It can be seen from the table that the results of the two methods were essentially the same.

Conclusions
Considering the problem of 3D line reconstruction in binocular or multiple view stereo vision, where there is only a pair of corresponding points on the line, a new method called PtD was proposed. Compared to DtP and TPS methods, PtD uses both point and line information for 3D line reconstruction. The simulation results in this paper demonstrate that our method achieves the best accuracy for determining direction; and although the positional accuracy of our method is worse than TPS, it is better than DtP. The physical experiment shows that under bad intersection conditions, the PtD method achieves better accuracy than the DtP method.
Author Contributions: Q.Y., X.Z., Y.S., X.Y., H.Z., and L.Z. participated the design of the method; L.Z. carried out the software implementation, experiments design, data analysis and manuscript writing; X.Y. checked the manuscript; J.Q. carried out the physical experiments and data analysis; X.Z. provided financial support; All authors have read and approved the final manuscript.