Prediction and Optimization Algorithm for Intersection Point of Spatial Multi-Lines Based on Photogrammetry

The basic theory of photogrammetry is mature and widely used in engineering. The environment in engineering is very complex, resulting in the corners or multi-line intersections being blocked and unable to be measured directly. In order to solve this problem, a prediction and optimization algorithm for intersection point of spatial multi-lines based on photogrammetry is proposed. The coordinates of points on space lines are calculated by photogrammetry algorithm. Due to the influence of image point distortion and point selection error, many lines do not strictly intersect at one point. The equations of many space lines are used to fit their initial value of intersection point. The initial intersection point is projected onto each image, and the distances between the projection point and each line on the image plane are used to weight the calculated spatial lines in combination with the information entropy. Then the intersection point coordinates are re-fitted, and the intersection point is repeatedly projected and recalculate until the error is less than the threshold value or reached the set number of iterations. Three different scenarios are selected for experiments. The experimental results show that the proposed algorithm significantly improves the prediction accuracy of the intersection point.


Introduction
Cranes are important equipment for the development of modern industry and are widely used in ports, metallurgy, urban construction, aerospace, petrochemical and other fields. The role of the crane in the process of cargo transportation is irreplaceable as it can transport high-load cargo, such as the hoisting of offshore drilling platforms, the assembly of heavy ships, and the hoisting of nuclear power plant containment domes [1][2][3][4][5].
With the continuous development of hydraulic technology [6], computer technology [7], communication technology [8], advanced control technology [9], and new energy technology [10][11][12], the intelligence level of cranes is also constantly improving, but its structural safety [13] is always an important part of its safety. Once there is a problem with the structural safety of large construction machinery, it is easy to cause significant loss of life and property [14]. Therefore, large-scale construction machinery such as hoisting machinery is also a highly dangerous mechanical equipment [15].
In the past 20 years, safety assessment has been gradually applied in structural engineering, chemical disasters, information security, and other fields, while there have been relatively few safety assessment theories for cranes [16][17][18]. The load of the crane is usually large, ranging from several tons to thousands of tons, and the lifting height ranges from tens of meters to hundreds of meters [19]. Therefore, with the increasing application of cranes, the number of accidents is also rising. Therefore, it is very important to carry out safety assessment and risk monitoring of cranes.
The structural safety assessment of cranes refers to the assessment of the potential hazards and severity of mechanical structures. Commonly used crane risk assessment methods include fuzzy comprehensive assessment method, risk assessment method based on combination weighting, comprehensive risk assessment method, etc. [20]. But no matter which risk assessment method is used, key dimensions or the coordinates of key points need to be obtained. For construction machinery in complex working environments, truss is a common local structure. The key points on the crane usually show a face on the image, so the selection of measuring tools is particularly critical. Especially for the complex local structure of the crane, the design scheme of the engineering test is also critical, which is related to whether the key point coordinates can be accurately measured. After the key points are measured, they can be used as the basic data for crane safety evaluation. These data will play an important role in crane safety evaluation.
In the field of engineering, the commonly used measurement methods are: coordinate measuring machine, articulated arm measuring machine, laser tracker, structured light measurement system, total station, laser scanner, photogrammetry, etc. [21]. The threecoordinate measuring machine (CMM) has high measurement accuracy, good flexibility, and strong reverse engineering capabilities [22]. It is widely used in the mold industry, and it is a modern intelligent tool that integrates design, testing, and statistical analysis. As a portable measuring device, the articulated arm measuring machine needs to touch the point to be measured in space to complete the measurement [23]. The laser tracker uses a spherical coordinate system and relies on single-frequency laser interferometric ranging, which has high measurement accuracy and fast measurement speed. Moreover, laser tracker has certain advantages in large scenes [23]. Structured light measurement systems are divided into line structured light measurement systems and surface structured light measurement systems, which use the principle of triangulation to obtain the three-dimensional coordinates of points [24]. The total station is a high-precision measurement device that integrates light, machinery, and electricity. Its basic measurement principle is the same as that of the laser tracker, but the distance measurement method is different. Its measurement distance is long and its application range is very wide [25]. Laser scanners use a large number of laser points to form point clouds, so that three-dimensional information of the outer surface of the object to be measured can be obtained [26]. Photogrammetry is based on the relative positional relationship between the photographic center, image point, and spatial point, using two or more images to complete the coordinates calculation of the spatial points, and the measurement distance can be close or far [27]. Among the above measurement methods, the CMM, the articulated arm measurement machine, and the structured light measurement system have shorter measurement distances. Both the articulated arm measuring machine and the laser tracker are contact measuring methods. The operation of the total station is complicated, and the coordinates of a large number of points must be measured one by one. Laser scanners are extremely expensive to purchase and expensive to maintain. None of these measurement methods can meet the measurement needs of large construction machinery such as port machinery. The structure of large construction machinery such as port machinery is usually relatively tall, the range to be measured is relatively wide, and its working environment is complex. These factors determine that the measurement method that can be widely used in port machinery and other large construction machinery must be efficient, non-contact, portable, and have long measuring distance. Among the existing measurement methods, photogrammetry is undoubtedly the most advantageous. With the development of computer technology and artificial intelligence technology, the accuracy and stability of image-based visual measurement are also constantly improving. The intersection of close-range photogrammetry and advanced technologies such as computer vision is getting deeper and deeper, so the application range of close-range photogrammetry is also expanding [28].
The demand for close-up photogrammetry in aerospace [29], automobile manufacturing [30], mold manufacturing [31], material science [32], biomedicine [33], cultural heritage protection [34], etc. is increasing. After close-range photogrammetry entered the digital stage in 1980, many digital close-range photogrammetry systems appeared. There are TRITOP system of German GOM company, DPA-Pro system of German AICON 3D company, etc., whose technologies all come from the IAPG Institute. In addition, there are the V-STARS system of GSI Corporation of the United States, the Australis system of Photometrix Corporation of Australia, and the Metronor system of Metronor Corporation of Norway. The V-STARS system was developed by GSI Corporation of the United States and is currently the most mature commercial digital close-range industrial photogrammetry system in the world [35]. The Lensphoto multi-baseline digital close-up photogrammetry system, which was directly participated and launched by Academician Zhang Zuxun of Wuhan University, has been applied in the fields of water conservancy and electric power measurement, building measurement, cultural relics protection, and other fields [36].

Related Work
The application of close-range photogrammetry in port machinery and other large construction machinery is also increasing. Lu Enshun from Wuhan University of Technology [37] developed a photogrammetry-based radius measurement algorithm for the structural characteristics of port machinery, he used the approximate distortion model of the camera to determine the weight of each midperpendicular, and compared the equal-weight fitting algorithm with the weighted fitting algorithm. Wang Qi [38] developed a focal length calibration algorithm based on photogrammetry for port cranes. The central idea is to obtain image information through the Internet of Things technology, and then obtain the attitude parameters by minimizing the error function. Lin Xuanxiang [39] disclosed a method for detecting the deformation of the main beam of a hoisting machine based on photogrammetry, which uses photogrammetry to obtain the coordinates of the point to be measured, and then fits a curve according to the obtained coordinates to determine the deformation of the main beam. At present, the application of photogrammetry in large-scale construction machinery is relatively small, and there is a relatively broad research space.
In the theory of close-range photogrammetry [40], there are usually observation errors, so redundant observations are introduced to increase the accuracy of the calculation. Using multiple images to complete the space intersection is a commonly used adjustment method. Li Zhongmei added the overall least squares estimation in the process of multi-image space intersection and removed the gross error. Li Jiatian [41] combined the space intersection with the neural network to reduce the influence of nonlinear errors on solving threedimensional coordinates. Faugeras and Mourrain [42] creatively gave a new derivation of the three-focal tensor equation, using three images to complete the solution; this method has been widely used in various fields. In the optimal solution triangulation of three views, Stewenius et al. used the method of interactive algebra to solve the polynomial matrix to obtain the triangulation result [43]. Agarwal et al. addressed the problem of global triangulation using fractional programming methods [44]. Dai et al. proposed a norm-based optimization method to improve the efficiency of triangulation [45]. In the multi-view triangulation, Zhang et al. selected some observation vectors as subsets, and then solved the subset data, which also improved the efficiency of triangulation [46].
Entropy is a thermodynamic concept proposed by physicist R. Clausius. C.E.Shannon first introduced entropy in thermodynamics into information theory. The appearance of information entropy is a sign of the generalization of entropy. It is widely used in physics, chemistry, medicine, water conservancy, communication, and mechanical safety assessment [47]. Since information entropy can measure the uncertainty of the appearance of things, it is also widely used in image processing to increase the accuracy of feature extraction, but there are few studies that can combine the basic principles of information entropy and photogrammetry.
For the safety assessment of large machinery such as port machinery, critical dimensions or the coordinates of key points are very important. Due to the limitation of the shooting angle or the occlusion of other mechanical structures, the key points are likely to be occluded or cannot be directly measured. Inspired by previous studies by scholars, the key points for dimension measurement or safety assessment are generally the intersection points of three straight lines or corner points with obvious features [48,49]. Combining structural features of large construction machinery, a prediction and optimization algorithm for the intersection point of multiple space straight lines is proposed [41,50,51]. In order to comprehensively consider the distortion of image points [52,53], the error of point selection, and the different shooting conditions of each image, the algorithm introduces information entropy [54,55]. Then the spatial lines involved in the calculation of fitting points are weighted [37] to further improve the fitting accuracy.
Therefore, compared with the previous work, the innovations of this paper are as follows: (1) we use multiple images to predict and optimize the intersection of straight lines; (2) an iterative optimization method based on reprojection is proposed; and (3) we introduce information entropy and weight the space line.
The rest of this paper is arranged as follows. Section 3.1 introduces the intersection point prediction algorithm of equal-weighted multiple lines. Section 3.2 introduces the intersection point prediction algorithm of weighted multiple lines. Section 4 introduces three experiments, all of which are used to verify the accuracy and stability of the algorithm in this paper. Section 5 provides a detailed and comprehensive analysis of the experimental results. Section 6 summarizes the entire paper.

The Traditional Intersection Point Prediction Algorithm of Multiple Straight Lines
The manufacturing and installation process of the camera is very precise, but there will still be certain errors [56]. At the same time, when calculating the coordinates of the spatial points corresponding to the image points through the basic principle of photogrammetry [40], the selection of the image points will also have certain errors. Various factors lead to errors in the calculation of the spatial coordinates of the points to be measured. Hence, there will also be errors between the coordinates of the prediction intersection point and the ideal point.
Taking the intersection point of three straight lines as an example, as shown in Figure 1, there are seven image points in the image plane, namely, a, b, c, d, e, f , and g, all of which are ideal image points, and the corresponding ideal space points are A, B, C, D, E, F, and G. The actual selected image points are a , b , c , d , e , f , and g , and the actual spatial points calculated by photogrammetry are A , B , C , D , E , F , and G . The straight line formed by points a and b is l ab , the straight line formed by points c and d is l cd , the straight line formed by points e and f is l e f , and the straight lines l ab , l cd , and l e f intersect at point g. Due to optical distortion and point selection error, the straight line formed by points a and b is l a b , the straight line formed by points c and d is l c d , and the straight line formed by points e and f is l e f . The straight lines l a b , l c d and l e f will not strictly intersect at the point g . Whether it is the image plane where the image plane lines l ab , l cd , and l e f are located or the object space where the space straight lines L AB , L CD , and L EF are located, their schematic diagrams of the intersection point can both be represented by Figure 1. According to the basic principles of photogrammetry [40], the coordinates of the six spatial points A , B , C , D , E , and F are obtained, and then the equations of the three straight lines L A B , L C D , and L E F are obtained according to the spatial coordinates of the six points. The three straight lines L A B , L C D , and L E F will not strictly intersect at the point G . To obtain the spatial coordinates of the fitting point G , a mathematical model is established to minimize the sum of the distances from the fitting point G to the three spatial straight lines. where D 1 , D 2 , and D 3 can be considered to be errors caused by camera distortion and point selection. Therefore, when the sum of (D 1 + D 2 + D 3 ) reaches the minimum value, the coordinates of point G are the desired coordinates. To facilitate the calculation, the minimum value of the sum of (D 1 + D 2 + D 3 ) is converted into the minimum value of the sum of (D 1 2 + D 2 2 + D 3 2 ). The calculation formula is as follows.
W is the function value, and the MATLAB optimization toolbox is used to constrain the nonlinear minimization function W to obtain coordinates of the point G . This method is the traditional algorithm (TA).

Intersection Point Prediction Algorithm of Weighted Multiple Lines
The initial coordinate value of the fitting point G can be obtained from the formula in Section 3.1, and the corresponding image point g can be obtained by projecting the spatial point G onto the image. The calculation formula is as follows [40]. (9) where X S , Y S , Z S , ϕ, ω, and κ are the exterior orientation elements of the camera, and ∆x, ∆y, and f are the intrinsic parameters of the camera. When the error of the image point is larger, the distance from point g to the straight line where the image point is located is also larger. When using multiple images to solve, due to the different shooting conditions of each image, the confidence of each image is different, and the confidence of the straight line composed of spatial points is also different. Thus, the information entropy is used to represent uncertainty. The larger the entropy value is, the smaller the weight of the indicator. The smaller the value is, the greater the weight. For n images, m space straight lines, the distance between the jth straight line on the ith image and the point g on the image is denoted as d g (i, j), and there is a negative correlation between the weight P e of each spatial line and the corresponding d g , that is, P e ∝ d g −1 . Therefore, the information entropy formula of the jth straight line in space is as follows.
Then, the weight formula of the jth spatial straight line is as follows.
where v i = 1 − H i . Therefore, the formula for calculating the sum of the errors is as follows.
where D j is the distance from point G to the jth straight line. This method is called the weighted intersection point prediction and optimization algorithm (WIPPOA). The iterative calculation flow of this method is shown in Figure 2.

Experimental Flow
Since the effect of the algorithm can not be presented by computer simulation, the physical model is taken, and three images and three straight lines are taken as examples for verification experiments. In order to verify the stability and accuracy of the algorithm, three scenes are selected in the experiment. The first experimental scene is for the black and white square model, the feature points of the model are very clear. The second experiment is aimed at engineering scene, the object to be measured is shot without obvious identification points. The third experimental scene is also aimed at engineering scene, but this object to be measured is shot with clearly marked points. The experimental flow chart is in Figure 3. The coordinates of the points forming the lines are calculated based on the basic principles of photogrammetry [40], and then the equations of the three lines are calculated. Finally, the intersection point of the lines are fitted by the method of Sections 3.1 and 3.2, and the experimental effects of the two algorithms are compared.

First Experiment
As shown in Figure 7, the experimental model is composed of black and white squares with a side length of 100 mm. Points 1, 2, 3, and 4 are control points, and points A, B, C, D, E, F, and G are to be measured. Points A and B determine the straight line L AB , points C and D determine the straight line L CD , points E and F determine the straight line L EF , and point G is the intersection point of the straight lines L AB , L CD , and L EF . There will be a certain error between point G fitted by the three straight lines and the ideal point G, and the error is used to measure the accuracy of the two methods. The Sony A7R4A camera is used to complete the experiment, and the camera is calibrated by the MATLAB calibration tool. The camera parameters are shown in Table 1, where f is the focal length of the image, ∆x and ∆y are the principal point deviations of the image, k 1 and k 2 are the radial distortions, q 1 and q 2 are the tangential distortions, and s 1 and s 2 are the poise prism distortions. The object space coordinates of the control points and points to be measured are shown in Table 2. Three images are taken at three camera stations, as shown in Figure 4. According to the basic principle of photogrammetry [40], the three images are used to complete the calculation of the points to be measured.
The image plane coordinates of the three images are shown in Table 3. The six exterior orientation elements of the three images are shown in Table 4. The calculated coordinates of the space points are shown in Table 5.

Second Experiment
As shown in Figure 8, points 1, 2, 3, and 4 are control points, and points A, B, C, D, E, F, and G are to be measured. A Cannon 5DS camera is used to complete the experiment. The camera parameters are shown in Table 6. As shown in Figure 5, three images are taken at three camera stations. The object space coordinates of the control points and points to be measured are shown in Table 7.  The image plane coordinates of the three images are shown in Table 8. The six exterior orientation elements of the three images are shown in Table 9. Table 9. Exterior orientation elements of three images. The calculated coordinates of the space points are shown in Table 10.

Third Experiment
As shown in Figure 9, points 1, 2, 3, and 4 are control points, and points A, B, C, D, E, F, and G are to be measured.
A Cannon 5DS camera is used to complete the experiment, and the camera parameters are shown in Table 11. The object space coordinates of the control points and points to be measured are shown in Table 12.  The image plane coordinates of the three images are shown in Table 13. The six exterior orientation elements of the three images are shown in Table 14. The calculated coordinates of the space points are shown in Table 15.

Results and Discussion
The three experimental results of the weighted algorithm based on information entropy proposed in Section 3.2 are shown in Tables 16-18. Across 10 iterations, the object space coordinates of the fitting point G and the weights of the space lines are updated.  Table 19 shows the coordinates of point G obtained by algorithm TA and algorithm WIPPOA. The object of first experiment is a model with simple background and obvious feature points, so the error is small. The second experiment and the third experiment are engineering experiments with complex background, so the error is large. As can be seen from Tables 16-18, the algorithm can converge after a few iterations, and the convergence speed is relatively fast. As can be seen from Figure 10, the results of weighted iteration algorithm based on information entropy proposed in Section 3.2 are superior to equal weight iteration algorithm in all three scenarios. Therefore, the above experiments can show that the proposed algorithm can effectively improve the fitting accuracy of the intersection point, and has good stability and wide applicability.

Conclusions
Based on the basic theory of photogrammetry and the structural characteristics of large engineering machinery such as port machinery, a prediction and optimization algorithm for intersection point of spatial multi-Lines based on photogrammetry is proposed under the condition of considering the influence of many factors such as image point distortion and point selection error. The algorithm takes the spatial fitting point calculated by the traditional method as the initial point of weighted iteration. The projection points on each image are obtained from the spatial fitting points, and the distance between the projection points and each line in the image plane is combined with the information entropy to determine the weight of the space line, and then more accurate spatial coordinates of the intersection points can be obtained. Experimental results show that the proposed iterative optimization algorithm based on information entropy can significantly improve the accuracy of fitting points, and this algorithm has strong practicability.
Due to the limitations of experimental conditions and in order to better present comprehensive experimental data, the experiment in this paper only uses three images to calculate the intersection point. In subsequent studies, more images can be used to solve the problem to obtain more accurate fitting point coordinates. This method is not only applicable to port machinery, but also to other large engineering machinery. At the same time, this paper also introduces the information entropy in the proposed algorithm and uses information entropy to determine the relative weights. In future studies, we will focus on other structural characteristics of port machinery and other large machinery, and more a targeted algorithm will be put forward. We will make the photogrammetry applied more widely in engineering, at the same time, we will try to apply information entropy to more aspects of photogrammetry.
Author Contributions: Z.Z. and G.W. guided the direction of the paper. H.X. provided the experimental platform and completed the experiment. C.Z. analyzed the experimental data and wrote the paper. All authors have read and approved the final manuscript.