1. Introduction
A shield tunneling machine [
1] is a special construction machine for tunneling, which has been widely used in subway, railway, highway, and other tunnel projects. With the significant improvement of human construction level and the vigorous development of the tunnel industry, intelligent tunneling equipment has become a development trend. The existing shield tunneling machine, which has been widely used in tunnel construction, is semi-automatic and needs both manual operation and automation execution, such as the changing of the worn disc cutter on the cutter head still needs human to do. Furthermore, the disc cutter consumption is so huge during the construction process that the disc cutter needs to be changed frequently. However, the environment for disc cutter changing is generally quite harsh, especially in mud balance shield machines of high humidity and high pressure, making the manual operation mode a significant potential safety hazard. For example, there have been many disc cutter changing accidents at home and abroad. The disc cutter detection and changing cost more than 10% of the construction period of the tunnel, so it is necessary to develop a disc cutter changing robot to realize the automatic operation. The robot’s vision system, which is used to obtain the disc cutter pose, should have high measuring accuracy and robustness to help the robot accurately grasp the disc cutter.
The visual measurement method includes measurement based on cooperative target [
2,
3] and that based on non-cooperative target [
4], the former is to locate the object by artificial markers fixed on it, and the latter is generally to locate the object by its features. The artificial marks processed on the disc cutter or its holder are easy to be worn; thus, the non-cooperative target method is more suitable for the actual situation. Usually, some prominent geometric features with simple structure and easy recognition are extracted from the object for positioning, such as point [
5], straight line [
6], circle [
7], etc. By using the corner points on the inside contour of the cutter holder, feature points are extracted to estimate the pose of the cutter holder. The feature extraction can be realized by the visual measurement system after image acquisition and processing of the object. According to the dimension information, visual measurement can also be divided into monocular measurement [
8,
9] and multi-ocular measurement [
10]. Monocular measurement is simple in structure, light in weight and low in cost, but it is difficult to obtain the depth information of objects. Multi-ocular measurement can obtain the depth information of objects, which is often used in the 3D reconstruction. Compared with multi-ocular measurement, monocular measurement has no image matching process and only needs a single image to obtain the pose, and has higher positioning efficiency [
11]. The actual disc cutter changing space is very narrow, and the vision system is installed on the end actuator of the disc cutter changing robot. The lower the load of the end actuator, the more dexterous the operation [
12]. In addition, the installation space of the end actuator is limited, and the small volume of the monocular measurement system is easier to install. Therefore, the monocular measurement mode is chosen forthe visual system in the disc cutter changing robot.
Pose measurement based on monocular vision is also called the Perspective-n-Point (PnP) problem [
13]. Currently, there are many methods proposed to solve this problem. Among these, the DLT [
14,
15] method is classical and efficient but vulnerable to noise interference and has poor robustness. Li et al. have proposed an RPnP [
16] method, which can always present stable calculation results regardless of the number of feature points, but it cannot get a unique solution when using the least square error, and its solving accuracy needs to be improved. The EPnP [
17,
18,
19] is a non-iterative method that uses virtual points to indirectly solve the pose parameters of the target, thus reducing its computational complexity, and it also has good anti-noise compared with other PnP methods. However, when the number of feature points is small, its solving accuracy and robustness are relatively low. The basic idea of the ASPnP [
20] and the OPnP [
21,
22] method is to formulate the PnP problem into a functional minimization problem and retrieve all its stationary points using the Gröbner basis technique. These two methods have high accuracy and robustness and are the best PnP methods so far.
The existing PnP methods can guarantee high accuracy when the noise is slight; otherwise, the accuracy is low. Since the image of the cutter holder collected in the actual industrial environment contains strong noise, a method with good robustness to solve the pose of the cutter holder is needed. The distance matching method [
23] is a template matching method that calculates the distance between the template and the measured object image to solve the object pose, which is robust but depends on the initial pose of the measured object. The key part of the distance matching method is the distance transform. Distance transform [
24,
25] is an operation for binary image, which can be used in the template matching method to estimate target pose [
26]. Different distance measures will produce different transformation results, among which the most commonly used distances are Euclidean distance [
27], Manhattan distance [
28], and chamfer distance [
29]. Euclidean distance represents the exact distance between the template and the object image, and the other distances are approximate expressions of Euclidean distance [
30]. The accuracy of the distance matching method depends on the accuracy of the given initial pose, and it is widely used to solve the pose based on contour features [
31]. Hu et al. [
32] introduced a method to estimate the pose of the pipe, which firstly used a template matching method to estimate the pipe’s initial pose, then used the least-squares method to obtain a more accurate pipe pose. A combination of the initial pose estimation method and the precise positioning method can be used to obtain the object pose.
In this paper, we propose the OPnP+DM method to obtain the cutter holder pose of the shield machine. First, before the pose measurement, the pose template library of the cutter holder is established offline. Second, the image coordinates of the feature points on the cutter holder surface can be obtained by the image processing method. We can use these feature points to solve the rough pose of the cutter holder by the OPnP method. Third, we take this initial pose as input and use the distance matching method to solve the pose accurately. Meanwhile, the distance between the collected image of the cutter holder and the template in the established pose template library is calculated. The template corresponding to the minimum distance is selected, and its corresponding pose is used as the final estimated pose of the cutter holder.
The innovations of this paper include:
- (1)
The OPnP+DM method, which consists of the optimal accuracy OPnP method and the highly robust distance matching method, takes advantage of the two methods.
- (2)
The OPnP+DM method extracts the characteristics of the cutter holder without processing the artificial marks, which reduces the labor and improves the stability of visual measurement.
- (3)
The step acceleration method is proposed to speed up the search of the optimal pose in the distance matching method.
- (4)
The proposed OPnP+DM method can achieve a positioning accuracy of 1 mm required by the vision measurement system of the disc cutter changing robot under the condition of strong noise, which cannot be achieved by the PnP methods.
The rest of this paper is organized as follows.
Section 2 states the pose estimation method of the cutter holder. In
Section 3, the simulation results are used to prove the accuracy and robustness of our method. In
Section 4, experiments are implemented to test the feasibility of the practical engineering application of our method. In
Section 5, the conclusions are given.
3. Simulation Results
Due to the harsh industrial environment of high humidity and high pressure, the cutter holder image collected contains intense noise. To meet the requirement of positioning accuracy of 1 mm for the cutter holder pose measurement, this paper focuses on the influence of noise intensity on the method under the fixed number of feature points.
In the simulation experiment, a virtual perspective camera with a resolution of 4112 × 3008 pixels is synthesized, and its focal length and pixel size are 8.5 mm and 3.45 μm, respectively. The movement range of the cutter holder in the
X and
Y directions is [−90, 90] mm, the movement range in the depth direction is [740, 760] mm, and the rotation range in the three degrees of freedom directions is [−60, 60] degrees. A cutter holder pose
p is randomly generated in this space range, and we can obtain the image coordinates corresponding to 20 feature points on the cutter holder surface under this pose state by the camera projection model. After that, different levels of Gaussian noise are added to the image feature points, and 500 test data sets are generated for each noise level. Then the cutter holder pose
pestimation of each test is calculated using PnP methods or the OPnP + DM method. The error between
pestimation and the truth value
p is calculated finally. We can solve the standard deviation of the 500 error values obtained, which is used to represent the solution accuracy of the methods. The variance of interference noise is successively increased from 0.5 to 10 pixels (the step is 0.5 pixels), and the solving accuracy of each degree of freedom direction of the monocular measurement methods is shown in
Figure 6.
The RPnP method has the worst performance among these PnP methods. In most degrees of freedom directions, its accuracy decreases fastest with increased noise intensity. Compared to the other PnP methods, one significant advantage of the RPnP method is that it has higher solution accuracy in the
Z-axis rotation direction. Among the remaining PnP methods, the OPnP method has the best performance in all the directions. Its accuracy is always the highest one among all PnP methods except in the
Z-axis rotation direction. However, it can be seen from
Figure 6 that the accuracy of the OPnP + DM method is higher than that of PnP methods, especially in the depth direction. In addition, with the increase of noise intensity, the solution accuracy of all PnP methods decreases, and the relationship between them is approximately linear. As shown in
Figure 6f, in the
Z-axis direction, the RPnP method can guarantee 1 mm measurement accuracy only when the variance of interference noise is less than 4 pixels. Among the PnP methods, the OPnP and ASPnP methods can also reach the same accuracy on the premise that the variance of interference noise is less than 7 pixels. However, the OPnP + DM method can guarantee an accuracy of 0.5 mm even if the noise variance is 10 pixels. In a word, its measurement accuracy in the depth direction under strong noise is far better than that of the PnP method. According to the results in
Figure 6, the OPnP + DM method has better robustness because its accuracy changes less than the PnP methods when the noise increases. It also has high solving accuracy under strong noise, which makes it more suitable for the disc cutter changing operation site with strong interference. The DM method relies on the initial pose provided by the OPnP method, it is only suitable for a small search range if the initial pose is given randomly. We set the search range of the cutter holder pose in the rotation direction and the movement direction to be [−0.5, 0.5] degrees, [−5, 5] mm, respectively. The average times required by the DM method and OPnP + DM method to search the cutter holder pose is 3.24 s and 19.32 s, respectively. Thus, it can be concluded that the solving speed of the OpnP + DM method is 5.96 times that of the DM method.
4. Experiments and Results
We set up a motion platform in the laboratory to verify that the OpnP+DM method has higher measurement accuracy than the PnP methods under complex conditions and can meet the positioning accuracy of 1 mm required by the visual system on the disc cutter changing robot in the moving directions. The input of the experiment is the cutter holder images collected by the vision system, and the output is the actual pose of the cutter holder measured by the pose estimation methods. We calculate the pose of the cutter holder through the pose estimation methods of the visual system and compare it with the pose recorded by the motion platform.
The four-axis motion platform used in the experiments is shown in
Figure 7. The platform is equipped with high-precision grating rulers in the
X-axis,
Y-axis, and
Z-axis directions, which enables the platform to achieve a movement accuracy of 0.01 mm in these three directions. Moreover, a DD motor with rotation accuracy of 0.02° is installed in the rotation direction of
Z-axis of the platform. The monocular vision system of the platform uses the Allied Vision’s Manta G-1236 camera, which has the same resolution, pixel size, and focal length as the virtual camera in the above simulation experiment. Before the measurement, we first use Zhang Zhengyou’s [
35] calibration method to calibrate the monocular camera, in which a 7 × 7 circular calibration board with a machining accuracy of 0.01 mm is used. After calibration, we can get the internal parameters of the camera: the focal length obtained is 8.56 mm, pixel size obtained is (3.45, 3.45) μm, and the principal point coordinate obtained is (2190.03, 1427.46) pixels.
We should first establish the pose template library of the cutter holder according to the internal parameters of the camera. Within the motion range of the cutter holder described in the simulation experiment, the step length of displacement direction is set as 0.1 mm, and the step length of rotation direction is set as 0.1° to change the pose of the cutter holder. By substituting the internal parameters of the calibrated camera into the projection model, the 20 image coordinates corresponding to the 20 feature points of the cutter holder in each pose state in the moving space can be obtained. The image coordinates of all feature points under each pose are stored, so the pose template library of the cutter holder is constructed, which means each template in the template library contains only the image coordinates of 20 feature points.
After setting up the pose template library, the vision system collects the actual image of the cutter holder and processes it with the image processing method mentioned above, and the results of image processing are shown in
Figure 8. The preprocessing block includes denoising, template matching, and binarization. For median filtering, the field shape is square and the convolution kernel size is 3. The feature points extraction block contains edge extraction, inside contour selection, contour segmentation, feature line segments screening, collinear processing, and intersection calculation. In the feature line segments screening, we set a length threshold of 130 pixels according to the image size of the bolt and other interference structures, and only the part with a length greater than it will be screened out. Finally, the obtained 20 feature points are shown in a small graph at the lower left of
Figure 8, represented by red crosses.
There will be a piece of washing equipment at the construction site to wash the disc cutter that needs to be replaced and its holder. However, the cutter holder cannot be completely cleaned, and there will always be residual silt to block the feature that needs to be detected. In order to simulate the state that the silt covers the cutter holder, we use soil to block the inside contour of the cutter holder randomly in the measurement experiments. We randomly select one or more straight-line segments on the inside contour of the cutter holder for occlusion in the experiments.
Figure 9 shows some screenshots of the occlusion of the straight-line segments, showing the state of occlusion of the inside contour in the experiments. We block the whole straight-line segment when adding soil to ensure that the OPnP+DM method still has good measurement accuracy under such extreme conditions.
The contaminated cutter holder images are processed by the above image processing method. We can obtain the DT image of the binary image containing only 20 feature points, similar to
Figure 4. The OPnP method is used to calculate the initial pose of the cutter holder, which along with the initial rotation step value of 3.2° and the initial translation step value of 3.2 mm are the inputs of the DM method. We can obtain the template with the smallest distance from the actual cutter holder in the pose template library by the DM method. The step acceleration method is used in the search process at the same time. The obtained template pose is taken as the final pose of the actual cutter holder. In the experiment, 100 cutter holder images were collected. The error in the directions of various degrees of freedom can be obtained by comparing the calculated cutter holder pose with the actual pose recorded by the platform.
The PnP methods are also used to solve the pose of the cutter holder in the experiments. We calculated the standard deviation of errors in the direction of 6 degrees of freedom for 100 groups of poses respectively, and the results are shown in
Figure 10. The rotation and translation errors of the cutter holder pose obtained by the OPnP + DM method in
X,
Y, and
Z axes are 0.13°, 0.28°, 0.09°, 0.28 mm, 0.30 mm, and 0.32 mm, respectively. Similar to the simulation results, the RPnP method has the worst accuracy among several PnP methods, especially in the depth direction, with an error of 2.57 mm. The errors of other PnP methods in the depth direction are also more than 1 mm, which cannot meet the requirements of practical visual measurement accuracy. In the
X-axis rotation,
X-axis translation, and
Z-axis rotation directions, the OPnP + DM method has the highest solving accuracy compared with the PnP methods. Besides this, its accuracy in the
Y-axis rotation direction and
Y-axis translation direction are not much different from that of other PnP methods except the RPnP method with poor accuracy. The experimental results show that all the pose estimation methods’ rotation accuracy and translation accuracy are similar in both
X-axis and
Y-axis directions, and the rotation accuracy of the
Z-axis direction is the highest one, which can reach 0.12°. Among the translational degrees of freedom, the translation accuracy in the
Z-axis direction is the lowest one, reaching 2.57 mm, far exceeding the accuracy of 1 mm required for engineering applications. The experimental results are the same as the simulation results when the variance of Gaussian noise is greater than 8 pixels. In a word, the OPnP + DM method presented in this paper has the highest accuracy, especially in the depth direction. It also has good robustness under strong interference because it can still achieve high precision and meet the actual accuracy requirement of 1 mm when the noise suddenly increases.