Visual Saliency Detection for Over-Temperature Regions in 3D Space via Dual-Source Images

Abstract

To allow mobile robots to visually observe the temperature of equipment in complex industrial environments and work on temperature anomalies in time, it is necessary to accurately find the coordinates of temperature anomalies and obtain information on the surrounding obstacles. This paper proposes a visual saliency detection method for hypertemperature in three-dimensional space through dual-source images. The key novelty of this method is that it can achieve accurate salient object detection without relying on high-performance hardware equipment. First, the redundant point clouds are removed through adaptive sampling to reduce the computational memory. Second, the original images are merged with infrared images and the dense point clouds are surface-mapped to visually display the temperature of the reconstructed surface and use infrared imaging characteristics to detect the plane coordinates of temperature anomalies. Finally, transformation mapping is coordinated according to the pose relationship to obtain the spatial position. Experimental results show that this method not only displays the temperature of the device directly but also accurately obtains the spatial coordinates of the heat source without relying on a high-performance computing platform.

1. Introduction

In the path planning of mobile robots, it is common to construct a map using the dynamic vision fusion of cameras and multi-sensors [1,2,3]. In a specific industrial environment, the robot needs to monitor the temperature of the equipment and work in the area of abnormal temperature points. The existing neural network control method shows high stability [4,5,6], but it also needs to accurately find the location of the abnormal temperature’s point. Traditionally, using a visible-light binocular camera to reconstruct the target is not possible, because it cannot accurately operate on the abnormal temperature point area [7,8,9,10]. At present, the most commonly used temperature detection methods use sensor contact measurements [11,12,13]. However, there are installation and use problems in engineering applications, so non-contact space measurements can be used to solve the installation problem. Visual target detection can solve this problem.

In the field of target detection, deep learning is a commonly used technology. At present, in 2D target detection, many methods of optimizing the structure of deep convolutional neural networks improve the accuracy of target detection [14,15,16], such as fully convolutional networks (FCN), progressive fusion [17], multi-scale depth encoding [18], and data set balancing and smearing methods [19,20]. In mobile robot navigation, precise positioning of the target often requires obtaining spatial coordinates. The depth camera can be used to obtain depth information for 2.5D target positioning. The deep network also plays an important role in this field. The variational autoencoder [21,22], the adaptive window and weight matching algorithm [23], the deep purifier, and the feature learning unit greatly improve the accuracy of detection. However, deep learning requires more sophisticated hardware and relies on a large number of training samples [24,25,26,27].

With the development of 3D reconstruction technology, the application of 3D reconstruction technology in real life has become extensive, attracting the attention of many experts and scholars [28,29]. Commonly used 3D visual reconstruction methods include feature extraction and matching, sparse point cloud reconstruction, camera pose solution, dense point cloud reconstruction, and surface reconstruction [30,31,32,33]. Through the research of different experts and scholars, related technologies such as feature matching, depth calculation, and mesh texture reconstruction have made great breakthroughs, which have resulted in a higher degree of reduction in visual 3D reconstruction [34,35,36].

The method proposed in this paper mainly uses ordinary and infrared cameras to take pictures of targets and then sparse point cloud reconstruction through ordinary pictures to obtain the pose of the camera when imaging. Then, image fusion is performed on ordinary pictures and infrared pictures. The original camera’s internal and external parameters do not change. The original image can be replaced with the fusion image to surface-map the dense point cloud in order to generate a three-dimensional surface. In addition, a three-dimensional reconstruction target that visually displays the surface temperature is obtained [37,38,39]. This paper uses an adaptive random sampling algorithm to obtain the main texture features, remove redundant point clouds, and finally, use the depth confidence to filter the wrong point clouds [40,41,42].

To reduce the calculation cost and dependence on training samples, this paper mainly uses the characteristics of infrared images to detect the center coordinates of the heat source. First, the infrared images are pre-processed by channel extraction and image segmentation. Then, the position of the two-dimensional plane temperature abnormal points is detected. Finally, the coordinate transformation is calculated based on the camera’s imaging pose relationship in order to calculate its spatial coordinates [43,44,45]. Therefore, it is possible to use the reconstructed target as an obstacle to plan the movement path of the robot and to work on the temperature abnormal point area according to the obtained spatial coordinate information. The schematic diagram is shown in Figure 1. The robot rotates around the target center once to reconstruct a complete target and quickly finds the center position of the heat source that needs to be operated using the above method.

2. Materials and Methods

The process of sparse point cloud reconstruction is as follows: Feature extraction, feature matching, elimination of mismatched pairs, 3D point cloud initialization, and camera pose calculation. Among these steps, the image mismatch elimination and pose solution have a great impact on the sparse point cloud reconstruction effect. The text uses the random sample consensus (RANSAC) algorithm to remove false matches and the beam adjustment method to recalculate the camera pose. The visible light camera used in this article is a 200W pixel POE DS-2CD3T25-I3 with a focal length of four millimeters. The device was manufactured by HIKVISION Company in Hangzhou, China.

2.1. Reconstruction of the Sparse Point Cloud to Obtain the Camera Attitude

2.1.1. Use of the Scale-Invariant Feature Transform (SIFT) Algorithm to Find Feature Points

The process of sparse point cloud reconstruction includes feature extraction, feature matching, elimination of mismatched pairs, 3D point cloud initialization, and camera pose calculation. Among these steps, the image mismatch elimination and pose solution have the greatest impact on the sparse point cloud reconstruction effect. The text uses the RANSAC algorithm to remove false matches and the beam adjustment method to recalculate the camera pose.

To realize 3D reconstruction, the feature points of the picture first need to be extracted. The scale-invariant feature transform (SIFT) algorithm is a computer vision algorithm that is used to detect and describe local features of images, find extreme points in the interscale, and extract their position, scale, and rotation invariants. It is divided into the following four steps:

• Multi-scale spatial extreme point detection: This searches image locations on all scales and uses Gaussian differential functions to identify potential rotation invariants and scale candidate points.
• Accurate positioning of key points: After determining candidate positions, a high-precision model is fitted to determine the scale and position. The stability of key points is used as the basis for selection.
• Calculation of the main direction of key points: Based on the local gradient direction of the image, each key point obtains one or more directions. In the future, the image processing will be transformed relative to the key-point scale, direction, and position to ensure the invariance of the transformation.
• Descriptor construction: In the field of key points, the direction of local gradients is measured according to the scale selected above, and these gradients are transformed into another representation.

The effect of feature point extraction is shown in Figure 2.

This shows the reconstruction of a potted plant on a 3.0 GHz CPU desktop computer, selecting 30 consecutive shots at a resolution size of $4000\times 3000\mathrm{ppi}$. The maximum calculation memory required during the reconstruction process, before using the adaptive sampling algorithm, is 5.3 GB. After adapting to the sampling algorithm, it is 3.2 GB, which proves that the algorithm effectively reduces the memory required for calculation.

2.1.2. Error Matching Elimination Based on the RANSAC Algorithm

There will be matching errors after feature matching. RANSAC is a commonly used error elimination algorithm. The grid-based motion (GMS) [46] algorithm, recently proposed by scholars, can match features in a short time and is very robust. It can remove wrong matches to a certain extent. However, the original author notes that the GMS algorithm is suitable for supplementing the RANSAC algorithm but not replacing it. Therefore, this article mainly uses the RANSAC algorithm to eliminate wrong feature matching. The algorithm works by using Equation (2) as the cost function to iteratively update the sample set.

$$\mathrm{s}\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(1)

$${\displaystyle \sum }_{i=1}^n{(x_i{}^{\prime }\frac{h_{11}{x}_i+h_{12}{y}_i+h_{13}}{h_{31}{x}_i+h_{32}{y}_i+h_{33}})}^2+{(y_i{}^{\prime }\frac{h_{21}{x}_i+h_{22}{y}_i+h_{23}}{h_{31}{x}_i+h_{32}{y}_i+h_{33}})}^2$$

(2)

In the above formula, $\left(x,y\right)$ represents the corner position of the target image, $\left(x^{\prime },y^{\prime }\right)$ is the corner position of the scene image, s is the scale parameter, and H is a 3 × 3 homography matrix.

Error matching elimination based on RANSAC is shown in Figure 3.

2.1.3. The Position Pose of the Phase Machine Is Solved by the Beam Adjustment Method

After the image alignment, the 3D point cloud and camera pose can be obtained. However, there will be interference noise when calculating the position and the 3D point, and there will be significant error in the subsequent calculation. Therefore, bundle adjustment is used to reduce the error [9], and the P matrix and F matrix of each picture after correction can be obtained. The reprojection error is defined as:

$$\mathrm{E}={\displaystyle \sum }_j{\rho }_j\left(\pi \left(P_C,X_k\right)-x_j{}^2\right)$$

(3)

where $\pi$ is a projection matrix from three-dimensional to two-dimensional, ${\rho }_j$ is a kernel function, and $\pi \left(P_C,X_k\right)-x_j{}^2$ is a cost function. Figure 4 shows the sparse point cloud obtained after the bundle adjustment (BA) algorithm is used to solve the position pose. The green dot is the posture of the solved camera.

2.2. Three-Dimensional Surface Generation

2.2.1. Adaptive Random Sampling

• A pixel point $\widehat{x_i}$ is randomly selected from the obtained point cloud image. $D_i\left(x_i\right)$ is the depth value of the pixel point and is inversely mapped into the three-dimensional space according to Equation (4). The tangent plane $P\left(\widehat{x_i}\right)$ is obtained according to the normal direction. $K_i$ is the camera internal parameter, $R_i$ is the rotation matrix, and $T_i$ is the translation vector.

  $$P\left(\widehat{x_i}\right)=R_i{}^T(K^{-1}{D}_i\left(\widehat{x_i}\right){\left[\widehat{x_i}1\right]}^T-T_i)$$

  (4)

Specific steps are as follows:

• Expand outwards with $\widehat{x_i}$ as the center, expand the radius r one pixel at a time, and calculate the three-dimensional coordinates $P\left(x_i{}^{\prime }\right)$ of each pixel $x_i{}^{\prime }$ in the expansion range.
• Calculate the distance $d_i$ of each pixel $x_i{}^{\prime }$ to the tangent plane within the current expansion range, and set the threshold size as $t_d$. If $d_i\le t_d$, the pixel point can be considered to be in the smooth area, and the point can be removed.
• When the expansion radius r is larger than the maximum expansion radius $r_{max}$, or a point cloud of a certain proportion of $p_i$ in the expansion range is removed, the expansion stops. $r_{max}$ and $p_i$ are tunable parameters. They can be determined according to the point cloud redundancy. During debugging, it is found that there are still many redundant point clouds after culling. $r_{max}$ can be increased and $p_i$ can be decreased. If the point cloud is over-eliminated, the parameter adjustment method is reversed.
• Then, randomly select a pixel point and repeat the above steps until all the sampling points in the current 3D point cloud image are sampled.

2.2.2. Deep Confidence Removes the Cloud of Error Points

$$E_d\left(P\left(\widehat{x_i}\right)\right)=\frac{{{\displaystyle \sum }}_{t^{\prime }ϵN\left(t\right)}{\left|\left|D_i\left(\widehat{x_i}\right)-D_i\left({\widehat{x}}_{i\to i^{\prime }}\right)\right|\right|}^2}{\left|N\left(i\right)\right|}$$

(5)

The above formula is the depth value estimation of the point cloud, i.e., the larger the estimated value, the smaller the error value and the higher the reliability. Among these values, $E_d\left(P\left(\widehat{x_i}\right)\right)$ is the depth value estimation of two adjacent frames, ${\widehat{x}}_{i\to i^{\prime }}$ represents the projection point of the $i^{\prime }$ pixel projected by the current pixel, and $N\left(i\right)$ represents the number of frames taken. The specific steps are as follows:

• The point cloud for the current frame k is sorted from high to low according to the estimated value, and the confidence threshold ${\epsilon }_d$ is set, starting from the point where the estimated value is the smallest. If $E_d\left(P\left(\widehat{x_i}\right)\right)<{\epsilon }_d$, the point is eliminated, the calculation continues until $E_d\left(P\left(\widehat{x_i}\right)\right)>{\epsilon }_d$ stops, and the remaining point clouds are stored in the sequence $S_k$. Then, the same calculation is performed on the next frame point cloud image until the point cloud image is calculated and the sequence set $S=\left\{S_k\right|k=1,\cdots ,\mathrm{n}\}$ is obtained.

• Starting from the k frame depth map, all three-dimensional points $\widehat{x_i}$ are mapped to ${\widehat{x}}_{i+1}$ on the k + 1 frame. Compare the estimated values of the two points, the s.

• maller three-dimensional coordinates of the larger estimated points of the estimated values, and so on, until all depth maps are completed.
• The three-dimensional sampling points of all depth maps are intersected to obtain the final three-dimensional point cloud image. Then, perform the mesh reconstruction and mesh texture generation on the filtered dense point cloud. The effect before and after filtering is shown in Figure 5.

The reconstruction details are shown in Figure 6.

2.3. Image Fusion

After reconstructing the sparse point cloud, the camera parameters are obtained. The original image can be corrected for distortion. The infrared image can be calibrated and corrected by itself. The image registration error is shown in the following formula:

$${\sigma }_x=\frac{f·d_x}{l_{pix}}(\frac{1}{D_{target}}-\frac{1}{D_{optimal}})$$

(6)

where f is the focal length, $l_{pix}$ is the pixel size, and $d_x$ is the baseline length.$D_{optimal}$ is the target distance, and the alignment error of the image is zero. Only objects that are far away from the camera will be precisely aligned.

2.3.1. Calculate Scale Factor

As the focal length and resolution of infrared and visible images are different, the imaging size of objects in space from the two camera types is not consistent. At the same time, the optical center of the hardware systems of the two camera types deviates in the Y direction. Therefore, it is not easy to scale the image by focal length.

The method adopted in this paper calculates the pixel difference between two corner points in infrared and visible images by using the checkerboard calibration board to obtain the image scale.

$$scale=\frac{infrare{d}_n-infrare{d}_{n-1}}{visibl{e}_n-visibl{e}_{n-1}}$$

(7)

It is assumed that the checkerboard calibration board corner with $k$ line, $l$ column, namely $kl$, is accumulated. n is the corner number on the checkerboard, the upper left corner is minimum 1, and the lower right corner is the maximum kl. The values increase from left to right, and from top to bottom,$infrare{d}_n$ is the x or y coordinate of the corner n on the infrared image, and $visibl{e}_n$ is the X or Y coordinate of the corner n on the visible light image.

2.3.2. Relative Offset of the Image

The factor scale is used to realize the unification of space objects in infrared and visible images. Then, the same corner point on the checkerboard is selected to calculate the relative offset of infrared and visible images.

$$X_{diff}={infrared}_x-{visible}_x$$

(8)

$$Y_{diff}=infrare{d}_y-visibl{e}_y$$

(9)

where $X_{diff}$ and $Y_{diff}$ are the offsets required for each pixel in the infrared image. The RGB color model is a color standard in the industrial world. It obtains various colors by changing the three-color channels of red (R), green (G), and blue (B) and by superimposing each on others. After the completion of each pixel offset, the values of the three channels of RGB of the infrared and visible pixel pairs in the same coordinate can be fused, and the fusion effect is shown in Figure 7. Figure 7 is the heating plate placed in the carton. An infrared camera with a resolution of $384\times 288\mathrm{ppi}$ is used. The infrared camera and visible light camera take pictures at the same time.

The camera pose is calculated based on the reconstructed sparse point cloud, and all the fused pictures are surface-reconstructed. The 3D reconstruction effect of the temperature display is shown in Figure 8.

2.4. 3D Target Detection

As shown in Figure 9, in this experiment, a high-temperature bottle is used as the temperature abnormal region of the overall device, and its spatial coordinates need to be calculated.

2.4.1. Target Detection of the Heat Source

In the infrared picture, the pixel temperature generated by the detection is proportional to the R channel value, so the image can be preprocessed first. The R channel value size of the original image is extracted, and all pixels are sorted according to the R value. However, noise in the image is unavoidable and will interfere with the sorting results. To avoid incorrect sorting, the extracted image can be cut and divided into sub-regions. The size of the region can be determined according to the input original image size. Then, the average value of the R channel in each area is calculated, and the area is sorted according to the average value to obtain the R channel size set of each area $R_{agg}=\left\{R_1,R_2,R_3,\cdots R_n\right\}$, assuming $R_{max}$ is its maximum value.

After the infrared image preprocessing is complete, the R channel value of each small area can be obtained. To allow the detection frame to be adaptively scaled, the size of the heat source needs to be calculated, so small squares (that meet the conditions) can be calculated and recorded for each small area location. The criteria are:

$$R_i>k\ast R_{max}$$

(10)

$$siz{e}_r=siz{e}_p\ast p_r$$

(11)

Among them, $R_i$ represents the value of the R channel region, and k is a proportionality coefficient that needs to be adjusted according to specific conditions. After calculating the situation of each sub-region, each region can be assessed, in order from left to right and top to bottom. Each sub-region is set to be square. The size of each sub-region $siz{e}_r$ can be determined according to Equation (11), where $siz{e}_p$ is the size of the infrared image used for detection, the proportion of $p_r$ sub-regions, and $p_r$ is an adjustable parameter. If four of the eight regions around the area meet the conditions, that area is a sub-area within the heat source range, and the position coordinate is recorded and evaluated. Finally, the size of the heat source border can be obtained from the coordinate position. The effect is shown in Figure 10.

2.4.2. Coordinate Transformation Mapping in 3D Space

After the detection of the heat source target, the coordinates of the heat source center in each infrared picture can be obtained; because the shoot is a head-up relationship, the horizontal deviation and the height deviation can also be obtained. The steps are as follows: Take the center of the first picture as the center point of the space and choose another angle during the shoot as the second position. As shown by two positions in Figure 11, calculate the deviation between the actual heat source and the ideal heat source. The following situations can occur:

Figure 12 is a top view of various situations. Taking Figure 12a as an example, cam1_center and cam2_center are the imaging center points of the camera at two positions, “ideal” is the most central position of the heat source processing experiment and is the intersection of the two imaging centers, and “real” is the actual position of the heat source. When the heat source reaches the imaging plane, the distance from the center of the camera is $bia{s}_1$ and $bia{s}_2$, where α is the angle of rotation of the second position relative to the first position. According to its geometric relationship, the rest of the same angle, that is, the angle shown in the figure, is obtained according to the geometric relationship.

$$\begin{gathered} x=bia{s}_1 \\ z=\left(z_1+z_2\right)/2 \\ ligh{t}_1=bia{s}_2/cos\alpha \\ ligh{t}_2=ligh{t}_1-bia{s}_1 \\ y=depth \\ depth=ligh{t}_2/tan\alpha \end{gathered}$$

(12)

In the above formula, z is the height position of the heat source, and $z_1$ and $z_2$ are the deviations from the origin of the space coordinates at the heights taken at the two positions. In order to reduce the operation error, the average of the two positions is taken as the height deviation. $ligh{t}_1$ and $ligh{t}_2$ are the distances in the calculation of geometric relations, respectively. According to the above formula, the head-up deviation x, depth deviation y, and height deviation z can be obtained. As the coordinates in the actual space of the idea are already known, the space coordinates of the actual heat source can be calculated.

Although detection speed has been greatly improved by the enhanced convolutional neural network structure, it still cannot provide high-precision results, and relies on high-performance GPUs. The method in this paper conducted 15 experiments, only running on a 3.0 GHz desktop computer, using the thermos randomly placed in the above figure as a simulated heat source. The camera is 10 m away from the ideal heat source. The error values were obtained from the actual measured coordinates and calculated coordinates. The error results of the experiment are shown in Figure 13. It can be seen from the experimental results that the error value is within ±20 mm, with high accuracy, and the calculation speed is 20 ms, which meets the detection requirements of industrial equipment.

3. Conclusions

The experimental results demonstrate that the method proposed in this paper can fuse target surface temperature information captured by infrared cameras into a three-dimensional point cloud while ensuring the accuracy and speed of the reconstruction and that the reconstructed object can intuitively display its surface temperature. The spatial coordinates of the heat source are calculated using the spatial transformation mapping relationship of the infrared picture. The experimental results demonstrate that the algorithm is highly accurate and meets the requirements of robot navigation and positioning.

4. Patents

A 3D reconstruction method based on point cloud optimization sampling; a 3D surface temperature display method based on infrared and visible image fusion is presented; the invention relates to a method for detecting the heat source center in three-dimensional space.