Adaptive Multi-View Image Mosaic Method for Conveyor Belt Surface Fault Online Detection

Abstract

In order to improve the accuracy and real-time of image mosaic, realize the multi-view conveyor belt surface fault online detection, and solve the problem of longitudinal tear of conveyor belt, we in this paper propose an adaptive multi-view image mosaic (AMIM) method based on the combination of grayscale and feature. Firstly, the overlapping region of two adjacent images is preliminarily estimated by establishing the overlapping region estimation model, and then the grayscale-based method is used to register the overlapping region. Secondly, the image of interest (IOI) detection algorithm is used to divide the IOI and the non-IOI. Thirdly, only for the IOI, the feature-based partition and block registration method is used to register the images more accurately, the overlapping region is adaptively segmented, the speeded up robust features (SURF) algorithm is used to extract the feature points, and the random sample consensus (RANSAC) algorithm is used to achieve accurate registration. Finally, the improved weighted smoothing algorithm is used to fuse the two adjacent images. The experimental results showed that the registration rate reached 97.67%, and the average time of stitching was less than 500 ms. This method is accurate and fast, and is suitable for conveyor belt surface fault online detection.

1. Introduction

Belt conveyor is a kind of continuous transportation equipment in modern production that has been widely used in the coal, mining, port, electric power, metallurgy, and chemical industries, as well as in other fields [1]. The conveyor belt is the key component of the belt conveyor for traction and load-bearing, which can easily produce surface scratch, longitudinal tear, or fracture [2]. If these faults are not detected and handled in time, they may cause equipment damage, material loss, and other substantial economic losses, and even cause personal casualties, affecting safety production [3]. At present, machine vision technology can be used for the online detection of conveyor belt surface fault [3,4,5,6,7]. However, due to the structural characteristics of the conveyor, the distance between the upper conveyor belt and the lower conveyor belt is small. The sight distance is small, and the bandwidth and field of view are large. Single-view conveyor belt surface fault detection is difficult to meet the bandwidth requirements, and there are blind areas and unsatisfactory imaging effects. Therefore, multi-view detection is needed, and multi-view conveyor belt images need to be stitched online.

At present, there are three main image mosaic methods: the method based on gray information, the method based on frequency domain correlation, and the method based on feature correlation. Among them, the method based on feature correlation has been widely focused upon because of its good robustness. The more mature feature operators are SIFT (scale-invariant feature transform), SURF (speeded up robust features), BRIEF (binary robust independent elementary features), ORB (oriented FAST and rotated BRIEF), etc. SIFT algorithm was proposed by Lowe in 1999 and improved in 2004. It not only has good robustness in image rotation and scale scaling, but also can maintain good stability against noise interference, brightness, and viewing angle changes [8,9]. However, the algorithm does not consider the distribution of feature points. It will extract too many feature points in the region with intricate details, which can easily produce false matching. The algorithm is complex in its calculation and is time-consuming. Thus, it is difficult to meet real-time performance.

For this reason, Li et al. [10] proposed an image mosaic algorithm based on the combination of region segmentation and SIFT, which improved the speed of the image. However, the amount of calculation is still large. Bay et al. [11] proposed SURF to overcome the shortcomings of the SIFT algorithm. The SURF algorithm inherits the scale and rotation invariance of the SIFT algorithm and adopts the Haar feature and the integral image concept, making the calculation more efficient. The speed is about three times faster than SIFT. However, the SURF algorithm also has the problem of wrong matching, which reduces the accuracy of the matching results and dramatically slows down the speed due to unstable feature points and wrong matching. Therefore, there are many studies on the performance improvement of the SURF algorithm. Jia et al. [12] constructed a k-dimensional (K-D) tree and improved the best bin first (BBF) algorithm instead of the linear algorithm to speed up SURF matching. Zhang et al. [13] proposed an improved SURF algorithm with a good application prospect in low light conditions. Zhang et al. [14] used Haar wavelet response to establish a descriptor for each feature point and calculated the normalized gray difference and two-step degree of the neighborhood to form a new feature descriptor. The SURF algorithm was improved to make it robust and stable to image blur, illumination difference, angle rotation, and field of view transformation. From overall technology development, image stitching technology has become more and more mature and stable. There are many excellent image stitching algorithms [15,16,17,18,19,20,21,22] that have achieved good results in some scenes. However, image stitching involves many research fields, and these stitching methods have many problems in dealing with special scenes such as conveyor belt surface fault detection. Due to the characteristics of surface material and image acquisition of the conveyor belt, it is not easy to extract and match the feature points of the conveyor belt image.

Moreover, due to a large amount of data in a multi-view image, it is not easy to accurately and quickly stitch online, affecting fault detection and recognition. The current image mosaic method is not suitable for multi-view conveyor belt surface fault online detection. According to the particularity of image mosaic, the accurate and fast image mosaic method is studied.

This paper presents an adaptive multi-view image mosaic method for conveyor belt surface fault online detection based on grayscale and feature combination. This method has a high registration rate and reduces the amount of stitching calculation and the time of the stitching process. It solves the problem of the difficulty of performing accurate and fast online stitching of the collected multi-view conveyor belt images, which affects fault detection and identification.

2. Multi-View Conveyor Belt Surface Fault Online Detection System

The multi-view conveyor belt surface fault online detection system based on machine vision comprises line-scan charge-coupled device (CCD) network cameras, linear light sources, Ethernet switch, and computer. The schematic diagram of the system components is shown in Figure 1.

The device produces diffuse reflection light when the light is emitted by a high brightness linear light source and irradiates on the surface of the conveyor belt. The light intensity of the diffuse reflection light is related to the surface characteristics of the conveyor belt [4,5,6]. The multi-view line-scan CCD camera senses the diffuse reflection light through line scanning. Each scan takes a line of images perpendicular to running direction of the conveyor belt and transmits it to the computer through Ethernet. The computer processes running image of the conveyor belt, analyzes and recognizes the conveyor belt fault, and gives a fault alarm or shutdown control signal when a fault is found.

The structural characteristics of the conveyor limit the installation space of the camera. Generally, the camera can only be installed close to the conveyor belt. If the image of the lower surface of the conveyor belt is to be collected for fault detection, the camera can only be installed in the narrow space between the upper and lower belts, close to the lower surface of the conveyor belt. The distance is less than 40 cm, and the object distance is small, as shown in Figure 2. The width of the conveyor belt is generally 0.8–2.4 m. Because the cross-section is arc and the field of view is large, it is difficult to detect the width and accuracy of the conveyor belt with a single camera. It is necessary to use multiple cameras to acquire images in a multi-viewpoint manner by selecting appropriate installation position and installation postures to realize the conveyor belt without blind area coverage.

However, two adjacent images have overlapping regions that image the same conveyor belt content. Determining the two images’ overlapping regions can effectively reduce the processing range of image, reduce the amount of calculation, and improve the stitching speed. The geometric shape of the conveyor belt and the influence of factors such as the parameters, installation position, and posture of the multi-view camera make it difficult to determine the exact overlapping region.

3. Materials and Methods

The adaptive multi-view image mosaic (AMIM) method proposed in this paper realizes multi-view conveyor belt surface fault online detection image stitching by the following sub-methods: grayscale-based overlapping region registration, the division of the IOI (image of interest), feature-based adaptive partition and block registration, improved weighted smooth fusion. The flow chart of the AMIM method is shown in Figure 3.

3.1. Grayscale-Based Overlapping Region Registration

3.1.1. Overlapping Region Estimation

The overlapping region estimation model of 2 adjacent images is established according to the geometry of the conveyor belt and the parameters, installation position, and posture of multi-view cameras. For a conveyor belt, the horizontal distance between its two ends is set as B, the speed is V. Suppose it runs smoothly and at a uniform speed. The sensor pixel size of the line-scan camera is w, the number of pixels is n, the line sampling frequency is f_h, and the lens focal length is f. For the conveyor belt, the world coordinate system O_w-X_wY_wZ_w is established. The X_w axis passes through the bottom end of the arc cross-section of the conveyor belt and is parallel to the ground, and the direction is from left to right. The Z_w axis passes through the leftmost endpoint of the arc cross-section of the conveyor belt and is perpendicular to the ground, and the direction is from bottom to top. The intersection point O_w of the X_w axis and the Z_w axis is the origin, and the origin coordinate is (0, 0, 0). The direction of V is the same as the Y_w axis. Then, in the world coordinate system O_w-X_wY_wZ_w, the coordinate of the camera installation position O_c is (X₀, Y₀, Z₀), the unit is millimeter; the camera installation posture is (α, β, γ), the unit is degrees (°). In the line-scan camera imaging process, the space points are projected onto the image sensing pixels to form an image, and the image pixel coordinate system is O_f-uv. The imaging schematic diagram of the line-scan camera is shown in Figure 4.

The static imaging of a line-scan camera can only obtain the one-dimensional image of the conveyor belt. Because of the relative motion between the conveyor belt and line-scan camera, the line-scan camera can obtain a two-dimensional image of the conveyor belt by dynamic imaging. Suppose that the coordinate of any point P on the conveyor belt in the world coordinate system O_w-X_wY_wZ_w is (X_P, Y_P, Z_P), and the coordinate of the point P imaged in the image pixel coordinate system O_f-uv is (u, v). According to the fundamental relationship of the perspective projection geometric imaging model of the imaging characteristics of the area-scan camera and the line-scan camera [23,24], one can obtain the dynamic imaging model of the line-scan camera, is seen in Equation (1):

$$\left\{\begin{array}{l}u=\frac{a_2(X_P-X_0)+b_2(Y_P-Y_0)+c_2(Z_P-Z_0)}{b_2{V}^{\prime }}\\ v=v_0+\mathsf{\Delta }v-f^{\prime }·\frac{c_3(X_P-X_0)-a_3(Z_P-Z_0)}{a_1(Z_P-Z_0)-c_1(X_P-X_0)}\\ s.t.\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{Z}_P=g(X_P)\end{array}\right.$$

(1)

where v₀ is the pixel coordinate value of the main imaging point O of sensor pixel center of the line-scan camera, which can be generally set as v₀ = n/2. f′ = f/w. V′ represents the relative motion between the conveyor belt and the line-scan camera in each image acquisition cycle, satisfying V′ = V/f_h. Compared with V′, the relative motion of the conveyor belt caused by vibration and offset can be ignored. The results show that △v is the imaging distortion, and △v increases gradually from v₀ to both sides. The Equation is as follows:

$$\mathsf{\Delta }v=k_1{(v-v_0)}^5+k_2{(v-v_0)}^3+k_3{(v-v_0)}^2$$

(2)

where k₁, k₂, and k₃ are distortion parameters, which can be obtained by camera calibration.

In Equation (1), a₁, a₂, a₃, b₁, b₂, b₃, c₁, c₂, c₃ are the elements of the rotation matrix of rigid body transformation between the world coordinate system O_w-X_wY_wZ_w and the line-scan camera coordinate system O_c-X_cY_cZ_c. The rotation matrix is an orthogonal matrix related to the camera installation posture (α, β, γ), as shown in Equation (3).

$$\left[\begin{array}{ccc}{a}_1& b_1& c_1\\ a_2& b_2& c_2\\ a_3& b_3& c_3\end{array}\right]=\left[\begin{array}{ccc}\cos \beta \cos \gamma & \cos \beta \sin \gamma & -\sin \beta \\ \sin \alpha \sin \beta \cos \gamma -\cos \alpha \sin \gamma & \sin \alpha \sin \beta \sin \gamma +\cos \alpha \cos \gamma & \sin \alpha \cos \beta \\ \cos \alpha \sin \beta \cos \gamma +\sin \alpha \sin \gamma & \cos \alpha \sin \beta \sin \gamma -\sin \alpha \cos \gamma & \cos \alpha \cos \beta \end{array}\right]$$

(3)

In Equation (1), the constraint condition Z_P = g(X_P) is the equation of conveyor belt arc cross-section in the world coordinate system by polynomial curve fitting.

Let the range of the conveyor belt content corresponding to the overlapping region of 2 adjacent images in the X_w axis direction of the world coordinate system be [X_P1, X_P2]. The overlapping region of 2 adjacent images only considers the v axis coordinates of image pixels. Let the pixel range of the overlapping region of the left image be [v₁, n]. Let the pixel range of the overlapping region of the right image be [1, v₂]. According to Equation (1), the overlapping region estimation model can be further established, which satisfies the relationship of Equation (4):

$$\{\begin{array}{l}{v}_1=v_0+\mathsf{\Delta }{v}_{v=v_1}-f^{\prime }·\frac{c_3(X_{P1}-X_{01})-a_3(Z_{P1}-Z_{01})}{a_1(Z_{P1}-Z_{01})-c_1(X_{P1}-X_{01})}\\ n=v_0+\mathsf{\Delta }{v}_{v=n}-f^{\prime }·\frac{c_3(X_{P2}-X_{01})-a_3(Z_{P2}-Z_{01})}{a_1(Z_{P2}-Z_{01})-c_1(X_{P2}-X_{01})}\\ 1=v_0+\mathsf{\Delta }{v}_{v=1}-f^{\prime }·\frac{c_3{}^{\prime }(X_{P1}-X_{02})-a_3{}^{\prime }(Z_{P1}-Z_{02})}{a_1{}^{\prime }(Z_{P1}-Z_{02})-c_1{}^{\prime }(X_{P1}-X_{02})}\\ v_2=v_0+\mathsf{\Delta }{v}_{v=v_2}-f^{\prime }·\frac{c_3{}^{\prime }(X_{P2}-X_{02})-a_3{}^{\prime }(Z_{P2}-Z_{02})}{a_1{}^{\prime }(Z_{P2}-Z_{02})-c_1{}^{\prime }(X_{P2}-X_{02})}\\ s.t.\begin{array}{cc}& Z_{P1}=g(X_{P1})\begin{array}{cc}& Z_{P2}=g(X_{P2})\end{array}\end{array}\end{array}$$

(4)

where, X₀₁, Y₀₁, Z₀₁ and X₀₂, Y₀₂, Z₀₂ are the installation position coordinates of the cameras corresponding to the 2 adjacent images, and a₁, a₃, c₁, c₃ and a₁′, a₃′, c₁′, c₃′ are the elements of the rotation matrix of the cameras corresponding to the 2 adjacent images. When v is v₁, n, 1, and v_2, the values of Δv_{v = v1}, Δv_{v = n}, Δv_{v = 1}, and Δv_{v = v2} are the values of Δv, respectively. The overlapping region of 2 adjacent images can be estimated by solving the values of unknown variables v₁, v₂, X_w1, and X_w2.

3.1.2. Overlapping Region Registration

Because the range of overlapping region obtained by the overlapping region estimation model is a rough estimation, it is necessary to carry out overlapping region registration. We used the grayscale-based method with gray correlation analysis to measure the correlation between the reference pixel and the comparison pixel in the local area of the image. We can find the position with a higher gray correlation degree in the estimated overlapping region and realize the overlapping region registration.

There are 2 adjacent m × n size images—the gray value of the left image pixel is f₁(u,v), and the gray value of the right image pixel is f₂(u,v), where u is the row coordinate, u∈[1,m], v is the column coordinate, v∈[1,n]. The overlapping region range of the left and right images estimated is [v₁,n] and [1,v₂]. Registration of overlapping regions can be divided into the following steps:

Step 1: Select the region image whose column coordinates are v∈[v₁−L,n] in the left image, and select the region image whose column coordinates are v∈[1,v₂ + L] in the right image, where L is the deviation value.

Step 2: Calculate the mean of pixels of each column for the 2 regional images obtained. The Equation is as follows:

$$\left\{\begin{array}{c}im{g}_1(v)={\displaystyle \sum }_{u=1}^m{f}_1\left(u,v\right)/m\\ im{g}_2(v)={\displaystyle \sum }_{u=1}^m{f}_2\left(u,v\right)/m\end{array}\right.$$

(5)

where img₁(v) and img₂(v) are the mean of pixels in column v.

Step 3: Affected by uneven illumination, weak imaging dark regions can appear at the edge of the image. To overcome its influence on stitching, for the left image, starting from v = n, one must look for the first one that satisfies img₁(v) ≥ T_d column by column in a descending manner and denote it as d₁, d₁ ≥ v₁. For the right image, starting from v = 1, one must look for the first column that satisfies img₂[v] ≥ T_d column by column in an increasing manner and denote it d₂, d₂ ≤ v₂; T_d is the threshold. For the columns of the left image v∈[d₁ + 1,n] and the right image v∈[1,d₂−1], the gray correlation degree calculation is not performed, and the corresponding gray correlation degree ε_l or ε_r is assigned the value of zero.

Step 4: The reference sequence X_l0, X_r0 and comparison sequence X_l1, X_r1 were determined.

$$\{\begin{array}{ll}{X}_l{}_0& =\{X_l{}_0\left(1\right),X_l{}_0\left(2\right),X_l{}_0\left(3\right),\dots ,X_l{}_0\left(k\right)\}\\ & =\{im{g}_2\left(d_2\right),im{g}_2\left(d_2+1\right),im{g}_2\left(d_2+2\right),\dots ,im{g}_2\left(d_2+k-1\right)\}\\ X_r{}_0& =\{X_r{}_0\left(1\right),X_r{}_0\left(2\right),X_r{}_0\left(3\right),\dots ,X_r{}_0\left(k\right)\}\\ & =\{im{g}_1\left(d_1-k+1\right),\dots ,im{g}_1\left(d_1-2\right),im{g}_1\left(d_1-1\right),im{g}_1\left(d_1\right)\}\end{array}$$

(6)

where k is an odd number.

For 2 images, take the current pixel img₁(p) and img₂(q) of the column mean as the center, and take k consecutive data to form a group of comparison sequences, where p∈[v₁−L, v₁ + L], q∈[v₂−L, v₂ + L]. The k consecutive data centered on img₁(p) and img₂(q) are divided into a group of comparison sequences as follows:

$$\{\begin{array}{ll}{X}_l{}_1& =\{X_l{}_1\left(1\right),X_l{}_1\left(2\right),X_l{}_1\left(3\right),\dots ,X_l{}_1\left(k\right)\}\\ & =\{im{g}_1\left(p-\left(k-1\right)/2\right),\dots ,im{g}_1\left(p-2\right),im{g}_1\left(p-1\right),im{g}_1\left(p\right),\\ & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}im{g}_1\left(p+1\right),im{g}_1\left(p+2\right),\dots ,im{g}_1\left(p+\left(k-1\right)/2\right)\}\\ X_r{}_1& =\{X_r{}_1\left(1\right),X_r{}_1\left(2\right),X_r{}_1\left(3\right),\dots ,X_r{}_1\left(k\right)\}\\ & =\{im{g}_2\left(q-\left(k-1\right)/2\right),\dots ,im{g}_2\left(q-2\right),im{g}_2\left(q-1\right),im{g}_2\left(q\right),\\ & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}im{g}_2\left(q+1\right),im{g}_2\left(q+2\right),\dots ,im{g}_2\left(q+\left(k-1\right)/2\right)\}\end{array}$$

(7)

Step 5: The average processing is used to dimensionless each group of sequence data.

$$\left\{\begin{array}{l}{X}_l{{}_0}^{\prime }=X_l{}_0/mean\left(X_l{}_0\right)\\ X_r{{}_0}^{\prime }=X_r{}_0/mean\left(X_r{}_0\right)\\ X_l{{}_1}^{\prime }=X_l{}_1/mean\left(X_l{}_1\right)\\ X_r{{}_1}^{\prime }=X_r{}_1/mean\left(X_r{}_1\right)\end{array}\right.$$

(8)

where “mean” means to get the average value of the sequence data.

Step 6: Calculate the difference sequence ∆_l, ∆_r, the maximum difference w_l1, w_r1, the minimum difference w_l2, w_r2.

$$\left\{\begin{array}{c}{\mathsf{\Delta }}_l=|X_l{{}_0}^{\prime }-X_l{{}_1}^{\prime }|\\ {\mathsf{\Delta }}_r=|X_r{{}_0}^{\prime }-X_r{{}_1}^{\prime }|\end{array}\right.$$

(9)

$$\left\{\begin{array}{c}{w}_l{}_1=max\left({\mathsf{\Delta }}_l\right)\\ w_r{}_1=max\left({\mathsf{\Delta }}_r\right)\end{array}\right.$$

(10)

$$\left\{\begin{array}{c}{w}_l{}_2=min\left({\mathsf{\Delta }}_l\right)\\ w_r{}_2=min\left({\mathsf{\Delta }}_r\right)\end{array}\right.$$

(11)

where “max” and “min” represent the maximum and minimum values in the sequence data, respectively.

Step 7: Calculation of gray correlation coefficient ${\delta }_l(i)$ and ${\delta }_r(i)$.

$$\left\{\begin{array}{c}{\delta }_l\left(i\right)=\frac{w_{l2}+\rho w_{l1}}{{\mathsf{\Delta }}_l(i)+\rho w_{l1}}\\ {\delta }_r\left(i\right)=\frac{w_{r2}+\rho w_{r1}}{{\mathsf{\Delta }}_r(i)+\rho w_{r1}}\end{array}\right.$$

(12)

where i = 1, 2, …, k, ρ = 0.5.

Step 8: Calculation of gray correlation degree ${\epsilon }_l$ and ${\epsilon }_r$.

$$\left\{\begin{array}{c}{\epsilon }_l=\frac{1}{k}{\displaystyle \sum _{i=1}^k{\delta }_l(i)}\\ {\epsilon }_r=\frac{1}{k}{\displaystyle \sum _{i=1}^k{\delta }_r(i)}\end{array}\right.$$

(13)

Step 9:p = p + 1. If p satisfies the range of p∈[v₁−L,v₁ + L], continue to calculate the gray correlation degree ε_l with img₁(p) as the current pixel according to step (4). q = q + 1. If q satisfies the range of q∈[v₂−L,v₂ + L], continue to calculate the gray correlation degree ε_r with img₂(q) as the current pixel according to step (4).

Step 10: Calculate the p value corresponding to the maximum value of ε_l and record it as p₀. Calculate the q value corresponding to the maximum value of ε_r and record it as q₀. Then, the range of overlapping region of the left image is determined as u∈[1,m], v∈[p₀,d₁], and the range of overlapping region of the right image is determined as u∈[1,m], v∈[d₂,q₀], and the overlapping region registration is completed.

As shown in Figure 5, regions R₁₂ and R₂₂ are the overlapping regions after registration, regions R₁₁ and R₂₁ are the non-overlapping regions, and regions R₁₃ and R₂₃ are the dark regions.

Select 2 adjacent images in the multi-view conveyor belt images. The image size is 2048 × 2048, as shown in Figure 6a,b. When L = 800, k = 353, T_d = 44 (T_d as half of the mean of image pixels), carry out the method in this paper, and obtain the curves of gray correlation degree ε_l and ε_r, as shown in Figure 6c,d. It can be seen that the value range of the correlation degree is [0,1]; the maximum value of the correlation value is used to register the overlapping region. The left and right image registration overlapping regions are [1269,1886] and [245,932], respectively. The registration of overlapping region is shown in Figure 6e,f.

3.2. The Division of The IOI

It is difficult to detect, select, and match feature points in image mosaic of multi-view conveyor belt surface fault online detection, which makes it impossible to use the feature-based method for more accurate image registration. Generally, the useful feature points of the conveyor belt are mainly distributed in the regions such as stripes caused by moving friction, conveyor belt joints, defects, etc. This paper uses an IOI detection algorithm to divide the IOI and the non-IOI image to improve feature-based registration effectiveness.

For the i-th image of m × n size images collected from N viewpoint, let the gray value of image pixel be f_i(u,v), and the overlapping region range be u∈[1, m], v∈[v_a, v_b]. The detection algorithm of the IOI is as follows: if the value of any i∈[1, N] satisfies E_i ≥ T_E, where T_E is the threshold, then the N images to be stitched are considered to be the IOIs with apparent features; otherwise, they are the images with no apparent features. The image evaluation parameter E_i is calculated as follows:

$$E_i=k_1\sqrt{\frac{1}{m}{{\displaystyle \sum _{u=1}^m\left(H{M}_i(u)-F_i\right)}}^2}+\frac{k_2}{v_b-v_b}{\displaystyle \sum }_{v=v_a}^{v_b}V{D}_i\left(v\right)$$

(14)

where k₁ and k₂ are proportional coefficients, which represent the contribution degree of factors and satisfy k₁ + k₂ = 1. Generally, k₁ and k₂ can be set to 0.5. HM_i(u) is the mean vector of row pixels, and F_i is the mean of image pixels, calculated as follows:

$$H{M}_i(u)=\frac{1}{v_b-v_a}{\displaystyle \sum }_{v=v_a}^{v_b}{f}_i\left(u,v\right)\hspace{0.17em}\hspace{0.17em}u=1,2,\dots ,m$$

(15)

$$F_i=\frac{1}{m}{\displaystyle \sum _{u=1}^{m}H{M}_i(u)}$$

(16)

VD_i(v) is calculated as follows:

$$V{D}_i(v)=\sqrt{\frac{1}{m}{{\displaystyle \sum _{u=1}^m\left(f_i\left(u,v\right)-V{M}_i\left(v\right)\right)}}^2}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}v=v_a,v_a+1,\dots ,v_b$$

(17)

where VM_i (v) is the mean vector of column pixels, which is calculated as follows:

$$V{M}_i(v)=\frac{1}{m}{\displaystyle \sum }_{u=1}^m{f}_i\left(u,v\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}v=1,2,\dots ,n$$

(18)

3.3. Feature-Based Adaptive Partition and Block Registration

For the IOI, the feature-based adaptive partition and block registration method is further used to improve stitching accuracy. The IOI is segmented adaptively to improve and enhance the efficiency and real-time stitching. The specific operation steps are as follows:

Step 1: A binarization curve is generated for the image partition of interest.

Select the HM_i(u) curve and the VD_i(v) curve described in Equation (15) and Equation (17). Construct a one-dimensional filter window with the size of L_f, and L_f is an odd number. Slide the filter window on the HM_i(u) curve and VD_i (v) curve, respectively. The mean of L_f pixels is taken as the value of the center pixel of the window. The HM′_i(u) curve and VD′_i(v) curve are obtained after the mean filtering. Then, perform binarization processing according to the following Equation to generate a binarization curve.

$$B_{H{M}_i}\left(u\right)=\left\{\begin{array}{c}1,\begin{array}{c}\end{array}H{M^{\prime }}_i\left(u\right)<F_i\\ 0,\begin{array}{c}\end{array}H{M^{\prime }}_i\left(u\right)\ge F_i\end{array}\right.$$

(19)

$$B_{V{D}_i}\left(v\right)=\left\{\begin{array}{c}1,\begin{array}{c}\end{array}V{D^{\prime }}_i\left(v\right)>V_i\\ 0,\begin{array}{c}\end{array}V{D^{\prime }}_i\left(v\right)\le V_i\end{array}\right.$$

(20)

where F_i is the mean of HM_i(u), as shown in Equation (16), and V_i is the mean of VD_i(v), which satisfies:

$$V_i=\frac{1}{v_b-v_b}{\displaystyle \sum }_{v=v_a}^{v_b}V{D}_i\left(v\right)$$

(21)

Step 2: Deal with the region of interest (ROI) of the binarization curve.

For B_VDi(v) and B_HMi(u) binarization curves, the region with a continuous value of “1” is found as the ROI of the curve, and the start position, end position, and length of the region are marked. The merging threshold T_C of adjacent ROIs and the removal threshold T_B of isolated small ROIs are set in order to solve the influence of scattered and small regions with continuous “1” on the partition effect of ROIs. Suppose the interval between 2 adjacent ROIs of B_VDi(v) and B_HMi(u) binarization curve satisfies the condition of less than T_C. In that case, the 2 ROIs are merged and connected. If the length of the region is less than T_B, the ROI is removed. After the processing, the binarization curves are B_VDi′(v) and B_HMi′(u).

Step 3: The ROI of IOI is generated and selected.

A template image with the same size as the overlapping region of the IOI is constructed, and its pixel value is g_i (u,v).

$$g_i\left(u,v\right)=B_{H{M}_i}{}^{\prime }\left(u\right)\times B_{V{D}_i}{}^{\prime }\left(v\right)=\left\{\begin{array}{l}1,\left(u,v\right)\in \mathrm{ROI}\\ 0,\left(u,v\right)\in background region\end{array}\right.$$

(22)

where u∈[1,m], v∈[v_a, v_b]; through Equation (23), the overlapping region of the IOI to be stitched is divided into 1 background region and K_i rectangular ROIs with effective feature points distribution.

$$f_i^{\prime }\left(u,v\right)=f_i\left(u,v\right)\times g_i\left(u,v\right)$$

(23)

If K_i ≥ 3, 2 ROIs with larger size are selected; otherwise, all ROIs are selected.

Step 4: The image block is generated and selected.

The ROI of the image is divided into N_i × N_i blocks, and the number of blocks is B_i. If B_i ≥ 3, 2 image blocks with larger standard deviation are selected; otherwise, all image blocks are selected.

The flow chart of adaptive partition and block operation for the IOI is shown in Figure 7.

We verify the effect of partition operation for the IOI. An IOI with a size of 2048 × 2048 (assuming that the whole image is an overlapping region), as shown in Figure 8a, is partitioned. The curves of HM₁(u) and its mean, VD₁(v) and its mean are shown in Figure 8c,d. Construct a one-dimensional window template of size 61 (L_f = 61) for mean filtering. The curves of HM′₁(u) and its mean are shown in Figure 8e. The curves of VD′₁(v) and its mean are shown in Figure 8f.The binarization curves B_HM1(u) and B_VD1(v) are obtained as shown in Figure 8g,h, respectively. Suppose T_C = 128 and T_B = 128, the binarization curves B_HM1′(u) and B_VD1′(v) obtained after merging and removing operations are shown in Figure 8i,j, respectively. Finally, the ROI is partitioned, and the result is shown in Figure 8b. It can be seen from 8b that most of the defect information of the conveyor belt is contained in the ROI, which reduces the scope of image registration.

We further verify the effect of adaptive partition and block operation for the IOI. The original images are shown in Figure 9a. L_f = 61, T_C = 128, T_B = 128 are used to process the original image and extract the ROI, as shown in Figure 9b. The ROI image is selected and divided into 128 × 128 blocks. The selected ROI images are shown in Figure 9(c1,c4,d1,e1). The result of 128 × 128 blocks for Figure 9(d1) is shown in Figure 10. The position of the image block is represented by (r_i, c_i); r_i and c_i are the numbers of rows and columns of the image block arranged according to the matrix, respectively, satisfying r_i ≤ R_i, c_i ≤ C_i, B_i = R_i × C_i. The number of blocks shown in Figure 10 is 5 × 4. Then, select the image block with a larger standard deviation, as shown in Figure 9c–e, where Figure 9(d2,d3) are the image blocks with the largest standard deviation (Block 1) and the second-largest standard deviation (Block 2), and the positions in Figure 10 are (5,3) and (4,3), respectively. It can be seen that the separated image block contains a high amount of feature information, such as conveyor belt joint or surface fault, which is conducive to the extraction of useful features. The results of the ROI adaptive partition and block method are shown in

Table 1. Results of the ROI adaptive partition and blocking method.

Image	ROI				Block 1		Block 2
	Number	Size	Blocks Number	Standard Deviation	Position	Standard Deviation	Position	Standard Deviation
Figure 9(a1)	2	896 × 1152	9 × 7	38.289	(7,7)	61.055	(7,6)	51.947
		256 × 1152	9 × 2	28.983	(1,5)	35.055	(1,4)	32.219
Figure 9(a2)	1	640 × 512	5 × 4	20.741	(5,3)	31.456	(4,3)	30.289
Figure 9(a3)	1	512 × 512	4 × 4	32.263	(2,2)	39.426	(1,1)	32.247

.

To speed up the calculation, we use the SURF algorithm for feature extraction and description of the selected image block. The SURF algorithm includes the following operations: building the Hessian matrix for feature point extraction, building scale space, feature point location, feature point main direction assignment, feature point descriptor generation, and feature point matching. The random sample consensus (RANSAC) algorithm is used to eliminate the outer points and retain the inner points for image registration. The transformation matrix of the image is constructed through the matching point pairs. According to the transformation matrix obtained by feature registration, the corresponding image is transformed. Feature point matching uses the distance function (Euclidean distance) to retrieve the similarity between high-dimensional vectors. In this paper, the K-D tree algorithm is used. Firstly, the K-D tree is established for each image with the feature descriptor as the tree node information. Then, the K data closest to the corresponding feature points in the other image is searched using the feature points of 1 of the 2 images to be stitched to complete the initial matching of feature points. The flow chart of the feature-based adaptive partition block registration method is shown in Figure 11.

3.4. Improved Weighted Smooth Fusion

The improved weighted smoothing algorithm is used for image fusion to get the stitched image. The expression of the improved weighted smoothing algorithm is as follows:

$$f(u,v)=\left\{\begin{array}{l}{f}_1(u,v),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(u,v)\in R_{11}\\ {\omega }_1(u,v)f_1(u,v)+{\omega }_2(u,v)f_2(u,v),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(u,v)\in R_{12}\cup R_{22}\\ f_2(u,v),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(u,v)\in R_{21}\end{array}\right.$$

(24)

where R₁₁ and R₂₂ are the non-overlapping regions of 2 adjacent images; R₁₂ is the overlapping region; and ω₁ and ω₂ are the weights of corresponding pixels in the overlapping region, respectively, satisfying 0 < ω₁, ω₂ < 1 and ω₁ + ω₂ = 1. To ensure the continuity of overlapping region transition and effectively remove stitching marks, one calculates the weight according to the following equation:

$${\omega }_1(u,v)=\frac{v-p_0}{d_1-p_0}\begin{array}{cc}& (u,v)\in R_{12}\end{array}$$

(25)

$${\omega }_2(u,v)=\frac{d_1-v}{d_1-p_0}\begin{array}{cc}& (u,v)\in R_{22}\end{array}$$

(26)

4. Experiments and Results

The experiment is carried out on the platform, as shown in Figure 12, in order to verify the effectiveness of the method proposed in this paper. Two 2048 × 2048 IOIs with the same surface damage content were selected, as shown in Figure 13(a1,b1). Simulation by using MATLAB software programming in Intel i5 CPU computer showed that the method was effective. The ROI images are shown in Figure 13(a2,b2). The block images are shown in Figure 13(a3,a4,b3,b4). For the original image, the ROI image, and the block image, the SURF algorithm with the Hessian matrix determinant response value of 1000 was used for feature extraction and description. Draw the circle on the image with the feature point as the center. The radius of the circle indicates the size of the feature point, and the straight line indicates the direction of the feature point. The feature points are described as shown in Figure 13(a5–a8,b5–b8).

The comparison of image feature extraction is shown in

Table 2. Comparison of image feature extraction when the threshold value of Hessian matrix determinant response value is 1000.

Parameters	Figure 13(a5)	Figure 13(a6)	Figure 13(a7)	Figure 13(a8)	Figure 13(b5)	Figure 13(b6)	Figure 13(b7)	Figure 13(b8)
Quantity (piece)	232	152	16	15	175	122	7	22
Time (ms)	578	234	8	8	546	203	8	8

. It can be seen that the number of feature points extracted in the ROI of Figure 13(a1,b1) were 65.52% and 69.71% of the whole image, respectively. The number of feature points extracted from the ROI accounts for a high proportion of the number of feature points in the whole image, reflecting that the ROI is the feature point concentration region of the whole image. However, it took 40.48% and 37.18% of the total image time, respectively, which was less time-consuming. Although the number of feature points extracted from two image blocks was less than the ROI, the feature points extracted had larger feature values and significant features, which met easy registration requirements. Moreover, the two image blocks took less time. The total time was only 16 ms, which was 2.77% and 2.93% of the whole image time, respectively. It is thus suitable for fast registration.

For the two images of Figure 13(a1,b1), the different threshold values of Hessian matrix determinant response value, such as 500, 1000, 1500, and 2000, were used to extract the feature points of each image. The comparison in

Table 3. Comparison of image feature extraction by different threshold values of Hessian matrix determinant response value.

Threshold	Parameters	Figure 13(a5)	Figure 13(a6)	Figure 13(a7)	Figure 13(a8)	Figure 13(b5)	Figure 13(b6)	Figure 13(b7)	Figure 13(b8)
500	Quantity (piece)	694	373	27	35	647	334	29	39
	Time (ms)	811	265	15	15	733	281	15	15
1000	Quantity (piece)	232	152	16	15	175	122	7	22
	Time (ms)	578	234	8	8	546	203	8	8
1500	Quantity (piece)	114	74	8	11	88	61	5	14
	Time (ms)	483	125	8	8	437	125	8	8
2000	Quantity (piece)	64	44	3	10	48	28	1	9
	Time (ms)	468	125	8	8	406	125	8	8

shows that with the increase of the threshold value, the number of feature points extracted and the time consumed was reduced. However, the threshold value is important for the block image, because of the small image size. If the threshold value is too large, too few feature points are extracted in the block, which is not conducive to feature point registration. For example, when the threshold value was 2000, the number of feature points shown in Figure 13(b7) was only one. The threshold value of the Hessian matrix determinant response value can be selected as 1000 by comparison.

For the feature points extracted from the original image, ROI image, and block image, we used the RANSAC algorithm to complete the matching of feature points. The matching results are shown in Figure 14. The comparison of registration is shown in

Table 4. Comparison of the registration.

Image	Number of Feature Points (Piece)	Registration Number (Pairs)	Registration Time (ms)
Original Image	407	14	12
ROI Image	274	10	9
Block Image	60	4	5

. It can be seen that a sufficient number of registration feature point pairs can be obtained from block images. The time of registration processing on an Intel i5 CPU computer was 5 ms, which was 41.67% of the total image time.

The proposed method was applied in the mine for conveyor belt surface fault online detection, as shown in Figure 15. To further verify the effectiveness of the AMIM method, we selected two groups of images collected by two line-scan cameras with different viewpoints. The image size was 2048 × 2048, as shown in Figure 16a,b. When L = 800, k = 353, T_d = 44 (T_d as half of the mean of image pixels), the overlapping regions of left and right images in Figure 16a were [1519,1788], [236,397], and the overlapping regions of left and right images in Figure 16(b) were [1526,1784], [257,546]. The parameters were set as T_E= 20, L_f = 61, T_C = 128, T_B = 128. The evaluation parameters E of the IOI calculated from the image registration overlapping region in Figure 16a were 11.22 and 8.14, respectively. The evaluation parameters E were calculated by Equation (14). According to the detection algorithm of the IOI, the images in Figure 16a were identified as the non-IOI. For the overlapping region estimated from the image in Figure 16b, the evaluation parameters E of the IOI calculated were 29.06 and 36.97, respectively. The images in Figure 16b were identified as the IOI. Only for the Figure 16b, using 128 × 128 block size, was the feature-based adaptive partition block registration method carried out. Finally, the improved weighted smoothing algorithm was used for image fusion, and the stitched image was obtained, as shown in Figure 16c,d. The subjective evaluation of the stitched image shows that the stitching effect of the AMIM method is good.

The AMIM method is evaluated from the registration rate and running time to verify the effectiveness of our method more objectively.

The registration rate measures the accuracy of the matched feature points.

$$Z=1-\frac{c_1}{c_2}$$

(27)

where Z is the registration rate, c₁ is the number of mismatched feature pairs, and c₂ is the number of feature pairs in the RANSAC algorithm.

The comparison between the AMIM method and other methods is shown in

Table 5. Comparison of our method with other methods.

Method	Image	Feature Points Number	Registration Number	Mismatches Number	Registration Rate	Time (ms)
SIFT	Figure 16a	1225	0	13	0%	5513
	Figure 16b	984	96	5	95.04%	4892
SURF	Figure 16a	232	0	11	0%	1270
	Figure 16b	175	94	5	94.95%	1231
AMIM	Figure 16a	31	-	-	-	278
	Figure 16b	42	42	1	97.67%	328

. For the images in Figure 16a, SIFT and SURF method cannot get the correct registration feature points and cannot realize the accurate image stitching operation on the basis of features. However, the AMIM method does not use the feature-based registration method after identifying the image as the non-IOI. It uses the overlapping region of registration to stitch the image directly. For the image in Figure 16b, although the number of feature points extracted by our method was small, the registration rate was high, reaching 97.67%. The average time of stitching two adjacent images was less than 500 ms. Thus, our method ensures image stitching accuracy, reduces the registration time, and improves image stitching efficiency. Therefore, our method has advantages in the process of multi-view conveyor belt image stitching.

5. Conclusions

Correct operation of the conveyor belt is very important for production and staff safety. The multi-view conveyor belt surface fault online detection system based on machine vision can provide a sufficient guarantee. This paper proposed an adaptive multi-view image mosaic method based on the combination of grayscale and feature to improve the accuracy and real-time of image mosaic and realize the multi-view conveyor belt surface fault online detection. Compared with the traditional mosaic methods, the proposed method has the following improvements and advantages:

The overlapping region of two adjacent images is preliminarily estimated by establishing the overlapping region estimation model, and then the grayscale-based method is used to register the overlapping region.

For the collected image, the IOI detection algorithm is used to divide it into the IOI and the non-IOI, and different registration methods are used.

For the IOI, its overlapping region may contain the fault information of the conveyor belt. The feature-based partition and block registration method is used to register the images more accurately, the overlapping region is adaptively segmented, the SURF algorithm is used to extract the feature points, and the random sample consensus RANSAC algorithm is used to achieve accurate registration. It is easy to realize the accurate measurement of the geometric shape of the fault, and it is conducive to the accurate and reliable detection and evaluation of the health status of the conveyor belt.

For the non-IOI, its overlapping region almost does not contain the fault information of the conveyor belt, and the features are not obvious. It is easy to have the wrong registration by using the SIFT algorithm, SURF algorithm, and other feature-based registration methods, and the registration rate is even 0%. Thus, they cannot obtain the right result, and the process is time-consuming, which is not conducive to real-time online detection. Accurate registration is not performed on the adjacent non-IOIs, which improves the accuracy and real-time of stitching.

The improved weighted smooth fusion algorithm is used to fuse the images to realize image stitching.

The experimental results show that when the SURF algorithm is used to extract and describe the block image features, the threshold value of the Hessian matrix determinant response value is 1000, which is better compared with 500, 1500, and 2000. It has the advantage of extracting enough feature points and consuming less time. Enough feature points can be extracted from the block image, and the features are significant, which is easy to register accurately. Compared with the whole image registration, it takes less time. The results of the experimental analysis showed the usefulness of the proposed method. The stitching effect is better compared with the SIFT and SURF algorithms. Simultaneously, it can reduce the amount of stitching calculation and the time of the stitching process, and the registration rate is high. The registration rate can reach 97.67%, and the average time of stitching two adjacent images is less than 500 ms. It is suitable for conveyor belt surface fault online detection. In addition, this method can also provide a reference for other application types of image mosaic.

The proposed method was applied in the mine for conveyor belt surface fault online detection and can be extended to port, electric power, metallurgy, chemical industry, and other fields for conveyor belt surface fault online detection. In the future, we will study the methods of accurate and reliable detection and evaluation of conveyor belt health. In order to improve the detection performance, we can further use the hybrid imaging method combining thermal imaging [25] and visible light imaging to explore ways to realize the fusion of heterogeneous images and multi-view image mosaic at the same time. It is also valuable to explore the application of deep learning in conveyor belt fault detection [26].

Multi-view is widely used in machine vision-based detection and has played an important role. The proposed method in this paper can be used in industry, agriculture, medical, and other fields of fault detection using the multi-view method [27,28,29,30]. Compared with other improved image mosaic methods, the proposed method is more suitable for solving the problem of conveyor belt surface fault online detection. For example, the IOI detection algorithm is proposed on the basis of the characteristics of the conveyor belt image, and it may not be adaptable to other detection object images for different applications. However, for other applications, the proposed method in this paper has a reference value for image mosaic problems that need to take into account both accuracy and real-time.