Adaptive Multi-View Image Mosaic Method for Conveyor Belt Surface Fault Online Detection

: In order to improve the accuracy and real-time of image mosaic, realize the multi-view conveyor belt surface fault online detection, and solve the problem of longitudinal tear of conveyor belt, we in this paper propose an adaptive multi-view image mosaic (AMIM) method based on the combination of grayscale and feature. Firstly, the overlapping region of two adjacent images is preliminarily estimated by establishing the overlapping region estimation model, and then the grayscale-based method is used to register the overlapping region. Secondly, the image of interest (IOI) detection algorithm is used to divide the IOI and the non-IOI. Thirdly, only for the IOI, the feature-based partition and block registration method is used to register the images more accurately, the overlapping region is adaptively segmented, the speeded up robust features (SURF) algorithm is used to extract the feature points, and the random sample consensus (RANSAC) algorithm is used to achieve accurate registration. Finally, the improved weighted smoothing algorithm is used to fuse the two adjacent images. The experimental results showed that the registration rate reached 97.67%, and the average time of stitching was less than 500 ms. This method is accurate and fast, and is suitable for conveyor belt surface fault online detection.


Introduction
Belt conveyor is a kind of continuous transportation equipment in modern production that has been widely used in the coal, mining, port, electric power, metallurgy, and chemical industries, as well as in other fields [1]. The conveyor belt is the key component of the belt conveyor for traction and load-bearing, which can easily produce surface scratch, longitudinal tear, or fracture [2]. If these faults are not detected and handled in time, they may cause equipment damage, material loss, and other substantial economic losses, and even cause personal casualties, affecting safety production [3]. At present, machine vision technology can be used for the online detection of conveyor belt surface fault [3][4][5][6][7]. However, due to the structural characteristics of the conveyor, the distance between the upper conveyor belt and the lower conveyor belt is small. The sight distance is small, and the bandwidth and field of view are large. Single-view conveyor belt surface fault detection is difficult to meet the bandwidth requirements, and there are blind areas and unsatisfactory imaging effects. Therefore, multi-view detection is needed, and multi-view conveyor belt images need to be stitched online.
At present, there are three main image mosaic methods: the method based on gray information, the method based on frequency domain correlation, and the method based on feature correlation. Among them, the method based on feature correlation has been widely focused upon because of its good robustness. The more mature feature operators are SIFT (scale-invariant feature transform), SURF (speeded up robust features), BRIEF (binary robust independent elementary features), ORB (oriented FAST and rotated BRIEF), etc. SIFT algorithm was proposed by Lowe in 1999 and improved in 2004. It not only has good robustness in image rotation and scale scaling, but also can maintain good stability against noise interference, brightness, and viewing angle changes [8,9]. However, the algorithm does not consider the distribution of feature points. It will extract too many feature points in the region with intricate details, which can easily produce false matching. The algorithm is complex in its calculation and is time-consuming. Thus, it is difficult to meet real-time performance.
For this reason, Li et al. [10] proposed an image mosaic algorithm based on the combination of region segmentation and SIFT, which improved the speed of the image. However, the amount of calculation is still large. Bay et al. [11] proposed SURF to overcome the shortcomings of the SIFT algorithm. The SURF algorithm inherits the scale and rotation invariance of the SIFT algorithm and adopts the Haar feature and the integral image concept, making the calculation more efficient. The speed is about three times faster than SIFT. However, the SURF algorithm also has the problem of wrong matching, which reduces the accuracy of the matching results and dramatically slows down the speed due to unstable feature points and wrong matching. Therefore, there are many studies on the performance improvement of the SURF algorithm. Jia et al. [12] constructed a k-dimensional (K-D) tree and improved the best bin first (BBF) algorithm instead of the linear algorithm to speed up SURF matching. Zhang et al. [13] proposed an improved SURF algorithm with a good application prospect in low light conditions. Zhang et al. [14] used Haar wavelet response to establish a descriptor for each feature point and calculated the normalized gray difference and two-step degree of the neighborhood to form a new feature descriptor. The SURF algorithm was improved to make it robust and stable to image blur, illumination difference, angle rotation, and field of view transformation. From overall technology development, image stitching technology has become more and more mature and stable. There are many excellent image stitching algorithms [15][16][17][18][19][20][21][22] that have achieved good results in some scenes. However, image stitching involves many research fields, and these stitching methods have many problems in dealing with special scenes such as conveyor belt surface fault detection. Due to the characteristics of surface material and image acquisition of the conveyor belt, it is not easy to extract and match the feature points of the conveyor belt image.
Moreover, due to a large amount of data in a multi-view image, it is not easy to accurately and quickly stitch online, affecting fault detection and recognition. The current image mosaic method is not suitable for multi-view conveyor belt surface fault online detection. According to the particularity of image mosaic, the accurate and fast image mosaic method is studied. This paper presents an adaptive multi-view image mosaic method for conveyor belt surface fault online detection based on grayscale and feature combination. This method has a high registration rate and reduces the amount of stitching calculation and the time of the stitching process. It solves the problem of the difficulty of performing accurate and fast online stitching of the collected multi-view conveyor belt images, which affects fault detection and identification.

Multi-View Conveyor Belt Surface Fault Online Detection System
The multi-view conveyor belt surface fault online detection system based on machine vision comprises line-scan charge-coupled device (CCD) network cameras, linear light sources, Ethernet switch, and computer. The schematic diagram of the system components is shown in Figure 1.
The device produces diffuse reflection light when the light is emitted by a high brightness linear light source and irradiates on the surface of the conveyor belt. The light intensity of the diffuse reflection light is related to the surface characteristics of the conveyor belt [4][5][6]. The multi-view line-scan CCD camera senses the diffuse reflection light through line scanning. Each scan takes a line of images perpendicular to running direction of the conveyor belt and transmits it to the computer through Ethernet. The computer processes The structural characteristics of the conveyor limit the installation space of the camera. Generally, the camera can only be installed close to the conveyor belt. If the image of the lower surface of the conveyor belt is to be collected for fault detection, the camera can only be installed in the narrow space between the upper and lower belts, close to the lower surface of the conveyor belt. The distance is less than 40 cm, and the object distance is small, as shown in Figure 2. The width of the conveyor belt is generally 0.8-2.4 meters. Because the cross-section is arc and the field of view is large, it is difficult to detect the width and accuracy of the conveyor belt with a single camera. It is necessary to use multiple cameras to acquire images in a multi-viewpoint manner by selecting appropriate installation position and installation postures to realize the conveyor belt without blind area coverage.
However, two adjacent images have overlapping regions that image the same conveyor belt content. Determining the two images' overlapping regions can effectively reduce the processing range of image, reduce the amount of calculation, and improve the stitching speed. The geometric shape of the conveyor belt and the influence of factors such as the parameters, installation position, and posture of the multi-view camera make it difficult to determine the exact overlapping region.

Materials and Methods
The adaptive multi-view image mosaic (AMIM) method proposed in this paper realizes multi-view conveyor belt surface fault online detection image stitching by the following sub-methods: grayscale-based overlapping region registration, the division of the The structural characteristics of the conveyor limit the installation space of the camera. Generally, the camera can only be installed close to the conveyor belt. If the image of the lower surface of the conveyor belt is to be collected for fault detection, the camera can only be installed in the narrow space between the upper and lower belts, close to the lower surface of the conveyor belt. The distance is less than 40 cm, and the object distance is small, as shown in Figure 2. The width of the conveyor belt is generally 0.8-2.4 m. Because the cross-section is arc and the field of view is large, it is difficult to detect the width and accuracy of the conveyor belt with a single camera. It is necessary to use multiple cameras to acquire images in a multi-viewpoint manner by selecting appropriate installation position and installation postures to realize the conveyor belt without blind area coverage. tion of the conveyor belt and transmits it to the computer through Ethernet. The computer processes running image of the conveyor belt, analyzes and recognizes the conveyor belt fault, and gives a fault alarm or shutdown control signal when a fault is found. The structural characteristics of the conveyor limit the installation space of the camera. Generally, the camera can only be installed close to the conveyor belt. If the image of the lower surface of the conveyor belt is to be collected for fault detection, the camera can only be installed in the narrow space between the upper and lower belts, close to the lower surface of the conveyor belt. The distance is less than 40 cm, and the object distance is small, as shown in Figure 2. The width of the conveyor belt is generally 0.8-2.4 meters. Because the cross-section is arc and the field of view is large, it is difficult to detect the width and accuracy of the conveyor belt with a single camera. It is necessary to use multiple cameras to acquire images in a multi-viewpoint manner by selecting appropriate installation position and installation postures to realize the conveyor belt without blind area coverage.
However, two adjacent images have overlapping regions that image the same conveyor belt content. Determining the two images' overlapping regions can effectively reduce the processing range of image, reduce the amount of calculation, and improve the stitching speed. The geometric shape of the conveyor belt and the influence of factors such as the parameters, installation position, and posture of the multi-view camera make it difficult to determine the exact overlapping region.

Materials and Methods
The adaptive multi-view image mosaic (AMIM) method proposed in this paper realizes multi-view conveyor belt surface fault online detection image stitching by the following sub-methods: grayscale-based overlapping region registration, the division of the However, two adjacent images have overlapping regions that image the same conveyor belt content. Determining the two images' overlapping regions can effectively reduce the processing range of image, reduce the amount of calculation, and improve the stitching speed. The geometric shape of the conveyor belt and the influence of factors such as the parameters, installation position, and posture of the multi-view camera make it difficult to determine the exact overlapping region.

Materials and Methods
The adaptive multi-view image mosaic (AMIM) method proposed in this paper realizes multi-view conveyor belt surface fault online detection image stitching by the Appl. Sci. 2021, 11, 2564 4 of 25 following sub-methods: grayscale-based overlapping region registration, the division of the IOI (image of interest), feature-based adaptive partition and block registration, improved weighted smooth fusion. The flow chart of the AMIM method is shown in Figure 3.

Overlapping Region Estimation
The overlapping region estimation model of 2 adjacent images is established according to the geometry of the conveyor belt and the parameters, installation position, and posture of multi-view cameras. For a conveyor belt, the horizontal distance between its two ends is set as B, the speed is V. Suppose it runs smoothly and at a uniform speed. The sensor pixel size of the line-scan camera is w, the number of pixels is n, the line sampling frequency is fh, and the lens focal length is f. For the conveyor belt, the world coordinate system Ow-XwYwZw is established. The Xw axis passes through the bottom end of the arc cross-section of the conveyor belt and is parallel to the ground, and the direction is from left to right. The Zw axis passes through the leftmost endpoint of the arc cross-section of the conveyor belt and is perpendicular to the ground, and the direction is from bottom to top. The intersection point Ow of the Xw axis and the Zw axis is the origin, and the origin coordinate is (0, 0, 0). The direction of V is the same as the Yw axis. Then, in the world coordinate system Ow-XwYwZw, the coordinate of the camera installation position Oc is (X0, Y0, Z0), the unit is millimeter; the camera installation posture is (α, β, γ), the unit is degrees (°). In the line-scan camera imaging process, the space points are projected onto the image sensing pixels to form an image, and the image pixel coordinate system is Of-uv. The imaging schematic diagram of the line-scan camera is shown in Figure 4.
The static imaging of a line-scan camera can only obtain the one-dimensional image of the conveyor belt. Because of the relative motion between the conveyor belt and line-scan camera, the line-scan camera can obtain a two-dimensional image of the conveyor belt by dynamic imaging. Suppose that the coordinate of any point P on the conveyor belt in the world coordinate system Ow-XwYwZw is (XP, YP, ZP), and the coordinate of the point P imaged in the image pixel coordinate system Of-uv is (u, v). According to the fundamental relationship of the perspective projection geometric imaging model of

Overlapping Region Estimation
The overlapping region estimation model of 2 adjacent images is established according to the geometry of the conveyor belt and the parameters, installation position, and posture of multi-view cameras. For a conveyor belt, the horizontal distance between its two ends is set as B, the speed is V. Suppose it runs smoothly and at a uniform speed. The sensor pixel size of the line-scan camera is w, the number of pixels is n, the line sampling frequency is f h , and the lens focal length is f. For the conveyor belt, the world coordinate system O w -X w Y w Z w is established. The X w axis passes through the bottom end of the arc crosssection of the conveyor belt and is parallel to the ground, and the direction is from left to right. The Z w axis passes through the leftmost endpoint of the arc cross-section of the conveyor belt and is perpendicular to the ground, and the direction is from bottom to top. The intersection point O w of the X w axis and the Z w axis is the origin, and the origin coordinate is (0, 0, 0). The direction of V is the same as the Y w axis. Then, in the world coordinate system O w -X w Y w Z w , the coordinate of the camera installation position O c is (X 0 , Y 0 , Z 0 ), the unit is millimeter; the camera installation posture is (α, β, γ), the unit is degrees ( • ). In the line-scan camera imaging process, the space points are projected onto the image sensing pixels to form an image, and the image pixel coordinate system is O f -uv. The imaging schematic diagram of the line-scan camera is shown in Figure 4.
The static imaging of a line-scan camera can only obtain the one-dimensional image of the conveyor belt. Because of the relative motion between the conveyor belt and line-scan camera, the line-scan camera can obtain a two-dimensional image of the conveyor belt by dynamic imaging. Suppose that the coordinate of any point P on the conveyor belt in the world coordinate system O w -X w Y w Z w is (X P , Y P , Z P ), and the coordinate of the point P imaged in the image pixel coordinate system O f -uv is (u, v). According to the fundamental relationship of the perspective projection geometric imaging model of the imaging characteristics of the area-scan camera and the line-scan camera [23,24], one can obtain the dynamic imaging model of the line-scan camera, is seen in Equation (1): where v 0 is the pixel coordinate value of the main imaging point O of sensor pixel center of the line-scan camera, which can be generally set as v 0 = n/2. f = f /w. V represents the relative motion between the conveyor belt and the line-scan camera in each image acquisition cycle, satisfying V = V/f h . Compared with V , the relative motion of the conveyor belt caused by vibration and offset can be ignored. The results show that v is the imaging distortion, and v increases gradually from v 0 to both sides. The Equation is as follows: where k 1 , k 2 , and k 3 are distortion parameters, which can be obtained by camera calibration.
Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 26 the imaging characteristics of the area-scan camera and the line-scan camera [23,24], one can obtain the dynamic imaging model of the line-scan camera, is seen in Equation (1): where v0 is the pixel coordinate value of the main imaging point O of sensor pixel center of the line-scan camera, which can be generally set as v0 = n/2. f′ = f/w. V′ represents the relative motion between the conveyor belt and the line-scan camera in each image acquisition cycle, satisfying V′ = V/fh. Compared with V′, the relative motion of the conveyor belt caused by vibration and offset can be ignored. The results show that △v is the imaging distortion, and △v increases gradually from v0 to both sides. The Equation is as follows: where k1, k2, and k3 are distortion parameters, which can be obtained by camera calibration. Figure 4. Imaging schematic diagram of the line-scan camera.

Optic Axis
In Equation (1), a1, a2, a3, b1, b2, b3, c1, c2, c3 are the elements of the rotation matrix of rigid body transformation between the world coordinate system Ow-XwYwZw and the line-scan camera coordinate system Oc-XcYcZc. The rotation matrix is an orthogonal matrix related to the camera installation posture (α, β, γ), as shown in Equation (3).
cos cos cos sin sin = sin sin cos cos sin sin sin sin cos cos sin cos cos sin cos + sin sin cos sin sin sin cos cos cos In Equation (1), the constraint condition ZP = g(XP) is the equation of conveyor belt arc cross-section in the world coordinate system by polynomial curve fitting. In Equation (1), a 1 , a 2 , a 3 , b 1 , b 2 , b 3 , c 1 , c 2 , c 3 are the elements of the rotation matrix of rigid body transformation between the world coordinate system O w -X w Y w Z w and the line-scan camera coordinate system O c -X c Y c Z c . The rotation matrix is an orthogonal matrix related to the camera installation posture (α, β, γ), as shown in Equation (3).
In Equation (1), the constraint condition Z P = g(X P ) is the equation of conveyor belt arc cross-section in the world coordinate system by polynomial curve fitting.
Let the range of the conveyor belt content corresponding to the overlapping region of 2 adjacent images in the X w axis direction of the world coordinate system be [X P1 , X P2 ]. The overlapping region of 2 adjacent images only considers the v axis coordinates of image pixels. Let the pixel range of the overlapping region of the left image be [v 1 , n]. Let the pixel Appl. Sci. 2021, 11, 2564 6 of 25 range of the overlapping region of the right image be [1, v 2 ]. According to Equation (1), the overlapping region estimation model can be further established, which satisfies the relationship of Equation (4): where, X 01 , Y 01 , Z 01 and X 02 , Y 02 , Z 02 are the installation position coordinates of the cameras corresponding to the 2 adjacent images, and a 1 , a 3 , c 1 , c 3 and a 1 , a 3 , c 1 , c 3 are the elements of the rotation matrix of the cameras corresponding to the 2 adjacent images. When v is v 1 , n, 1, and v 2, the values of ∆v v = v1 , ∆v v = n , ∆v v = 1 , and ∆v v = v2 are the values of ∆v, respectively. The overlapping region of 2 adjacent images can be estimated by solving the values of unknown variables v 1 , v 2 , X w1, and X w2 .

Overlapping Region Registration
Because the range of overlapping region obtained by the overlapping region estimation model is a rough estimation, it is necessary to carry out overlapping region registration. We used the grayscale-based method with gray correlation analysis to measure the correlation between the reference pixel and the comparison pixel in the local area of the image. We can find the position with a higher gray correlation degree in the estimated overlapping region and realize the overlapping region registration.
There are 2 adjacent m × n size images-the gray value of the left image pixel is f 1 (u,v), and the gray value of the right image pixel is f 2 (u,v), where u is the row coordinate, u∈ [1,m], v is the column coordinate, v∈ [1,n]. The overlapping region range of the left and right images estimated is [v 1 ,n] and [1,v 2 ]. Registration of overlapping regions can be divided into the following steps: Step 1: Select the region image whose column coordinates are v∈[v 1 −L,n] in the left image, and select the region image whose column coordinates are v∈[1,v 2 + L] in the right image, where L is the deviation value.
Step 2: Calculate the mean of pixels of each column for the 2 regional images obtained. The Equation is as follows: where img 1 (v) and img 2 (v) are the mean of pixels in column v.
Step 3: Affected by uneven illumination, weak imaging dark regions can appear at the edge of the image. To overcome its influence on stitching, for the left image, starting from v = n, one must look for the first one that satisfies img 1 (v) ≥ T d column by column in a descending manner and denote it as d 1 , d 1 ≥ v 1 . For the right image, starting from v = 1, one must look for the first column that satisfies img 2 [v] ≥ T d column by column in an increasing manner and denote it d 2 , d 2 ≤ v 2 ; T d is the threshold. For the columns of the left image v∈[d 1 + 1,n] and the right image v∈[1,d 2 −1], the gray correlation degree calculation is not performed, and the corresponding gray correlation degree ε l or ε r is assigned the value of zero.
Step 4: The reference sequence X l0 , X r0 and comparison sequence X l1 , X r1 were determined.
where k is an odd number. For 2 images, take the current pixel img 1 (p) and img 2 (q) of the column mean as the center, and take k consecutive data to form a group of comparison sequences, where The k consecutive data centered on img 1 (p) and img 2 (q) are divided into a group of comparison sequences as follows: Step 5: The average processing is used to dimensionless each group of sequence data.
where "mean" means to get the average value of the sequence data.
Step 6: Calculate the difference sequence ∆ l , ∆ r , the maximum difference w l1 , w r1 , the minimum difference w l2 , w r2 .
where "max" and "min" represent the maximum and minimum values in the sequence data, respectively.
Step 7: Calculation of gray correlation coefficient δ l (i) and δ r (i).
Step 9: p = p + 1. If p satisfies the range of p∈[v 1 −L,v 1 + L], continue to calculate the gray correlation degree ε l with img 1 (p) as the current pixel according to step (4). q = q + 1. If q satisfies the range of q∈[v 2 −L,v 2 + L], continue to calculate the gray correlation degree ε r with img 2 (q) as the current pixel according to step (4).
Step 10: Calculate the p value corresponding to the maximum value of ε l and record it as p 0 . Calculate the q value corresponding to the maximum value of ε r and record it as q 0 . Then, the range of overlapping region of the left image is determined as u∈ [1,m], v∈[p 0 ,d 1 ], and the range of overlapping region of the right image is determined as u∈ [1,m], v∈[d 2 ,q 0 ], and the overlapping region registration is completed.
As shown in Figure 5, regions R 12 and R 22 are the overlapping regions after registration, regions R 11 and R 21 are the non-overlapping regions, and regions R 13 and R 23 are the dark regions.
Step 8: Calculation of gray correlation degree  l and  r .
Step 9: p = p + 1. If p satisfies the range of p∈[v1−L,v1 + L], continue to calculate the gray correlation degree εl with img1(p) as the current pixel according to step (4). q = q + 1.
If q satisfies the range of q∈[v2−L,v2 + L], continue to calculate the gray correlation degree εr with img2(q) as the current pixel according to step (4).
Step 10: Calculate the p value corresponding to the maximum value of εl and record it as p0. Calculate the q value corresponding to the maximum value of εr and record it as q0. Then, the range of overlapping region of the left image is determined as u∈ [1,m], v∈ [p0,d1], and the range of overlapping region of the right image is determined as u∈ [1,m], v∈ [d2,q0], and the overlapping region registration is completed.
As shown in Figure 5, regions R12 and R22 are the overlapping regions after registration, regions R11 and R21 are the non-overlapping regions, and regions R13 and R23 are the dark regions.

The Division of The IOI
It is difficult to detect, select, and match feature points in image mosaic of multi-view conveyor belt surface fault online detection, which makes it impossible to use the featurebased method for more accurate image registration. Generally, the useful feature points of the conveyor belt are mainly distributed in the regions such as stripes caused by moving friction, conveyor belt joints, defects, etc. This paper uses an IOI detection algorithm to divide the IOI and the non-IOI image to improve feature-based registration effectiveness. Step 1: A binarization curve is generated for the image partition of interest. Select the HMi(u) curve and the VDi(v) curve described in Equation (15) and Equation (17). Construct a one-dimensional filter window with the size of Lf, and Lf is an odd number. Slide the filter window on the HMi(u) curve and VDi (v) curve, respectively. The mean of Lf pixels is taken as the value of the center pixel of the window. The HM′i(u) For the i-th image of m × n size images collected from N viewpoint, let the gray value of image pixel be f i (u,v), and the overlapping region range be u∈ [1, m], v∈[v a , v b ]. The detection algorithm of the IOI is as follows: if the value of any i∈ [1, N] where T E is the threshold, then the N images to be stitched are considered to be the IOIs with apparent features; otherwise, they are the images with no apparent features. The image evaluation parameter E i is calculated as follows: where k 1 and k 2 are proportional coefficients, which represent the contribution degree of factors and satisfy k 1 + k 2 = 1. Generally, k 1 and k 2 can be set to 0.5. HM i (u) is the mean vector of row pixels, and F i is the mean of image pixels, calculated as follows: VD i (v) is calculated as follows: where VM i (v) is the mean vector of column pixels, which is calculated as follows:

Feature-Based Adaptive Partition and Block Registration
For the IOI, the feature-based adaptive partition and block registration method is further used to improve stitching accuracy. The IOI is segmented adaptively to improve and enhance the efficiency and real-time stitching. The specific operation steps are as follows: Step 1: A binarization curve is generated for the image partition of interest. Select the HM i (u) curve and the VD i (v) curve described in Equation (15) and Equation (17). Construct a one-dimensional filter window with the size of L f , and L f is an odd number. Slide the filter window on the HM i (u) curve and VD i (v) curve, respectively. The mean of L f pixels is taken as the value of the center pixel of the window. The HM i (u) curve and VD i (v) curve are obtained after the mean filtering. Then, perform binarization processing according to the following Equation to generate a binarization curve.
where F i is the mean of HM i (u), as shown in Equation (16), and V i is the mean of VD i (v), which satisfies: Step 2: Deal with the region of interest (ROI) of the binarization curve. For B VDi (v) and B HMi (u) binarization curves, the region with a continuous value of "1" is found as the ROI of the curve, and the start position, end position, and length of the region are marked. The merging threshold T C of adjacent ROIs and the removal threshold T B of isolated small ROIs are set in order to solve the influence of scattered and small regions with continuous "1" on the partition effect of ROIs. Suppose the interval between 2 adjacent ROIs of B VDi (v) and B HMi (u) binarization curve satisfies the condition of less than T C . In that case, the 2 ROIs are merged and connected. If the length of the region is less than T B , the ROI is removed. After the processing, the binarization curves are B VDi (v) and B HMi (u).
Step 3: The ROI of IOI is generated and selected. A template image with the same size as the overlapping region of the IOI is constructed, and its pixel value is g i (u,v).
where u∈ [1,m], v∈[v a , v b ]; through Equation (23), the overlapping region of the IOI to be stitched is divided into 1 background region and K i rectangular ROIs with effective feature points distribution.
If K i ≥ 3, 2 ROIs with larger size are selected; otherwise, all ROIs are selected.
Step 4: The image block is generated and selected. The ROI of the image is divided into N i × N i blocks, and the number of blocks is B i . If B i ≥ 3, 2 image blocks with larger standard deviation are selected; otherwise, all image blocks are selected.
The flow chart of adaptive partition and block operation for the IOI is shown in Figure 7.  We verify the effect of partition operation for the IOI. An IOI with a size of 2048 × 2048 (assuming that the whole image is an overlapping region), as shown in Figure 8a, is partitioned. The curves of HM1(u) and its mean, VD1(v) and its mean are shown in Figure  8c,d. Construct a one-dimensional window template of size 61 (Lf = 61) for mean filtering. The curves of HM′1(u) and its mean are shown in Figure 8e. The curves of VD′1(v) and its mean are shown in Figure 8f.The binarization curves BHM 1 (u) and BVD 1 (v) are obtained as We verify the effect of partition operation for the IOI. An IOI with a size of 2048 × 2048 (assuming that the whole image is an overlapping region), as shown in Figure 8a, is partitioned. The curves of HM 1 (u) and its mean, VD 1 (v) and its mean are shown in Figure 8c,d. Construct a one-dimensional window template of size 61 (L f = 61) for mean filtering. The curves of HM 1 (u) and its mean are shown in Figure 8e. The curves of VD 1 (v) and its mean are shown in Figure 8f.The binarization curves B HM1 (u) and B VD1 (v) are obtained as shown in Figure 8g,h, respectively. Suppose T C = 128 and T B = 128, the binarization curves B HM1 (u) and B VD1 (v) obtained after merging and removing operations are shown in Figure 8i,j, respectively. Finally, the ROI is partitioned, and the result is shown in Figure 8b. It can be seen from 8b that most of the defect information of the conveyor belt is contained in the ROI, which reduces the scope of image registration.
We further verify the effect of adaptive partition and block operation for the IOI. The original images are shown in Figure 9a. L f = 61, T C = 128, T B = 128 are used to process the original image and extract the ROI, as shown in Figure 9b. The ROI image is selected and divided into 128 × 128 blocks. The selected ROI images are shown in Figure 9(c1,c4,d1,e1). The result of 128 × 128 blocks for Figure 9(d1) is shown in Figure 10. The position of the image block is represented by (r i , c i ); r i and c i are the numbers of rows and columns of the image block arranged according to the matrix, respectively, satisfying The number of blocks shown in Figure 10 is 5 × 4. Then, select the image block with a larger standard deviation, as shown in Figure 9c-e, where Figure 9(d2,d3) are the image blocks with the largest standard deviation (Block 1) and the second-largest standard deviation (Block 2), and the positions in Figure 10 are (5,3) and (4,3), respectively. It can be seen that the separated image block contains a high amount of feature information, such as conveyor belt joint or surface fault, which is conducive to the extraction of useful features. The results of the ROI adaptive partition and block method are shown in Table 1. Complete the adaptive partition and block operation We verify the effect of partition operation for the IOI. An IOI with a size of 204 2048 (assuming that the whole image is an overlapping region), as shown in Figure 8a partitioned. The curves of HM1(u) and its mean, VD1(v) and its mean are shown in Figu 8c,d. Construct a one-dimensional window template of size 61 (Lf = 61) for mean filteri The curves of HM′1(u) and its mean are shown in Figure 8e. The curves of VD′1(v) and mean are shown in Figure 8f.The binarization curves BHM 1 (u) and BVD 1 (v) are obtained shown in Figure 8g,h, respectively. Suppose TC = 128 and TB = 128, the binarization curv BHM 1 ′(u) and BVD 1 ′(v) obtained after merging and removing operations are shown in F ures 8i,j, respectively. Finally, the ROI is partitioned, and the result is shown in Figure  It can be seen from 8b that most of the defect information of the conveyor belt is co tained in the ROI, which reduces the scope of image registration.
We further verify the effect of adaptive partition and block operation for the IOI original images are shown in Figure 9a. Lf = 61, TC = 128, TB = 128 are used to proces original image and extract the ROI, as shown in Figure 9b. The ROI image is selected divided into 128 × 128 blocks. The selected ROI images are shown in Figure 9(c1,c4,d The result of 128 × 128 blocks for Figure 9(d1) is shown in Figure 10. The position o image block is represented by (ri, ci); ri and ci are the numbers of rows and columns o image block arranged according to the matrix, respectively, satisfying ri ≤ Ri, ci ≤ Ci, Bi Ci. The number of blocks shown in Figure 10 We further verify the effect of adaptive partition and block operation for the IOI. The original images are shown in Figure 9a. Lf = 61, TC = 128, TB = 128 are used to process the original image and extract the ROI, as shown in Figure 9b. The ROI image is selected and divided into 128 × 128 blocks. The selected ROI images are shown in Figure 9(c1,c4,d1,e1). The result of 128 × 128 blocks for Figure 9(d1) is shown in Figure 10. The position of the image block is represented by (ri, ci); ri and ci are the numbers of rows and columns of the image block arranged according to the matrix, respectively, satisfying ri ≤ Ri, ci ≤ Ci, Bi = Ri × Ci. The number of blocks shown in Figure 10 is 5 × 4. Then, select the image block with a larger standard deviation, as shown in Figure 9c-e, where Figure 9(d2,d3) are the image blocks with the largest standard deviation (Block 1) and the second-largest standard deviation (Block 2), and the positions in Figure 10 are (5,3) and (4,3), respectively. It can be seen that the separated image block contains a high amount of feature information, such as conveyor belt joint or surface fault, which is conducive to the extraction of useful features. The results of the ROI adaptive partition and block method are shown in Table 1   To speed up the calculation, we use the SURF algorithm for feature extraction and description of the selected image block. The SURF algorithm includes the following operations: building the Hessian matrix for feature point extraction, building scale space, feature point location, feature point main direction assignment, feature point descriptor generation, and feature point matching. The random sample consensus (RANSAC) algorithm is used to eliminate the outer points and retain the inner points for image registration. The transformation matrix of the image is constructed through the matching point pairs. According to the transformation matrix obtained by feature registration, the corresponding image is transformed. Feature point matching uses the distance function (Euclidean distance) to retrieve the similarity between high-dimensional vectors. In this paper, the K-D tree algorithm is used. Firstly, the K-D tree is established for each image with the feature descriptor as the tree node information. Then, the K data closest to the corresponding feature points in the other image is searched using the feature points of 1 of the 2 images to be stitched to complete the initial matching of feature points. The flow chart of the feature-based adaptive partition block registration method is shown in Figure 11.  Figure 11. Flow chart of feature-based adaptive partition block registration method.

Improved Weighted Smooth Fusion
The improved weighted smoothing algorithm is used for image fusion to get the stitched image. The expression of the improved weighted smoothing algorithm is as follows: where R11 and R22 are the non-overlapping regions of 2 adjacent images; R12 is the overlapping region; and ω1 and ω2 are the weights of corresponding pixels in the overlapping region, respectively, satisfying 0 < ω1, ω2 < 1 and ω1 + ω2 = 1. To ensure the continuity of overlapping region transition and effectively remove stitching marks, one calculates the weight according to the following equation:

Experiments and Results
The experiment is carried out on the platform, as shown in Figure 12, in order to verify the effectiveness of the method proposed in this paper. Two 2048 × 2048 IOIs with the same surface damage content were selected, as shown in Figure 13(a1,b1). Simulation by using MATLAB software programming in Intel i5 CPU computer showed that the method was effective. The ROI images are shown in Figure 13(a2,b2). The block images

Improved Weighted Smooth Fusion
The improved weighted smoothing algorithm is used for image fusion to get the stitched image. The expression of the improved weighted smoothing algorithm is as follows: where R 11 and R 22 are the non-overlapping regions of 2 adjacent images; R 12 is the overlapping region; and ω 1 and ω 2 are the weights of corresponding pixels in the overlapping region, respectively, satisfying 0 < ω 1 , ω 2 < 1 and ω 1 + ω 2 = 1. To ensure the continuity of overlapping region transition and effectively remove stitching marks, one calculates the weight according to the following equation:

Experiments and Results
The experiment is carried out on the platform, as shown in Figure 12, in order to verify the effectiveness of the method proposed in this paper. Two 2048 × 2048 IOIs with the same surface damage content were selected, as shown in Figure 13(a1,b1). Simulation by using MATLAB software programming in Intel i5 CPU computer showed that the method was effective. The ROI images are shown in Figure 13(a2,b2). The block images are shown in Figure 13(a3,a4,b3,b4). For the original image, the ROI image, and the block image, the SURF algorithm with the Hessian matrix determinant response value of 1000 was used for feature extraction and description. Draw the circle on the image with the feature point as the center. The radius of the circle indicates the size of the feature point, and the straight line indicates the direction of the feature point. The feature points are described as shown in Figure 13(a5-a8,b5-b8). was used for feature extraction and description. Draw the circle on the image with the feature point as the center. The radius of the circle indicates the size of the feature point, and the straight line indicates the direction of the feature point. The feature points are described as shown in Figure 13(a5-a8,b5-b8). The comparison of image feature extraction is shown in Table 2. It can be seen that the number of feature points extracted in the ROI of Figure 13(a1,b1) were 65.52% and 69.71% of the whole image, respectively. The number of feature points extracted from the ROI accounts for a high proportion of the number of feature points in the whole image, reflecting that the ROI is the feature point concentration region of the whole image. However, it took 40.48% and 37.18% of the total image time, respectively, which was less time-consuming. Although the number of feature points extracted from two image blocks was less than the ROI, the feature points extracted had larger feature values and significant features, which met easy registration requirements. Moreover, the two image blocks took less time. The total time was only 16 ms, which was 2.77% and 2.93% of the whole image time, respectively. It is thus suitable for fast registration. The comparison of image feature extraction is shown in Table 2. It can be seen that the number of feature points extracted in the ROI of Figure 13(a1,b1) were 65.52% and 69.71% of the whole image, respectively. The number of feature points extracted from the ROI accounts for a high proportion of the number of feature points in the whole image, reflecting that the ROI is the feature point concentration region of the whole image. However, it took 40.48% and 37.18% of the total image time, respectively, which was less time-consuming. Although the number of feature points extracted from two image blocks was less than the ROI, the feature points extracted had larger feature values and significant features, which met easy registration requirements. Moreover, the two image blocks took less time. The total time was only 16 ms, which was 2.77% and 2.93% of the whole image time, respectively. It is thus suitable for fast registration.
For the two images of Figure 13(a1,b1), the different threshold values of Hessian matrix determinant response value, such as 500, 1000, 1500, and 2000, were used to extract the feature points of each image. The comparison in Table 3 shows that with the increase of the threshold value, the number of feature points extracted and the time consumed was reduced. However, the threshold value is important for the block image, because of the small image size. If the threshold value is too large, too few feature points are extracted in the block, which is not conducive to feature point registration. For example, when the threshold value was 2000, the number of feature points shown in Figure 13(b7) was only one. The threshold value of the Hessian matrix determinant response value can be selected as 1000 by comparison.   For the two images of Figure 13(a1,b1), the different threshold values of Hessian matrix determinant response value, such as 500, 1000, 1500, and 2000, were used to ex- For the feature points extracted from the original image, ROI image, and block image, we used the RANSAC algorithm to complete the matching of feature points. The matching results are shown in Figure 14. The comparison of registration is shown in Table 4. It can be seen that a sufficient number of registration feature point pairs can be obtained from block images. The time of registration processing on an Intel i5 CPU computer was 5 ms, which was 41.67% of the total image time. Table 2. Comparison of image feature extraction when the threshold value of Hessian matrix determinant response value is 1000. Figure 13(a5) Figure 13(a6) Figure 13(a7) Figure 13(a8) Figure 13(b5) Figure 13(b6) Figure 13(b7) Figure 13   algorithm of the IOI, the images in Figure 16a were identified as the non-IOI. For the overlapping region estimated from the image in Figure 16b, the evaluation parameters E of the IOI calculated were 29.06 and 36.97, respectively. The images in Figure 16b were identified as the IOI. Only for the Figure 16b, using 128 × 128 block size, was the feature-based adaptive partition block registration method carried out. Finally, the improved weighted smoothing algorithm was used for image fusion, and the stitched image was obtained, as shown in Figure 16c,d. The subjective evaluation of the stitched image shows that the stitching effect of the AMIM method is good.   Figure 13(a6) and Figure 13(b6) registration. (c) Figure 13(a7) and Figure 13(b7) registration. (d) Figure 13(a8) and Figure 13(b8) registration. The proposed method was applied in the mine for conveyor belt surface fault online detection, as shown in Figure 15. To further verify the effectiveness of the AMIM method, we selected two groups of images collected by two line-scan cameras with different viewpoints. The image size was 2048 × 2048, as shown in Figure 16a Figure 16a were 11.22 and 8.14, respectively. The evaluation parameters E were calculated by Equation (14). According to the detection algorithm of the IOI, the images in Figure 16a were identified as the non-IOI. For the overlapping region estimated from the image in Figure 16b, the evaluation parameters E of the IOI calculated were 29.06 and 36.97, respectively. The images in Figure 16b were identified as the IOI. Only for the Figure 16b, using 128 × 128 block size, was the feature-based adaptive partition block registration method carried out. Finally, the improved weighted smoothing algorithm was used for image fusion, and the stitched image was obtained, as shown in Figure 16c,d. The subjective evaluation of the stitched image shows that the stitching effect of the AMIM method is good.  The AMIM method is evaluated from the registration rate and running time to verify the effectiveness of our method more objectively.

Parameters
The registration rate measures the accuracy of the matched feature points.
where Z is the registration rate, c1 is the number of mismatched feature pairs, and c2 is the number of feature pairs in the RANSAC algorithm. The AMIM method is evaluated from the registration rate and running time to verify the effectiveness of our method more objectively.
The registration rate measures the accuracy of the matched feature points.
where Z is the registration rate, c1 is the number of mismatched feature pairs, and c2 is the number of feature pairs in the RANSAC algorithm. The AMIM method is evaluated from the registration rate and running time to verify the effectiveness of our method more objectively.
The registration rate measures the accuracy of the matched feature points.
where Z is the registration rate, c 1 is the number of mismatched feature pairs, and c 2 is the number of feature pairs in the RANSAC algorithm.
The comparison between the AMIM method and other methods is shown in Table 5. For the images in Figure 16a, SIFT and SURF method cannot get the correct registration feature points and cannot realize the accurate image stitching operation on the basis of features. However, the AMIM method does not use the feature-based registration method after identifying the image as the non-IOI. It uses the overlapping region of registration to stitch the image directly. For the image in Figure 16b, although the number of feature points extracted by our method was small, the registration rate was high, reaching 97.67%. The average time of stitching two adjacent images was less than 500 ms. Thus, our method ensures image stitching accuracy, reduces the registration time, and improves image stitching efficiency. Therefore, our method has advantages in the process of multi-view conveyor belt image stitching.

Conclusions
Correct operation of the conveyor belt is very important for production and staff safety. The multi-view conveyor belt surface fault online detection system based on machine vision can provide a sufficient guarantee. This paper proposed an adaptive multi-view image mosaic method based on the combination of grayscale and feature to improve the accuracy and real-time of image mosaic and realize the multi-view conveyor belt surface fault online detection. Compared with the traditional mosaic methods, the proposed method has the following improvements and advantages: The overlapping region of two adjacent images is preliminarily estimated by establishing the overlapping region estimation model, and then the grayscale-based method is used to register the overlapping region.
For the collected image, the IOI detection algorithm is used to divide it into the IOI and the non-IOI, and different registration methods are used.
For the IOI, its overlapping region may contain the fault information of the conveyor belt. The feature-based partition and block registration method is used to register the images more accurately, the overlapping region is adaptively segmented, the SURF algorithm is used to extract the feature points, and the random sample consensus RANSAC algorithm is used to achieve accurate registration. It is easy to realize the accurate measurement of the geometric shape of the fault, and it is conducive to the accurate and reliable detection and evaluation of the health status of the conveyor belt.
For the non-IOI, its overlapping region almost does not contain the fault information of the conveyor belt, and the features are not obvious. It is easy to have the wrong registration by using the SIFT algorithm, SURF algorithm, and other feature-based registration methods, and the registration rate is even 0%. Thus, they cannot obtain the right result, and the process is time-consuming, which is not conducive to real-time online detection. Accurate registration is not performed on the adjacent non-IOIs, which improves the accuracy and real-time of stitching.
The improved weighted smooth fusion algorithm is used to fuse the images to realize image stitching.
The experimental results show that when the SURF algorithm is used to extract and describe the block image features, the threshold value of the Hessian matrix determinant response value is 1000, which is better compared with 500, 1500, and 2000. It has the advantage of extracting enough feature points and consuming less time. Enough feature points can be extracted from the block image, and the features are significant, which is easy to register accurately. Compared with the whole image registration, it takes less time. The results of the experimental analysis showed the usefulness of the proposed method. The stitching effect is better compared with the SIFT and SURF algorithms. Simultaneously, it can reduce the amount of stitching calculation and the time of the stitching process, and the registration rate is high. The registration rate can reach 97.67%, and the average time of stitching two adjacent images is less than 500 ms. It is suitable for conveyor belt surface fault online detection. In addition, this method can also provide a reference for other application types of image mosaic.
The proposed method was applied in the mine for conveyor belt surface fault online detection and can be extended to port, electric power, metallurgy, chemical industry, and other fields for conveyor belt surface fault online detection. In the future, we will study the methods of accurate and reliable detection and evaluation of conveyor belt health. In order to improve the detection performance, we can further use the hybrid imaging method combining thermal imaging [25] and visible light imaging to explore ways to realize the fusion of heterogeneous images and multi-view image mosaic at the same time. It is also valuable to explore the application of deep learning in conveyor belt fault detection [26].
Multi-view is widely used in machine vision-based detection and has played an important role. The proposed method in this paper can be used in industry, agriculture, medical, and other fields of fault detection using the multi-view method [27][28][29][30]. Compared with other improved image mosaic methods, the proposed method is more suitable for solving the problem of conveyor belt surface fault online detection. For example, the IOI detection algorithm is proposed on the basis of the characteristics of the conveyor belt image, and it may not be adaptable to other detection object images for different applications. However, for other applications, the proposed method in this paper has a reference value for image mosaic problems that need to take into account both accuracy and real-time.