Safety Warning of Mine Conveyor Belt Based on Binocular Vision

Abstract

For the wear and damage to mine conveyor belts by foreign materials, such as large materials, this paper designs a volume measurement method for large materials based on binocular vision by the volume measurement of the large material inon the conveyor belt, provides safety-warning data of over-volume limits for mine conveyor belts, and prevents damage to the mine conveyor belts. In this design, first, by obtaining the binocular camera parameters to improve the image-edge information, images taken with a binocular camera were then aligned by the Bouguet stereo correction algorithm. Finally, the disparity map was calculated by the semi-global-block matching (SGBM) stereo matching, and to get the target 3D coordinates, volume measurements were performed by the micrometric method. The study shows that the experimental error remains at about 5.65%, which provides a low-cost measurement method for raw-coal-volume measurement of the well industry and coal-mine belt conveyor.

1. Introduction

As the world’s largest coal power producer and consumer, coal resources have played an indelible role in the development of China’s national economy [1]. Coal output is an important index of the economic benefit of coal mining enterprises, and the intelligent development of coal mines in China is in the primary stage. The innovation of theory and technology requires our continuous efforts to promote development [2,3]. As the key equipment of coal transportation, the mine conveyor belt is in the harsh environment of the mine. In the long-term use of a high-load conveyor belt, it is easily affected by foreign bodies such as large coal and gangue if not cleaned up in time, and in the process of falling they may cause partial deviation, skids, belt breaks, and longitudinal tear accidents. Light damage to the mining conveyor belt affects the safety of production and the economic benefits of the coal mine; heavy damage causes the conveyor belt to tear and deviate and other conditions occur, causing a foreign-body splash onto pedestrians near the belt, or it is a direct threat to the life and safety of mine workers. Concerning the occurrence of mining conveyor belt tears, deviations, and other abnormal phenomena, there are two detection methods: contact and non-contact types. Contact detection mainly determines the nonlinear deformation caused by pressure; this method has a low accuracy rate and is easy to damage. The non-contact detection method mainly includes an embedding method, ultrasonic method, X-ray detection, frequency emission method, and so on. The above methods are reliable, but the price is expensive and complex. The contactless detection method used in this paper is computer vision, setting the large material volume threshold as the limit, carrying an early warning of mine conveyer-belt safety; this method is low-cost, easy to maintain, and highly reliable. In an accurate measurement of a large material volume, it is helpful to monitor real-time damage of large materials and coal-storage materials to the mining conveyor belt, to maintain the safe use of mine conveyor belts and the safety of miners. It is of great significance.

Currently, the volume measurement method mainly adopts manual and instrument measurement methods. There are many limitations in manual measurement, such as many difficulties in measurement, long measurement time consumption, lack of real-time performance, large influence of artificial subjective factors, and serious limitations in the measurement environment. There are still many deficiencies in the contact electronic belt scale [4] in the instrument measurement, but the use of image processing and other technologies simplifies the production process, helps to control costs, and reflects the trend of digital visualization [5]. It meets the needs and development trend for the national construction of intelligent coal mines. Mi Yizhou [6] used binocular stereo vision to reconstruct the regular wrapping object in 3D, and obtained the coordinates of the four corners of the regular wrapping to calculate the wrapping volume, and this method is only valid for measuring regular objects; Shao Baofeng [7] proposed combining point lasers to obtain depth information and locate image feature points, but this method is only limited to regular objects, and the cost is relatively high compared to simple binocular cameras; Gao Ruxin et al. [8] used binocular stereo vision and the SURF (speed-up features) feature-matching algorithm to extract feature points to achieve volume calculation. However, the feature points in this method are relatively sparse, so a large error is generated to measure volume. In this paper, combined with the principle of binocular vision, this noncontact method for measuring irregular raw coal is proposed. The method includes camera calibration, image acquisition, image processing, correction and stereo matching, parallax acquisition, and volume calculation. The image processing and stereo matching are compared with various methods, and experimental verification is carried out.

2. The principle of Binocular Vision

2.1. Binocular Camera Model

Binocular stereo vision is established on the basis of the human visual system. Through the principle of parallax and similar triangles, the three-dimensional information of the object to be measured is obtained from multiple images [9]. As shown in Figure 1, the distance from a point in space to the imaging plane of the camera can be obtained according to the proportional relationship.

The points P_l and P_r in the figure are the image points of the point P on the imaging planes of the left and right cameras, respectively, and Z is expressed as the distance from the point P to the camera, which can be calculated by the following Equations (1)–(4):

$$d=\left|x_l-x_r\right|$$

(1)

$$s=T-\left(x_l-x_r\right)$$

(2)

$$\frac{T-\left(x_l-x_r\right)}{z-f}=\frac{T}{z}$$

(3)

$$z=\frac{fT}{x_l-x_r}$$

(4)

where the distance from the left and right image points to the imaging surface is X_l and X_r, respectively, and d represents the parallax of point P on the binocular camera; the distance from point P_l to P_r is s; and the distance between the left and right cameras is T.

2.2. Camera Calibration

The parameters of the camera are obtained by Zhang Zhengyou’s calibration method [10] and images of different angles are taken for the checkerboard of Zhang’s calibration method, which makes the calibration results more accurate. In this paper, an 8 × 12 checkerboard is selected, and the grid size is 22 mm × 22 mm. The checkerboard for the experiment is shown in Figure 2.

During the shooting process, make sure that the binocular cameras are on the same horizontal line and then collect the calibration boards in different positions to ensure that the calibration boards in each photo account for about 1/4 of the camera. In the collected photos, name the left and right cameras respectively and store in two folders respectively.

Run the MATLAB program, input the left camera image collected by the binocular camera into the Camera Calibrator calibration toolbox, and preset the size and size of the checkerboard, and the calibration toolbox will automatically extract the corners of each image [11]. The calibration effect is shown in Figure 3.

The calibration process of the right camera is the same. The calibration results of the left and right cameras are shown in

Table 1. The calibration result of the camera.

	Internal Parameter Matrix	Distortion Parameter
left camera	$M_l=\left[\begin{array}{ccc}459.349& 1.143& 329.263\\ 0& 456.678& 225.593\\ 0& 0& 1\end{array}\right]$	$k_l=\left[\begin{array}{ccccc}0.0154& -0.0344& -0.00385& -0.00033& 0\end{array}\right]$
right camera	$M_r=\left[\begin{array}{ccc}461.474& 0.893& 321.159\\ 0& 458.286& 254.798\\ 0& 0& 1\end{array}\right]$	$k_r=\left[\begin{array}{ccccc}0.0125& -0.0303& -0.0038& 0.00033& 0\end{array}\right]$

.

Binocular cameras are mainly in the same world frame, unified with the respective world coordinate systems of the left and right cameras. Using the stereo-camera calibrator toolbox in MATLAB, the rotation matrix R and the translation vector T were obtained for the binocular camera; the obtained parameters are shown in

Table 2. The rotation matrix of the binocular camera works with the translation vector.

$R=\left[\begin{array}{ccc}1.000& 0.000959& 0.000867\\ -0.000958& 1.000& -0.00105\\ -0.000868& 0.00105& 1.000\end{array}\right]$

$T=\left[\begin{array}{ccc}-25.586& -2.2287& 1.387\end{array}\right]$

.

3. Image Preprocessing and Stereo Correction

3.1. Image Preprocessing

In the process of collecting images, to reduce the noise pollution caused by some mis-operation on the collected images, such as environmental background interference, camera or human environmental factors, or high exposure caused by strong light, the image processing process [12] is shown in Figure 4.

Figure 5 shows the original image of the object to be tested collected by the binocular camera.

First of all, grayscale processing is used. The process of compressing and expressing image information is grayscale processing. The grayscale pixel information is single channel, which greatly compresses the data volume of the image and replaces the color information of the RGB three channels. The grayscale rendering is shown in Figure 6.

Then binarization is based on grayscale, directly discarding the detailed information of the grayscale change of the image and only retaining the two grayscale values of black and white in the picture, which greatly simplifies the image information. Commonly used are the OTSU threshold segmentation method, adaptive threshold segmentation, etc. OTSU threshold segmentation is an efficient algorithm that uses thresholds to divide the original image into foreground and background images, and maximizing the inter-class variance is its essential idea. The adaptive threshold segmentation method used in this paper is a local threshold segmentation method. The segmentation object is the domain window on the image, which can clearly distinguish the coal block from the background area and can better deal with images with large differences in light and shade. The binarization effect diagram is shown in Figure 7. Compared with the adaptive threshold segmentation method, the OTSU algorithm cannot better highlight the coal block edge.

Mean filtering uses the mean value of the gray values of all pixels in the window instead, while Gaussian filtering uses a Gaussian function to calculate the weighted average coefficient of the filtering window. Since the weighted average coefficient is calculated according to the Gaussian function, the closer to the center of the filter window, the greater the weight, and the greater blurring of the image will not be produced. The effect after filtering is shown in Figure 8. Compared with Gaussian filtering, the average filtering image is blurred, and the feature points of the detected object are lost.

Finally, edge detection is performed on the image [13] to highlight the details of the target contour. Common methods include Canny and Sobel operators. The Canny operator calculates the gradient value of the image for the horizontal and vertical gradients, respectively, and performs non-maximum suppression and elimination. With duplicate data, two thresholds with different heights are preset, and the pixels that are smaller than the lowest threshold and those that are higher than the highest threshold are removed.

The Sobel operator finds the edge of the image by calculating the first-order gradient among the pixels. The gradient calculation direction is divided into vertical and horizontal directions. The detection process of the Sobel operator mainly refers to the convolution operation between the pair of operators and the image window, as shown in Equations (5)–(7).

$$G_x=\left(z_7+z_8+z_9\right)-\left(z_1+2z_2+z_3\right)$$

(5)

$$G_y=\left(z_3+2z_6+z_9\right)-\left(z_1+2z_4+z_7\right)$$

(6)

$$\nabla f=\sqrt{G_x^2+G_y^2}$$

(7)

where z₁, z₂, …, z₉ are the pixel values in the window, G_x and G_y are the horizontal and vertical convolution results, respectively, and ▽f is the new pixel in the center of the window. The comparison chart of the edge-detection effect is shown in Figure 9. It can be seen that the Canny operator cannot better reflect the detailed outline of the coal block surface compared with the Sobel operator.

3.2. Stereo Correction

The main purpose of stereo correction is to correct the left and right images and remove distortion, so that the two are on the exact same plane. Correction was performed in OpenCV using Bouguet limit correction [14]. The composite rotation matrices Rl and Rr of the left camera and the right camera, respectively, can be obtained by decomposing the rotation matrix of the right camera relative to the left camera.

To make the baseline and the image parallel, a transformation matrix Rrect can be established. The method is implemented by the offset matrix T of the binocular camera [15]. In e₁, to make the epipolar line in a horizontal state, Rrect shifts the pole of the left view to infinity. It can be seen that the direction of the pole of the left view is the translation vector of the left and right cameras.

$$e_1=\frac{T}{\Vert T\Vert }$$

(8)

$$T={\left[\begin{array}{ccc}{T}_x& T_y& Tz\end{array}\right]}^T$$

(9)

$$e_2=\frac{\left[\begin{array}{ccc}-T_y& T_x& 0\end{array}\right]}{\sqrt{T_x^2+T_y^2}}$$

(10)

In the case of obtaining the directions of e₁ and e₂, the direction of e₃ is perpendicular to e₁ and e₂, and their cross product is e₁. The matrix Rrect can be obtained by moving the pole of the left camera to infinity, as shown in Equation (11):

$$R_{rect}=\left[\begin{array}{c}{e}_1^T\\ e_2^T\\ e_3^T\end{array}\right]$$

(11)

After correction, the corresponding points in the image are on the same horizontal line, and the effect is shown in Figure 10:

4. Stereo-Matching Algorithm

SAD (sum of absolute differences), BM (block matching), and SGBM algorithms are three commonly used algorithms in OpenCV.

The SAD algorithm is to take the source-matching point of the image acquired by the left camera as the center point, and then define a window D, calculate the sum of the gray values in D, and then determine the window according to the corresponding point of the right-eye image along the polar direction, according to different inspections, and calculate the difference between the gray value and the difference between the gray level of the left and right windows one by one, and determine the central pixel of the area with the smallest difference as the matching point [16].

The basic idea of the BM method is to divide the frames of the left and right cameras into multiple small cubes, then move a small cube to compare the pixel position of the small cube in another graph, and then calculate the parallax information from the calibration results of the binocular camera.

The SGBM algorithm breaks the matching cost algorithm of mutual trust in the SGM (semi-global matching) algorithm and uses the BT algorithm that approximates the global two-dimensional smoothness constraint, so that it has the characteristics of good parallax effect and fast speed [17].

The gradient parallax plot of the image is obtained by using the Sobel operator, which has the influence on the image to reduce the uneven lighting and noise.

$$s\left(x,y\right)=2\left[P\left(x+1,y\right)-P\left(x-1,y\right)\right]+P(x+1,y-1)-P\left(x+1,y+1\right)-P\left(x-1,y+1\right)$$

(12)

where s(x, y) is the pixel value processed by the 3 × 3 operator, and P is the pixel value in the unprocessed image. After processing the horizontal Sobel operator, each pixel value on the image is processed by the following function.

$$P=\{\begin{array}{l}0, P<-preFilterCap\\ P+preFilterCap,-preFilterCap\le P\le preFilterCap\\ 2+preFilterCap,P\ge preFilterCap\end{array}$$

(13)

where preFilterCap is a constant parameter, and the value is adjusted by example to obtain the image gradient information of the subsequent cost calculation [18].

According to Equation (14), the census transformation makes the pixel values of the traversal domain window against the pixel values of the center point of the window, and maps the resulting Boolean value on a little string, if the field window size is m × n (both odd), then m′ and n′ are positive integers.

$$C_s\left(u,v\right)=\underset{i=-n\prime }{\overset{n\prime }{\otimes }}\underset{j=-m\prime }{\overset{m\prime }{\otimes }}\xi \left(I\left(u,v\right),I\left(u+i,v+j\right)\right)$$

(14)

$$\xi \left(x,y\right)=\{\begin{array}{l}0,x\le y\\ 1,x>y\end{array}$$

(15)

where the census transformation value C_s is expressed as the value of the bit string, and where m and n are twice as great as m′ and n′.

The dynamic programming algorithm is ineffective for places where the parallax changes greatly, often resulting in matching errors, and it is difficult to avoid using the wrong parallax information for subsequent calculations, affecting the parallax accuracy. Its energy function is as follows:

$$E\left(D\right)={\displaystyle \sum _P\left(C\left(P,D_P\right)+{\displaystyle \sum _{q\in N_p}\left[\left|D_P-D_q\right|=1\right]}+{\displaystyle \sum _{q\in N_p}{P}_{2}T\left[\left|D_P-D_q\right|>1\right]}\right)}$$

(16)

where D is the pre-matching of the parallax graph, E(D) corresponds to the energy function of the parallax graph, p and q are certain pixels in the image, Np is the set of pixels in the field of pixel p, and C(p, D_p) is the matching cost [19].

In SGBM, the SGM algorithm proposes to combine information in multiple directions to minimize the impact of error and obtain the optimal energy function. SGM is a Markov-based energy equation that accumulates information in different directions and ends up with the winner-take-all (WTA) algorithm [20] determining the parallax size of each pixel, which will affect the accuracy of the method.

$$L_r\left(p,d\right)=C\left(p,d\right)+\min \left\{\begin{array}{l}{L}_r\left(p-r,d\right)\\ L_r\left(p-r,d-1\right)+P_1\\ L_r\left(p-r,d+1\right)+P_1\\ \underset{i}{\min L_r}\left(p-r,i\right)+P_2\end{array}\right\}-\min L_r(p-r,i)$$

(17)

$$S\left(p,d\right)={\displaystyle \sum _r{L}_r}\left(p,d\right)$$

(18)

where L_r is the new path generation value; S is the result of the aggregation of the total path cost; both P₁ and P₂ are penalty coefficients, where P₁ applies to those pixels in the field and the P-spot parallax is equivalent to 1, while P₂ is those pixels that differ by more than 1. As for the role of I as a judgment function, if I is judged to be true, 1 is returned; otherwise, 0 is returned.

As shown in Figure 11, the three-dimensional matching-effect diagram, it can be clearly seen that the SAD algorithm divides the measured object, and it is difficult to see the general outline of the object to be tested. Moreover, under the SAD algorithm, the ground-depth-information collection is blurred, so the accurate distance between the binocular camera and the ground cannot be judged, and the effect is very poor. The BM algorithm has an obvious graph segmentation effect on the measured object. The outline of the object to be measured is very clear, but the processing of the ground-depth information is not excellent. There is still a large number of black areas that cannot be identified, but the running speed is fast. The image segmentation of the SGBM algorithm is not as effective as that under the BM algorithm, but it still can clearly identify the specific outline of the object to test, and, relative to the above two methods, the running speed is slower than the BM algorithm, but still within the accepted range. The ground-depth information processing is perfect; the black (where it cannot identify the area) is greatly reduced.

5. Calculation of Target Volume

This article uses a WSD-2022-V binocular camera (Shenzhen Weishida Technology Co., Ltd., Shenzhen, China), an Intel(R) Core-i5 laptop, and other supporting equipment. Under the Win10 system, MATLAB is used for raw-coal image processing, Visual Studio 2022 (VS) code writing platform is used, and the OpenCV4.1.0 visual open-source library is used. The camera used in the test is shown in Figure 12.

In this experiment, the extrinsic and extrinsic parameters of the camera are first obtained. Complete the preprocessing of the raw coal image in MATLAB, and then complete the stereo correction and matching work in the VS and OpenCV environment, analyze the acquired depth information, and realize the volume calculation.

After the algorithm runs, adjust the image smoothing and other processing operations to make the object in the image stand out. Click the part of the object to be measured in the window with the mouse to obtain the target distance. The effect diagram is shown in Figure 13.

In the process of ranging, the binocular camera is kept horizontal, and the distance from the ground is always maintained at a height of 400 mm. For irregular coal blocks, 10 random points are taken on it, their heights are measured, and the actual average height of the coal blocks is obtained. Then, in the image after the algorithm runs, 10 points are collected and divided into five groups, and the average height of the measurement is calculated. The error measurement is shown in

Table 3. Data results for the target ranging.

	Height from Coal Block/mm	Height above Ground/mm	Object Height Measurement/mm	Relative Error
Group One	338.313	397.34	59.027	1.79%
Group Two	345.335	406.019	60.684	0.97%
Group Three	338.994	401.31	62.316	3.68%
Group Four	337.066	397.38	60.314	0.37%
Group Five	344.435	402.823	58.388	2.85%

.

From this table, it can be seen that the error range of the height measurement of the object is about 0.37~3.68%, and the average error is about 1.93%. The reasons for the error are analyzed:

(1)

The accuracy of the camera is not high enough, and the camera used is an entry-level commercial camera, which is far lower than the accuracy of an industrial camera;

(2)

The irregular shape on the coal block and the reason why its color distinction is not obvious enough cause the disparity map generated by the coal block in the algorithm to be flawed;

(3)

The experimental environment is simple, which has a certain influence on the measurement accuracy of the binocular camera.

The origin of the coordinate system is mainly the left camera, the object area is set as the area to be measured, and the remaining area is the background area, which provides ground depth information for the camera’s measurement. Then, the area to be measured is divided and accumulated as the final volume of the target by the micro-element method. As shown in Figure 14, the laboratory mine conveyor belt and large raw coal block used in this paper are marked as A, B, and C, respectively from left to right. This experiment was carried out in the mining conveyor belt platform built in the laboratory.

Count the depth information in the background area, discard the maximum and minimum values, then calculate the average value of the remaining depth information, and then calculate the size of the object to be measured according to the horizontal and vertical coordinates of the pixels in the area to be measured, and divide the area into n sizes for the sub-region of i × j; the maximum and minimum values are also discarded, the depth value is obtained, the average depth value that can represent the region is calculated, and then the volume of the sub-region is obtained according to the formula, and finally the volume of the sub-region of the region to be measured is accumulated to obtain the total volume, such as in Equations (19)–(21):

$$V_1=\left(H_0-{\overline{h}}_1\right)\cdot i\cdot j$$

(19)

$$V_s={\displaystyle \sum _1^m{V}_m}$$

(20)

$$V_s={\displaystyle \sum \left(H_0-h\right)}$$

(21)

where V₁ is the volume of the sub-region; i and j represent the size of the sub-region; ${\overline{h}}_1$ is the depth of the sub-region; V_s represents the sum of the accumulated sub-regions; and H₀ represents the height of the current scene from the ground.

The volume of the coal block was obtained by binocular vision, and the actual volume was measured by the drainage method. The data are shown in

Table 4. Results of the target volume measurements.

	Coal Measurement Volume/(mm3)	Actual Volume of Coal/(mm3)	Relative Error/%
A	674,177.828945	650,000	2.95%
B	455,641.367206	430,000	5.96%
C	244,690.001879	230,000	6.39%
B + C	708,409.589607	660,000	7.33%

.

It can be seen that the smaller the coal block, the greater the error, and when the two pieces of coal are put together, the error increases. Analyzing the above reasons for error, the distance makes small coal measurements more accurate than large coal measurements; the average error was calculated to be 5.65%. When two coal blocks are measured at the same time, the interference between objects and coal blocks are not rule bodies, and the uneven surface, coal and the ground gap, and coal surface color disparity are not obvious, the coal block is hollow on the side and top view occlusion of a series of factors such as false volume leads to error, but for irregular shapes of coal this still has a certain reference value.

6. Conclusions

This low-cost, noncontact method of coal-block volume measurement is adopted in this paper; image processing was performed using MATLAB, the Bouguet algorithm corrected for images acquired by binocular vision using VS and in OpenCV environment, the SGBM algorithm was used for stereo matching, and parallax maps were calculated and optimized to get depth information.

(1)

Experiments show that the measurement error is about 5.65%; on the basis of this error, the volume of foreign bodies such as irregular large coal blocks and gangue is the set threshold. In this way, a mine conveyor belt safety-warning system is designed to achieve the timely maintenance of the mine conveyor belt for the normal safety of mine production and to protect the life and safety of mine workers.

(2)

This design was performed in a laboratory environment, with low cost, and simple and easy operation as the basic idea to use. In the practical application process, in natural light situations, the recognition effect is quite good. Further research is still needed in actual mines.