Detection Method of Crushing Mouth Loose Material Blockage Based on SSD Algorithm

Abstract

With the advancement of smart mines technology, unmanned and Shojinka have received widespread attention, among which unattended crushing station is one of the research directions. To realize unattended crushing station, first of all, it is necessary to detect loose material blockage at the crushing mouth. Based on deep learning (DL) and machine vision (MV) technology, an on-line detection method is studied to trace the blockage in a swift and accurate manner, so that the corresponding detection system can be designed accordingly. The charge coupled device (CCD) industrial camera set above the crushing mouth is used to collect images and input them to the edge computing equipment. The original Single Shot MultiBox Detector (SSD) preprocessing model is trained and optimized before it is combined with the MV technology to detect and then the MV technology is combined to detect whether the crushing mouth is covered. In Ansteel Group GUANBAOSHAN mine, the accuracy of recognition and detection system with human observation was examined for one month, and the tested accuracy is 95%. The experimental results show that the proposed method can detect the crushing mouth blockage in real time, which would solve the problem that the blockage can only be identified by human eyes in traditional method, and provides basic support for the unattended crushing station.

1. Introduction

For most open-pit mines, the ore needs to be roughly crushed in the crushing station before entering the next processing flow. The crushing station is equipped with a crushing mouth, and the on-site monitoring personnel send a dumping signal. The ore truck unloads the ore into the crushing mouth, and there is a certain probability that the crushing mouth would become blocked in the process. Once the crushing mouth is blocked, the efficiency of mine production would be lowered, leaving truck and the conveying belt lay idled. At present, loose material blockage identification is mainly conducted by on-site monitoring personnel, who detect the blockage by observation and send out an alarm. With the advancement of smart mine technologies, unmanned and Shojinka have received widespread attention, among which unattended crushing station is one of the research directions.

With the development of image recognition technology, image recognition technology has been applied in different fields to replace the manual work. Znamenskaya et al. shows the possibility of conducting research in gas dynamics based on big data analysis of digital recordings using MV and machine learning [1]. Yang et al. proposed the detection method of bubble defects on tire surfaces based on line lasers and MV [2]. Li et al. presented both a MV based method and a prototype of an industrial hardware design for stress-crack detection in maize kernels [3]. Mirbod et al. presented the industrial parts change recognition model using MV image processing in the framework of industrial information integration for research purpose [4]. Dlamini et al. introduced a real-time MV system designed for detecting defects in functional textile fabrics with good detection precision and high detection speed to assist in textile industry quality control [5]. Singh et al. proposed automation surface detection framework using MV and convolutional neural networks [6]. Yang et al. analyzed the formation and growth mechanism of Exchanger surface frosting and condensation, then summarized the current research status of MV technology in defrosting and decondensation [7]. MV has been applied in various industries with remarkable results.

The theory of DL has been widely used in more and more industries. Wang et al. through the integration of DL into the workflow, from image processing to simulating physical processes, pore-scale imaging and modeling has advanced greatly [8]. Haghighi et al. proposed a stacking ensemble model of DL and its application to Persian/Arabic handwritten digits recognition [9]. Abed et al. studied automated prediction of solar flares from SDO images using DL [10]. Hasan et al. proposed a survey of DL techniques for weed detection from images [11]. Said et al. proposed a DL model to evaluate and predict the performance of an internet of things (IoT) communication system [12]. Rohila et al. proposed an automated diagnosis of COVID-19 infection from CT scans of the patients using DL technique [13]. Pawar et al. proposed model in their study proved to be suitable for the classification of mural images on the basis of every performance parameter [14]. Al Duhayyim et al. presented an artificial ecosystem-based optimization with an improved DL model for IoT-assisted sustainable waste management [15].

DL and MV have been increasingly applied in the mining industry, which helped solve some problems in mining practice and achieve research result. Patel et al. developed a MV-based ore classification model using support vector machine (SVM) algorithm [16]. Wang TH et al. studied mine conveyor belt deviation detection system based on MV [17]. Xiao et al. quantified particle size and size distribution of mine tailings through DL approach of auto encoders [18]. Zhang et al. achieved a luminance equalization-based DL mineral identification method [19]. Lesego et al. proposed a new and reliable method to detect drill bit failure in rotary percussion drills using DL [20]. Hoang et al. predicted blast-induced ground vibration in open-pit mines using different nature-inspired optimization algorithms and deep neural network [21]. Liu et al. with the development of artificial intelligence (AI) and computer technology, the DL based mineral image classification system is gradually applied to ore sorting [22].

In recent years, the DL based detection methods mainly include the Fast R-CNN [23], the Faster R-CNN [24], YOLO [25], SSD [26] etc. Among them, SSD, as a typical target detection algorithm, is a new end-to-end detection algorithm. It greatly improves the speed while maintaining high accuracy and real-time performance, and has excellent detection effects on small targets and multi targets [26]. Jiang et al. proposed a fast traffic accident identification method based on SSD model [27]. Sun et al. presented an advanced model based on DL trying to explore good recognition performance for industrial environment [28]. Li et al. put forward a multi-block SSD method based on small object detection for railway scenes in unmanned aerial vehicle surveillance [29]. Nevertheless, currently there is little research that uses cheap cameras and the SSD model based on DL technology, crushing mouth blockage can be monitored in real time, which could effectively replace expensive sensor equipment and manual work.

In mine production, ore blockage occur frequently as there are so many links in mine production procedure involve ore transportation. Wang used proximity switch to detect chute material blockage [30]. LUO et al. proposed the method of ore identification and detection based on Mask RCNN, which could provide precise visual guidance for the intelligent blocking manipulator [31]. XIAO et al. used photoelectric sensors to make hopper blockage detection devices [32]. These methods can only be used in specific scenarios, which cannot be widely applied. This paper proposes a method of detecting the blockage of rotary crushers widely used in open-pit mining. The method is not only widely applicable, but it is also beneficial for replacing the manual work required for blockage detection. At present, the AI-based blockage detection of the crushing mouths is still blank in both the research and the mining practice. At present, the blockage detection of the crushing mouth mainly depends on depends on manual work, and the recognition application based on artificial intelligence is still blank.

The overall objective of the study is to propose a low-cost and intelligent monitoring method based on DL techniques using camera to detect crushing mouth blockage. This paper, supported by MV technology, edge computing technology and IoT technology with only a CCD camera to achieve real-time monitoring. Based on DL and MV technology, an on-line detection method is studied to detect the blockage quickly and accurately, so that the corresponding detection system can be designed accordingly. The CCD industrial camera set above the crushing mouth is used to collect images and input them to the edge computing equipment. The recognition model of the blockage is trained through DL first, and then the MV technology is combined to detect whether the crushing mouth is covered and send out alarm if blockage is detected. Finally, 5G router is employed to output the result to IoT server by MQTT protocol, then the user can view the real-time and historical data of blockage through web server. The experimental results show that the proposed method can detect the blockage at the crushing mouth in real time, which solves the problem that the traditional method can only identify blockage by human eyes, and provides a basic support for the unattended crushing station.

2. SSD Algorithm

At present, with the development of image recognition technology and DL technology, detection algorithms based on DL object detection have become the mainstream research direction in the image recognition, DL detection algorithms such as the Fast R-CNN [23], the R-FCN [33], the Faster R-CNN [24], SSD [26] and others have emerged rapidly. An object detection named the application programming interface (API) project that can provide a variety of network structures is trained by the COCO database. Based on these structures, an object detection model can be trained by modifying relevant parameters. Therefore, it is unnecessary to build an individual framework. The SSD model, which excels in recognition accuracy and swiftness in object detection models, is selected as the detection framework in this study.

SSD is a precise, efficient and real-time detection algorithm proposed by Liu W et al. in 2016 [26]. The network structure of SSD is shown in Figure 1. A 3-channel RGB image is used as the input, and the VGG16 network is employed as the basic network for feature extraction. An additional convolution layer and pooling layer are added as auxiliary components to classify and locate the features, respectively. Next, a multiscale feature layer group composed of Conv4_3, Conv6, Conv7, Conv8_2, Conv9_2, Conv10_2 and Conv11_2 is formed. The feature map of the shallow network in a larger size contains more details, so that it is designed for detecting small targets. The feature map extracted by the deep network, which is mainly used to detect large targets, contains less information due to its small size. The numbers of channels in the six feature maps are 512, 1024, 1024, 512, 256, 256 and 256, respectively; and their corresponding sizes are 38 × 38, 19 × 19, 19 × 19, 10 × 10, 5 × 5, 3 × 3 and 1 × 1, respectively. The 1 × 1 and 3 × 3 convolution kernels are used to connect each feature layer, and the test results are integrated. After inputting a single image, the SSD algorithm generates a certain number of candidate boxes with different aspect ratios through CNN training. Each candidate box has position offset and target category score information, and then a Non-Maximum Suppression (NMS) operation to generate prediction results is used.

The objective function of the SSD detection algorithm is extended on the basis of the Multi-Box objective function, so that it can handle multiple categories of objects at the same time. The loss function is weighed by the location loss and confidence loss, and if $x_{ij}^p=1$ matches the i prediction box with the j real box of class p, while $x_{ij}^p=0$ does not match, then [26]:

$$L\left(x,c,l,g\right)=\frac{1}{N}\left(L_{conf}\left(x,c\right)+a{L}_{loc}\left(x,l,g\right)\right)$$

(1)

Among them, x = {1,0} represents whether the target exists, and x is 1 when the ratio of intersection to merger (IoU) is greater than the set threshold, otherwise it is 0. c is confidence; l is the prediction box; g is the true box; N represents the number of prediction boxes that match the real box, and when N = 0, the default loss is 0; a is the weight coefficient.

The location loss function is defined as follows [26]:

$$L_{loc}\left(x,l,g\right)={\displaystyle \sum }_{i\in Pos}^N{\displaystyle \sum }_{m\in \left\{cx,cy,w,h\right\}}^N{x}_{ij}^{k}smoot{h}_{L1}\left(l_i^m-{\widehat{g}}_j^m\right)$$

(2)

Among them, ${\widehat{g}}_j^{cx}=\frac{\left(g_j^{cx}-d_i^{cx}\right)}{d_i^w}$, ${\widehat{g}}_j^{cy}=\frac{\left(g_j^{cy}-d_i^{cy}\right)}{d_i^h}$, ${\widehat{g}}_j^w=\log \left(\frac{g_i^w}{d_i^w}\right)$, ${\widehat{g}}_j^h=\log \left(\frac{g_i^h}{d_i^h}\right)$.

The confidence loss is defined as follows [26]:

$$L_{conf}\left(x,c\right)=-{\displaystyle \sum }_{i\in Pos}^N{x}_{ij}^p\log \left({\widehat{c}}_i^p\right)-{\displaystyle \sum }_{i\in Neg}^N\log \left({\widehat{c}}_i^0\right)$$

(3)

where ${\widehat{c}}_i^p$ is the probability that the $i$ prediction box matches $p$, ${\widehat{c}}_i^p=\frac{\exp \left({\widehat{c}}_i^p\right)}{{{\displaystyle \sum }}_p\exp \left({\widehat{c}}_i^p\right)}$.

3. Composition of Crushing Mouth Blockage Detection

3.1. System Structure of Crushing Mouth Blockage Detection

Crushing mouth blockage detection system structure is shown in Figure 2, the hardware part by a CCD industrial camera, light source, bracket, edge computing device, 5G router and other accessory parts. The CCD industrial camera collects the image of the crushing mouth and inputs the collected image into the edge computing device.

The crushing mouth works for 24 h a day, two light sources are configured for the intensity of light. The bracket is made of 4040c industrial aluminum profile, with a shape of which is a goal structure. The crossbeam part can move up and down to adjust the distance between the camera and crushing mouth, the camera and light source are installed on the crossbeam and it can adjust to the most favorable shooting angle and catch the best light by moving horizontally. The edge computing device is used to process the image, output the result through the serial port of the edge computing device, and transmit it to the 5G router. The 5G router is used to transmit the results to the IoT server on cloud platform via the 5G wireless network with the MQTT protocol of the IoT for data visualization and display.

Web server and IoT server are deployed on the cloud platform. The former is for providing web service and the latter is receiving and storing the data sent by 5G router. Both the real-time data and the historical data of crushing mouth blockage can be accessible through 5G network on mobile terminals and via the internet on PC, as shown in Figure 3.

The title in the upper part of the figure shows that the crushing mouth functions as the image acquisition device, and the image acquisition frequency is once every 10 s. Next, the original image obtained by the camera is displayed. In addition, at lower part, the information of the image acquisition time, latest status and number of alarms is displayed. At the lower part, the curve of alarming time is displayed by default. Historical data is available by pressing the button “<” or “>”.

3.2. The Flow Diagram of Crushing Mouth Blockage Detection

The flow diagram of crushing mouth blockage detection is presented in Figure 4. The crushing mouth blockage detection method mainly consists of the following steps: (1) Collect the images which contain the crushing mouth in the crushing stations; (2) Mark the image to acquire the XML file with tag information, which can be divided into training set and verification set; (3) Convert XML file to CSV file first, and then to TFRecord format file to form PBTXT file; (4) Set the pipeline configuration file, acquire three results from the training, and generate the frozen inference diagram file; (5) Collect the video of the crushing mouth and input it into the LabVIEW development environment, and conduct real-time monitoring of the crushing mouth blockage according to the judgment algorithm; (6) When the crushing mouth blockage is detected, an alarm is triggered warn the operator.

4. Data Acquisition and Model Building

Install a camera above the crushing mouth to capture video. After collecting a large amount of image data with crushing mouth through the camera equipment, the LabelImg tool was used to mark the crushing mouth in the image and take it as the input of the SSD pretraining model. The parameters were adjusted by repeatedly training the model. The weight representing the characteristics of the crushing mouth is generated, the frame image of the video is read, and the weight is read. Ultimately, the specific location and range of the crushing mouth in the image are identified, so that whether the crushing month is blocked can be judged by algorithm. The block diagram is shown in Figure 5.

4.1. Collect the Image of the Crushing Mouth

Capture videos from the camera installed above the crushing mouth, including videos in various weather conditions, as well as videos in different times within a day. In the captured video, the images of the crushing mouth can be seen and saved as JPG image files, which are 5300 JPG files in total.

4.2. Image Preprocessing

Use LabelImg tool to manually mark the saved image file to the crushing mouth to obtain XML files with marking information for each JPG file, altogether 5300 XML files are obtained, some examples as shown in Figure 6. The JPG file and XML file are divided into training set and verification set. The training set consists of about 2/3 of all files or 7066 files in total, which are used to train the identification model of the crushing mouth. The verification set is about 1/3 of all documents, totaling 3534 files, used to verify the training effect of the crushing mouth.

4.3. Training Preparation

Write a python program to convert the XML file with tag information in training set and verification set into CSV file format, as shown in

Table 1. CSV file format with tag information.

Filename	Width	Height	Class	xmin	ymin	xmax	ymax
20220613154238.jpg	1920	1200	mouth	566	402	1312	925
20220613154244.jpg	1920	1200	mouth	587	397	1307	926
20220613154250.jpg	1920	1200	mouth	579	391	1324	926
20220616122934.jpg	1920	1200	mouth	565	382	1351	975
20220616131045.jpg	1920	1200	mouth	557	371	1361	963
20220616131239.jpg	1920	1200	mouth	564	380	1357	955
20220616133704.jpg	1920	1200	mouth	572	378	1365	965
20220616133904.jpg	1920	1200	mouth	566	386	1381	957
20220616134104.jpg	1920	1200	mouth	580	366	1349	963
20220617101303.jpg	1920	1200	mouth	584	408	1307	928

.

Write python programs to convert CSV files with marked information in training set and verification set into TFRecord format dedicated to TensorFlow. The PBTXT file is prepared, which contains the name of the crushing opening to be identified. The format is as follows:

• item {id: 1 name: ‘mouth’};

4.4. Start Training

Use TensorFlow’s Object Detection API to train dataset, use the target detection algorithm SSD based on DL to implement training, set the pipeline configuration file “ssd_mobilenet_v1_coco.config”, modify the number of classifications “num_classes” to 1, the number of training samples in each batch “batch_size” to 32, and the total number of training steps “num_steps” to 1400000. The path of TFRecord format dataset for training and verification is set to the TFRecord format file generated in step 3.3., Label map path set to the PBTXT file generated in step 3.3.

Start training and acquire three training result files. The first file “model.ckpt-1400000.data-00000-of-0001” stores the values of each weight, biases, gradients and other variables. The second file “model.ckpt-1400000.index” is an index, which is used to match the data in the first file with the structure of the third file, that is, to which part of the graph does the value belongs. The third file “model.ckpt-1400000.meta” is stores the structure of TensorFlow calculation graph, or, the network structure of neural network. Next, use the three training result files to generate the frozen inference diagram “frozen_inference_graph.pb” file. This file is the solidified final neural network model, which contains graph definition and training parameters.

4.5. Monitoring of Crushing Mouth Blockage

Collect video from the camera installed above the crushing mouth and input it into the LabVIEW development environment. Based on the LabVIEW development environment combined with DL function, the DL function mainly combines the solidified final neural network model with the development environment. The freeze inference diagram file obtained in 3.4 is used as input, before it is combined with the blockage identification algorithm to conduct real-time crushing mouth blockage detection.

The specific identification algorithm is shown in the Figure 7.

Step1: Initialize the recognition rate of the crushing mouth to 0.5, the check number of light alarm to 5, the check number of critical alarm to 30. The recognition rate is used to represent the matching degree of the MV combined with the frozen inference map file for the identified object. The maximum value is 1, representing the complete matching of the identified object. The minimum value is 0. Light alarm is used to indicate that the crushing mouth is not covered for a long time, the detection system acquires an image every 5 s for analysis, that is to say, and the light alarm will be triggered when the crushing mouth is not recognized within 25 s. As the ore continues to enter the crushing mouth, the crushing mouth may return to normal state. Light alarm signal is transmitted to the automatic control system of the crushing mouth through the interface for automatic dumping control. The critical alarm is used to indicate that the crushing mouth has been covered more than three minutes, and the crushing mouth can no longer be restored to the normal state, experience shows that a truck of ore should enter the crushing mouth within 150 s, that is to say, the critical alarm will be triggered when the crushing mouth is not recognized within 150 s. It is transmitted to the remote-control center through the interface, prompting manual processing.

Step2: Reset identification number to 0.

Step3: Get images from camera.

Step4: Judge whether the crushing mouth is identified and recognition rate greater than 0.5? If yes, go to the next step; otherwise, return to step 2.

Step5: Identification number plus 1.

Step6: Judge whether the identification number is greater than the check number of light alarm (5)? If yes, go to the next step; otherwise, start Step 8.

Step7: Trigger the light alarm of crushing mouth blockage.

Step8: Judge whether the identification number is greater than the check number of critical alarm (30)? If yes, go to the next step; otherwise, return to step 3.

Step9: Trigger the critical alarm of crushing mouth blockage, return to step 3.

4.6. Trigger Alarm

The alarm is triggered according to the detection results.

5. Experiment and Results

5.1. Experimental Procedure and Parameters

The size of crushing mouth image collected through the CCD industrial camera is 1920 × 1200. The camera model is osg 230–150 μm with 2.3 million pixels, 150 frames and USB3.0. As the dust on the site is relatively large, the camera is sealed in a protective box with tempered glass with high light transmittance, which can protect the camera from the dust. The edge computing device is connected with the camera through USB3.0, and the specific configuration is: CPU core i74500U, memory 8G, hard disk solid-state 128G, USB × 4, LAN × 2, HDMI × 1, COM × 2, which supports the automatic power on function with win10 operating system and LabVIEW 2020 runtime. IoT server of ThingsBoard and HTTP server of Tomcat are deployed on the cloud, both ThingsBoard and Tomcat are open-source applications, the former is used for data collection, data processing, visualization, and device management, while the latter for the Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket technologies.μ

5.2. Experimental Results

The model training works on the Windows operating system, and based on the TensorFlow platform, the SSD model was tested on a NVIDIA GeForce GTX 1060 6GB. The training is using the TensorBoard tool, and the model is built by comparing evaluation indexes such as the total loss value, AR (Average Recall) and mAP (mean Average Precision). Due to the total loss value, the AR and mAP tend to be flat after approximately 50,000 iterations, the total loss value is approximately 0.58, the mAP is 0.92 and the AR is approximately 0.91. In addition, the crushing mouth recognition model is trained based on the Faster R-CNN algorithm under the same input conditions and parameters. When the number of iterations is same 50,000, the total loss of the model is approximately 0.75, the mAP is approximately 0.86, and the AR is 0.84. According to the comparison result in this paper, the SSD model is better than the Faster R-CNN model, as shown in Figure 8. Therefore, the object detection model is derived and generated based on the SSD model.

Based on DL technology, the model built in the third section was employed to detect crushing mouth in different scenarios. Some identification results are summarized as shown in Figure 9 and Figure 10. Figure 9 shows position and score of the crushing mouth in the image identified by the object detection model. Figure 10 shows the image of crushing mouth blockage by identification algorithm. Furthermore, light alarm and critical alarm can also be triggered off by identification model and algorithm.

In order to verify the applicability and stability of the method, comparative study of the critical alarm information with the data recorded manually on site was conducted in August, 2022. In August, there were 96 critical alarms, including 0 missing alarm and 5 false alarms. Missing alarm indicates the blockage whereas the system does not send off alarm, therefore, false alarm scenario is where the system triggers the alarm when no actual blockage happens, some false alarm samples as shown in Figure 11. By analyzing false alarm pictures, it can be concluded that the false alarm might be caused by the sunlight to which the crushing mouth is exposed. The solution is to collect as many pictures as possible of the crushing mouth exposed to sunlight in different times before they are used to train the model.

6. Discussion

This research draws on the strength of DL in object detection and applies it to detecting crushing mouth blockage. According to the position information of the identification frame of the crushing mouth in the image, the detection algorithm is used to judge whether there is any loose material. This case study shows that this method can effectively identify blockage, and it is also suitable for 24-h production. First, compared with other methods using cameras to capture images and analyzing them based on images, this method does not require a series of image processing steps, such as edge detection, gray scale stretching and binary processing, can save considerable computing memory and time. In addition, this method has the advantages of low cost, low impact and less maintenance. Second, no existing DL technology dominates to date has explored the field of crushing mouth blockage. In addition, if missing alarm and false alarm are found in operation, it only requires the corresponding images to be collected for retraining models. After a period of iteration, the recognition accuracy can be continuously increased. Thus, this method possesses excellent potential for future application in the crushing mouth blockage.

Although these studies reveal some important findings, they also have limitations. First of all, for early data collection and processing, it requires onerous and intensive work. As it is necessary to collect a large number of images and mark off the positions crushing mouths in all the images manually. Therefore, there might be some errors in the manual recognition procedure due to the insufficient image data collected. In addition, bad weather may increase the difficulty of automatic recognition of crushing mouth in images. Therefore, it is necessary to collect as much image data as possible of various monitoring points that cover all sorts of situations.

We will further conduct follow-up studies on these issues. We will combine our research with the automatic dumping system at the crushing station to realize the early warning of the blockage. Once the trend of blockage is traced, the amount of ore dumped by the truck would be reduced so that the bulk of the ore would enter the crushing mouth and the rest of the ore would be dumped gradually to avoid the real blockage. Moreover, after the recognition accuracy is enhanced, the unmanned crushing station can be realized, and the automatic dumping command can be carried out.

7. Conclusions

Based on DL and MV technology, an on-line detection method is studied to detect crushing mouth blockage quickly and accurately, so that the corresponding detection system can be accordingly designed. In Ansteel Group GUANBAOSHAN mine, the difference between human eye recognition and detection system is compared for a period of one month, and the accuracy is 95%. The experimental results show that the proposed method can detect the crushing mouth blockage in real time, which solves the problem that the traditional method can only identify blockage through manual work, and provides a basic support for the unattended crushing station. The main conclusions are as follows:

(1)

This work obtains an automatic crushing mouth blockage identification model by adopting the SSD model with training and verification functions using a series of images. The results indicate that the mAP of the model on the data set reached approximately 0.92. Moreover, compared with the Faster RCNN, the SSD algorithm is more suitable for this research.

(2)

This study solves the problem that the traditional method can only identify the crushing mouth blockage by human eyes. Can meet the requirement of continuous video capture in the mine around the clock all year round.

(3)

This study is based on MV to detect the crushing mouth blockage, to provide basic support for unattended crushing station.

(4)

This study can provide some reference for the identification of tarpaulin in other similar concentrators, and provide support for the construction of dark factory in the future.

Based on the current research results, in the future, it is considered to develop an automatic dredge robot for crushing mouth blockage to replace the existing dredge method of excavator.