Gangue is a solid waste with low carbon content, which is usually mixed into the raw coal during production. Due to the inefficiency of gangue separation, much gangue flows out of the mining area, which increased the transporting costs and caused severe pollution to the environment [1
]. Therefore, effectively sorting gangue from the raw coal is essential in improving the quality of the coal and reducing the costs of transport. With the increase of labor costs and the need to avoid hazards to worker’s health, automatically separating gangue from raw coal has become a critical issue in recent years. As a contactless inspection technology, computer vision has been widely applied in driverless cars [2
], medical diagnostics, remote sensing, mineral processing, and many other fields. From an engineering perspective, it seeks to automate many tasks the human vision can perform. Human beings can distinguish gangue and coal by the differences in brightness, color, morphology, texture, and other features. So far in the literature, a number of studies have been conducted on the gangue image features extracting algorithms to separate gangue from coal.
] suggested a vision-based gangue sorting model based on the analysis of color texture and a multilayer perceptron (MLP) neural network. Color texture features were extracted from hue saturation value (HSV) and luminance chrominance (YCbCr) color spaces, respectively, which were used as inputs to the MLP neural network to sort gangue. Hong [4
] built a deep learning model using a convolutional neural network (CNN) and transfer learning to distinguish coal and gangue images. The typical workflow for CNN image recognition was presented and the model was tested with photos from a washing plant. Su [5
] improved the LeNet-5 coal gangue identification model to achieve a recognition rate of 95.88%. Many other similar kinds of literature are not listed because of the limited length. These studies have primarily focused on classifying the image category, (i.e., detecting whether gangue or coal existed in the image). The results of these studies would not provide the position and shape information of gangue in the images to guide the sorting manipulators.
Image segmentation is an important task in computer vision facilitated areas. The results of image segmentation can divide the image into object regions that can provide the position of the target in the image. Gao [6
] proposed an algorithm based on grayscale feature decision theory to recognize gangue over a moving belt conveyor. The threshold of the grayscale distribution histogram was calculated by the Bayesian decision theory. Wei [7
] proposed a method of combining image feature extraction and artificial neural network to identify gangue, and the performance of the algorithm was tested on a dataset consisted of 20 images. Liu [8
] proposed a computer vision approach for feature extraction of coal and gangue. Grayscale and texture features were extracted using multifractal detrending fluctuation analysis (MFDFA). Sun [9
] proposed a morphology-based method to separate gangue from coal, and the supplementary texture is extracted based on morphology. Pre-treatments of coal and gangue images have been introduced. Li [10
] proposed an image-based hierarchical deep learning framework for coal and gangue detection. Gaussian pyramid principle was applied to construct the training dataset, and convolution neural networks were used as a feature extractor to generate the classification and detection box.
Though previous research has reported acceptable detection or segmentation accuracy for dealing with gangue images, the existing approaches still need to be improved because the gangue and coal in their images datasets were sparse, the entire image even contained individual categories. However, images collected on the conveyor belt in a coalfield are often complicated, with coal and gangue randomly heaped with ash. Thus, comprehensive studies on the gangue segmentation methods with regard to complex environmental conditions call for much attention.
Deep learning, as one of the most interested scientific research trends [11
], has brought revolutionary advances in computer vision and machine learning [13
]. Unlike traditional machine learning techniques, it can automatically learn representations from images without introducing hand-coded rules or human domain knowledge [14
]. Convolutional neural networks (CNNs), one of the most popular methodologies in deep learning, have been widely applied as an efficient architecture for feature extraction [15
]. A fully convolutional network (FCN) is a particular type of CNN, which replaces the fully connected layer with convolution operation, and has been proven to be an ideal way to accomplish the image segmentation tasks [16
]. The U-Net has further improved this method, a network structure first created by Ronneberger [17
] and applied to biomedical image segmentation. The outstanding performance of U-Net in biomedical image segmentation without the need for a massive number of training images inspired the authors to examine the performance of U-Net in gangue image segmentation. To the best of our knowledge, this is the first research work considering U-Net in gangue sorting.
In this article, a fully convolutional neural network model called U-Net was applied to segment gangue from raw coal images collected under complex conditions. The main approach was to build an end-to-end deep learning model, which could be generalized to the complex conditions in applying to a coalfield. The U-Net model was trained using 54 manually labeled images. The trained U-Net model was able to provide probability maps from six unseen testing images. The model was successfully deployed to guide two manipulators sorting gangue from a moving conveyor belt. The results of the present research indicated that the proposed solution is able to segment gangue from raw coal effectively. The rest of this paper is organized as follows. First, in Section 2
, the collection and preparation of the raw coal images used in our study are described. This is followed by a detailed description of the proposed approach. The obtained results are presented and discussed in Section 4
. Finally, Section 5
concludes the research.
The method for the segmentation of gangue images presented in this paper features a novel structure of convolutional neural networks, which have the capability of extracting multiple features of the images. A fully convolutional neural network called U-Net was constructed initially, followed by training the network using the coal image dataset that have been collected under complex environmental conditions. The trained U-Net is able to segment gangue pixels with an accuracy of a human capability (AUC = 0.96). For some testing images (see the third row in Figure 6
), the results were perfect, meaning that almost each gangue pixel within the image was correctly predicted. It is also worth noting that a lump of small coal heaped on the edge of a gangue (see the third row in Figure 6
), and the predicted borders around the coal were in good agreement with the contours of the ground truth.
These results demonstrate that U-Net is promising for the industry application of coal image segmentations. However, this method still needs future improving to be used in more areas. It should be noted that the method developed in this research is based on the image data collected from Datong Coalfield, Shanxi, China. The coals obtained in this region are Middle Jurassic coals that are characterized by low ash yield content, low moisture content, low-medium volatile bituminous, and ultra-low sulfur content. Since the coals from the different areas should differ significantly from each other, the method proposed in this research may not be able to obtain similar results for other images, especially for the situation where the geological condition of two coalfields were significantly different from each other. However, the results of this research provide an insight into using a similar method for more complex coal image segmentation. Additionally, the model can be improved by training on more comprehensive datasets, which should give more satisfying results in multiple coal deposits, which are also the future work of this research.