Next Article in Journal
Table-Based Adaptive Digital Phase-Locked Loop for GNSS Receivers Operating in Moon Exploration Missions
Next Article in Special Issue
A Compact IIoT System for Remote Monitoring and Control of a Micro Hydropower Plant
Previous Article in Journal
Retrieval of Suspended Sediment Concentration from Bathymetric Bias of Airborne LiDAR
Previous Article in Special Issue
Possibilities of Real Time Monitoring of Micropollutants in Wastewater Using Laser-Induced Raman & Fluorescence Spectroscopy (LIRFS) and Artificial Intelligence (AI)
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Automated Identification of Overheated Belt Conveyor Idlers in Thermal Images with Complex Backgrounds Using Binary Classification with CNN

AMC Vibro Sp. z o.o., Pilotow 2e, 31-462 Kraków, Poland
Department of Robotics and Mechatronics, AGH University of Science and Technology, Al. Mickiewicza 30, 30-059 Kraków, Poland
Department of Mining, Faculty of Geoengineering, Mining and Geology, Wrocław University of Science and Technology, 50-370 Wroclaw, Poland
Author to whom correspondence should be addressed.
Sensors 2022, 22(24), 10004;
Submission received: 4 November 2022 / Revised: 2 December 2022 / Accepted: 12 December 2022 / Published: 19 December 2022
(This article belongs to the Special Issue Application of Wireless Sensor Networks in Environmental Monitoring)


Mechanical industrial infrastructures in mining sites must be monitored regularly. Conveyor systems are mechanical systems that are commonly used for safe and efficient transportation of bulk goods in mines. Regular inspection of conveyor systems is a challenging task for mining enterprises, as conveyor systems’ lengths can reach tens of kilometers, where several thousand idlers need to be monitored. Considering the harsh environmental conditions that can affect human health, manual inspection of conveyor systems can be extremely difficult. Hence, the authors proposed an automatic robotics-based inspection for condition monitoring of belt conveyor idlers using infrared images, instead of vibrations and acoustic signals that are commonly used for condition monitoring applications. The first step in the whole process is to segment the overheated idlers from the complex background. However, classical image segmentation techniques do not always deliver accurate results in the detection of target in infrared images with complex backgrounds. For improving the quality of captured infrared images, preprocessing stages are introduced. Afterward, an anomaly detection method based on an outlier detection technique is applied to the preprocessed image for the segmentation of hotspots. Due to the presence of different thermal sources in mining sites that can be captured and wrongly identified as overheated idlers, in this research, we address the overheated idler detection process as an image binary classification task. For this reason, a Convolutional Neural Network (CNN) was used for the binary classification of the segmented thermal images. The accuracy of the proposed condition monitoring technique was compared with our previous research. The metrics for the previous methodology reach a precision of 0.4590 and an F1 score of 0.6292. The metrics for the proposed method reach a precision of 0.9740 and an F1 score of 0.9782. The proposed classification method considerably improved our previous results in terms of the true identification of overheated idlers in the presence of complex backgrounds.

1. Introduction

The automation and robotization of condition monitoring (CM) processes in the mining sector are inevitable steps for increasing the industrial equipment life cycle and improving the safety of operations. Modern automation methods provide solutions to reduce or totally exclude the presence of humans in hazardous environments. Using mobile robots for CM applications can improve the inspection process, as they apply more sophisticated techniques for identification and localization of faults in industrial infrastructures located in mining sites [1,2,3,4,5,6].
Conveyor systems have been developed throughout the past decades and used as the most common method for transporting raw materials in mining sites. The idlers are responsible for supporting the loaded bulk materials’ weights that are carried by the belt [7,8]. Faulty idlers can cause serious damage in conveyor systems [9,10].
The minimum L10 life requirement (the amount of time in which 10% of the bearings will fail) or, in other words, the lifetime of idler bearing in normal environmental conditions, should be 50,000 h or 5–7 years. However, this time can considerably decrease due to environmental conditions and belt speed [11,12].
Harsh environmental conditions in mining sites can considerably reduce the life span of the conveyor system idlers. Dust, high temperature, and sunlight can be considered as environmental factors that can affect the life span of idlers. The idler sealings are usually poor; therefore, the accumulation of dust in conveyor systems can increase the rotation resistance of the conveyor idlers. Similarly, high ambient temperatures and sun reflection in hot, dry climates can increase the bearing temperature, which reduces the bearing life span [13]. The idler bearings’ condition can be monitored by analyzing their surface temperatures, noise emissions, and vibrations [14,15]. Robotic-based infrared thermography (IRT) can be considered a reliable method for automatic CM of idlers [16,17].
Different fault signatures in rotating machinery can be caused by misalignment, and bearing damage and mass imbalance are recognizable by IRT methods [18,19,20,21]. Analyzing the temperature patterns on the outer surface of idlers can give us information to evaluate the degree of deterioration on inspected idler bearings [22,23]. Detection of anomalous temperature signatures in thermal or infrared (IR) images can help us to determine the overheated idlers [24].
Two different methods can be used to measure the thermal condition of idlers. In quantitative methods, we need to determine the exact temperature of the idler surfaces. On the other hand, in the qualitative methods, the acquired thermal signatures are analyzed to find the characteristic patterns based on relative temperatures of equipment in a captured scene. In our study, we chose to identify the overheated idlers base on qualitative methods; therefore, the relative temperature values of detected hotspots in IR images were analyzed with respect to other areas [25,26,27,28].
Modern IR image analysis software has the capability to prepare an inspection report. However, despite its ease of use and functionality, the manual evaluation of data is time-consuming for inspectors [29]. Furthermore, in our case, for conducting inspection programs in a large mining site, where series of conveyor systems need to be inspected individually, the manual evaluation of reports can become more complicated.
Over-centralized distribution of pixels and their low-intensity contrast are the main features of the IR images that bring difficulties in correct identification of temperature anomalies within the region of interest (ROI). The classical threshold-based segmentation techniques can accurately segment the ROI areas in images that are captured in visual light. However, due to the nature of IR images, the accuracy of final results can be affected by over- and undersegmentation [30,31,32]. Furthermore, in our case study, the presence of reflective objects with high emissivity values that are not related to idlers can create complex background conditions which directly affect the accuracy of segmentation results [33].
In this paper, we developed solutions for improving the proposed CM method in our previous study [24]. Therefore, we focused on proposing a classification method for accurate detection and classification of the overheated idlers in segmented frames from other thermal sources that were not related to idlers. The development of a methodology for the classification of thermal defects in segmented IR images can allow us to improve the number of true detections and the overall accuracy of the proposed CM method. So, the main contributions are:
A novel method based on binary classification with CNN was developed for classification of overheated idlers in segmented frames.
We showed that our proposed CM method can accurately classify the overheated idlers, even in IR images with complex backgrounds.
Our novel CM method based on the deep learning approach was compared against an IRT method that was specifically developed for identification of the overheated idlers in IR images.

2. Literature Review

In this section, we firstly focus on the importance of application of robotic-based IRT for the reliability of conveyor systems in mining environments. The length of a conveyor belt in mining sites can reach several kilometers, while hundreds of idlers that support the belt, and the loads need to be monitored by technicians [34,35]. The manual condition monitoring methods are time-consuming and inaccurate due to low frequency of inspections. Furthermore, working in high-pollution environments can be harmful for human health [36,37,38]. By using automatized CM solutions in mining sites, operators can considerably reduce the number of unplanned shutdowns and increase Overall Equipment Efficiency (OEE) [39,40,41].
Autonomous or semi-autonomous robotics IRT are categorized as a non-destructive CM method that can be used as a proper solution for automatic detection of faulty idlers in conveyor systems. In IRT-based techniques, the emitted IR radiation from objects’ surfaces is analyzed for identification of temperature anomalies [42].
IRT methods are used in a wide range of fields, including medical imaging, non-destructive testing, defect detection in civil structures, and so on [43]. However, due to the complexity of mining environments and various factors that need to be considered in developing IRT methods, the application of machine learning methods in automated diagnoses of IR images captured in mining environments is still in the early stages [44,45,46].
Most of the proposed automated diagnostic system for analyzing IR images of industrial infrastructures consist of three steps: estimation of ROI, extraction of relevant features and, finally, classification of extracted features [18,47,48]. The definition of ROI can reduce the complexity of the analyzed images and improve the feature extraction results in the next steps. Afterward, the desirable features should be processed and extracted in such a way that it gives us enough information to understand the current state of the monitored equipment. Finally, in the classification stage, the extracted features from previous stages are studied to understand whether or not the images contain possible signs of thermal defects.
In [16,20,24,49], researchers proposed a different CM method using the IRT method for identification of overheated idlers in conveyor systems located in mining sites. From the investigations with IR images taken with mobile robots, some researchers only focused on the segmentation of hotspots without discriminating against the defected idlers. In [16], for identifying thermal defects in idlers, researchers proposed a method based on Canny edge detection and blob detection techniques. In other researches, alternatively, overheated idlers were segmented by the following methods: color-based thresholding [20], an outlier detection technique [24], and a simple thresholding method based on maximum and minimum temperatures [49]. The mentioned methods have been referred to as classical methods. Classical IRT methods only focus on excluding hotspots from backgrounds, but they do not classify them based on their IR sources.
For the detection of the idlers from their background, different methods based on deep learning are proposed. For instance, a CM method based on object detection, Hough line transform, and template matching algorithms are proposed by [36]. However, the presence of other thermal sources and objects with high emissivity values in complex backgrounds lead to hotspots with atypical geometries that the mentioned method could not address.
The weaknesses of the related works are grouped into three categories: the first category focuses only on the detection of hotspots; the second category uses stationary camera systems for CM of equipment; and the third category handles deep learning techniques but does not propose methods for classification of other thermal sources that can be wrongly identified as thermal defects. Information is presented in more detail in Table 1.

3. Materials and Methods

The main idea of the methodology is described in Figure 1. Firstly, the captured data by mobile robot including IR and RGB videos were loaded and prepared for preprocessing. Afterward, the total number of frames from both sources were extracted. Furthermore, to reduce the size and complexity of captured frame, we choose to convert the colored IR frames into 8-bit grayscale ones.
In some frames, due to the existence of different thermal sources that are not related, the segmentation algorithm cannot always precisely exclude hotspots (overheated idlers) due to oversegmentation. For improving the oversegmentation issue, we choose to preprocess the ROIs using different preprocessing techniques. In this study, intensity normalization, Contrast Limited Adaptive Histogram Equalization (CLAHE), and the Fast Non-Local Means Algorithm (FNLM) were used.
For segmenting overheated idlers in preprocessed frames, an outlier detection method based on the interquartile range (IQR) technique was used to calculate the optimal threshold value in each frame.
Based on the visual analysis of the segmented frames, we noticed that in some frames other thermal sources including sun reflection or other overheated equipment were wrongly segmented and recognized as overheated idlers. This can reduce the accuracy of the CM method. For the identification of hotspots in segmented frames, we proposed a classification method based on binary classification technique using a CNN. The different stages of the proposed methodology were developed in the Python programming language using OpenCV library for image processing and did not have major changes. Furthermore, the Keras and TensorFlow libraries were used for developing the proposed image classifier.

3.1. Region of Interest Estimation

Through the conducted experiments, the inspection mobile robot moved alongside the conveyor systems and captured different data, including IR and RGB videos. The mobile robot camera system was installed on a fixed hand above the robot and pointed toward the conveyor system. The mobile robot followed a straight line with constant speed; therefore, we did not have major changes in the camera system POV toward the conveyor system. The captured RGB and IR videos from the conveyor systems contain information on target pixels that are related to idlers and redundant areas. For improving the segmentation accuracy, we excluded the non-ROI areas from the original captured frames.
Through analyzing the extracted frames, we understand that idlers that are located on the right side of the conveyor (from mobile robot POV) were not always visible for being analyzed in extracted IR and RGB frames; therefore, for improving idler detection, we chose to define ROI on extracted frames for capturing idlers that were on the left side of the conveyors.
By observing neglectable changes in the camera system POV through the sequence of extracted frames, we could accurately predict the location of idlers. Different detection algorithms can be used for following the idlers in the sequence of the frame. In our work, we chose to consider a rectangular ROI with predetermined size: 200 × 200 pixels with a fixed position to extract ROIs from captured IR and RGB frames. The size of the extracted ROIs covers the idler sections, while it is considerably smaller than the originally captured frames Figure 2.

3.2. Pre-Processing of Region of Interests

The captured IR images are characterized by noise due to illumination and contrast variation; therefore, the use of automatic segmentation methods for detection of overheated idlers can become a challenging task. To address this issue, in this paper, intensity normalization, CLAHE, and FNLM were used for improving the overall quality of extracted IR images.

3.2.1. Normalization

The intensity value of each pixel in an original frame is predefined by the IR camera, considering the hottest and coldest point within the frame borders. For defining statistical parameters that work well for analyzing ROIs, we need to recalculate the pixels’ values with respect to the pixel distribution in defined ROIs. Normalization of the ROIs can reduce the seasonal differences and lets us define statistical parameters that work in ROIs with varying temperature ranges. The normalization was performed using the following equation:
N i = T S i T S min T S max T S min
In Equation (1), N i can be defined as the normalized value of pixel i, where T S i is defined as the intensity value of pixel i. Furthermore, the maximum and minimum values of pixel intensities within the ROIs are defined as T S m a x and T S m i n .

3.2.2. CLAHE Method

Histogram equalization (HE) is an elementary method to improve the contrast of underexposed and overexposed images. However, in the HE method, changes in natural brightness of the processed images can cause undesirable noise in the results [50]. Adaptive Histogram Equalization (AHE) is the improved version of the HE method [51]. In AHE, the input image is split into smaller areas, while the cumulative distributive function (CDF) is generated for each of these smaller images. In this method, the noise can considerably increase when the image histogram slope is steep. For addressing this issue, the CLAHE method is proposed. CLAHE is an improved version of the AHE that works in the same way. In the CLAHE method, the extracted histogram of the input image is clipped at specific values for limiting the amplification before computing the CDF, which can considerably reduce the amount of the unwanted noise in the final results. The calculation of CLAHE is performed as follows:
p = p max p min P ( f ) + p min
In Equation (2), the maximum and minimum intensity value of pixels in an image are defined by p max and p min , while P ( f ) defines the cumulative probability distribution function, and p represents the assigned value to each pixel after applying CLAHE.

3.2.3. Fast Non-Local Means Algorithm

The Non-Local Means Denoising Algorithm (NLM) is an effective method for reducing the unwanted noise in images Figure 3. As opposed to Gaussian, median, and Wiener filters, the NLM method uses the Euclidean distance as a weight; therefore, it can provide better results. The NLM denoising algorithm can be defined as follows:
N L [ I ] ( m ) = N I ω ( m , n ) I ( n )
In Equation (3), the weight: ω ( m , n ) can be expanded as follows:
ω ( m , n ) = 1 Z ( m ) e G σ ( τ ) I ( M + τ ) I ( n + τ ) 2 2 d 2
where τ describes the number of pixels in an image and G σ ( τ ) can be defined as Gaussian distribution of pixels with size σ 2 of the number of available pixels in the background. Furthermore, I ( M + τ ) I ( n + τ ) 2 2 describes the differences in intensities between adjacent pixels based on calculating the euclidean distance values. The leveling constant is set by Z(m):
Z ( m ) = n e G σ ( τ ) I ( m + τ ) I ( n + τ ) 2 2 d 2
The calculation of ω ( m , n ) in FNLM is modified from one dimension to two dimensions. The modified version of ω ( m , n ) can be defined as follows:
ω ( m , n ) = 1 Z ( m ) H i ( I ( m + s ) I ( m s ) )
In Equation (6), τ is defined based on n m , s is defined as m + τ , and H i can be describe as follows:
H i ( s ) = q = 0 s e I ( q ) I ( q + τ ) 2 2 d 2 .
By simplifying the process through one-dimensional computations, the FNLM method can denoise the image four times faster than the NLM method [52,53,54,55,56,57,58,59,60].

3.3. IR Image Segmentation

Image segmentation is a process of dividing images into regions that may be meaningful for extracting the desirable objects or constituent areas. Thresholding techniques are considered as the simplest way of performing image segmentation in IR images. Generally, in IR images, regions with high temperatures are correlated to the foreground; therefore, pixels with high intensity represent the heated objects. In the proposed method, the optimal threshold value for captured IR images can be defined based on analyzing the statistical features that are extracted from the IR images’ histogram.

3.3.1. Histogram Analysis of IR Images

An IR image histogram describes the tonal or color distribution of pixels. The tonal distribution of pixels in grayscaled IR images refers to discrete temperature values; therefore, the analysis of IR images’ histogram from a statistical point of view is a useful approach for detecting temperature anomalies in monitored equipment. Different statistical features such as: mean value, variance, and standard deviation can be extracted to describe the pixel tonal distribution in IR images. For a gray-leveled IR image, the first-order histogram probability P ( g ) is computed as follows [61]:
P ( g ) = L ( g ) M
In Equation (8), L ( g ) is the number of gray levels g, where the total number of pixels in a processed image is defined by M. In gray-scaled IR images, the total number of available levels for the pixel L span into [ 0 , 256 ] . The general brightness, or in other words average temperature, in a captured frame can be defined by mean value as follows:
g ¯ = g = 0 L 1 g · P ( g )
Furthermore, the dispersion of a set of data points around their mean can be defined by variance value as follows:
σ g 2 = g = 0 L 1 ( g g ¯ ) 2 · P ( g )
The standard deviation or the square root of the variance indicates the image contrast. It can be considered as an important factor for identifying the thermal anomaly, as analyzing temperature distribution in IR images is a key index of detection of possible defects.

3.3.2. Anomaly Detection

In this work, a qualitative-based method is proposed for identification of hotspots in preprocessed frames. In IR images with uniform background, the defected idlers can be recognized as hotspots. For separation of the hotspots from the background, thermal anomalies in idlers’ surfaces are treated as outliers. Outlier detection can be described as a problem of finding data that cannot be defined in the range of normal behavior. The interquartile range (IQR) can be used to extract histogram features that can help us to define the outliers.
The IQR defines the difference between the first and the third quartile: I Q R = Q 3 Q 1 where Q 1 and Q 3 can be calculated as follows [62,63]:
Q 1 = g ¯ 0.675 σ Q 3 = g ¯ + 0.675 σ
The outliers can be extracted and classified in two different groups, namely: mild and extreme outliers. The lower and upper bounds can be calculated as Q 1 3 I Q R and Q 3 + 3 I Q R . The values that are either among inner or outer bounds are classified as mild outliers; on the other hand, extreme outliers can be considered as values that are beyond outer bounds [64,65].
The optimal thresholder value can be defined as T in each frame, while α and β refer to the segmented frames and preprocessed ROIs, respectively. Furthermore, W and H can be considered as the processed ROI height and width, while x and y represent the pixel coordination.
β ( x , y ) = 1 if α ( x , y ) > T 0 if α ( x , y ) T 0 x < W , 0 y < H
To separate the hotspots from background in α , the pixel values were set to 1 when they were greater than computed T, otherwise they were set to 0:
γ ( x , y ) = α ( x , y ) if β ( x , y ) = 1 0 if β ( x , y ) = 0 0 x < W , 0 y < H
The pixels that are related to overheated idlers cannot always be defined as mild outliers. Furthermore, the definition of overheated idlers based on mild outliers can cause oversegmentation. For addressing the oversegmentation issue, we choose to extract extreme outlier values in extracted IR frames and consider them as the optimal threshold values for segmentation of possible hotspots, as shown in Figure 4.

3.4. Convolutional Neural Networks

During the visual inspection of the results, we found out that the proposed segmentation method could accurately segment the overheated idlers from the background in the majority of extracted frames. However, due to overestimation in some frames, other thermal sources were wrongly segmented as overheated idlers.
The objective of this section is to propose a method to accurately classify the segmentation results in two different classes considering the differences between hotpot shapes in segmented frames. The first class consists of frames where the segmentation method could accurately separate overheated idlers from the background, while the other class consists of hotspots that were not related to idlers.
A CNN is an ANN (artificial neural network) with different numbers of intermediate layers. The CNN models are able to automatically detect the specific features on each image and then train a classifier model on them [66,67]. CNNs performance in classification tasks can be improved when the size of the given dataset continues to increase. A developed CNN can learn to extract specific features from a given dataset and classify them at the same time.
Deep learning models that use CNN architecture are suitable for image processing classification problems. In CNNs, different layers in a network work as a detection filter for the recognition of specific patterns or features. The first layer of a CNN model can detect major features in an image that can be easily interpreted, while in middle layers, more abstract features can be detected. Finally, in the last layer, the collected details from previous layers are accurately merged and classified [68].

3.4.1. Data Collection and Description

Through the experiments, two conveyor systems that were located in an opencast mining site were inspected. The inspected conveyor systems were used for the continuous transportation of mineral materials from the mine pit to the bunker. The length of the investigated conveyor systems reached one hundred meters and is designed to operate in harsh environmental conditions. The opencast mining site in this study is located in Jaroszów, 50 km to the west of Wrocław. The length of the inspected parts of the conveyor systems were 150 m, where there was a 1.45 m space between each idler. Furthermore, the idlers’ diameter was 1.33 m and the belt length was 0.8 m.
A mobile inspection robot was used for data accusation Table 2. The inspection mobile robot is specially designed for Wrocław University of Science and Technology as a mobile platform that can conduct inspection missions in harsh environmental conditions, as shown in Figure 5.
The precision of an IRT method is directly related to environmental parameters and the IR camera specifications Table 3. The parameters such as environmental conditions and emissivity values of the objects should be taken into account for accurate identification of overheated equipment [33]. In our case study, one can notice that in a raw IR image there are different radiation sources that can be wrongly identified as overheated idlers Figure 6.
In opencast mining sites, solar radiation can warm up the conveyor system surfaces, especially modules that can absorb the considerable amount of sun energy. Our experiment was conducted on a sunny day. As long as we organized the measurement session a few weeks before the experiment, it was almost impossible to predict the weather. However, in the proposed method, we tried to provide a solution that can be applicable in different environmental conditions. Furthermore, in our case, the solar radiation was mostly blocked by ceiling; therefore, their affect on the idler surface temperature was negligible.
Examples presented here show the application of IRT in industrial data sets is always challenging. As a result, the classical image segmentation techniques cannot provide accurate detection results due to instances of different thermal sources in mining areas, as shown in Figure 7. prances of different thermal

3.4.2. Binary Classification

In this work, we developed a binary classification model for classification of the segmented hotspots to identify frames where overheated idlers are correctly segmented Figure 8.
The proposed classification model includes three blocks, where each block is composed of a convolution layer with a 3 × 3 filter, followed by a maximum grouping layer. The ReLU activation function was used in each conventional layer. The depths of our architecture for the three blocks are 32, 32, and 64 respectively. A value of zero (0) or one (1) should be predicted for each input image, as we used a binary classification. The sigmoid type of activation was used to regularize the deep neural network at the final output layer. Regularization simulates the different possible changes on the structure of the network and increases the strength of the nodes in the developed network. Finally, for compiling the data, the binary cross-entropy method was used as a loss function, and the Adam method was used as the optimizer to form a classifier on the image data sets Figure 9.

3.5. Training and Testing the CNN

Three different data sets, which consist of segmented IR images captured from two different conveyor systems, were used for training the CNN model. Through the experiments, the inspection robot moved alongside conveyor system number one (back and forth) and conveyor system number two (forth). As a result, three different data sets, 6275 frames from conveyor one (moving forward), 6135 frames from conveyor 2 (moving backward), and 10896 frames from conveyor two (moving forward), were selected and underwent preprocessing stages and segmentation. Afterward, the segmented frames were filtered based on the size of the segmented hotspots; therefore, in each segmented frame, the number of non-black pixels was calculated, and frames that the number of non-black pixels was above 400 pixels were kept for further analysis.
The classifier was trained using images labeled by experts. We labeled each segmented image as “contained an overheated idler” or “other thermal sources”, as shown in Figure 10. Furthermore, the filtered and labeled frames from the mentioned data sets are combined together for training the CNN model. A total of 1216 frames from the three different data sets was selected for the training process. The images are randomly split, and 20% of the total images were used for testing images and the remainder as training images.

4. Evaluation

For evaluation of the proposed classifier, we computed the following performance metrics: sensitivity, precision, accuracy, and F1-score.
Accuracy = ( T P + T N ) ( T P + F N ) + ( F P + T N )
Sensitivity = ( T P ) ( T P + F N )
Precision = ( T P ) ( T P + F P )
F 1 Score = ( 2 P r e c i s i o n S e n s i t i v i t y ) ( P r e c i s i o n + S e n s i t i v i t y )
Here, accuracy is the proportion of correctly classified overheated idlers among the whole population. Sensitivity is measured as the proportion of true positive cases that are correctly predicted by the classifier, while specificity is the prediction of true negative cases that are correctly predicted. Precision is the proportion of the correct predictions in the confusion matrix out of all positive predictions. Furthermore, the F1-score is the harmonic mean of precision and sensitivity. The coefficient takes into account the TP (true positive), TN (true negative), FP (false positive), and FN (false negative) factors for scoring the model. The ideal value of these metrics is 1 and is the target for the models in this study.

5. Results

In this section, we evaluate the performance of the proposed method in the detection of overheated idlers. Furthermore, the performance of the proposed method is compared with our previous research using test data randomly chosen from 20% of the whole data set [24].
The confusion matrix is used to display the results of classifications and generate all of the performance metrics. Our binary classification model could accurately classify the segmentation results in over 97% of studied cases, as shown in Figure 11.
The training and validation loss graph of the trained model is shown in Figure 12. The loss of validation data significantly decreased until the 6th epoch, but on the next epoch, the trends tended to increase. This means that our model converged fast until the 6th epoch, and the greater epoch would not give any significant change on the classification result.
The training and validation accuracy curves are shown in Figure 13. The curves indicate that through each training epoch the accuracy parameter varies and then continuously converges to a certain accuracy level with a lot of small oscillation. The trends also show that the classifier did not affect by significant overfitting or underfitting.
The receiver operating characteristic curve (ROC) for a binary classification model is shown in Figure 14. ROC is a receiver operating curve, which is one of the model’s measuring metrics. The area under curve can be defined as the AUC that indicates the performance of the model. It is a graph that shows how the true positive rate (TPR) and the false positive rate (FPR) are related. For the studied binary classification model, the ROC was 100%.
Table 4 compares the performance metrics of the proposed method and our previous work in the identification of overheated idlers in captured IR frames. The precision value shows that the binary classification method has few errors. In this table, it is shown how the shape-based classification of segmented hotspots affects the accuracy metrics. It is demonstrated that the false identification of other thermal sources as overheated idlers in our previous research had a major effect on the performance of the CM method. Reducing the portion of false negatives by more than 54% led to an increase in the F1 score value from 0.62 to 0.97 in this work.

6. Conclusions

In this paper, we solve the problems related to the identification of overheated idlers in IR images with complex backgrounds. We demonstrate that the identification of overheated idlers in IR images captured in real case scenarios is a challenging task. For the proposed methodology, the following points were covered: Firstly, in Section 3.2.2, we replaced our previous noise reduction method with the FNLM method, which could help us to reduce the amount of presented noise more efficiently. Secondly, the proposed binary classification methodology allows the detection of frames that other thermal sources wrongly segmented as overheated idlers.
The value of the accuracy metrics in Table 4 for the proposed method is greater than 0.9795, which implies that the discussed binary classification method is reliable, confirmed by the precision value being greater than 0.9740.
Future work assumes the improvement of the CM method in terms of adaptability to different conveyor systems that are located in different mining sites. Furthermore, our team is interested in working on methods that can be used for the detection and localization of overheated idlers in global navigation satellite system denied environments, such as deep underground mines, by fusion of LIDAR and IR images captured by an inspection mobile robot.

Author Contributions

Conceptualization, R.Z. and M.S.; methodology, M.S.; software, M.S.; validation, R.Z., J.W. and T.B.; formal analysis, M.S.; investigation, M.S. and J.W.; resources, R.Z. and J.W.; data curation, J.W. and M.S.; writing original draft preparation, M.S.; writing review and editing, M.S., J.W., R.Z. and T.B.; visualization, M.S.; supervision, R.Z. and T.B.; project administration, R.Z.; funding acquisition, R.Z. All authors have read and agreed to the published version of the manuscript.


This work was supported by the European Commission via the Marie Sklodowska Curie program through the ETN MOIRA project (GA 955681)—Mohammad Siami. This activity has received funding from the European Institute of Innovation and Technology (EIT), a body of the European Union, under the Horizon 2020, the EU Framework Program for Research and Innovation. This work is supported by EIT RawMaterials GmbH under Framework Partnership Agreement No. 19018 (AMICOS. Autonomous Monitoring and Control System for Mining Plants). Scientific work published within the framework of an international project co-financed from the funds of the program of the Minister of Science and Higher Education titled “PMW” 2020–2021; contract no. 5163/KAVA/2020/2021/2.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Archived data sets cannot be accessed publicly according to the NDA agreement signed by the authors.


The authors (M. Siami) gratefully acknowledge the European Commission for its support of the Marie Sklodowska Curie program through the ETN MOIRA project (GA 955681).

Conflicts of Interest

The authors declare no conflict of interest.


  1. Zimroz, R.; Hutter, M.; Mistry, M.; Stefaniak, P.; Walas, K.; Wodecki, J. Why Should Inspection Robots be used in Deep Underground Mines? In Proceedings of the 27th International Symposium on Mine Planning and Equipment Selection, Las Condes, Chile, 19–23 November 2018; Selection-MPES 2018. Widzyk-Capehart, E., Hekmat, A., Singhal, R., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 497–507. [Google Scholar] [CrossRef]
  2. Zimroz, R.; Hardygóra, M.; Blazej, R. Maintenance of Belt Conveyor Systems in Poland—An Overview. In Proceedings of the 12th International Symposium Continuous Surface, Aachen, Germany, 1 October 2014; Niemann-Delius, C., Ed.; Springer International Publishing: Cham, Switzerland, 2015; pp. 21–30. [Google Scholar] [CrossRef]
  3. Bołoz, Ł.; Biały, W. Automation and Robotization of Underground Mining in Poland. Appl. Sci. 2020, 10, 7221. [Google Scholar] [CrossRef]
  4. Trybała, P.; Blachowski, J.; Błażej, R.; Zimroz, R. Damage detection based on 3d point cloud data processing from laser scanning of conveyor belt surface. Remote Sens. 2021, 13, 55. [Google Scholar] [CrossRef]
  5. Błażej, R.; Kirjanów, A.; Kozłowski, T. A high resolution system for automatic diagnosing the condition of the core of conveyor belts with steel cords. Diagnostyka 2014, 15, 41–45. [Google Scholar]
  6. Zimroz, R.; Król, R. Failure analysis of belt conveyor systems for condition monitoring purposes. Min. Sci. 2009, 128, 255–270. [Google Scholar]
  7. Król, R.; Kisielewski, W. Research of loading carrying idlers used in belt conveyor-practical applications. Diagnostyka 2014, 15, 67–74. [Google Scholar]
  8. Peruń, G.; Opasiak, T. Assessment of technical state of the belt conveyor rollers with use vibroacoustics methods—Preliminary studies. Diagnostyka 2016, 17, 75–81. [Google Scholar]
  9. Dąbek, P.; Krot, P.; Wodecki, J.; Zimroz, P.; Szrek, J.; Zimroz, R. Measurement of idlers rotation speed in belt conveyors based on image data analysis for diagnostic purposes. Measurement 2022, 202, 111869. [Google Scholar] [CrossRef]
  10. Bortnowski, P.; Król, R.; Nowak-Szpak, A.; Ozdoba, M. A Preliminary Studies of the Impact of a Conveyor Belt on the Noise Emission. Sustainability 2022, 14, 2785. [Google Scholar] [CrossRef]
  11. Yardley, E.; Stace, L. 4-Design of belt conveyors 2-hardware (Idlers, structure, pulleys, drives, tensioning devices, transfer points and belt cleaning). In Belt Conveying of Minerals; Woodhead Publishing Series in Metals and Surface Engineering; Yardley, E., Stace, L., Eds.; Woodhead Publishing: Sawston, Cambridge, UK, 2008; pp. 44–70. [Google Scholar] [CrossRef]
  12. Król, R. Studies of the Durability of Belt Conveyor Idlers with Working Loads Taken into Account. IOP Conf. Ser. Earth Environ. Sci. 2017, 95, 42054. [Google Scholar] [CrossRef] [Green Version]
  13. Zakharov, A.; Gerike, B.; Shiryamov, D. The Ultimate Rotating Resistance of the Belt Conveyors Rollers. In Proceedings of the 9th China-Russia Symposium “Coal in the 21st Century: Mining, Intelligent Equipment and Environment Protection”, Qingdao, China, 18–21 October 2018; Atlantis Press: Qingdao, China, 2018; pp. 226–229. [Google Scholar] [CrossRef] [Green Version]
  14. Shiri, H.; Wodecki, J.; Ziȩtek, B.; Zimroz, R. Inspection robotic UGV platform and the procedure for an acoustic signal-based fault detection in belt conveyor idler. Energies 2021, 14, 7646. [Google Scholar] [CrossRef]
  15. Grzesiek, A.; Zimroz, R.; Śliwiński, P.; Gomolla, N.; Wyłomańska, A. Long term belt conveyor gearbox temperature data analysis—Statistical tests for anomaly detection. Meas. J. Int. Meas. Confed. 2020, 165, 108124. [Google Scholar] [CrossRef]
  16. Dabek, P.; Szrek, J.; Zimroz, R.; Wodecki, J. An Automatic Procedure for Overheated Idler Detection in Belt Conveyors Using Fusion of Infrared and RGB Images Acquired during UGV Robot Inspection. Energies 2022, 15, 601. [Google Scholar] [CrossRef]
  17. Szrek, J.; Jakubiak, J.; Zimroz, R. A Mobile Robot-Based System for Automatic Inspection of Belt Conveyors in Mining Industry. Energies 2022, 15, 327. [Google Scholar] [CrossRef]
  18. Carvalho, R.; Nascimento, R.; D’angelo, T.; Delabrida, S.; Bianchi, A.G.; Oliveira, R.A.; Azpúrua, H.; Garcia, L.G. A UAV-based framework for semi-automated thermographic inspection of belt conveyors in the mining industry. Sensors 2020, 20, 2243. [Google Scholar] [CrossRef] [Green Version]
  19. Yang, W.; Zhang, X.; Ma, H. An inspection robot using infrared thermography for belt conveyor. In Proceedings of the 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence, URAI 2016, Xi’an, China, 19–22 August 2016; pp. 400–404. [Google Scholar] [CrossRef]
  20. Szrek, J.; Wodecki, J.; Błazej, R.; Zimroz, R. An inspection robot for belt conveyor maintenance in underground mine-infrared thermography for overheated idlers detection. Appl. Sci. 2020, 10, 4984. [Google Scholar] [CrossRef]
  21. Błazej, R.; Sawicki, M.; Kirjanów, A.; Kozłowski, T.; Konieczna, M. Automatic analysis of thermograms as a means for estimating technical of a gear system. Diagnostyka 2016, 17, 43–48. [Google Scholar]
  22. Qu, Z.; Jiang, P.; Zhang, W. Development and application of infrared thermography non-destructive testing techniques. Sensors 2020, 20, 3851. [Google Scholar] [CrossRef]
  23. Abdel-Qader, I.; Yohali, S.; Abudayyeh, O.; Yehia, S. Segmentation of thermal images for non-destructive evaluation of bridge decks. NDT E Int. 2008, 41, 395–405. [Google Scholar] [CrossRef]
  24. Siami, M.; Barszcz, T.; Wodecki, J.; Zimroz, R. Design of an Infrared Image Processing Pipeline for Robotic Inspection of Conveyor Systems in Opencast Mining Sites. Energies 2022, 15, 6771. [Google Scholar] [CrossRef]
  25. Hurley, T.J. Infrared qualitative and quantitative inspections for electric utilities. In Thermosense XII: An International Conference on Thermal Sensing and Imaging Diagnostic Applications; Semanovich, S.A., Ed.; International Society for Optics and Photonics, SPIE: Orlando, FL, USA, 1990; Volume 1313, pp. 6–24. [Google Scholar] [CrossRef]
  26. Griffith, B.; Türler, D.; Goudey, H. IR Thermographic Systems: A Review of IR Imagers and Their Use; Lawrence Berkeley National Laboratory: Berkeley, CA, USA, 2001. [Google Scholar]
  27. Wurzbach, R.N.; Hammaker, R.G. Role of comparative and qualitative thermography in predictive maintenance. In Thermosense XIV: An International Conference on Thermal Sensing and Imaging Diagnostic Applications; SPIE: Orlando, FL, USA, 1992; Volume 1682, pp. 3–11. [Google Scholar]
  28. Jadin, M.S.; Taib, S.; Kabir, S.; Yusof, M.A.B. Image processing methods for evaluating infrared thermographic image of electrical equipments. In Proceedings of the Progress in Electromagnetics Research Symposium, Suzhou, China, 12–16 September 2011. [Google Scholar]
  29. Prasad, B.S.; Prabha, K.A.; Kumar, P.G. Condition monitoring of turning process using infrared thermography technique—An experimental approach. Infrared Phys. Technol. 2017, 81, 137–147. [Google Scholar] [CrossRef]
  30. Cong-ping, C.; Wu, Q.; Zi-fan, F.; Yi, Z. Infrared image transition region extraction and segmentation based on local definition cluster complexity. In Proceedings of the 2010 International Conference on Computer Application and System Modeling (ICCASM 2010), Taiyuan, China, 22–24 October 2010; Volume 3, pp. V3-50–V3-54. [Google Scholar] [CrossRef]
  31. Li, Y.; Mao, X. An efficient method for target extraction of infrared images. In International Conference on Artificial Intelligence and Computational Intelligence; Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in, Bioinformatics); Wang, F.L., Deng, H., Gao, Y., Lei, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2010; Volume 6319, pp. 185–192. [Google Scholar] [CrossRef]
  32. Fan, S.; Yang, S.; He, P.; Nie, H. Infrared Electric Image Thresholding Using Two-Dimensional Fuzzy Renyi Entropy. Energy Procedia 2011, 12, 411–419. [Google Scholar] [CrossRef] [Green Version]
  33. Harrap, M.J.; De Ibarra, N.H.; Whitney, H.M.; Rands, S.A. Reporting of thermography parameters in biology: A systematic review of thermal imaging literature. R. Soc. Open Sci. 2018, 5, 181281. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Nascimento, R.; Carvalho, R.; Delabrida, S.; Bianchi, A.G.; Oliveira, R.A.R.; Garcia, L.G. An integrated inspection system for belt conveyor rollers advancing in an enterprise architecture. In Proceedings of the 19th International Conference on Enterprise Information Systems, ICEIS 2017, Porto, Portugal, 26–29 April 2017; Volume 2, pp. 190–200. [Google Scholar] [CrossRef]
  35. Opasiak, T.; Peruń, G. Influence of construction the rollers C type on resistance of rotating driven system of the belt conveyor. Diagnostyka 2016, 17, 81–86. [Google Scholar]
  36. Liu, Y.; Miao, C.; Li, X.; Ji, J.; Meng, D. Research on the fault analysis method of belt conveyor idlers based on sound and thermal infrared image features. Meas. J. Int. Meas. Confed. 2021, 186, 110177. [Google Scholar] [CrossRef]
  37. Liu, T.; Liu, S. The impacts of coal dust on miners’ health: A review. Environ. Res. 2020, 190, 109849. [Google Scholar] [CrossRef] [PubMed]
  38. Bortnowski, P.; Nowak-Szpak, A.; Król, R.; Ozdoba, M. Analysis and Distribution of Conveyor Belt Noise Sources under Laboratory Conditions. Sustainability 2021, 13, 2233. [Google Scholar] [CrossRef]
  39. Morales, A.S.; Aqueveque, P.; Henriquez, J.A.; Saavedra, F.; Wiechmann, E.P. A technology review of idler condition based monitoring systems for critical overland conveyors in open-pit mining applications. In Proceedings of the 2017 IEEE Industry Applications Society Annual Meeting, Cincinnati, OH, USA, 1–5 October 2017; pp. 1–8. [Google Scholar] [CrossRef]
  40. Konieczna, M.; Gładysiewicz, L. Theoretical basis for determining rolling resistance of belt conveyors. Min. Sci. 2016, 23, 105–119. [Google Scholar] [CrossRef]
  41. Bortnowski, P.; Kawalec, W.; Król, R.; Ozdoba, M. Types and causes of damage to the conveyor belt—Review, classification and mutual relations. Eng. Fail. Anal. 2022, 140, 106520. [Google Scholar] [CrossRef]
  42. Zhu, L.; Spachos, P.; Pensini, E.; Plataniotis, K.N. Deep learning and machine vision for food processing: A survey. Curr. Res. Food Sci. 2021, 4, 233–249. [Google Scholar] [CrossRef]
  43. Meola, C. Infrared Thermography: Recent Advances and Future Trends; Bentham Science Publishers: Sharjah, United Arab Emirates, 2012. [Google Scholar] [CrossRef]
  44. Szrek, J.; Zimroz, R.; Wodecki, J.; Michalak, A.; Góralczyk, M.; Worsa-Kozak, M. Application of the infrared thermography and unmanned ground vehicle for rescue action support in underground mine—The amicos project. Remote Sens. 2021, 13, 69. [Google Scholar] [CrossRef]
  45. Zimroz, P.; Trybała, P.; Wróblewski, A.; Góralczyk, M.; Szrek, J.; Wójcik, A.; Zimroz, R. Application of UAV in search and rescue actions in underground mine—A specific sound detection in noisy acoustic signal. Energies 2021, 14, 3725. [Google Scholar] [CrossRef]
  46. Uth, F.; Polnik, B.; Kurpiel, W.; Kriegsch, P.; Baltes, R.; Clausen, E. An innovative person detection system based on thermal imaging cameras dedicate for underground belt conveyors. Min. Sci. 2019, 26, 263–276. [Google Scholar] [CrossRef]
  47. Stypułkowski, K.; Gołda, P.; Lewczuk, K.; Tomaszewska, J. Monitoring System for Railway Infrastructure Elements Based on Thermal Imaging Analysis. Sensors 2021, 21, 3819. [Google Scholar] [CrossRef]
  48. Shao, H.; Xia, M.; Han, G.; Zhang, Y.; Wan, J. Intelligent Fault Diagnosis of Rotor-Bearing System Under Varying Working Conditions With Modified Transfer Convolutional Neural Network and Thermal Images. IEEE Trans. Ind. Inform. 2021, 17, 3488–3496. [Google Scholar] [CrossRef]
  49. Szurgacz, D.; Zhironkin, S.; Vöth, S.; Pokorný, J.; Sam Spearing, A.J.; Cehlár, M.; Stempniak, M.; Sobik, L. Thermal imaging study to determine the operational condition of a conveyor belt drive system structure. Energies 2021, 14, 3258. [Google Scholar] [CrossRef]
  50. Pizer, S.M.; Amburn, E.P.; Austin, J.D.; Cromartie, R.; Geselowitz, A.; Greer, T.; ter Haar Romeny, B.; Zimmerman, J.B.; Zuiderveld, K. Adaptive Histogram Equalization and Its Variations. Comput. Vis. Graph. Image Process. 1987, 39, 355–368. [Google Scholar] [CrossRef]
  51. Hummel, R. Image Enhancement By Histogram Transformation. Comput. Graph. Image Process. 1977, 6, 184–185. [Google Scholar] [CrossRef]
  52. Choi, W.H.; Choi, H.R.; Seo, E.; Hwang, J.; Oh, H.; Kim, M.R.; Han, S.R.; Kim, M.S.; Kang, S.H.; Lee, Y. Feasibility of fast non-local means noise reduction algorithm in magnetic resonance imaging using 1.5 and 3.0 T with diffusion-weighted image technique. Optik 2019, 183, 241–246. [Google Scholar] [CrossRef]
  53. Buades, A.; Coll, B.; Morel, J.M. A non-local algorithm for image denoising. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–26 June 2005; Volume 2, pp. 60–65. [Google Scholar] [CrossRef]
  54. Naegel, B.; Cernicanu, A.; Hyacinthe, J.N.; Tognolini, M.; Vallée, J.P. SNR enhancement of highly-accelerated real-time cardiac MRI acquisitions based on non-local means algorithm. Med. Image Anal. 2009, 13, 598–608. [Google Scholar] [CrossRef]
  55. Lee, S.; Jin Park, S.; Jeon, J.M.; Lee, M.H.; Ryu, D.Y.; Lee, E.; Kang, S.H.; Lee, Y. Noise removal in medical mammography images using fast non-local means denoising algorithm for early breast cancer detection: A phantom study. Optik 2019, 180, 569–575. [Google Scholar] [CrossRef]
  56. Coupé, P.; Yger, P.; Barillot, C. Fast Non Local Means Denoising for 3D MR Images. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2006; Larsen, R., Nielsen, M., Sporring, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2006; pp. 33–40. [Google Scholar]
  57. Dauwe, A.; Goossens, B.; Luong, H.Q.; Philips, W. A fast non-local image denoising algorithm. In Image Processing: Algorithms and Systems VI; SPIE: Orlando, FL, USA, 2008; Volume 6812, pp. 324–331. [Google Scholar] [CrossRef]
  58. Shim, J.; Yoon, M.; Lee, Y. Feasibility of newly designed fast non local means (FNLM)-based noise reduction filter for X-ray imaging: A simulation study. Optik 2018, 160, 124–130. [Google Scholar] [CrossRef]
  59. Huang, K.; Zhang, D.; Wang, K. Non-local means denoising algorithm accelerated by GPU. In MIPPR 2009: Medical Imaging, Parallel Processing of Images, and Optimization Techniques; SPIE: Orlando, FL, USA, 2009; Volume 7497, pp. 263–270. [Google Scholar] [CrossRef]
  60. Kim, B.G.; Kang, S.H.; Park, C.R.; Jeong, H.W.; Lee, Y. Noise Level and Similarity Analysis for Computed Tomographic Thoracic Image with Fast Non-Local Means Denoising Algorithm. Appl. Sci. 2020, 10, 7455. [Google Scholar] [CrossRef]
  61. Younus, A.M.; Widodo, A.; Yang, B.S. Image histogram features based thermal image retrieval to pattern recognition of machine condition. In Engineering Asset Lifecycle Management, Proceedings of the 4th World Congress on Engineering Asset Management, WCEAM 2009, Athens, Greece, 28–30 September 2009; Springer: Athens, Greece, 2009; pp. 943–949. [Google Scholar] [CrossRef]
  62. Le, T.M.; Vo, T.M.; Pham, T.N.; Dao, S.V.T. A Novel Wrapper-Based Feature Selection for Early Diabetes Prediction Enhanced with a Metaheuristic. IEEE Access 2021, 9, 7869–7884. [Google Scholar] [CrossRef]
  63. Bergamasco, M.; Della Rossa, F.; Piroddi, L. Active noise control of impulsive noise with selective outlier elimination. In Proceedings of the American Control Conference, Washington, DC, USA, 17–19 June 2013; pp. 4165–4170. [Google Scholar] [CrossRef]
  64. Sasmal, S.; Chowdhury, S.; Kundu, S.; Politis, D.Z.; Potirakis, S.M.; Balasis, G.; Hayakawa, M.; Chakrabarti, S.K. Pre-seismic irregularities during the 2020 samos (Greece) earthquake (m = 6.9) as investigated from multi-parameter approach by ground and space-based techniques. Atmosphere 2021, 12, 1059. [Google Scholar] [CrossRef]
  65. Pincus, R.; Barnett, V.; Lewis, T. Outliers in Statistical Data. 3rd edition. J. Wiley & Sons 1994, XVII. 582 pp., £49.95. Biom. J. 1995, 37, 256. [Google Scholar] [CrossRef]
  66. Li, Z.; Liu, F.; Yang, W.; Peng, S.; Zhou, J. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Trans. Neural Netw. Learn. Syst. 2021, 33, 6999–7019. [Google Scholar] [CrossRef]
  67. Nishani, E.; Çiço, B. Computer vision approaches based on deep learning and neural networks: Deep neural networks for video analysis of human pose estimation. In Proceedings of the 2017 6th Mediterranean Conference on Embedded Computing (MECO), Bar, Montenegro, 11–15 June 2017; pp. 1–4. [Google Scholar] [CrossRef]
  68. Voulodimos, A.; Doulamis, N.; Bebis, G.; Stathaki, T. Recent developments in deep learning for engineering applications. Comput. Intell. Neurosci. 2018, 2018, 8141259. [Google Scholar] [CrossRef]
Figure 1. The proposed procedure flowchart.
Figure 1. The proposed procedure flowchart.
Sensors 22 10004 g001
Figure 2. The location of a predefined ROI on an original frame.
Figure 2. The location of a predefined ROI on an original frame.
Sensors 22 10004 g002
Figure 3. Comparison of a preprocessed and an original IR image after modifications through preprocessing stages.
Figure 3. Comparison of a preprocessed and an original IR image after modifications through preprocessing stages.
Sensors 22 10004 g003
Figure 4. Comparison of detected outliers by mid and extreme values in a preprocessed ROI.
Figure 4. Comparison of detected outliers by mid and extreme values in a preprocessed ROI.
Sensors 22 10004 g004
Figure 5. (a) Mobile robot during inspection. (b) A general picture of the mining site.
Figure 5. (a) Mobile robot during inspection. (b) A general picture of the mining site.
Sensors 22 10004 g005
Figure 6. Examples of IR sources that are not related to idlers.
Figure 6. Examples of IR sources that are not related to idlers.
Sensors 22 10004 g006
Figure 7. Segmentation results of overheated objects that were not related to idlers.
Figure 7. Segmentation results of overheated objects that were not related to idlers.
Sensors 22 10004 g007
Figure 8. Simplified flowcharts of binary classification procedure.
Figure 8. Simplified flowcharts of binary classification procedure.
Sensors 22 10004 g008
Figure 9. Summary of the proposed CNN architecture.
Figure 9. Summary of the proposed CNN architecture.
Sensors 22 10004 g009
Figure 10. Image annotation process base on fusion of IR and RGB images. (a) Segmentation of other thermal sources; (b) segmentation of an overheated idler.
Figure 10. Image annotation process base on fusion of IR and RGB images. (a) Segmentation of other thermal sources; (b) segmentation of an overheated idler.
Sensors 22 10004 g010
Figure 11. Confusion matrix of the binary classification.
Figure 11. Confusion matrix of the binary classification.
Sensors 22 10004 g011
Figure 12. Training and validation loss plot for the binary classification model.
Figure 12. Training and validation loss plot for the binary classification model.
Sensors 22 10004 g012
Figure 13. Accuracy plot for the binary classification model.
Figure 13. Accuracy plot for the binary classification model.
Sensors 22 10004 g013
Figure 14. ROC Curves of the binary classification model.
Figure 14. ROC Curves of the binary classification model.
Sensors 22 10004 g014
Table 1. Comparison between research that proposed different IRT methods for identifying overheated idlers in conveyor systems.
Table 1. Comparison between research that proposed different IRT methods for identifying overheated idlers in conveyor systems.
Dabek et al. [16]Identifies hotspots in captured IR videos based on Canny edge and blob detection methods.Lacks a method to classify segmented hotspots. It does not propose a solution for frames with complex backgrounds.
Siami et al. [24]Proposed an image processing pipelines for identification of hotspots based on outlier detection method.It does not present a solution for classification of segmented hotspots based on their IR sources.
Szurgacz et al. [49]Proposed a CM method based on simple thresholding technique.Tested on limited number of IR images. IR images were captured manually by inspectors.
Liu et al. [36]Proposed a method for identification hotspots in IR images based on Hough line transform and template matching algorithm.Tested on limited number of IR images on a controlled environment. The data were captured by stationary camera systems.
Table 2. The main characteristics of the mobile inspection robot.
Table 2. The main characteristics of the mobile inspection robot.
Locomotion typeWheeled, skid steering
Navigation systemsAutonomous (internal computer)
Manual (pilot using remote computer connection)
Internal operating softwareRobot Operating System (ROS)
Power systemInternal battery, 24 V
Max gross weight140 kg
Maximum payload capacity75 kg
Table 3. The IR camera system specification.
Table 3. The IR camera system specification.
Resolution640 × 480 pixles
Frames per second25 fps
Observation angle45
Mounting height100 cm above shelf
Table 4. Comparison of the precision, recall, and F1 score metrics for overheated idler detection in IR images using binary classification and Siami et al.’s work [24].
Table 4. Comparison of the precision, recall, and F1 score metrics for overheated idler detection in IR images using binary classification and Siami et al.’s work [24].
MeasuresSiami et al.Siami et al. + FNLM + Binary ClassificationImprovement
True Positive112112
False Positive1323
False Negative02
Sensitivity1 0.9825 1.75 %
Precision 0.4590 0.9740 51.50 %
Accuracy 0.4590 0.9795 52.05 %
F1 Score 0.6292 0.9782 34.90 %
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Siami, M.; Barszcz, T.; Wodecki, J.; Zimroz, R. Automated Identification of Overheated Belt Conveyor Idlers in Thermal Images with Complex Backgrounds Using Binary Classification with CNN. Sensors 2022, 22, 10004.

AMA Style

Siami M, Barszcz T, Wodecki J, Zimroz R. Automated Identification of Overheated Belt Conveyor Idlers in Thermal Images with Complex Backgrounds Using Binary Classification with CNN. Sensors. 2022; 22(24):10004.

Chicago/Turabian Style

Siami, Mohammad, Tomasz Barszcz, Jacek Wodecki, and Radoslaw Zimroz. 2022. "Automated Identification of Overheated Belt Conveyor Idlers in Thermal Images with Complex Backgrounds Using Binary Classification with CNN" Sensors 22, no. 24: 10004.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop