Coal Mine Personnel Safety Monitoring Technology Based on Uncooled Infrared Focal Plane Technology

Abstract

In an effort to overcome the difficulty of real-time early warning via traditional infrared imaging technology caused by the complex working environment in coal mines, this paper proposes a mine early warning method based on uncooled infrared focal plane technology. The infrared thermal spectrogram of the detected object was visually displayed in a pseudo-color image with high resolution and high sensitivity, which can realize the real-time detection and early warning of personnel safety in modern mines. The multipoint compression correction algorithm based on human visual characteristics divided the response units of all acquisition units into gray intervals according to a threshold value, then the corresponding parameters were set in different intervals, and finally, each interval was compressed using a two-point correction algorithm. The volume of stored data was the sum of the calibration curve and the data from an encode table corrected by a MATLAB simulation, and the number of CPU cycles was run by a CCS 3.3 clock calculation algorithm. The results showed that when the temperature of the blackbody reached 115 °C, the nonuniformity before correction was 6.32%, and the nonuniformity after the multipoint correction of human eyes was 2.99%, which implied that the algorithm proposed in this paper had good denoising ability. The number of CPU cycles occupied by this algorithm was 18,257,363 cycles/frame with a frequency of 29.97 Hz. The sharpness of the compressed infrared images was obviously improved, and the uniformity was better. The method proposed in this paper can meet the need for modern mine personnel search and rescue, equipment supervision and dangerous area detection and other early warning requirements so as to achieve the goal of developing smart mines and ensuring safety in coal mine production.

1. Introduction

With increasing national requirements for workplace safety, coal mine enterprises have successively installed safety monitoring systems. However, traditional workplace safety monitoring systems can only issue a failure alarm when one or several indicators within the monitoring scope exceed the established standards rather than predict the incoming danger and issue a warning in advance [1,2,3]. Therefore, to realize the goal of developing smart mines and ensuring the safety of coal mine production, it is becoming more and more important to find a portable and automatic way to monitor the dangerous areas and equipment status in a coal mine in real time. Zhu et al. [4] proposed a mine equipment monitoring and early warning scheme based on wireless sensing, but this method did not integrate multiple signal types, leading to a low utilization rate of the information and the system. Xie et al. [5] used optical fiber sensing and infrared image technology to predict accidents in a mine, however, it could not meet the early warning requirements inside the complex and changeable working environment of a mine. Xu et al. [6] devised a mine disaster early warning model with the help of big data. The system architecture of most domestic enterprises has many defects such as poor scalability, low resource utilization, low environmental adaptability, high cost and poor energy-saving efficiency when processing a large amount of data. Sun et al. [7] put forward an early warning method in mines by combining visible light and infrared image. However, it was difficult to obtain a real-time and effective early warning in practice because the alarm would only be sent out after a real-time comparison between the visible light detection results and the infrared image detection results [8,9,10].

Infrared imaging technology has good anti-interference ability and allows target detection in complex weather without contact. At present, the most effective way to improve the system sensitivity and image resolution is the application of uncooled infrared imaging system and image processing technology, because the system can improve the sensitivity by shortening the detector response time, while reducing the interference of nonuniformity in the infrared images with the image preprocessing technology. At present, mainstream nonuniformity correction algorithms [11] are scaling-based and scene-based nonuniformity correction algorithms. The former is widely used in practice because its low algorithm complexity makes it possible to run on a hardware platform, while the latter cannot due to its high algorithm complexity. At present, such algorithms mainly include neural network algorithm [12,13,14,15], time-domain high-pass filtering method [16,17], Kalman filtering method [18] and other algorithms. Qian et al. [19] proposed a pseudo-color fusion algorithm of infrared and visible light image based on improved color delivery strategy and nonsubsampled contourlet transform (NSCT) and obtained a gray scale fusion image using NSCT and a Canny edge detection algorithm. Then, the fusion grayscale image was inserted into the Y channel, and the residual image between the source image and the fusion grayscale image was inserted into the Cb and Cr channels, respectively, to generate the YCbCr source color image [20,21,22,23]. Li et al. proposed a new method of image coloring fusion in the oRGB color space to enhance infrared thermal and visible light night vision images [24]. To sum up, in the development of infrared imaging technologies, in addition to the continuous research and performance improvement in infrared focal plane arrays, the infrared image processing technology is constantly developing. In particular, algorithms for the nonuniformity correction algorithm of uncooled infrared images and pseudo-color mapping have a great significance and application value [25,26,27,28].

In this paper, the proposed technology of monitoring coal mine personnel safety based on uncooled infrared focal plane technology visually displays the infrared thermal spectrum of the detected object in the form of pseudo-color images with high resolution and high sensitivity. Thus, the real-time detection and early warning of personnel safety in modern mines can be achieved. A multipoint compression correction algorithm based on human visual characteristics divides the response units of all acquisition units into gray intervals according to a threshold value, then the corresponding parameters are set in different intervals, and finally the compression of each interval is obtained by using a two-point correction algorithm. Through infrared images, the specific location of personnel can be accurately detected, and a real-time alarm can be issued if the personnel enter the dangerous area. Infrared images can clearly detect the real-time status of the personnel. If the operation and working status of the personnel are abnormal, a quick alarm can be triggered. This method can meet the need for modern mine personnel search and rescue, equipment supervision and dangerous area detection, and other early warning requirements, so as to achieve the goal of developing smart mines and ensuring safety in coal mine production.

2. Uncooled Infrared Imaging System

When the target enters the detection range of the optical lens, the optical lens projects the thermal radiation released by the target to the entire infrared focal plane array through the lens. The detection unit of the infrared focal plane induces the radiation energy to realize the conversion between the photoelectric signals.

The algorithm finally generates a thermal image from the detected grayscale image through digital signal processing. The wavelength of infrared radiation is higher than that of visible light, so the radiation intensity depends mainly on the actual temperature of the object. At the beginning of the twentieth century, Planck, based on quantum theory, studied the change law of the blackbody radiation spectrum and discovered the following relationship among the radiosity, wavelength $\lambda$ and absolute temperature $T$ [29]:

$$M_{\mathrm{b}}(\gamma ,T)=c_1{\gamma }^{-5}{[e^{c_2/(\gamma T)}-1]}^{-1}$$

(1)

where $c_1=3.743\times {10}^8 (\mathrm{W}\cdot \mathsf{\mu }{\mathrm{m}}^2/\mathrm{m})$ and $c_2=1.439\times {10}^4 (\mathsf{\mu }\mathrm{m}\mathrm{K})$ are the Planck first and second constant, respectively, and $\gamma$ is the radiation rate. The Planck radiation spectrum is shown in Figure 1. From Figure 1 we can see that the radiance of the blackbody increases with the increasing temperature; the radiance at the same temperature is continuous and reaches a peak. As the wavelength increases, the radiance increases first and then decreases [30,31].

The target infrared radiation needs air as medium to transmit energy to the optical lens, so it is necessary to consider the characteristics of radiation transmission in the atmosphere. The absorption of the radiation in the atmosphere is concentrated in the infrared and thermal infrared regions, and carbon dioxide and water vapor are the main factors affecting their atmospheric permeability. Based on the transmittance curves of the atmosphere in Figure 2, two atmospheric windows [32] are found in the atmosphere, namely the 3.0~5.0 μm and 8.0~14.0 μm bands. The transmittance of the 3.0~5.0 μm band is due to electric resonance and the high transmittance of the 8.0~14.0 μm band is due to the joint electric resonance with magnetic resonance. Through both windows, the thermal images of the target can be clearly observed at lightless nights or in cloudy weather [33,34,35,36].

The infrared imaging process can be briefly described as the energy signal of the object entering the infrared detector after going through the medium and optical system. Therefore, it is shown that the infrared detector output is affected by the combination of the object’s radiation characteristics, radiation transmission characteristics, optical system, array output characteristics and other factors. Based on the above study of infrared radiation characteristics, the factors leading to the reduced image quality are summarized into three categories: the output nonuniformity, blind pixels, and the contrast of the image.

3. Architecture Design of the Early Warning System

An uncooled infrared detector, not equipped with a cooling device, directly reduces the system power consumption and the system volume. However, the uncooled focal plane array greatly increases the data volume and indirectly increases the running time of the algorithm, thus reducing the system resolution and real-time performance. In view of these deficiencies, we improved the platform architecture and DSP (digital signal processor) processing algorithm and adopted the hardware architecture of CPLD (complex programmable logic device) + DSP + MCU (microcontroller unit) to improve the sensitivity and real-time performance of the system. CPLD mainly provides driving timing and register configuration information for the uncooled focal plane array, completes the preparation of the image acquisition and transmits the image data. An uncooled focal plane array detection pixel will always be at a uniform and constant temperature through the MCU module. Finally, the DSP processes the algorithm, connects and controls the units.

After the imaging system is turned on, the power supply circuit allocates three different voltages to each circuit, so that each chip can complete the startup. When the CPLD program is running, the drive timing configuration and register configuration information are transmitted to the infrared detector, and the infrared detector completes the startup. At the same time, the infrared radiation light of the object is transmitted through the air, and the optical system allows the radiation of 8.0~14.0 μm bands to pass through and absorbs or reflects the radiation from other bands [37,38]. When the infrared light is transmitted to the entire infrared focal plane and reaches the integration time, the array-induced radiation energy is converted to a 6-bit electrical signal by using a readout circuit and transmitted back to the 12-bit electrical signal synthesized by the CPLD. The front-end interface VPFE (video processing front end) of the VPSS (video processing submodule) of the DSP master chip receives the integrated 12-bit digital signal. The DSP processes the video signal with nonuniformity correction and pseudo-color mapping, stores the processed signal loop in memory, and finally the output thermal images are shown on the LCD (liquid crystal display) display or a PC (personal computer). The overall diagram of the uncooled imaging system is shown in Figure 2.

The core of the uncooled system consists of the front-end CPLD acquisition module, DSP main chip module, power supply module, expansion module and memory module. The schematic diagram is shown in Figure 2. The front-end CPLD acquisition module includes the optical lens, the CPLD and the MCU temperature control circuit, in which the optical lens is responsible for the collection of the infrared radiation energy; the CPLD for the transmission of the drive timing of the detector configuration and the register configuration information; and the temperature control circuit for the control of the infrared detector temperature at 30 °C. The power supply module has three major functions: charging, conversion voltage and monitoring voltage. The charging function is to recharge the lithium battery, making the infrared imaging system portable. The conversion voltage function is to provide power to the entire imaging system. Voltage monitoring is mainly for real-time monitoring of the power supply module to prevent voltage failure. The extension module reserves various interfaces for a USB port and an SD card, to provide a function for different user needs, and these interfaces facilitate the subsequent development of other needs. The memory module includes LCD displays, NORFLASH, and DDR2 memory, where the NORFLASH is primarily used to store the DSP boot master and nonuniform correction factor tables. The raw data, the intermediate calculation results and the final operation results are stored in the DDR2 memory. The operation results are presented on the LCD display. The VPBE (video processing backend end) converts the processed video signal into a hot image and then outputs it.

4. Infrared Image Preprocessing

The detector’s output could vary greatly due to the differences in production process and the materials used, and this phenomenon results in the nonuniformity of the infrared focal plane array, which causes a fixed mode noise on the infrared image, seriously affecting the temperature resolution ability of the system [39]. Therefore, it is required to have a correction algorithm to overcome the nonuniformity effect of the infrared focal panel arrays.

4.1. Two-Point Correction Algorithm

The two-point correction algorithm is based on two assumptions: (1) The error at both ends is ignored and a linear mathematical relationship is established between the output and the amplitude. (2) The output of each detection unit is time-invariant, so the gain and offset coefficients of the detection unit are calculated directly through the blackbody radiation source calibration before the system application. Thus, the general equation for the two-point nonuniformity correction can be expressed as:

$$Y_{i,j}^{\prime }=G_{i,j}{Y}_{i,j}+O_{i,j}$$

(2)

where: $Y_{i,j}$ is the gray value before the correction, $Y_{i,j}^{\prime }$ is the corrected gray value, $G_{i,j}$ is the gain coefficient and $O_{i,j}$ is the offset coefficient. When adjusting the blackbody temperature to $T_L$, we fix the position of the blackbody and infrared imaging systems and align the blackbody and infrared optical lenses to ensure that the radiation shines in the infrared focal plane [40]. When the blackbody temperature is stable, we collect and save the gray value output from each detection unit of the infrared focal plane array $Y_{i,j}(T_L)$, and the average $\overline{Y}(T_L)$ is calculated by Equation (3).

$$\overline{Y}(T_L)=\frac{1}{M\times N}{Y}_{i,j}(T_L)$$

(3)

When the blackbody temperature is adjusted to $T_H$, the position of the blackbody and infrared imaging system is fixed and the blackbody and infrared optical lens are aligned, the infrared focal plane is fully subject to radiation, and the output gray value $Y_{i,j}(T_H)$ of the detection unit is collected and stored when the blackbody temperature change tends to be stable, and the average $\overline{Y}(T_H)$ is calculated by Equation (4). M and N are the number of elements in the flat array.

$$\overline{Y}(T_H)=\frac{1}{M\times N}{Y}_{i,j}(T_H)$$

(4)

Thus, the gain and offset coefficients are obtained:

$$G_{i,j}=\frac{\overline{Y}(T_H)-\overline{Y}(T_L)}{Y_{i,j}(T_H)-Y_{i,j}(T_L)}$$

(5)

$$O_{i,j}=\frac{\overline{Y}(T_H)\cdot Y_{i,j}(T_L)-\overline{Y}(T_L)\cdot Y_{i,j}(T_H)}{Y_{i,j}(T_L)-Y_{i,j}(T_H)}$$

(6)

The two-point correction algorithm is simple to realize, occupies less hardware resources and the algorithm running time is short, which is suitable to real-time hardware processing. However, the two-point correction algorithm is based on the two hypothetical conditions mentioned above, while the response curve of each probe unit does not satisfy the linear model relationship and the time-invariant characteristic of actual processing [41]. Therefore, the two-point correction algorithm is suitable for applications when correction accuracy is low and the working environment does not change a lot.

4.2. Five-Point Correction Algorithm

We selected appropriate points on the response curve generated from data acquisition from the probe unit. The curves were then segmented and processed with the two-point correction algorithm. The response is represented by $V$, the radiation flux by $\varphi$, and the calibration points by ${\varphi }_1$, ${\varphi }_2$, ${\varphi }_3$, ${\varphi }_4$, ${\varphi }_5$.

The correction formula of the correction algorithm is:

$$V_{i,j}^{\prime }(\varphi )=K_{i,j}({\varphi }_l)V_{i,j}(\varphi )+O_{i,j}({\varphi }_l)$$

(7)

where: $\varphi =[{\varphi }_l,{\varphi }_{l+1}]$ and $l\in [1,n]$; $K_{i,j}({\varphi }_l)$ is a gain coefficient; $O_{i,j}({\varphi }_l)$ is an offset coefficient; $V_{i,j}(\varphi )$ is the precorrection response; and $V_{i,j}^{\prime }(\varphi )$ is the postcorrection response. The embedded platform can realize the operation of the multipoint correction algorithm. The calculation of the gain and offset coefficients from all the data collected in the infrared focal plane array is large. The occupation of hardware resources by the algorithm results in an increase in running time, affecting the real-time performance of the system, and the range of application is also greatly limited.

The trend of change of the infrared focal plane detector is affected by the integrated state. In the same integrated state, the response curve of the pixel is close to the parabola at different temperatures. If the infrared focal plane detector works in the multi-integrated state, the image quality is greatly affected by nonuniformity. Thus, the problem of how to reduce the image nonuniformity must be considered. The detector pixel response curve is shown in Figure 3.

4.3. Multipoint Correction Algorithm Based on Human Eyes

The human visual system is the main channel for receiving external information, and the human eye is only related to the brightness of the contrast between the target and the background and has nothing to do with the overall image. When the gray level is higher or lower, the human eye’s ability to distinguish gray levels decreases, and when the image gray level is medium, the human eye’s ability to distinguish gray levels is stronger. The human eye can only distinguish six gray levels near the gray level of 0, that is, the human eye cannot distinguish between the 0-level gray and the 5-level gray, and it is considered to be the same gray color. When the gray level is 191, the distinguishing ability is strong, and two gray levels can be distinguished; when the gray level is around 255, three gray levels can be distinguished. It can be concluded that the ability of the visual system to discriminate images has a threshold limit.

The traditional multipoint correction algorithm directly calculates the 0–255 gray values and needs to know the coefficients of all detector units (110,592 × 10 bit = 1080 kbit) and the data of five calibration points (110,592 × 5 bit = 540 kbit); the total volume of data to be stored is 1080 + 540 kbit = 1620 kbit. At the same time, the correction coefficient needs to be calculated and selected, and a multiplication and addition calculation can be performed to complete the correction. A large amount of storage and complex calculation processes occupy storage space and reduce the running speed of the algorithm.

The idea of the multipoint correction algorithm based on the human eye was, on the basis of the multipoint correction algorithm and according to the characteristics of the human eye’s ability to distinguish gray levels, to first set an appropriate threshold to divide the entire gray level and divide the same or similar gray levels into intervals. Degree-valued pixels were grouped together. According to the grayscale resolution capability curve of the human eye, the threshold was set to 6, and the 0–255 grayscale values were divided into 43 intervals, each interval corresponding to six grayscale ranges. The corresponding correction coefficient was determined according to the divided 43 intervals. During the operation, the DSP only needs to find the corresponding correction coefficient according to the interval in which the gray scale falls. The traditional multipoint correction algorithm changes the search for the entire gray value and the complex operation of recalculating the correction coefficient every time. This way, the computational complexity of the algorithm is reduced, thereby improving the computational speed of the algorithm.

The range of the image gray scale was divided into 43 different intervals according to the gray resolution threshold, and each interval was numbered for the convenience of subsequent operations. The same interval numbering consisted of pixels with similar gray values, each interval having six gray scale values. The DSP was responsible for loading the data in the coefficient table, realizing fast alignment and search according to the gray scale interval of the pixel, so as to determine the corresponding coefficient in real time. Each determined cell segment was considered linear and a two-point correction was performed separately for each interval to achieve compression of the infrared focal plane array.

At the same temperature, 384 × 288 response values can be obtained at any calibration point ${\varphi }_1$. The maximum can be obtained by Equation (8):

$$V^{\prime }({\varphi }_{\mathrm{n}})=\max \{V_{1,1}({\varphi }_n),V_{1,2}({\varphi }_n),\cdots V_{M,N}({\varphi }_n)\}$$

(8)

The maximum obtained in Equation (8) served as the correction value of this interval; based on the two-point correction principle, the gain and offset lineage arrays were generated at calibration point ${\varphi }_1$, and the latter four points were processed in the same way, so that 384 × 288 × 5 sets of gain and offset coefficients were obtained. As shown in Equations (9) and (10)

$$K_{i,j}({\varphi }_l)=\frac{V^{\prime }({\varphi }_{l+1})-V^{\prime }({\varphi }_l)}{V_{i,j}({\varphi }_{l+1})-V_{i,j}({\varphi }_l)}$$

(9)

$$O_{i,j}({\varphi }_1)=\frac{V_{i,j}^{\prime }({\varphi }_{l+1})V^{\prime }({\varphi }_l)-V_{i,j}({\varphi }_l)V^{\prime }({\varphi }_{l+1})}{V_{i,j}({\varphi }_{l+1})-V_{i,j}({\varphi }_l)}$$

(10)

when the image is compressed, the gray value $V_{i,j}$ of the calibration point $(i,j)$ is encoded with number $H_{i,j}$ in the corresponding interval. The encode number could effectively help avoid repeated operations. The coefficients generated by these five calibration points were calculated by Equations (11) and (12):

$$K_{i,j}^{\prime }(H_{i,j})=\frac{V^{\prime }({\varphi }_{l+1})-V^{\prime }({\varphi }_l)}{(H_{i,j}({\varphi }_{l+1})-H_{i,j}({\varphi }_l)}$$

(11)

$$O_{i,j}^{\prime }(H_{i,j})=\frac{H_{i,j}({\varphi }_{l+1})V^{\prime }({\varphi }_l)-H_{i,j}({\varphi }_l)V^{\prime }({\varphi }_{l+1})}{H_{i,j}({\varphi }_{l+1})-H_{i,j}({\varphi }_l)}$$

(12)

The resolution of the nonuniformity in measuring methods for the parameters of infrared focal plane arrays (GB/T17444-2013) is shown in Equation (13):

$$\begin{array}{ll}{U}_R=& \frac{1}{\overline{R}}\sqrt{\frac{1}{M\times N-(d+h)}}•\\ & \sqrt{{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{(R(i,j)-\overline{R})}^2}}}\times 100\%\end{array}$$

(13)

where $R(i,j)$ is the response value of the detection unit and $\overline{R}$ is the response average of the infrared focal plane. The infrared image can clearly detect the real-time status of the staff and compare the movement trajectory and activity status of the staff with the real-time detected image results under normal conditions. If the operation and working status of the staff are abnormal, a quick alarm can be triggered. The nonuniformity (NU) is used to measure the correction ability of the algorithm proposed in this paper. If the value of NU is small, it means that it has a better correction ability. The nonuniformity by the two algorithms is shown in

Table 1. Nonuniformity by the two algorithms.

Blackbody temperature/°C	35	55	75	95	115
After correction	10.96	8.95	8.02	7.12	6.32
After five-point correction (%)	4.01	4.52	2.49	2.89	3.31
After multipoint correction (%)	3.98	3.96	2.27	2.63	2.99

.

It can be easily found that the compression of the image can improve the image quality and obtain better correction from the data in Table 1. The encoding table was corrected by a MATLAB simulation, and the volume of data stored was the sum of the correction curve and the encoded table data. Thus, the traditional algorithm can 1620 kbit. The correction algorithm proposed in this paper fundamentally reduces the amount of data, performs compression well and yields high quality image data. The result in

Table 2. The number of CPU cycles occupied by the two algorithms.

	Five-Point Correction	Multipoint Correction of Human Eyes
Cycle/frame	52 462 248	18 257 363
Frequency (Hz)	12.16	29.97

is the number of CPU cycles run by the two algorithms, calculated by the clock from CCS (Code Composer Studio) 3.3.

5. Field Test

The experimental results of sending a warning when the personnel approached dangerous areas via noncontact detection in coal mines and sending an emergency rescue are shown in Figure 4. We can see that the nonuniformity of the image is effectively corrected, reducing the influence of the interference noise, and the target can be clearly seen. After collecting and comparing the nonuniformity values conducted at different blackbody temperature intervals using the five-point correction algorithm and the multipoint correction algorithm of human eyes, respectively, it shows that when the blackbody temperature reaches 115 °C, the nonuniformity is 6.32% before the multipoint correction of human eyes, and 2.99% after it. It is not difficult to see that the multipoint correction algorithm based on human eyes proposed in this paper has good denoising ability, improves the detection accuracy of the algorithm, and reduces the data processing amount, calculation and workload of the processor CPU, so as to effectively improve the detection speed of the algorithm.

This specific result shows that the proposed algorithm can well improve the contrast between target and background, boost the system’s resolution ability, effectively enhance the image resolution and accuracy, and provide a solid basis for the reliable early warning of the system. Thermal image technology is used to search and identify the exact location of the victims for prompt rescue in dark, smoky and foggy environments. The thermal image instrument can not only measure the surface temperature of the object, but also show the temperature distribution of the object, forming a “heat map” to provide the rescuers with more information about the object status and to improve rescue efficiency.

6. Conclusions

• This paper studied the advantages and disadvantages of both traditional nonuniformity correction algorithms and nonuniformity correction algorithms based on human visual characteristics and proposed a multipoint compression correction algorithm based on human visual properties to improve the image resolution.
• In view of the disadvantages, such as poor contrast and low resolution, of infrared images, this paper adopted a pseudo-color mapping algorithm with fixed thresholds to improve the contrast and visual effect of images.
• Preliminary modeling, simulation and field tests showed that the proposed algorithm could satisfy the need for the analysis of the surface temperature of an object and noncontact temperature measurement in modern mines.