Intelligent System Study for Asymmetric Positioning of Personnel, Transport, and Equipment Monitoring in Coal Mines

Abstract

The paper presents a study of an intelligent system for personnel positioning, transport, and equipment monitoring in the mining industry using convolutional neural network (CNN) and OpenPose technology. The proposed framework operates through a three-stage pipeline: OpenPose-based skeleton extraction from surveillance video streams, capturing 18 key body joints at 30fps; multimodal feature fusion, combining skeletal key points and proximity sensor data to achieve environmental context awareness and obtain relevant feature values; and hierarchical pose alert, using attention-enhanced bidirectional LSTM (trained on 5000 annotated fall instances) for fall warning. The experiment conducted demonstrated that the combined use of the aforementioned technologies allows the system to determine the location and behavior of personnel, calculate the distance to hazardous areas in real time, and analyze personnel postures to identify possible risks such as falls or immobility. The system’s capacity to track the location of vehicles and equipment enhances operational efficiency, thereby mitigating the risk of accidents. Additionally, the system provides real-time alerts, identifying abnormal behavior, equipment malfunctions, and safety hazards, thus promoting enhanced mine management efficiency, improved safe working conditions, and a reduction in accidents.

1. Introduction

In the context of mining operations, the inherent complexity of the environment, the potential hazards posed by production equipment, and the confined nature of the workplace contribute to the high-risk levels characteristic of coal mines. To enhance the safety management level of such facilities and reduce the incidence of accidents, there has been a significant focus on the study and implementation of intelligent mine systems. A key component of such systems is the identification of potential hazards for personnel in the early stages [1,2,3]. By monitoring the position and behavior of personnel in real time, the system can detect safety hazards in a timely manner, issue early warnings, and prevent accidents [4,5,6].

In the 1990s, the research was mainly based on static image analysis, and traditional methods such as edge detection and color analysis were adopted to achieve simple human body recognition. However, these methods were constrained by the limited scenario variability (e.g., fixed camera angles and controlled lighting conditions), resulting in inadequate robustness when applied to complex dynamic environments. This limitation prompted a paradigm shift around 2000, when video-based dynamic analysis emerged as the dominant research direction. Machine learning algorithms such as Support Vector Machine (SVM) and Hidden Markov Model (HMM) were used for the classification of typical behaviors such as running and falling, but they relied on manual features and had limited generalization ability. After the breakthrough of deep learning technology in 2012, object detection (such as YOLO, Faster R-CNN) and behavior recognition models centered on CNN have significantly improved accuracy and real-time performance. The proposal of OpenPose technology in 2016 enabled real-time 2D human pose estimation, laying the foundation for behavior analysis in high-risk scenarios such as mines through the capture of skeletal key points in 2020. In recent years, multimodal sensing (RGB+D) and graph neural networks (GNNs) have further enhanced the ability to understand behaviors in complex environments, promoting the development of security monitoring toward intelligence and self-adaptation.

The proposal entails an investigation into an intelligent system that integrates convolutional neural network (CNN) technology [7,8,9] and OpenPose technology [10,11] with the objective of enhancing the safety and efficiency of mining operations. The system is designed to detect and identify the location and behavior of personnel, calculate the distance to hazardous areas in real time, and perform posture analysis to detect risks such as falls or immobility.

Furthermore, the system incorporates vehicle and equipment monitoring to track movements, prevent collisions, and ensure operational efficiency [12,13,14]. Real-time alerts are generated for abnormal behavior, equipment malfunctions, and safety hazards, thus constituting a comprehensive solution to improve mine management, reduce accidents and enhance overall safety in complex mining environments.

The proposed study aims to investigate the architecture of a mining system consisting of personnel positioning, transport process detection, and safe equipment fault detection modules as part of a study of an intelligent system for personnel positioning, as well as equipment transport and monitoring in a coal mine working environment. The simulation of the above processes will confirm the merits of the intelligent system, as well as evaluate and identify ways to optimize the performance of the model.

The novelty of this work lies in the combination of CNN and OpenPose to achieve multi-task monitoring (personnel location, behavior analysis, equipment failure detection) in complex mining environments. Meanwhile, compared with the traditional OpenPose detection of 18 key components, this study innovatively proposes a five-key point screening strategy based on OpenPose. By calculating features such as centroid height and joint acceleration, combined with the lightweight LSTM network, real-time fall detection in complex mining scenarios is achieved, reducing the computational complexity while maintaining high accuracy. It has solved the limitations of traditional methods that rely on all key points or high-cost equipment.

2. Materials and Methods

• Intelligent system architecture and functions:

The system under study is an intelligent mine management platform that integrates advanced technologies and aims to improve safety, production efficiency, and the level of operational intelligence at the mine by monitoring and analyzing the state of personnel, transport processes, and equipment in real time (Figure 1). The system combines three modules: personnel location and behavior detection, transport process detection, and equipment operation monitoring, providing a comprehensive digital solution for mine management.

The advantages of this system include:

(1)

Comprehensive safety: real-time positioning and monitoring functions provide control over personnel and equipment, while hazardous behavior detection and early warning mechanisms effectively reduce accidents.

(2)

Efficient resource management: intelligent route planning and optimization greatly improves the efficiency of rock mass transportation, reducing resource wastage and operating costs.

(3)

Intelligent operation support: the system’s data recording and analysis function provides a visual report for mine management and supports scientific decision-making and management optimization.

(4)

Modular scalability: the system is very flexible and can be expanded according to the actual needs of the mine, such as environmental monitoring, energy management, etc.

(5)

This study implements the positioning and fall detection of personnel through video analysis of existing surveillance cameras, without the need to install additional underground positioning equipment, effectively avoiding installation difficulties in special environments.

The combination of 5G functionality with remote working capability allows miners to perform their duties with minimal delays, as well as providing real-time, high-definition images from the mine site. The use of artificial intelligence algorithms plays a key role in this issue, as it allows personnel to make informed decisions about the process in real time in a ground-based, well-air-conditioned environment. This capability extends to early detection and real-time intervention for any abnormalities or sub-optimal performance, which helps to build confidence in the process. Ensuring safe operation is facilitated by the real-time transmission of high-definition images of scenes on the conveyor belt.

The module of personnel location and behavior detection enables the execution of its primary function through the implementation of three stages:

(1)

Personnel positioning: the system receives input images or videos, detects personnel in the mining area using convolutional neural network (CNN) technology [15,16,17], and accurately locates them.

(2)

Behavioral safety detection: in conjunction with known hazardous areas in mining areas (such as high-temperature equipment or hazardous work zones), the system calculates the distance between personnel and these hazardous areas and marks them in real time. When a person approaches a hazardous area, the system emits an alarm.

(3)

Behavioral analysis and safety early warning: the system utilizes OpenPose technology [18] to analyze the behavior of an identified person, identifying dangerous behaviors such as falling, immobility, etc., and triggering an appropriate safety early warning based on the distance to the danger zone.

The transport process detection module has been developed for the purpose of monitoring and optimizing the transportation of minerals in real time using intelligent detection technology [19], with the objectives of ensuring transportation efficiency and reducing safety risks. The functionality of this module is characterized by the ability to perform the following tasks:

(1)

The detection of foreign bodies during the transportation of minerals on the conveyor belt: utilizing computer vision and sensor technology to monitor minerals on the conveyor belt in real time, accurately detecting foreign objects or anomalies and ensuring a smooth and safe transfer process;

(2)

Real-time foreign object identification (Figure 2): conveyor belt images are captured by high-resolution cameras and deep learning algorithms (e.g., YOLO [20,21,22] or Faster R-CNN [23,24,25]) are used to detect foreign objects (e.g., tools, metal fragments, etc.) mixed with ore;

(3)

Automatic shutdown and signaling: if foreign objects are detected, the system automatically sounds an alarm and shuts down to prevent equipment damage or accidents;

(4)

Data recording and analysis: records the time, location, and category of foreign object detection to provide the data needed to further improve ore sorting and handling processes;

(5)

Transport detection and route planning: using positioning technology and a transport planning algorithm [26], the speed and path of mining machines are dynamically tracked and optimized to improve transport efficiency and reduce road safety hazards;

(6)

Speed monitoring and warning: using GPS and sensors to monitor vehicle speed in real time, detect dangerous activities such as speeding or emergency stops, and provide timely safety warnings;

(7)

Route planning and optimization: intelligent planning of an optimal vehicle route based on real-time traffic data and transport tasks in mining areas reduces congestion and fuel consumption;

(8)

Collision avoidance systems: use of UWB sensors or vision sensors to monitor the distance between the vehicle and surrounding objects or people to avoid collisions;

(9)

Data visualization and planning: dynamic map of the mining area, real-time display of vehicle location, task status, and transport efficiency, and intuitive management tools for planners.

With the above functions, the module can comprehensively enhance the intelligent management of the mine transport process and provide a reliable guarantee for safe production and efficient operation of the mining site.

The safe equipment fault detection module is realized through the use of an artificial intelligence (AI) image processing algorithm, with the intelligent video analysis platform monitoring and warning of underground equipment faults in the mine shaft in real time. The system can automatically detect abnormal equipment status, such as abnormal running speed, abnormal temperature, etc., and send an alarm in advance to reduce equipment failures and production accidents.

In the event that personnel fail to rectify the unsafe behavior despite the system’s warning, the control system is programmed to initiate the shutdown sequence. Concurrently, the control personnel receive the requisite shutdown information (Figure 3).

• Features of system architecture design:

When designing the system, consider that it can support the following forms of information input:

(1)

Image input: the system can process a single frame image of the mine area and analyze the position and behavior of people in the image;

(2)

Video input: real-time analysis of the location, behavior, and interaction of personnel with hazardous areas using video streams.

It is proposed to investigate the design of a personnel location module based on CNN.

Convolutional Neural Network (CNN) is one of the most widely used deep learning models in computer vision, especially in object detection. By learning on large datasets, CNN can learn the features of different targets in images to achieve accurate target identification and positioning. In this system, a CNN-based object detection algorithm is used to determine the location of each individual and infer its boundaries and type [28].

Implementation phases as shown in Figure 4:

(1)

Model selection and training: A pre-trained CNN model such as YOLO [29], Faster R-CNN [30], Mask R-CNN [31], or RetinaNet [32] is selected, and further tuned to adapt to the mining scene. To ensure that the model can identify people in different environments, large-scale mining image data are employed;

(2)

Image processing: the input image or video frame is pre-processed, including scaling, normalization, and enhancement, to ensure that the input image meets the requirements of the model;

(3)

Personnel detection: the processed image is input to the CNN model and the model outputs information about the location (bounding box), type (e.g., workers, visitors, etc.), and confidence level of the personnel;

(4)

Transmission of results: the location information of the person is then transmitted for further processing to the module for detection of subsequent behavior and early warning of security breaches.

The working environment in a mine is complex, with the presence of both mechanical and high-temperature equipment, as well as high-hazard areas such as those designated for hazardous chemical handling or heavy machinery operations. In these areas, the improper behavior of personnel can lead to severe accidents, emphasizing the necessity for real-time monitoring of personnel’s position relative to hazardous areas by the system. The following steps have been identified for the implementation of the behavioral safety detection module:

(1)

Define the danger zone: In the geographical information system (GIS) of the mine area, the specific location of the danger zone is marked in advance and the safety radius is defined. For example, for a piece of equipment, define a danger zone within 5 m of the equipment;

(2)

Real-time distance calculation: By comparing the location of the person identified by the CNN model and the location of the marked danger zone, the distance between the person and the danger zone is calculated in real time. Simple calculations can be performed using Euclidean distance or Manhattan distance;

(3)

Hazard Zone Marking: when personnel enter or approach a hazardous area, the system will highlight the hazardous area and send a warning signal to remind the mine manager to pay attention.

The Behavior Detection and Danger Alert module is based on OpenPose technology as shown in Figure 5:

OpenPose is a widely used deep learning framework for human pose estimation, capable of analyzing a person’s posture by identifying key body points such as head, shoulders, elbows, knees, etc. [33]. In this system, OpenPose is mainly used to identify safety hazards in personnel behavior such as falling or prolonged immobility. The system combines a person’s location and behavior to determine if a hazard needs to be warned.

As shown in the figure, the training process first builds a dataset based on the mine monitoring video. Through OpenPose, 18 key points of the human body in each frame of the image are extracted, and 5 core key points, including the head, hip, and knee are selected to reduce redundancy. Aiming at the spatiotemporal characteristics of the falling behavior, calculate the features such as the centroid height, joint acceleration, and hip–knee angle of each frame, and construct the time series data. The dataset is divided into the training set, the validation set, and the test set according to 7:1.5:1.5, and the lightweight LSTM network is adopted for modeling and training.

Implementation steps:

(1)

Loading the OpenPose model: using the pre-trained OpenPose model, initialize and load the required parameters;

(2)

Pose Recognition: Analyze the pose of the person detected and extract the coordinates of key points, including head, torso, limbs, etc. The OpenPose model is able to identify 18 key points of the human body based on a photo or video frame and generate a human skeleton map based on these points;

(3)

Behavioral Analysis: Based on the pose data, the system can determine if a person has abnormal behavior. For example, if a person’s horizontal body posture is detected to be too low (e.g., limbs pressed against the ground), the system can assume that the person has fallen; if a person’s posture is detected to be unchanged for a long time, the system recognizes that the person may be in a motionless state;

(4)

Hazard Warning: in combination with the hazardous area analysis in the second step, when a person falls or remains motionless near a hazardous area for a long time, the system issues a hazard warning and notifies the relevant management personnel so that they can take action in time.

OpenPose is a state-of-the-art open-source tool designed to accurately estimate body posture using computer vision. Using deep learning algorithms, it can identify key points of the human body in 2D images or videos to recognize skeletal structure and posture [34].

The working principle of the algorithm is summarized in Figure 4:

• part (a): first, the whole image is fed to CNN, which generates prediction;
• part (b): confidence maps to locate the body parts;
• part (c): object affinity fields to establish relationships between body parts;
• part (d): using these results, a two-part matching algorithm is applied to connect the detected body parts;
• part (e): finally, the body parts are assembled into full poses of all people in the image.

The initial setup in the CNN structure is as follows: the first stage is responsible for prediction and the last stage focuses on generating the trusted output. The graph combines and passes the predictions of each stage along with the corresponding image features to the next stage. Furthermore, the original 7 × 7 convolutional layer was replaced by three consecutive 3 × 3 convolutional layers that were combined in the output (Figure 5).

In order to better reveal the interconnected structure of the human body and to analyze human pose behavior, the following processing steps are performed for the confidence graph of part detection (Figure 6):

(1)

Candidate points for both body part types (red and blue lines) and all possible connections (gray lines) are identified;

(2)

The connections are optimized using midpoints (yellow dots) so that correct connections are represented by black lines and incorrect ones by green lines while satisfying the association constraints;

(3)

The final result is obtained using the PAF (yellow arrow), which encodes the position and orientation of the limb, effectively eliminating false connections.

During training, consensus maps S^* are generated from annotated 2D key points. Each credibility map is a 2D representation of the belief that a particular body part can be located in any specified pixel. If there is one person in the image, each concordance map should have one peak if the corresponding body part is visible. If there are multiple people in the image, for each person k there should be a peak corresponding to each visible part j. First, individual concordance maps S_j,k* are created for each person k. Let x_j,k ∈ R₂ be the true position of body part j for person k in the image. The value at location p ∈ R₂ in S_j,k* is defined as:

$$S_{j,k}^{\ast }\left(p\right)=exp exp \left(-{\displaystyle \frac{\parallel p-x_{j,k}{\parallel }_2^2}{{\sigma }^2}} \right),$$

(1)

where σ controls the spread of the peak. The true confidence map predicted by the network is the union of the individual confidence maps using the max operator

$$S_j^{\ast }\left(p\right)=\underset{k}{max}{S}_{j,k}^{\ast }\left(p\right)$$

(2)

In this study, the maximum of the concordance maps is utilized as opposed to the mean value, thereby ensuring the accuracy of nearby peaks remains distinct, as illustrated in Figure 6. During the testing phase, the concordance maps are predicted and candidates for body parts are obtained through suppression without the maximum [35].

The OpenPose output, which detects key points of the body, legs, arms, and face in real time, exhibits robustness to occlusions, including during human–object interaction.

3. Results and Discussions

• Technical implementation details:

The following technologies and techniques were favored in the image recognition experiment:

(1)

CNN model: CNN model for personnel location detection was selected from Faster R-CNN because this model has high detection accuracy and good generalization ability and is suitable for real-time personnel detection in the mining industry;

(2)

OpenPose: It was used for pose estimation and behavior analysis. OpenPose analyzes human poses in real time by identifying human key points and is a key tool for behavior detection and early warning of danger;

(3)

Python.3,8.3 + OpenCV: A basic framework for image and video processing. OpenCV is a powerful image-processing library that efficiently processes frames and integrates with deep learning models;

(4)

PyTorch/TensorFlow: as a deep learning framework for the CNN model and OpenPose model, it provides model loading, training, and inference functions.

CNN model training. In the context of personnel detection systems in mining areas, it is imperative to first train the CNN model on the specific scene of the mining area. This can be facilitated by utilizing publicly available COCO datasets [36,37] as a foundation, and subsequently employing images of mining areas for transfer training, thereby enhancing the model’s performance in such environments. As shown in

Table 1. The csv format dataset obtained through OpenPose processing.

Frame_ID	Ratio_Bbox	Log_Angle	re	Ratio_Derivative	gf	Label
1	0.81	0.39	0.88	0.05	0.9	0
2	0.79	0.41	0.85	0.06	0.88	0
3	0.79	0.50	0.80	0.10	0.84	0
4	0.68	0.70	0.73	0.15	0.76	1
5	0.65	0.85	0.60	0.20	0.69	1

, it is the csv format data obtained by me through preprocessing the coco format data.

The advantages of the OpenPose model can be seen through the comparison in

Table 2. Correlation technology comparison.

Technology	Key Point Detection	Object Detection	Speed	Suitable for Fall Detection
OpenPose	Yes	No	intermediate	Behavior analysis, multi-person posture recognition
YOLO	No	Yes	fast	Monitoring, real-time applications
Faster RCNN	No	Yes	slow	Accurate detection, small target recognition

. The utilization of OpenPose models can be achieved through direct inference using pre-trained models. However, in complex mining scenarios, such as those involving dense crowds (Figure 7) or poor lighting conditions, the model’s capacity to estimate attitude may be compromised [35]. The red box in the picture marks the positions of the detected personnel, the blue line Outlines the upper body area, and the green line Outlines the lower body area.

• Simulation result:

The images were fed to the input of the system (Figure 8, Figure 9 and Figure 10).

The graph in the figure is a label for the personnel showing their real-time position, and the prediction result on the graph shows their real-time state, such as normal at the beginning, indicating that the personnel are safe (Figure 10). When the block detects that the position of the key node of the personnel meets the definition of falling, the prediction result shows falling, indicating that the personnel will receive an early warning of danger (Figure 11 and Figure 12). The value returns to normal under normal conditions.

In practical applications, the following aspects should be considered when evaluating and optimizing system performance:

(1)

Accuracy: the accuracy of the inspection model is the primary metric used to ensure that all personnel in the mine are correctly identified and located;

(2)

Coverage: evaluation of the model’s ability to detect complex conditions such as illumination changes, occlusion, etc., to fully capture all target individuals;

(3)

Response time: it is necessary to ensure that the system has a high level of real time so that monitoring and early warning of mine personnel can be provided in a timely manner, typically requiring each frame to be processed in less than 100 milliseconds.

Accuracy of behavior analysis: The behavior recognition capabilities of OpenPose should be tested on standard datasets to ensure that the system can accurately identify abnormal behavior such as falls or immobility.

As shown in

Table 3. Comparison of different key point selection models in OpenPose.

Key Point Strategy	Accuracy (%)	Inference Time (ms)	Memory Usage (MB)
Full 18-Key point	91.2	120	350
Selected 5-Key Point	89.8	65	150

, compared with the traditional 18-key point detection of OpenPose, the five-key point strategy achieved considerable accuracy (only a 1.4% decrease) while reducing the inference time by 45% and memory consumption by 57%, proving its applicability to resource-constrained mining environments.

At present, the research has been verified in the three-dimensional simulated mine scene (50 m × 30 m × 10 m) constructed in the laboratory environment. The fall detection module has achieved an accuracy rate of 89.8% in the test of the standard action library of simulated miners. For the practical application underground, the research team has established a cooperation intention with the JIE Coal Industry and plans to advance it in two phases: in the first phase, edge computing nodes based on this algorithm will be deployed to fixed monitoring areas such as underground waiting rooms, and infrared supplementary lighting will be used to enhance low-light adaptability. The second stage combines the transformation of intrinsically safe cameras for mining to achieve mobile area coverage. The extended functions such as transportation monitoring will be developed as independent modules and connected to the existing mine Internet of Things platform through the OPC UA protocol.

4. Conclusions

The article examines an intelligent system for the real-time positioning of personnel, transport, and equipment monitoring, utilizing convolutional neural networks and OpenPose technologies. The project combines the high-precision target positioning of CNN with the real-time pose analysis of OpenPose to achieve personnel positioning, fall detection, and danger warning (fall warning) within a single framework, solving the efficiency bottleneck of traditional methods that rely on the independent operation of multiple modules. Meanwhile, five core nodes were innovatively selected: the head, both hips, and both knees, from the 18 skeletal key points of OpenPose. By calculating features such as the sudden drop in the height of the center of mass and the sudden change in the hip–knee angle, a lightweight spatiotemporal sequence was constructed, which increased the inference speed of the LSTM network by 45% (65ms/frame) while maintaining a detection accuracy rate of 89.8%. This significantly reduces the computational complexity in complex mining environments. The adaptability of the current system to extremely low-light conditions (such as underground areas without auxiliary light sources) still needs to be improved. In the future, it is planned to enhance environmental robustness by integrating infrared thermal imaging sensors. The transportation and equipment monitoring functions proposed in the article are theoretical design modules. Subsequently, the Internet of Things (IoT) and multi-agent collaboration technologies will be combined to achieve complete closed-loop management.

It has been determined that this system encompasses the fundamental elements of mine operations and is capable of facilitating integrated intelligent management of personnel, materials, and equipment. The system has been shown to enhance mine safety and provide substantial technical support for efficient operation and sustainable development of the mine by optimizing resource allocation and intelligent planning. At the same time, this system provides an expandable technical framework for the intelligent transformation of mine safety management, and its lightweight design is particularly suitable for underground environments with limited edge computing resources.