Skip to Content
BiomimeticsBiomimetics
  • Article
  • Open Access

25 May 2026

22 Pages

Bio-Inspired Adaptive Multimodal Decision Fusion for Intelligent Safety Monitoring in Confined Spaces

,
,
and
1
The Zhongshan Power Supply Bureau, Guangdong Power Grid Co., Ltd., Zhongshan 528400, China
2
School of Electrical and Electronic Engineering, Shandong University of Technology, Zibo 255049, China
*
Author to whom correspondence should be addressed.

Abstract

To improve operational safety in confined spaces, this study proposes an intelligent safety monitoring framework that utilizes multimodal data from wearable devices. The framework comprises two core components: a human activity recognition (HAR) module and a bio-inspired adaptive multimodal decision fusion (BA-MDF) module. The HAR module processes accelerometer and gyroscope data through an enhanced FFT–LSTM architecture that integrates time- and frequency-domain features for real-time activity classification. The BA-MDF module, inspired by biological multisensory integration mechanisms—particularly the inverse effectiveness principle observed in the superior colliculus—evaluates contextual risk by adaptively fusing HAR outputs, heart rate variability, and geospatial constraints without additional computational overhead. Experimental testing demonstrated 92.4% overall HAR accuracy and 94.3% identification accuracy for emergency scenarios under a simulated sensor degradation environment. These results validate the framework’s effectiveness in mitigating risks from anomalous events in visually constrained environments.

1. Introduction

With the development of information and communication technologies (ICT), intelligent safety monitoring methods have attracted global attention in recent years, particularly within high-risk confined environments such as underground mines, industrial tanks, and confined construction zones [1]. While existing ICT-based approaches have demonstrated their ability to detect potential anomalous situations, most of them rely on visual surveillance systems [1,2]. However, such vision-dependent approaches face critical limitations in visually degraded conditions—such as fog, smoke, dust, or low-light scenarios—commonly encountered in industrial settings. These limitations underscore the need for alternative strategies to reliably detect anomalous situations in visually degraded cases.
Human activity recognition (HAR) serves as the fundamental step in intelligent safety monitoring. Extensive research efforts have focused on HAR methodologies that fuse accelerometer and gyroscope measurements [3]. The strength of these methodologies lies in their capability to classify activities within visual surveillance blind zones, where traditional vision-based monitoring exhibits inherent limitations. Several studies have utilized wearable sensors to collect accelerometer and gyroscope data for analyzing walking patterns (e.g., level walking, ascending/descending stairs) and stationary postures (e.g., sitting, standing, lying) [4,5,6]. These studies have predominantly employed deep learning architectures, with long short-term memory (LSTM) networks serving as the primary architecture for activity classification. Parallel to this, researchers have developed fall detection methods through machine learning, including support vector machines (SVM), recurrent neural networks (RNN), and LSTM, using inertial sensor data from smartphones or smartwatches [7,8,9]. Further, technologies for detecting anomalous activities were introduced in prior research [10,11], which rely on analyzing acceleration magnitudes derived from smartwatch-based accelerometers and gyroscopes. Collectively, these LSTM-based HAR implementations demonstrate approximately 90% accuracy.
Despite these advancements, a critical limitation persists: for a recognized activity, existing approaches based on accelerometer and gyroscope data face challenges in distinguishing between normal and anomalous patterns. A particularly challenging scenario arises when distinguishing between human falls and accidental device drops, as the sensor readings captured during a person’s fall closely resemble those recorded when a smartphone is simply dropped to the ground. Consequently, existing techniques are unable to effectively discriminate between the critical event of human falling and the commonplace occurrence of device dropping. This ambiguity leads to an unacceptably high rate of false alarms, eroding system trustworthiness in life-critical applications.
The cross-analysis of multimodal data can be used to improve safety monitoring, and, to overcome this fundamental limitation, the present work puts forward a novel bio-inspired adaptive multimodal decision fusion (BA-MDF) framework that transcends single-sensor HAR by integrating three complementary data streams:
-
IMU-based activity classification enhanced through attitude and heading reference system (AHRS) for orientation-aware feature extraction, followed by fast Fourier transform (FFT) for spectral pattern recognition and LSTM for temporal sequence modeling;
-
Physiological monitoring via continuous heart rate variability (HRV) from wearable smartwatches, serving as a biomarker for stress-induced physiological response;
-
Spatial context derived from GPS positioning, enabling detection of group-level anomalies (e.g., multiple individuals in prolonged supine positions within a 5 m radius).
The integration of these modalities enables context-aware risk inference: a detected “lying” posture is not interpreted in isolation. Instead, it is validated through cross-modal correlation—elevated HRV indicates physiological distress, while spatial adjacency confirms potential multi-person collapse. This multi-layered inference significantly reduces false positives while enhancing sensitivity to true emergencies.
Unlike prior works treating HAR as a standalone classification task, our approach redefines activity recognition as the first layer of a risk-inference pipeline, where contextual and physiological cues serve as essential validation signals. This paper makes three key contributions:
(1)
A novel AHRS–FFT–LSTM pipeline for orientation-resilient HAR in dynamic, body-mounted IMU configurations;
(2)
The first fusion of HRV and GPS spatial clustering with HAR outputs for hazard discrimination in confined spaces, enabling adaptive weight adjustment based on real-time sensor reliability;
(3)
A lightweight, real-time BA-MDF decision logic with adaptive risk scoring, validated under field-deployable constraints.
The remainder of this paper is structured as follows: Section 2 provides an overview of related work in HAR and safety monitoring. Section 3.1 outlines the overall system architecture. Section 3.2 details the HAR module design, including an enhanced FFT–LSTM model for activity recognition. Section 3.3 details the BA-MDF module for multimodal fusion and decision-making. Section 4.1 presents the performance evaluation of HAR using public datasets. Section 4.2 presents system validation through controlled experiments. Section 5 concludes the paper and outlines future research directions.

3. Methodology

3.1. Overall Architecture for BA-MDF-Based Safety Monitoring

Figure 1 illustrates the comprehensive architecture of the proposed multimodal safety monitoring system designed for real-time hazard detection among workers in confined spaces. The system integrates multiple sensor modalities—accelerometer, gyroscope, GPS, and heart rate—to enable comprehensive situational awareness. The architecture is structured into two tightly coupled modules: human activity recognition (HAR) and bio-inspired adaptive multimodal decision fusion (BA-MDF), each serving distinct yet synergistic functions.
Figure 1. Schematic overview of the proposed safety monitoring framework.
The HAR module performs fine-grained activity classification by processing high-frequency IMU data (accelerometer and gyroscope) sampled at 50 Hz from wearable devices. Raw triaxial acceleration and angular velocity signals are preprocessed and fed into a hybrid pipeline comprising:
-
Attitude and heading reference system (AHRS): reconstructs 3D device orientation in real time;
-
Fast Fourier transform (FFT): extracts spectral features from time-series inertial data;
-
Long short-term memory (LSTM): captures temporal dynamics to distinguish among activities including walking, sitting, and lying.
Concurrently, GPS data provides geospatial coordinates and velocity estimates, while HRV from a smartwatch serves as a physiological indicator of stress, enabling the detection of both physical exertion and cognitive load.
The BA-MDF module serves as the system’s risk inference and alerting layer, integrating HAR outputs, HRV trends, and geospatial constraints into a unified decision framework. It evaluates contextual risk using an adaptive scoring mechanism that maps activity states against predefined safety thresholds (e.g., prolonged stationary posture in restricted zones, elevated heart rate during high-risk tasks). Deviations exceeding predefined thresholds trigger tiered alarm signals via deterministic rule-based logic.
The proposed framework is designed as a centralized processing architecture. Wearable devices (e.g., smartwatches) function exclusively as sensing peripherals, responsible solely for raw data acquisition. All computationally intensive tasks—including FFT–LSTM-based HAR inference, bio-inspired adaptive fusion scoring, and anomalous state decision-making—are executed on the industrial PC.
Implementation of the HAR module is described in Section 3.2, with emphasis on model enhancement and implementation of HAR using the enhanced model. The full specification of BA-MDF, including data-fusion rules and hazard-tiered responses, is detailed in Section 3.3.

3.2. Human Activity Recognition (HAR) Module

3.2.1. An Enhanced FFT–LSTM Model for HAR

In this section, the foundational LSTM architecture for time-series analysis is outlined, followed by a concise exposition of Fourier transformation principles. The proposed FFT–LSTM model is then formulated to synergistically integrate time- and frequency-domain features, thereby enhancing the accuracy of long-term forecasting.
  • A. Long Short-Term Memory Network
As a specialized variant of RNNs, LSTM effectively addresses the long-term dependency limitation commonly encountered in conventional RNN architectures when training long sequences of data [34]. Its unique gating mechanism enables information retention across long sequences, rendering LSTM highly suitable for temporal prediction tasks. The model demonstrates its effectiveness in capturing temporal patterns and modeling complex long-term dependencies within sequential data.
Figure 2 presents how the LSTM unit transitions its state at step t, relying on xt, ht−1, and ct−1 through four essential stages [34]:
Figure 2. LSTM unit structure.
1. Forget Gate Step—determines which components of the prior cell state are to be eliminated. The forget gate ft is computed by:
f t = σ W f · x t + U f · h t − 1 + b f
2. Input Gate Step—controls which fresh information can enter cell memory. The input gate it and the input modulation gate c ^ t are computed using the formula:
i t = σ W i · x t + U i · h t − 1 + b i
c ^ t = t a h n W c · x t + U c · h t − 1 + b c
3. State Update Step—updates the current state based on the information from the input gate and the forget gate. The memory gate ct is computed by:
c t = f t ⊙ c t − 1 + i t ⊙ c ^ t
4. Output Gate Step—determines the output information ot from the current cell state. Subsequently, based on the updated cell state ct, the hidden state ht is computed.
o t = σ W o · x t + U o · h t − 1 + b o
h t = o t ⊙ t a h n c t
where {W, U, b} denote input weights, hidden state weights, and bias terms, respectively; σ represents the sigmoid activation function, and tanh represents the hyperbolic tangent activation function.
LSTM excels at time-series prediction tasks owing to its ability to process sequential data while preserving long-term dependencies. Accurate time-series prediction heavily relies on previous observations. Through its gating mechanism (forget/input/output gates), LSTM efficiently manages information flow, either preserving relevant information in the current state or discarding obsolete or irrelevant information. This intelligent memory management not only prevents the overfitting problem in extended sequences but also focuses attention on meaningful patterns, proving especially valuable when modeling data with prolonged temporal dependencies or cyclical seasonal behavior [34].
  • B. FFT-Based Time Series Preprocessing
Consider a function space X and its transformed counterpart Ψ. Suppose f t ∈ X is a function in the original space, while F ( ω ) ∈ Ψ represents its counterpart in the transformed space.
The function f: R→X maps from the time domain R to space X, with its argument t (in seconds) representing the temporal variable. Correspondingly, the function F: Z→Ψ operates from the complex domain Z to space Ψ, taking frequency ω (in rad/s) as its argument. A functional transformation φ mapping from space X to Ψ is hereby introduced as follows:
φ : X → Ψ
The discrete one-dimensional Fourier transform is a specific instance of the transformation form (7), designed for analyzing sampled data sequences within the temporal domain X. Like its continuous version, the discrete Fourier transform maps signals between the temporal domain X and the spectral domain Z. A distinctive characteristic of this transform lies in its application to discrete data sequences, including both finite and infinite time series. For a finite numerical sequence with N samples, the discrete Fourier transform is mathematically expressed as follows:
X k = ∑ n = 0 N − 1 x n e − i k n 2 π N , k ∈ [ 0 ,   N − 1 ]
  • x: a discrete sequence supplied as input to the transform.
  • Xk: the spectral coefficients obtained through the discrete Fourier transform.
  • N: the total count of samples spanning the discrete sequence.
The spectral parameter collection {Xk|k = 0, …, N − 1} characterizes the frequency composition of the signal. These parameters reside in the domain of complex numbers Z, a space defined by the real and imaginary axes. For any given frequency component Xk within this set, the following relation applies:
X k = R e X k ± i I m X k ,   k ∈ [ 0 ,   N − 1 ]
For efficient transmission and processing of spectral parameters within neural network architectures, their polar coordinate representation is employed:
X k = X k c o s a r g X k + i s i n a r g X k
where
X k = R e ( X k ) 2 + I m ( X k ) 2
a r g X k = a r c t a n I m X k R e X k
Consequently, Formula (9) is replaced with Formulas (10)–(12), wherein each spectral parameter is uniquely characterized by a pair of quantities derived from the complex number’s polar representation.
Formula (11) determines the modulus (complex number magnitude), representing the spectral parameter’s amplitude. Formula (12) computes the argument (complex number angle), indicating the spectral parameter’s phase. These polar coordinates provide clearer frequency characterization, as they explicitly define the signal’s amplitude and phase components. Magnitude and phase values derived from the frequency components are utilized as features for subsequent machine learning-based time series analysis.
  • C. Enhanced FFT–LSTM Architecture
LSTM effectively addresses long-term dependency challenges through its memory cell architecture, making it well-suited for classifying activities characterized by distinct temporal patterns [35]. Consequently, the HAR module proposed in this work leverages LSTM to recognize human activities using accelerometer and gyroscope data acquired from a smart device.
Time-series signals are frequently corrupted by sensor noise, data gaps, and irregular artifacts, all of which can degrade the predictive capability of an LSTM [36,37]. Consequently, careful preprocessing of raw data is essential for ensuring robust and accurate recognition outcomes. A study by [38] introduced an enhanced LSTM variant that integrates an FFT-based preprocessing stage. Through extensive comparative experiments, the authors validated the efficacy of this FFT–LSTM fusion. Subsequent investigations [39,40] have further examined FFT–LSTM architectures across diverse application domains. Nevertheless, these prior approaches depend solely on frequency-domain features derived via FFT, a constraint that may limit recognition accuracy when applied to the complex, multimodal time series data examined in this study. To address this limitation, this study proposes an enhanced FFT–LSTM architecture that jointly exploits both temporal and spectral representations, as illustrated in Figure 3.
Figure 3. Enhanced FFT–LSTM model.
In the proposed architecture, time-domain features and frequency-domain characteristics derived via FFT are jointly supplied to the network, enabling it to simultaneously leverage information from both domains. Consequently, the model gains a richer input representation that encapsulates both temporal and spectral properties.
The following time-domain statistical metrics are employed for feature extraction:
s d = 1 n ∑ i = 1 n a i − a ¯ 2 m a d = 1 n ∑ i = 1 n a i − a ¯ k u = ∑ i = 1 n a i − a ¯ 4 n − 1 s d 4 s k = ∑ i = 1 n a i − a ¯ 3 n − 1 s d 3 r m s = ∑ i = 1 n a i 2 n
where sd, mad, ku, sk, and rms represent the standard deviation, mean absolute deviation, kurtosis, skewness, and root mean square, respectively; ai denotes the i-th data point within the window, a ¯ denotes the mean value of all data points in that window.
The frequency-domain features comprise the three highest peaks of cepstrum coefficients, denoted as C1, C2, and C3. The calculation formula is as follows:
C n = 1 N ∑ k = 0 N − 1 log X k 2 ⋅ e j 2 π k n / N ,   n = 0,1 , … , N − 1
Let xt = [sd, mad, ku, sk] denote the time-domain feature vector, φ t = [C1, C2, C3] represent the frequency-domain feature vector. The operational pipeline of the enhanced FFT–LSTM architecture proceeds as outlined below.
In the forget gate computation, both magnitude and phase components are explicitly incorporated:
f t = σ W f · x t , φ t + U f · h t − 1 + b f
The update steps of cell state are as follows:
i t = σ W i · x t , φ t + U i · h t − 1 + b i
c ^ t = t a h n W c · x t , φ t + U c · h t − 1 + b c
The magnitude and phase components extracted from the time series are subsequently forwarded to the output stage of the network:
o t = σ W o · x t , φ t + U o · h t − 1 + b o
At each step, it can be seen that the proposed model will enhance the neural network’s input by incorporating frequency-domain features while preserving the original temporal characteristics. This approach will help to improve recognition reliability across extended time spans for sequential data.

3.2.2. HAR Using the Proposed FFT–LSTM Model

The model recognizes human activities (i.e., the activities of arms or legs) using accelerometer and gyroscope data from wearable devices such as smartphones or smartwatches. The accelerometer and gyroscope respectively capture tri-axial acceleration and angular velocity along the x, y, and z axes. Following [6], the model processes acceleration values with absolute orientation derived from sensor fusion of accelerations and angular velocities via Mahony’s AHRS algorithm [41]. This absolute orientation, represented as a quaternion relative to the device’s local coordinate system, is computed through the AHRS process. By integrating inertial sensor data, the AHRS process improves LSTM-based action recognition by providing robust orientation estimates while eliminating raw sensor noise and drift.
Sliding window sampling is applied to the output of the AHRS process. The continuous time-series stream is partitioned into fixed-length windows of 2.56 s using this technique, with a 50% overlap maintained between adjacent segments. Each window has 128 readings, resulting in a time step of 128.
The LSTM module used in the model consists of two stacked layers, which are more effective in representing complex sequential relationships than basic single-Layer LSTMs. Figure 4 illustrates the LSTM architecture. The network extracts and transforms the input into new feature representations, which are subsequently passed to a SoftMax classifier that assigns each feature vector to its corresponding activity category.
Figure 4. Architecture of the LSTM.
The LSTM is optimized using a cross-entropy loss augmented with an L2 regularization term, formulated as follows:
l o s s = − 1 K ∑ i = 1 K y i log   y ^ i + λ ∑ j = 1 M ω j 2 2
where K denotes the number of activity classes; yi and y ^ i correspond to the ground-truth and predicted labels for the i-th sample, respectively; λ represents the L2 regularization coefficient; M indicates the total count of trainable weights within the LSTM; and ω j refers to the j-th individual weight parameter. An adaptive moment estimation (ADAM) optimizer was used for network optimization. The hyperparameter configurations for the LSTM are summarized in Table 1.
Table 1. LSTM Configuration Specifications.

3.3. Bio-Inspired Adaptive Multimodal Decision Fusion (BA-MDF) Module

The core of the multimodal safety monitoring framework is its bio-inspired adaptive fusion and decision-making mechanism, which integrates HAR results, heart rate, and location data to accurately detect hazardous worker states. As illustrated in Figure 5, this mechanism is implemented within the BA-MDF module. Activity recognition, heart rate monitoring, and spatial proximity analysis operate as independent modalities with separate thresholds in a temporal window. Their outputs are unified via a weighted fusion mechanism that combines module-specific scores into a single safety assessment.
Figure 5. Bio-inspired adaptive fusion and decision-making mechanism.
The BA-MDF module implements a bio-inspired adaptive fusion mechanism based on the inverse effectiveness principle observed in biological multisensory systems. Unlike conventional fixed-weight fusion, the system dynamically adjusts modality weights based on real-time reliability estimates—mirroring how the superior colliculus enhances weak sensory inputs through cross-modal integration.
  • A. Activity Recognition Fusion
As described in Section 3.2, the activity recognition module leverages an LSTM-based neural network to classify worker activities from IMU data. A 20 s time window is used to monitor activity duration, enabling the detection of prolonged lying (≥20 s) or simultaneous lying by multiple workers. The activity score (Scoreact) is calculated as follows:
S c o r e a c t = 0.1   if   Normal   activity 1.0   if   Prolonged   lying
  • B. Heart Rate Analysis Fusion
Heart rate data is analyzed using HRV metrics, calculated as |Hcurrent − Havg|/Havg. A threshold (θheart = 0.3) identifies significant heart rate fluctuations indicative of physiological stress or distress. The 20 s time window captures temporal heart rate patterns, allowing the system to detect prolonged anomalies (≥10 s) or severe anomalies. The heart rate score (Scoreheart) is assigned as follows:
S c o r e h e a r t = 0.05   if   Normal   heart   rate 0.7   if   prolonged   anomalies 0.9   if   severe   anomaly
This scoring system prioritizes prolonged and severe anomalies, ensuring that physiological stress is promptly detected.
  • C. Spatial Proximity Fusion
Spatial proximity is analyzed using GPS coordinates and the Haversine formula to compute geographic distances between workers. A threshold (Dthreshold = 5 m) defines spatial adjacency. If a worker is in proximity to another anomalous worker, their activity score is weighted by 1.2 (Scoregroup = Scoreact × 1.2). This integration enhances the detection of group-related hazards, such as multiple workers simultaneously lying in close proximity. In situations where GPS is not available, such as in a closed environment, UWB positioning technology can be used as a substitute.
  • D. Bio-inspired Adaptive Fusion Scoring
The system employs a weighted scoring approach to combine activity, physiological, and spatial data:
S c o r e t o t a l = w 1 ( t ) × S c o r e a c t + w 2 ( t ) × S c o r e h e a r t + w 3 ( t ) × S c o r e g r o u p
For the i-th modality at time t, its inverse effective weight w i ( t ) is calculated as:
w i ( t ) = R i ( t ) ∑ j R j ( t )
where R i ( t ) ∈ [ 0,1 ] denotes the real-time reliability of modality i. Following [42], the reliability is computed as the inverse of the normalized signal variance, R i t = 1 / ( σ n o r m 2 + ϵ ) , where ϵ is a small positive constant added for numerical stability. When a modality’s reliability decreases due to environmental interference, its weight automatically decreases, compelling the system to rely on alternative reliable modalities. Consequently, the benefits of multimodal integration are actually enhanced. This adaptive mechanism ensures robust fusion under sensor degradation without requiring manual threshold tuning.
  • E. Decision Mechanism
The decision mechanism triggers alarms based on the total score:
S = 0   ( Normal   state ) if   S c o r e t o t a l ≤ 0.3 1   ( Level   1   alarm ) if   0.3 < S c o r e t o t a l ≤ 0.6 2   ( Level   2   alarm ) if   S c o r e t o t a l > 0.6
Normal state only needs routine monitoring; Level 1 alarm suggests moderate risks that need monitoring; and Level 2 alarm indicates serious hazards requiring immediate attention. This multi-level alarm system ensures that the most critical hazards are prioritized while maintaining sensitivity to moderate risks.
The multimodal safety monitoring process, which integrates the HAR and BA-MDF modules, is formalized in Algorithm 1.
Algorithm 1 Multimodal Safety Monitoring
Input: IMU sensor data D, Heart rate data H, GPS coordinates C
Output: Anomalous state flag S
1. Initialize parameters:
      - Time window T = 20 s
      - Activity history record Histact = []
      - Heart rate history record Histheart = []
2. Process IMU sensor data D:
      - Preprocess IMU sensor data D
      - Predict activity P using the enhanced FFT-LSTM model
      - If P is lying
        * Record lying start time tstart
        * Monitor lying duration Dur = tcurrent − tstart
      - Update Histact
3. Analyze heart rate data H:
      - Get current heart rate Hcurrent
      - Calculate HRV = | Hcurrent − Havg|/Havg
      - If HRV > θheart, mark as heart rate anomaly
      - Update Histheart
4. Detect GPS proximity:
      - Get current GPS coordinates C
      - Calculate distance D to nearby workers using Haversine formula
      - If D ≤ Dthreshold, mark as nearby worker
5. Bio-inspired adaptive fusion scoring using Equation (22)
6. Anomalous state decision making using Equation (24), and return S
Unlike classic adaptive fusion techniques that often require pre-defined thresholds or heuristic rules for weight adjustment [27,28], the proposed BA-MDF formalizes weight adaptation through a continuous, uncertainty-driven mechanism (Equation (23)) inspired by biological sensory integration. This provides a principled mathematical framework for weight dynamics. For instance, in contrast to static weighted averaging or switching logic, our model offers a smooth, gradient-like transition of influence based on real-time uncertainty estimates.
The BA-MDF method is distinguished from standard reliability-weighted fusion by three principal differences. First, its weight adaptation rule is directly derived from the neuroscientific inverse effectiveness principle, providing a biological justification for the fusion strategy rather than an empirical or optimization-based one. Second, weights are continuously adjusted based on real-time uncertainty estimates, not just triggered by binary sensor failure flags. Third, it is specifically designed to maintain performance under partial sensory degradation (simulating conditions like smoke, dust, or signal loss common in confined spaces), a scenario where conventional fusion often fails catastrophically, as validated in our experiments. This makes BA-MDF not merely a heuristic rule-set but a formally modeled, bio-inspired adaptive fusion framework for safety-critical environments.

4. Results and Discussion

4.1. Performance Evaluation of HAR

4.1.1. Dataset Description and Preprocessing

The HAR experiments using the enhanced FFT–LSTM model were conducted on Reyes’s dataset [43], which covers six activity categories: walking, ascending stairs, descending stairs, sitting, standing, and lying. Measurements were obtained from accelerometer and gyroscope sensors embedded in smartphones secured at the waist of participating volunteers. All signals were sampled at a frequency of 50 Hz. To address overfitting problems, as in [35], the dataset was randomly divided into two subsets for the LSTM process, with 70% for training and the remaining 30% for evaluation.
Some sample data in the dataset are shown in Figure 6. For the original data shown in Figure 6, Figure 7 depicts the result of the AHRS process.
Figure 6. Sample accelerometer and gyroscope data.
Figure 7. AHRS results for sample data in Figure 6.

4.1.2. Comparative Analysis

The performance of the enhanced FFT–LSTM model for HAR was benchmarked against several established approaches. The primary baseline is a classical time-domain LSTM architecture [44], which processes accelerometer signals derived from the AHRS preprocessing pipeline. The second baseline is the FFT–LSTM model proposed in [38], which operates entirely in the frequency domain. This approach transforms time-series signals by substituting the original temporal sequence with a compact frequency-domain representation. Specifically, it utilizes the FFT results computed from the AHRS-processed accelerometer signals as input to the LSTM. Our approach leverages both time-domain and frequency-domain representations, combining the AHRS-processed accelerometer signals with their corresponding FFT-derived spectral features.
Figure 8 presents a comparative summary of the performance across different LSTM variants. The FFT–LSTM model outperforms the original LSTM by approximately 5% across all four evaluation metrics—accuracy, precision, recall, and F1 score. Furthermore, the enhanced FFT–LSTM achieves an even more substantial improvement, surpassing the original LSTM by roughly 11% on the same metrics.
Figure 8. Performance of different versions of the LSTM.
The training time also varies with the version of LSTMs. Figure 9 illustrates the accuracy trajectories of different LSTM variants across training epochs. The original LSTM attains 90% accuracy by the 21st epoch and further climbs to 95% at the 24th epoch. In contrast, the FFT–LSTM reaches 90% accuracy considerably earlier, at the 8th epoch, and achieves 95% by the 13th epoch. The enhanced FFT–LSTM reaches 90% accuracy at the 10th epoch and further improves to 95% by the 13th epoch. Consequently, both the FFT–LSTM and the enhanced FFT–LSTM reduce training time by approximately 50% relative to the original LSTM.
Figure 9. Training accuracy of different versions of LSTM over training epochs.
Generally, the enhanced FFT–LSTM demonstrates dual improvements in HAR performance and computational efficiency, as validated by its superior accuracy/F1 scores (Figure 8) and a 50% reduction in training time compared to baseline methods (Figure 9).
The confusion matrices in Table 2 demonstrate the classification performance using different versions of LSTM. As can be seen from the confusion matrices, the proposed approach attains superior performance across all six activity categories, although recognition accuracy varies substantially among individual activities. Lying achieves the highest recognition rate in the matrices, possibly due to its distinct triaxial acceleration and angular velocity patterns compared to other activities. Sitting and Standing exhibit a modest degree of mutual confusion within the matrices, though the overall misclassification rate between them remains relatively low.
Table 2. Confusion matrices using different versions of LSTM.

4.1.3. Impact of Hidden Unit Count on Model Performance

To examine the influence of the hidden node count—a critical hyperparameter in the proposed FFT–LSTM architecture—on HAR performance, a dedicated experiment was conducted. The corresponding results are presented in Figure 10.
Figure 10. Recognition accuracies with different hidden counts.
With only eight hidden nodes, the model yields a modest accuracy of 82%. Expanding the hidden node count from 8 to 64 drives accuracy above 94%. Beyond this point, further increments produce negligible gains. Balancing predictive performance against computational cost—since additional hidden nodes prolong training duration—a configuration of 32 hidden nodes is adopted as optimal for the proposed FFT–LSTM model for HAR tasks.

4.2. System Experiments

The system was validated under controlled industrial conditions to detect prolonged supine postures (≥20 s) and co-occurring multi-worker hazards. Performance was assessed using data from simulated enclosed workspaces, with detection accuracy, false alarm rate, and temporal response latency as key metrics. All experiments were conducted on a Nuvo-6108GC embedded industrial computer (Intel® Core™ i7-6700TE, NVIDIA® GeForce GTX 1050Ti GPU, 32GB RAM) running Python 3.8, which was sourced from Neousys Technology Inc., New Taipei City, China.

4.2.1. Experimental Design and Data Collection

A controlled test environment was established at a power construction site. Wearable IMUs were deployed to capture six categories of typical worker behaviors (Figure 11) from 12 construction personnel (aged 25–45) under real-world working conditions:
Figure 11. Typical behaviors in real-world scenarios.
-
Walking (average velocity: 0.8–1.2 m/s)
-
Standing (including static working postures)
-
Lying (simulating accidental falls)
-
Sitting (resting state)
-
Ascending stairs (30-degree incline)
-
Descending stairs (30-degree incline)
Data sampling was configured at 50 Hz over a 6 h continuous acquisition period, yielding 1550 validated data samples.
Dual-triggering criteria were set for the safety warning system:
-
Single-person lying state ≥ 20 s (medically confirmed threshold for unconsciousness)
-
More than two simultaneous lying postures in the detection zone (preventing mass incidents)

4.2.2. Experimental Results and Discussion

  • A. Field Validation of the Model Performance
The enhanced FFT–LSTM model developed in this study was applied to the collected data for behavior recognition, with five-fold cross-validation revealing an overall accuracy of 92.4% (±1.7%), the highest recognition accuracy for lying postures (96.2%), and a misjudgment rate of 7.3% for stair motions, primarily occurring under tool-carrying conditions.
  • B. Field Validation of Anomaly Detection
During on-site accident simulation at the construction site, the system successfully identified all pre-configured emergency scenarios, including five individual unconsciousness events and two simulated group fall incidents. The alarm response latency was maintained under 800 ms, while the average rescue arrival time following triggered alerts was reduced to 43 s. Notably, the proposed multimodal monitoring system remains fully operational in zero-light or smoke-filled confined spaces where conventional video surveillance is inherently non-functional.
The adaptive fusion mechanism demonstrated particular robustness under simulated sensor degradation. When GPS signals were artificially attenuated (underground scenario), the system automatically increased IMU and heart rate weights, maintaining 94.3% (±1.8%) detection accuracy versus 78.2% (±3.1%) for fixed-weight fusion—validating the bio-inspired inverse effectiveness design. The results are reported with 95% confidence intervals and standard deviations across five runs.
To further validate the advantage of adaptive fusion, the full BA-MDF pipeline was compared against three alternative fusion strategies under identical conditions. As summarized in Table 3, BA-MDF achieved 94.3% accuracy, substantially outperforming fixed-weight fusion (78.2%), majority voting (81.6%), and Bayesian fusion (85.9%). This 8.4–16.1% improvement confirms that the bio-inspired adaptive weighting mechanism provides a decisive advantage over both static and classical probabilistic fusion approaches, particularly under partial sensor degradation.
Table 3. Full-pipeline fusion strategy comparison.
Several limitations of the field validation should be noted: all findings derive from controlled simulations rather than real-world deployment, and the modest dataset (12 participants) warrants broader validation. Expanded real-world testing is planned.

5. Conclusions

To ensure operational safety in confined spaces, this study proposes an intelligent anomalous situation detection method based on multimodal data analysis. The proposed safety monitoring framework consists of two core components: the human activity recognition (HAR) module and the bio-inspired adaptive multimodal decision fusion (BA-MDF) module.
The BA-MDF module collects and processes multimodal sensor data—including accelerometer, gyroscope, heart rate, and GPS signals—from wearable devices like smartphones or smartwatches to monitor human activities. HAR serves as the fundamental technology for human activity monitoring. To improve HAR performance, an enhanced FFT–LSTM architecture was developed that integrates both time-domain and frequency-domain features. Experimental results for HAR demonstrate that this model significantly outperforms baseline methods in terms of accuracy, precision, recall, and F1-score metrics.
The BA-MDF module integrates HAR outputs, HRV trends, and geospatial proximity constraints into a deterministic rule-based scoring system. It dynamically evaluates risk by mapping activity states (e.g., prolonged lying, elevated HRV) against context-aware safety thresholds and triggers tiered alerts via an adaptive fusion mechanism.
The system demonstrated a 92.4% overall accuracy in real-world HAR testing. During simulated accident scenarios, the system achieved reliable identification accuracy for all pre-defined emergency cases (including unconsciousness and fall events) with inference latency below 800 ms. The average rescue arrival time following triggered alerts was reduced to 43 s. Critically, the proposed multimodal monitoring system remains fully operational in environments where conventional video surveillance is inherently non-functional. The experimental results demonstrate that the proposed method can effectively reduce risks from anomalous events in visually restricted confined spaces, where conventional surveillance systems face operational constraints, thereby enhancing overall safety management in such challenging environments.
Beyond technical contributions, this work demonstrates the value of bio-inspired design principles in industrial AI systems. The inverse effectiveness mechanism—borrowed from neuroscience—provides a theoretically grounded solution to sensor degradation that would be difficult to discover through purely empirical optimization. Future work will explore additional biological principles, such as spike-based efficient coding and neuromodulatory gain control, for next-generation safety monitoring systems.
The thresholds (e.g., θheart = 0.3, Dthreshold = 5 m) and scoring values (e.g., 0.1, 0.7, 1.0) used in the BA-MDF module were determined empirically during system development to demonstrate the framework’s feasibility. While effective in the controlled experiments, these parameters may benefit from further theoretical optimization or data-driven calibration to improve generalizability across diverse environments. Future work will focus on establishing a more rigorous theoretical basis for these parameters, potentially through probabilistic modeling or machine learning techniques, to enhance the system’s adaptability and performance.

Author Contributions

X.L.: conceptualization, methodology; Z.L.: software, writing—original draft; H.Z.: investigation, funding acquisition; Q.Z.: supervision, writing—review and editing. All authors have read and agreed to the published version of the manuscript.

Funding

The authors gratefully acknowledge the financial support from the Science and Technology Project of China Southern Power Grid Company Limited (Project No. 032000KC23110038 (GDKJXM20231238)).

Institutional Review Board Statement

The human subjects shown in Figure 11(4) are student volunteers from our university laboratory. The experiments were conducted in a simulated on-campus environment for human motion recognition, not involving real patients.

Data Availability Statement

The original contributions presented in this study are included in the article. The full source code is not publicly available due to its integration into a broader proprietary software ecosystem but may be made available from the corresponding author upon reasonable request.

Conflicts of Interest

Authors Xinhai Li, Zhibin Lian, and Heng Zhou were employed by the company Guangdong Power Grid Co., Ltd. The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Sim, H.; Kim, H. Establishment of a Real-Time Risk Assessment and Preventive Safety Management System in Industrial Environments Utilizing Multimodal Data and Advanced Deep Reinforcement Learning Techniques. Int. J. Adv. Sci. Eng. Inf. Technol. 2025, 15, 328–337. [Google Scholar] [CrossRef] [Scilit]
  2. Yang, M.; Wu, C.; Guo, Y.; Jiang, R.; Zhou, F.; Zhang, J.; Yang, Z. Transformer-based deep learning model and video dataset for unsafe action identification in construction projects. Autom. Constr. 2023, 146, 104703. [Google Scholar] [CrossRef] [Scilit]
  3. Munid, A.; Raghdah, S.A.; Mohammad, I. Effectiveness of Machine Learning on Human Activity Recognition Using Accelerometer and Gyroscope Sensors: A Survey. In Proceedings of the 26th World Multi-Conference on Systemics, Cybernetics and Informatics (WMSCI 2022), Virtual, 12–15 July 2022; pp. 1–6. [Google Scholar]
  4. Chevalier, G. LSTMs for Human Activity Recognition. Available online: https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition (accessed on 17 March 2026).
  5. Wang, L.; Liu, R. Human Activity Recognition Based on Wearable Sensor Using Hierarchical Deep LSTM Networks. Circuits Syst. Signal Process. 2020, 39, 837–856. [Google Scholar] [CrossRef] [Scilit]
  6. Park, H.; Kwon, E.; Byon, S.; Shin, W.-J.; Jung, E.-S.; Lee, Y.-T. Danger detection based on fusion analysis of multimodal data from a smartphone and smartwatch. In Proceedings of the 2019 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Republic of Korea, 16–18 October 2019; pp. 1143–1145. [Google Scholar]
  7. Hakim, A.; Huq, M.S.; Shanta, S.; Ibrahim, B. Smartphone based data mining for fall detection: Analysis and design. Procedia Comput. Sci. 2017, 105, 46–51. [Google Scholar] [CrossRef] [Scilit]
  8. Mauldin, T.R.; Canby, M.E.; Metsis, V.; Ngu, A.H.; Rivera, C.C. SmartFall: A smartwatch-based fall detection system using deep learning. Sensors 2018, 18, 3363. [Google Scholar] [CrossRef] [Scilit]
  9. Wu, X.; Cheng, L.; Chu, C.-H.; Kim, J. Using deep learning and smartphone for automatic detection of fall and daily activities. In Proceedings of the Smart Health: International Conference, ICSH 2019, Shenzhen, China, 1–2 July 2019; pp. 61–74. [Google Scholar]
  10. Keeffe, H. Wrist-Based Device Accelerometer Analysis for Establishing Thresholds of Various Violent Attacks. Master’s Thesis, Murdoch University, Perth, Australia, 2018. [Google Scholar]
  11. Franck, T. Recognition and Classification of Aggressive Motion Using Smartwatches. Master’s Thesis, Université d’Ottawa/University of Ottawa, Ottawa, ON, Canada, 2018. [Google Scholar]
  12. Kamal, S.; Alshehri, M.; AlQahtani, Y.; Alshahrani, A.; Almujally, N.A.; Jalal, A.; Liu, H. A novel multi-modal rehabilitation monitoring over human motion intention recognition. Front. Bioeng. Biotechnol. 2025, 13, 1568690. [Google Scholar] [CrossRef] [Scilit]
  13. ProHawk. Overcoming Video Degradation from Light Absorption by Particulates with AI-enabled Computer Vision. Available online: https://prohawk.ai/nvidia-review/ (accessed on 17 March 2026).
  14. Rehouma, H.; Boukadoum, M. Fall Detection by Deep Learning-Based Bimodal Movement and Pose Sensing with Late Fusion. Sensors 2025, 25, 6035. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  15. Islam, M.M.; Nooruddin, S.; Karray, F.; Muhammad, G. Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things. Inf. Fusion 2023, 94, 17–31. [Google Scholar] [CrossRef] [Scilit]
  16. Hu, X.; Yu, S.; Zheng, J.; Fang, Z.; Zhao, Z.; Qu, X. A hybrid CNN-LSTM model for involuntary fall detection using wrist-worn sensors. Adv. Eng. Inform. 2025, 65, 103178. [Google Scholar] [CrossRef] [Scilit]
  17. Mae, Y.; Yuki, R.; Kojima, M.; Arai, T. Extraction of mental stress scene in driving car by wearable heart rate sensor. In Proceedings of the 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR), Shenyang, China, 24–27 August 2018; pp. 480–485. [Google Scholar]
  18. Pataca, A.O.; Zdravevski, E.; Coelho, P.J.; Garcia, N.M.; Deryuck, M.; Albuquerque, C.; Pires, I.M. Use of machine learning for predicting stress episodes based on wearable sensor data: A systematic review. Comput. Biol. Med. 2025, 198, 111166. [Google Scholar] [CrossRef] [Scilit]
  19. Ganesan, P.; Thota, Y.R.; Shehata, H.; Nikoubin, T. TinyML Based Stress Detection utilizing PPG Signals: A Lightweight Approach for Smart Wearable Devices. In Proceedings of the Great Lakes Symposium on VLSI 2025, New Orleans, LA, USA, 30 June–2 July 2025; pp. 941–946. [Google Scholar]
  20. Shterev, V. Enhancing Stress Detection Through HRV in Wearable Technology. Bachelor’s Thesis, University of Twente, Enschede, The Netherlands, 2025. [Google Scholar]
  21. Oxmaint. Wearable Technology for Maintenance Workers: Safety and Productivity. Available online: https://oxmaint.com/blog/post/wearable-technology-maintenance-workers-safety-productivity-smart-devices (accessed on 17 March 2026).
  22. Bi, L.; Liu, Y.; Zhong, D.; Wen, L. Multi-Objective Real-Time Planning of Evacuation Routes for Underground Mine Fires. Appl. Sci. 2024, 14, 7521. [Google Scholar] [CrossRef] [Scilit]
  23. Casha, O. A Comparative Analysis and Review of Indoor Positioning Systems and Technologies. In Innovations in Indoor Positioning Systems (IPS); Sabban, A., Ed.; IntechOpen: London, UK, 2024. [Google Scholar]
  24. SafetyCulture. What is man-Down Detection, and Why Is It Vital for Lone Worker Safety. Available online: https://safetyculture.com/topics/man-down-system (accessed on 17 March 2026).
  25. Group, T.V. Lone Worker Safety Devices for Employee Protection. Available online: https://vestigeview.com/lone-worker-safety-devices/ (accessed on 17 March 2026).
  26. OKALONE. Man Down Alarm Devices for Lone Worker Safety. Available online: https://www.okaloneworker.com/solutions/man-down/man-down-alarm/ (accessed on 17 March 2026).
  27. Xie, D.; Zhang, X.; Gao, X.; Zhao, H.; Du, D. MAF-Net: A multimodal data fusion approach for human action recognition. PLoS ONE 2025, 20, e0319656. [Google Scholar] [CrossRef] [Scilit]
  28. Fuad, Z. Improving Human action Recognition Using Decision Level Fusion of Classifiers Trained with Depth and Inertial Data. Master’s Thesis, Sabanci University, İstanbul, Turkey, 2018. [Google Scholar]
  29. Boiler World Update. Raksha-IoT: Revolutionizing Worker Safety in Confined Spaces with RFID and BLE Beacon Technology. Available online: https://boilerworldupdate.com/raksha-iot-revolutionizing-worker-safety-in-confined-spaces-with-rfid-and-ble-beacon-technology/ (accessed on 17 March 2026).
  30. Corporate. 5G IoT and AI: Managing Confined Spaces and Enhancing Safety. Available online: https://www.1010corporate.com/en/news-updates/edigest-5g-iot-ai-manage-confined-space-safety/ (accessed on 17 March 2026).
  31. Stein, B.; Stanford, T. Multisensory integration: Current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 2008, 9, 255–266. [Google Scholar] [CrossRef] [Scilit]
  32. Bean, N.L.; Stein, B.E.; Rowland, B.A. Cross-modal exposure restores multisensory enhancement after hemianopia. Cereb. Cortex 2023, 33, 11036–11046. [Google Scholar] [CrossRef] [Scilit]
  33. He, X.; Dongcheng, Z.; Li, Y.; Kong, Q.; Yang, X.; Zeng, Y. Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence. arXiv 2025, arXiv:2505.10176. [Google Scholar] [CrossRef] [Scilit]
  34. Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  35. Bahadur, E.H.; Masum, A.K.M.; Barua, A.; Alam, M.G.R.; Chowdhury, M.A.U.Z.; Alam, M.R. LSTM based approach for diabetic symptomatic activity recognition using smartphone sensors. In Proceedings of the 2019 22nd International Conference on Computer and Information Technology (ICCIT), Dhaka, Bangladesh, 18–20 December 2019; pp. 1–6. [Google Scholar]
  36. Usmani, M.; Memon, Z.A.; Zulfiqar, A.; Qureshi, R. Preptimize: Automation of Time Series Data Preprocessing and Forecasting. Algorithms 2024, 17, 332. [Google Scholar] [CrossRef] [Scilit]
  37. Chen, J.; Yang, Z. Revolutionizing Time Series Data Preprocessing with a Novel Cycling Layer in Self-Attention Mechanisms. Appl. Sci. 2024, 14, 8922. [Google Scholar] [CrossRef] [Scilit]
  38. Hachemi, M.L.; Ghomari, A.; Hadjadj-Aoul, Y.; Rubino, G. Mobile traffic forecasting using a combined FFT/LSTM strategy in SDN networks. In Proceedings of the 2021 IEEE 22nd International Conference on High Performance Switching and Routing (HPSR), Paris, France, 7–10 June 2021; pp. 1–6. [Google Scholar]
  39. Houenou, A.G.H.; Nounangnonhou, C.T.; Agbokpanzo, R.G.; Didavi, K.B.A.; Sedjro, J.; Agbomahena, B.M. Hybrid FFT-Hyperband-LSTM model for direct short-term PV power forecasting. In Proceedings of the 2024 IEEE 10th International Conference on Smart Instrumentation, Measurement and Applications (ICSIMA), Bandung, Indonesia, 30–31 July 2024; pp. 42–47. [Google Scholar]
  40. Zhao, J.; Liu, N.; Ye, J.; Liu, C.; Sun, F.; Lv, L. CEEMD-FFT-BiLSTM based model for power line loss prediction. J. Phys. Conf. Ser. 2024, 2846, 012003. [Google Scholar] [CrossRef] [Scilit]
  41. Ludwig, S.A.; Burnham, K.D.; Jiménez, A.R.; Touma, P.A. Comparison of attitude and heading reference systems using foot mounted MIMU sensor data: Basic, Madgwick, and Mahony. In Proceedings of the Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2018, Denver, CO, USA, 5–8 March 2018; pp. 644–650. [Google Scholar]
  42. Jeong, S.; Park, J.; Imani, M. Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection. arXiv 2025, arXiv:2505.02393. [Google Scholar]
  43. Reyes-Ortiz, J. Human activity recognition using smartphones data set. Available online: https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+using+smartphones (accessed on 17 March 2026).
  44. Prater, R.; Hanne, T.; Dornberger, R. Generalized Performance of LSTM in Time-Series Forecasting. Appl. Artif. Intell. 2024, 38, 2377510. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.