Safety Behavior Recognition for Substation Operations Based on a Dual-Path Spatiotemporal Network

Abstract

The integration of large-scale renewable energy sources has increased the complexity of operation and maintenance in modern power systems, causing on-site substation operation and maintenance activities to exhibit stronger continuity and dynamics, and thereby placing higher demands on real-time operational perception and safety judgment. However, existing behavior recognition methods have difficulty accurately identifying operational states in complex scenarios involving continuous actions, partial occlusions, and fine-grained manipulations. To address these challenges, this paper proposes a safety behavior recognition method for substation operations based on a dual-path spatiotemporal network. Personnel localization is achieved using YOLOv8, while behavior classification is performed through the SlowFast framework. In the Slow pathway, an ECA attention mechanism is integrated with residual structures to enhance the representation of sustained operational postures. In the Fast pathway, a multi-path excitation residual network is introduced to fuse temporal, channel, and motion information, improving the multi-scale representation of local action variations. Furthermore, to mitigate the issue of class imbalance in substation operation data, Focal Loss based on binary cross-entropy is incorporated to adaptively down-weight easily classified samples. Experimental results demonstrate that the proposed method achieves a recognition accuracy of 87.77% and an F1-score of 85.56% across multiple operation scenarios. The results further indicate improved recognition stability and adaptability, supporting safe substation operation and maintenance in renewable energy-integrated power systems.

1. Introduction

With the rapid integration of large-scale renewable energy, the complexity of operation and dispatch in modern power systems has increased significantly. As a result, substation operation and maintenance activities exhibit more frequent task switching and enhanced process continuity, placing higher demands on on-site operational safety as well as the standardization and stability of operational behaviors. Traditional manual inspection and monitoring approaches have difficulty achieving real-time and accurate identification of operator behaviors in complex operational environments, thereby constraining further improvements in risk prevention and control capabilities [1,2]. In recent years, incidents such as the 2023 Yuanjiang “11.15” accident, the 2024 Nanjing “7–27” accident, and the 2024 Lanzhou “5.14” and “9.12” accidents have resulted in severe casualties and substantial negative societal impacts [3,4]. Therefore, the development of efficient behavior recognition methods tailored to substation operational scenarios is essential for enhancing on-site safety awareness and risk prevention capabilities.

Current research on behavior recognition algorithms mainly falls into three categories: pose-estimation-based methods, object-detection-based methods, and video action recognition methods [5,6,7,8]. For example, Liu et al. [9] proposed a lightweight pose estimation method combined with the YOLO framework and employed a spatiotemporal graph convolutional network (ST-GCN) to classify skeleton sequences, achieving behavior recognition for live working activities in distribution networks. Wang et al. [10] constructed a skeleton-sequence-driven violation detection model based on a graph convolutional network, achieving detection of irregular operator actions. However, pose-estimation-based methods usually rely on complete visibility and accurate detection of key points. When operators are partially occluded or undergo large pose variations, skeleton key points are prone to deviation or loss, which significantly limits the overall recognition performance. Li et al. [11] constructed a safety generative pre-trained Transformers model, which encodes visual and textual features and uses a cross-attention mechanism to achieve multimodal feature alignment and fusion, thereby completing automatic recognition and response generation for unsafe behaviors. Chandramouli et al. [12] fused CNN and BiLSTM to construct a downsampled action recognition model for early warning of abnormal behaviors, but it remains limited under occlusion, motion blur, and small sample conditions. In contrast, video action recognition methods can extract complementary spatial and temporal features through two-stream structures and exhibit stronger capability in representing dynamic behaviors. For example, Wei et al. [13] proposed a dual-attention feature fusion module, CMDA, which enhances spatiotemporal information interaction between the two pathways of the SlowFast network and improves video action recognition performance. Chen et al. [14] proposed the action granularity pyramid network (AGPN), which builds a hierarchical action granularity pyramid module and fuses multi-granularity spatiotemporal features to improve feature representation in complex spatiotemporal scenarios. However, the recognition objectives of the aforementioned improved methods remain primarily focused on clip-level behavior classification, and their performance gains are mainly attributed to enhanced inter-path feature interactions. These approaches do not explicitly address the recognition stability challenges arising from behavior state transitions in continuous operation scenarios.

In typical substation environments, Ma et al. [15] addressed the complex multi-factor interactions involved in distribution network operations by proposing a ladder-climbing behavior recognition method based on semantic feature analysis. By extracting and analyzing semantic features within the operational scene, their approach enabled accurate identification of interactions between personnel and ladders. Feng et al. [16] employed pose sequence quantization features in combination with spatiotemporal graph convolutional networks (ST-GCNs) for spatial feature extraction and further incorporated object detection tools to identify equipment usage states, thereby achieving precise recognition of operational behaviors. Zhang et al. [17] proposed a vision-based method for recognizing high-altitude work behaviors that integrates human pose estimation with a one-dimensional convolutional neural network (1D CNN). The method extracts action features through a human key points encoding structure, enabling automatic recognition of high-altitude operations even under partial occlusion. As shown in

Table 1. Performance comparison of behavior recognition models in power operation scenarios.

Reference	Model	Recognized Behaviors	AP/%	Parameters/M	GFLOPs	Limitations
[9]	YOLOv8n-Pose + ST-GCN	Live working	88.00	6.23	4.50	Not optimized or validated under complex background clutter and occlusion conditions
[15]	ResNet101 + Transformer	Ladder climbing	91.2	121.40	202.30	Limited behavior categories and insufficient generalization capability
[16]	OpenPose + ST-GCN + YOLOv5s	Live working	88.9	—	—	Human pose estimation is constrained under occlusion conditions
[17]	PCT-1DCNN	High-altitude operation	90.3	—	—	Limited generalization to other behavior types

, although existing methods can achieve relatively high recognition accuracy in specific scenarios, they are mostly designed for single task types. Such methods inadequately model the sequential structure of continuous posture variations and fine-grained local actions that are prevalent in substation operations, and their adaptability remains limited under complex conditions, such as partial personnel occlusion and multi-person collaboration.

In summary, this study investigates the temporal characteristics of continuous operational behaviors involving the alternating coupling of long-term posture states and short-term local actions and extends the recognition objective to the temporal perception and cooperative discrimination of continuous operation behavior transitions. Accordingly, taking power maintenance scenarios as the application context, this study proposes a dual-path spatiotemporal network-based method for recognizing safety behaviors in substation operations. The proposed aims to simultaneously capture long-duration continuous actions and critical fine-grained local variations of operators, thereby meeting the dual requirements of high accuracy and high efficiency in practical applications. The main contributions of this work are as follows:

• To model the temporal structure formed by continuous posture variations and the alternation of fine-grained local actions, this paper proposes a behavior recognition method based on a dual-path spatiotemporal network. By combining static target localization from YOLOv8 with dynamic behavior recognition from SlowFast, the method achieves accurate detection and continuous action analysis of operator behaviors in substation operation scenarios for both rapid movements and long-duration sequences, thereby improving multi-scenario adaptability.
• To address the problems of partial occlusion of operators and limited computational efficiency, an attention mechanism is integrated with the original residual block in the Slow Pathway to form a lightweight ECA-Res module. This design enables efficient channel weight allocation, thereby strengthening the temporal representation capability of long-duration operational postures, while effectively mitigating occlusion-induced interference in dynamic scenes and reducing overall network complexity.
• Considering that operator behaviors exhibit continuity and similarity, a multi-path excitation residual network (SCM-Res) is introduced in the Fast Pathway. By performing weighted fusion of spatial–temporal, channel, and motion information, this module effectively captures fine-grained temporal variations of rapid behaviors between adjacent frames and enhances multi-scale feature fusion capability for fast motions.
• To address the class imbalance problem in substation operation datasets, a binary cross-entropy-based Focal Loss function is designed to effectively reduce the loss weight of easily classified samples and improve the attention and recognition ability for infrequent classes.

2. Basic Theory of Behavior Recognition

2.1. YOLOv8 Network

During behavior detection, target detection results directly influence the extraction of spatiotemporal behavioral features, thereby affecting subsequent operator behavior classification. In addition, the detection speed of the target detection algorithm also affects the efficiency of the overall video behavior detection. SlowFast, a 3D convolutional neural network model for video behavior recognition and detection tasks, usually adopts the Faster R-CNN as the target detector [18,19]. However, Faster R-CNN has limitations in capturing rich spatiotemporal information from video data and requires more fine-grained features to further improve classification accuracy. Compared with the Faster R-CNN algorithm, YOLOv8 is superior in terms of detection speed, accuracy and target tracking [20,21]. Therefore, YOLOv8 is selected as the object detector in this study and integrated with the improved SlowFast model to provide accurate and efficient target localization for operation behavior detection.

Figure 1 illustrates the network architecture of YOLOv8. The backbone part of YOLOv8 consists of Conv module, C2f module and SPPF module. The C2f module mainly consists of two convolutional layers and multiple bottleneck layers, which adjust the number of different channels for different scale models, so that YOLOv8 can obtain a lightweight structure while capturing richer gradient flow information. The SPPF module uses a serial structure of MaxPool2d, which reduces the number of parameters while preserving the ability to aggregate multi-scale features [22]. The head adopts a decoupled structure that separates the classification and localization branches and replaces the traditional anchor-based mechanism with an anchor-free strategy that directly predicts object locations and sizes. This design enables YOLOv8 to capture information from targets at various scales, thereby improving the accuracy of detecting power operation personnel within bounding boxes.

2.2. SlowFast Network

In most cases, human actions are completed within a short duration, whereas the surrounding environment remains largely stable or undergoes only slow variations during the action period. Based on this observation, Feichtenhofer et al. proposed the two-stream 3D convolutional network SlowFast, which models slow spatial semantic information (such as color and shape) and fast temporal dynamic information separately to mimic the human visual system’s selective attention to motion cues of different speeds during action recognition [23].

The structure of this network is shown in Figure 2. The Slow Pathway operates with low temporal and high spatial resolution, mainly capturing the spatial semantic information in the video. The Fast pathway operates with high temporal and low spatial resolution, mainly capturing the dynamic information of the rapid change of the behavior, and adopts smaller convolutional width to lower the channel capacity in order to reduce the amount of computation. The length of the video processed by SlowFast network is τ × T frames each time (τ denotes the time step, and T denotes the number of sampled frames). In the Slow Pathway, frames are sampled with a step size of τ (τ is usually taken as 16), inputting T frames into the network. The Fast Pathway samples αT frames into the network according to the step τ/α (α is usually taken as 8). Meanwhile, in order to reduce the computational effort, the Fast Pathway adopts a smaller convolution width, usually with a frame rate ratio of 1: α (where α > 1) and a channel number ratio of 1: β (where β > 1). The network fuses the dual frames by connecting them multiple times horizontally. The network fuses the features of the dual-stream branches through multiple lateral connections, and finally, the fused output is pooled as a global average and used as an input to the fully connected classifier.

For the action detection of onsite personnel, the extraction of global spatial information by the SlowFast network helps determine the interaction between workers and surrounding equipment [24]. The SlowFast network structure parameter table is shown in

Table 2. SlowFast network architecture parameters.

Stage	Slow Pathway	Fast Pathway	S/F Output T × S2
Raw clip	—	—	64 × 2242
Data layer	Stride16, 12	Stride2, 12	4/32 × 2242
Conv1	1 × 72, 64	5 × 72, 64	4/32 × 1122
Pool1	1 × 32, max	1 × 32, max	4/32 × 562
Res2	{1 × 12, 64 1 × 32, 64 1 × 12, 256} × 3	{3 × 12, 8 1 × 32, 8 1 × 12, 32} × 3	4/32 × 562
Res3	{1 × 12, 128 1 × 32, 128 1 × 12, 512} × 4	{3 × 12, 64 1 × 32, 64 1 × 12, 256} × 4	4/32 × 282
Res4	{3 × 12, 256 1 × 32, 256 1 × 12, 1024} × 6	{3 × 12, 32 1 × 32, 32 1 × 12, 128} × 6	4/32 × 142
Res5	{3 × 12, 512 1 × 32, 512 1 × 12, 2048} × 3	{3 × 12, 64 1 × 32, 64 1 × 12, 256} × 3	4/32 × 72
Global average pool, concate, fc			#classes

, with ResNet50 as the backbone network. Here, T is the number of time-dimensioned video frames, S is the image size of the video frames, and Res₂-Res₅ denotes the residual modules.

3. Methods

3.1. Method Overview

To address the need for identifying safe work behaviors in substation field operations, this paper proposes a video-aware dual-path spatiotemporal network. Using video frames captured by surveillance cameras as input, the proposed method consists of three stages: worker region extraction, temporal feature learning, and behavior classification. In complex substation operation scenarios, operation behaviors exhibit diversity in temporal scale and action patterns. Operation processes typically involve long-duration postural states as well as short-duration, localized operational behaviors. To accommodate these characteristics, this study enhances the SlowFast network through three key modifications, with the overall network architecture illustrated in Figure 3.

Specifically, when video is input into the model, the video frames are cut at a rate of 30 frames per second, with frames extracted at both 1 frame per second and 30 frames per second. At the data layer, frame sequences with different temporal resolutions are generated using two distinct temporal sampling intervals and are routed to the Slow Pathway and the Fast Pathway, respectively. The Slow Pathway adopts a lower temporal sampling rate of 1 frame per second (FPS), corresponding to a temporal sampling interval of τ = 16. Under a 30 FPS video setting, one frame is sampled every 16 frames, corresponding to a temporal sampling interval of approximately 0.5 s. This pathway is used to model operational postural states that require sustained observation over longer time windows for reliable recognition, such as standing and pole climbing operations. In contrast, the Fast Pathway processes video frames at a higher frame rate of 30 FPS, enabling the capture of rapid and localized motion details that occur within short-duration intervals, such as squatting movements. The frames then enter the 3D residual network, where the Slow Pathway and Fast Pathway use ECA-Res and SCM-Res modules, respectively, to extract motion and spatial information from the operator’s actions. Meanwhile, frames at the position of (τ × T)/2 are selected as key frames within that time period, which are then input into the YOLOv8 model to detect the operator’s location. The results are fed into the RoIAlign module, which projects the bounding boxes onto the 3D feature map to obtain the corresponding feature matrices. By expanding each 2D RoI along the time dimension, a 3D RoI is formed. Finally, after pooling and fully connected layers, the feature maps of uniform dimensions are classified using a Sigmoid classifier to recognize the behavior of onsite personnel.

3.2. Improvements to the SlowFast Network Architecture

3.2.1. Multi-Path Excitation Residual Network

The Fast Pathway captures operators’ rapid and fine-grained behaviors at a higher frame rate, and its feature variations are more dependent on short-term correlations within local spatial regions. However, in substation maintenance scenarios, different task categories exhibit certain similarities in local motion patterns. Under these conditions, directly adopting a standard 3D residual structure in the Fast Pathway tends to over-smooth rapidly changing local behaviors, thereby reducing its ability to distinguish fine-grained behavioral differences. Therefore, this study utilizes three complementary attention modules from ACTION—namely spatial-temporal excitation (STE), channel excitation (CE), and motion excitation residual (ME) [25]—combined with the original residual block to form the multi-path excitation residual network (SCM-Res). By strengthening the representation of spatial context, this design enables the network to better focus on localized differences in fast behaviors, thereby improving the accuracy of recognition. The structure of SCM-Res is illustrated in Figure 4. In this study, the SCM-Res block consists of three convolutions and a residual edge, with the ACTION module added before each convolution and residual connection. This allows the network to capture multi-dimensional spatiotemporal patterns, channel information, and motion data, which are then convolved to obtain finer-grained features.

The structure of the STE module is shown in Figure 5a. The STE module effectively captures the spatiotemporal dynamic features of power operation personnel behaviors, enhancing the spatial and temporal relationships in the video and improving the model’s ability to recognize behaviors at different scales. It primarily generates a spatiotemporal mask M to produce a spatiotemporal attention map, which is then used to extract spatiotemporal features from the video.

Traditional spatiotemporal feature extraction typically uses 3D convolutions. However, directly using 3D convolutions significantly increases the computational load of the model. Therefore, for the input features $X\in {ℝ}^{N\times T\times C\times H\times W}$, the STE module first performs global pooling along the channel dimension to obtain a compact channel-level spatiotemporal representation. Here, N, T, C, H and W denote the batch size, temporal length, number of channels, height, and width, respectively. The features are then rearranged to ensure compatibility with 3D convolution operations. The processed features are subsequently fed into a 3D convolutional layer with a kernel size of 3 × 3 × 3 to capture local spatiotemporal correlations, thereby generating enhanced spatiotemporal feature representations map. The mathematical expression is given by Formula (1).

$$F_{\mathrm{o}1}^{*}=K\times F_1^{*}$$

(1)

Here, K denotes the three-dimensional convolution kernel and $F_1^{*}$ represents the intermediate spatiotemporal feature representation obtained from the input features after global pooling along the channel dimension and subsequent dimension rearrangement.

The resulting spatiotemporal feature map is activated using a Sigmoid function to generate a spatiotemporal attention mask, which is then fused with the original input features through element-wise weighting. In addition, a residual connection is introduced to preserve the original information, yielding the final output Y₁ of the STE module, as defined in Equation (2).

$$Y_1=X+X\odot M_1$$

(2)

The CE attention module is used to extract appropriate channel-level features to emphasize information from different channels within the network. The CE module is similar to the squeeze and excitation (SE) attention mechanism [26]. The key difference lies in the incorporation of temporal information inherent in video actions. Specifically, a one-dimensional convolution layer is introduced between two fully connected layers to capture temporal variations in channel features, thereby enhancing the temporal interdependence among behavior-related feature representations. The structure of the CE module is shown in Figure 5b. The structure of the CE module is shown in Figure 5b.

For a given input $X\in {ℝ}^{N\times T\times C\times H\times W}$, a global spatial information tensor $F_2\in {ℝ}^{N\times T\times C\times 1\times 1}$ is obtained through average pooling, and the global spatial information tensor is defined as follows:

$$F_2={\displaystyle \frac{1}{H\times W}}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^{W}X\left[:,:,:,i,j\right]}}$$

(3)

Subsequently, a lightweight convolution is applied to the channel features F₂ for feature mapping and compression, where a channel reduction ratio r is used to reduce the feature dimensionality and obtain a new channel representation F_r. The representation F_r is then reshaped to incorporate temporal information, and a one-dimensional convolution is applied to generate an intermediate feature representation F_temp.

Finally, the intermediate feature representation F_temp is projected using a two-dimensional convolution kernel K₁ of size 1 for feature expansion, followed by a Sigmoid activation to generate an action mask for the operator. The resulting mask is then multiplied element-wise with the original features, and the outputs are fused via a residual connection to produce the final output Y₂ of the CE module.

$$Y_2=X+X\odot \delta \left(K_1\times F_{temp}\right)$$

(4)

Here, K₁ denotes a two-dimensional convolution kernel of size 1 × 1, δ (•) denotes the Sigmoid activation function, and ⊙ denotes element-wise multiplication.

The ME attention module primarily focuses on extracting motion-related information between two adjacent frames, in order to better capture the motion features of operators’ behaviors during power operations. Its structure is shown in Figure 5c. For a given input, the dimensionality is first reduced using a 1 × 1 convolutional kernel K₂, and then the features of adjacent frames F_m is calculated using Equation (5):

$$F_m=K_2\times F_r[:,t+1,:,:,:]-F_r[:,t,:,:,:]$$

(5)

In the formula, F_m represents the motion feature map, $F_r\left[:,t,:,:,:\right]$ and $F_r[:,t+1,:,:,:]$ represent the features of the current frame and the previous frame, respectively.

Subsequently, the differences between each pair of adjacent frames are aggregated along the temporal dimension to obtain F_M. The resulting features are then processed through spatial average pooling and lightweight convolutional mapping to generate an intermediate response F_o3 for motion attention. A Sigmoid activation is applied to F_o3 to produce a motion mask, which is used to reweight the input features. Finally, a residual connection is employed to yield the output Y₃, as shown in Equation (6).

$$Y_3=X+X\odot \delta \left(F_{o3}\right)$$

(6)

The feature information of operation behaviors is processed through the STE, CE, and ME attention mechanisms. The three generated excitation features are then element-wise summed to obtain the final output Y of the multi-path excitation residual network. The calculation process is shown in Formula (7):

$$Y=Y_1+Y_2+Y_3$$

(7)

3.2.2. ECA-Res

In maintenance scenarios, the working environment is often complex and variable, with potential occlusions caused by surrounding objects or background clutter, which can hinder the accurate capture of changes in the operator’s movements. Such problems may affect the accuracy of behavior detection, and therefore effective mechanisms need to be employed to enhance the ability to capture critical channel information in response to the interference of environmental factors. In this context, the ECANet attention mechanism was proposed based on the SENet attention mechanism [27]. ECANet adopts a cross-channel interaction strategy without dimensionality reduction, enabling more effective information exchange among channels. Compared with traditional attention mechanisms, it introduces only a small number of parameters while maintaining computational efficiency, thereby simplifying the model structure and reducing overall complexity.

Because the Slow Pathway operates at a lower frame rate to capture pose variations over extended time intervals, its global channel features play a more critical role. Accordingly, integrating the ECA module into the Slow Pathway enhances the effectiveness of channel-wise information interaction, enabling more accurate modeling of fine-grained spatial and channel characteristics—particularly in complex operational environments—while mitigating interference from background noise. Its network structure is shown in Figure 6.

The network structure diagram of the ECA attention module is shown in Figure 7. The model is able to adapt itself to the size of the convolution kernel during training. The specific approach is as follows:

• The feature map is first compressed through convolution to obtain a new feature map χ, which is then processed by global average pooling to produce a 1 × 1 × C vector.
• Calculate the size of the adaptive one-dimensional convolution kernel as shown in Equation (8).

  $$k=\phi \left(C\right)=\left|{\displaystyle \frac{{\log }_2\left(C\right)}{\gamma }}+{\displaystyle \frac{b}{\gamma }}\right|$$

  (8)

  where γ and b are hyperparameters, with γ = 2 and b = 1; k represents the kernel size, and C denotes the channel size. The parameters γ and b govern the scaling relationship between the convolution kernel size and the number of channels. This design maintains a small and stable odd-sized convolution kernel across different channel scales, thereby achieving a good balance between channel-wise coverage and computational efficiency.
• Use the adaptive convolution kernel into the one-dimensional convolution to get the weights of each channel, allowing layers with larger number of channels to interact more between neighboring channels.
• The generated weights σ are multiplied channel-by-channel with the original feature map χ to obtain the final channel-weighted feature map $\tilde{\chi }$. This process enhances the response of important channels while suppressing background noise or unimportant information.

3.2.3. Focal Loss Function

There are more checking electricity behaviors and standing behaviors of the operators in the dataset, while the number of behaviors such as ladder climbing and squatting is small, which leads to an obvious category imbalance problem. So, the original CE Loss function for some small samples of the operator’s behavioral accuracy is not guaranteed [28,29]. In order to improve this problem, this paper applies the Focal Loss function to replace the original loss function, by adding a moderating factor to the standard cross-entropy criterion to reduce the relative loss of the well-classified samples, and to improve the classification performance for the few samples of the class, so as to help the model to better deal with the problem of category imbalance [30,31], in order to quickly obtain accurate training results. The formula of Focal Loss is formulated as follows.

$$FL(p_t)=-{\alpha }_t{(1-p_t)}^{\gamma }\log (p_t)$$

(9)

where p_t is the predicted probability score of the model, α_t is the weighting factor and ${\alpha }_t\in \left[0,1\right]$, ${\left(1-P_t\right)}^{\gamma }$ is the adjustment factor, and γ is the adjustment parameter. When P_t tends to 1, the sample is easy to classify samples, at this time the adjustment factor approaches 0, indicating a smaller contribution to the loss, that is, reduces the proportion of the loss of easy to classify samples; when P_t tends to 0, meaning a sample is misclassified as a positive sample, at this time the adjustment factor tends to 1, increasing the proportion of the loss of difficult to classify samples.

However, Focal Loss only addresses the class imbalance problem and does not explicitly capture behavior states across consecutive video frames. Therefore, this paper introduces a Temporal Smoothing Loss (TSL) based on Focal Loss, aiming to enhance the model’s ability to maintain continuity in behavior prediction by penalizing discrepancies between predictions of adjacent frames. The Temporal Smoothing Loss is defined as follows:

$$L_{\mathrm{smooth}}={\displaystyle \frac{1}{T-1}}{\displaystyle \sum _{\mathrm{t}=1}^{T-1}{‖f_t-f_{t+1}‖}_1}$$

(10)

Let T be the total number of frames in a video segment, ${‖·‖}_1$ denotes the L1 norm, f_t and f_t+1 represent the output feature vectors of adjacent frames.

The total loss function is composed of the weighted sum of Focal Loss and Temporal Smoothing Loss:

$$L_{Loss}=FL\left(p_t\right)+\lambda L_{smooth}$$

(11)

where λ is the weighting coefficient for the smoothing term.

4. Experimental Design and Result Analysis

4.1. Dataset Construction

In this paper, a dataset was constructed based on videos collected from power maintenance scenarios involving electric power operators at a substation in a specific city. The dataset contains 1520 video clips with a total of 10,139 behavior annotations, covering six typical operation behaviors. It includes both indoor and outdoor scenes, different lighting conditions, as well as single-operator and multi-operator cooperative scenarios. The dataset composition is shown in

Table 3. Composition of the behavior dataset.

Behavior Category	Videos		Labels	Annotation Strategy
	Train	Test
Check electricity	307	77	2296	Operator holds a voltage detector and performs electricity-checking behaviors near equipment.
Stand	406	102	3615	Operator holds a voltage detector and checks for the absence of voltage to ground near the equipment.
Squat	77	19	554	Operator adopts a crouched posture with a lowered center of gravity.
Ladder climb	64	16	520	Operator maintains contact with a ladder while moving upward or downward, or remaining stationary.
Live working	283	71	2580	Operator performs maintenance, testing, or operational tasks in close proximity to energized equipment.
Pole climb	78	20	574	Operator climbs along a utility pole or performs tasks while in contact with the pole.
Total	1215	305	10,139	——

. Regarding the class distribution, stand and check electricity account for 33.4% and 25.3% of the dataset, respectively, indicating relatively sufficient samples. In contrast, ladder climb and squat represent only 5.3% and 6.3%, respectively, and thus have comparatively fewer samples.

The data preprocessing pipeline is as follows: (a) segment the video in the form of 30 frames per second; (b) extract key frames in the video according to the step size of 1s, and input these key frames into the YOLOv8 target detector to automatically annotate the positional information of electric power operators in the image; (c) the detected bounding boxes of power workers were manually refined using the VIA annotation tool, after which behavior category labels were assigned to generate AVA format dataset annotation files [32]. Behavior annotations in this dataset were performed by a single annotator. To reduce potential subjectivity associated with individual annotation, label quality was constrained through standardized behavior definitions, a unified annotation workflow, and a combination of automated pre-detection and manual refinement, with the corresponding behavior discrimination criteria summarized in Table 3. In addition, under the same annotation guidelines, a second annotator independently conducted frame-level behavior annotation. Inter-annotator agreement was evaluated using Cohen’s Kappa coefficient, yielding an agreement rate of 84.95% with k = 0.72.

Regarding data partitioning, the dataset is randomly split at the video level into training and testing sets, with 80% of the videos used for model training and the remaining 20% for evaluation. Due to privacy and security management constraints, the dataset is not publicly available at this time. But this paper provides detailed descriptions of the data construction process and behavior definitions to ensure research reproducibility.

4.2. Experimental Environment

This study primarily focuses on algorithm design and performance evaluation, and all experiments were conducted in PC computing environment. The hardware platform was built on a general-purpose computing system to support model training and inference on video data for the proposed method. The experimental platform operated on a 64-bit Windows 11 operating system and was equipped with an NVIDIA GeForce RTX 4060 Ti GPU with 8 GB of memory. Regarding the software environment, the experiment utilized Python version is 3.8, and the deep learning framework used is PyTorch 1.10.0. The GPU acceleration libraries include CUDNN 8.2.0 and CUDA 11.3. The model was trained for a total of 300 epochs, with a batch size of 8. The model training process employed the SGD optimizer. Based on the variation in the loss curve during the initial training phase and balancing the requirements for convergence speed and overfitting prevention, the initial learning rate and weight decay coefficient were set to 0.1125 and 0.00001, respectively. For the learning rate scheduling strategy, a cosine annealing scheme was applied to achieve smooth decay, thereby maintaining stable convergence in the later stages of training.

4.3. Experimental Results and Analysis

4.3.1. Ablation Studies

To evaluate the contributions of the SCM-Res and ECA-Res modules, ablation experiments are conducted. The detection results for the four behavior categories are shown in

Table 4. Ablation study results.

Model	SCM-Res	ECA-Res	Accuracy of Different Behaviors Under Operating Conditions (%)						mAP /%	Params /Million	GFLOPs/G
			Check Electricity	Stand	Squat	Ladder Climb	Live Working	Pole Climb
1	×	×	81.03	83.71	75.03	79.91	83.27	82.19	80.86	33.568	53.10
2	×	√	83.57	84.93	75.43	82.01	83.68	84.13	82.29	33.572	53.12
3	√	×	86.97	88.65	80.25	84.99	85.77	87.79	85.74	33.631	53.30
4	√	√	88.92	90.83	82.44	87.90	87.36	89.15	87.77	33.634	53.37

, where √ indicates that the module was used, and × indicates that the module was not used.

From Table 4, it can be seen that the original model (i.e., Model 1) has an average recognition accuracy of 80.86% for the behaviors “check electricity,” “squat,” “ladder climb,” and “live working,” showing only moderate performance. In Model 2, Introducing the ECA-Res module leads to a 1.43% improvement in mAP@0.5, with notable gains in the “check electricity” and “pole climb” categories, indicating enhanced channel-wise feature discrimination. In Model 3, the SCM-Res module is introduced into the Fast Pathway, which improves the mAP@0.5 by 4.88% compared to the original model. Specifically, the average accuracy for “squat” increased by 5.22%, demonstrating its effectiveness in capturing spatiotemporal dependencies. Model 4 combines both the SCM-Res and ECA-Res modules, leveraging the enhanced capabilities of both pathways to achieve superior overall performance. Compared to the original model, the mAP@0.5 increased by 6.91%, and the average precision for each behavior category exceeding that of the previous models. The synergistic and complementary effects of these two attention mechanisms led to cumulative performance improvements. From the comparison results of model parameters (Params) and GFLOPs, it can be observed that the introduction of the SCM-Res and ECA-Res modules leads to only marginal increases in both the number of parameters and computational cost. Specifically, both the parameter count and computational cost increase by less than 1%, indicating that the proposed model achieves effective performance improvement while maintaining controlled model size and computational complexity.

The overall mean average precision variation curves are shown in Figure 8, where the horizontal axis represents the number of iterations and the vertical axis indicates the overall mAP value. Each curve corresponds to the performance of the respective model. From the trend of the curves, SlowFast-SCM-ECA converges faster than both SlowFast-SCM and SlowFast-ECA, and achieves slightly better performance in mAP@0.5 compared to other models. This indicates that the joint application of SCM and ECA modules provides complementary enhancement of spatial and channel features, demonstrating stronger adaptability and discriminative power in complex action recognition tasks. Consequently, the model achieves notable improvements in both feature extraction and recognition capabilities.

4.3.2. Stability and Robustness Analysis

To quantitatively evaluate the recognition stability of the proposed model across different operational behaviors, the bootstrap resampling method was employed to analyze the recognition results for each behavior category, as summarized in

Table 5. Stability of recognition accuracy for different behaviors.

Behavior Category	Variance	Std/%	95% CI/%
Check electricity	0.00021	1.45	[86.10, 91.63]
Stand	0.00018	1.34	[88.42, 93.10]
Squat	0.00039	1.97	[78.66, 85.98]
Ladder climb	0.00026	1.61	[84.82, 90.77]
Live working	0.00015	1.22	[85.01, 89.74]
Pole climb	0.00031	1.76	[85.90, 92.11]

. The results indicate that the standard deviations of recognition performance for all behavior categories are constrained within a narrow range of 1–2%, and the corresponding 95% confidence intervals are relatively concentrated, without exhibiting pronounced fluctuations.

Building upon the above analysis, Figure 9 presents the confusion matrix results of the proposed method for six substation operation behaviors. The observed confusion primarily occurs between behaviors with similar short-duration motion patterns, such as squatting. These misclassifications are mainly attributed to transitional postures and overlapping motion characteristics.

Additionally, this paper constructs a subset of the dataset under different illumination levels, camera viewpoints, and motion intensity conditions. Specifically, illumination enhancement was implemented using linear brightness scaling, and a controlled brightness enhancement factor of 1.3 was applied to video frames during the testing phase to simulate potential lighting variations encountered in real substation environments. Camera viewpoints were divided into frontal and rear views. Motion conditions were categorized based on the continuity of behaviors during operation into continuous motion and intermittent motion. With the model architecture and parameters kept unchanged, quantitative evaluations were conducted on these condition-specific subsets, and the results are summarized in

Table 6. Robustness evaluation under different operating conditions.

Scenario Factor	Condition	Accuracy of Different Behaviors Under Operating Conditions (%)
		Check Electricity	Stand	Squat	Ladder Climb	Live Working	Pole Climb
Lighting	Original	90.24	92.39	83.70	87.95	87.65	90.31
	Brightness enhancement	89.44	91.10	83.50	87.44	87.40	89.85
Camera viewpoints	Front view	89.15	92.72	84.15	86.83	87.46	90.21
	Back view	88.91	91.21	82.14	84.42	85.43	89.96
Motion Pattern	Continuous motion	90.51	91.45	83.81	88.58	87.34	89.42
	Intermittent motion	89.32	91.70	83.23	87.75	86.78	89.10

.

4.3.3. Analysis of Experimental Results on Public Datasets

Additional comparative experiments were conducted on two widely used public behavior recognition benchmarks, HMDB51 and UCF101, as summarized in

Table 7. Experimental results of different behavior recognition models on public datasets.

Model	Accuracy/%
	HMDB51	UCF101
Slow Only	66.8	80.3
TimeSformer	69.5	82.5
AMP-HOI	76.8	89.9
Proposed method	77.6	91.6

. The results show that the recognition accuracies of all compared models are generally higher on UCF101 than on HMDB51, which is consistent with the advantages of UCF101 in terms of video quality and dataset scale. The proposed method achieves superior recognition accuracy among all models, indicating that it is not limited to specific substation operation scenarios and can maintain stable performance under different data distribution conditions.

4.3.4. Comparative Experiments

In order to verify the superiority of the improved SlowFast model proposed in this study for the safety behavior recognition effect of electric power operators, this paper selects different behavioral detection techniques and model architectures to conduct comparative experiments with the proposed improved model. Figure 10 presents the visualization results of various models, while

Table 8. Comparison results with other methods.

Model	Accuracy of Different Behaviors Under Operating Conditions (%)						mAP/%
	Check Electricity	Stand	Squat	Ladder Climb	Live Working	Pole Climb
Slow Only	78.92	87.77	74.36	76.75	84.67	71.03	78.92
TimeSformer	84.17	85.98	77.33	80.29	85.37	76.42	81.59
AMP-HOI	87.30	88.71	79.50	84.31	86.59	82.52	84.82
Proposed method	88.92	90.83	82.44	87.90	87.36	89.15	87.77

summarizes the comparative experimental results of recognition accuracy across different models.

The comparison models include mainstream architectures such as Slow Only, TimeSformer, and AMP-HOI, each demonstrating their respective strengths and limitations in the behavior recognition task. Slow Only is a model that performs behavior recognition using only slow-frame sequences, emphasizing the extraction of fine-grained features from high-quality images. However, by neglecting temporal dynamic features, it struggles to provide a comprehensive understanding of complex behaviors, resulting in an mAP@0.5 value of 78.92%. TimeSformer employs a global attention mechanism to capture long-range temporal dynamics, demonstrating superior recognition capability in tasks involving complex temporal dynamics, such as “squat” and “ladder climb”. Its mAP@0.5 value reaches 81.59%. AMP-HOI integrates human–object interaction detection with temporal segment prediction, explicitly modeling the interactions between humans and objects to enhance the understanding of fine-grained behaviors. This approach attains an mAP@0.5 of 84.82%, yet it demonstrates limited effectiveness in recognizing fast dynamic behaviors such as “squat”. In contrast, the method proposed in this study exhibits notable advantages in key feature representation as well as in capturing long-term temporal dependencies and fine-grained local dynamics, achieving the highest mAP@0.5 of 87.77% among all compared methods.

Table 9. Performance of different models on multiple metrics.

Model	Precision/%	Recall/%	F1-Score/%	Params/M	GFLOPs	FPS (Frame/s)
Slow Only	78.64	76.68	77.64	31.63	63.92	101.50
TimeSformer	81.72	77.82	79.72	115.76	146.54	75.20
AMP-HOI	82.91	80.00	81.43	265.16	129.83	82.60
Proposed method	89.07	82.47	85.56	33.63	53.37	117.90

presents the performance of the proposed method and the comparison models in terms of Precision, Recall, and F1-Score. The experimental results show that the proposed method achieves Precision, Recall, and F1-Score values of 89.07%, 82.47%, and 85.56%, respectively, attaining a good balance between precision and recall. Meanwhile, the proposed method outperforms all comparison models across all metrics, fully demonstrating its superior target-capturing capability and recognition accuracy.

Additionally, as shown by the computational efficiency comparison results in Table 8, the main issues with the TimeSformer and AMP-HOI models lie in its high computational complexity, large memory consumption, and slow inference speed, making them less suitable for scenarios with high real-time requirements. The memory usage of each trained model weight file is approximately three times that of the proposed model, significantly increasing the computational and storage burden during deployment and inference. In contrast, the proposed method reduces computational and storage overhead while maintaining recognition performance.

In summary, the proposed substation operation safety behavior recognition method based on a dual-path spatiotemporal network is capable of jointly modeling spatiotemporal information and dynamic behavioral features, achieving an mAP@0.5 of 87.77%. Both the recognition accuracy and computational efficiency outperform all comparative models mentioned above, demonstrating the effectiveness and superiority of the proposed approach in complex substation operation scenarios.

4.3.5. Operational Scenario Testing

The proposed method combines features from both slow and fast frame sequences and incorporates the ACTION and ECA attention mechanisms. Figure 11 presents the detection results for different operation types. It can be observed that the proposed method accurately identifies both the personnel and their behaviors across various operation categories and maintains stable performance under different scene conditions, confirming the effectiveness of the method.

Meanwhile, to thoroughly evaluate the effectiveness of the proposed method under various challenging conditions, multiple experiments were conducted on our self-constructed dataset. As shown in Figure 12a, for fast-motion categories such as “check electricity,” “ladder climb,” and “pole climb,” video segments containing drastic motion changes were selected for evaluation, highlighting the model’s capability to capture high-dynamic behaviors and subtle motion variations. As shown in Figure 12b, for partial-occlusion cases involving “squat” and “live working,” test data containing localized obstructions such as buildings or trees were employed, reflecting the model’s robustness when parts of the target are missing. As shown in Figure 12c, for complex scene interference, datasets incorporating diverse backgrounds and long-range targets were used to validate the model’s discriminative ability. As shown in Figure 12d, to examine class imbalance, experiments were conducted on video segments of “ladder climb” and “pole climb” with highly skewed class distributions, demonstrating the model’s stability in recognizing small-sample categories. In summary, the experimental results across these challenging scenarios intuitively verify that the proposed method maintains prediction accuracy and adaptability when addressing practical issues such as rapid motion, partial occlusion, complex backgrounds, and class imbalance.

5. Conclusions

In the context of deep integration of renewable energy and the new power system, power system operation and control have become increasingly complex, resulting in substation operation and maintenance scenarios characterized by strong action continuity, partial occlusions, and multi-operator collaboration. Therefore, this paper proposes a substation operation safety behavior recognition method based on a dual-path spatiotemporal network, which effectively improves behavior detection accuracy by processing continuous action changes and fine-grained differences. The main conclusions are as follows:

• YOLOv8 is first used to replace the original Faster R-CNN region detection head, enabling precise localization of operators in video frames and providing reliable spatial information for subsequent behavior feature extraction and recognition.
• By incorporating the ECA attention mechanism into the Slow Pathway and the multi-path excitation residual network into the Fast Pathway, the two pathways form a complementary structure during multi-scale spatiotemporal feature fusion, thereby enabling effective representation of long-term postural states and rapid dynamic behaviors in complex operation scenarios.
• A binary cross-entropy-based Focal Loss is designed to balance the weight distribution of positive and negative samples across different categories, supporting more stable recognition performance for categories with limited samples.

The experimental results indicate that the proposed method maintains stable recognition performance under representative substation operating modes, with an overall mAP of 87.77% and an F1-score of 85.56%, while operating at approximately 118 frames per second with moderate computational complexity. The results validate the effectiveness of the proposed method in achieving stable recognition of complex substation operation and maintenance behaviors.

6. Limitations

Although the proposed substation operation safety behavior recognition method based on a dual-path spatiotemporal network achieves promising experimental performance, several aspects of the method remain to be further improved in the context of deep integration of renewable energy and the new power system. First, the current model mainly relies on human motion features and does not explicitly model human–object interactions between operators and tools, which are important for accurate safety risk assessment. Second, while the model shows relatively high inference efficiency on GPU platforms, real-world deployment on resource constrained edge devices may still be limited by computational and power constraints. Future work will focus on incorporating model compression techniques, such as pruning and quantization, to improve deployment efficiency.

Building upon the above considerations, future work is expected to proactively identify and intervene in potential hazardous situations at an early stage, thereby enabling proactive risk prevention and meeting the growing demand for intelligent safety operation and maintenance under the integrated operation of renewable energy and the new power system.