Improving Safety in High-Altitude Work: Semantic Segmentation of Safety Harnesses with CEMFormer

Abstract

The symmetry between production efficiency and safety is a crucial aspect of industrial operations. To enhance the identification of proper safety harness use by workers at height, this study introduces a machine vision approach as a substitute for manual supervision. By focusing on the safety rope that connects the worker to an anchor point, we propose a semantic segmentation mask annotation principle to evaluate proper harness use. We introduce CEMFormer, a novel semantic segmentation model utilizing ConvNeXt as the backbone, which surpasses the traditional ResNet in accuracy. Efficient Multi-Scale Attention (EMA) is incorporated to optimize channel weights and integrate spatial information. Mask2Former serves as the segmentation head, enhanced by Poly Loss for classification and Log-Cosh Dice Loss for mask loss, thereby improving training efficiency. Experimental results indicate that CEMFormer achieves a mean accuracy of 92.31%, surpassing the baseline and five state-of-the-art models. Ablation studies underscore the contribution of each component to the model’s accuracy, demonstrating the effectiveness of the proposed approach in ensuring worker safety.

1. Introduction

In industries such as power and construction, workers frequently need to perform tasks at heights. Such work poses a risk of falls, which may result in considerable harm to workers’ health and safety. To mitigate these risks, regulations mandate the use of personal protective equipment, such as safety harnesses. However, workers may tend to improperly wear safety harnesses for various reasons. Among those injured due to falls, the proportion of workers not wearing safety harnesses correctly is notably high, reaching up to 95.9%, as reported by Anantharaman et al. [1]. To better safeguard workers’ health and safety, continuous monitoring is crucial to ensure proper use of safety harnesses during work at heights, and to promptly issue reminders or warnings upon detecting incorrect usage.

Currently, monitoring at engineering sites can be managed by installing surveillance cameras and aggregating the captured video data, enabling a small number of personnel to supervise a large number of workers. Devices such as loudspeakers or radios can be used to issue timely reminders or alerts to workers. However, continuously monitoring multiple screens and observing numerous workers’ conditions places a significant mental burden on supervisors, which may lead to a gradual decline in efficiency and an increase in erroneous judgments.

There exists a symmetry between production efficiency and production safety. While the application of artificial intelligence in production to enhance efficiency is significant and holds great potential, it is equally important to explore the use of AI for safety supervision. In this context, replacing or supplementing human supervision of workers’ safety conditions with machine vision offers a viable alternative. In recent years, advancements in deep learning have led to the gradual application of neural network models across various industries, resulting in continuous improvements in the accuracy of machine vision tasks. Semantic segmentation is a task within machine vision that aims to classify each pixel in an image. It provides pixel-level delineation of object boundaries, precisely outlining their shapes. Given the complex backgrounds in high-altitude work environments, where workers and safety harnesses are relatively small and vary in shape, semantic segmentation is particularly suitable for this scenario.

Previously, we employed a deep learning-based object detection approach to individually detect ground workers, aerial workers, and safety harnesses [2]. Given that safety harnesses are relatively small objects for bounding-box-based object detection, which hinders model learning, and considering the potential to improve dataset annotations, we re-annotated the dataset using semantic segmentation masks. Additionally, we introduced a new class for safety ropes, representing the part of the harness that is secured between the body and the anchor point, to distinguish between correct usage and incorrect usage when the harness is worn but not secured. With the new dataset, we explored the potential of semantic segmentation for harness detection and proposed a novel model, CEMFormer. The primary contributions of this paper are as follows:

• Given that safety ropes are often overlooked by workers and researchers due to their small scale, we propose a new annotation principle by adding safety ropes as a new category in the dataset and utilizing masks for precise annotation.
• ConvNeXt is utilized as the backbone; as a next-generation convolutional neural network (CNN) with Transformer-like characteristics, it demonstrates improved accuracy over its predecessor, ResNet.
• Efficient Multi-Scale Attention (EMA) is integrated at the end of the backbone, combining channel and spatial information in the feature maps to provide the segmentation head with enriched feature representations.
• Mask2Former is utilized as the segmentation head; it is a versatile architecture designed for various image segmentation tasks, demonstrating excellent performance on our new dataset.
• Poly Loss is employed as the classification loss, with parameters adjusted according to the dataset to improve the model’s accuracy during training.
• Log-Cosh Dice Loss is utilized as part of the mask loss to address gradient issues associated with the original Dice Loss, leading to more effective training.

CEMFormer was compared to its baseline and five state-of-the-art (SOTA) models. Ablation studies were conducted to demonstrate the effectiveness of the proposed performance improvements.

In summary, our approach, which incorporates simplified annotations to enhance attention to safety ropes, establishes new detection criteria to protect workers from fall hazards. We have experimentally validated the effectiveness of our annotation method and demonstrated the efficacy of semantic segmentation in detecting safety ropes. This foundational work paves the way for future research aimed at ensuring worker safety.

2. Related Works

2.1. Vision-Based Safety Monitoring

For the critical issue of production safety, numerous researchers have applied the rapidly advancing machine vision technology to safety monitoring. Xiong and Tang [3] employed a lightweight OpenPose model for pose estimation to identify various parts of the human body. They treated the head and upper body as local attention regions for detecting safety helmets and vests and utilized a shallow CNN-based classifier to determine whether the corresponding safety gear was being worn. Khan et al. [4] utilized Mask R-CNN for instance segmentation of workers and scaffolding, subsequently analyzing the presence and use of safety outriggers on scaffolds through an object association detection module to assess worker safety. Gong’s team [5] enhanced YOLOv3 with random erasing data augmentation to detect workers on offshore platforms, applied a regional multi-person pose estimation algorithm to locate keypoints, and then utilized an improved ResNet50 model for deep transfer learning to determine whether workers were wearing helmets and work clothes.

Riaz et al. [6] combined Unet and Swin Transformers to detect helmets and vests, capturing high-quality feature representations. Subsequently, a segmentation self-attention mechanism was employed for the precise localization of personal protective equipment (PPE). Shi and colleagues [7] integrated global attention, a bidirectional feature pyramid, SimC2f, and GhostConv into YOLOv8n for detecting helmets, goggles, vests, and masks, achieving an accuracy improvement with a 13.6% reduction in computational cost. Ludwika and Rifai [8] developed a dataset containing seven types of PPE—helmets, lab coats, safety shoes, masks, protective goggles, earmuffs, and gloves—and two states of proper or improper use for each. They evaluated YOLOv4, YOLOv5, and YOLOv6 on this dataset, with YOLOv5 (trained for 100 epochs) demonstrating the best performance among the three models.

Zhang et al. [9] developed a dataset for six types of PPE used in substations and utilized an improved YOLOv8n, incorporating multi-scale channel attention, EC2f, GhostConv, and adaptive spatial feature fusion modules for PPE detection. The enhanced model achieved a 2.4% increase in accuracy while reducing computational cost by 7.3%. Zaidi’s team [10] utilized YOLOv8 to detect helmets, boots, and vests worn by workers, then employed a rule-based module to verify compliance based on detection results and coordinates, followed by a temporal analysis module to reduce false positives in dynamic environments. Sanjeewani et al. [11] extracted features using MobileNet V2 and conducted detection using SSD, achieving real-time PPE detection on edge devices. Although the proposed model was less accurate than six contemporary models, it demonstrated the highest recall and speed. Chen and Demachi [12] utilized OpenPose for human pose estimation, YOLOv3 for helmet and mask detection, and subsequently compared the PPE and human keypoint positions to determine whether nuclear plant workers were properly wearing PPE. Tang et al. [13] utilized Faster R-CNN to detect workers and PPE, such as helmets and safety harnesses, followed by a human–object interaction model to determine PPE compliance. Although safety harnesses, a key component of fall protection systems, were mentioned, detection results for them were not provided.

The aforementioned studies evaluate worker safety using machine vision methods. Among these, Khan et al. [4] specifically focused on scaffolding used by workers, whereas the other studies concentrated primarily on common PPE. PPE, such as safety helmets, is suitable for various engineering sites, providing protection to workers by preventing head injuries to some extent, thereby contributing to their safety.

2.2. Safety Harness Monitoring Based on Deep Learning

In the literature [1], it is stated that falls are the most common cause of severe injuries in the construction industry. Furthermore, in the power industry, which is of particular interest, falls are also the most frequent cause of fatalities [14]. Safety harnesses are effective in reducing casualties resulting from falls; however, research on safety harnesses remains relatively limited compared to other PPE, such as helmets. Ma et al. [15] applied channel pruning and layer pruning to YOLOv4, enabling the model to run on mobile devices for real-time detection of helmets, reflective vests, and safety harnesses. Additionally, fine-tuning was performed to minimize the impact of pruning on model accuracy. Chen and Demachi [16] adopted a similar approach to that described in the literature [12] to determine whether workers were wearing PPE correctly. Unlike the literature [12], they included goggles and safety harnesses as detection targets in the construction site context, addressing different potential hazards compared to those in nuclear power plants.

Fang’s team [17] utilized Faster R-CNN to detect worker positions, subsequently inputting the detected content into a deep CNN to identify whether workers were wearing safety harnesses. Although the literature [15,16,17] detected safety harnesses, they focused on the harness as a whole, merely determining its presence without considering whether it was being used correctly. Specifically, they did not consider whether the harness was properly secured—namely, whether the safety rope was connected to a fixed anchor point, which is crucial for providing protection in the event of a fall.

Chern et al. [18] initially utilized YOLOv5 to detect workers and PPE at construction sites, followed by recognizing the working context of the workers and analyzing whether they were using the appropriate PPE correctly. They proposed two methods for working context recognition: a depth estimation model and a semantic segmentation model for work scenes. Although they considered the detailed components of the safety harness, their segmentation approach was excessively fine-grained, potentially leading to recognition errors when parts of the harness were occluded and not fully detectable. Fang’s team [19] utilized object detection and tracking algorithms to identify workers and windows from video footage, then employed a spatial interaction-based classifier to determine whether a worker was transitioning from an indoor to an outdoor high-altitude scenario. Subsequently, they detected PPE for workers entering high-altitude environments and issued an alarm if PPE was not properly worn. The study noted that detecting the safety harness, safety rope, and anchor point as a whole resulted in excessively large bounding boxes, which included a significant amount of irrelevant background, introducing noise and interference when extracting features.

Li and colleagues [20] extracted images from videos, detected worker and PPE positions using YOLOv5, and identified keypoints of workers’ bodies using OpenPose. They then employed a 1D CNN to compare keypoints with PPE positions to determine whether workers were properly wearing helmets and attaching harness hooks to anchor points. However, they did not consider other parts of the safety harness or the possibility of hook occlusion, which could impact the accuracy of their method.

2.3. Deep Learning-Based Semantic Segmentation

In 2015, Long et al. [21] introduced Fully Convolutional Networks (FCNs) for semantic segmentation based on classification networks, marking the beginning of pixel-wise classification as the primary method for semantic segmentation and ushering in an era of deep learning-based approaches. In the same year, U-Net [22] effectively utilized contextual information with its distinctive U-shaped architecture, becoming an early application of FCNs that not only achieved high accuracy in medical image segmentation but also inspired many subsequent models.

Subsequently, as a representative of CNNs, DeepLab proposed by Chen et al. [23] employed atrous convolution to increase the receptive field and introduced atrous spatial pyramid pooling to capture objects and context at multiple scales, evolving into a series of models. Mask R-CNN [24] expanded Faster R-CNN into a multi-task model with a mask segmentation branch, achieving high precision in instance-level segmentation.

After Dosovitskiy’s team [25] integrated Transformers into the field of computer vision, numerous semantic segmentation algorithms based on Transformers emerged. Among them, Mask2Former [26] is an outstanding semantic segmentation algorithm that has been widely applied and improved upon in various domains since its inception.

For example, Zhang et al. [27] developed Mask2Former+ based on Mask2Former to identify rock movement cracks, enabling real-time acquisition of clear crack outlines, outperforming classical U-Net and DeepLabV3+. García and colleagues [28] fine-tuned Mask2Former initialized with a pre-trained backbone on the Cityscape dataset for segmentation of photovoltaic panels, achieving an accuracy of 98.38%, surpassing eight existing models. Li et al. [29] expanded the lychee branch dataset using denoising diffusion probabilistic models and trained Mask2Former on the augmented dataset for segmentation in outdoor backgrounds. They employed specified thresholds in the Hue, Saturation, and Value color space to prevent Mask2Former from extracting incorrect regions, ultimately generating multi-view images of lychees with Wonder3D, achieving an average pixel accuracy of 85.82%. Guo’s team [30] proposed IQ2Former, which integrated a query scenario module into Mask2Former to enhance scene adaptability, strengthening feature extraction through a query attention module, resulting in high-quality semantic segmentation of remote sensing images, outperforming the baseline and six other models.

The widespread use of Mask2Former demonstrates its effectiveness and generalizability; however, in the field of safety harness segmentation, the large-scale differences between masks necessitate further improvements.

3. Method

This section initially introduces the overall structure of CEMFormer. Subsequently, we provide a detailed explanation of the new criteria established for safety harness and safety rope detection, as well as modifications made to the original dataset based on these criteria. We also describe the model’s backbone, ConvNeXt; the integrated Efficient Multi-Scale Attention (EMA) mechanism; the detection head, Mask2Former; and the improvements made to the loss function within the detection head. The overall structure of the model is illustrated in Figure 1.

A three-channel image with a size of 1024 × 1024 pixels (3 × 1024 × 1024) is input into CEMFormer. After passing through Conv1 and Layer Norm1, a feature map of size 96 × 256 × 256 is obtained. The feature map is then processed by the ConvNeXt block, retaining its dimensions and serving as the backbone for outputting multi-scale feature maps. As the feature map passes through Downsample Modules 1, 2, and 3—each consisting of Layer Norm and a 2D convolution with a kernel size of 2 × 2—the feature map sizes change to 192 × 128 × 128, 384 × 64 × 64, and 768 × 32 × 32, respectively. The feature maps output from the backbone are fed into the EMA module for cross-dimensional interaction before being used as input for Mask2Former.

The multi-scale feature maps are fed into Mask2Former, both into the Pixel Decoder for multi-scale feature fusion and pixel-level prediction, as well as into the Transformer Decoder for single-scale predictions. In the Pixel Decoder, the multi-scale feature maps are unified to 256 channels via convolution and, after position encoding, are fed into the DETR Encoder to capture global context. The fused feature map is then obtained via the nearest neighbor interpolation, which is subsequently used for mask prediction. In the Transformer Decoder, each scale of the feature map is fed into different Decoder layers, where class and mask predictions are made alongside the input query features associated with position encoding.

In Figure 1, the three convolution modules utilize different kernel sizes: Conv1 utilizes a 4 × 4 kernel, Conv2 utilizes a 1 × 1 kernel, and Conv3 utilizes a 3 × 3 kernel. Both the Downsample Modules and Conv Modules employ 2D convolutions. The Positional Encoding module uses sinusoidal positional encoding. The modifications to the dataset, along with the details of the ConvNeXt Block, EMA, Decoder, and DETR Encoder in the model, as well as the changes to the loss function, will be introduced in the following subsections.

3.1. New Dataset

The original dataset was annotated using detection bounding boxes for ground-level workers (Ground), workers at height (Offground), and safety harnesses (Safebelt). An example of the annotations, along with an enlarged view of a local region, is presented in Figure 2.

In Figure 2a, the worker is wearing the safety harness without securing the safety rope, indicating improper use of the harness. The annotations in this dataset indicate only the presence of the safety harness and whether the worker is wearing it; however, they cannot determine whether the harness is being used correctly. Consequently, this limitation hinders the accurate assessment of the worker’s safety risk.

To address this issue, we propose a new annotation principle. For workers at height, the portion of the safety harness that extends beyond the worker’s outline and is secured to the anchor point is annotated as Saferope, while the remaining parts are still annotated as Safebelt. For workers on the ground, all portions of the safety harness are labeled as Safebelt. Considering the frequent occurrence of occlusion, we opted not to further subdivide the harness, allowing the model to learn to identify categories even when a significant portion is occluded, and to avoid cases where the finer structures are completely hidden. Annotations are created using polygonal masks to minimize noise and facilitate model learning. Any areas not manually annotated are automatically labeled as background. An example of the new dataset annotations, along with an enlarged view of a local region, is presented in Figure 3, where the background represents unannotated areas.

3.2. ConvNeXt

ConvNeXt was used as the backbone to extract features from the images. ConvNeXt, introduced by Liu et al. [31], is a modernized CNN based on ResNet [32]. As an outstanding backbone that has emerged in recent years, it has been widely utilized and validated in various fields, including skin disease classification [33], peptide-binding affinity prediction [34], identification of winter wheat seedlings [35], and rolling bearing defect diagnosis [36]. The structure of the ConvNeXt Block, the fundamental module of ConvNeXt depicted in Figure 1, is presented in Figure 4.

In the ConvNeXt Block, the depthwise convolution utilizes a 7 × 7 kernel without altering the dimensions of the feature map. The other two convolution layers utilize a 1 × 1 kernel: Conv1 expands the number of channels in the feature map by four times, while Conv2 restores the expanded channels to their original size.

Compared to ResNet, ConvNeXt integrates design elements inspired by Transformers, resulting in the following adjustments:

The number of blocks in each stage has been adjusted from (3, 4, 6, 3) to (3, 3, 9, 3), where each block represents the ConvNeXt Block depicted in Figure 4. This implies that the basic structure of the ConvNeXt Block is repeated three, three, nine, and three times in ConvNeXt Blocks 1, 2, 3, and 4, respectively.

The convolution layer that processes the input image, Conv1 in Figure 1, has been adjusted from a 7 × 7 kernel with a stride of 2 to a 4 × 4 kernel with a stride of 4.

Depthwise convolution is employed in the ConvNeXt Block in place of standard convolution.

An inverted bottleneck structure is employed, wherein the feature map in the ConvNeXt Block first undergoes depthwise convolution without altering the number of channels, followed by two regular convolutions that sequentially expand the channels by four times and restore them.

A large convolution kernel is employed in the ConvNeXt Block, specifically a 7 × 7 kernel for depthwise convolution.

GELU is used in place of ReLU as the activation function, and the number of activation functions within the model has been reduced.

Batch normalization is replaced with layer normalization, and the use of normalization layers is reduced.

Downsampling is introduced between stages, as depicted in Downsample in Figure 1.

With these modifications, ConvNeXt, as a CNN, integrates advanced design features from Transformers while simplifying its structure, thereby improving accuracy compared to the baseline ResNet.

3.3. Efficient Multi-Scale Attention (EMA)

In our previous research, we integrated the Efficient Multi-Scale Attention (EMA) proposed by Ouyang et al. [37] into an object detection network, which led to improved accuracy. At the conclusion of [37], the authors suggested that EMA could also be applied to tasks such as semantic segmentation. Inspired by this, we reintroduced EMA to improve feature extraction. Additionally, following the recommendation in [38], EMA was placed at the end of the backbone. Figure 5 presents the internal structure of EMA.

After the feature map is input into EMA, it is divided into n groups along the channel (C) dimension, and each group is subsequently split into two branches for computation. In this study, n is set to 8. Conv1 utilizes a 3 × 3 kernel, forming the “3 × 3 branch”, while Conv2 utilizes a 1 × 1 kernel, forming the “1 × 1 branch”. In the 1 × 1 branch, the feature map undergoes symmetric average pooling along both width (W) and height (H) directions, effectively capturing positional information along these vertical dimensions. The pooled results are concatenated along the W (H) dimension, resulting in a size of C//n × (W + H), and subsequently passed through a convolution. Conv2 does not alter the dimensions of the feature map, thereby ensuring no reduction in dimensionality and preventing information loss. The output of the convolution is split back into two vectors along the concatenated direction, each passing through a sigmoid function to approximate a two-dimensional binomial distribution. These vectors are subsequently multiplied to facilitate inter-channel interaction.

The 3 × 3 branch utilizes Conv1 to capture multi-scale features, after which it enters the cross-spatial learning module. In cross-spatial learning, the outputs of the two branches pass through a symmetric structure consisting of 2D global average pooling for spatial information encoding, a softmax function, and multiplication of the resulting vectors to generate two spatial attention maps. At the end of EMA, the two spatial attention maps are summed, and a sigmoid function is applied to map the output features according to the attention weights.

Throughout the entire EMA, these symmetric structures aid in capturing spatial semantic information and facilitating inter-channel interaction, thereby providing pixel-level global context for the feature map. Furthermore, the number of channels remains unchanged, preventing information loss due to channel reduction.

3.4. Mask2Former

Mask2Former, introduced by Cheng et al. [26], aims to provide a flexible, general-purpose, and high-performance architecture for various image segmentation tasks, including semantic segmentation and instance segmentation. The overall structure of Mask2Former is presented in Figure 1, where the DETR Encoder in the Pixel Decoder and the Decoder in the Transformer Decoder share a similar structure, imparting a sense of symmetry to Mask2Former’s design. DETR was introduced by Carion’s team [39], and the structure of the DETR Encoder is depicted in Figure 6.

After the feature map enters the DETR Encoder, it is divided into three components: Key (K), Query (Q), and Value (V). K and V are associated with positional encoding, and these three components are then used as inputs to the multi-head self-attention (MHSA) module. The output of the MHSA is added to the feature map and undergoes layer normalization, after which the resulting feature map is split into two branches. One branch passes through a feed-forward network (FFN) and is then added to the other branch, followed by another layer normalization before output. The Pixel Decoder captures global information via the DETR Encoder, which is subsequently used for the final pixel-level mask prediction.

Transformer Decoder of Mask2Former primarily consists of three Decoder blocks, and the structure of each Decoder block is depicted in Figure 7.

The structure of the Decoder is similar to that of the DETR Encoder, but it incorporates masked attention along with corresponding layer normalization, serving as a substitute for cross-attention. This adjustment directs the model’s attention more specifically to the foreground.

3.5. Loss Function

The loss function of Mask2Former is represented by Equation (1).

$$L={\lambda }_{ce}{L}_{ce}+{\lambda }_{dice}{L}_{dice}+{\lambda }_{cls}{L}_{cls}$$

(1)

In this context, $L_{ce}$ represents the cross-entropy loss, $L_{dice}$ denotes the Dice Loss, and $L_{cls}$ refers to the classification loss. The respective weights for ${\lambda }_{ce}$, ${\lambda }_{dice}$, and ${\lambda }_{cls}$ are set to 5, 5, and 2, respectively. ${\lambda }_{ce}{L}_{ce}+{\lambda }_{dice}{L}_{dice}$ is the mask loss. $L_{ce}$ adopts a binary cross-entropy loss, which is defined by Equation (2).

$$L_{ce}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N[y_i\log (\sigma (p_i))+(1-y_i)\log (1-\sigma (p_i))]}$$

(2)

In this equation, $y_i$ represents the ground truth value of sample $i$, which can be either 0 or 1. $\sigma (p_i)$ is the predicted value of sample $i$ after passing through the sigmoid function, and $N$ is the total number of samples.

$L_{cls}$ uses the basic cross-entropy loss, which is defined by Equation (3).

$$L_{cls}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{c=1}^C{y}_{i,c}\log (p_{i,c})}}$$

(3)

In this equation, $y_{i,c}$ represents the ground truth value of sample $i$, $p_{i,c}$ is the probability that sample $i$ is predicted as class $c$, $C$ is the total number of classes, and $N$ is the total number of samples. Leng et al. [40] found that, when optimizing cross-entropy loss using gradient descent, the Taylor expansion of the cross-entropy loss does not yield optimal results for polynomial coefficients. They proposed that adjusting the first polynomial coefficient could achieve significant gains with minimal effort, leading to the development of a new loss function, Poly Loss, as expressed in Equation (4).

$$L_{poly}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N\left[{\displaystyle \sum _{c=1}^C{y}_{i,c}\log (p_{i,c})}+\epsilon \left(1-{\displaystyle \sum _{c=1}^C{p}_{i,c}{y}_{i,c}}\right)\right]}$$

(4)

In this equation, $\epsilon$ is a hyperparameter used to adjust the polynomial coefficient. We used Poly Loss instead of cross-entropy loss as the classification loss for the model and found that setting $\epsilon$ to −1 effectively improved the model’s accuracy.

$L_{dice}$ uses the Dice Loss [41], which is defined by Equation (5).

$$L_{dice}=1-{\displaystyle \frac{2{\displaystyle \sum _{i=1}^N{p}_i{y}_i}}{{\displaystyle \sum _{i=1}^N{p}_i^2}+{\displaystyle \sum _{i=1}^N{y}_i^2}}}$$

(5)

In this equation, $p_i$ represents the predicted value of pixel $i$, $y_i$ is the ground truth value of pixel $i$, and $N$ is the total number of pixels. Jadon [42] argued that, due to the non-convexity of the Dice coefficient, Dice Loss cannot achieve optimal results. Based on Dice Loss, he proposed Log-Cosh Dice Loss, a loss function that retains the symmetry of Dice while being smooth, thereby facilitating gradient computation. The expression for Log-Cosh Dice Loss is presented in Equation (6).

$$L_{lc-dce}=\log (\cosh (L_{dice}))$$

(6)

We used Log-Cosh Dice Loss to replace the original Dice Loss. The final expression for our loss function is given by Equation (7).

$$L={\lambda }_{ce}{L}_{ce}+{\lambda }_{lc-dce}{L}_{lc-dce}+{\lambda }_{poly}{L}_{poly}$$

(7)

Following [26], we retained the coefficients from the original loss function, with ${\lambda }_{ce}$, ${\lambda }_{lc-dce}$, and ${\lambda }_{poly}$ set to 5, 5, and 2, respectively.

4. Experiments

4.1. Dataset

The dataset used in this study consists of images from a power plant in Guangdong, China, with annotated categories including ground-level workers (Ground), workers at height (Offground), safety harnesses worn by workers (Safebelt), safety ropes secured at anchor points (Saferope), and other regions (Background). The annotation principles were introduced in Section 3.1, and an example of the annotations is presented in Figure 3. The dataset contains a total of 2016 images, with 1413 used for the training set, 198 for the validation set, and the remaining 404 for the test set. All models mentioned in this paper were trained on the training set, validated on the validation set, and evaluated on the test set for final results.

4.2. Evaluation Metrics

Rather than focusing on whether the predicted mask shape closely matches the ground truth, our primary concern is whether the model can accurately classify each pixel—specifically, whether it can determine from the visible portion of the safety rope in the image if it is being used correctly, thereby assessing whether the worker is properly secured. Therefore, Accuracy (Acc) and mean Accuracy (mAcc) are used as evaluation metrics for the model. Their respective expressions are provided in Equations (8) and (9).

$$Ac{c}_i={\displaystyle \frac{T{P}_i}{T{P}_i+F{N}_i}}$$

(8)

$$mAcc={\displaystyle \frac{1}{C}}{\displaystyle \sum _{i=1}^{C}Ac{c}_i}$$

(9)

In these equations, $Ac{c}_i$ represents the Accuracy of class $i$, $T{P}_i$ is the number of pixels correctly predicted by the model for class $i$, $F{N}_i$ is the number of pixels from class $i$ incorrectly predicted as other classes by the model, and $C$ is the total number of classes.

4.3. Environment and Parameter Settings

All experiments in this study were conducted on a 64-bit Windows 11 operating system, featuring an NVIDIA GeForce RTX 4090 D GPU and an AMD Ryzen 7 7800X3D CPU. Python 3.8, PyTorch 2.0.1, and mmsegmentation 1.2.2 were used as the experimental environment. All models were trained for 200 epochs, with batch sizes ranging from two to sixteen depending on GPU memory usage. For the first 10 epochs, the learning rate was set to 0.1, and a polynomial learning rate scheduler was subsequently used to gradually reduce the learning rate to zero. The input image size for the models was set to 1024 × 1024.

4.4. Results

CEMFormer was compared with its baseline, Mask2Former (using ResNet50 as the backbone), as well as five SOTA models: PIDNet [43], SegNext [44], PoolFormer [45], ConvNeXt (with Uper Head [46] as the segmentation head), and DDRNet [47]. The comparison results are presented in

Table 1. A model performance comparison.

Model	AccBackground	AccOffground	AccGround	AccSafebelt	AccSaferope	mAcc	Time
PIDNet-L	99.76%	92.54%	88.34%	86.65%	74.46%	88.35%	95.2 ms
SegNext- MSCAN-L	99.64%	92.56%	88.09%	83.85%	70.64%	86.96%	119.1 ms
PoolFormer-m36	99.76%	92.51%	88.54%	83.14%	72.57%	87.31%	151.4 ms
ConvNeXt-T	99.77%	93.54%	89.04%	84.97%	75.63%	88.59%	124.2 ms
DDRNet-23	99.80%	67.52%	55.18%	55.84%	38.77%	63.42%	154.3 ms
Mask2Former	99.72%	92.58%	89.82%	85.55%	78.14%	89.16%	122.7 ms
CEMFormer (Ours)	99.51%	90.91%	89.33%	90.14%	91.68%	92.31%	119.1 ms

, where the variants of all models are also listed.

The mAcc of CEMFormer is 92.31%, surpassing PIDNet, SegNext, PoolFormer, ConvNeXt, DDRNet, and the baseline by 3.96%, 5.35%, 5.00%, 3.72%, 28.89%, and 3.15%, respectively. For individual class accuracies, compared to the baseline, our model demonstrates a loss of 0.21%, 1.67%, and 0.49% for the Background, Offground, and Ground classes, respectively. However, it achieves improvements of 4.59% and 13.54% in the Safebelt and Saferope classes. This trade-off is considered worthwhile, particularly since the Acc for the Saferope class reaches 91.68%, which is our primary focus. Except for DDRNet, all other SOTA models achieved an mAcc above 85%, indicating the effectiveness of our annotation principle. The Time column indicates the average inference time per image for each model on the test set. Our model achieves an average of 119.1 ms per image, matching SegNext, second only to PIDNet at 23.9 ms, and outperforming the other four models, including the baseline. However, achieving real-time detection will require targeted lightweight optimizations. Example images and the semantic segmentation results obtained using CEMFormer are presented in Figure 8 and Figure 9, respectively.

Figure 8 and Figure 9 demonstrate that, after training on the dataset with the new annotation principle, our model effectively distinguishes between workers in different positions, safety harnesses, and safety ropes, achieving good segmentation performance.

4.5. Ablation Study

To assess the effectiveness of all the improvements made to the model in enhancing its accuracy, an ablation study was conducted. The ablation study results are presented in

Table 2. An ablation study of CEMFormer.

Experiment	1	2	3	4	5
ConvNeXt	√	√	√	√	×
EMA	√	√	√	×	×
Poly Loss	√	√	×	×	×
Log-Cosh Dice Loss	√	×	×	×	×
AccBackground	99.51%	99.55%	99.76%	99.69%	99.72%
AccOffground	90.91%	84.55%	92.51%	92.33%	92.58%
AccGround	89.33%	91.85%	90.72%	91.53%	89.82%
AccSafebelt	90.14%	85.27%	85.43%	86.61%	85.55%
AccSaferope	91.68%	88.87%	80.18%	77.05%	78.14%
mAcc	92.31%	90.02%	89.72%	89.44%	89.16%

. In the table, “√” indicates that the model includes the corresponding component in the experiment, whereas “×” indicates that the model does not include the component, using the corresponding structure from the baseline instead.

When comparing Experiments 1 and 2 in Table 2, replacing Log-Cosh Dice Loss with the baseline Dice Loss resulted in a 2.29% decrease in mAcc. This decrease is primarily attributed to the decline in Acc for the Offground, Safebelt, and Saferope classes, which dropped by 6.36%, 4.87%, and 2.81%, respectively. This suggests that the smoother Log-Cosh Dice Loss offers an advantage during training. When comparing Experiments 2 and 3, reverting Poly Loss to cross-entropy loss reduced mAcc by 0.3%. Cross-entropy loss exhibited advantages for the Background, Offground, and Safebelt classes, while Poly Loss significantly improved classification accuracy for Saferope by 8.69%, representing a substantial improvement for identifying safety ropes.

Similarly, removing EMA from Experiment 3 resulted in a significant loss in Acc, particularly for the Saferope class, which dropped from 80.18% to 77.05% in Experiment 4, representing a decrease of 3.13%. This reduction is attributed to the absence of high-resolution feature maps provided by EMA, making it more challenging to segment small-scale objects such as safety ropes. Lastly, although ConvNeXt, as a backbone incorporating numerous advanced designs, does not offer a distinct advantage, it is still worth using compared to the classic ResNet. When comparing Experiments 4 and 5, removing ConvNeXt resulted in changes in Acc for individual classes within 2%, with an overall Acc decrease of 0.28%.

5. Conclusions

This paper proposes the following new annotation principle: for workers at height, the portion of the safety harness that extends from the anchor point to the worker’s outline is categorized as Saferope, and the dataset is re-annotated accordingly. Based on this new dataset, the semantic segmentation model CEMFormer is proposed. The proposed model employs ConvNeXt as the backbone, Mask2Former as the segmentation head, and integrates EMA between the backbone and the head. Poly Loss is employed for classification loss in the segmentation head, while binary cross-entropy loss and Log-Cosh Dice Loss are employed for mask loss. Experimental results indicate that the proposed model outperforms both the baseline and SOTA models. Ablation studies demonstrate the contribution of each enhancement to model accuracy.

Nevertheless, there is still room for improvement in our approach. The dataset only includes annotations for workers and safety harnesses, without labeling other objects, such as platforms, utility poles, sky, and ground. Although the model achieved high accuracy under these conditions, the dataset does not facilitate learning the worker’s position in relation to the ground or height, which may limit further improvements in accuracy. Therefore, enhancing the dataset will be prioritized. Additionally, semantic segmentation tends to use a single mask to cover multiple interacting objects, which is suboptimal for scenarios involving multiple workers collaborating. The previous object detection algorithm will be used as a complement to CEMFormer—specifically, to detect large-scale objects such as human bodies to determine each worker’s location, while delegating smaller-scale objects, such as safety harnesses, to CEMFormer. Lastly, since this study focused on exploring the feasibility of the new annotation principle, model parameter and computational complexity were not considered. Evaluating computational cost and lightweighting the model will be prioritized in our future research.