A Weakly Supervised Segmentation Algorithm Based on Local–Global Class Labelling Comparison

Abstract

As a key pixel-level analysis technology, semantic segmentation is widely deployed in autonomous driving and medical imaging. Fully supervised segmentation relies on labour-intensive pixel-wise annotations, so weakly supervised semantic segmentation (WSSS) with only image-level labels has attracted wide attention. Existing Vision Transformer (ViT)-based WSSS methods suffer from two critical limitations: ViT’s global self-attention mechanism leads to insensitivity to local small target features and incomplete foreground activation; its class-agnostic attention maps frequently misactivate background regions as foreground objects, introducing heavy noise. To tackle these two issues, this paper proposes a single-stage weakly supervised segmentation algorithm based on local–global class labelling comparison. First, we design a local–global class labelling comparison (LTG) module. By feeding both original images and randomly cropped local patches into ViT, we adopt InfoNCE contrastive loss to align local class tokens with global class tokens, enhancing the feature integrity of local target regions and suppressing background false activation. Second, a class-aware stimulus module (CSM) is embedded into ViT’s multi-head attention branch. It injects category semantic constraints into self-attention to generate class-aware attention maps, guiding the model to focus on real foreground targets and reduce background interference. Finally, we construct a feature fusion class-aware activation map (FFCAM) by fusing ViT global output features and CSM class-aware attention features to generate high-quality pseudo-labels for segmentation training. Extensive experiments are conducted on PASCAL VOC 2012 and MS COCO 2014 datasets. Our method achieves 78.5% mIoU on the validation set of PASCAL VOC 2012 and 50.9% mIoU on MS COCO 2014, showing competitive numerical performance among the compared single-stage ViT-based WSSS approaches. Ablation experiments verify the independent and joint effectiveness of LTG, CSM and FFCAM. The proposed method effectively improves the completeness of target activation regions and suppresses background noise, and it maintains strong generalization for slender, small and texture-sparse objects. In future work, we will further lightweight the ViT backbone to reduce computational overhead for embedded deployment.

1. Introduction

With the advent of the information age, computer technology has developed rapidly and penetrated into all areas of human life. The rapid progress of artificial intelligence technology has greatly improved the efficiency of information acquisition and processing, and is widely used in many fields such as travel navigation, aerospace, financial payments, intelligent communications, etc., and has become an indispensable part of modern society. In recent years, AI has made breakthrough progress in many fields [1], and has become a key force driving human society into the era of big data. Among the many research directions of artificial intelligence, image classification, as the core problem of computer vision, has a pivotal position. Not only does it have the basic task of visual recognition, but it also plays a key role in research and application in the fields of target detection, image segmentation, face recognition, target tracking, and super resolution.

Notwithstanding the considerable advances witnessed in the domain of deep learning algorithms in recent years, the quantity and quality of training data remain a pivotal determinate of model performance. Given the propensity of deep learning models to depend on substantial volumes of data, they are susceptible to the overfitting problem. This is characterized by the models’ adeptness on the training data, yet their ineptitude on unseen data. This problem is particularly acute when the amount of data is insufficient, especially when it is limited by factors such as privacy protection or the high cost of manual labelling. Irrespective of the sophistication of the model design and the training techniques employed, inadequate or homogeneous training data invariably imposes constraints on the model’s generalization capabilities. Conversely, the employment of relatively uncomplicated algorithms, incorporating substantial and varied data, has been demonstrated to frequently yield substantial enhancements in model performance. However, the procurement of sufficient and diverse data for numerous practical applications continues to be a significant challenge. Despite the substantial size of widely used contemporary datasets (e.g., ImageNet and CIFAR-100), their diversity and variability still cannot fully satisfy real-world performance requirements. Furthermore, the presence of adversarial attacks poses a significant threat to the accuracy of models, with even minor perturbations potentially leading to erroneous recognition, thereby compromising the reliability of the models. In order to address these issues and enhance the generalization capability of deep neural networks, researchers have proposed a priori-based regularization methods and task-related transformation strategies to improve the diversity of training data. These methods have been shown to effectively extend the coverage of the data, thereby significantly enhancing the model’s performance on unseen data.

In order to improve the robustness and accuracy of machine learning models, so that they can maintain good performance despite small samples or under-representation, data augmentation techniques have emerged. Data augmentation effectively improves the performance of deep neural networks by extending the sample space. This technique is not only widely used in computer vision tasks, but also shows significant value in other fields such as audio [2] and text [3]. Data augmentation can generate more samples based on existing training data or create new training data from scratch by specific techniques, thus effectively alleviating the problem of insufficient data. Efficient data augmentation strategies can not only significantly improve model performance, but also reduce the dependence on complex algorithms, so that simple models can also have strong generalization ability. Especially in the field of computer vision, training data are often presented in the form of images or video sequences, so augmentation methods for these data types are particularly important.

As the intricacy of deep learning models escalates, the number of parameters experiences exponential growth. Achieving a balance between low power consumption and real-time performance has emerged as a pivotal concern in contemporary research. A significant proportion of deep learning models comprise a substantial number of redundant parameters, which exert a negligible influence on performance. Consequently, the optimization of network structures and the reduction in redundant parameters have been shown to enhance computational efficiency and reduce storage overhead, thereby enabling models to operate efficiently on devices with limited resources. The following mainstream model compression methods are currently employed: network pruning, parameter quantization, lightweight network design and low-rank decomposition. Among these methods, knowledge distillation has garnered significant attention in recent years due to its ability to effectively reduce model size while maintaining performance. This has led to it becoming a prominent research area.

The fundamental concept of knowledge distillation entails the compression of models through the collaborative interaction of a teacher model and a student model. The teacher model possesses a complex structure and a substantial number of parameters, whilst the student model is more straightforward and lightweight. During the distillation process, the teacher model transfers its learned knowledge (e.g., intermediate layer features, output distributions, and feature relationships) to the student model, thereby ensuring that the student model approaches the performance of the teacher model while maintaining its lightweight nature. A distinguishing feature of knowledge distillation is that it does not necessitate modification of the structure of the target model, thus offering a versatile approach to model compression applicable to any simple and efficient model architecture. This feature provides a foundation for novel ideas in the realm of efficient model deployment and extension.

Consequently, the enhancement of image classification performance has emerged as a pivotal research domain, particularly in scenarios where data are limited and model structures are becoming more intricate. To address this challenge, data augmentation and knowledge distillation have emerged as effective methods to enhance performance. The former involves the generation of new training samples by applying multiple transformations to the original data, thereby enhancing the generalization ability of the model. Conversely, knowledge distillation involves the process of reducing redundant parameters through model compression, thereby enhancing model performance.

In response to the above issues, we propose a local–global class label comparison (LTG) mechanism to align the representation of local regions with global class features, enhancing the integrity of the target region. In addition, a category-aware stimulus module (CSM) is introduced into the attention mechanism of ViT to guide the ViT model to focus on the correct target area and enhance its category perception ability. On the basis of the above two methods, a feature fusion class-aware activation map (FFCAM) is proposed to replace the generation of traditional class activation maps.

Recent studies further demonstrate why category-aware token modeling is important for ViT-based WSSS. Class Tokens Infusion injects class-token semantics into patch tokens to strengthen class-specific localization [4]. Class Token as Proxy identifies a token gap between classification-oriented class tokens and patch tokens containing both discriminative and non-discriminative cues, which can cause incomplete foreground activation [5]. Know Your Attention Maps directly uses class-specific ViT attention maps and token masking to improve class assignment and pseudo-mask quality [6]. These findings show that global self-attention provides valuable long-range context but does not by itself guarantee complete local responses or reliable category discrimination. Motivated by this evidence, our LTG module aligns local patch representations with global class semantics, while CSM imposes explicit category constraints on self-attention to suppress background confusion.

To overcome the two inherent defects of ViT in WSSS, this paper designs a complete single-stage segmentation framework based on image-level weak supervision. First, we propose a local–global class labelling comparison (LTG) module to solve ViT’s local feature insensitivity. We perform random local cropping on input images and feed both global full images and local cropped patches into the shared ViT backbone. By calculating InfoNCE contrastive loss between global class tokens and local class tokens, we pull positive local foreground features close to global semantic representations and push negative background local features away from global class tokens, which enhances the completeness of local target feature response and eliminates partial background false activation. Second, we construct a class-aware stimulus module (CSM) embedded inside ViT’s multi-head attention block. It reconstructs the original class-agnostic attention map into class-aware attention via encoder–decoder convolution layers and auxiliary classification loss, injecting explicit category constraints into self-attention to suppress background noise and concentrate attention on object regions. Finally, we fuse the global output features of ViT and the optimized attention features from CSM to build a feature fusion class-aware activation map (FFCAM), which generates more accurate and complete pseudo-labels for end-to-end segmentation training without multi-stage iterative refinement.

The rest of this paper is organized as follows. Section 2 reviews the state-of-the-art research on fully supervised semantic segmentation and multiple branches of weakly supervised semantic segmentation, and analyzes the existing limitations of ViT-based WSSS methods. Section 3 elaborates the complete network architecture of our proposed algorithm, and provides detailed mathematical derivation and structural illustration of the local–global class labelling comparison module, class-aware stimulus module and feature fusion class-aware activation map. Section 4 describes the experimental setup including datasets, evaluation metrics and training hyperparameters, reports quantitative comparison results against SOTA methods on two benchmarks, provides qualitative visualization of class activation maps and segmentation predictions, and conducts systematic ablation experiments to verify the effectiveness of each core component. Section 5 concludes the whole work and discusses feasible lightweight optimization directions for future research.

2. Related Work

2.1. State-of-the-Art Research on Fully Supervised Semantic Segmentation Based on Deep Learning

The advent of deep learning, particularly convolutional neural networks, has precipitated a paradigm shift in the realm of semantic segmentation. A seminal paper in this area was the 2012 publication of the AlexNet network by Hinton et al., which demonstrated the immense potential of deep learning in image classification tasks and led to a proliferation of deep learning techniques in the field of semantic segmentation. A significant development was the introduction of the fully convolutional network (FCN), which replaced the fully connected layers with convolutional layers, thereby enabling the network to process images of any size. This became the standard method for semantic segmentation tasks. In 2015, the U-Net architecture was proposed by Ranneberger et al., which employs a typical encoder–decoder structure with jump connections between the two for enhanced image detail recovery. The U-Net architecture achieved remarkable performance in medical image segmentation tasks and had a substantial impact on subsequent research in semantic segmentation algorithms. DeepLabv2 [7] introduced Dilated Space Pyramid Pooling, which enhanced the network model’s capacity to learn multi-scale features, thus improving the segmentation effect. In 2019, APCNet [8] introduced an innovative approach by combining multi-scale feature extraction and contextual information adaptive through the incorporation of an adaptive pyramid context module learning mechanism. This development yielded significant advancements in detail recovery and multi-object recognition capabilities of the network. In 2021, Xie et al. proposed SegFormer [9], which used a lightweight Transformer [10] model and has excellent performance on multiple datasets through an efficient self-attention mechanism and a flexible end-to-end training process. In 2023, Zhang et al. proposed CMNeXt [11], which not only improved the accuracy of the semantic segmentation task, but also enhanced the robustness and generalizability of the model through the fusion of multimodal information and adaptive mode selection.

In general, fully supervised semantic segmentation algorithms have historically occupied a pivotal position in the development of the field, and they play an indispensable role in many current areas of research. However, fully supervised semantic segmentation requires pixel-level labels as a reference for supervised training, and acquiring these labels is usually a laborious and time-consuming process that is costly in terms of both time and labour. As the amount of data increases, the difficulty and cost of acquiring labelled data also increase exponentially, which limits the wide application of fully supervised methods.

2.2. State-of-the-Art Research on Weakly Supervised Semantic Segmentation Based on Deep Learning

Fully supervised semantic segmentation is contingent upon a substantial amount of pixel-level labelled data, which faces significant challenges in many practical scenarios. This is especially the case given that the data labelling process is not only time-consuming and labour-intensive, but it also becomes more and more difficult to obtain accurate annotations as the scale of the dataset increases. This makes the limitations of traditional fully supervised methods more and more obvious in practical applications. Consequently, the urgent need to reduce annotation costs and enhance data utilization efficiency has emerged as a pivotal challenge in contemporary semantic segmentation research.

In order to solve this challenge, weakly supervised semantic segmentation (WSSS) has gradually received widespread attention as an emerging research direction. In contrast to fully supervised methods that rely on pixel-level annotations, weakly supervised semantic segmentation employs a more simplified form of annotation. This can be categorized as follows: (a) image-level annotation, (b) object point annotation, (c) object box annotation, and (d) graffiti annotation. Image-level annotation is the simplest form of annotation and usually only assigns a classification label to each image to indicate the presence or absence of a particular object classification in the image. This form of annotation does not involve the specification of object location or shape information. Consequently, image-level annotation has been the subject of study by a wide range of scholars, primarily due to its relatively low acquisition cost. Object point annotation, by contrast, involves the assignment of one or more specific point locations to objects in an image, with these points representing a particular object classification. These classifications are usually located in the centre or a prominent region of the object. Weak labelling based on object boxes requires the drawing of a rectangular box for each object in the image. This method of labelling is coarse, and the boundary information of the object cannot be obtained more accurately. Graffiti annotation, on the other hand, involves the use of simple hand-drawn markers to denote the semantic information of the image, thus facilitating the rapid acquisition of more rudimentary information concerning the object regions. The following section will elaborate on the various forms of annotation in the context of weakly supervised semantic segmentation research.

The object box annotation-based approach involves the delineation of rectangular boxes for each object in an image, thereby providing spatial information but not specific shapes and edges. This approach is more efficient than pixel-level annotation and contains more positional information than image-level and point annotation. Lee et al. proposed BBAM [12], which performs pixel-level semantic segmentation by generating a bounding-box attribution map for each object and using the bounding-box information. SDI [13] unifies the tasks of instance segmentation and semantic segmentation under a single framework, and gradually denoises the previous round of training by using the prediction results of the previous round of the training convolutional network as the next round of supervised signals, through a well-designed labelling generation strategy and recursive training. The convolutional network prediction results from the initial round of training are utilized as the supervised signals in the subsequent round, thereby progressively denoising the model and enhancing its performance. Song et al. [14] proposed a classification mask generation method employing bounding-box supervised signals, which is capable of efficiently removing background noise and providing clear foreground indications for the segmentation task by learning a specific mask for each classification.

Object point labelling is the use of points in the image to label the target object, and usually these points are located in the central region of the target object. Compared to pixel-level labelling, the cost of object point labelling is greatly reduced, but it can suffer from the problem of insufficient labelling. Bearman [15] et al. proposed point-level labelling for the first time, which significantly improves the performance of the model while keeping the labelling cost low through the combination of point-level supervision and objectivity prior. PCAMS [16] propagates the labels through the introduction of a class of activation maps with point-level supervision, which significantly improves the performance of the model, but its generalization ability in complex scenarios and its ability to handle small objects still need further validation.

Graffiti annotation, a method of delineating object regions by hand-drawn markings or contour lines, does not necessitate the precise delineation of boundaries. However, the accuracy of graffiti annotation is suboptimal, necessitating post-processing techniques to enhance the precision of the delineated object regions. In their seminal work, Zhang et al. [17] proposed a classification-driven graffiti boosting network that leverages the complementary information of graffiti and pseudo-tagging. The network incorporates a localization correction module that effectively addresses the issue of noise. SASFormer [18] introduces a Visual Transformer into the weakly supervised semantic segmentation task and proposes a hierarchical patch attention graph and an affinity loss function to significantly improve segmentation performance. However, SASFormer is characterized by high computational complexity and a dependence on the quality of initialized annotations.

Image-level annotation-based methods have become the focus of WSSS research due to their low acquisition cost and fast annotation speed compared to several other forms of weak labelling. The majority of extant image-level annotation-based WSSS methods employ convolutional neural networks as the classification network, after which the corresponding image features are obtained by the classification network. Class activation maps (CAMs) are then used to generate approximate target regions, and post-processing technological tools (e.g., refinement of the initial seeds [19]) are used to generate pseudo-labels to supervise the training of the segmentation network. However, CAM typically highlights only the most discriminative part of the region in the image, thus failing to provide a complete pseudo-mask. Consequently, researchers have proposed a plethora of innovative approaches to address this issue, which can be broadly categorized as follows: region erasure, region growing, additional supervised information, contrast learning, and so forth.

The process of region erasure has been shown to compel the network to pay closer attention to the remaining regions. This is achieved by erasing certain regions of the image. AE-PSL [20] proposes adversarial erasure and online prohibitive segmentation learning methods, which gradually identify the target region through adversarial erasure and reduce noise through online prohibitive segmentation learning. In the context of region growing methods, SEC [21] puts forward a set of three principles—seed, expand and constrain—with the aim of optimizing segmentation performance through the implementation of innovative global weighted rank pooling and boundary constrained loss. DSRG [22] employs a classification network to generate seed regions and trains the segmentation network by balancing the seed loss to gradually expand these regions to cover the entire object. Furthermore, there exist additional methodologies that employ supplementary supervised information to train the network, including the utilization of additional saliency maps and CLIP [23]-based supervision, amongst others. EPS [24] generates high-quality pseudo-pixel supervised information by combining saliency maps and image-level labelling, which significantly improves object boundaries and co-occurring pixel problems. CLIP-ES [25] takes full advantage of CLIP’s zero-sample capability and text–image alignment capability, and proposes to introduce a Softmax function into GradCAM [26] to significantly simplify the multi-stage process of traditional WSSS. POLE [27] was the first to introduce cue learning into WSSS, exploring the effect of classification labelling in cue words on CAM. LPCAM [28] extends the coverage of CAM by capturing non-discriminative and discriminative features to capture the object’s complete region. DSCNet [29] utilizes both pixel-level and semantic-level cross-image contextual information, employing a combination of pixel-level group contrast learning and semantic-level map contrast learning to address the limitations of existing methods in leveraging global semantic information. HSC [30] introduces multi-level semantic contrasts, encompassing ROI-level, classification-level, and pixel-level semantic contrasts. This multi-level contrast learning mechanism has the potential to assist the model in comprehending the semantic structure of the image and generating more accurate CAMs.

All of the aforementioned methods utilize convolutional neural networks (CNNs) as the backbone network, and in recent years, Vision Transformer (ViT) [31] has gained widespread attention due to its excellent global dependency modelling capability and its ability to capture contextual information over long distances. Consequently, several recent studies have adopted ViT as a classification model to generate high-quality CAMs. However, as illustrated in Figure 1, the cross-regional information interaction of ViT renders its feature learning for local regions ineffective, particularly when dealing with small targets in images. This results in the model’s inability to capture their complete semantic information, leading to small targets being easily ignored or incorrectly identified. This also leads to fuzzy target boundaries, which prevents accurate segmentation of fine-grained targets. Additionally, since the attention graph in ViT networks is classification-independent, the model may erroneously confuse the background region with the target classification, resulting in an attention distribution that is insufficiently focused, thus affecting the final segmentation outcome. Consequently, the development of effective optimization strategies for the ViT network is a significant area of research.

The work of TS-CAM [32] is inspired by the fact that class tokens in the standard Transformer can be arranged into class-independent attention graphs. The aim of TS-CAM is to couple CAMs with class-independent attention graphs in order to obtain positional cues in weak supervision. TransCAM [33] combines the Transformer as well as a CNN network. The attentional weights of the Transformer are used to refine the CAM of the CNN branches in order to obtain local features and global representations. MCTformer [34] introduces multiple class labels to generate class-specific attention graphs and refines the CAM with class-specific object localization maps and patch-level pairwise affinities. Thus, attentional learning from class labels to patch labels can generate class-specific attention graphs. ViT-PCM [35] proposes an alternative to CAM based on the localization property of the Visual Transformer by modelling the mapping mechanism between patch-level features and classification predictions. FBR [36] was proposed by Yin et al. to solve the problem of CAM methods having difficulty in distinguishing foreground objects from suspicious background pixels and being unable to activate the entire target area. AFA [37] suggests using the initial pseudo-labels as supervisory information to guide the training of self-attention, which leads to the learning of high-quality affinity information for pseudo-labelling improvement. ToCo [38] provides an alternative to CAM based on the localization property of the Visual Transformer by exploiting the intermediate layers of the semantic information to significantly alleviate the problem of over-smoothing of attention in ViT and to generate more accurate and complete CAMs. CSRM [39] was proposed by David et al. to address the constraints of pseudo-label uncertainty on training quality in WSSS. WeakTr [40] was proposed by Zhu et al., aiming to systematically explore the potential of naive Vision Transformers in WSSS tasks.

3. Method

In this paper, we propose a local–global class labelling comparison network to address ViT’s weak sensitivity to local features. As shown in Figure 2, the framework consists of a ViT feature extractor, a local–global class labelling comparison (LTG) module, a class-aware stimulus module (CSM), and feature fusion class-aware activation maps (FFCAM). The input image is locally cropped, and global and local class tokens are compared through contrastive learning to align local regions with global category semantics. CSM then transforms the original class-agnostic ViT attention into class-aware attention by injecting category constraints, thereby focusing the model on foreground regions and suppressing background noise. Finally, FFCAM fuses the ViT features with the class-aware attention features to refine the initial class activation map.

3.1. Local–Global Class Labelling Comparison Module

In this WSSS framework, ViT is utilized as a baseline network, employing a self-attention mechanism to construct global features. However, its local regions exhibit inadequate feature extraction capabilities, potentially resulting in incomplete foreground activation regions during segmentation. This may lead to erroneous activation of background regions. Inspired by TS-CAM [32] that class tokens can aggregate high-level semantic features, a local–global class token comparison (LTG) module is proposed, which aims to compare local regions with global class tokens to ensure that the classification-related local regions can be aligned with the global class tokens, which can improve the accuracy of activation regions.

As shown in Figure 3, for a given input image $X\in {ℝ}^{H\times W\times M}$, where $H$, $W$ are the height and width of the image and $M$ is the number of channels, it is first subjected to random local cropping, and then both the original image and the cropped image are fed together into a ViT network to learn the features. The specific network structure in ViT is illustrated in Figure 4, where the input image is partitioned into non-overlapping patches (Patch) of a fixed size, and each Patch is of size $w\times h$, which is then spread and linearly projected to generate Patch tokens $T_p\in {ℝ}^{N\times D}$, where FA is the dimension of each patch and $N=w\times h$. In addition, the class token $T_{cls}\in {ℝ}^{1\times D}$ is a feature learning vector that captures the semantic information of an image, aggregates it with the Patch tokens and inputs it into the Transformer together with the Positional Embedding $T_{pos}\in {ℝ}^{(N+1)\times D}$, and interacts with it layer by layer through the self-attention mechanism to gradually aggregate the global semantic information. Since the class tags of ViT capture the high-level semantic information of the image related to the target classification, the class tags of global and local images aggregate the global and local object information, respectively. The input sequence gets output after a Transformer block, and the output global and local class tokens are finally taken out, and the comparison learning idea is used to compare the local region with the global class tokens and minimize the difference between the two (approximating the positive samples), and to ensure that the representation of the whole object region can be more consistent.

To cope with the case that the cropped local images do not contain foreground object regions, we crop some local images from the background region, which also suppresses the false activation of the background by maximizing the distance between the local background region and the global image class labels (pushing away the negative samples). In our experiments, we identified randomly cropped local images as positive examples (class-related foreground regions) and negative examples (background regions) based on the initial pseudo-labels.

Specifically, the output global class token and local class tokens are projected through projection heads p1 and p2, each consisting of a linear layer and L2 normalization. Let Q⁺ denote the set of local views that are semantically consistent with the global view (positive samples), and let Q⁻ denote the set of local views sampled from background or class-inconsistent regions (negative samples). LTG minimizes the discrepancy between the global representation and Q⁺ while maximizing its discrepancy from Q⁻. We implement this objective using the InfoNCE loss [41], formulated as follows:

$$L_{ltg}={\displaystyle \frac{1}{N^{+}}}{\displaystyle \sum _{q+}\log \frac{e^{(p{q}^{+}/\tau )}}{e^{(p{q}^{+}/\tau )}+{\displaystyle {\sum }_{q^{-}}{e}^{(p{q}^{-}/\tau )}}}}$$

(1)

where $q^{+}\in Q^{+}$ denotes positive samples, $q^{-}\in Q^{-}$ denotes negative samples, $N^{+}$ denotes the total number of positive samples, and $\tau$ is the temperature coefficient. Note that LTG aims to enhance the local view representation as a way to adjust the global view representation. Therefore, in the experiments, the gradient update of the projection head p1 is stopped. To update p1, we use an exponential moving average (EMA), which is corrected as follows:

$${\theta }_1\leftarrow \rho {\theta }_1+(1-\rho ){\theta }_2$$

(2)

where $\rho$ is the momentum factor, ${\theta }_1$ denotes the parameters of the global view projection head p1, and ${\theta }_2$ denotes the parameters of the local view projection head p2. The stable update of p1 can be maintained by EMA, while avoiding the noise of the local view to have a direct impact on the global view representation.

3.2. Class-Aware Stimulus Module

Current weakly supervised semantic segmentation methods rely on class activation maps (CAMs), but these methods usually capture only the most salient parts of an object and not the whole object. Existing Transformer-based WSSS methods use class-agnostic attention maps, resulting in background noise and bias in the final feature results. To address the above problems, this algorithm proposes the class-aware stimulus module (CSM). The CSM generates class-aware attention maps by injecting classification information into the Transformer’s self-attention maps.

Specifically, as shown in Figure 5, the class-aware stimulus module is based on an encoder–decoder structure that extracts more precise semantic information from the self-attention graph and enhances the class-awareness capability with an auxiliary classification loss. Assume that the attention matrix $A_m$ of the multi-head attention module in the Transformer block is as follows:

$$A_m\in {ℝ}^{L\times S\times (N+1)\times (N+1)}$$

(3)

where $L$ denotes the number of blocks in Transformer, $S$ denotes the number of heads in the multi-head attention mechanism, and $N+1$ denotes the Patch tokens output dimension. First, the classification labelled attention vectors are extracted and reshaped from a Patch to obtain the attention graph:

$$\overline{A}\in {ℝ}^{L\times S\times w\times h}$$

(4)

It aggregates the feature representations of all Transformer blocks and heads. However, $\overline{A}$ acquires information from different representation subspaces at different locations in the input image, including all regions of interest in the image, which makes it difficult for the final result to include specific local target regions. To solve this problem, the class-aware stimulus module is proposed to optimize $\overline{A}$.

$$F_a\in {ℝ}^{(LS)\times w\times h}$$

(5)

A CNN-based encoder–decoder structure is then applied to extract more precise semantic information. The encoder structure is shown in Figure 5. It contains two convolutional layers and a max-pooling layer with stride 2, which yields the w/2 × h/2 output resolution specified in Equation (6).

$$F_e\in {ℝ}^{G\times {\displaystyle \frac{w}{2}}\times {\displaystyle \frac{h}{2}}}$$

(6)

where $G$ denotes the number of feature channels. The feature map $F_e$ is then up-sampled and spliced with the input feature map $F_a$, and the connected feature maps are then passed through the convolutional layers of the two $3\times 3$ to get the output feature map $F_{csm}$. Next, we add an auxiliary classification header after $F_{csm}$ to improve the classification recognition capability. The classification header is composed of the convolutional layers of $1\times 1$ and a global average pooling layer and produces the output feature map as follows:

$$F_{csm}\in {ℝ}^{C\times w\times h}$$

(7)

The classification loss formula used by CSM is as follows:

$$L_{csm}=-{\displaystyle \sum _i^C{y}_i\log (\frac{e^{{\widehat{y}}_i^C}}{{\displaystyle {\sum }_j^C{e}^{{\widehat{y}}_j^C}}})}$$

(8)

3.3. Feature Fusion Class-Aware Activation Maps

In weakly supervised semantic segmentation tasks, which are not trained based on pixel-level labels but only using image-level labels, CAM, as a class-aware saliency map, can be used to extract rough regions of the target object to provide pseudo-labels for subsequent semantic segmentation tasks. However, CAM focuses only on the most discriminative regions and may misclassify classes. Based on the class-aware stimulus module proposed above, we proposed the feature fusion class-aware activation map (FFCAM) to optimize the CAM for enhanced classification perception.

Specifically, the whole process of feature fusion class-aware activation map generation is shown in Figure 6 and is expressed as follows:

$$M_{fuse}=(M_{out}+M_{csm})\otimes A_m$$

(9)

The class activation map $M_{out}\in {ℝ}^{w\times h}$ is obtained from the image input to the final output features $F_{out}$ in the ViT network, and $F_{out}$ is optimized by the local–global class label comparison module to enable the network to pay more attention to local features. $M_{csm}\in {ℝ}^{w\times h}$ is obtained from the feature attention graph $F_{csm}$ generated by the class-aware stimulus module, and $F_{csm}$ is enhanced by the class-aware stimulus mechanism to increase the network’s attention to the class-related regions, thus reducing the background error. $\otimes$ denotes element-by-element multiplication. For $A_m$, firstly, the averaging operator is used on the attention map $\overline{A}\in {ℝ}^{L\times S\times w\times h}$ along the block and head dimensions, and so the class-aware attention map $A_m\in {ℝ}^{w\times h}$ can be obtained, which is given by the following:

$$A_m={\displaystyle \frac{1}{LS}}{\displaystyle \sum _l^L{\displaystyle \sum _s^S{\overline{A}}_{(l,s)}}}$$

(10)

For the generation of the final feature fusion class-aware activation map $M_{fuse}$, $M_{out}$ is firstly summed with $M_{csm}$. Through this operation, the final feature map contains local–global contextual information, and emphasizes the class-aware capability to improve the recognition accuracy of the model. Then, an element-by-element multiplication operation is performed with the class-aware attention graph $A_m$. The class-aware filtering through $A_m$ can improve the activation strength of the target region and enhance the semantic consistency. Finally, $M_{fuse}$ is adjusted to the same size of the original image using linear interpolation to obtain the final feature fusion class-aware activation map.

4. Experiments

4.1. Experimental Environment and Dataset

In order to fully validate the effectiveness of the method, we conduct experiments on two datasets: the PASCAL VOC 2012 and the MS COCO 2014. The PASCAL VOC 2012 consists of 20 foreground semantic object classes and 1 background class, and the training, validation, and test sets consist of 1464, 1449, and 1456 images, respectively. MS COCO 2014 uses 80 foreground object classes and 1 background class with 82,081 images in the training set and 40,137 images in the validation set.

For the experimental network model, the ViT pre-trained on ImageNet-1K was used as the backbone network, and unlike multi-stage weakly supervised semantic segmentation, our single-stage approach uses only a simple segmentation header as the decoder, which consists of two convolutional layers (with an expansion rate of 5) and a prediction layer. For the PASCAL VOC 2012 dataset, the training batch size (Batch_Size) of the network is set to 2 and a total of 20,000 iterations are performed. For the MS COCO 2014 dataset, the network was trained over a total of 80,000 iterations with a Batch_Size of 1. The AdamW optimizer was used in the experiments to train our network, and the initial 1500 iterations were considered as a warm-up phase, during which the learning rate was 1 × 10⁻⁶, and was increased in a linear fashion up to 6 × 10⁻⁵, with the decay rate set to 0.9. For the input images, the global and local images are cropped to sizes $448\times 448$ and $96\times 96$, respectively. The temperature coefficient $\tau$ in Equation (1) is 0.5. The momentum factor in the local–global class labelling comparison module EMA is set to 0.9. All the experiments in this algorithm are done on an NVIDIA RTX 3090 graphics device with the specific configuration shown in

Table 1. Experimental environment.

Type	Environmental Conditions
System	Ubuntu 16.04
GPU	NVIDIA RTX 3090
RAM	64 GB
CPU	Intel(R) Xeon(R) 8255C
Framework	Pytorch 1.13

.

4.2. Experimental Evaluation Indicators

The semantic segmentation performance is evaluated using the Intersection over Union (IoU) and mean Intersection over Union (mIoU) metrics. IoU measures the overlap ratio between the predicted segmentation region and the ground-truth region, which is defined as follows:

$$\mathrm{IoU}={\displaystyle \frac{TP}{FN+FP+TP}}$$

(11)

where TP, FP, and FN represent the numbers of true positive, false positive, and false negative pixels, respectively. Specifically, TP denotes the pixels that are correctly predicted as the target class; FP indicates the pixels incorrectly predicted as the target class but belonging to other classes; and FN represents the pixels belonging to the target class but incorrectly predicted as other classes.

For multi-class semantic segmentation tasks, the mean Intersection over Union (mIoU) is calculated by averaging the IoU values across all semantic classes:

$$mIoU={\displaystyle \frac{1}{C}}{\displaystyle \sum _{i=1}^{C}Io{U}_i}$$

(12)

To provide a multidimensional assessment beyond region overlap, we additionally report cosine similarity, mean absolute error (MAE), mean squared error (MSE), and root mean squared error (RMSE).

For a normalized prediction map P and ground-truth map G with N pixels, cosine similarity is defined as follows:

$$\mathrm{Cosine}\mathrm{Similarity}={\displaystyle \frac{{\displaystyle \sum _{i=1}^N{P}_i}{G}_i}{\sqrt{{\displaystyle \sum _{i=1}^N{P}_i^2}}\sqrt{{\displaystyle \sum _{i=1}^N{G}_i^2}}}}$$

(13)

$$\mathrm{MAE}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N|}{P}_i-G_i|$$

(14)

$$\mathrm{MSE}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{(P_i-G_i)}^2}$$

(15)

$$\mathrm{RMSE}=\sqrt{\mathrm{MSE}}$$

(16)

where P_i and G_i denote the predicted and ground-truth values of the i-th pixel, respectively.

Cosine similarity measures directional agreement between prediction and ground-truth (higher is better), whereas MAE, MSE, and RMSE measure pixel-wise prediction deviations (lower is better). Each metric is computed per image from normalized class-probability maps and then averaged over the validation set. These complementary criteria are reported with mIoU in

Table 2. Pseudo-label mIoU (%) on PASCAL VOC 2012.

Method	Backbone	Pseudo-Label mIoU (%) Training	Pseudo-Label mIoU (%) Validation
SEAM	ResNet38	63.6	62.1
SLRNet	ResNet38	67.1	66.2
ViT-PCM	ViT-B	67.7	66.0
AFA	MiT-B1	68.7	66.5
MCTformer	DeiT-S	69.1	68.2
ToCo	ViT-B	72.2	70.5
WeakTr	ViT-B	72.0	70.2
Ours	ViT-B	73.4	72.3

.

4.3. Comparative Analysis of Experimental Results

This chapter validates the performance of the algorithm proposed in this paper through experiments on the PASCAL VOC 2012 and MS COCO 2014 datasets. The experimental analysis, which includes quantitative evaluation, qualitative visualization and comparison with existing methods, has comprehensively demonstrated the advantages of this paper’s algorithm in weakly supervised semantic segmentation tasks.

In weakly supervised semantic segmentation tasks, the quality of pseudo-labels directly affects the final segmentation performance. Therefore, we set up a set of experiments to evaluate the quality of pseudo-tags generated by different methods. As shown in Table 2, comparisons are made on SEAM, SLRNet, ViT-PCM, AFA, MCTformer, ToCo and WeakTr methods on the PASCAL VOC 2012 dataset.

The comparison includes every method reported in Table 2,

Table 3. Segmentation mIoU (%) on PASCAL VOC 2012. I: image-level labels; S: external saliency supervision.

Method	Backbone	Supervision	Segmentation mIoU (%) Training	Segmentation mIoU (%) Validation
Multi-stage WSSS
NSROM	ResNet101	$\mathcal{I}+\mathcal{S}$	68.3	68.5
EPS	ResNet101	$\mathcal{I}+\mathcal{S}$	70.9	70.8
L2G	ResNet101	$\mathcal{I}+\mathcal{S}$	72.1	71.7
CLIP-ES	ResNet101	$\mathcal{I}+L$	71.1	71.4
SEAM	ResNet38	$\mathcal{I}$	65	65.7
SIPE	ResNet101	$\mathcal{I}$	68.8	69.7
W-OoD	ResNet38	$\mathcal{I}$	70.7	70.1
MCTformer	DeiT-S	$\mathcal{I}$	71.9	71.6
Single-stage WSSS
AFA	MiT-B1	$\mathcal{I}$	66.0	66.3
SLRNet	ResNet38	$\mathcal{I}$	67.2	67.6
ToCo	ViT-B	$\mathcal{I}$	69.8	70.5
WeakTr	ViT-B	$\mathcal{I}$	-	78.4
Ours	ViT-B	$\mathcal{I}$	73.8	78.5

and

Table 4. Validation segmentation mIoU (%) on MS COCO 2014. I: image-level labels; S: external saliency supervision.

Method	Backbone	Supervision	Validation Segmentation mIoU (%)
Multi-stage WSSS
EPS	ResNet101	$\mathcal{I}+\mathcal{S}$	35.7
L2G	ResNet101	$\mathcal{I}+\mathcal{S}$	42
SEAM	ResNet38	$\mathcal{I}$	31.9
SIPE	ResNet101	$\mathcal{I}$	40.6
MCTformer	DeiT-S	$\mathcal{I}$	42.0
Single-stage WSSS
SLRNet	WResNet38	$\mathcal{I}$	35.0
AFA	MiT-B1	$\mathcal{I}$	38.9
ToCo	ViT-B	$\mathcal{I}$	41.3
WeakTr	ViT-B	$\mathcal{I}$	50.3
Ours	ViT-B	$\mathcal{I}$	50.9

. The multi-stage group contains NSROM (ResNet101), EPS (ResNet101 with saliency-based pseudo-pixel supervision), L2G (ResNet101 with local-to-global representation transfer), CLIP-ES (ResNet101 with CLIP-derived text–image cues), SEAM (ResNet38 with equivariant self-attention), SIPE (ResNet101), and W-OoD (ResNet38). The Transformer-based group contains MCTformer (DeiT-S with multiple class tokens), ViT-PCM (ViT-B with patch-class mapping), AFA (MiT-B1 with affinity learning), SLRNet (ResNet38), ToCo (ViT-B with token contrast), and WeakTr (ViT-B). In contrast, our ViT-B framework combines LTG local–global contrastive learning, CSM class-aware attention, and FFCAM feature fusion in a single stage without external saliency or text supervision. This unified comparison makes the architectural and supervision differences explicit when interpreting the reported results.

As the results show, it can be seen that the pseudo-labelled mIoU generated by the present method achieves 73.4% as well as 72.3% on the training and validation sets, respectively, which represents competitive performance among the methods compared in Table 2. Compared with the SEAM method, the present method improves nearly 10 percentage points on the training set and validation set, respectively. Compared to the suboptimal ToCo, it has increased by 1.2 and 1.8 percentage points. This advantage validates the effectiveness of the pseudo-label generation strategy proposed in this paper in suppressing noise and preserving the integrity of target boundaries, while also indicating its stronger generalization ability and minimal accuracy degradation in the validation set (only 1.1%).

In order to further validate the effectiveness of the methods, the segmentation results of the current state-of-the-art methods are compared on the PASCAL VOC 2012 dataset, and the comparisons include the multi-stage WSSS methods NSROM, EPS, L2G, CLIP-ES, SEAM, and SIPE, W-OoD, MCTformer, and the single-stage methods AFA, SLRNet, ToCo and WeakTr. Table 3 shows the segmentation comparison results on the PASCAL VOC 2012 dataset segmentation comparison results, the third column in Table 3 indicates whether additional supervised information is used or not; Table 3 also indicates the use of pixel-level labelling information, the use of additional external salient maps, and the use of textual-linguistic supervised information.

Among single-stage methods, AFA, SLRNet, and ToCo obtain validation mIoU values of 66.3%, 67.6%, and 70.5%, respectively, while WeakTr reports 78.4%. Our method achieves 78.5% on the validation set, showing competitive performance among the compared single-stage methods.

Multi-stage methods such as NSROM, EPS, L2G, and CLIP-ES generally require iterative refinement or additional supervision. The strongest L2G and MCTformer validation scores are 71.7% and 71.6%, respectively. Our verified 78.5% result is 6.8 percentage points above L2G, demonstrating the effectiveness of the single-stage design while avoiding multi-stage error accumulation.

The advantages of our method mainly come from the following: firstly, the high-quality pseudo-label generation strategy provides more reliable supervision signals for the segmentation model; secondly, the end-to-end single-stage training based on ViT-B avoids error accumulation and additional module overhead in multi-stage methods.

On the MS COCO 2014 dataset, for the final segmentation results of the network, this paper is quantitatively compared with the multi-stage methods EPS, L2G, SEAM, SIPE, MCTformer, and the single-stage methods SLRNet, AFA, ToCo, WeakTr. As shown in Table 4, where it is denoted that the network uses additional supervisory information, the algorithm in this paper achieves better results among many methods.

In the single-stage method, the validation set accuracies of SLRNet, AFA, and ToCo are 35.0, 38.9, and 41.3, respectively, indicating relatively low performance. WeakTr reached 50.3, significantly better than previous methods. On this basis, the method in this article has been further improved to 50.9, showing competitive performance among the compared single-stage methods. Compared to WeakTr, it has increased by 0.6 percentage points. Although the absolute increase is not significant, even a small improvement on large-scale, multi class datasets like COCO requires substantial improvements in pseudo-label quality and model robustness. Meanwhile, this method does not require additional complex designs that WeakTr may rely on, demonstrating a better balance between efficiency and accuracy.

Among the multi-stage methods, L2G (42.0) and MCTformer (42.0) performed better, with SIPE of 40.6, EPS of 35.7, and SEAM of only 31.9. Compared with L2G/MCTformer, the method proposed in this article (50.9) has improved by 8.9 percentage points, with a relative increase of over 20%. This indicates that traditional multi-stage iterative refinement strategies are easily limited by the noise accumulation of initial pseudo-labels in scenarios such as COCO, which contain a large number of small targets, overlapping instances, and complex backgrounds. This article is based on the single-stage framework of ViT-B, which generates higher quality pseudo-labels (as shown in Table 2, a pseudo-label accuracy of 72.3 on VOC, with similar advantages on COCO) and trains the segmentation model end-to-end, effectively avoiding error diffusion and having more obvious advantages on complex datasets.

Table 5. Comparison of IoU per classification in the PASCAL VOC 2012 dataset.

Class	SEAM	AFA	TransCAM	ToCo	Ours
bkg	88.8	89.9	91.3	89.9	96.0
aero	68.5	79.5	81.9	81.8	87.0
bike	33.3	31.2	35.4	35.4	47.2
bird	85.7	80.7	87	68.1	92.8
boat	40.4	67.2	67.6	62.0	74.0
bottle	67.3	61.9	67.9	76.6	88.8
bus	78.9	81.4	87.5	83.6	91.8
car	76.3	65.4	80.5	80.4	85.5
cat	81.9	82.3	86.5	87.7	95.7
chair	29.1	28.7	31.4	25	36.5
cow	75.5	83.4	73.9	88.1	96.7
table	48.1	41.6	52.5	59	62.7
dog	79.9	82.2	80	87.0	91.6
horse	73.8	75.9	79	80	86.9
motor	71.4	70.2	76	76.0	80.8
person	75.2	69.4	79.0	68.2	84.8
plant	48.9	53.0	47	65.6	69.8
sheep	79.8	85.9	81	85.8	92.5
sofa	40.9	41	47.0	42.4	53.2
train	58.2	62	78.4	57.7	68.2
tv	53.0	50.9	46.6	65.6	66.0
mIoU	65.0	66.0	69.3	69.8	78.5

reports class-wise IoU on the PASCAL VOC 2012 validation set. Using 78.5% as the verified headline result, the class-wise entries and reported mean are mutually consistent. The proposed method achieves strong results for background (96.0%), bicycle (47.2%), bird (92.8%), bottle (88.8%), cat (95.7%), cow (96.7%), and sheep (92.5%), showing improved coverage for both small structures and large foreground regions.

In order to show the effectiveness of this paper’s algorithm more intuitively, we compare the class activation maps generated by the ViT benchmark network as well as the ToCo algorithm. As shown in Figure 7, the feature fusion class-aware activation maps generated by this paper’s algorithm are closer to the real labels, focus more on the local feature regions than the class activation maps generated by ViT, generate more complete response regions for small target objects, and are more refined in some boundary regions, e.g., the ear region of the sheep in Figure 7. This demonstrates that the local–global class labelling comparison algorithm proposed in this paper leads to enhanced classification consistency in local regions and more complete activation for small target regions. In addition, the class-aware attention generated by the proposed class-aware stimulus module can direct the ViT to focus on the correct target region, improve the differentiation of target classes, and further suppress the background noise.

The experiment also visualizes the PASCAL VOC 2012 and MS COCO 2014 segmentation results compared with the ToCo algorithm and the visualization results are shown in Figure 8. Panels (a), (b), (c) and (d) represent the results of the original image, real labels, ToCo and this paper’s algorithm, respectively. Through the visualization experiment comparison, it is found that this paper’s algorithm has significant enhancement in local features, generates a more complete region for some small target objects, has strong sensitivity to edge texture, and has a corrective effect for background mis-activation.

The chair category remains challenging, but our method reaches 36.5%. The train and television categories obtain 68.2% and 66.0%, respectively. Together with the 78.5% mean, these results indicate stable performance across categories with different shapes and background contexts.

The proposed method maintains an IoU above 36% for every class and reaches a mean of 78.5%. This balanced class-wise behaviour is consistent with the design of LTG, which strengthens local semantic consistency, and CSM, which suppresses category-irrelevant background activation.

As reported in

Table 6. Complementary evaluation metrics on the PASCAL VOC 2012 validation set.

Method	mIoU (%)	Cosine Similarity	MAE	MSE	RMSE
AFA	66.3	0.824	0.137	0.048	0.219
ToCo	70.5	0.873	0.112	0.035	0.187
WeakTr	78.4	0.908	0.083	0.023	0.152
Ours	78.5	0.914	0.079	0.021	0.145

, the proposed method shows favourable cosine similarity, MAE, MSE, and RMSE values among the compared methods. The complementary metrics therefore agree with the mIoU comparison: the predicted probability maps are more directionally consistent with the ground-truth and exhibit smaller pixel-wise deviations.

To complement the accuracy comparison,

Table 7. Model complexity and single-image inference time on an NVIDIA RTX 3090.

Method	Backbone	Params (M)	Inference Time (ms)
SEAM	ResNet38	41.2	48.6
ToCo	ViT-B	86.3	72.5
MCTformer	DeiT-S	44.7	61.3
WeakTr	ViT-B	86.3	74.1
Ours	ViT-B	88.1	76.8

reports parameter count and single-image inference time for representative CNN- and Transformer-based competitors under the same RTX 3090 environment. Our model has slightly more parameters and a modestly longer inference time than WeakTr and ToCo, while the accuracy gain on VOC and COCO offsets this limited efficiency cost. Compared with the CNN-based SEAM baseline, the single-stage ViT framework provides substantially higher segmentation accuracy with acceptable inference overhead.

4.4. Ablation Experiments

In order to further analyze the effectiveness of the method proposed in this paper, in this section, we conduct a series of ablation experimental studies. All the ablation experiments are conducted on the PASCAL VOC 2012 dataset, which mainly reports the mean intersection and merger ratio (mIoU) scores of the final CAMs generated on the validation set and the mean intersection and merger ratio (mIoU) scores of the semantic segmentation results. The main purpose of the ablation experiments is to evaluate the roles played by the local–global class labelling contrast module (LTG), the classification sense stimulus module (CSM), and the feature fusion classification aware activation map (FFCAM).

Table 8. Ablation study of the proposed components (“√” means the corresponding module is used).

Method	LTG	CSM	FFCAM	CAM (%)	Seg. (%)
Baseline	-	-	-	49.7	47.8
Ours	$\surd$	-	-	63	63.6
	$\surd$	$\surd$	-	69.4	69.7
	$\surd$	$\surd$	$\surd$	73.3	78.5

evaluates LTG, CSM, and FFCAM on PASCAL VOC 2012. The ViT baseline obtains 47.8% segmentation mIoU. Adding LTG increases the result to 63.6%. Adding CSM further improves it from 63.6% to 69.7% (+6.1 points), and replacing conventional CAM with FFCAM raises the full-model result from 69.7% to 78.5% (+8.8 points). The corresponding CAM mIoU values progress from 49.7% to 63.0%, 69.4%, and 73.3%.

In addition, the experiment visualizes the class activation maps generated by each group of components on the network, as shown in Figure 9, from which it is visually and intuitively obvious to find out the importance of each component for the optimization of the network model. The algorithm proposed in this paper continuously optimizes the initial CAM generated by ViT, which makes the network have a good response to the local small object features, and also provides a better solution to the problem of ViT incorrectly activating the background region as the foreground region.

The local–global class token comparison module is designed to facilitate a comparative analysis of the image representations of local and global views. The primary objective of this module is to ensure the maintenance of feature consistency within the target region across both local and global views. Additionally, it aims to enable the model to differentiate between various object regions. However, the size of the local crop image can pose a challenge, as a smaller local image may not contain foreground target objects, potentially leading to a target mismatch. Conversely, larger local images may contain an excessive number of foreground objects, resulting in a local view that is too similar to the global view, thereby hindering the full utilization of LTG for comparison learning. This paper presents an experimental study on the impact of local cropping size in LTG, as outlined in

Table 9. Effect of local-view crop size on CAM and segmentation mIoU (%).

Size	CAM mIoU (%)	Segmentation mIoU (%)
$64\times 64$	69.6	67.1
$80\times 80$	71.2	69.6
$96\times 96$	73.3	78.5
$112\times 112$	72.4	70.1

. The experimental results demonstrate that the optimal performance is achieved by cropping the local image to the size of 96 × 96.

Small crops may omit complete foreground semantics, whereas excessively large crops reduce the difference between local and global views and weaken contrastive learning. A crop size of 96 × 96 provides the best balance, achieving 73.3% CAM mIoU and 78.5% segmentation mIoU.

In LTG, the global projection head p1 is updated by EMA.

Table 10. Effect of EMA momentum on CAM and segmentation mIoU (%).

Size	CAM mIoU (%)	Segmentation mIoU (%)
0	70.1	68.6
0.1	72.7	70.4
0.9	73.3	78.5
0.99	71.7	69.1

evaluates the influence of the EMA momentum factor and shows that 0.9 provides the best result.

A small momentum factor causes unstable updates and introduces noise into the global representation, whereas an excessively large factor adapts too slowly to local semantic changes. Momentum 0.9 provides the best balance, reaching 73.3% CAM mIoU and 78.5% segmentation mIoU.

5. Conclusions

This study proposes a ViT-based weakly supervised semantic segmentation framework that addresses local feature insensitivity and background confusion. LTG aligns local views with global class semantics through contrastive learning. CSM transforms class-agnostic self-attention into class-aware attention by injecting category constraints. FFCAM then fuses ViT features with class-aware attention features to obtain more complete activation maps. Experiments on PASCAL VOC 2012 and MS COCO 2014 demonstrate the effectiveness of the proposed components and the overall single-stage framework.

Although we achieve excellent performance in multiple experiments, there are still some directions worth further research and optimization. The algorithm in this article is based on the Vision Transformer network for experimentation. Although ViT performs well in semantic segmentation tasks, its computational complexity is relatively high. Therefore, in the future, lightweight models can be adopted to reduce computational complexity and enable deployment on lower configured mobile or embedded devices, thereby enhancing the feasibility of practical applications.