A Multi-Scale Structure with Improved Reverse Attention for Polyp Segmentation

Abstract

Colorectal cancer (CRC) is the second most common global malignancy with high mortality, and timely early polyp detection is critical to halt its progression. Yet, polyp image segmentation—an essential tool—faces challenges: blurred edges, small sizes, and artifacts from intestinal folds, bubbles, and mucus. To address these, we proposed a novel segmentation model with multi-scale feature extraction. Its encoder uses Multiscale Attention-based Pyramid Vision Transformer v2 (PVTv2) for hierarchical features (lower-stage modules expand receptive field), while the decoder adopts a Parallel Multi-level Aggregation structure, plus multi-branch and improved reverse attention modules. Ablation experiments validated key modules. Compared to nine state-of-the-art networks across five benchmarks, the model showed superiority: optimal mDice/mIoU on polyp datasets, 0.2% higher mDice than MEGANet on Kvasir-SEG, and outperformance over UHA-Net and CSCA-U-Net on CVC-ClinicDB.

1. Introduction

Colorectal cancer (CRC) is one of the leading causes of cancer-related death and is the second most common cancer worldwide, with a persistently high mortality rate [1,2]. Early detection and removal of polyps are critical measures to prevent the progression of colorectal cancer [3]. Colonoscopy is the gold standard for the detection of colorectal lesions [4], and accurate localization of early-stage polyps is crucial for the clinical prevention of CRC.

However, this manual detection paradigm is prone to significant missed diagnoses—attributed to factors such as operator fatigue, subtle polyp morphology, and interference from intestinal physiological structures. To address this clinical bottleneck, integrating AI-driven deep learning and image segmentation techniques bridges subjective manual detection with objective computational analysis. This translates the clinical need for precise early polyp localization into a technical task of medical image segmentation, enabling quantitative polyp analysis to enhance diagnostic efficiency and accuracy [5], while mitigating manual limitations through synergistic integration with traditional inspection. Image segmentation represents a fundamental yet challenging research area within the domain of computer science [6]. These challenges are visually exemplified in Figure 1:

Figure 1. Polyp segmentation challenges in colonoscopy. In images presented by colonoscopy, the differences between polyp and non-polyp regions are usually not obvious. Meanwhile, factors such as air bubbles, mucus and intestinal folds further increase the difficulty of image analysis. The highlighted area in the Polyp Object indicates the polyp region.

As demonstrated in the figure, existing polyp segmentation methods still face substantial hurdles due to the diverse shapes and sizes of colon polyps. For instance, some polyps have blurred boundaries that are difficult to distinguish from normal tissues [7], while smaller polyp targets are easily overlooked or misidentified as intestinal folds—issues that also plague manual clinical inspection. Current methods are continuously addressing these challenges, exploring solutions, and achieving steady progress.

Here, we propose a novel approach, using a polyp segmentation model leveraging multi-scale feature extraction. It employs Multiscale Attention-based Pyramid Vision Transformer v2 (PVTv2) [8] as the encoder and a Parallel Multi-level Aggregation Decoder composed of attention and multi-scale parallel convolutions, incorporating an improved Scale Aggregation Reverse Attention. This strategy enhances learning and generalization capabilities, overcoming the weaknesses of existing models in challenging environments and better addressing the issues faced by current polyp segmentation models.

(1)

We introduce a Multiscale Attention (MA) module at the encoder’s lower stages. This module utilizes multiscale attention mechanisms and dilated convolutions with multiple sizes to expand the receptive field, improving performance in low-layers feature map extraction.

(2)

The proposed method introduces an integrated decoder that incorporates an Attention Gate (AG) module and multi-branch convolution modules to enhance segmentation accuracy for polyps of varying sizes and shapes, leveraging hierarchical feature supplementation and refinement.

(3)

The proposed method integrates an Improved Reverse Attention module, which specializes in polyp boundary segmentation tasks via enhanced feature extraction and reverse attention mechanisms.

(4)

To verify the practical effectiveness of the proposed model, experiments were conducted on five challenging benchmark datasets. The results demonstrate that the model exhibits reliable performance and strong versatility as a segmentation tool.

To address the key challenges prevalent in existing polyp segmentation, such as unclear boundary segmentation and inadequate detailed feature extraction, this paper adopts a systematic research framework and arranges the specific structure with core contents as follows: Section 2 systematically reviews existing deep learning architectures and combs through previous research achievements in the field of polyp segmentation, clarifying the current technical pain points in boundary and detail extraction, and providing targeted directions for subsequent model design; Section 3 deeply elaborates on the design of the three core components of the proposed model, including the Multiscale Attention (MA) module deployed in the lower stages of the encoder to enhance feature extraction, the integrated decoder that improves the segmentation accuracy of diverse polyps, and the Improved Reverse Attention module focusing on boundary segmentation; Section 4 details the experimental procedures and results, comparing our model with the current state-of-the-art (SOTA) models to evaluate its performance. It also includes ablation experiments to assess the contribution of each core component. Finally, Section 5 discusses the effectiveness of the model and summarizes the contributions made in this paper.

2. Related Work

In recent years, deep neural networks have been widely and deeply used in the domain of image processing due to their excellent performance [9,10]. Especially in medical image segmentation, the convolutional neural network (CNN) framework forms the core cornerstone of most advanced models. Long et al. [11] took the lead in making a breakthrough by replacing all linear layers with convolutional layers to design a fully convolutional network (FCN). Later, Ronneberger et al. [12] constructed a symmetric structure and proposed a U-shaped architecture (U-Net) with shrinking paths and expanding paths. Because of its simple structure, few parameters, and rapid convergence speed, U-Net performs well as a benchmark model in this field. Zhou et al. [13] put forward a modified variant of U-Net, known as U-Net++, which incorporates dense skip connections linking the encoder and decoder components. Oktay et al. [14] added attention mechanisms to enhance spatial perception and developed Attention U-Net. In addition, Jin et al. [15] cleverly combined residual learning and attention mechanisms to propose that residual attention U-Net (RA-U-Net) has achieved significant results in multiple organ classification tasks.

Transformer [16], as a large semantic network model specifically designed to capture global semantic information in language, has garnered widespread academic attention due to its exceptional global semantic extraction capabilities and spatially adaptive information aggregation properties. Dosovitskiy et al. [17] introduced Transformer to the field of computer vision by incorporating positional encoding and patch embedding techniques, pioneering the Vision Transformer (ViT). Chen et al. [18] proposed a novel network structure combining Transformer with U-Net—the TransUNet model—which integrates Transformer into the encoder stage and upsamples encoder features in the decoder before combining them with CNN outputs. Building on this, Wang et al. [19] further integrated the pyramid structure into the Transformer architecture, thereby proposing the Pyramid Visual Transformer (PVT), which offers new approaches for the multi-scale representation of image features. Base on PVT, optimized models like SS-Former [20] and FCBFormer [21] have shown remarkable performance in related tasks, further promoting the application of Transformer in the field of image segmentation.

However, Transformer-based models come with their own challenges. They possess a vast number of parameters and demand substantial computational resources, particularly exhibiting relatively insufficient capability in capturing local features when processing small datasets like medical images. The multi-scale pyramid structure design of CNNs (Convolutional Neural Networks) significantly enhances the model’s ability to handle multi-scale problems, improving performance across multiple dimensions such as multi-scale representation, local feature extraction, feature reuse, robustness, and interpretability. Wang et al. [19] ingeniously integrated the pyramid structure from CNN architectures with the original advantages of ViT (Vision Transformer), aiming to optimize ViT ’s efficiency in processing high-resolution images, thereby giving rise to the Pyramid Vision Transformer (PVT) model. Unlike strategies such as VGGNet [22] and ResNet [23], which obtain multi-scale feature maps through varying convolutional strides, PVT adopts a progressive shrinking approach. It flexibly adjusts the scale of feature maps through patch embedding layers and encoders with built-in spatial reduction mechanisms, preserving the traditional CNN’s ability to generate multi-scale feature maps while effectively reducing computational overhead. Subsequently, in PVTv2 [8], Wang et al. improved PVT’s performance by incorporating linear-complexity attention layers, overlapping patch embedding, and convolutional feedforward networks. UNeXt [24] and Uformer [25] drew inspiration from the UNet architecture from different perspectives to optimize Transformer models for computer vision tasks. CSWin Transformer [26] employs a multi-layer Transformer structure and introduces residual-like connections between modules. HRFormer [27] integrates Transformer modules based on the multi-resolution feature fusion architecture of HRNet [28]. Swin UNETR [29] uses only Swin Transformer [30] as the encoder and CNN as the decoder, where local and global features are alternately extracted, leading to potential confusion and incorrect feature acquisition.

Several methods in polyp segmentation showcase varied strengths and constraints, affecting their segmentation efficacy. PraNet [31] Initially applies high-level features to generate a coarse prediction of polyp regions, then employs a Reverse attention module (RA) to progressively refine the edge details of polyps from the background. Inspired by PraNet, UACANet [32] designs a novel parallel axial attention and introduces uncertain regions to the boundary information. CaraNet [33] adopts an Axial reverse attention (A-RA) module to tackle segmentation issues involving diminutive polyp entities. This module effectively filters out disturbances and sharpens boundaries, although it lacks the capability to supplement and explore additional features. ADS_UNet [34] integrates AdaBoost and a heuristic stepwise training strategy into the iterative construction of a collective predictive model, thereby achieving higher accuracy than other UNet-like architectures. M²SNet [35] leverages pyramid aggregation based on its proposed intra-layer multi-scale subtraction unit, enabling cross-layer complementarity between low-layer and high-level features. UHA-Net [36] introduces a hierarchical feature fusion module and an uncertainty-induced cross-level fusion module for feature aggregation, ultimately combining multi-scale features. EFA-Net [37] improves the accuracy of polyp segmentation through the exploitation of inter-layer features for enhanced edge recognition. Texture traits also have a key role in telling polyp foregrounds apart from backgrounds. From the above polyp segmentation models, it is evident that feature extraction for polyp regions and boundary issues have consistently been focal points.

However, most of the SOTA polyp segmentation methods have difficulties in representing multi-scale features and are unable to draw the feature boundaries of polyps under problematic imaging conditions: U-Net variants (U-Net++, Attention U-Net) lack adaptive multi-scale fusion, leading to blurred boundary segmentation; Transformer-based models (TransUNet, PVT) suffer from high computational costs and weak local feature capture on medical small datasets; reverse attention methods (PraNet, CaraNet) fail to supplement discriminative features, resulting in incomplete small polyp segmentation.

The proposed model achieves superiority via targeted innovations: it adopts a PVTv2 encoder with Multiscale Attention to balance global semantics and local features, resolving Transformer’s inefficiency. Its Parallel Multi-level Aggregation decoder with Attention Gate and multi-branch convolutions enables dynamic hierarchical feature refinement for diverse polyp sizes and shapes, outperforming U-Net variants’ fixed skip connections. The Improved Reverse Attention module enhances edge extraction, mitigating artifacts (mucus, bubbles) that hinder PraNet and CaraNet.

3. Methodology

3.1. Overall Architecture

The inconsistency in defining segmentation targets in medical imaging is a well-documented issue, leading to suboptimal segmentation outcomes, poorly defined boundaries, and considerable challenges in achieving accurate diagnosis and the subsequent treatment planning [38]. To address these challenges, a novel multi-scale architectural framework is proposed, as shown in Figure 2.

Figure 2. Architecture of the proposed polyp segmentation model. Illustrates the architecture of the proposed model, which includes the backbone PVTv2, Parallel Multi-level Aggregation Decoder (PMAD), and Improved Reverse Attention Module.

The proposed model adopt the same structural design as U-net, featuring an encoder-decoder architecture, which is of the utmost importance for diagnostic and treatment planning tasks in medical imaging. This allows the network to the features that the image contains, as a process of image encoding is being undertaken. Subsequently, during the decoding phase, the segmentation results captured in the encoding phase must then be reinstated. Our model primarily consists of the following key components: the encoder, the decoder and the Improved Reverse Attention Module, as as illustrated.

In the encoder stage, the model employ PVTv2, which adapts to visual tasks by combining attention mechanisms adept at handling spatial and multi-scale features. This structured approach ensures that the model learns to identify polyp tissues with a high degree of precision. It achieves this by considering both the local context, such as the fine details of polyp edges and texture differences between lesions and surrounding mucosa, and the global context, including the overall distribution of polyps in the intestinal lumen and their positional relationships with large-scale anatomical structures within the image. Additionally, we introduce the Multiscale Attention module at the bottom stage of the encoder, which analyzes multi-level characteristics across various viewpoints, synthesizing both localized and extensive data for enhanced feature extraction compared to those that depend exclusively on single-scale cues. The decoder is specifically designed to leverage pyramid features—multi-scale feature maps extracted from the encoder backbone of the segmentation model. We combine attention with various parallel convolutional blocks to propose the Parallel Multi-level Aggregation Decoder, enabling the decoder to extract refined pyramid features as comprehensively as possible and further aggregate these features. Finally, through the Improved Reverse Attention module, we process the decoder output using the Scale Aggregation module, followed by reverse attention feature extraction, to further address boundary issues in polyp segmentation.

3.2. Multiscale Attention Module

Vision Transformer employs a global attention mechanism, facilitating the model’s comprehension of interconnections among diverse image spatches. However, computational costs grow rapidly as the number of image patches increases. For many typical computer vision tasks, modeling relationships between all image patches is not invariably essential. Some relationships may be redundant or have minimal impact on the final results.

By default, we use a convolution kernel size of 3 × 3 with a size and use hole convolutions with dilation ratios of 1, 2, and 3, which will bring the receptive field sizes of each head to 3 × 3, 5 × 5, and 7 × 7, respectively.The 3 × 3 convolution kernel is selected as the base due to its balance of effective local feature extraction, lower computational complexity compared to larger kernels such as 5 × 5 and 7 × 7, and flexibility in receptive field expansion via dilation. Dilation ratios 1, 2, and 3 are adopted to generate multi-scale receptive fields for each head, enabling adaptive capture of fine polyp boundaries, medium-sized regions, and large polyp structures while avoiding parameter redundancy.

Local attention [39] reduces computational complexity but compromises long-range dependency modeling and global receptivity, presenting an inherent trade-off. To address this limitation, we propose a novel Multiscale Attention (MA) module, which is uniquely integrated into the multi-head attention of PVTv2’s bottom encoder stages, as shown in Figure 3.

Figure 3. Multiscale Attention (MA) module architecture schematic. The feature map’s channels are segmented into various components, and the self-attention process takes place among the blue blocks nestled in the vicinity of the red query block. This is all done while employing a range of dilation rates throughout the different segments. Additionally, concatenation of various section features is then presented to a linear processing layer. By default, we use a kernel size of 3 × 3, with dilation rates r = 1, 2, and 3, resulting in receptive field sizes of 3 × 3, 5 × 5, and 7 × 7 for the respective heads.

This module innovatively extends multi-head attention with flexible, cost-efficient sparse global attention: by incrementally expanding the receptive field without extra computation, it captures broader contextual dependencies while preserving fine-grained local details. Its tailored adaptation to PVTv2’s bottom layers specifically enhances low-level feature perception—critical for resolving subtle structures like small polyp edges and texture variations—thereby strengthening downstream segmentation precision. This design overcomes the limitations of conventional local attention in modeling long-range interactions, marking a distinct advance in optimizing encoder.

To formalize the computational workflow of the Multiscale Attention (MA) module, as shown in Algorithm 1, the following algorithm details its step-by-step process, including channel segmentation, multi-dilation convolution operations, self-attention computation, and feature fusion, with specific parameter settings as described.

Algorithm 1 Multiscale Attention (MA) Module Algorithm

Input: Feature map $X\in {\mathbb{R}}^{H\times W\times C}$ (where H = height, W = width, C = channels) Output: Enhanced feature map $X_{\mathrm{out}}\in {\mathbb{R}}^{H\times W\times C}$   1: // 1. Channel Segmentation   2: Split X’s channels C into 4 equal segments: $X_1,X_2,X_3,X_4$ (each $\in {\mathbb{R}}^{H\times W\times C/4}$)   3: Designate $X_1$ as red query block, $X_2$-$X_4$ as blue neighbor blocks   4: // 2. Multi-Dilation Self-Attention   5: Define dilation rate set $\mathcal{R}=[1,2,3,3]$ (corresponding to $X_1$-$X_4$)   6: for $i=1$ to 4 do   7:     Apply 3 × 3 dilated convolution with rate $\mathcal{R}\left[i\right]$ to $X_i$: $X_i^{\mathrm{dilated}}=\mathrm{DilatedConv}(X_i,3,\mathcal{R}\left[i\right])$   8:     if $i=1$ then                     ▹ Red query block (Q)   9:         $Q=X_i^{\mathrm{dilated}}$ 10:     else                   ▹ Blue neighbor blocks (K/V) 11:         $K_i=X_i^{\mathrm{dilated}}$, $V_i=X_i^{\mathrm{dilated}}$ 12:         Compute attention score: $A_i=\mathrm{Softmax}\left({\displaystyle \frac{Q\times K_i^T}{\sqrt{C/4}}}\right)$ 13:         Attended feature: $X_i^{\mathrm{attended}}=A_i\times V_i$ 14:     end if 15: end for 16: $X_1^{\mathrm{attended}}=Q$                  ▹ Retain query block feature 17: // 3. Feature Fusion & Linear Processing 18: Concatenate attended segments: $X_{\mathrm{concat}}=\mathrm{Concat}(X_1^{\mathrm{attended}},X_2^{\mathrm{attended}},X_3^{\mathrm{attended}},X_4^{\mathrm{attended}})$ 19: Linear projection via 1×1 convolution: $X_{\mathrm{linear}}=\mathrm{Conv}1\text{×}1\left(X_{\mathrm{concat}}\right)$            ▹ Maintain $H\times W\times C$ 20: // 4. Output Enhanced Feature Map 21: $X_{\mathrm{out}}=X_{\mathrm{linear}}$ return $X_{\mathrm{out}}$

3.3. Parallel Multi-Level Aggregation Decoder

The proposed parallel multi-level aggregation decoder includes UP blocks, an Attention Gate (AG), a Parallel Multi-level Aggregation (PMA) module, and an Adjacent Supplement Partial Decoder (ASPD). The UP blocks enhance subdivision by preserving spatial details during upsampling, avoiding loss that blurs boundaries. The AG module boosts subdivision through cascaded fusion, focusing on critical cross-level features to sharpen fine-grained distinctions. The PMA module deepens subdivision via robust refinement and integration, extracting nuanced details beyond basic segmentation. The ASPD module enhances subdivision by aggregating final refined features, strengthening contextual coherence to clarify subtle subdivisions—each module’s originality lies in targeted, non-segmentation-based enhancement of feature subdivision. The parallel multi-level aggregation decoder streamlines segmentation by combining precise upsampling, focused attention, robust refinement, and contextual enhancement, ensuring sharp, fine-grained subdivisions and clear boundaries throughout the process.

3.3.1. Attention Gate (AG)

Low-layer features are considered to contain substantial redundant information, making them unsuitable for direct integration into subsequent feature extraction processes. Additionally, when low-layer features are directly upsampled and added to higher-layer features, it may lead to the loss of fine-grained details. Therefore, the model incorporated the Attention Gate module. Similar to Attention U-Net, the AG can assign varying degrees of weighting to different parts of the input data, enabling the network to focus on more critical regions.

$$q_{att}={\sigma }_1\left(BN\left(Conv1\times 1_g\left(g\right)\right)+BN\left(Conv1\times 1_x\left(x\right)\right)\right)$$

(1)

$$AG(g,x)=x\times {\sigma }_{2}BN(Conv1\times 1_q\left(q_{att}(g,x)\right))$$

(2)

Here, ${\sigma }_1(·)$ and ${\sigma }_2(·)$ correspond to the ReLU and Sigmoid activation functions, and $q_{att}(·)$ denotes the process of computing the attention query vector based on the inputs, respectively. $BN(·)$ denotes batch normalization. Respectively g and x represent the upsampled and encoder output features. By incorporating the AG, nonlinearity can be significantly enhanced without altering the scale, refining the encoder’s output and enabling cross-channel information integration.

3.3.2. Parallel Multi-Level Aggregation (Pma)

Encoder convolution and downsampling increases feature dimensionality but reduces spatial scale, risking loss of small polyp details unrecoverable via decoder upsampling. Following the AG module, the novel Parallel Multi-level Aggregation (PMA) module includes Polarized Self-Attention (PSA) [40] and a Multi-branch Feature Aggregation (MFA) module, as shown in Figure 4. PSA enhances subdivision by fusing channel and spatial attention to pinpoint and refine critical features and their positions, sharpening fine-grained distinctions without segmentation. MFA boosts subdivision via parallel multi-scale convolutional blocks, integrating diverse visual scales to capture nuanced details—both modules’ originality lies in non-segmentation-based, targeted enhancement of feature subdivision.

Figure 4. The architecture of the Parallel Multi-level Aggregation (PMA) Module. The PMA module comprises a Polarized Self-Attention module and a feature refinement block, where the latter further consists of Multi-branch Feature Aggregation (MFA), a shortcut connection, and a residual connection.

To better refine feature maps, CBAM’s sequential spatial and channel attention structure [41] is adopted: Channel Attention (CA) focuses on and optimizes key feature maps; Spatial Attention (SA) highlights critical locations. Unlike CBAM (relying on fully connected and convolutional layers, limiting information mining), we adopt PSA’s sequential design to enhance features and suppress noise. The self-attention module converts input x into query Q (fully compressed channels) and value V (retaining ($C/2$) channels), enabling efficient long-range modeling via Q-dimensionality reduction—ensuring computational efficiency, specific-direction feature compression, and perpendicular high resolution. Q is then enhanced via High Dynamic Range (HDR), with dimensionality increased through matrix multiplication and 1 × 1 convolution. By leveraging PSA’s efficient self-attention mechanism and the targeted enhancement of key features through Channel and Spatial Attention, the module refines feature maps and improves segmentation precision, particularly in capturing fine-grained spatial details and reducing noise interference. The entire process is as follows:

$$PSA\left(x\right)=A^{ch}{\odot }^{ch}\left(A^{sp}{\odot }^{sp}x\right)$$

(3)

In Equation 3, $A^{ch}$ and $A^{sp}$ are the channel attention matrix and spatial attention matrix, respectively; ${\odot }^{ch}$ and ${\odot }^{ch}$ are the channel-wise multiplication operator and spatial multiplication operator.

The Multi-branch Feature Aggregation (MFA) module, a clever multi-convolutional setup, takes segmentation to the next level through three ingenious pathways. Borrowing a page from RFB Net [42], it leverages various hole convolutions to beef up receptive fields, nailing down those all-important multi-scale features that make or break fine-grained segmentation edges. The first pathway stacks 3 × 3 dilated convolutions to weave together multi-scale details, giving object contours a much-needed sharpening. The second strikes a balance between scale variety and computational efficiency by employing 5 × 5 convolutions alongside diagonal ones, zeroing in on spatial subtleties that often define tricky or oddly shaped regions. The third pathway, armed with 3 × 3 diagonal convolutions and feature-organizing layers, keeps spatial information intact—a must-have for dead-on localization. This multi-pronged approach cranks up non-linearity, letting the network flex both global and local feature muscles; optimizing each branch independently also lightens the load, paving the way for deeper networks that pack richer segmentation insights. When you throw these aggregated features into the mix with smart shortcut and residual connections that keep the original data intact, the module really hits its stride, telling fine-grained segments apart rather than just carving up space willy-nilly. By strategically combining multi-scale features and enhancing spatial details through each pathway, the MFA module significantly improves segmentation accuracy, ensuring precise delineation of fine-grained boundaries and achieving sharper, more reliable object contours.

To clarify the operational mechanism of the MFA module, the following pseudocode formalizes its multi-branch convolutional design, receptive field expansion via dilated convolutions, and feature aggregation strategy, along with residual shortcut connections.

3.3.3. Adjacent Supplement Partial Decoder

The Adjacent Supplement Partial Decoder is added after the PMA module as a partial encoder. Due to the nature of pyramid features, multi-level pyramid features are generated at both the encoder and decoder stages. However, two key issues remain when aggregating multiple feature pyramids: how to maintain semantic consistency within layers and how to bridge context across layers. Here, we draw inspiration from the Cascaded Partial Decoder (CPD) [43] by introducing a partial encoder to aggregate high-level output features. More specifically, we use the Adjacent Supplement Partial Decoder (ASPD) to combine these discriminative features through concatenation. Here, $x_1$, $x_2$, $x_3$, and $x_4$ denote the multi-level pyramid features generated at different stages of the encoder and decoder, with $x_1$ representing the lowest-layer feature (rich in original details) and $x_4$ the highest-layer feature (more refined in semantics). Lower-layer features are upsampled to the same scale as the higher-layer features and then optimized via a 3 × 3 convolutional layer. These features are multiplied level-by-level with the higher-layer features, and the three resulting products are concatenated to obtain the final feature map. As shown in Figure 5, the computation involves processing $x_1$, $x_2$, $x_3$ and $x_4$ sequentially. Lower-layer features contain more original information, while higher-layer features are more refined. Multiplying the processed lower-layer features with higher-layer features and then combining multi-level features allows for better integration of contextual information, further highlighting the characteristics of each layer. Concatenating the results helps avoid losing optimized feature information.

Figure 5. The Adjacent Supplement Partial Decoder (ASPD) Module structure description. Illustrates the structure of the ASPD module, where UP represents a combination of a 3 × 3 convolutional layer and an upsampling operations. The input consists of lower-layer features processed through the UP block, which are then multiplied with higher-layer features. The outputs from each layer will be connected together to form the final output result.

The Algorithm 2 is shown below:

Algorithm 2 Multi-branch Feature Aggregation (MFA) Module Algorithm

Input: Input feature map $X\in {\mathbb{R}}^{H\times W\times C}$ Output: Aggregated and refined feature map $Y\in {\mathbb{R}}^{H\times W\times C}$   1: // Branch 1: Stacked 3×3 dilated convolutions   2: $B_1=\mathrm{Conv}3\text{×}3(X,\mathrm{dilation}=1)$               ▹ Receptive field: 3 × 3   3: $B_1=\mathrm{Conv}3\text{×}3(B_1,\mathrm{dilation}=2)$              ▹ Receptive field: 5 × 5   4: $B_1=\mathrm{Conv}3\text{×}3(B_1,\mathrm{dilation}=3)$              ▹ Receptive field: 7 × 7   5: // Branch 2: 5×5 convolution + diagonal convolution   6: $B_2=\mathrm{Conv}5\text{×}5\left(X\right)$                ▹ Capture large-scale features   7: $B_2=\mathrm{DiagConv}3\text{×}3\left(B_2\right)$           ▹ Focus on diagonal spatial details   8: // Branch 3: Diagonal convolution with feature organization   9: $B_3=\mathrm{DiagConv}3\text{×}3\left(X\right)$               ▹ Diagonal kernel, size 3 × 3 10: $B_3=\mathrm{Conv}3\text{×}3\left(B_3\right)$                   ▹ Feature organization 11: $B_3=\mathrm{Conv}3\text{×}3\left(B_3\right)$                   ▹ Enhance non-linearity 12: // Feature aggregation and residual connections 13: $B_{\mathrm{concat}}=\mathrm{Concat}(B_1,B_2,B_3)$            ▹ Merge multi-branch features 14: $B_{\mathrm{proj}}=\mathrm{Conv}1\text{×}1\left(B_{\mathrm{concat}}\right)$               ▹ Channel alignment to C 15: // Shortcut and residual connections 16: $\mathrm{Shortcut}=\mathrm{Conv}1\text{×}1\left(X\right)$               ▹ Preserve original features 17: $\mathrm{Residual}=\mathrm{ResBlock}\left(\mathrm{ResBlock}\right(\mathrm{ResBlock}\left(X\right)\left)\right)$    ▹ Three residual blocks 18: // Final refinement 19: $Y=\mathrm{ReLU}(B_{\mathrm{proj}}+\mathrm{Shortcut}+\mathrm{Residual})$ ▹ Element-wise addition return Y

3.4. Improved Reverse Attention Module

The decoder’s output is further processed by the Scale Aggregation (SA) module to attain a more accurate and comprehensive prediction map, as shown in Figure 6. The SA module consists of convolutions of different scales connected via concat. The reverse attention module is enhanced by a novel residual axial attention block for gradual correction of polyp boundaries. Therefore, we believe convolutional layers are crucial for the refinement module, as they emphasize high-frequency features and offer more detailed edge information for boundary refinement. The refinement module learns multi-scale features using convolutional kernels of varying sizes to handle scale variations effectively. To aggregate multi-scale information, features are concatenated along a specific dimension. The detailed flowchart of scale aggregation is illustrated in the figure. For convenience, we denote the input to SA as x. Specifically, spatial features are extracted from different scales by sequentially performing dilated convolutions, including 3 × 3, 5 × 5, and 7 × 7 operations. The features across various scales are subsequently concatenated. Additionally, a 1 × 1 convolutional layer is added for skip connections, ensuring the final aggregated features retain the original input information and effectively integrate the results. To reduce computational overhead, multiple 3 × 3 convolutions are used instead of 5 × 5 and 7 × 7 operations. Stacked convolutional layers provide more activation functions, enhancing the network’s nonlinearity and improving feature extraction and fusion.

Figure 6. Architecture of the Scale Aggregation (SA) module. Implements multi-scale convolutional operations using consecutive 3 × 3 convolutional layers and adds shortcuts to enhance residual connections.

The employment of the SA module’s multi-scale convolutional layers and residual axial attention block within the Improved Reverse Attention Module facilitates the refinement of polyp boundaries. This module enhances edge clarity, mitigates blurring via multi-scale feature aggregation, enables precise boundary correction in ambiguous regions, and achieves more accurate, detailed segmentation. The aforementioned process can be expressed as follows:

$$SA\left(x\right)=Cat(Conv3\times 3(x),Conv5\times 5(x),Corv7\times 7(x\left)\right)+Conv1\times 1\left(x\right)$$

(4)

In Equation 4, $Cat(·)$ denotes the concatenation operation. Subsequently, reverse attention is incorporated, drawing inspiration from PraNet’s parallel reverse attention network design. Starting from the saliency map generated at the deepest layer, existing estimated polyp regions are erased to compute the weights of the reverse attention module. Higher-layer output features are then subjected to element-wise dot products, using deeper features to sequentially mine complementary regions and details.

3.5. Loss Function

When analyzing polyp images, considering both the boundaries and polyp regions significantly contributes to segmentation performance. Specifically, the global loss optimizes overall polyp segmentation accuracy by focusing on the entire polyp region and background, while the local loss enhances boundary precision by assigning higher weights to edge pixels to address indistinct polyp edges in colonoscopy images. The proposed loss function employs a weighted combination [44] of Intersection over Union (IoU) [45] and Binary Cross-Entropy (BCE). The BCE effectively mitigates class imbalance through pixel-wise classification penalties, while IoU guarantees spatial consistency by quantifying regional overlap—both are critical for clinically relevant polyp segmentation—with their weighted formulation enabling adaptive alignment with task-specific challenges. Our loss calculation is as follows:

$$L_{out}=L_{IoU}^w+L_{BCE}^w$$

(5)

During loss computation, to better address boundary issues by increasing weights, encouraging the model to focus more intently on the edge regions, we introduce multiple prediction heads to separately calculate global and local losses. For model training, we use the decoder’s final aggregated output and the outputs of three reverse attention modules as prediction heads, and then use additive aggregation to calculate the final prediction map, as shown in Equation (6):

$$L_{out}=a\times L_1+b\times L_2+c\times L_3+d\times L_4$$

(6)

where $L_1$, $L_2$, $L_3$ and $L_4$ are the feature maps of the four prediction heads; a, b, c and d are the weights of each prediction head, all set to 1.0 here.

4. Experiments

4.1. Dataset

Experiments are conducted on five popular benchmark polyp segmentation datasets: Kvasir-SEG [46], CVC-ClinicDB [47], CVC-ColonDB [48], Endoscene [49] and ETIS-LaribPolypDB (also known as CVC-T) [50]. The details of this five datasets are described below:

As shown in Table 1, following the configuration of PraNet, the Kvasir-SEG and CVC-ClinicDB datasets are utilised for training purposes, with 80% of the data allocated for this purpose. Validation is performed using 10% of the dataset, and testing is conducted with a further 10% of the data. To assess generalisation-related performance, the model is tested on three unseen small datasets: EndoScene, CVC-ColonDB, and ETIS-LaribPolypDB. Due to varying image sizes in these three datasets, all exceeding 500 × 500, we resize all training images to 256 × 256 to conserve computational resources and accelerate convergence.

Table 1. The number and size of five challenging benchmark polyp datasets is provided, in addition to the division of the train and test datasets followed the Pranet.

Dataset	Number	Size (Pixels)	Train	Test
Kvasir-SEG [46]	1000	range from 332 × 487 to 1920 × 1072	900	100
CVC-ClinicDB [47]	612	384 × 288	548	64
CVC-ColonDB [48]	380	574 × 500	-	380
Endoscene [49]	60	1225 × 966	-	60
ETIS-LaribPolypDB [50]	196	1226 × 996	-	196

4.2. Implementation Details

Our experiments were conducted using PyTorch 1.11.0, torchvision 0.12.0, and Python 3.9.18 via the Anaconda command shell. The experimental platform for training and testing on a computer equipped with 13th Gen Intel(R) Core(TM) i7-13700KF @ 3.40 GHz and NVIDIA GeForce RTX 4080 with 32 GB of memory.

During training, the pre-trained weights derived from ImageNet dataset [51] were utilised for the PVTv2-b2 backbone network for faster convergence, higher training efficiency, and enhanced performance in intestinal polyp segmentation. AdamW combined with weight decay effectively suppresses overfitting, while a 0.0001 learning rate balances convergence speed and stability; the 256 × 256 size achieves a trade-off between feature retention and computational efficiency. And multi-scale training was utilized with scales of 0.75, 1.0, and 1.25 to enhance the model’s adaptability to variations in input size is imperative, as is the improvement of its generalisation performance, serving as an alternative to data augmentation. Additionally, for better data augmentation, gradient clipping was set to 0.5, with a decay cycle of 200, and random flipping was applied to the images. All models in our tests made use of publicly available code, with the datasets being public and the pre-trained weights provided by the original authors.

Following the reverse attention mechanism module, our final prediction map $P_1$, $P_2$, $P_3$ and $P_4$ is generated by summation. The entire of the network is trained end-to-end, taking roughly 600 min to converge over 100 epochs with batches configured to a size of 16. To avoid underfitting or overfitting, we store the train-weights at the end of each epoch if performance on the validation set increases.

4.3. Evaluation Metrics

We primarily adopted the Mean Dice Similarity Coefficient (mDice) [52] and Mean Intersection over Union (mIoU) as evaluation metrics to evaluate the model’s performance on the target dataset. The mDice is a set similarity measure function that is frequently employed to calculate the degree of similarity between two samples. It operates on a scale that varies from 0 to 1. The IoU metric is widely used in semantic segmentation, referring to the prediction accuracy for each category. This evaluation measure is a frequently used to gauge the models’ effectiveness in image segmentation, where increased values reflect a stronger correspondence between the segmented outputs and the actual ground truth.

Additionally, we employed other evaluation metrics such as the weighted F-measure ($F_{\beta }^w$) [53], Structural measure ($S_{\alpha }$) [54], Enhanced-alignment measure ($E_{\varphi }^{max}$) [55] and Mean Absolute Error ($MAE$). The $F_{\beta }^w$ calculates a weighted harmonic mean to balance precision and recall. The $S_{\alpha }$ gauges structural resemblance between the prominence map and the reference standard. The $E_{\varphi }^{max}$ employs an improved alignment matrix $\varphi$ reflecting two binary map characteristics. The $MAE$ is the average absolute error between observed and true values, assessing pixel-level segmentation accuracy. Parameters (Params) refer to the total number of trainable variables in a model, directly related to its capacity and parameter efficiency. Floating-point operations (FLOPs) quantify the computational complexity of model training/inference, reflecting the required computing resources. Speed (fps, frames per second) denotes the number of input frames a model processes per second during inference, a key metric for evaluating real-time performance in latency-sensitive scenarios.

4.4. Results

Learning and generalization capability analysis: An evaluation was conducted for the purpose of determining the model’s performance when applied to five distinct datasets, with Kvasir-SEG and CVC-ClinicDB datasets used primarily to assess the capability of feature modeling, while the other unseen datasets were used to evaluate generalization ability. To assess model lightweighting, we compared FLOPs, Speed, and Params metrics, with experimental results.

The present study constitutes an experimental comparison of disparate network models on the Kvasir-SEG and CVC-ClinicDB datasets. The results on Kvasir-SEG are shown in Table 2. For ease of visualization, the two highest-scoring responses are distinguished by red and blue, respectively.

Table 2. The results on the Kvasir-SEG for different models in a few noteworthy cases.

Methods	Year	Kvasir-SEG
		$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}^{\mathit{max}}\uparrow$	$\mathit{MAE}\downarrow$
U-Net [12]	2015	0.818	0.746	0.794	0.858	0.893	0.055
U-Net++ [13]	2018	0.821	0.743	0.808	0.862	0.910	0.048
SFA [56]	2019	0.723	0.611	0.670	0.782	0.849	0.075
PraNet [31]	2020	0.898	0.840	0.885	0.915	0.948	0.030
MSNet [57]	2021	0.907	0.862	0.893	0.922	0.944	0.028
Enhanced U-Net [58]	2021	0.908	0.854	0.893	0.917	0.954	0.028
CFA-Net [59]	2023	0.915	0.861	0.903	0.924	-	0.023
MEGANet(Res2Net-50) [60]	2024	0.913	0.863	0.907	0.918	0.959	0.025
CSCA U-Net [61]	2024	0.903	0.846	0.890	0.918	0.951	0.031
Proposed		0.923	0.870	0.917	0.925	0.967	0.023

We compared the metric data with nine other state-of-the-art (SOTA) models. The comparative results are presented in Table 2 and Table 3, including reports on U-Net [12], U-Net++ [13], SFA [56], PraNet [31], MSNet [57], CFA-Net [59], MEGANet(Res2Net-50) [60] and CSCA U-Net [61]. All comparative data were sourced directly from the original papers. The experimental results demonstrate that our model significantly outperforms both traditional and modern segmentation techniques. As shown in Table 2 and Table 3, our model exhibits improved feature modeling capabilities compared to other models and demonstrates better generalization across other datasets. Particularly in cross-dataset evaluations, our model outperformed the second-best CFA-Net model by 0.8% in the mDice metric on the Kvasir-SEG dataset and surpassed MEGANet and CSCA U-Net on the CVC-ClinicDB dataset. It also maintained strong performance in mIoU, leading significantly on the Kvasir-SEG dataset and trailing MEGANet by only 0.2% on the CVC-ClinicDB dataset. Additionally, our model performed well on other metrics, validating the algorithm’s robustness, stability, and generalization capabilities. Consistent progress over these multiple polyp datasets confirms the model’s robust generalisation under endoscopic imaging conditions.

Table 3. The results on the CVC-ClinicDB for different models in a few noteworthy cases.

Methods	Year	CVC-ClinicDB
		$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}^{\mathit{max}}\uparrow$	$\mathit{MAE}\downarrow$
U-Net [12]	2015	0.823	0.755	0.811	0.889	0.954	0.019
U-Net++ [13]	2018	0.794	0.729	0.785	0.873	0.931	0.022
SFA [56]	2019	0.700	0.607	0.647	0.793	0.885	0.042
PraNet [31]	2020	0.899	0.849	0.896	0.936	0.979	0.009
MSNet [57]	2021	0.921	0.879	0.914	0.941	0.972	0.008
Enhanced U-Net [58]	2021	0.902	0.846	0.891	0.936	0.965	0.011
CFA-Net [59]	2023	0.933	0.883	0.924	0.950	-	0.007
MEGANet(Res2Net-50) [60]	2024	0.938	0.894	0.940	0.950	0.986	0.006
CSCA U-Net [61]	2024	0.915	0.864	0.912	0.942	0.975	0.010
Proposed		0.939	0.892	0.942	0.952	0.989	0.006

The experimental comparison of different network models on CVC-ColonDB, Endoscene, and ETIS-LaribPolypDB datasets. For ease of visualization, the two highest-scoring responses are distinguished by red and blue, respectively.

Since the datasets in Table 4, Table 5 and Table 6 were not included in the training set, all methods’ performance generally declined on these datasets, particularly on Endoscene and ETIS-LaribPolypDB. The model’s performance on the ETIS-LaribPolypDB dataset is demonstrated by an mDice score of 0.756 and an mIoU score of 0.673, with $S_{\alpha }$ reaching 0.844 (the highest across all datasets) and $F_{\beta }^w$ at 0.709, both leading comparative models. Although on Endoscene, our metrics were slightly lower than MEGANet and marginally higher than CFA-Net, our proposed method still improved in boundary clarity and detail preservation. The visualization reveals that our model provides clearer boundary judgments, enabled by the Improved Reverse Attention module that optimizes edge entropy and reduces segmentation blur, allowing the segmentation of complex and subtle details in ambiguous cases and performing well even in challenging scenarios with uncertain regions—this is further validated by the optimal MAE (0.028) on CVC-ColonDB and 0.006 on CVC-ClinicDB, achieved through the synergy of MA module’s detail error reduction and reverse attention’s boundary optimization.

Table 4. The results on the CVC-ColonDB for different models in a few noteworthy cases.

Methods	Year	CVC-ColonDB
		$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}^{\mathit{max}}\uparrow$	$\mathit{MAE}\downarrow$
U-Net [12]	2015	0.512	0.444	0.498	0.712	0.776	0.061
U-Net++ [13]	2018	0.483	0.410	0.467	0.691	0.760	0.064
SFA [56]	2019	0.456	0.337	0.366	0.629	0.765	0.094
PraNet [31]	2020	0.709	0.640	0.696	0.819	0.869	0.045
MSNet [57]	2021	0.755	0.678	0.737	0.836	0.883	0.041
Enhanced U-Net [58]	2021	0.756	0.681	0.730	0.831	0.872	0.045
CFA-Net [59]	2023	0.743	0.665	0.728	0.835	-	0.039
MEGANet(Res2Net-50) [60]	2024	0.793	0.714	0.779	0.854	0.895	0.040
CSCA U-Net [61]	2024	0.788	0.703	0.760	0.857	0.906	0.036
Proposed		0.820	0.738	0.805	0.869	0.927	0.028

Table 5. The results on the Endoscene for different models in a few noteworthy cases.

Methods	Year	Endoscene
		$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}^{\mathit{max}}\uparrow$	$\mathit{MAE}\downarrow$
U-Net [12]	2015	0.710	0.627	0.684	0.843	0.875	0.022
U-Net++ [13]	2018	0.707	0.624	0.687	0.839	0.898	0.018
SFA [56]	2019	0.467	0.329	0.341	0.640	0.817	0.065
PraNet [31]	2020	0.871	0.797	0.843	0.925	0.972	0.010
MSNet [57]	2021	0.869	0.807	0.849	0.925	0.943	0.010
Enhanced U-Net [58]	2021	0.837	0.765	0.805	0.904	0.933	0.015
CFA-Net [59]	2023	0.893	0.827	0.875	0.938	-	0.008
MEGANet(Res2Net-50) [60]	2024	0.899	0.834	0.882	0.935	0.969	0.007
CSCA U-Net [61]	2024	0.868	0.789	0.835	0.920	0.969	0.012
Proposed		0.895	0.829	0.878	0.932	0.972	0.007

Table 6. The results on the ETIS-LaribPolypDB for different models in a few noteworthy cases.

Methods	Year	ETIS-LaribPolypDB
		$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{w}}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}^{\mathit{max}}\uparrow$	$\mathit{MAE}\downarrow$
U-Net [12]	2015	0.398	0.335	0.366	0.684	0.740	0.036
U-Net++ [13]	2018	0.401	0.344	0.390	0.683	0.776	0.035
SFA [56]	2019	0.297	0.217	0.231	0.557	0.633	0.109
PraNet [31]	2020	0.628	0.567	0.600	0.794	0.841	0.031
MSNet [57]	2021	0.719	0.664	0.678	0.840	0.830	0.020
Enhanced U-Net [58]	2021	0.687	0.609	0.636	0.793	0.841	0.068
CFA-Net [59]	2023	0.732	0.655	0.693	0.845	-	0.014
MEGANet(Res2Net-50) [60]	2024	0.739	0.665	0.702	0.836	0.858	0.037
CSCA U-Net [61]	2024	0.688	0.608	0.644	0.814	0.882	0.026
Proposed		0.756	0.673	0.709	0.844	0.894	0.020

On the CVC-ColonDB dataset, our model outperformed all alternative methods across the majority of evaluation metrics, including mDice 0.820, mIoU 0.738, $F_{\beta }^w$ 0.805, and $S_{\alpha }$ 0.869, demonstrating excellent segmentation performance. Despite the noteworthy segmentation capabilities demonstrated by MEGANet and CSCA U-Net, and the incorporation of cross-level feature fusion in CFA-Net, the efficacy of these methods in segmenting polyps with blurred edges was found to be unsatisfactory due to insufficient low-level detail extraction. n contrast, our model achieves precise polyp segmentation and superior results by integrating the MA module at the encoder’s lower stages—its multi-scale attention and dilated convolutions capture fine-grained features, avoid small target/weak boundary omission, and enhance effective feature weighting to suppress background interference. It is paired with hierarchical feature supplementation from the integrated decoder and edge-focused optimization from the Improved Reverse Attention module. These two components collectively boost overlap rate, positioning accuracy, and structural consistency across datasets, as evidenced by the standout metrics on Kvasir-SEG: mDice 0.923, mIoU 0.870, and $E_{\varphi }^{max}$ 0.967.

The visual analysis results of the comparative experiments are shown in Figure 7, Figure 8 and Figure 9. The proposed model is compared with five SOTA polyp segmentation methods (PraNet, MSNet, CFA-Net, MEGANet, CSCA U-Net) across five datasets, showing superior performance in segmenting challenging targets. On Kvasir-SEG as shown in Figure 7, CSCA U-Net and CFA-Net lack fine texture restoration for small polyps and show slight thin edge discontinuities, while our model captures low-level micro-textures with continuous edges via the MA module’s multi-scale attention and dilated convolutions. For irregular, low-contrast polyps as shown in Figure 8, MEGANet and MSNet exhibit detail loss and blurred edges at contour turns, whereas our PMA and Improved Reverse Attention modules synergistically enhance boundary discrimination, achieving near-ground truth segmentation. In Figure 9, PraNet and CFA-Net present blurred edges and background missegmentation; our Improved Reverse Attention module (with a pre-added SA module for precise localization) effectively distinguishes polyp-background pixels, ensuring sharp, continuous edges with high ground truth alignment.

Figure 7. Qualitative analysis on the Kvasir-SEG dataset for different models in several noteworthy cases.

Figure 8. Qualitative analysis on the CVC-ClinicDB dataset for different models in several noteworthy cases.

Figure 9. Qualitative analysis on three Benchmark Datasets for different models in several noteworthy cases. The first and second rows belong to the CVC-ClinicDB dataset, the third row to the CVC-300 dataset, and the last two rows to the ETIS-LaribPolypDB dataset.

As shown in Figure 8, for the fourth input image, it can be observed that the segmentation results of CSCA U-Net, MSNet, and PraNet are incomplete, whereas our method achieves segmentation results closer to the ground truth.

Based on the validation of segmentation results, further analysis is conducted from the perspective of computational efficiency. The model’s lightweight nature and operational performance are quantified through metrics including FLOPs, Speed (FPS), and Params (M) presented in Table 7. The proposed model exhibits balanced performance in terms of parameter quantity, computational cost, and inference speed, which lays a solid foundation for its deployment in real-world hospital environments. The model comes in at 45.2 million trainable parameters—small enough to prevent hospital edge devices from being overwhelmed by memory demands, retaining sufficient capacity to learn fine-grained polyp features from medical images. When saved as a standard .pth file (only storing model weights), the file size is approximately 172 MB, which is easy to transmit and deploy across hospital internal systems without consuming excessive storage resources.

Table 7. Comparison of FLOPs, Speed and Params for different models.

Methods	Year	$\mathit{FLOPs}$	$\mathit{Speed}\left(\mathit{fps}\right)$	$\mathit{Params}\left(\mathit{M}\right)$
U-Net [12]	2015	123.87	123.11	34.52
U-Net++ [13]	2018	262.16	82.51	36.63
SFA [56]	2019	-	-	-
PraNet [31]	2020	13.15	25.05	32.55
MSNet [57]	2021	17.00	31.08	29.74
Enhanced U-Net [58]	2021	-	-	-
CFA-Net [59]	2023	55.36	23.50	25.24
MEGANet(Res2Net-50) [60]	2024	-	-	44.19
CSCA U-Net [61]	2024	-	72.79 (Kvasir-SEG)	-
Proposed (PVTv2-B2)		9.8 (256 × 256)	170	45.2

In terms of inference computational cost and speed (tested on NVIDIA RTX 4080, with 256 × 256 input size), the model achieves an average of 9.8 GFLOPs per test image, which effectively sidesteps the bogged-down inference times that typically plague systems with excessive computational demands. The proposed model achieves an average inference time of 5.88–5.90 ms per test image (derived from 170 FPS), which is far faster than the clinical real-time requirement of at least 30 FPS. Ensuring low latency in real-time endoscopic examinations minimizes imaging-segmentation display delays for doctors, a key guarantee for the continuity of clinical operations.

4.5. Ablation Study

In this section, we present a detailed account of the ablation experiments that were conducted on the model’s key modules. The purpose of these experiments was to demonstrate the roles of these modules in the context of the entire network. Table 8 summarizes the specific test results, where the Improved Reverse Attention module is denoted as SARA in tables and figures. Figure 10 displays the final segmentation results after removing or replacing modules. In the single-module ablation experiments, we directly removed the Improved Reverse Attention Module, replaced the MA module with the original attention module from PVTv2, and substituted the PAM module with a combination of 3 × 3 convolutional block, BN, and ReLU functions, while keeping other structures unchanged. We selected two metrics to evaluate our method: mDice and mIoU. The ablation study was conducted on two datasets used for training. Table 8 shows that in single-module ablation experiments, the PAM module contributed more to the model than other modules. This is likely due to its convolutions with varying receptive fields capturing multi-scale features and the improved attention mechanism enhancing feature selection and extraction. Notably, on the Kvasir-SEG dataset, mDice improved by 0.011. Similarly, the Improved Reverse Attention Module significantly aided the model in edge segmentation, as evidenced by Figure 10, where the model with this module exhibited clearer boundaries on the CVC-ClinicDB dataset compared to the model without it, which also underperformed in mDice and mIoU. The MA module contributed to the model but showed smaller improvements compared to the other two proposed modules. As seen in the first and third rows of Figure 10, it had certain limitations in polyp segmentation refinement, failing to effectively capture relationships among features across larger regions, resulting in segmentation that did not fully encompass the polyp area.

Table 8. The results of the ablation study on Kvasir-SEG and CVC-ClinicDB datasets.

Methods			Kvasir-SEG		CVC-ClinicDB
MA	PAM	SARA	$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$	$\mathit{mDice}\uparrow$	$\mathit{mIoU}\uparrow$
	✓	✓	0.913	0.855	0.928	0.880
✓		✓	0.904	0.845	0.922	0.876
✓	✓		0.910	0.852	0.925	0.877
✓	✓	✓	0.915	0.858	0.930	0.883

Figure 10. Core module ablation impact on the proposed Polyp segmentation accuracy. Results of the ablation study, where the last three images in the horizontal row respectively show the outcomes after removing the corresponding modules. The first row displays results from the Kvasir-SEG dataset, the second row from the CVC-ClinicDB dataset, the third and fourth rows from the CVC-ColonDB dataset, and the final row from the ETIS-LaribPolypDB dataset. Green, red, and white regions represent the ground truth, final mask, and the false negative. It can be observed that removing any module affects segmentation accuracy, leading to missed or erroneous detection results. The proposed model can accurately locate and segment polyps regardless of size.

Overall, the PAM module significantly enhances the model’s ability to extract and aggregate polyp features from the decoder output, laying a foundation for subsequent work. Additionally, the Improved Reverse Attention module also contributes substantially to the overall network performance.

5. Conclusions

5.1. Discussion

In this paper, the proposed model dedicated to the task of polyp image segmentation. Currently, some classical polyp segmentation networks still struggle to address the challenges in this field, such as indistinct polyp boundaries, dispersed polyp regions, and image blurring or reflections. To tackle these issues, we incorporate more parallel multi-branch convolutional blocks and an enhanced reverse attention mechanism.

We utilize the traditional classic U-net encoder-decoder architecture, which is crucial for medical image diagnosis and treatment planning, as it captures image features during encoding and reconstructs segmentation results during decoding. The model consists of three core components:

(1)

The encoder employs PVTv2, leveraging spatial and multi-scale feature attention mechanisms for visual tasks. A Multiscale Attention module is introduced at the bottom stage of the encoder to optimize multi-scale feature extraction by fusing local and global information, improving polyp tissue recognition accuracy.

(2)

The parallel multi-level aggregation decoder is specifically designed for pyramid features, combining attention mechanisms with multi-convolutional blocks in parallel through residual convolutions to efficiently integrate and re-extract multi-scale feature maps from the encoder output.

(3)

The Scale Aggregation Reverse Attention module first processes decoder outputs via Scale Aggregation and then reinforces feature extraction using reverse attention, focusing on optimizing boundary recognition accuracy for polyp segmentation.

Evidently, the model demonstrates a high level of proficiency when applied to five polyp datasets, and we validated the effectiveness of the proposed three modules through ablation experiments. To evaluate our model’s performance, we compared it with nine SOTA segmentation networks. Table 2, Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8 and Figure 7, Figure 8, Figure 9 and Figure 10 clearly demonstrate that our model achieves the best segmentation results on five small polyp target datasets, scoring high in both mDice and mIoU, indicating superior feature capture capability. Particularly in cross-dataset evaluations, our model outperforms the second-best MEGANet by 0.2% in mDice on the Kvasir-SEG dataset and surpasses UHA-Net and CSCA U-Net on the CVC-ClinicDB dataset. It also maintains strong performance in mIoU, trailing only slightly behind MEGANet. On the Endoscene dataset, compared to CFA-Net, the proposed model outperforms it in mIoU (0.829 vs. 0.827, +0.2%), while maintaining competitive mDice (0.895 vs. 0.893, +0.2%) and $F_{\beta }^w$ (0.878 vs. 0.875, +0.3%), and its MAE (0.007) is 0.1% lower than CFA-Net’s (0.008), highlighting its superior ability to capture polyp structural integrity and reduce prediction errors. On the CVC-ColonDB and ETIS-LaribPolypDB dataset, our proposed model achieves the best performance across key evaluation metrics—securing the highest mDice, mIoU, $F_{\beta }^w$, and $E_{\varphi }^{max}$, while maintaining competitive scores in $E_{\varphi }^{max}$ and $MAE$, demonstrating its superior ability to accurately segment polyps even on this challenging dataset. These results validate the algorithm’s robustness, stability, and generalization capability. The model’s strong generalisation under diverse endoscopic imaging conditions has been confirmed by consistent progress across multiple independent datasets.

In summary, our proposed model has yielded satisfactory segmentation outcomes for small polyps, but it is imperative to acknowledge that the method still has shortcomings when dealing with complex and challenging samples. From the third row of Figure 8 and the fourth row of Figure 9, it can be seen that there is still significant room for improvement in segmentation accuracy between our method and SOTA segmentation methods. However, in terms of overall comparison, the segmentation results of the proposed model are the closest alignment with the ground truth.

5.2. Conclusions

In this paper, we propose a model that differs from other polyp segmentation methods. Utilizing PVTv2 as the backbone architecture, we integrate a Multiscale Attention module in initial encoder phases to enhance the receptive field for lower-layer feature discernment, and introduce an Improved Reverse Attention module to address the segmentation of polyp edges. We conducted ablation and comparative experiments on five small polyp target datasets, all achieving promising results, demonstrating the performance of our model in polyp segmentation tasks.