AHLFNet: Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network

Abstract

Dense image prediction tasks require both strong semantic category information and precise boundary delineation in order to be effectively applied to downstream applications. However, existing networks typically fuse deep coarse features with adjacent fine features directly through upsampling. Such a straightforward upsampling strategy not only blurs boundaries due to the loss of high-frequency information, but also amplifies intra-class conflicts caused by high-frequency interference within the same object. To address these issues, this paper proposes an Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network(AHLFNet), which consists of an Adaptive Low-Frequency Multi-Kernel Smoothing Unit(ALFU), a Gate-Controlled Selector(GCS), and an Adaptive High-Frequency Edge Enhancement Unit(AHFU). The ALFU suppresses high-frequency components within objects, mitigating interference during upsampling and thereby reducing intra-class conflicts. The GCS adaptively chooses suitable convolutional kernels based on the size of similar regions to ensure accurate upsampled features. The AHFU preserves high-frequency details from low-level features, enabling more refined boundary delineation. Extensive experiments demonstrate that the proposed network achieves state-of-the-art performance across various downstream tasks.

1. Introduction

Dense image prediction plays a crucial role in computer vision and has been widely applied to tasks such as segmentation [1,2,3], classification [4,5], and object detection [6]. These tasks further serve as the foundation for real-world scene understanding in applications including autonomous driving, smart agriculture, and intelligent agents. In such applications, it is essential to assign each pixel to a known category in order to leverage semantic information for classification, while precise boundary information is required for accurate localization. To achieve feature representations that simultaneously capture richer semantic information and finer boundary details, popular models [7,8] commonly adopt downsampling-based feature fusion techniques [9]. Feature fusion refers to the process of combining shallow, high-resolution detail features with deep, low-resolution coarse features. Since feature maps at different levels have mismatched spatial resolutions, deep coarse features are typically upsampled to the same size as shallow detail features using methods such as nearest-neighbor or bilinear interpolation, after which they are either added or concatenated.

However, nearest-neighbor and bilinear interpolation introduce two critical issues that severely affect downstream tasks: intra-class conflict and edge drift. Intra-class conflict refers to inconsistencies in feature representation within the same object during interpolation or feature fusion, caused by variations in illumination, shape, or structural differences. For example, the skin region of a human face exhibits smooth textures and gradual color changes, while the eyes contain strong edges and high-contrast details. Similarly, car wheels and windows often exhibit pronounced intra-class conflict. Such conflict within the same category result in disorganized feature representations after interpolation, ultimately weakening the model’s discriminative ability. As shown in Figure 1, edge drift refers to the phenomenon where object boundary features become excessively blurred or shifted due to the smoothing effects of simple interpolation methods such as nearest-neighbor or bilinear interpolation, resulting in boundary information that is inconsistent with the actual object contour. This misalignment significantly degrades the performance of downstream tasks that rely on precise boundaries, such as segmentation and detection. FPN [9] has pointed out that traditional feature fusion, due to its sampling mechanism, fails to effectively handle intra-class conflict. Similarly, Sapa [10] and FADE [11] demonstrate that simple interpolation algorithms tend to oversmooth features, resulting in edge drift. Furthermore, high-frequency noise within objects leads to nonsmooth fluctuations that disrupt intra-class consistency [12]. In addition, SSFA [13] shows that the absence of high-frequency information in boundary regions causes semantic contour displacement errors. The design philosophy of this paper is inspired by a form of functional symmetry: just as a complete visual perception system must simultaneously handle the two complementary aspects of semantic consistency and boundary precision, our AHLFNet also adopts a symmetrical dual-pathway architecture. we propose the Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network (AHLFNet), a novel method for enhancing feature fusion. This network comprises three key components: an Adaptive Low-Frequency Multi-Kernel Smoothing Unit (ALFU), a Gate-Controlled Selector (GCS), and an Adaptive High-Frequency Edge Enhancement Unit (AHFU). The ALFU selectively applies different convolution kernels depending on the size of similar regions to suppress high-frequency components within objects, thereby reducing intra-class conflict. During upsampling, this module maintains smoothness while overcoming the limitations of fixed kernel sizes, which are unable to balance both global smoothness and local detail preservation. Specifically, large-scale feature inconsistencies can be alleviated by emphasizing large kernels to smooth and upsample coarse high-level features, thereby reducing pixel-level differences and mitigating feature inconsistency. In contrast, fine-grained boundaries require small kernels to preserve details and avoid blurring. The GCS adaptively determines kernel size based on regional similarity. It conducts local similarity analysis using three dilated convolutions with different dilation rates. If the similarity among the three kernels is consistent, the large kernel is selected. If the similarities of the small and medium kernels are consistent, the medium kernel is chosen. If the small kernel differs significantly from the other two, or in other ambiguous cases, the small kernel is selected. This mechanism ensures that, during upsampling, sampled pixels exhibit higher similarity with their neighboring pixels, thereby reducing intra-class conflict. As a result, the enhanced similarity and discriminability of intra-class features make this mechanism particularly effective for dense prediction tasks. Although the ALFU and GCS effectively reconstruct upsampled high-level features with low intra-class conflict and clear boundaries, fine-grained edge details in low-level features remain difficult to fully recover in high-level representations during downsampling. According to the Nyquist–Shannon sampling theorem, any frequency component above the Nyquist frequency is irreversibly lost during downsampling. To address this, we introduce the AHFU, which compensates for boundary details beyond the Nyquist frequency that cannot be restored through upsampling. By capturing low-level high-frequency details lost during downsampling, this unit significantly improves boundary localization accuracy. Together, these three modules reconstruct fused features that are both semantically consistent and boundary-sharp. The main contributions of this study can be summarized as follows:

Figure 1. Edge shifts among three different upsampling methods. Compared with the other two methods, our approach achieves better suppression of edge drift.

• We propose a novel frequency-centric design paradigm. Motivated by the complementary roles of frequency components in images—where low frequencies govern semantic consistency and high frequencies dictate boundary precision—we introduce a principled Adaptive High–Low Frequency Collaborative framework. This framework is specifically designed to address the root causes of intra-class conflict and edge drift in dense prediction tasks, ensuring both strong semantic discriminability and fine boundary precision.
• Under this paradigm, we instantiate the AHLFNet which integrates three key components: (1) an Adaptive Low-frequency Unit (ALFU), which smooths object interiors to suppress high-frequency interference and reduce intra-class conflict; (2) a Gate-Controlled Selector (GCS), which adaptively determines the kernel size for smoothing through similarity analysis to ensure feature consistency; and (3) an Adaptive High-frequency Unit (AHFU), which compensates for boundary details lost during downsampling to improve edge localization accuracy.
• Extensive experiments demonstrate that the proposed method yields significant performance improvements in object detection, instance segmentation, and semantic segmentation. Consistent gains are achieved across benchmark datasets including COCO, ADE20K, and Cityscapes, validating the effectiveness and robustness of our approach.

2. Related Work

2.1. Feature Fusion and Upsampling Methods

Feature fusion aims to effectively combine high-resolution shallow features with low-resolution deep features, thereby preserving both fine-grained local details and abstract semantic representations [3]. In most existing frameworks, deep features are first upsampled to match the spatial resolution of shallow features, after which they are fused through weighted summation or concatenation. Traditional upsampling methods, such as nearest-neighbor and bilinear interpolation, rely on fixed manually designed convolution kernels. As a result, they often suffer from over-smoothing and edge drift when applied to complex semantic scenes. To overcome these limitations, researchers have proposed a variety of improved feature fusion strategies. Methods such as deconvolution [14], pixel shuffle [15], and DUpsampling [16] allow convolution kernel weights to be learned during training. However, once training is complete, the kernels remain fixed, lacking adaptability to spatial variations. To address this issue, several studies have explored dynamic upsampling mechanisms. For instance, LHC [17] jointly learns both kernel shapes and weights, while unpooling [18] leverages the indices from max pooling for spatial reconstruction, reflecting location-dependent dynamics. CARAFE [19] employs a context-aware mechanism to dynamically generate kernel weights, enabling adaptive local feature reassembly. These methods share a common characteristic: they adopt data-dependent, predictable kernels whose parameters are generated by sub-networks, thereby providing greater expressive flexibility for feature upsampling. Similarly, GUM [20] predicts guidance offsets to adjust upsampling coordinates, aligning high- and low-level features. AlignSeg [21] further extends this idea by introducing predicted sampling offsets for multi-resolution feature alignment and contextual modeling. Mask2Former [22] employs a deformable attention module that combines spatial offsets with adaptive attention weights to achieve more efficient multi-scale fusion.

In summary, existing feature fusion approaches during upsampling can be broadly categorized into two paradigms: kernel learnability and sampling coordinate optimization. The former enhances representational capacity by learning convolution kernels but generally lacks spatial adaptability; the latter improves feature alignment via offsets or attention, yet still struggles to reduce intra-class conflict and precise boundary information. A comparative analysis with several major methods has been elaborated. For instance, while CARAFE achieves general content-aware upsampling through predicted kernels, our proposed GCS introduces similarity-driven kernel selection specifically designed for low-frequency smoothing, establishing a dedicated mechanism that fundamentally differs in both purpose and implementation for handling intra-class conflicts. Similarly, whereas adaptive filtering techniques are typically applied in general image processing, our ALFU implements task-specific adaptive multi-kernel smoothing for feature consistency in dense prediction tasks. The integration of three scalable kernels with the GCS mechanism represents a significant architectural advancement. The proposed Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network approaches the problem from a frequency-domain perspective, combining low-frequency dynamic smoothing with high-frequency edge enhancement. This integrated design aims to further alleviate both intra-class inconsistency and boundary blurring issues in feature fusion.

2.2. Intra-Class Conflict and Boundary Issues

In dense prediction tasks such as semantic segmentation and object detection, intra-class conflict and boundary accuracy remain critical bottlenecks to performance. Prior studies [10] have shown that although deep features possess strong semantic representation capabilities, they often exhibit intra-class conflict within the same category due to variations in illumination, texture irregularities, or structural complexity. Meanwhile, commonly used interpolation-based upsampling methods tend to oversmooth feature representations, leading to semantic misalignment along object boundaries [11] and consequently reducing the model’s ability to distinguish fine-grained structures. To mitigate intra-class conflict, one line of research introduces regularization or contrastive learning techniques to bring the feature representations of samples from the same class closer together. For example, clustering regularization and center loss explicitly constrain intra-class samples toward their category centers, thereby enhancing compactness [18,23]. Another line of work leverages channel attention or feature re-weighting mechanisms, enabling the model to emphasize category-discriminative regions while suppressing redundant features [24,25]. However, these methods mainly focus on optimizing global representations and lack effective handling of intra-class conflict in boundary regions. To alleviate edge drift, multi-scale feature fusion and high-resolution branch architectures [26] improve boundary quality to some extent. Nevertheless, these approaches rely on fixed-scale fusion or interpolation operations, which often lead to blurred fine-grained boundaries. Other methods explicitly impose semantic contour constraints through boundary-aware loss functions or edge enhancement modules. However, such approaches are largely limited to task-level optimization and seldom incorporate feature similarity in analyzing edge drift. Different from the above methods, our work introduces a cosine-similarity-based analytical framework.

2.3. Applications of Frequency-Domain Methods in Visual Tasks

Frequency-domain analysis has long been a fundamental tool in the field of image processing [27], and in recent years, it has also demonstrated significant theoretical and practical value in deep learning. For example, frequency-domain methods have been applied to study the optimization characteristics [28] and generalization ability [29] of deep neural networks (DNNs). OSBN [30] and DFPF [31] observed that DNNs tend to learn low-frequency patterns first during training, a phenomenon known as spectral bias or the frequency principle. AdaBlur [32] introduced content-aware low-pass filtering to suppress aliasing effects during downsampling, while FLC [33] further confirmed that frequency aliasing degrades model robustness. ISID [34] showed that low-pass filtering effectively suppresses high-frequency interference caused by image noise. FreD [12] specifically pointed out that high-frequency perturbations significantly improve intra-class conflict. In terms of methodological innovation, researchers have explored diverse applications of frequency-domain techniques. DHP [35] and FcaNet [36] utilized discrete cosine transform coefficients to enhance channel attention mechanisms. AFFT [37] applied the classical convolution theorem to DNNs, demonstrating that adaptive frequency filters can serve as efficient global feature integrators. It has been shown that intra-class conflict arises from disturbances in high-frequency components, whereas edge drift is caused by the loss of high-frequency information. To address these issues, we propose an innovative design that employs an ALFU and a GCS to reduce feature conflicts, while leveraging an AHFU to strengthen useful high-frequency details and sharpen object boundaries.

Existing frequency-based methods typically process frequency components in isolation or sequentially. Such serial or segregated architectures inevitably lead to the loss or distortion of one frequency component while processing the other. In contrast, our framework establishes a synchronous dual-path processing mechanism, where the ALFU+GCS pathway is dedicated to low-frequency purification, and the AHFU pathway focuses on high-frequency preservation. Both paths operate simultaneously and are fused through our enhanced fusion strategy.

3. Method

We first present the overall improvements in feature fusion. Specifically, features from different stages are initially combined through basic fusion, and the resulting fused features are then fed into three distinct components for further enhanced fusion, as illustrated in Figure 2. We then provide a detailed explanation of the implementation principles of the AHLFNet. To facilitate readability, we have listed the symbols used in the formulas and their corresponding meanings in Table 1.

Figure 2. Implementation process of the Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network. The upper part above the dashed line illustrates the basic fusion, while the lower part below the dashed line depicts the enhanced fusion. ALFU, AHFU, and GCS denote the Adaptive Low-Frequency Multi-Kernel Smoothing Unit, the Adaptive High-Frequency Edge Enhancement Unit, and the Gate-Controlled Selector, respectively. PS represents PixelShuffle, and Proj indicates projection. “*” in a circle indicates element-wise multiplication, while ⊕ indicates element-wise addition.

Table 1. Symbols and Their Meanings.

Symbols	Meanings	Symbols	Meanings
UpS	Upsampling	Proj	1 × 1 Convolution
AL	ALFU	AH	AHFU Convolution
Conv	Convolution	exp	Exponential Function
Sim	Cosine Similarity	cen	center
nei	neighbor	$\tau$	Hyperparameter
$\sigma$	activation function

3.1. Overall Improvement Through Feature Fusion

Traditional feature fusion methods typically concatenate or add features from the l-th layer with upsampled features from the (l + 1)-th layer, where upsampling is generally implemented using bilinear interpolation or nearest-neighbor methods. To optimize this process, we first construct a preliminary fusion approach through channel adjustment, which can be mathematically formulated as follows:

$${\mathrm{F}}^{\mathrm{l}}={\mathrm{C}}^{\mathrm{l}}+\mathrm{UpS}\left({\mathrm{F}}^{\mathrm{l}+1}\right)$$

(1)

$${\mathrm{Z}}^{\mathrm{l}}=\mathrm{UpS}\left(\mathrm{Proj}\left({\mathrm{F}}^{\mathrm{l}+1}\right)\right)+\mathrm{Proj}\left({\mathrm{C}}^{\mathrm{l}}\right)$$

(2)

where ${\mathrm{C}}^{\mathrm{l}}\in {\mathrm{R}}^{\mathrm{C}\times 2\mathrm{H}\times 2\mathrm{W}}$ and ${\mathrm{F}}^{\mathrm{l}+1}\in {\mathrm{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$ denote the l layer feature extracted from the backbone and the feature to be fused at the l layer, respectively. UpS represents the upsampling operation, ${\mathrm{Z}}^{\mathrm{l}}\in {\mathrm{R}}^{\mathrm{C}\times 2\mathrm{H}\times 2\mathrm{W}}$ denotes the compressed fused feature. Proj refers to channel adjustment, which can be implemented using a 1 × 1 convolution. Although the preliminary design in Equations (1) and (2) addresses the channel dimension matching problem, it suffers from two inherent limitations: (1) It can still exacerbate intra-class conflict, as the fusion process does not suppress high-frequency interference within the feature maps. (2) It remains ineffective in mitigating edge drift, since valuable low-level details are not explicitly enhanced during fusion. These limitations motivate our proposed enhanced fusion mechanism presented below. Enhanced fusion is achieved by collaboratively integrating the outputs of a dual-path processing strategy: the l-th layer feature undergoes high-frequency edge enhancement to yield ${\overline{\mathrm{C}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}}\in {\mathrm{R}}^{\mathrm{C}\times 2\mathrm{H}\times 2\mathrm{W}}$, while simultaneously, the $(l+1)$-th layer feature is processed through adaptive low-frequency smoothing and upsampling to produce ${\overline{\mathrm{F}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}+1}\in {\mathrm{R}}^{\mathrm{C}\times 2\mathrm{H}\times 2\mathrm{W}}$. This process is mathematically formulated as follows:

$${\mathrm{F}}_{i,j}^l={\overline{\mathrm{F}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}+1}+{\overline{\mathrm{C}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}}$$

(3)

$${\overline{\mathrm{F}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}+1}=\mathrm{UpS}\left(\mathrm{AL}\left({\mathrm{F}}_{i,j}^{l+1}\right)\right)$$

(4)

$${\overline{\mathrm{C}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}}=\mathrm{AH}\left({\mathrm{C}}_{i,j}^l\right)+{\mathrm{C}}_{i,j}^l$$

(5)

Here, AL (the Adaptive Low-frequency Multi-kernel Smoothing Unit) performs adaptive low-frequency smoothing on high-level features to mitigate intra-class conflict by enhancing consistency in intra-class feature similarity. Meanwhile, AH (the Adaptive High-frequency Edge Enhancement Unit) focuses on amplifying high-frequency boundary details in low-level features, thereby effectively suppressing edge drift. The notation (i, j) denotes spatial coordinates in the feature map. Through this high–low frequency division-and-conquer strategy [11], our collaborative design simultaneously addresses the two fundamental challenges of intra-class conflict and edge drift.

3.2. AHLFNet

Adaptive Low-Frequency Multi-Kernel Smoothing Unit(ALFU). The design objective of ALFU is to perform adaptive low-frequency smoothing on high-level features ${\mathrm{F}}^{\mathrm{l}+1}$ during the upsampling process [15], thereby enhancing intra-class feature consistency. As illustrated in Figure 3a, the workflow can be decomposed into the following four steps. (1) Weight Generation: The unit takes the preliminary fused feature ${\mathrm{Z}}^{\mathrm{l}}$ as its input. First, the input is processed through a convolutional layer to generate the spatially variable base weights ${\overline{\mathrm{U}}}^{\mathrm{l}}\in {\mathrm{R}}^{{\mathrm{K}}^2\times 2\mathrm{H}\times 2\mathrm{W}}$. Then, a kernel-wise softmax operation is applied to each kernel scale (3, 5, 7) within ${\overline{\mathrm{U}}}^{\mathrm{l}}$, producing independent, normalized weight tensors ${\overline{\mathrm{W}}}_{\mathrm{i},\mathrm{j},\mathrm{p},\mathrm{q}}^{\mathrm{l},\left(\mathrm{K}\right)}\in {\mathrm{R}}^{{\mathrm{K}}^2\times 2\mathrm{H}\times 2\mathrm{W}}$ for each scale. This operation ensures the non-negativity and normalization (i.e., summation to one) of all kernel coefficients. (2) Weight Reorganization and Assignment: The generated weight tensor is reorganized along the channel dimension and evenly divided into four groups, denoted as ${\overline{\mathrm{W}}}_{\mathrm{i},\mathrm{j},\mathrm{p},\mathrm{q}}^{\mathrm{l},\left(\mathrm{K}\right),\mathrm{g}}\in {\mathrm{R}}^{{\mathrm{K}}^2\times \mathrm{H}\times \mathrm{W}}$, where g ∈ (1, 2, 3, 4). Each group corresponds to a spatially varying low-pass filter kernel for a specific sub-pixel location in the upsampled output. (3) Multi-scale Filtering and Fusion: The high-level feature ${\mathrm{F}}^{\mathrm{l}+1}$ is convolved with the filter kernels of the three scales (K = 3, 5, 7) belonging to the same group g, yielding filtered features ${\mathrm{F}}_{\mathrm{l}+1,\mathrm{i},\mathrm{j}}^{\mathrm{k},\left(\mathrm{g}\right)}\in {\mathrm{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$. The gate-controlled selector then assigns weights $a^k$ to the outputs of each scale, and a weighted summation is performed to obtain the final intermediate feature ${\overline{\mathrm{F}}}_{\mathrm{l}+1,\mathrm{i},\mathrm{j}}^1$ for that group. (4) Sub-pixel Reorganization and Output: The intermediate features of the four groups, ${\overline{\mathrm{F}}}_{\mathrm{l}+1,\mathrm{i},\mathrm{j}}^1$ to ${\overline{\mathrm{F}}}_{\mathrm{l}+1,\mathrm{i},\mathrm{j}}^4$ are spatially rearranged according to sub-pixel rules, ultimately producing the 2× upsampled smoothed feature ${\overline{\mathrm{F}}}^{\mathrm{l}+1}\in {\mathrm{R}}^{\mathrm{C}\times 2\mathrm{H}\times 2\mathrm{W}}$. Through this design, ALFU integrates feature upsampling and adaptive smoothing into a single operation, efficiently enhancing feature consistency. This process can be formally expressed as

$${\overline{\mathrm{U}}}^{\mathrm{l}}=\mathrm{Conv}\left({\mathrm{Z}}^l\right)$$

(6)

$${\overline{\mathrm{W}}}_{i,j,p,q}^{l,\left(k\right)}={\displaystyle \frac{exp\left({\overline{\mathrm{U}}}_{i,j,p,q}^{l,\left(k\right)}\right)}{{\sum }_{p,q\in w\left(k\right)}exp\left({\overline{\mathrm{U}}}_{i,j,p,q}^{l,\left(k\right)}\right)}}$$

(7)

$${\mathrm{F}}_{l+1,i,j}^{k,\left(g\right)}=\sum _{p,q\in w\left(k\right)}{\overline{\mathrm{W}}}_{i,j,p,q}^{l,\left(k\right),g}\ast {\mathrm{F}}_{i,j,p,q}^{l+1}$$

(8)

$${\overline{\mathrm{F}}}_{l+1,i,j}^{\left(g\right)}=\sum _k{\mathrm{a}}_{i,j}^k\ast {\mathrm{F}}_{l+1,i,j}^{k,\left(g\right)}$$

(9)

$${\overline{\mathrm{F}}}^{l+1}=\mathrm{PS}({\overline{\mathrm{F}}}_{l+1}^1,{\overline{\mathrm{F}}}_{l+1}^2,{\overline{\mathrm{F}}}_{l+1}^3,{\overline{\mathrm{F}}}_{l+1}^4)$$

(10)

where $w\left(k\right)$ denotes the neighborhood of scale k ($k\in \{3,5,7\}$). The ALFU can adaptively predict spatially varying filtering kernels based on the feature content, effectively enhancing feature consistency.

Figure 3. (a) Details of the Adaptive Low-Frequency Multi-Kernel Smoothing Unit. (b) Details of the Gate-Controlled Selector. (c) Details of the Adaptive High-Frequency Edge Enhancement Unit. Here, HFEE denotes High-Frequency Edge Enhancement, and E represents the identity matrix.

Gate-Controlled Selector(GCS). Although fixed-kernel low-frequency smoothing units can reduce intra-class conflicts through feature smoothing, they face significant limitations when dealing with large inconsistent regions and fine boundaries. Large kernels can improve consistency across broad regions but tend to degrade fine details, whereas small kernels preserve details but struggle to correct widespread inconsistencies. This necessitates the adaptive selection of appropriate kernels. To address this, we propose the Gate-Controlled Selector, which is based on a similarity-driven principle. As illustrated in Figure 3b, for each pixel, three similarity $S^t$ (t ∈ (small(3), middle(5), large(7))) are computed, representing the average similarity of the pixel with its eight neighboring pixels under three different dilation rates (1, 2, 3). The similarities are then converted into soft decisions $H_t$ in the range [0,1], which are used to construct unnormalized weights ${\overline{\mathrm{a}}}_{\mathrm{t}}$. The process for generating the three weights is as follows: if the average similarity of the small and medium matrices is higher, ${\overline{\mathrm{a}}}_{\mathrm{m}}$ approaches 1 while ${\overline{\mathrm{a}}}_{\mathrm{l}}$ and ${\overline{\mathrm{a}}}_{\mathrm{s}}$ approach 0, resulting in the selection of the medium kernel. Other cases are handled similarly, ensuring that one weight in $\overline{a}$ is close to 1 while the other two are near 0, thereby selecting the appropriate kernel. Finally, the normalized weights ${\mathrm{a}}_{\mathrm{t}}$(t ∈ 3, 5, 7) are constructed. This process can be formally expressed as

$${\mathrm{S}}_{i,j}^t={\mathrm{Sim}}_{\mathrm{k}}(\mathrm{cen},\mathrm{nei})$$

(11)

$${\mathrm{h}}_t=\sigma \left({\displaystyle \frac{{\mathrm{S}}^t-{\mathrm{T}}_t}{\tau }}\right),\mathrm{t}\in (\mathrm{small},\mathrm{middle},\mathrm{large})$$

(12)

$${\overline{\mathrm{a}}}_{\mathrm{l}}={\mathrm{h}}_s{\mathrm{h}}_m{\mathrm{h}}_l,{\overline{\mathrm{a}}}_{\mathrm{m}}={\mathrm{h}}_s{\mathrm{h}}_m(1-{\mathrm{h}}_l)$$

(13)

$${\overline{\mathrm{a}}}_{\mathrm{s}}={\mathrm{h}}_s(1-{\mathrm{h}}_m)+(1-{\mathrm{h}}_s)(1-{\mathrm{h}}_m)$$

(14)

$${\mathrm{a}}_t={\displaystyle \frac{{\overline{\mathrm{a}}}_{\mathrm{t}}}{{\overline{\mathrm{a}}}_{\mathrm{s}}+{\overline{\mathrm{a}}}_{\mathrm{m}}+{\overline{\mathrm{a}}}_{\mathrm{l}}}},\mathrm{t}\in (\mathrm{small},\mathrm{middle},\mathrm{large})$$

(15)

where ‘Sim’ denotes cosine similarity, ‘cen’ denotes the center pixel, ‘nei’ represents the neighboring pixels, $\sigma$ denotes the activation function and $\tau$ is a hyperparameter with a value of 0.3. ${\mathrm{T}}_t$ represents a learnable threshold parameter, initialized to 0.7. During training, ${\mathrm{T}}_t$ is optimized end-to-end alongside the rest of the network through backpropagation, enabling the model to adaptively learn optimal decision boundaries for kernel selection across diverse scenes and object scales.

Adaptive High-Frequency Edge Enhancement Unit(AHFU). Although the ALFU and the GCS can effectively reconstruct upsampled high-level features with low intra-class conflict and clear boundaries, fine-grained edge information lost in low-level features during downsampling is difficult to fully restore in high-level representations. To address this limitation, as shown in Figure 3c, we design the AHFU to recover detailed edge information lost during downsampling. Specifically, this unit takes the initial fused feature $Z^l$ as input and predicts spatially varying high-frequency kernels. To preserve fine edge details, the unit employs only $3\times 3$ convolutional kernels, a softmax layer, and a filter inversion operation. In implementation, low-frequency kernels are first obtained via kernel-level softmax, and then inverted [35] to produce high-pass kernels, where E represents a 3 × 3 identity kernel, in which the center element is set to 1 while all eight surrounding elements are set to 0. Finally, the high-frequency kernels are applied and combined with residual addition to produce the enhanced output ${\overline{\mathrm{C}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}}$. The residual structure ensures the preservation of original feature information while enhancing high-frequency details. This process can be expressed as

$${\overline{\mathrm{U}}}^{\mathrm{l}}=\mathrm{Conv}\left({\mathrm{Z}}^l\right)$$

(16)

$${\overline{\mathrm{W}}}_{i,j,p,q}^l=\mathrm{E}-{\displaystyle \frac{exp\left({\overline{\mathrm{U}}}_{i,j,p,q}^l\right)}{{\sum }_{p,q\in w\left(k\right)}exp\left({\overline{\mathrm{U}}}_{i,j,p,q}^l\right)}}$$

(17)

$${\overline{\mathrm{C}}}_{\mathrm{i},\mathrm{j}}^{\mathrm{l}}=\sum _{p,q\in w\left(k\right)}{\overline{\mathrm{W}}}_{i,j,p,q}^l\ast {\mathrm{C}}_{i,j}^l+{\mathrm{C}}_{i,j}^l$$

(18)

We visualize the four-stage output features obtained using AHLFNet and bilinear interpolation upsampling methods with ResNet50 as the backbone in Figure 4. The figure clearly shows that in the first two stages, AHLFNet produces features with sharper boundaries and more accurate localization. However, the last two stages contain higher-level semantic information that falls beyond the scope of human interpretation.

Figure 4. The first row displays features from the ResNet50 backbone, the second row shows features with added FPN (using nearest-neighbor interpolation for upsampling), and the third row presents features from AHLFNet. Our method demonstrates superior visualization results.

4. Experiment

We first evaluate the generalizability of the proposed Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network (AHLFNet) across three representative dense prediction tasks: semantic segmentation, object detection, and instance segmentation. Subsequently, we conduct ablation studies on the three proposed modules to thoroughly validate their effectiveness. We provide the definitions of the evaluation metrics used here. IOU refers to the ratio of the intersection between the predicted region and the ground truth to their union. Among these, mIOU means the mean intersection over union across all categories, APb refers to the detection accuracy at the bounding box level, and APm calculates the IOU between segmentation masks.

4.1. Semantic Segmentation

Semantic segmentation aims to predict the correct category label for each pixel while ensuring effective aggregation of pixels belonging to the same class. Existing methods typically adopt progressive upsampling and multi-level feature fusion [38,39], highlighting the critical role of feature fusion in segmentation. Motivated by this, we select semantic segmentation as a task to evaluate the effectiveness of the AHLFNet. This task demands low intra-class conflict and precise boundary localization, and the network’s optimized feature alignment and fusion help reduce intra-class conflict and mitigate edge drift, thereby improving overall segmentation performance.

Experimental Settings. We evaluate our method on mainstream segmentation datasets, including Cityscapes, ADE20K [40], and COCO-Stuff [41]. In implementation, the proposed modules are integrated into baseline models such as SegFormer [38], Mask2Former [42], and SegNeXt [43], while keeping their original training configurations. We use the AdamW [44] optimizer with an initial learning rate of $6\times {10}^{-5}$ and poly decay, combined with standard data augmentation strategies including random flipping, scaling, and cropping. The batch size is set to 8 for Cityscapes and 16 for the other datasets. Training iterations are 160 K for Cityscapes and ADE20K, and 80 K for COCO-Stuff. Three AHLFNets are inserted into the multi-scale feature fusion stages of SegFormer and Mask2Former, while two are inserted into SegNeXt, to comprehensively validate the effectiveness of the proposed method.

Experimental Results. Comparison with strong baselines: As shown in Table 2, SegFormer-B1 is used as the baseline segmentation model, upon which various state-of-the-art methods are compared on the ADE20K dataset. The proposed AHLFNet achieves the best improvement, with a gain of 2.9% mIoU.

Table 2. Comparison of various state-of-the-art methods on the ADE20K dataset, using SegFormer-B1 as the base segmentation model.

Methods	Params (M)	mIoU (%)	bIoU (%)
SegFormer-B1 [38] (BL)	13.7	41.7	27.8
Carafe [45] (2019ICCV)	14.2	42.8	29.8
IndexNet [46] (2020TPAMI)	26.3	41.5	28.3
PixelShuffle [15] (2016CVPR)	27.9	41.5	26.6
A2U [47] (2021CVPR)	13.9	41.5	27.3
Dysample-S [48] (2023ICCV)	13.8	43.6	29.9
Sapa-B [10] (2022NeurlPS)	13.8	43.2	31.0
FADE [11] (2022ECCV)	14.0	43.1	31.7
AHLFNet (Ours)	14.2	44.6	32.7

Integration with advanced methods: As reported in Table 3, when combined with Mask2Former using a ResNet-50 backbone on the Cityscapes dataset, the AHLFNet achieves a 1.1% mIoU improvement, outperforming IFA, FaPN, and the original Mask2Former with offset-based fusion. Replacing the backbone with ResNet-101 further demonstrates the consistent effectiveness of the method.

Table 3. Performance comparison on the Cityscapes using Mask2Former as the base segmentation model, with the proposed method integrated, against various segmentation methods.

Methods	Backbone	mIoU (%)
IFA [49] (2022ECCV)	ResNet-50	78.0
AlignSeg [21] (2021TPAMI)	ResNet-50	78.5
FaPN [50] (2021ICCV)	ResNet-50	80.0
Mask2Former [42]	ResNet-50	79.4
Mask2Former + AHLFNet	ResNet-50	80.5 (+1.1)
Mask2Former	ResNet-101	80.1
Mask2Former + AHLFNet	ResNet-101	80.9 (+0.8)

Generalization across multiple mainstream models: Table 4 reports experiments integrating the AHLFNet with a variety of mainstream architectures, covering both convolutional networks and Transformer-based frameworks. UPerNet [39] uses an FPN structure, while SegFormer and SegNeXt rely on feature concatenation for fusion. Despite the clear architectural differences, the AHLFNet consistently enhances performance, demonstrating strong adaptability to diverse model designs.

Table 4. Performance improvement on the Cityscapes when integrating AHLFNet using single-scale inference across 3 segmentation models. The values in parentheses indicate the improvement over the original model.

Methods	Backbone	Params (M)	mIoU (%)
UPerNet [39]	ResNet-50	29.2	79.8 (+1.0)
UPerNet	DAMamba-T	55.3	80.4 (+0.7)
SegFormer [38]	MiT-B1	14.1	80.2 (+1.7)
SegNeXt [43]	MSCAN-T	4.4	80.7 (+0.9)

Experiments on challenging datasets: Using SegNeXt as the base model, we conduct experiments on multiple challenging datasets. As shown in Table 5, the AHLFNet consistently improves performance across all three datasets. Specifically, it increases mIoU by 0.9%, 2.3%, and 2.1% on Cityscapes, ADE20K, and COCO-Stuff, respectively. Further experiments with SegMAN-L [51] achieve optimal results of 84.6%, 53.5%, and 49.1% mIoU across the three datasets.

Table 5. Evaluation of the generalizability and effectiveness of AHLFNet on the Cityscapes, ADE20K, and COCO-Stuff datasets, using SegNeXt-T and SegMAN-L as the base segmentation models.

Methods	Params (M)	Cityscapes/mIoU	ADE20K	COCO-Stuff
SegNeXt-T [43]	4.3	79.8	41.1	38.7
SegNeXt-T + AHLFNet	4.6	80.7 (+0.9)	43.4 (+2.3)	40.8 (+2.1)
SegMAN-L [43]	92.4	84.2	53.2	48.8
SegMAN-L + AHLFNet	92.6	84.6 (+0.4)	53.5 (+0.3)	49.1 (+0.3)

4.2. Object Detection

In object detection, models are required to simultaneously address two key challenges: precise localization and accurate classification. Most mainstream detectors are built upon Feature Pyramid Networks (FPNs), making the effectiveness of feature fusion critical to overall performance. High-quality fusion produces semantically consistent and boundary-preserving feature representations, which significantly enhance both localization accuracy and class discrimination.

Experimental Settings. We conduct object detection experiments on the MS COCO [52] dataset, using the classical Faster R-CNN framework with a ResNet-50 backbone for evaluation. The experiments are implemented based on the MMDetection [53] codebase. A 1× training schedule, corresponding to 12 epochs, is adopted. Only the feature fusion modules within the FPN are replaced with our proposed design, while all other settings remain unchanged.

Experiment Results. As shown in Table 6, the AHLFNet demonstrates superior performance on the COCO dataset, achieving an overall AP improvement of 1.9%, significantly outperforming comparison methods such as Carafe and FADE. Figure 5 presents the visualization of detection results from three different methods on the COCO dataset, where AHLFNet results are closest to the ground truth. These experimental results fully demonstrate the effectiveness of the proposed high–low frequency collaborative feature alignment approach in enhancing object detection performance.

Table 6. Performance comparison on the MS COCO dataset for Faster R-CNN with a ResNet-50 backbone, integrating various upsampling methods.

Methods	Params (M)	${\mathit{AP}}^{\mathit{b}}$ (1×)	${\mathit{AP}}_{50}^{\mathit{b}}$	${\mathit{AP}}_{75}^{\mathit{b}}$
Nearest(BL)	46.8	37.5	58.2	40.8
PixelShuffle [15]	56.2	37.5	58.5	40.4
IndexNet [46]	55.2	37.6	58.4	40.9
Carafe [45]	47.1	38.6	59.9	42.2
A2U [47]	46.9	37.3	58.7	40.0
FADE [11]	47.0	38.5	59.6	41.8
DySample+ [19]	46.9	38.7	60.0	42.2
AHLFNet (Ours)	47.2	39.4	60.9	42.8

Figure 5. Detection results of three different object detection methods on the COCO dataset. The results of AHLFNet are closest to the ground truth.

4.3. Instance Segmentation

Experimental Settings. We conduct instance segmentation experiments on the MS COCO benchmark dataset, using the classical Mask R-CNN [1] framework with a ResNet-50 backbone. The experimental setup is consistent with that of object detection, except that our proposed improvements are applied only to the feature fusion modules within the FPN. The implementation is based on MMDetection. Considering that the FPN structure fuses features at 4×, 8×, 16×, and 32× downsampling levels, we insert three AHFLNets for evaluation.

Experimental Results. As shown in Table 7, the AHLFNet achieves significant improvements on the COCO dataset. Specifically, $A{P}^m$ increases by 1.2% and bounding $A{P}^b$ by 1.7%, outperforming all compared methods. Figure 6 presents visualizations of three instance segmentation methods, where our method produces superior results. These findings fully validate the effectiveness of AHLFNet for instance segmentation tasks.

Table 7. Performance comparison on the MS COCO dataset for Mask R-CNN with a ResNet-50 backbone, integrating various fusion methods. AP^b and AP^m denote box AP and mask AP, respectively, and 1× corresponds to 12 epochs.

Methods	${\mathit{AP}}^{\mathit{b}}$ (1×)	${\mathit{AP}}_{50}^{\mathit{b}}$	${\mathit{AP}}_{75}^{\mathit{b}}$	${\mathit{AP}}^{\mathit{m}}$ (1×)	${\mathit{AP}}_{50}^{\mathit{m}}$	${\mathit{AP}}_{50}^{\mathit{m}}$
Nearest (BL)	38.3	58.7	42.0	34.7	55.8	37.2
Carafe [45]	39.2	60.0	43.0	35.4	56.7	37.6
IndexNet [46]	38.4	59.2	41.7	34.7	55.9	37.1
PixelShufffe [15]	38.5	59.4	41.9	34.8	56.0	37.3
FADE [11]	39.1	60.3	42.4	35.1	56.7	37.2
A2U [47]	38.2	59.2	41.4	34.6	56.0	36.8
DySample+ [19]	39.6	60.4	43.5	35.7	57.3	38.2
AHLFNet (Ours)	40.0	61.1	43.6	35.9	57.9	38.1

Figure 6. Comparison of three different instance segmentation methods. The results of AHLFNet are closest to the ground truth.

4.4. Ablation Studies

Experimental Settings. In this section, we conduct systematic ablation studies to evaluate the contributions of each component in the AHLFNet. We select the high-performance SegNeXt as the baseline model and perform evaluations on the ADE20K dataset.

Module Ablation Results. Using SegNeXt-T as the baseline, Table 8 analyzes the individual and combined effects of the ALFU, the GCS, and the AHFU. When used individually, the ALFU and AHFU improve mIoU by 0.9% and 0.6%, respectively. Since the GCS serves as guidance for the ALFU, it cannot be used independently and thus, no separate ablation is conducted. When two modules are combined, performance surpasses that of any single module, and the use of all three modules achieves the best performance of 43.4% mIoU (+2.3), demonstrating the optimal synergistic effect of the complete framework.

Table 8. Ablation study on ADE20K using SegNeXt-T as the baseline, evaluating the contributions of the ALFU, GCS, and the AHFU.

ALFU	GCS	AHFU	Params (M)	mIoU
×	×	×	4.26	41.1
✓	×	×	4.43	42.0 (+0.9)
×	×	✓	4.28	41.7 (+0.6)
✓	×	✓	4.45	42.6 (+1.5)
×	✓	✓	4.40	42.1 (+1.0)
✓	✓	×	4.55	42.7 (+1.6)
✓	✓	✓	4.57	43.4 (+2.3)

The impact of the number of groups and layers on performance in ALFU. Table 9 investigates the grouping strategy of the ALFU using SegNeXt-T as the baseline. Regardless of whether the kernel is fixed at 3 × 3, 5 × 5, or 7 × 7, a group number of 4 achieves the best performance. Therefore, this study adopts a group size of 4 for all experiments.

Table 9. Ablation study on the number of groups for three different kernel scales in the ALFU, using SegNeXt-T as the baseline.

	1	2	4	8	16
3 × 3	41.2	41.6	42.0	41.7	41.6
5 × 5	41.1	41.4	41.7	41.5	41.3
7 × 7	41.2	41.5	41.7	41.5	41.4

5. Conclusions

In this study, we propose the Adaptive High–Low Frequency Collaborative Auxiliary Feature Alignment Network (AHLFNet) to address intra-class conflict and edge drift in dense image prediction tasks. The network comprises three key components: the AHLFNet, which smooths high-level features; the GCS, which ensures that pixels with high similarity are sampled within their neighborhoods to prevent increased edge drift and intra-class conflict; and the AHFU, which enhances the high-frequency components of low-level features. The synergistic operation of these three components effectively mitigates intra-class conflict and inaccurate boundary localization. Extensive experiments demonstrate that AHLFNet achieves superior performance across multiple dense prediction tasks, including semantic segmentation, object detection, and instance segmentation.

6. Limitations and Future Work

Limitations in Computational Efficiency: We candidly note that since AHLFNet includes adaptive frequency processing and multi-scale selection mechanisms, its computational overhead is somewhat increased compared to some lightweight upsampling methods (such as bilinear interpolation).

Although AHLFNet demonstrates excellent performance on standard benchmark datasets, we acknowledge that its performance is somewhat dependent on input image quality. Model performance may degrade significantly when the input is highly degraded due to extreme environmental factors, such as severe low-light conditions or dense haze. This limitation highlights the gap between the current work and a fully robust visual perception system. As the reviewer astutely noted, integrating advanced image enhancement and restoration methods (such as FreqSpatNet [54] for low-light conditions or VNDHR [55] for dehazing) as preprocessing modules with AHLFNet to form an end-to-end enhancement–understanding pipeline is key to overcoming this limitation and improving the model’s applicability in real-world scenarios. This provides a clear direction for our future research.