Edge-Supervised Attention-Aware Fusion Network for RGB-T Semantic Segmentation

Abstract

To address the limitations in the efficiency of modality feature fusion in existing RGB-T semantic segmentation methods, which restrict segmentation performance, this paper proposes an edge-supervised attention-aware algorithm to enhance segmentation capabilities. Firstly, we design a feature fusion module incorporating channel and spatial attention mechanisms to achieve effective complementation and enhancement of RGB-T features. Secondly, we introduce an edge-aware refinement module that processes low-level modality features using global and local attention mechanisms, obtaining fine-grained feature information through element-wise multiplication. Building on this, we design a parallel structure of dilated convolutions to extract multi-scale detail information. Additionally, an EdgeHead is introduced after the edge-aware refinement module, with edge supervision applied to further enhance edge detail capture. Finally, the optimized fused features are fed into a decoder to complete the RGB-T semantic segmentation task. Experimental results demonstrate that our algorithm achieves mean Intersection over Union (mIoU) scores of 58.52% and 85.38% on the MFNet and PST900 datasets, respectively, significantly improving the accuracy of RGB-T semantic segmentation.

1. Introduction

Semantic segmentation, as a core research topic in computer vision, aims to achieve the classification and understanding of each pixel within an image. This technology plays an irreplaceable role in various cutting-edge fields such as security monitoring [1], autonomous driving [2], and robot navigation [3]. Traditional semantic segmentation methods primarily rely on processing RGB images, distinguishing different objects by analyzing visual elements such as color features, geometric structures, and texture characteristics. However, these unimodal approaches often face challenges in real-world applications due to factors like changes in ambient lighting, occlusions, and complex scenes, making it difficult to ensure stable and reliable segmentation results. With the rapid advancement of multimodal sensing technologies, RGB-T semantic segmentation has emerged as a research hotspot [4,5,6]. By leveraging the complementary advantages of fusing RGB visible light images with thermal images, this approach significantly enhances segmentation system performance. Thermal imaging possesses the unique advantage of being independent of ambient lighting, allowing for accurate capture of an object’s thermal distribution even in low-light conditions or adverse weather such as fog, effectively compensating for the limitations of RGB images in these scenarios. Despite the promising potential of RGB-T semantic segmentation, numerous technical challenges remain. The primary challenge lies in the differences in modality characteristics; RGB images reflect the object’s appearance features, while thermal images record temperature information. This fundamental disparity leads to significant inconsistencies between the two modalities at the pixel level. Additionally, designing an efficient multimodal feature fusion mechanism is another critical challenge. Inappropriate information interaction strategies may reduce segmentation accuracy, necessitating the development of advanced algorithmic frameworks to achieve the seamless integration of the two modalities, thereby maximizing their respective strengths.

The current research on RGB-T image semantic segmentation primarily focuses on two core directions: multimodal feature enhancement and multimodal feature fusion. In the realm of feature enhancement, Deng et al. [7] introduced a methodology leveraging channel and spatial attention mechanisms. The channel attention mechanism targets the foreground object regions, while the spatial attention mechanism concentrates on global small targets, thereby extracting more discriminative feature information. Li et al. [8] concentrated on multiscale feature processing, proposing an edge-sharpening module to enhance low-level texture features, an auxiliary activation module to optimize mid-level features, and a collaborative localization module to process high-level semantic features, achieving a multilevel optimization of feature representation. Wu et al. [9] employed a gating mechanism to filter and integrate intra-modal effective information, using symmetric gating to acquire complementary information within the modality, and effectively reduced modality differences through weighted aggregation. Zhang et al. [10] innovatively proposed generating cross-modal pseudo-images using multilayer unimodal features, reducing modality differences by enforcing similarity constraints between real and pseudo-image features. However, in practical scenarios where the proportion of foreground region pixels is small, this global processing approach may struggle to adequately reduce modality differences in the foreground regions.

In the domain of multimodal feature fusion, Sun et al. [11] utilized a fusion strategy based on ResNet feature extraction and simple element-wise summation. However, this straightforward fusion method may not fully exploit the complementary information between modalities. To address this issue, Frigo et al. [12] proposed a confidence-based adaptive weighted fusion method that dynamically adjusts the contribution weights based on the reliability of different modal features, and introduced a matching layer mechanism to improve the consistency of feature fusion. Zhou et al. [13] captured shared semantic and spatial information by computing inter-channel and inter-pixel relationship matrices, effectively reducing the interference of modality-specific information and enhancing segmentation performance. Although these studies have made significant progress in the field of RGB-T semantic segmentation, challenges remain in feature enhancement and fusion in complex scenes. Particularly, in scenarios where the foreground regions occupy a small proportion, further exploration is needed to more precisely extract and fuse effective feature information.

Considering the differences in feature properties resulting from the imaging mechanisms of RGB and thermal images, this paper proposes an Edge-Aware Attention Fusion Network (EAAFNet). The feature fusion module of EAAFNet employs global max pooling and average pooling strategies to integrate features with different statistical properties, mitigating the adverse effects of noise. Additionally, the dual attention mechanism in both channel and spatial domains effectively leverages complementary feature information between RGB and thermal images, allowing unique feature information from one modality to supplement the other. In retaining more low-level features, an edge-aware refinement module is designed, enabling the network to learn the edge correspondence between different modalities, thereby aiding the model in better aligning and fusing the features of the two modalities.

The primary contributions of this study are as follows:

• We propose an edge-supervised attention-aware fusion network that effectively leverages complementary information across modalities, mitigates the adverse effects of interfering noise, and achieves superior performance through the efficient integration of multimodal feature information.
• We introduce a feature fusion module that employs Channel Attention and Spatial Attention mechanisms, enabling it to capture richer complementary feature information across both channel and spatial dimensions, thereby enhancing the effectiveness of the fusion process.
• We design an edge-aware refinement module that learns the edge correspondence between different modalities, enhancing the accuracy and clarity of segmentation boundaries.

2. Proposed Method

This paper employs the classical encoder–decoder network framework to perform segmentation tasks. The proposed RGB-T semantic segmentation algorithm is depicted in Figure 1. First, the RGB and thermal modalities are separately input into two parallel ResNet-152 encoder branches, each consisting of five feature extraction stages. To accommodate the encoder’s input requirements, the single-channel thermal modality is replicated and expanded to three channels before input. In the first feature extraction stage, an Edge-Aware Refinement Module (ERM), is designed to preserve fine structural details of the target objects, enhancing the accuracy and clarity of segmentation boundaries. Subsequently, the Feature Fusion Module (FFM) utilizes channel and spatial attention mechanisms to complement the features between the RGB and thermal modalities, forming an adaptive fused feature representation. Finally, the decoder processes the fused multi-scale features to produce the desired segmentation result.

2.1. Feature Fusion Module

In multimodal semantic segmentation, different modalities exhibit distinct feature representations, each contributing unique advantages. For instance, the RGB modality provides rich color information, crucial for discerning object details and textures. However, RGB cameras are susceptible to performance degradation in low-light or low-contrast environments and are vulnerable to noise. Conversely, the thermal modality, unaffected by lighting conditions, captures thermal radiation emitted by objects, proving invaluable in scenarios with limited visibility, such as nighttime. Therefore, integrating the strengths of both modalities effectively mitigates their individual limitations, achieving robust performance under complex environmental conditions.

This paper introduces a novel feature fusion module, illustrated in Figure 2, designed to embed salient features from one modality into its complementary counterpart while simultaneously reducing the impact of modality-specific noise. This module leverages both Channel Attention (CA) and Spatial Attention (SA) mechanisms, enabling the capture of richer feature information across both channel and spatial dimensions. This facilitates a deeper understanding of the unique attributes and complementary nature of RGB and thermal images. By independently analyzing the features of each modality and subsequently employing a cross-multiplication approach, the module not only effectively suppresses background noise but also significantly enhances the model’s adaptability to diverse environmental conditions by integrating the strengths of both modalities.

To comprehensively extract and leverage features $F_{rgb}$ and $F_t$ from the two modalities, both are initially subjected to Global Average Pooling (GAP) and Global Max Pooling (GMP) along the spatial axis. These pooling operations effectively capture global statistical information of the background while preserving crucial local details, thereby mitigating the influence of interfering noise. GAP focuses on extracting holistic background features, whereas GMP emphasizes the most salient foreground features. The outputs of these two pooling strategies are summed to integrate features with distinct statistical properties. This integration enhances the model’s sensitivity to key signals and effectively reduces noise interference.

To precisely modulate inter-modal features, the combined RGB modality features are fed into a Multilayer Perceptron (MLP). This MLP includes a linear transformation followed by a non-linear ReLU activation function, further refining the feature representation. Subsequently, a Sigmoid function is applied to generate the channel attention weight vector, $V_{rgb}^c$, for the RGB modality. Concurrently, the combined thermal (T) modality features are concatenated along the spatial axis. This concatenated output is then processed through a structure comprising two $1\times 1$ convolutional layers and a ReLU activation function for further feature refinement. Finally, a Sigmoid function is applied to generate the spatial attention weight vector, $V_t^s$. The method employs the cross-multiplication of two attention weight vectors to generate complementary feature information $F_{rgb}^s$ and $F_t^c$. This approach enhances the contrast between the foreground and background within each modality, effectively leveraging the complementary information from both modalities. The formula is as follows:

$$V_{rgb}^c=\sigma \left(MLP(GAP\left(F_{rgb}\right)+GMP\left(F_{rgb}\right))\right)$$

(1)

$$V_t^s=\sigma \left({Conv}_{1\times 1}\left(ReLU\left({Conv}_{1\times 1}\left((GAP\left(F_t\right)+GMP\left(F_t\right))\right)\right)\right)\right)$$

(2)

$$F_t^c=V_{rgb}^c⊛F_t$$

(3)

$$F_{rgb}^s=V_t^s⊛F_{rgb}$$

(4)

where ⊛ denotes channel multiplication, and $\sigma$ represents the Sigmoid function.

Upon identifying the complementary characteristics of channel and spatial attention, we add the enhanced features to the original features element-wise, specifically denoted as $F_{rgb}^e$ and $F_t^e$. The relationship can be expressed as follows:

$$F_{rgb}^e=F_{rgb}^s+F_{rgb}$$

(5)

$$F_t^e=F_t^c+F_t$$

(6)

This integration results in a more enriched and detailed feature representation. Subsequently, these enhanced modal features are fed into the feature fusion module. The ultimate goal of this process is to achieve optimal feature fusion, thereby enhancing the segmentation performance. To achieve the final modality fusion, the two modalities are merged using a $1\times 1$ convolution and then separated after applying the SoftMax activation function to obtain an adaptive fused feature representation. This approach integrates the feature maps from the two branches into a single feature map, enhancing the model’s ability to capture detailed information and improving the accuracy and efficiency of semantic segmentation. This phenomenon can be mathematically represented as:

$${\overline{F}}_{rgb},{\overline{F}}_t=F_{split}\left(\mathrm{SoftMax}\left(\mathrm{Cat}\left({Conv}_{1\times 1}\left(F_{rgb}^e\right),{Conv}_{1\times 1}\left(F_t^e\right)\right)\right)\right)$$

(7)

$$F_{fuse}={\overline{F}}_{rgb}\otimes F_{rgb}^e+{\overline{F}}_t\otimes F_t^e$$

(8)

2.2. Edge-Aware Refinement Module

Due to the often ambiguous or unclear boundaries of targets in RGB-T images, the edge-aware refinement module plays a crucial role in enhancing the model’s ability to accurately identify and localize target boundaries, thereby improving the precision of segmentation results. The first layer of stage features retains a significant amount of detailed information, enabling the edge-aware refinement module to learn the correspondence of edges between different modalities. This facilitates better alignment and fusion of features from the two modalities. The fused features are then input into the edge-aware refinement module, as illustrated in Figure 3. To achieve fine-grained feature representation, this module is designed with two branches. The first branch incorporates a global attention mechanism that employs global average pooling and a multi-layer perceptron (MLP) to preserve global channel context information. The second branch utilizes a local attention mechanism that leverages max pooling (MP) and an MLP to capture critical detail information. Through this method, we obtain two feature vectors R and T, which can be expressed as:

$$R=MLP\left(GAP\left(F_{fuse}\right)\right)$$

(9)

$$T=MLP\left(MP\left(F_{fuse}\right)\right)$$

(10)

To generate a weight vector rich in multimodal information, we perform element-wise multiplication of the two processed feature vectors. Subsequently, we apply channel-wise multiplication to further refine these multimodal feature representations:

$$F_{fuse}^{\prime }=\sigma (R\otimes T)⊛F_{fuse}$$

(11)

Subsequently, we design a parallel structure of dilated convolutions to extract multi-scale detail information. Finally, we concatenate and aggregate these multi-scale features to generate the output features. Following the edge-aware refinement module, we attach an EdgeHead and apply edge supervision to obtain more accurate edge information. The aforementioned process can be succinctly summarized as follows:

$$F_{erm}=MHDC\left(F_{fuse}^{\prime }\right)$$

(12)

2.3. Loss Function

Our network incorporates two types of supervision: edge supervision $L_{erm}$ and final semantic segmentation supervision $L_{seg}$. Specifically, for edge supervision, we employ Dice Loss. For the final semantic segmentation results, we utilize both Dice Loss and Soft Cross Entropy, with the coefficient weights set to 0.5. Let $O_{seg}$ denote final semantic segmentation results, $O_{erm}$ denote edge-aware refinement module results, and the overall loss function can be expressed as follows:

$$L_{total}=L_{erm}+L_{seg}$$

(13)

$$L_{erm}=DiceLoss(GT,O_{erm})$$

(14)

$$L_{seg}=DiceLoss(GT,O_{seg})+SoftCrossEntropy(GT,O_{seg})$$

(15)

3. Experimental Results and Analysis

To comprehensively evaluate the performance of the proposed algorithm in this study, we conducted an in-depth analysis using the MFNet and PST900 datasets. We implemented the proposed EAAFNet using the PyTorch 1.9.0, CUDA 11.1, and Python 3.8.17 libraries. Our model was trained on two A100 GPUs, each with 40GB of memory, utilizing the SGD optimizer. The initial learning rate was set to 0.02, with momentum decay and weight decay parameters set to 0.9 and 0.0005, respectively. We trained the model for 300 epochs with a batch size of 5. In order to enhance the generalization ability of the model, we also apply a variety of data augmentation techniques, including random image resizing and random horizontal flipping, to further improve the robustness and performance of the model.

3.1. Datasets and Experimental Metrics

The MFNet dataset, as the first publicly available large-scale RGB-Thermal (RGB-T) image semantic segmentation dataset, holds significant research value and practical implications in the field of urban scene understanding. This dataset comprises a total of 1569 precisely aligned RGB-T image pairs, each with a resolution of 640 × 480 pixels, including 820 daytime scenes and 749 nighttime scenes. Notably, the MFNet dataset provides detailed annotations for nine common target categories in urban street scenes, such as background, traffic cones, pedestrians, and car blockers. These comprehensive annotations offer crucial support for model training and evaluation in the autonomous driving domain. Currently, MFNet has become one of the most representative benchmark datasets in the field of RGB-T image semantic segmentation research. To maintain consistency in our study, we adopted the same data partitioning strategy as MFNet [14].

The PST900 dataset contains 894 precisely aligned RGB-T image pairs, providing pixel-level annotations for five target categories, including background, hand drills, backpacks, fire extinguishers, and survivors. It serves as a vital data resource for image analysis and understanding in challenging environments. In this study, we followed the original partitioning method of PST900 [15], dividing the dataset into training and testing sets.

It is important to note that current multimodal image semantic segmentation datasets commonly suffer from class imbalance issues, which may lead to the Class Average Accuracy metric inadequately reflecting the true performance of models. Therefore, the mIoU has emerged as a more representative evaluation metric and has become a key objective standard for assessing semantic segmentation model performance. In this research, we primarily employ the mIoU metric for model performance evaluation and optimization to ensure the reliability and scientific validity of our findings.

3.2. Comparison of MFNet Dataset

This paper conducts a comparative analysis of the EAAFNet method against several state-of-the-art approaches on the MFNet dataset, including PSTNet [15], RTFNet [11], FuseSeg [16], MFFENet [17], GMNet [18], DHFNet [19], MFTNet [20], DooDLeNet [12], EGFNet [21], DBCNet [22], FDCNet [23], MMDRNet [24], SGFNet [25], CCFFNet [9]. The comparative results of these methods are detailed in

Table 1. Comparison with SOTA methods on the testing sets of MFNet dataset.

Models	mIoU	Car	Person	Bike	Curve	Car Stop	Guardrail	Color Cone	Bump
PSTNet [15]	48.4	76.8	52.6	55.3	29.6	25.1	15.1	39.4	45.0
RTFNet [11]	53.2	87.4	70.3	62.7	45.3	29.8	0.0	29.1	55.7
FuseSeg [16]	54.5	87.9	71.7	64.6	44.8	22.7	6.4	46.9	47.9
MMDRNet [24]	56.0	85.7	70.3	61.5	46.9	32.7	7.7	48.2	53.4
DBCNet [22]	56.2	87.4	73.6	62.9	47.1	33.8	6.8	50.9	45.4
FDCNet [23]	56.3	87.5	72.4	61.7	43.8	27.2	7.3	52.0	56.6
MFFENet [17]	57.1	88.2	74.1	62.9	46.2	37.1	7.6	52.4	47.4
GMNet [18]	57.3	86.5	73.1	61.7	44.0	42.3	14.5	48.7	47.4
DHFNet [19]	57.2	87.6	71.7	61.1	39.5	42.4	9.5	49.3	56.0
MFTNet [20]	57.3	87.9	66.8	64.4	47.1	36.1	8.4	55.5	62.2
DooDLeNet [12]	57.3	86.7	72.2	62.5	46.7	28.0	5.1	50.7	65.8
EGFNet [21]	57.5	89.8	71.6	63.9	46.7	31.3	6.7	52.0	57.4
SGFNet [25]	57.6	88.4	77.6	63.4	45.8	31.0	6.0	57.1	55.0
CCFFNet [9]	57.6	89.6	74.2	63.1	50.5	31.9	4.8	49.7	56.3
Ours	58.5	87.8	71.2	63.6	47.0	29.7	10.8	53.4	64.6

. The findings demonstrate that the proposed FFM, in conjunction with ERM, significantly enhances the fusion efficacy of RGB-T modalities, with EAAFNet achieving a leading mIoU of 58.5%. Furthermore, the paper contrasts the performance of these methods under both daytime and nighttime conditions, with quantitative results indicating the superiority of the proposed method across various scenarios. To further elucidate the advantages of our approach, Figure 4 presents visual comparisons under different lighting conditions. According to the visualization results, our method can effectively capture the boundary segmentation of small targets even in the presence of occlusion. For instance, in the first column of Figure 4, where a person is interacting with a vehicle and is partially obscured by other objects, our method still successfully segments the category of ’person,’ thereby validating the effectiveness of our approach. However, in the long-range images presented in the third and fourth columns, there may be instances of minor missegmentation. Therefore, it is necessary to enhance the reliability of segmentation predictions in distant scenes and reduce uncertainty.

3.3. Comparison of PST900 Datasets

In this study, we conducted a comprehensive comparison of the EAAFNet method against several state-of-the-art techniques on the PST900 dataset, including RTFNet [11], CCNet [26], PSTNet [15], ABMDRNet [10], FDCNet [23], ACNet [27], DHFNet [19], DBCNet [22], MMDRNet [24], SGFNet [25], CCFFNet [9], GCNet [28], GCGLNet [29] and GMNet [18]. The comparative results of these methods are meticulously detailed in

Table 2. Comparison with SOTA methods on the testing sets of PST900 dataset.

Model	mIoU	Background	Fire-Extinguisher	Backpack	Hand-Drill	Survivor
UNet [30]	52.80	98.00	43.00	52.90	38.30	31.60
RTFNet [11]	57.60	98.90	52.00	75.30	25.40	36.40
CCNet [26]	61.42	99.05	51.84	66.42	32.27	57.50
PSTNet [15]	68.40	98.90	70.10	69.20	53.60	50.00
ABMDRNet [10]	71.30	99.00	66.20	67.90	61.50	62.00
ACNet [27]	71.81	99.25	59.95	83.19	51.46	65.19
MMDRNet [24]	74.50	98.90	52.40	71.10	40.60	62.30
FDCNet [23]	77.11	99.15	71.52	72.17	70.36	72.36
DBCNet [22]	81.78	99.40	72.95	82.67	76.68	77.19
CCFFNet [9]	82.10	99.40	79.90	75.80	82.80	72.70
DHFNet [19]	82.19	99.44	78.15	87.34	71.18	74.81
GCNet [28]	82.58	99.35	77.68	79.37	82.92	73.58
SGFNet [25]	82.80	99.40	75.60	85.40	76.70	76.70
GCGLNet [29]	83.24	99.39	77.57	81.01	81.90	76.31
GMNet [18]	84.12	99.44	73.79	83.82	85.17	78.36
Ours	85.38	99.59	80.55	85.90	83.74	77.14

. The findings reveal that EAAFNet achieved the highest mIoU of 85.38%, which robustly demonstrates the superior performance and practical efficacy of our proposed method.

3.4. Comparison of NYUDv2 Datasets

To evaluate the generalization ability of EAAFNet, we use the NYUD V2 dataset for training and testing. We use RGB and depth images as input, and the comparison results are shown in

Table 3. Comparison with other methods on the testing sets of NYUDv2 dataset.

Models	mIoU
RTFNet [11]	49.1
ECGFNet [31]	51.5
CAINet [32]	52.6
PDCNet [33]	53.5
Ours	53.8

. Benefiting from EAAFNet’s proposed FFM and REM modules, we obtained 53.8% mIoU on the NYUDv2 test set, confirming its competitiveness and showing generalization to RGB-D data.

3.5. Computational Complexity Analysis

To evaluate the computational performance and efficiency of the proposed model, we further conducted experiments. As shown in

Table 4. Results of computational complexity analysis.

Models	FLOPs/G	Params/M	mIoU
FuseSeg [16]	142.22	100.11	54.5
CCFFNet [9]	527.48	109.05	57.6
GEBNet [34]	216.38	126.18	56.2
Ours	203.86	147.35	58.5

, the results show that the model proposed in this paper demonstrates good efficiency in terms of FLOPs, significantly outperforming CCFFNet [9] and slightly surpassing GEBNet [34]. Our model has a parameter size of 147.35 M, suggesting that alongside its high performance, it requires certain storage and memory resources.

3.6. Ablation Experiment

This study conducts ablation experiments to evaluate the performance of key modules in the proposed EAAFNet. Under consistent experimental conditions, we obtained the results as shown in

Table 5. Results of ablation experiment on the MFNet dataset.

Baseline	Type	ERM	FFM	mIoU
✔	RGB	×	×	53.43
✔	RGB-T	✔	×	57.82
✔	RGB-T	×	✔	55.37
✔	RGB-T	✔	✔	58.52

. When only the RGB modality was used for semantic segmentation, the model achieved an mIoU of 53.43%. Upon incorporating the FFM module, the mIoU of the algorithm increased to 57.82%. When only the ERM module was retained, the mIoU of the algorithm rose to 55.37%. The simultaneous utilization of both the FFM and ERM modules significantly enhanced the discriminability of the fused features, leading to a notable improvement in semantic segmentation performance, with the mIoU increasing by 5.09%.

4. Conclusions

This study proposes an innovative algorithm to address the limitations of existing RGB-T semantic segmentation methods in feature fusion efficiency. The proposed algorithm enhances inter-modal feature complementarity through a dedicated feature fusion module, which effectively captures and amplifies critical information from each modality by leveraging channel attention and spatial attention mechanisms, thereby significantly reducing noise interference. Furthermore, an edge-aware refinement module is incorporated to enhance the accuracy and clarity of segmentation boundaries by learning edge correspondence relationships across different modalities. Extensive experimental evaluations on MFNet and PST900 datasets demonstrate the superior performance of our algorithm, with the mIoU metric significantly outperforming existing state-of-the-art methods. These results substantiate the effectiveness of our feature complementarity enhancement and cross-modal fusion strategy in improving the accuracy of RGB-T semantic segmentation tasks. This research represents a significant advancement in the field of RGB-T semantic segmentation, offering substantial value for performance enhancement in applications such as autonomous driving and robotic navigation. Future work will focus on exploring optimization strategies for diverse scenarios to improve the algorithm’s generalizability and robustness, thereby contributing to the continuous development of this field.