Depth Adaption SegNet for RGB-T Segmentation

Abstract

RGBT segmentation is a challenging task in the area of computer vision. Current advanced networks for RGBT segmentation focus on extracting deeper discriminative features from RGB and thermal images to provide richer semantic information for the fusion features to the decoder. However, excessively mining deeper semantic features only makes the model redundant. Simultaneously, lacking shallow spatial features leads to difficulties in guaranteeing accurate localization of targets. We believe that the features provided by images can be categorized into three types: edge, patch, and semantics. Only by synchronously taking into account the extraction of all three types of features can models achieve accurate classification on the basis of precise localization. Therefore, we propose Depth Adaption SegNet for RGB-T Segmentation (DASNet). According to the characteristics of the three types of features, we extract semantics, patch, and edge features from the deep, middle, and shallow stages respectively. We specifically design the cross-attention semantics module, patch activation module, and edge enhancement module to perform feature extraction. In addition, in order to efficiently fuse features from different categories, we design a deep-emphasis fusion module to fuse the output features of the modules. Compared to advanced methods, qualitative and quantitative experiments show that DASNet exhibits state-of-the-art performance on the CNN-based RGBT segmentation task.

1. Introduction

RGBT segmentation is a challenging task of computer vision [1,2,3]. The objective is to assign a class label to each pixel in the RGB image using both the RGB and corresponding thermal infrared (TIR) images In conditions of adverse nocturnal weather and low-light environments, RGB images exhibit limitations within the realm of applicability. In light of this, the task incorporates TIR images to provide additional discriminative features for segmentation tasks under conditions such as solar glare and low-light environments. Common RGBT models are typically constructed in a bottom-up structure, wherein input features are passed through predefined modules, with the output of each module serving as the input to the next. Throughout this process, progressively deeper feature representations of the input data are extracted, culminating in the application of upsampling operations to generate the final mask.

While this architecture enables the model to learn more abstract and higher-level features from the raw data, it overlooks the distinctions among features at different levels. We take inspiration from [4] that features at different levels exhibit varying degrees of sensitivity to different types of information. For instance, shallow-level features are most compatible with edge information and can extract rich boundary details. Similarly, intermediate-level features are effective at capturing patch information, while deep-level features exhibit higher sensitivity to semantic information. Simultaneously, a module is necessary to merge different categories of features.

Upon this foundation, we propose DASNet for RGBT segmentation. We design three distinct feature extraction modules, denoted as cross-attention semantics module (CSM), patch activation module (PAM), and edge enhancement module (EEM), tailored for processing feature maps at different depths, capturing boundary, patch, and semantic information, respectively. Additionally, we design a deep-emphasis fusion module (DEF) to amalgamate the output features from these various modules. Ultimately, the final mask is generated through a decoder.

The main characteristics and contributions of the proposed method are summarized as follows:

• We propose DASNet to accomplish the RGBT segmentation task. This network employs a novel hierarchical divergence structure, utilizing differentiation modules to handle features at various depths, maximizing the exploitation of feature potentials, thereby providing an enhanced set of discriminative features for segmentation.
• We design three distinct differentiation modules: The CSM focuses on semantic feature extraction, the PAM concentrates on capturing patch features within features, and the EEM is specifically for extracting boundary information from input images. Furthermore, these modules incorporate fusion operations for RGB and TIR modalities to comprehensively capture multi-modal characteristics.
• We design a DEF module to integrate the outputs of the differentiation modules. This fusion module not only accomplishes the fusion of diverse output features but also introduces multi-scale insights into the model, thereby enhancing model diversity.

The remaining sections of this paper are organized as follows: Section 2 reviews related work about our study. Section 3 introduces the structure of our model and details of each module in the model. Section 4 presents quantitative and qualitative experimental results. Section 5 concludes our work.

2. Related Work

In this section, we review the literature about RGB semantic segmentation and RGB-T semantic segmentation.

2.1. RGB Semantic Segmentation

Recently, RGB semantic segmentation based on convolutional neural networks has made amazing progress by many researchers. Long et al. proposed the Fully Convolutional Network (FCN) [5]. FCN is the first end-to-end CNN-based semantic segmentation method, which integrates the semantic features from the deep, coarse network layers with the surface-level details gleaned from the shallow layers. Badrinarayanan et al. proposed SegNet [6]. The encoder of SegNet employs the computed pooling indices from the corresponding encoder’s max-pooling steps to perform nonlinear upsampling. Ronneberger et al. proposed U-Net [7], which comprises a contracting pathway that captures contextual information and a symmetric expansive pathway that facilitates precise localization.

Inspired by the above works, RGB semantic segmentation has made great progress. Furthermore, researchers endeavor to employ more efficient means for feature extraction. Chen et al. introduced DeepLab v1 [8], where they first proposed dilated convolutions. The authors also employed a conditional random field as a post-processing step. Chen et al. introduced Atrous Spatial Pyramid Pooling in DeepLab v2 [9], employing varying dilation rates in different branches to obtain multi-scale image representations. Chen et al. proposed DeepLab v3 [10]. To introduce diverse dilation rates, the authors employed a multigrid approach within residual blocks. They also incorporated image-level features into the Atrous Spatial Pyramid Pooling module.

In addition, researchers have made significant strides in improving the architecture of neural networks. Zhao et al. proposed PSPNet [11]. The model modifies the underlying ResNet [12] architecture by introducing dilated convolutions, which incorporates auxiliary losses in the middle layers of ResNet to optimize holistic learning. Lin et al. proposed RefineNet [13], which utilizes multi-resolution inputs by merging extracted features. The network effectively captures background information from a larger image region by chained residual pooling.

2.2. RGB-T Semantic Segmentation

With the popularization of sensors and imaging technology, many researchers have introduced TIR images to complete the challenging segmentation task in RGB semantic segmentation. Ha et al. proposed MFNet [14], which was the first network designed for RGBT segmentation. The authors established a novel RGB-T dataset for that task. Sun et al. proposed RTFNet [15], which employs a novel encoder–decoder architecture. Sun et al. proposed FuseSeg [16], which utilizes a Monte Carlo dropout technique to construct a Bayesian uncertainty algorithm, enabling the analysis of uncertainty in semantic segmentation results.

In addition, the cross-attention mechanism has been widely used in RGB-T semantic segmentation. Zhang et al. proposed ABMDRNet [17]. The model initially employs a method based on image-to-image bidirectional transformation to bridge the modality gap in multimodal data. It adaptively selects distinct multimodal features for RGB-T semantic segmentation. Zhou et al. proposed GMNet [18]. The authors, taking into account the characteristics of low-level and high-level features, designed two distinct fusion modules to enhance pertinent details. Deng et al. proposed FEANet [19]. The model explores and enhances multi-level features from both channel and spatial perspectives, directing increased attention towards high-resolution features obtained from the fusion images.

3. Approach

In this section, we present the proposed DASNet in detail. Firstly, we introduce the network overview of DASNet. Secondly, we elaborate the proposed CSM, PAM, EEM, and DEF, respectively. Thirdly, we present the loss function.

3.1. Overview of the Proposed Method

As depicted in Figure 1, the proposed DASNet is based on the feature fusion paradigm, including a feature extractor, three specific modules, and a decoder. For the feature extractor, we adopt two parallel ResNet-152 backbones, named RGB branch and TIR branch, to extract multi-modality features from RGB and TIR images. We denote the five convolution blocks in the RGB branch and TIR branch as $R^i$ and $T^i$$(i=1,2,3,4,5)$ respectively. The corresponding output RGB and TIR features are denoted as $\{{\widehat{F}}_r^i,{\widehat{F}}_t^i\}\in {\mathbb{R}}^{h_i\times w_i\times c_i}$. The input size is denoted as $H\times W$, so $h_i$ is $H/\left(2^i\right)$, $w_i$ is $W/\left(2^i\right)$, and $c_i\in \{64,256,512,1024,2048\}$. Here, the RGB and TIR branches share parameters, which not only keeps the extracted features in the same feature space but also reduces the number of parameters. Furthermore, we employ two identical convolutional layers (i.e., shared parameters) to project the same level of RGB and TIR features, $\{{\widehat{F}}_r^i,{\widehat{F}}_t^i\}$ to $\{F_r^i,F_t^i\}$, respectively, with fewer channels (i.e., $c_i\in \{64,128,256,256,512\}$).

Figure 1. DASNet’s core design principle is that features at different depths carry qualitatively different information. Multiple sets of RGB and TIR features from different stages are passed to the corresponding modules (EEM, PAM, and CSM). Then, the output features of each module are passed to the decoder. The corresponding transmission relationship in the figure uses the same color, shape, and dotted line type. Before passing the output features of each module to the decoder, we additionally set corresponding heads for supervision. Heads are convolutional layers. Among them, the number of channels for the features of the CSM head outputs is the total number of the dataset categories, while for the PAM and EEM, it is 2.

We divide the extracted basic cross-modal features into three levels, that is, $\{F_r^{4\sim 5},F_t^{4\sim 5}\}$ as the high level, $\{F_r^{2\sim 3},F_t^{2\sim 3}\}$ as the middle level, and $\{F_r^1,F_t^1\}$ as the low level. We arrange the CSM, PAM, and EEM for these three levels of features to achieve semantics extraction, patch activation, and edge enhancement respectively.

Concretely, the CSM enables the network to enhance the semantic information of all objects under semantic supervision. The PAM enables the network to highlight the position regions of all objects of different categories. The EEM enables the network to extract edge information of objects with the help of edge supervision. These three specific modules are the core components of our DASNet. With the informative features generated by the above modules from the five-level features, we use a deep-emphasis fusion module (DEF) in the decoder for the fusion of different informative features to achieve the accurate segmentation result $S_{sem}$; the decoder is composed of three DEF modules, an element-wise addition operation, a convolutional layer, and an upsampling operation.

3.2. Cross-Attention Semantics Module

As we all know, high-level features contain rich semantic information. Cross-modality RGB and TIR features are complementary, which is more conducive to semantics extraction. Inspired by object segmentation works exploring the cross-attention modeling of target objects in two different modalities, consecutive video frames and features of successive levels, we propose a cross-attention semantics module to model the pixel-level correlation of multi-modality high-level semantic features to highlight the targeted semantics. We illustrate the CSM in Figure 2. The input features of the CSM are $\{F_r^{4\sim 5}$ and $F_t^{4\sim 5}\}$. The whole process of CSM can be divided into correlation fore fusion, cross-attention modeling, and multi-modality fusion.

Figure 2. The PAM and CSM are complementary rather than interchangeable: the PAM adapts its attention type to the depth of the features it receives (spatial attention for shallow stages, channel attention for deeper stages), while the CSM must resolve cross-modal misalignment before feature interaction between modalities. The (right) part is the PAM, while the other is the CSM. In the CSM, the weight matrix obtained through the correlation fusion is spatial-wise multiplied by the features before the multi-modality fusion, in order to highlight the interested position. The convolution layers in the figure are omitted.

3.2.1. Correlation Fore Fusion

Due to significant disparities between two modalities, direct fusion yields suboptimal results, which is confirmed by the corresponding ablation study listed in Table 1. Therefore, we designed the correlation fore fusion (where ‘Fore’ indicates ‘Before’) to narrow the inter-modality gap in the feature space. We reshape $\{F_r^i,F_t^i\}\in {\mathbb{R}}^{h_i\times w_i\times c_i}$ into $\{A_r^i,A_t^i\}\in {\mathbb{R}}^{h_i{w}_i\times c_i}$. Then, we use matrix multiplication to get the correlation matrix $M_{rt}^i$ between two different modalities, which can be formulated as follows:

$$M_{rt}^i=A_r^i\times {\left(A_t^i\right)}^{\mathrm{T}},$$

(1)

where × is the matrix multiplication, ${(\cdot )}^{\mathrm{T}}$ is the matrix transpose operation and $M_{rt}^i\in {\mathbb{R}}^{h_i{w}_i\times h_i{w}_i}$. We reshape $M_{rt}^i$ into ${\overline{F}}_{rt}^i\in {\mathbb{R}}^{h_i\times w_i\times h_i{w}_i}$. Then, we use convolution to adjust the channel of ${\overline{F}}_{rt}^i$ and get the $F_{rt}^i\in {\mathbb{R}}^{h_i\times w_i\times 1}$ as the spatial attention of the multi-modality.

Table 1. Ablation experiment about the module details. The top result is highlighted in red color.

Aspect	Method	IoU
EEM	w/o dilated conv	56.4
	w/o multi-modality fusion	57.7
	w/o channel attn	56.6
	w/o spatial attn	57.6
PAM	swap attn	57.8
	w/o co-attn	56.8
	w/o correlation fusion	57.2
CSM	w/o multi-modality fusion	57.5
DEF	w/o res-structure	57.4
	Ours	58.1

3.2.2. Cross-Attention Modeling

Owing to environmental factors such as illumination, certain categories prove challenging to discern within RGB features. Element-wise multiplication serves to accentuate regions of interest from the TIR features within the RGB feature space. Therefore, we first perform an element-wise multiplication operation on $F_r^i$ and $F_t^i$, generating $F_{mul}^i\in {\mathbb{R}}^{h_i\times w_i\times c_i}$. Then, we model the pixel-level correlation of $\{F_{mul}^i,F_r^i\}$ and $\{F_{mul}^i,F_t^i\}$, respectively, to collaboratively identify objects in cross-modal features, which can be formulated as follows:

$$\begin{array}{c}\hfill F_{r-corr}^i=Corr(F_{mul}^i,F_r^i),\\ \hfill F_{t-corr}^i=Corr(F_{mul}^i,F_t^i),\end{array}$$

(2)

where $Corr(\cdot )$ is the co-attention operation, and $\{F_{r-corr}^i,F_{t-corr}^i\}\in {\mathbb{R}}^{h_i\times w_i\times c_i}$ are the multi-modality correlation features. The co-attention operation computes the correlation matrix of two input features through matrix multiplication. Then, it transfers the valuable semantics of the correlation matrix to the output feature.

3.2.3. Multi-Modality Fusion

$F_{rt}^i$ is the spatial attention matrix of multi-modality features. $F_{r-corr}^i$ and $F_{t-corr}^i$ have a strong representation of the targeted semantics. We design the multi-modality fusion to fuse them effectively. We adopt a fusion scheme that mixes multiple multiplications and additions to combine multi-modality features, which can be formulated as follows:

$$\begin{array}{c}\hfill F_{CSM}^i=((F_{r-corr}^i\otimes F_{rt}^i)\oplus (F_{t-corr}^i\otimes F_{rt}^i))\copyright \\ \hfill ((F_{r-corr}^i\otimes F_{rt}^i)\ast (F_{t-corr}^i\otimes F_{rt}^i)),\end{array}$$

(3)

where ⊗ is the spatial-wise multiplication, ⊕/∗ are the element-wise summation/multiplication, © is the concatenation, and $F_{CSM}^i\in {\mathbb{R}}^{h_i\times w_i\times c_i}$ is the output feature of the CSM. We omit the convolutional layer.

Through the above multi-modality fusion operation, we can obtain informative features of targeted semantics. Furthermore, as shown in Figure 3, we attach a semantic head (SemHead) after the CSM to achieve more accurate targeted semantics with the semantic supervision.

Figure 3. EEM exploits the asymmetry between modalities rather than simply combining them. When the features of the two modalities are element-wise subtracted, the subtraction relationship is exchanged to obtain two output features. Specifically, of the two outputs from the element-wise subtraction, one is to use RGB features to subtract TIR features, while the other uses TIR to subtract RGB features. The convolution layers in the figure are ignored.

3.3. Patch Activation Module

Middle-level multi-modality features (i.e., $\{F_r^{2\sim 3},F_t^{2\sim 3}\}$) facilitate the localization of all targets within the image, offering sufficient patch information. To achieve this, we devised the PAM to activate patch regions of objects within middle-level features at different scales. Concretely, our PAM is based on the attention mechanism, which enables us to activate specific regions and establish stronger connections between features.

We illustrate the PAM in Figure 2, whose inputs are $F_r^i$ and $F_t^i$. We first use element-wise summation and multiplication to get ${\{F_{sum}^i,F_{mul}^i\}}_{i=2,3}\in {\mathbb{R}}^{h_i\times w_i\times c_i}$ to take advantage of the two types of complementary feature combinations.

Then, distinct attention mechanisms are employed for features at various stages. For the 2nd stage, given its proximity to the pixel level, we employ bidirectional spatial attention. To enhance the sharpness of edges within the patch, we further integrate the obtained fused features with edge features from the 1st stage.

To sum up, we briefly formulate the above process as follows:

$$F_{PAM}^2=(SA\left(F_{mul}^2\right)\otimes F_{sum}^2)\oplus (SA\left(F_{sum}^2\right)\otimes F_{mul}^2)\oplus {\overline{F}}_{EEM}^1$$

(4)

where ⊗ is the spatial-wise multiplication, ⊕ is the element-wise summation, $SA(\cdot )$ is the spatial attention operation, ${\overline{F}}_{EEM}^1$ is $F_{EEM}^1$ after the convolution and downsampling layers, and $F_{PAM}^2\in {\mathbb{R}}^{h_2\times w_2\times c_2}$ is the output feature of the PAM from the 2nd stage.

For the 3rd stage, in order to extract deeper discriminative features, we employ bidirectional channel attention. We incorporate $F_{PAM}^2$ with $F_{PAM}^3$ output from the 3rd stage for the multi-scale fusion. To sum up, we briefly formulate the above process as follows:

$$F_{PAM}^3=(CA\left(F_{mul}^3\right)\odot F_{sum}^3)\oplus (CA\left(F_{sum}^3\right)\odot F_{mul}^3)\oplus {\overline{F}}_{PAM}^2$$

(5)

where ⊙ is the channel-wise multiplication, $CA(\cdot )$ is the channel attention operation, ${\overline{F}}_{PAM}^2$ is $F_{PAM}^2$ after the convolution and downsampling layers, and $F_{PAM}^3\in {\mathbb{R}}^{h_3\times w_3\times c_3}$ is the output feature of the PAM from the 3rd stage.

3.4. Edge Enhancement Module

Low-level multi-modality features, being the closest to the pixel level, encapsulate rich spatial edge features. Incorporating edge features into the model offers two advantages. First, it introduces a third discriminative feature into the entire model, enhancing its hierarchical structure and robustness. Second, it provides additional edge features to the PAM, improving the results of the patch analysis.

We illustrate the EEM in Figure 3, whose inputs are $\{F_r^1,F_t^1\}$. We employ element-wise subtraction to obtain modality-specific regions of interest. For instance, subtracting $F_t^1$ from $F_r^1$, and subsequently applying the ReLU activation function, yields $F_{rt}^1$, representing regions of interest for RGB but disinterest for TIR. Similarly, we obtain $\{F_{rt}^1,F_{t-r}^1\}\in {\mathbb{R}}^{h_1\times w_1\times c_1}$. To enable a single modality to capture regions of interest from the other modality, we perform element-wise multiplication between $F_r^1$ and $F_{t-r}^1$, as well as between $F_t^1$ and $F_{rt}^1$, resulting in $\{{\overline{F}}_r^1,{\overline{F}}_t^1\}\in {\mathbb{R}}^{h_1\times w_1\times c_1}$. Similar to the PAM, we use element-wise summation and multiplication between ${\overline{F}}_r^1$ and ${\overline{F}}_t^1$ to get $\{F_{mul}^1,F_{sum}^1\}\in {\mathbb{R}}^{h_1\times w_1\times c_1}$. Then, we add $F_{mul}^1$ and $F_{sum}^1$, transferring to a parallel structure with dilated convolutions to extract multi-scale detail information. Finally, we aggregate multi-scale features by concatenation, generating the output feature of the EEM, $F_{EEM}^1\in {\mathbb{R}}^{h_1\times w_1\times c_1}$. We briefly summarize the above process as follows:

$$F_{EEM}^1=MHDC(F_{mul}^1\oplus F_{sum}^1)$$

(6)

where $MHDC(\cdot )$ represents the multi-head dilated convolutions.

3.5. Deep-Emphasis Fusion Module

To efficiently fuse the output features from different modules, as illustrated in Figure 4, we designed a deep-emphasis fusion module. We first adopted a hybrid scheme including summation, multiplication, and concatenation to combine multi-scale features. Then, a residual structure was designed to amplify the weight of deep features. Finally, upsampling was employed to adjust the size of the output features. We briefly summarize the above process as follows:

$$F_{DEF}=(\left(F_i\copyright {\widehat{F}}_{i+1}\right)\ast F_i)\oplus (\left(F_i\copyright {\widehat{F}}_{i+1}\right)\ast {\widehat{F}}_{i+1})\oplus {\widehat{F}}_{i+1}$$

(7)

where $F_i$ is the output feature of the module at the ith stage, ${\widehat{F}}_{i+1}$ is the output feature of DEF from the $i+1$th stage, $i=2,3,4$. We omit the convolutional and upsampling layers.

Figure 4. DEF’s key design choice is to enhance rather than merely concatenate deep features. $F_i$ is the output feature of the module at the ith stage. ${\widehat{F}}_{i+1}$ is the output feature of DEF from the $i+1$th stage. These two features are fused through DEF to obtain ${\widehat{F}}_i$, which is the output feature of DEF from the ith stage.

4. Experiments and Results

All experiments used an input resolution of 640 × 480 for both training and testing. A single random seed of 42 was fixed for all runs, and all reported results were obtained from one training run per configuration. Implementation was in Python 3.10.16 with PyTorch 2.1.0 and torchvision 0.16.0, on an NVIDIA GeForce RTX 3090 card. The MFNet dataset is a publicly available RGB-T semantic segmentation benchmark for autonomous driving. It comprises 1569 spatially aligned RGB–thermal image pairs collected in urban environments under both daytime and nighttime conditions, officially partitioned into 820 daytime and 749 nighttime pairs. Eight obstacle classes commonly encountered in driving scenarios, including car, person, bike, curve, car stop, guardrail, color cone, and bump, are annotated. The dataset is divided into training, validation, and test sets, with the training set containing 50% of the daytime and nighttime images, and the validation and test sets each containing 25% of the daytime and nighttime images.

Following the empirical studies in [20,21], all quantitative results reported in this paper were obtained from a single training run with a fixed random seed. We do not report means, standard deviations, or significance tests, and therefore we do not claim that the observed performance differences are statistically significant. Known sources of run-to-run variability in our setting include random weight initialization of the non-pretrained layers, batch ordering, and the stochastic augmentation pipeline. Our use of the official MFNet split removes data-partitioning variability, and ImageNet-pretrained backbone weights reduce sensitivity to initialization, but these measures do not substitute for repeated runs. Related experimental results on MFNet benchmark are shown in Table 2. We therefore present our results as a single-run comparison under a fixed, fully reported protocol, and we explicitly refrain from any claim of statistical superiority over the compared methods.

During the training phase, we utilized RGB and TIR images at their original resolutions as inputs to the network, employing random flipping and cropping for data augmentation. The batch size during model training was set to four, with an initial learning rate of $1\times {10}^{-5}$. The parameters of the feature extractor were initialized with a pretrained ResNet-152 model. The parameters of other convolutional layers were initialized with the KAIM method [22]. We set the number of training epochs to 500. During the testing phase, we directly input the RGB-T image pairs at their original resolutions into the trained DASNet without any post-processing, obtaining segmentation results.

Table 2. Quantitative comparison on the test set of the MFNet dataset. ‘-’ means that the authors do not provide the corresponding information. The top two results in each column are highlighted in red and blue.

Method	Type	Backbone	Car	Person	Bike	Curve	Car Stop	Guardrail	Color Cone	Bump	mIoU
SwinT [23]	RGB	-	85.2	57.6	61.0	33.2	28.0	2.4	42.7	33.5	49.0
BiSeNet [24]	RGB	-	84.1	63.2	60.1	36.7	25.3	5.0	42.2	35.9	50.0
DANet [25]	RGB	-	84.5	55.0	62.6	33.4	27.4	0.9	41.9	44.5	49.7
HRNet [26]	RGB	-	84.9	55.4	60.3	33.3	28.3	2.5	40.3	46.9	49.9
SA-Gate [27]	RGB-D	-	73.8	59.2	51.3	38.4	19.3	0.0	24.5	48.8	45.8
D-CNN [28]	RGB-D	-	77.0	53.4	56.5	30.9	29.3	6.4	30.1	32.3	46.1
FuseNet [29]	RGB-D	VGG-16 [30]	75.6	66.3	51.9	37.8	15.0	0.0	21.4	45.0	45.6
ACNet [31]	RGB-D	ResNet-101 [12]	79.4	64.7	52.7	32.9	28.4	0.8	16.9	44.4	46.3
FuseSeg-161 [16]	RGB-T	DenseNet-161 [32]	87.9	71.7	64.6	44.8	22.7	6.4	46.9	47.9	54.5
ABMDRNet [17]	RGB-T	ResNet-50 [12]	84.8	69.6	60.3	45.1	33.1	5.1	47.4	50.0	54.8
EGFNet [33]	RGB-T	ResNet-152 [12]	87.6	69.8	58.8	42.8	33.8	7.0	48.3	47.1	54.8
LASNet [4]	RGB-T	ResNet-152 [12]	84.2	67.1	56.9	41.1	39.6	18.9	48.8	40.1	54.9
MTANet [34]	RGB-T	ResNet-152 [12]	88.1	71.5	60.7	40.9	38.9	13.7	45.9	47.2	56.1
MFFENet [35]	RGB-T	DenseNet-121 [32]	88.2	74.1	62.9	46.2	37.1	7.6	52.4	47.4	57.1
GMNet [18]	RGB-T	ResNet-50 [12]	86.5	73.1	61.7	44.0	42.3	14.5	48.7	47.4	57.3
CGFNet [36]	RGB-T	ResNet [12]	84.3	72.6	59.2	45.6	29.7	7.7	50.3	54.5	55.7
GCNet [37]	RGB-T	DenseNet [32]	86.0	72.0	60.0	42.8	30.7	6.2	49.5	52.6	55.3
GEBNet [38]	RGB-T	ConvNeXt-S [39]	88.7	70.7	63.3	42.5	29.7	6.6	52.5	54.4	56.2
GCGLNet [40]	RGB-T	MobileNet-V2 [41]	83.8	72.4	59.6	40.9	43.0	8.7	43.3	48.6	55.4
DHFNet [42]	RGB-T	ConvNeXt-S [39]	87.6	71.7	61.1	39.5	42.4	9.5	49.3	56.0	57.2
MDRNet+ [43]	RGB-T	ResNet-50 [12]	87.1	69.8	60.9	47.8	34.2	8.2	50.2	55.0	56.8
FDCNet [44]	RGB-T	ResNet-50 [12]	87.5	72.4	61.7	43.8	27.2	7.3	52.0	56.6	56.3
DooDLeNet [45]	RGB-T	DeepLabV3+ [46]	86.7	72.2	62.5	46.7	28.0	5.1	50.7	65.8	57.3
CACFNet [47]	RGB-T	ConvNeXt-B [39]	89.2	69.5	63.3	46.6	32.4	7.9	54.9	58.3	57.8
SpiderMesh-152 [48]	RGB-T	ResNet-152 [12]	88.1	72.8	63.7	48.4	28.2	8.8	48.2	64.2	57.9
Ours	RGB-T	ResNet-152 [12]	89.0	70.7	62.4	45.2	44.1	5.2	53.3	55.1	58.1

Table 2. Quantitative comparison on the test set of the MFNet dataset. ‘-’ means that the authors do not provide the corresponding information. The top two results in each column are highlighted in red and blue.

Method	Type	Backbone	Car	Person	Bike	Curve	Car Stop	Guardrail	Color Cone	Bump	mIoU
SwinT [23]	RGB	-	85.2	57.6	61.0	33.2	28.0	2.4	42.7	33.5	49.0
BiSeNet [24]	RGB	-	84.1	63.2	60.1	36.7	25.3	5.0	42.2	35.9	50.0
DANet [25]	RGB	-	84.5	55.0	62.6	33.4	27.4	0.9	41.9	44.5	49.7
HRNet [26]	RGB	-	84.9	55.4	60.3	33.3	28.3	2.5	40.3	46.9	49.9
SA-Gate [27]	RGB-D	-	73.8	59.2	51.3	38.4	19.3	0.0	24.5	48.8	45.8
D-CNN [28]	RGB-D	-	77.0	53.4	56.5	30.9	29.3	6.4	30.1	32.3	46.1
FuseNet [29]	RGB-D	VGG-16 [30]	75.6	66.3	51.9	37.8	15.0	0.0	21.4	45.0	45.6
ACNet [31]	RGB-D	ResNet-101 [12]	79.4	64.7	52.7	32.9	28.4	0.8	16.9	44.4	46.3
FuseSeg-161 [16]	RGB-T	DenseNet-161 [32]	87.9	71.7	64.6	44.8	22.7	6.4	46.9	47.9	54.5
ABMDRNet [17]	RGB-T	ResNet-50 [12]	84.8	69.6	60.3	45.1	33.1	5.1	47.4	50.0	54.8
EGFNet [33]	RGB-T	ResNet-152 [12]	87.6	69.8	58.8	42.8	33.8	7.0	48.3	47.1	54.8
LASNet [4]	RGB-T	ResNet-152 [12]	84.2	67.1	56.9	41.1	39.6	18.9	48.8	40.1	54.9
MTANet [34]	RGB-T	ResNet-152 [12]	88.1	71.5	60.7	40.9	38.9	13.7	45.9	47.2	56.1
MFFENet [35]	RGB-T	DenseNet-121 [32]	88.2	74.1	62.9	46.2	37.1	7.6	52.4	47.4	57.1
GMNet [18]	RGB-T	ResNet-50 [12]	86.5	73.1	61.7	44.0	42.3	14.5	48.7	47.4	57.3
CGFNet [36]	RGB-T	ResNet [12]	84.3	72.6	59.2	45.6	29.7	7.7	50.3	54.5	55.7
GCNet [37]	RGB-T	DenseNet [32]	86.0	72.0	60.0	42.8	30.7	6.2	49.5	52.6	55.3
GEBNet [38]	RGB-T	ConvNeXt-S [39]	88.7	70.7	63.3	42.5	29.7	6.6	52.5	54.4	56.2
GCGLNet [40]	RGB-T	MobileNet-V2 [41]	83.8	72.4	59.6	40.9	43.0	8.7	43.3	48.6	55.4
DHFNet [42]	RGB-T	ConvNeXt-S [39]	87.6	71.7	61.1	39.5	42.4	9.5	49.3	56.0	57.2
MDRNet+ [43]	RGB-T	ResNet-50 [12]	87.1	69.8	60.9	47.8	34.2	8.2	50.2	55.0	56.8
FDCNet [44]	RGB-T	ResNet-50 [12]	87.5	72.4	61.7	43.8	27.2	7.3	52.0	56.6	56.3
DooDLeNet [45]	RGB-T	DeepLabV3+ [46]	86.7	72.2	62.5	46.7	28.0	5.1	50.7	65.8	57.3
CACFNet [47]	RGB-T	ConvNeXt-B [39]	89.2	69.5	63.3	46.6	32.4	7.9	54.9	58.3	57.8
SpiderMesh-152 [48]	RGB-T	ResNet-152 [12]	88.1	72.8	63.7	48.4	28.2	8.8	48.2	64.2	57.9
Ours	RGB-T	ResNet-152 [12]	89.0	70.7	62.4	45.2	44.1	5.2	53.3	55.1	58.1

4.1. Comparisons with SOTA Methods

We compared DASNet with advanced CNN-based models on RGB/RGB-D/RGB-T tasks. For a fair comparison, building upon the foundations laid by [17,33], we further adapted certain RGB semantic segmentation methods to accommodate RGB-T image pairs. We employed default parameter settings for the RGB-T, RGB-D, and modified RGB models, subsequently retraining them on the same dataset. On the MFNet dataset, we compared our approach with state-of-the-art CNN-based models, including RGB-T (i.e., SpiderMesh-152 [48] and CACFNet [47]), RGB-D (i.e., ACNet [31]) and RGB (i.e., HRNet [26]). The quantitative experimental results of our method and all other approaches on the MFNet dataset are presented in Table 2, encompassing performance metrics for eight classes and overall performance. Overall, our approach achieves the best performance on the crucial mIoU metric, demonstrating robust adaptability to diverse scenes. Specifically, on the mIoU metric, our approach outperforms the second-best method (SpiderMesh-152 [48]) by 0.2%. Across all categories, our approach demonstrates excellent performance in the Car Stop, Car, and Color Cone classes. Particularly, the Car Stop category secures the first position, surpassing the second-best method (GCGLNet [40]) by 1.1%. The Car and Color Cone categories achieve the second position. We observe that DASNet achieves a notably low IoU on the Guardrail class (5.2%). We attribute this to the extreme class imbalance (Guardrail is one of the rarest classes in MFNet) and its thin, elongated structure, which is difficult for standard convolutions and pooling operations to capture. While our EEM enhances edge features, it is not specifically designed for extremely thin objects. This highlights a limitation of our current design and suggests that specialized strategies—such as class-reweighted loss functions, dedicated thin-structure modules, or synthetic data augmentation—are needed to improve performance on such underrepresented classes.

Moreover, the quantitative experimental results of our model and other methods on the daytime and nighttime test sets of the MFNet dataset are presented in Table 3. Our approach demonstrated superiority in both of these scenarios, particularly on the mIoU metric. This suggests that the strategy employed by our segmentation model is effective. The edge–patch–semantic feature extraction pathway progressively enables our method to extract discriminative features, achieving RGB-T segmentation. The above quantitative analysis clearly indicates the effectiveness of our DASNet on the MFNet dataset.

Table 3. Comparative results (%) in daytime and at nighttime. The top two results in each column are highlighted in red and blue.

Method	Daytime	Nighttime
FRRN [49]	40.0	37.3
BiSeNet [24]	44.5	45.0
DFN [50]	42.2	44.6
GCMA [51]	47.2	49.1
CCNet [52]	43.5	38.1
APCNet [53]	42.4	46.4
MFNet [14]	36.1	36.8
FuseNet [29]	41.0	43.9
D-CNN [28]	42.4	43.2
RTFNet [15]	45.8	54.8
FuseSeg-161 [16]	47.8	54.6
EGFNet [33]	47.3	55.0
MFFENet [35]	49.5	57.8
GMNet [18]	49.0	57.7
CGFNet [36]	48.6	54.5
FDCNet [44]	47.8	56.8
Ours	51.5	58.3

We acknowledge that the total FLOPs of DASNet are higher than lightweight backbones due to the employment of the dual ResNet-152 feature extractor. However, benefiting from the shared-weight strategy, the parameter count is only 89.7M, which is even lower than CACFNet (198.6M). As shown in Table 4, we compare DASNet with GCGLNet and CACFNet from the perspectives of params, FLOPs, FPS, and mIoU. While DASNet’s FLOPs are substantially higher than GCGLNet, it attains the highest mIoU of the three methods with less than half the parameters of CACFNet. DASNet achieves an inference speed of 24 FPS. This indicates that our method maintains feasible efficiency for real-world autonomous driving deployment.

Table 4. Complexity comparisons of various models. FPS indicates frames per second.

Method	FLOPs/G	Params/M	FPS	mIoU
GCGLNet [40]	7.4	7.9	53	55.4
CACFNet [47]	101.4	198.6	-	57.8
DASNet	283.7	89.7	24	58.1

In addition, we conducted a partial visualization experiment. Figure 5 presents the segmentation results of our model. The first row corresponds to the original RGB images. The second row depicts the original TIR images. The third row illustrates the segmentation results obtained by our model. The fourth row represents the ground truth. The first to fourth columns correspond to daytime scenes, while the fifth to eighth columns represent nighttime scenes. These segmented images visually showcase the effectiveness of our model on the MFNet dataset, demonstrating robust segmentation performance in both daytime and nighttime scenarios.

Figure 5. Results of qualitative experiments.

Additionally, we visualize the outputs of the EEM and PAM in Figure 6 and Figure 7. Figure 6 corresponds to daytime scenes, while Figure 7 represents nighttime scenes. The first and second rows depict the original RGB and TIR images, respectively. The third row visualizes the output features from the EEM. The fourth row visualizes the output features from the PAM. From the output feature maps of the EEM, we observe that our model accurately extracts and sharpens the edge features of the objects in both daytime and nighttime scenes. In the feature maps from the PAM, we find that the regions of interest in both scenarios are highlighted, with particularly accurate delineation in nighttime scenes. This validates the effectiveness of the EEM and PAM.

Figure 6. Experiment results on daytime scenes with corresponding feature maps. We used the output mask of the EEM head as the feature map of the EEM for visualization. The EEM highlights the edge areas of the target effectively. Considering that visualization pays more attention to spatial features, we chose the output mask of the head of the 2nd-stage PAM as the feature map of the PAM for visualization. Also, for easy observation, we set the visualization threshold to 0.5. The PAM effectively pays more attention to the area around the target position.

Figure 7. Experiment results on nighttime scenes with corresponding feature maps. The settings are the same as for the daytime scenes. Compared with the daytime scenes, the performance of the PAM is better at nighttime. The position of interest is more precise.

4.2. Ablation Experiments

We conducted a comprehensive ablation study to demonstrate the effectiveness of individual modules and key structures within the modules on the MFNet dataset. Specifically, we firstly employed ablation experiments to showcase the separate and combined contributions, as well as the effectiveness, of the four modules (EEM, PAM, CSM, and DEF). Subsequently, we established the effectiveness of individual components within each of the four modules. For all ablation experiments, we trained the model with the same training parameters as described in Section 4. We employed mIoU as the performance evaluation metric.

4.2.1. The Individual and Combined Effectiveness of the Modules

We introduced three modules, the CSM, PAM, and EEM, to extract semantic, patch, and edge features, respectively. Initially, we proposed four different variants to assess the segmentation performance when each of the three modules was used separately. These variants were: (1) baseline, (2) baseline + EEM, (3) baseline + PAM, and (4) baseline + CSM.

We employed element-wise addition and convolution to replace the modules that were not equipped in each variant. The experimental results are presented in Table 5. The “Baseline” achieved an mIoU of only 55.4%, which is 2.7% lower than our complete DASNet, indicating that these three modules indeed contributed to improving segmentation accuracy. With the assistance of the EEM, PAM or CSM, the segmentation performance of Variants 2, 3, and 4 improved compared to that of the “Baseline,” with increases of 0.5%, 1.0%, and 0.8%, respectively.

Table 5. Ablation experiments. The top result is highlighted in red color. ✓ denotes the adoption of the corresponding module.

No.	Baseline	EEM	PAM	CSM	DEF	IoU
1	✓				✓	55.4
2	✓	✓			✓	55.9
3	✓		✓		✓	56.4
4	✓			✓	✓	56.2
5	✓		✓	✓	✓	56.5
6	✓	✓		✓	✓	57.0
7	✓		✓	✓		57.6
8	✓	✓	✓	✓		56.8
Ours	✓	✓	✓	✓	✓	58.1

Next, we introduced three variants to assess the combined contributions of the three modules: (5) Baseline + PAM + CSM, (6) Baseline + EEM + CSM, and (7) Baseline + EEM + PAM. The segmentation metrics for the three variants demonstrated further improvements when two modules were combined compared to using a single module. Subsequently, we simultaneously incorporated all three modules in the model. The segmentation metrics for that variant showed improved performance compared to Variants 5, 6, and 7, with increases of 1.6%, 1.1%, and 0.5%, respectively. These experimental results confirm the collective contributions of the three modules when used in conjunction.

We then introduced the DEF module for multi-scale fusion, integrating features of two different types (patch and semantic). To demonstrate the effectiveness of the fusion module, we designed Variant 8. That variant was based on the simultaneous use of the three modules, replacing the DEF module in the decoder with concatenation and convolution. The segmentation metrics for Variant 8 decreased by 1.3%. This result demonstrates the effectiveness of the DEF module. The above experimental results collectively confirm the effectiveness of all the modules designed in the DASNet.

4.2.2. Effectiveness of Each Component of EEM

To validate the effectiveness of individual components within the EEM, we introduced two variants: (1) w/o dilated convolution and (2) using element-wise addition instead of the multi-modality fusion. The experimental results are illustrated in Table 1. Firstly, we observe a decrease of 1.7% in segmentation metrics when we use regular convolution instead of dilated convolution. This is primarily attributed to the smaller receptive field of regular convolution compared to dilated convolution. The smaller receptive field causes the EEM to overlook some boundaries of target objects.

Secondly, compared to the multi-modality fusion, the segmentation metrics of the model with element-wise addition in the EEM decrease by 0.4%. This is mainly because the multi-modality fusion incorporates element-wise addition, element-wise multiplication, and matrix concatenation. These three methods have corresponding advantages in feature fusion, which are equally important.

4.2.3. Effectiveness of Each Component of PAM

To validate the effectiveness of individual components within the PAM, we used three variants: (1) w/o channel attention, using spatial attention instead, (2) w/o spatial attention, using channel attention instead, and (3) swapping the application positions of the two attention mechanisms. The experimental results are illustrated in Table 1. Firstly, we observe a 1.5% decrease in mIoU when only using the spatial attention mechanism compared to the segmentation metrics of the original network. When only using the channel attention mechanism, the mIoU decreases by 0.5%. Due to the difference in the extent of the decrease, we conclude that the spatial attention is more crucial than the channel attention in the PAM. This is because the features processed by the PAM are closer to the pixel level, which is more suitable for spatial attention.

Secondly, when we swap the positions of the two attention mechanisms in the network, the segmentation metrics decrease by 0.3%. We attribute this to the characteristics of the two attention mechanisms. Spatial attention is suitable for operating on feature maps close to the pixel level, effectively highlighting regions of interest. Channel attention is more suitable for operating on feature maps farther from the pixel level, extracting deeper information. The usage positions of the two attention mechanisms differ.

4.2.4. Effectiveness of Each Component of CSM

To validate the effectiveness of individual components within the CSM, we used three variants: (1) w/o co-attn, (2) w/o correlation fusion, and (3) using element-wise addition instead of the multi-modality fusion. The experimental results are illustrated in Table 1. Firstly, using element-wise addition results in a decrease of 1.3% in segmentation metrics compared to co-attn. We believe that the co-attn mechanism calculates the similarity between different positions of two features. The co-attn mechanism, compared to element-wise addition, allows for a tighter integration of the two features, optimizing segmentation performance.

Secondly, we observe a decrease of 0.9% in segmentation metrics when the correlation fusion module is excluded. This is because the correlation fusion module enables the module to explore deep positional features, passing these features to the subsequent fused features. The absence of these positional features results in a CSM lacking localization capabilities, leading to inaccurate semantic positioning. Then, as mentioned in the ablation experiments for the EEM, the multi-modality fusion incorporates element-wise addition, element-wise multiplication, and matrix concatenation. The effectiveness of the multi-modality fusion surpasses the use of any single method.

4.2.5. Effectiveness of Each Component of DEF

To validate the effectiveness of the residual structure within the DEF module, we introduced a single variant: w/o res-structure. We found that when removing the residual structure in the DEF module, the segmentation metrics of the network decreased by 0.7%. This is because the residual structure is designed to strengthen the weight of deep features in the final output fusion features. The absence of the residual structure results in the dominance of shallow edge and patch features, leading to a decrease in semantic accuracy.

5. Conclusions

In this paper, we proposed a novel Depth Adaption SegNet for RGB-T Segmentation. In order to fully utilize the characteristics of features from different stages, our network employed three different modules to extract corresponding discriminative information from features of varying levels of depth. Our network extracted semantics, patch, edge and discriminative information from features on deep-to-shallow stages. The network also utilized an effective approach to integrate these features. Specifically, a cross-attention semantics module elicited semantic discriminative features for target classification from deep features. We employed attention mechanisms to construct a patch activation module extracting patch-based discriminative characteristics from middle-level features. In addition, we designed an edge enhancement module leveraging dilated convolutions to extract boundary information from shallow features. Moreover, to effectively integrate output features of various categories, we proposed a deep-emphasis fusion module in the decoder to fuse them. Notably, DASNet maintains a balance between spatial localization and depth-wise semantics, which has not been effectively addressed by other existing RGB-T segmentation methods. Qualitative and quantitative experiments showed that our Depth Adaption SegNet exhibited impressive segmentation performance.

We acknowledge several limitations in the current study. First, our quantitative evaluation was limited to the MFNet dataset. While MFNet provides rich day/night variability and is the most widely used benchmark for RGB-T urban scene parsing, we recognize that evaluating on an additional dataset such as PST900 would further strengthen our generalization claims. Second, while our computational analysis showed that DASNet achieved practical inference speed, further optimization for embedded deployment remains an important direction. Third, while our method achieved impressive mIoU, its performance on certain challenging categories, notably Guardrail, remained suboptimal, which we attribute to limited training samples and the structural complexity of such objects. Finally, our results were based on a single training run and were not accompanied by standard deviations or significance testing. We therefore do not claim statistical significance for the reported margins over competing methods. Conducting multiple runs with different random seeds and reporting mean performance with standard deviation are important directions for future work.