Dual-Branch Transformer–CNN Fusion for Enhanced Cloud Segmentation in Remote Sensing Imagery

Abstract

Cloud coverage and obstruction significantly affect the usability of remote sensing images, making cloud detection a key prerequisite for optical remote sensing applications. In existing cloud detection methods, using U-shaped convolutional networks alone has limitations in modeling long-range contexts, while Vision Transformers fall short in capturing local spatial features. To address these issues, this study proposes a dual-branch framework, TransCNet, which combines Transformer and CNN architectures to enhance the accuracy and effectiveness of cloud detection. TransCNet addresses this by designing dual encoder branches: a Transformer branch capturing global dependencies and a CNN branch extracting local details. A novel feature aggregation module enables the complementary fusion of multi-level features from both branches at each encoder stage, enhanced by channel attention mechanisms. To mitigate feature dilution during decoding, aggregated features compensate for information loss from sampling operations. Evaluations on 38-Cloud, SPARCS, and a high-resolution Landsat-8 dataset demonstrate TransCNet’s competitive performance across metrics, effectively balancing global semantic understanding and local edge preservation for clearer cloud boundary detection. The approach resolves key limitations in existing cloud detection frameworks through synergistic multi-branch feature integration.

1. Introduction

Clouds serve as critical components of meteorological systems, with their dynamic variations serving as essential indicators for climate analysis. It has been estimated that clouds obscure more than 65% across the globe at any particular moment, presenting substantial challenges for remote sensing applications that rely on optical imagery [1]. Accordingly, the accurate delineation of cloud cover has become a subject of considerable research interest. However, owing to the heterogeneity and complexity inherent in remote sensing data, current cloud detection techniques are often susceptible to both omission and commission errors, making methodological advancement particularly complex.

Among established cloud detection approaches, threshold-based methods have been extensively utilized. For instance, the Fmask algorithm introduced by Zhu et al. employs a decision tree for pixel-level discrimination between cloudy and non-cloudy regions [2]. Other notable thresholding strategies include the NOAA Advanced Very-High-Resolution Radiometer (AVHRR) cloud detection algorithm and the ACCA method [3,4]. These methods typically apply empirically defined thresholds to multispectral imagery for cloud discrimination. However, their effectiveness is often limited by their reliance on expert-driven parameter selection and ongoing calibration, which may constrain the detection accuracy.

The use of convolutional neural networks (CNNs) has greatly accelerated progress in automated cloud identification, largely owing to their powerful image feature extraction capabilities. Approaches including Cloud-Net and Cloud-Net+ [5,6] have delivered significant improvements in detection outcomes. Furthermore, research has shown that CNNs are adept in learning low-level cues, and, with the introduction of more advanced networks (e.g., CDNet, CDNetV2), refined fusion and attention mechanisms have been incorporated to further enhance the accuracy [7,8,9]. Nevertheless, CNNs often struggle to represent long-range dependencies and the global semantic context, which still impacts the performance of these models.

Recently, Transformer-based architectures—initially originating in the context of natural language processing—have seen growing adoption in computer vision tasks, where they deliver remarkable results. With the introduction of self-attention mechanisms, Transformers can model relationships between distant parts of an image, thus capturing comprehensive global information. Networks such as DBNet and Cloudformer [10,11] have demonstrated improved results over traditional CNN models. By leveraging both CNNs’ detailed local representations and Transformers’ global modeling, next-generation frameworks are being developed to further advance the cloud detection accuracy.

This study presents a novel feature extraction system named TransCNet, which includes two parallel segmentation branches: a CNN path and a Transformer path. The Transformer path adopts the Pyramid Vision Transformer (PVT) for the modeling of global dependencies, while the CNN path—based on ResNet—specializes in capturing detailed local features from remote sensing images [12,13,14]. Since Transformer models focus primarily on the global context and may miss fine-grained object boundaries, incorporating the CNN path helps to ensure more complete feature representation. The feature aggregation module combines multi-scale features from both branches, supporting the precise joint prediction of cloud regions. Channel attention mechanisms are further applied in decoding to maximize semantic and spatial feature fusion. Validation experiments on the 38-Cloud and SPARCS datasets, as well as a new high-resolution Landsat-8 collection, confirm that the proposed method achieves state-of-the-art performance. The principal contributions of this work are outlined below.

(1)

A novel cloud detection approach is introduced, combining Transformer and CNN architectures to capture both local details and global contexts. This framework improves the precision of cloud region segmentation by aggregating features from both models.

(2)

The proposed feature aggregation module (FAM) efficiently merges multi-level outputs from both pathways, integrating spatial and contextual information. This design overcomes the limitations of using CNN or Transformer models alone, retaining more relevant details for accurate cloud delineation.

(3)

A large-scale, high-resolution cloud detection dataset, named CHLandsat-8, was assembled from 64 full scenes of Landsat-8 satellite imagery, covering various regions in China from January to December 2021. Each image includes pixel-level cloud annotations through manual labeling. The dataset will be publicly available, serving as a benchmark for the development and testing of cloud recognition techniques and supporting ongoing research in the field.

2. Related Works

2.1. Cloud Detection in Remote Sensing Images

Machine and deep learning have transformed remote sensing, as they allow algorithms to learn patterns from data and make predictions, leading to important advances in many applications [15,16]. Recently, deep learning approaches leveraging convolutional neural networks (CNNs) have become the standard for cloud recognition within remote sensing data. As a landmark, Mohajerani et al. put forward an end-to-end framework built upon fully convolutional architectures, laying the groundwork for subsequent improvements in the area [5,6]. Among these, Cloud-Net+ (2021) is notable for its effectiveness on Landsat-8 data, achieving improved detection performance. In pursuit of more precise boundary and spatial characterization, Hu et al. introduced CDUNet in 2021, employing deep learning techniques dedicated to cloud extraction [14]. Later, Pu et al. (2022) advanced the field with a new detection system that employs self-attention layers and a spatial pyramid pooling scheme, with DenseNet serving as the core feature encoder to enrich semantic representation [17]. It has been established that low-level CNN features sharpen the delineation of cloud edges, while higher-level abstractions from deeper layers help to pinpoint the overall cloud location. Yang et al. (2019) presented CDNet, combining codec structures, feature pyramids, and boundary refinement modules to improve the segmentation effectiveness [8]. Building on this, Guo et al. (2020) introduced CDNetV2, which incorporates adaptive feature fusion and semantic flow modules to better distinguish between clouds and snow in satellite imagery [9]. Building upon residual network architectures, Lu et al. presented a multi-scale strip-pool aggregation approach (2022), supplemented by pyramid pooling and strip-based boundary refinement to retain cloud edge information [18]. Additional research has focused on reinforcing global context modeling in CNNs. For instance, Xia et al. devised a global attention fusion residual network featuring specialized modules for accurate cloud region and edge delineation [19]. Guo et al. also presented a streamlined CNN-based network employing multi-feature fusion from visible and near-infrared data, achieving effective segmentation with a fully convolutional structure, termed ClouDet [20]. Despite these advances, limitations in modeling long-range dependencies persist, as the intrinsic characteristics of CNNs impede global contextual integration, thus constraining the segmentation accuracy. This limitation has motivated the exploration of Vision Transformer architectures, which are recognized for their superior global modeling capabilities and form the basis for the present study’s investigation into Transformer-based cloud segmentation [21].

2.2. Vision Transformer

Initially developed for natural language processing, the introduction of Transformer models has led to significant progress in computer vision. Originally de-signed for natural language processing, the Transformer—with its stacked multi-head self-attention and multi-layer perceptron modules—was first presented by Vaswani et al. in 2017 [22]. Dosovitskiy subsequently applied Transformer networks to image classification, demonstrating strong results on ImageNet and driving broad adoption for visual recognition tasks [21,23]. Numerous Transformer-based vision models have since emerged, frequently outperforming traditional CNN architectures across a range of computer vision applications. For example, Liu et al. developed a Transformer-only framework for both RGB and RGB-D saliency detection [24]. Qiu et al. designed an asymmetric bilateral U-shaped Transformer network to simultaneously model global contexts and local details for saliency tasks [25]. In medical image analysis, Zhang et al. constructed an intra-branch parallel architecture that integrates both CNN and Transformer components, yielding improved segmentation robustness [26]. In the context of remote sensing image cloud detection, hybrid architectures integrating CNNs and Transformers have been proposed. Lu et al. described a dual-branch framework for the separate extraction of semantic and spatial detail features, addressing the challenges of false positives and missed detections in high-resolution imagery [18]. Zhang et al. further advanced this approach by designing a detection framework that fuses CNN and Transformer outputs for accurate cloud identification in optical remote sensing data [26]. Nevertheless, the direct application of non-remote sensing-oriented cloud detection methods to satellite imagery remains suboptimal, largely due to the high spatial resolution and heterogeneity of satellite data. Current Transformer-based methods often fail to fully exploit the global contextual features captured by Transformers and the local detail features provided by CNNs, resulting in limited segmentation precision and increased rates of omission and false detection.

3. Method

The core architecture of TransCNet consists of a dual-branch feature extraction network that operates in parallel, along with a highly efficient feature aggregation module. This section first introduces the overall framework of TransCNet and then provides a detailed explanation of the design and construction of the feature aggregation module.

3.1. Problem Formulation

The primary objective of cloud detection in remote sensing imagery is to generate a binary mask that highlights the cloud regions within the input image. The input is represented in the RGB color space, and the output is a binary map that distinguishes between cloud and non-cloud areas. TransCNet, as introduced in this study, is formulated as a composite function that combines CNN and Transformer architectures. The central goal is to develop an effective mapping function that leverages the complementary strengths of both CNNs and Transformers, enabling the model to learn discriminative features for more accurate cloud region segmentation [27,28].

3.2. Overall Framework

The proposed framework is designed to simultaneously acquire global contextual information and local feature representations, which are critical for both the accurate localization and detailed boundary refinement of clouds. Conventional CNN-based methods for cloud detection in satellite imagery (Hu et al., 2021; Pu et al., 2022; Lu et al., 2022; Xia et al., 2021; Guo et al., 2021) have been shown to lack sufficient capabilities for context modeling [14,17,18,19,20], whereas Transformer-based approaches in various vision applications (Qiu et al., 2021; Zhang et al., 2021; Samplawski et al., 2021; Sun et al., 2021; Wang et al., 2021; Dang et al., 2022; Botach et al., 2022) have demonstrated limitations in effectively representing local spatial details [25,26,29,30,31,32,33]. To address these objectives, TransCNet is developed as a parallel dual-branch feature extraction network that leverages the complementary strengths of Transformer and CNN architectures. The network inherits the Transformer’s capacity for global context modeling alongside the CNN’s effectiveness in local feature extraction. TransCNet consists of a parallel dual-branch encoder and decoder.

As illustrated in Figure 1, the architecture features two distinct encoding branches: the grey Transformer path and the orange CNN path. There are four levels in the well-known Transformer network PVT, called the Transformer path. The issue with this path is that it ignores local knowledge in favor of understanding only long-term dependencies. By using the CNN path, which is a well-known CNN network, ResNet-34, this issue is lessened. A dedicated feature aggregation module is constructed to effectively integrate hierarchical features from the two branches. For cloud region segmentation, the decoding network incorporates a channel attention mechanism to enhance the accuracy of the final output.

Transformer Path. In this study, the widely recognized Transformer network PVT is utilized as the backbone for the Transformer path. The four stages of the PVT adopt a uniform architecture consisting of a patch embedding layer followed by multiple Transformer blocks. Specifically, for an input image of size $H\times W\times 3$, the patch embedding layer divides it into $H/4\times W/4$ patches, each with dimensions $4\times 4\times 3$. Positional embeddings are then added to the flattened patches, which are subsequently processed by Transformer blocks. From the second stage onward, the feature map is further partitioned into halves through an additional patch embedding layer, followed by the addition of positional embeddings and processing by a Transformer block. With 64, 128, 320, and 512 channels, respectively, T1, T2, T3, and T4 in Figure 1 show the output features of the PVT from top to bottom for the four stages.

CNN Path. For the CNN branch, the widely used ResNet-34 network is adopted. As indicated in Figure 1, the features from stages C1, C2, C3, and C4 in the CNN path are fused with the corresponding outputs from the Transformer path. Notably, the CNN path is designed to be adaptable, allowing for the substitution of other convolutional architectures if desired. The input to the CNN path is the original remote sensing image, enabling the extraction of rich spatial detail features.

To improve the expressiveness of decoded features, an adapted channel attention (CA) module is integrated within the decoder. As illustrated in Figure 2, this module adopts the conventional squeeze-and-excitation (SE) block framework [34]. Specifically, the CA unit receives both the decoded feature D_i and the fused feature F_i from the aggregation module, generating D_i−1 as input for the following decoding layer.

The mathematical formulation describing channel attention is presented as follows:

$$\mathrm{CA}\left(v^h,W\right)=\sigma \left(con{v}_2\left(R\left(con{v}_1\left(G\left(v^h\right),W_1\right)\right),W_2\right)\right),$$

(1)

where $v^h$ denotes the channel feature vector, $W_1$ and $W_2$ denote the parameters learned, $\sigma (\cdot )$ is the Sigmoid function, $R(\cdot )$ is the ReLU activation function, $G(\cdot )$ denotes the global average pooling operation, and $con{v}_1(\cdot )$ and $con{v}_2(\cdot )$ are the convolution operations.

The inputs and outputs of the channel attention module are defined as

$$D_{i-1}=\mathrm{CA}\left(con{v}_3\left(Cat\left[F_i,D_i\right]\right),W\right), for i\in \{4,3,2,1\},$$

(2)

where $Cat[\cdot ,\cdot ]$ denotes the feature cascade operation, $con{v}_3(\cdot )$ is the convolution operation, and $\mathrm{CA}(v^h,W)$ is the above-mentioned channel attention mechanism.

The outputs generated by each decoder stage (P1–P5) are aggregated to produce the final prediction mask. To enhance training, strong supervision is imposed on these outputs. During the decoding phase, the features undergo batch normalization and activation, followed by two convolutional layers that reduce them to a single channel. A Sigmoid function is then used to produce the probability map, with values ranging from 0 to 1. For joint optimization, TransCNet is trained end-to-end using the binary cross-entropy (BCE) loss [35], which supervises all output layers during training. The overall loss function for deep supervision is given as

$$L={\displaystyle \sum _{i=1}^5\left(\frac{i}{5}{L}_{BCE}\left(P_i,G\right)\right)},$$

(3)

Here, $G$ denotes the ground-truth cloud region, and $L_{BCE}$ refers to the binary cross-entropy loss function, defined as

$$L_{BCE}=-{\displaystyle \sum _{\left(r,c\right)}\left[G\left(r,c\right)\log \left(S\left(r,c\right)\right)\right.}\left.+\left(1-G\left(r,c\right)\right)\log \left(1-S\left(r,c\right)\right)\right],$$

(4)

where $G(r,c)$ is the mask label of the pixel $(r,c)$ and $S(r,c)$ is the predicted cloud area.

3.3. Feature Aggregation Module

To combine the representations obtained from the CNN and Transformer branches, this study introduces an original feature aggregation module (FAM), illustrated in Figure 3. The FAM incorporates both an attention mechanism and a multimodal feature fusion strategy. In Figure 3, ${\mathrm{C}}_i$ denotes the features extracted by the CNN branch, ${\mathrm{T}}_i$ refers to those from the Transformer branch, and ${\mathrm{F}}_i$ represents the output features generated by the FAM. Here, the index $i$ covers values of 1, 2, 3 and 4.

Specifically, in this paper, fused features are obtained by the following operations. First, inspired by DANet, the features of the two branches are fused in a channel-like attention mechanism, defined as follows [36]:

$$\left\{\begin{array}{l}{\mathrm{C}}_i^{\prime }=\mathrm{softmax}\left(con{v}_4\left({\mathrm{C}}_i\right)\times con{v}_5\left({\mathrm{T}}_i\right)\right)\times con{v}_6\left({\mathrm{C}}_i\right){+\mathrm{C}}_i\\ {\mathrm{T}}_i^{\prime }=\mathrm{softmax}\left(con{v}_4\left({\mathrm{C}}_i\right)\times con{v}_5\left({\mathrm{T}}_i\right)\right)\times con{v}_7\left({\mathrm{T}}_i\right){+\mathrm{T}}_i\end{array}\right.,$$

(5)

Then, to further fuse the two branch features, while enhancing the cloud region features and suppressing noise, the fused features are processed using the channel note SE-Block [34]:

$${\mathrm{C}\mathrm{T}}_i^{\prime }=\mathrm{ChannelAttn}\left(Cat\left[con{v}_4\left({\mathrm{C}}_i\right),con{v}_5\left({\mathrm{T}}_i\right)\right]\right),$$

(6)

where $\mathrm{ChannelAttn}(\cdot )$ denotes the channel note SE-Block, and $Cat[\cdot ,\cdot ]$ denotes the feature cascade operation. The final output features of the FAM are as follows:

$${\mathrm{F}}_i=con{v}_8\left(Cat\left[{\mathrm{C}}_i^{\prime },{\mathrm{T}}_i^{\prime },{\mathrm{C}\mathrm{T}}_i^{\prime }\right]\right),$$

(7)

4. Experimental Results and Analysis

4.1. Experimental Setup

TransCNet is implemented in PyTorch 1.8.1. For training, the CNN branch is initialized with ResNet-34 weights, while the Transformer branch uses weights from PVT-Tiny. All other convolutional layers and modules are randomly initialized. The Adam optimizer is selected, with an initial learning rate of 5 × 10⁻³ and weight decay of 0.0005. The learning rate is reduced by a factor of ten every ten epochs. Models are trained for 40 epochs with a batch size of 16. All training is conducted on an NVIDIA Tesla V100 GPU, and evaluation is carried out on an NVIDIA RTX 3060 GPU with 12 GB RAM [25,26].

4.2. Dataset

To comprehensively evaluate the approach, three widely used remote sensing cloud detection datasets are utilized: the 38-Cloud and 95-Cloud public datasets, the SPARCS dataset, and a bespoke CHLandsat-8 dataset. Among these, CHLandsat-8 is notable for its higher spatial resolution and increased diversity of land cover types, posing greater challenges for cloud classification.

The 38-Cloud dataset, as described by Mohajerani S et al. (2019), consists of 38 Landsat-8 scenes, with 18 allocated for training and 20 for testing. Each image is preprocessed to 384 × 384 pixels, producing 8400 samples for training and 9201 for testing [5]. The SPARCS dataset includes 1000 × 1000 pixel patches extracted from 80 different Landsat-8 scenes [35] and is commonly used as a remote sensing benchmark.

CHLandsat-8 comprises 64 large-scale images collected by Landsat-8 satellites from various regions of China in 2021, representing the first open-access high-resolution dataset for cloud detection in the country. Its coverage extends over the northwest, north, Qinghai–Tibet, and southern regions, and it contains a diverse array of environments, including urban areas, snow, ice, grasslands, mountains, forests, oceans, and deserts. Each image is approximately 8000 × 8000 × 3 pixels. All cloud masks were manually annotated at the pixel level and are freely available to support further research.

Annotation was performed by a team of 4 annotators, each responsible for labeling the cloud regions. The process involved assigning a value of 1 for cloud pixels and 0 for background pixels. During annotation, a quality control procedure was implemented, where each annotation was continuously validated and adjusted based on real-time feedback, ensuring consistent and accurate labeling. In case of disagreements, the final decision was made by selecting the annotation with the better performance in tests, ensuring the most reliable output. Of the 64 images, 44 were randomly selected for training (CHLandsat-8-TR), while the remaining 20 were reserved for testing (CHLandsat-8-TE).

To accommodate GPU memory limitations, all images were resized to 352 × 352 × 3 prior to model training. Further information about each dataset is summarized in

Table 1. Dataset details [27].

Dataset	Scenes	Images	Train/Test
CHLandsat-8-TR	44	22,616	Train
CHLandsat-8-TE	20	10,080	Test
38-Cloud-Test	20	10,906	Test
SPARCS	80	720	Test

.

4.3. Evaluation Indicators

Model evaluation is based on six quantitative criteria commonly used in remote sensing cloud detection: the maximum F-measure (MaxFm), mean absolute error (MAE), weighted F-measure (WFm), average F-measure (AvgFm), S-measure (Sm), and E-measure (Em).

The F-measure (F_β) balances precision and recall, with ${\beta }^2$ = 0.3 used to emphasize precision in this study. Both the peak and average values are reported for a comprehensive assessment [37,38]:

$${\mathrm{F}}_{\beta } = {\displaystyle \frac{\left(1 + {\beta }^2\right)\cdot \mathrm{precision}\cdot \mathrm{recall}}{{\beta }^2\cdot \mathrm{precision} + \mathrm{recall}}},$$

(8)

Here, ${\beta }^2$ is configured as 0.3 to highlight accuracy. For the evaluation in this research, the maximum F-measure and the average F-measure are computed [39].

The MAE evaluates the average absolute difference, pixel by pixel, between the predicted and true cloud masks [40]:

$$\mathrm{MAE} = {\displaystyle \frac{1}{W\times H}}{\displaystyle {\sum }_{i=1}^W{\displaystyle {\sum }_{j=1}^H\left|\mathrm{G}\left(i,j\right)-\mathrm{S}\left(i,j\right)\right|}},$$

(9)

where $\mathrm{S}\in {\left[0,1\right]}^{\mathrm{W}\times \mathrm{H}}$ is the predicted mask and $\mathrm{G}\in {\left[0,1\right]}^{\mathrm{W}\times \mathrm{H}}$ is the ground truth.

The weighted F-measure builds upon the standard F-measure by mapping TP, FP, TN, and FN to continuous values, assigning different error penalties [41], and leveraging local spatial neighborhood information $\omega$.

$${\mathrm{F}}_{\beta }^{\omega }={\displaystyle \frac{\left(1+{\beta }^2\right)\cdot {\mathrm{precision}}^{\omega }\cdot {\mathrm{recall}}^{\omega }}{{\beta }^2\cdot {\mathrm{precision}}^{\omega }{+\mathrm{recall}}^{\omega }}}$$

(10)

The S-measure offers a foreground structural assessment by blending region-aware ($S_r$) and object-aware ($S_o$) similarity, with the formula

$${\mathrm{S}}_{\mathrm{m}}=\alpha \ast {\mathrm{S}}_{\mathrm{o}}+\left(1-\alpha \right)\ast {\mathrm{S}}_{\mathrm{r}},$$

(11)

where $\alpha$ is set to 0.5 in this study [42].

The E-measure quantifies both global agreement and local alignment in the segmentation output, given by [43]

$${\mathrm{E}}_m={\displaystyle \frac{1}{W\times H}}{\displaystyle \sum _{c=1}^W{\displaystyle \sum _{r=1}^H\mathsf{\Phi }\left(r,c\right)}},$$

(12)

where $\mathsf{\Phi }$ is the enhanced alignment matrix, H and W denote the image size, and $(r,c)$ are coordinates.

In summary, the MAE evaluates the average absolute difference between the predicted and true cloud masks. Sm primarily focuses on structural evaluation, blending region-aware and object-aware similarity, emphasizing the shapes and boundaries of clouds. Em quantifies both global consistency and local alignment, with a focus on pixel-level alignment. Meanwhile, WFm enhances the traditional F-measure by introducing a weighted mechanism and leveraging local spatial information, improving its performance in handling class imbalance and spatial structure in cloud detection.

4.4. Comparative Experiments

To provide a comprehensive evaluation, the proposed method is benchmarked against multiple leading models, including ResNet-34 [13], PVT [12], FCN8S [44], UNet [45], PSPNet [46], SEGNet [47], GFRNet [48], CDNet [8], CDNetV2 [9], Cloud-Net [5], ClouDet [20], Mask2former [49], Segformer [50], and Cloudformer [11]. All models are trained and tested using the same open-source framework and experimental configuration. Training utilizes the CHLandsat-8-TR set, with the evaluation conducted on three distinct test datasets.

4.4.1. Quantitative Experiments

The segmentation performance on the CHLandsat-8-TE test set is summarized in

Table 2. Quantitative comparison with SOTA methods on the CHLandsat-8-TE dataset [27].

Model	CHLandsat-8-TE (20)
	MAE	MaxFm	AvgFm	WFm	Sm	Em
FCN8s	0.106	0.874	0.858	0.784	0.742	0.827
UNet	0.113	0.862	0.826	0.745	0.729	0.789
PSPNet	0.097	0.879	0.869	0.799	0.767	0.858
SEGNet	0.102	0.874	0.850	0.778	0.754	0.824
GFRNet	0.123	0.852	0.827	0.736	0.716	0.779
Cloud-Net	0.101	0.875	0.846	0.764	0.736	0.795
ClouDet	0.095	0.884	0.870	0.796	0.764	0.828
CDNet	0.129	0.848	0.814	0.722	0.709	0.763
CDNetV2	0.125	0.842	0.823	0.735	0.714	0.790
ResNet-34	0.133	0.841	0.823	0.716	0.691	0.763
PVT	0.144	0.850	0.818	0.706	0.753	0.824
Mask2former	0.102	0.877	0.839	0.765	0.737	0.797
Segformer	0.109	0.867	0.831	0.752	0.733	0.791
Cloudformer	0.112	0.856	0.828	0.742	0.766	0.781
TransCNet	0.082	0.893	0.874	0.815	0.850	0.844

. The results show that TransCNet consistently provides segmentation outputs that closely match the manual labels and achieves top rankings in most evaluation metrics (MaxFm, MAE, WFm, AvgFm, Sm, and Em). The results for each metric are color-coded to indicate the ranking: red (best), blue (second), green (third). While TransCNet achieves leading results in five out of six indicators, its E-measure is only slightly lower than the top score, further validating the effectiveness of the proposed framework for cloud detection.

Table 3. Quantitative comparison with SOTA methods on the 38-Cloud-Test dataset [27].

Model	38-Cloud-Test (20)
	MAE	MaxFm	AvgFm	WFm	Sm	Em
FCN8s	0.069	0.857	0.833	0.768	0.768	0.817
UNet	0.064	0.879	0.862	0.797	0.785	0.856
PSPNet	0.065	0.831	0.823	0.759	0.777	0.889
SEGNet	0.056	0.857	0.850	0.800	0.806	0.912
GFRNet	0.079	0.843	0.824	0.751	0.754	0.817
Cloud-Net	0.055	0.890	0.878	0.761	0.798	0.870
ClouDet	0.052	0.896	0.882	0.819	0.824	0.899
CDNet	0.106	0.835	0.823	0.738	0.727	0.793
CDNetV2	0.108	0.817	0.805	0.718	0.721	0.781
ResNet-34	0.090	0.840	0.797	0.704	0.719	0.770
PVT	0.101	0.851	0.697	0.647	0.737	0.735
Mask2former	0.078	0.846	0.841	0.754	0.761	0.845
Segformer	0.080	0.842	0.831	0.751	0.746	0.821
Cloudformer	0.082	0.841	0.835	0.752	0.742	0.831
TransCNet	0.045	0.867	0.866	0.814	0.859	0.892

summarizes the results obtained when training on CHLandsat-8-TR and testing on the 38-Cloud-Test dataset, following the same evaluation protocol. TransCNet ranks first in two of the six metrics. For MaxFm, it is 0.012 below UNet; for Em, it is 0.02 lower than SEGNet; for AvgFm, it is 0.012 lower than Cloud-Net; and, for WFm, it lags by 0.005 compared to ClouDet.

Table 4. Quantitative comparison with SOTA methods on the SPARCS dataset [27].

Model	SPARCS (80)
	MAE	MaxFm	AvgFm	WFm	Sm	Em
FCN8s	0.143	0.464	0.386	0.307	0.517	0.456
UNet	0.131	0.527	0.451	0.365	0.542	0.506
PSPNet	0.126	0.543	0.480	0.376	0.541	0.550
SEGNet	0.110	0.631	0.555	0.470	0.592	0.595
GFRNet	0.131	0.516	0.444	0.363	0.547	0.512
Cloud-Net	0.121	0.547	0.462	0.380	0.553	0.517
ClouDet	0.105	0.566	0.502	0.452	0.581	0.554
CDNet	0.116	0.616	0.546	0.459	0.592	0.595
CDNetV2	0.122	0.587	0.514	0.425	0.570	0.560
ResNet-34	0.148	0.403	0.364	0.301	0.490	0.442
PVT	0.180	0.512	0.442	0.355	0.479	0.495
Mask2former	0.115	0.618	0.557	0.466	0.595	0.602
Segformer	0.122	0.545	0.471	0.378	0.562	0.546
Cloudformer	0.112	0.624	0.563	0.475	0.602	0.611
TransCNet	0.105	0.645	0.582	0.490	0.606	0.627

reports the results on the SPARCS dataset following training on CHLandsat-8-TR. TransCNet consistently delivers the best performance across all six metrics, confirming its superior cloud detection capabilities relative to existing advanced approaches.

4.4.2. Qualitative Experiments

To thoroughly illustrate the segmentation performance, qualitative visualizations are presented for TransCNet and several established baseline models, such as FCN8S, UNet, PSPNet, and SEGNet, among others. These results demonstrate that TransCNet can reliably recognize a variety of cloud structures across different remote sensing contexts. These categories are systematically summarized in

Table 5. Confusion matrix [28].

	Cloud Region	Non-Cloud Region
Predicted: cloud	True Positive	False Positive
Predicted: non-cloud	False Negative	True Negative

.

TransCNet demonstrates robust adaptability to a range of scene types, as evidenced by the results on the CHLandsat-8-TE evaluation set, which encompasses environments such as grasslands, deserts, plateaus, forests, oceans, and snowfields. Figure 4 displays the side-by-side visual detection outcomes for TransCNet and other competing approaches. It is observed that many CNN-based methods tend to produce more false alarms and omissions, especially when faced with challenging or heterogeneous landscapes, whereas TransCNet substantially reduces such errors.

Figure 5 displays the detection performance on the 38-Cloud-Test test dataset, featuring scene types such as simple, cloudy, partly cloudy, ice and snow, and ocean. TransCNet demonstrates stable and accurate detection, excelling especially in challenging ice and snow scenarios.

On the SPARCS dataset (Figure 6), the visual comparison covers scenes such as thin cloud, simple, complex, snow, cloudy, cloud shadow, confusing background, and partly cloudy. TransCNet consistently yields clearer, more precise cloud delineation than its counterparts.

The results across all three test datasets demonstrate that TransCNet consistently surpasses competing algorithms in overall cloud detection accuracy for a wide range of remote sensing scenes. While thick cloud layers are readily identified by both TransCNet and the baseline models, the detection of cirrus and thin clouds remains challenging due to their high transparency, resulting in increased false negatives. Notably, TransCNet achieves a lower false negative rate than other methods, owing to its integration of Transformer and CNN modules, which jointly enhance both local detail extraction and global contextual understanding. This complementary architecture enables more precise cloud localization and segmentation. However, thin clouds are often misclassified as bright surfaces such as water or snow due to spectral similarity, leading to residual errors. Incorporating additional spectral bands (e.g., NIR, SWIR) or frequency-domain features may help to alleviate these misclassifications, providing a potential direction for further improvement.

In contrast, a standalone CNN model is limited by the size of the convolutional kernels, making it difficult to effectively capture global semantic information. On the other hand, a standalone Transformer model lacks sufficient local information, which weakens its ability to detect small-sized clouds. TransCNet, through its feature aggregation module, cleverly integrates multi-level spatial features and contextual information from both branches, enabling the network to more accurately distinguish cloud regions, reduce false positives, and deliver clearer and more accurate cloud detection results under challenging conditions.

Moreover, TransCNet excels in scenarios characterized by cloud–snow coexistence and complex backgrounds, outperforming the comparison algorithms. This advantage is largely attributed to the feature aggregation module, which effectively combines multi-level spatial and contextual features from both branches. As a result, the network is able to better distinguish cloud regions, reduce false positives, and deliver clearer, more accurate cloud detection even in challenging conditions.

4.5. Ablation Experiments

To systematically examine the contributions of each architectural component, a series of ablation studies is performed. The process begins with training baseline networks that utilize either ResNet-34 (for the CNN branch) or PVT-Tiny (for the Transformer branch), as detailed in

Table 6. Results of the proposed network structure ablation experiment.

Backbone	FAM	CA	CHLandsat8-TE (20)			38-Cloud-Test (20)			SPARCS (80)
			MAE	MaxFm	Sm	MAE	MaxFm	Sm	MAE	MaxFm	Sm
ResNet-34	✗	✗	0.133	0.841	0.691	0.090	0.840	0.719	0.148	0.403	0.490
PVT-Tiny	✗	✗	0.144	0.850	0.753	0.101	0.851	0.737	0.180	0.512	0.479
ResNet-34	✗	✓	0.128	0.840	0.698	0.082	0.851	0.708	0.149	0.408	0.451
PVT-Tiny	✗	✓	0.137	0.855	0.747	0.096	0.848	0.743	0.174	0.522	0.483
ResNet-34 + PVT-Tiny	✗	✗	0.106	0.859	0.794	0.068	0.827	0.785	0.136	0.602	0.571
ResNet-34 + PVT-Tiny	✓	✗	0.088	0.886	0.853	0.051	0.869	0.828	0.108	0.697	0.637
ResNet-34 + PVT-Tiny	✓	✓	0.082	0.893	0.850	0.045	0.867	0.859	0.098	0.728	0.662
ConvNeXt-Tiny + Swin-T	✗	✗	0.097	0.871	0.813	0.061	0.832	0.819	0.125	0.647	0.608
ConvNeXt-Tiny + Swin-T	✓	✗	0.081	0.894	0.859	0.047	0.871	0.856	0.097	0.731	0.669
ConvNeXt-Tiny + Swin-T	✓	✓	0.076	0.902	0.854	0.042	0.869	0.887	0.09	0.758	0.693

. Next, the channel attention (CA) module is incorporated into both baselines, producing the CNN + CA and Transformer + CA variants. Finally, both the CNN and Transformer branches are merged with the proposed feature aggregation module (FAM). Performance is assessed using the MAE, MaxFm, and Sm metrics, with the results summarized in Table 6. Building upon this, we replaced the backbone with ConvNeXt-Tiny [51] and Swin-T [52] for further ablation studies. After replacing the backbones, all metrics showed improvements.

4.6. Interpretive Experiments

To substantiate the contribution of the developed feature aggregation module (FAM), comparative visualizations of intermediate features from the dual-branch aggregation are presented in Figure 7 and Figure 8. In Figure 7, the second-stage feature maps, C2, T2, and F2, correspond to the outputs of the CNN branch, Transformer branch, and FAM, respectively, as defined in Figure 1. “Concatenation” refers to the direct concatenation of C2 and T2 features. The visualization results reveal that the F2 features provide a more accurate and distinct representation of cloud regions compared to simple concatenation, highlighting the effectiveness of the FAM in combining information from both branches.

The comparative results of the third-stage feature visualization of the two-branch feature aggregation are shown in Figure 8. Similarly, C3, T3, and F3 denote the CNN network branch features, the Transformer network branch features, and the FAM output features, respectively, as shown in Figure 1. The clouds in the visualization results in the figure are bright pixel points, i.e., the brighter the color of the pixel point, the higher the probability of it being a cloud. The feature visualization results in Figure 5, comparing C3 with T3 and F3, the FAM outputs features that are more focused on the cloud region, while the background noise is effectively suppressed, resulting in the whole network learning more features that are valuable for cloud detection.

5. Discussion and Analysis

The experimental results across multiple benchmark datasets clearly demonstrate that TransCNet consistently outperforms conventional CNN-based and Transformer-based models in cloud detection tasks. This superiority can be attributed to the complementary strengths of CNNs in capturing fine-grained local spatial details and Transformers in modeling long-range contextual dependencies. By combining these two types of feature representations, TransCNet achieves robust and reliable performance across diverse scene types, as confirmed by both qualitative comparisons and quantitative evaluations.

It is worth noting that, compared with widely used architectures such as UNet and PSPNet, TransCNet produces more accurate segmentation under complex background conditions, particularly in regions with subtle cloud–background transitions. Moreover, in contrast to single-backbone models such as CDNetV1 and CDNetV2, which mainly rely on scaling the backbone complexity to improve the performance, TransCNet employs the feature aggregation module (FAM) to effectively integrate features from its dual backbones. This strategy enables the more efficient utilization of network capacity and achieves a favorable balance between accuracy and computational demands.

Building on the above discussion, Section 5.1 provides a quantitative analysis of the computational complexity to further investigate the trade-off between accuracy and efficiency across different models. This is followed by Section 5.2, which discusses the limitations of the proposed method, thereby offering a more comprehensive perspective on its applicability and outlining potential directions for future research.

5.1. Computational Complexity Analysis

Building upon the complexity analysis of the CDNetV1 [8] and CDNetV2 [9] models, and further comparing them with other models in

Table 7. Computational complexity analysis of different methods.

Model	Params (M)	FLOPs (G) (224 × 224)	Running Time (s) (1 k × 1 k)
UNet	8.6	25.2	1.09
PSPNet	46.6	19.3	1.05
SEGNet	29.7	90.2	1.28
CDNetV1	64.8	48.5	1.26
CDNetV2	65.9	31.5	1.31
ConvNeXt-Tiny + Swin-T	31.6	33.8	1.38
TransCNet	23.0	24.6	1.29

, the results show that TransCNet outperforms other models in cloud detection while maintaining moderate computational complexity. This is reflected in its trainable parameters (Params), floating point operations (FLOPs), and running time at a resolution of 1 k × 1 k. Although TransCNet adopts a dual-backbone structure with ResNet-34 and PVT-Tiny for multi-level feature extraction and fuses the dual-backbone features through the FAM, it gradually restores the cloud mask during the decoding stage using a channel attention mechanism. Its computational complexity is mainly determined by the size of the backbone network. By using ResNet-34 and PVT-Tiny as backbones, TransCNet ensures sufficient depth for the extraction of richer semantic information while maintaining reasonable computational complexity compared to ConvNeXt-Tiny [51] and Swin-T [52], which has dimension settings of (96, 192, 384, 768). The dimensions of (64, 128, 320, 512) in TransCNet effectively prevent an excessive computational burden.

5.2. Limitations

TransCNet adopts a dual-backbone structure as the encoder, which combines the strengths of CNNs and Transformers. While this hybrid architecture has shown strong performance in cloud detection tasks, it also introduces a higher runtime due to the computational cost associated with processing both convolutional and attention-based mechanisms. The use of two backbones, while enhancing feature extraction and enabling the model to capture both local and global dependencies, inevitably leads to an increase in computational complexity. As a result, the model’s runtime may not be ideal for real-time applications, especially when working with larger datasets or higher-resolution images.

Additionally, the current model is limited to cloud detection using RGB images and has not yet been extended to multispectral data. This restricts the model’s applicability to scenarios where multispectral or hyperspectral information could provide additional valuable insights, such as in detecting different types of cloud cover or distinguishing between clouds and other atmospheric phenomena. The potential for expanding TransCNet to handle multispectral data is considerable, but it remains an area for future work, as multispectral images would introduce greater complexity in terms of data preprocessing, feature extraction, and model training.

6. Conclusions

In this study, we propose TransCNet, a novel cloud detection model that integrates a dual-backbone architecture with CNN and Transformer branches. The feature aggregation module (FAM) effectively fuses information from both branches, as confirmed by intermediate feature visualizations (Figure 7 and Figure 8), where FAM-enhanced features (F2) yield more precise cloud region representations than simple concatenation. The experimental results demonstrate that TransCNet consistently outperforms competing models; however, it also exhibits a higher computational cost and is currently restricted to RGB inputs, which limits its applicability to multispectral data. These findings reveal a trade-off between performance and efficiency. Future research will focus on reducing the computational complexity, extending the framework to multispectral imagery, enabling real-time applications, and exploring cloud removal techniques to further improve the detection accuracy and overall model robustness.