Exploring Uncertainty-Based Self-Prompt for Test-Time Adaptation Semantic Segmentation in Remote Sensing Images

Abstract

Test-time adaptation (TTA) has been proven to effectively improve the adaptability of deep learning semantic segmentation models facing continuous changeable scenes. However, most of the existing TTA algorithms lack an explicit exploration of domain gaps, especially those based on visual domain prompts. To address these issues, this paper proposes a self-prompt strategy based on uncertainty, guiding the model to continuously focus on regions with high uncertainty (i.e., regions with a larger domain gap). Specifically, we still use the Mean-Teacher architecture with the predicted entropy from the teacher network serving as the input to the prompt module. The prompt module processes uncertain maps and guides the student network to focus on regions with higher entropy, enabling continuous adaptation to new scenes. This is a self-prompting strategy that requires no prior knowledge and is tested on widely used benchmarks. In terms of the average performance, our method outperformed the baseline algorithm in TTA and continual TTA settings of Cityscapes-to-ACDC by 3.3% and 3.9%, respectively. Our method also outperformed the baseline algorithm by 4.1% and 3.1% on the more difficult Cityscapes-to-(Foggy and Rainy) Cityscapes setting, which also surpasses six other current TTA methods.

1. Introduction

In the actual operation of intelligent vehicles, it is necessary to establish a high-level understanding of the environment to assist decision making [1,2]. There is a need to build the real-time and accurate perception capability through effective data management and processing methods [3,4]. Among various perception methods, machine vision-based methods have become an important part due to their comprehensive, intuitive, and cost-effective advantages [5]. Semantic segmentation [6], in particular, can directly determine the positions and occupancy of various entities in the scene, thereby effectively aiding intelligent vehicles. Real-world scenes are often complex and diverse, varying in city styles and weather (such as rain, fog, snow, and nighttime) [7,8,9]. However, segmentation models trained offline and based on a fully supervised paradigm often lack generalization in these unseen or adverse conditions. Therefore, a lot of works adopted Domain Adaptation (DA) [10,11,12,13,14,15,16,17,18] to train, attempting to transfer the knowledge from existing datasets to unlabeled environments. Although DA effectively improves the model’s generalization, it still follows an offline training approach, which fixes the model’s performance in real-world scenarios once training is complete, making it difficult to adapt to other more complex scenes. Considering that some acronyms will be used frequently, we have listed the acronyms at the end for facilitating reading and understanding the paper.

Another direction is to explore the commonalities between data with different distributions, which is called Domain Generalization (DG, or OpenDA) [19,20,21,22,23,24,25]. This strategy aims to extract domain-invariant knowledge from a wide range of data, hoping to achieve better performance on unseen target domains. In addition to using real data, synthetic datasets [24,25] are also widely used in DG. Various training augmentation [20,21] and test augmentation methods [22,23] also fall into this category. Although DG attempts to address the generalization problem from a more macro-perspective, the common features extracted by DG are often weak to capture the variations in real-world scenes.

Test-time adaptation (TTA) [26,27,28,29,30,31] aims to make models adapt different conditions during testing. TTA can deal with newly arriving data, thereby updating the knowledge stored in the model. According to the trend of changes in the test data, TTA can be divided into continuous TTA [32] and non-continuous TTA. Continuous TTA is specifically designed to handle scenarios with continuous changes, making it particularly suitable for automated vehicle perception. However, continuous TTA faces two challenges: (1) how to effectively extract knowledge from unlabeled data, and (2) how to avoid or mitigate catastrophic forgetting. This is because the model can be easily affected by wrong predictions and noise during long-time adaptation.

Previous TTA approaches can be categorized into distribution alignment strategies [28,29,30,31] and consistency learning strategies [27,32]. Distribution alignment strategies assume that the distribution differences between two domains can be characterized by the parameters of normalization layers. Therefore, continuous domain discrepancy alignment can be achieved by adjusting normalization layers such as Batch Normalization (BN) [33]. However, it should be noted that not every neural network has BN layers, and fine-tuning BN layers usually cannot accurately capture domain differences in most cases. On the other hand, consistency learning employs semi-supervised learning by using a Mean-Teacher architecture [34] to generate pseudo-labels for multiple augmented images based on the teacher model and calculate the consistency loss between the student model and the teacher model. Here, the motivation for this adaptation method is not clear, and the introduction of test-time augmentation [22] significantly slows down the training speed. To address catastrophic forgetting [35], stochastic parameter restore [32] and regularization [27] are commonly used. Random parameter recovery periodically reassigns a portion of the original model’s parameters to the current model to counteract forgetting, while regularization attempts to constrain the model to make it less sensitive to domain changes. Although these two methods effectively alleviate forgetting, they also conservatively limit the model.

With the rise of large-scale models [36,37,38], the use of prompt tuning [36] techniques in Natural Language Processing (NLP) has been introduced to computer vision as a highly promising direction. Prompt allows for the preservation of the original model while using additional learnable parameters to handle the features of newly encountered data [39]. Recently, some studies have proposed the concept of Visual Domain Prompt to address domain gaps [40,41,42], but they often focus on classification tasks. There is also the problem that domain prompts lack clear meaning (as is shown in Figure 1). It is worth noting that the prompt holds great potential for building domain cues for models, which will help acquire knowledge from unlabeled data, and persistent prompts with additional learnable parameters are expected to alleviate the forgetting.

We believe that the domain gap is the fundamental reason for increased uncertainty in model predictions. The role of the prompt should be to guide the model’s attention to areas of high uncertainty, thereby helping the model achieve stability during testing. To address this problem, we propose a Uncertainty-based Self-Prompting (USP) method to assist the model to adapt during testing. The method still utilizes the Mean-Teacher architecture, where the teacher model not only generates pseudo-labels but also produces uncertainty maps. Then, the testing images, uncertainty maps and the prompt module are collectively used as visual prompts for the student model’s computation. This method effectively guides the model to focus on regions with high uncertainty (i.e., areas with significant domain differences) and helps the model explore cross-domain information. Regarding alleviating forgetting, we do not follow VPT [39] (this method freezes the backbone and only trains the prompt and segmentation heads) and adjust all the parameters. Additionally, after each adaptation, the prompt module updates the next prediction’s prompt using the exponential moving average (EMA) technique. We compared our algorithm with six other online TTA algorithms, one continuous TTA algorithm [32] as well as five applied on intelligent vehicle segmentation methods. Specifically, we conduct experiments on three benchmarks (Cityscapes [43], Foggy and Rainy-Cityscapes [7], and ACDC [9]) and demonstrate its superiority in terms of accuracy and efficiency. The structure of the paper is as follows. In Section 3, we describe the detail of our method, including an uncertainty prompt and domain prompt update strategy, etc. In Section 4, we set TTA and CTTA experiments to compare the performance of our method with other state-of-the-art methods; then, the ablation studies and discussion are conducted in Section 5.

Our contributions can be summarized as follows:

• We propose a self-prompted test-time adaptation method based on uncertainty, which does not require any prior knowledge, and effectively directs the model’s attention to domain gap adaptation. At the same time, our method effectively improves the anti-forgetting ability and maintains effective performance over multiple rounds of continuous adaptation.
• Our proposed method surpasses six previous online TTA methods and one continuous TTA semantic segmentation method, demonstrating its effectiveness in cross-domain learning.

2. Related Work

2.1. Road Scene Semantic Segmentation

There have been numerous studies on understanding road scenes using deep learning methods. A survey [6] highlights the importance of semantic segmentation in autonomous driving scene understanding. DSiV [4] and [44] integrate visual data into the field of autonomous driving data science. Building upon these works, [45] proposes a hierarchical and interpretable approach to autonomous driving, providing a multi-scale perspective for future research. FCIS [46] introduces an instance-level pixel inference method for more accurate contour detection, while [47] presents a scene-adaptive multi-scale semantic perception framework for 3D semantic segmentation, estimating spatial occupancy in the scene. MPCNet [48] proposed a multi-pool contextual segmentation network with high speed. FT-Net [49] tries to use few-shot learning to improve segmentation results. However, these methods do not consider the segmentation performance under adverse imaging conditions and need to improve adaptability to scene variations.

2.2. Domain Adaptation

Domain adaptation aims to transfer knowledge from one (source) domain to another (target domain). Typically, labels in the source domain are easily obtained, such as synthetic data (e.g., GTA5 [50] and SYNTHIA [51]) and annotated real data (e.g., Cityscapes [43]), while labels in the target domain are difficult to acquire. This leads to a poor generalization of models trained on the source domain datasets [52]. Early domain adaptation methods primarily used generative adversarial networks [53], treating the domain gap as perturbations or noise, trying to align the two domains through regularization [11,12,54]. However, this approach was too coarse in computing discriminative probabilities. Subsequently, many works attempted contrastive learning for domain adaptation, using the source domain as a reference to guide model learning in the target domain. DISE [55] separates image features into domain-invariant and domain-specific features. ProDA [56] utilizes prototypes for learning. DAFormer [10] is the first to introduce the Transformer [57] into domain adaptation tasks and adaptively adjusts the target domain loss weights during training to guide capability transfer. HRDA [17], built upon DAFormer, integrates high-resolution attention prediction. MIC [18] introduces masked learning [58], aiming to let the student model recover the masked spatial structure and enhance spatial consistency in domain adaptation. ALST [59] summarizes the domain adaptive segmentation algorithms applied to intelligent vehicles. However, domain adaptation still relies on offline learning and struggles with the contradiction between unlimited test data and limited training data.

2.3. Test-Time Adaptation

Test-time adaptation (TTA) aims to make models adapt to newly coming data during testing. Theoretically, TTA enables infinite testing data be used for training, achieving lifelong learning [60]. During TTA, no source domain data are needed, forcing the TTA algorithm to self-supervised train entirely in the target domain. The initial challenge for TTA is how to acquire knowledge from unlabeled test data. Some approaches use generative models for feature alignment [61], while others utilize regularization methods to fine-tune existing models [28,29]. For instance, TENT [28] adjusts the BN layer through entropy minimization. Another problem in TTA is overcoming the forgetting [35] caused by continuous training. Some approaches employ periodic replay-based correction [62], while most methods adopt regularization techniques [27,30,31] aiming to constrain the model’s attention to domain-invariant robust features, reducing interference from erroneous details. CoTTA [32] tackles the scenario of continuous domain changes by developing a domain adaptation architecture suitable for any model. It mitigates forgetting by utilizing random parameter restoration and improves performance on the testing set through test-time augmentation and weight-averaged teacher knowledge collection. Nonetheless, these methods lack an explicit handling of domain discrepancy in their motivations, resulting in weakness when address the difficult challenges posed by domain gaps.

2.4. Visual Domain Prompt

Prompt originated from large-scale natural language processing models [36] and has been introduced to the computer vision domain. In NLP, prompts are commonly presented as questions or fill-in-the-blank forms, aiming to guide the model to recover the missing parts. Therefore, early applications of prompts in computer vision with large models were primarily task prompts [38,63], attempting to guide the models to complete the desired tasks through task examples. Subsequently, some works attempted to tune the models using prompts. VPT [39] introduced learnable parameters into prompts and achieved good optimization results with freezing the backbone network. Later, DePT [41] incorporated the idea of VPT into TTA by adding prompts to the memory pool for feature aggregation. VDP [40] first proposed the concept of Visual Domain Prompt, which froze the model’s backbone, utilized learnable parameters to randomly mask the model, and set domain-specific and domain-agnostic prompts. However, this kind of prompt still lacks interpretable domain specificity. SVDP [64] employed uncertainty maps generated by MC-Dropout to guide the model to focus on areas with significant domain discrepancies sparsely. SVDP [64] enables prompts to possess domain-aware properties for the first time. In this paper, we use entropy to represent uncertainty, with the predicted image, entropy map and learnable parameters as components of the Visual Domain Prompt, enhancing the model’s ability to fuse global domain discrepancy features.

3. Method

3.1. Preliminaries

3.1.1. Test-Time Adaptation (TTA)

TTA aims to utilize a pretrained model on the source domain (${\mathcal{X}}_S$, ${\mathcal{Y}}_S$) to adapt to unlabeled and potentially differently distributed, ${\mathcal{X}}_{T_1}$, ${\mathcal{X}}_{T_2}$ … ${\mathcal{X}}_{T_n}$ during testing, where ${\mathcal{X}}_{T_i}={\left\{x_i^T\right\}}_{i=1}^{N_t}$, $N_t$ represent the scale of the target domain. The overall training only has access to target domain data once and cannot access the source domain. The target domain for TTA can be a single one or multiple continually changing unknown domains (CTTA) with the latter more in line with that of the real world.

3.1.2. Visual Domain Prompt

VDP [40] first introduced Visual Domain Prompt to serve the continual adaptation of classification tasks. Specifically, VDP($\mathbf{p}$) is a dense set of learnable parameters added to the input image:

$$\tilde{\mathbf{x}}=\mathbf{x}+\mathbf{p}$$

(1)

3.2. Overview

We try to let the model handle the domain gap according to uncertainty by itself. The overall framework is shown in Figure 2, which still adopts the Mean-Teacher [34] architecture, in which a robust weighted-averaged teacher model $f_{{\theta }_{\mathcal{T}}}$ generates prompts by the uncertainty, thus guiding the student model (main model) $f_{{\theta }_{\mathcal{S}}}$ to reinforce learning in regions with significant domain differences. Our approach is a compromise between the Dense Rectangular Prompt (VDP) [40] and the Sparse Prompt (SVDP) [64]. To mitigate the overconfidence of the model (where high uncertainty does not necessarily mean wrong predictions), prompts are randomly dropped, referred to as Prompt Dropout. After each testing adaptation, the parameters of the current prompt are transferred next time using an Exponential Moving Average (EMA) [65], which is shown in Figure 3. Following previous work [32,64], the loss function still employs consistency loss between the student output and pseudo-labels generated by the teacher model. Concretely, the original target images will undergo strong augmentation to obtain a pseudo-label, and the entropy of teacher’s prediction will be the original hint. Next, the uncertainty-based prompt module will process the prompt features, which are then randomly dropped. Finally, the prompt will be added on the original image. During the test-time adaptation, the prompt module will be updated by a historical version of the last iteration. After being prompted, the student model will pay more attention to high-entropy regions and attempts to restore the original image features compared to the pseudo-label, achieving similar effects to Masked Image Learning.

3.3. Uncertainty Prompted and Dropout

For single test data $x_t$, we adopt Shannon entropy [66] $H\left(\widehat{y}\right)$ of the teacher prediction $H\left(\widehat{y}\right)=-{\sum }_c{f}_{{\theta }_{\mathcal{T}}}\left(\widehat{y}\right)log{f}_{{\theta }_{\mathcal{T}}}\left(\widehat{y}\right)$ as a measure of uncertainty. During the adaptation process, the weighted-averaged teacher model $f_{{\theta }_t}$ accumulates knowledge, allowing its entropy distribution to express the model’s adaptability to the current target domain. In sparse (or image-level) tasks such as classification, optimizing the loss function with respect to entropy may result in the model assigning probability to all classes [28]. However, in dense (or pixel-level) tasks like semantic segmentation, the distribution of entropy possesses clear spatial properties. Furthermore, our method treats entropy merely as a prompt without considering its specific semantic information, thereby facilitating the model to conduct more targeted optimization.

After obtaining the entropy map $H\left(\widehat{y}\right)$, we employ a prompt module E consisting of convolutional layers to jointly process with the image $x_t$. Specifically, we concatenate the entropy map as the fourth channel of $x_t$ and then encode the image and the entropy map to the prompt feature $\mathbf{f}$:

$$\mathbf{f}=E\left([x_t,H\left(\widehat{y}\right)]\right)$$

(2)

Although the entropy indicates where the model should pay more attention, high entropy cannot confirm the model’s predictions here are incorrect. To avoid over-fitting, we employ random dropout, which proposed stochastic abandon uncertain features in $\mathbf{f}$. Specifically, we generate a mask $\mathbf{M}$ using the Bernoulli distribution with probability p and randomly drop out features from $\mathbf{f}$ that exceed a certain threshold $\tau$.

$$\mathbf{M}\sim \mathrm{Bernoulli}\left(p\right)$$

(3)

$${\mathbf{p}}_{\mathrm{r},\mathrm{c}}=\left\{\begin{array}{cc}{\mathbf{f}}_{\mathrm{r},\mathrm{c}}\hfill & {\mathbf{M}}_{r,c}=1\cup {\mathbf{f}}_{\mathrm{r},\mathrm{c}}<\tau \\ 0\hfill & {\mathbf{M}}_{r,c}=0\cap {\mathbf{f}}_{\mathrm{r},\mathrm{c}}>\tau \end{array}\right.$$

(4)

where the r and c are the row and column number of $H\left(\widehat{y}\right)$, respectively. The processed prompts are added to the image, constituting the final image which will be fed into the main model. Thus, the domain information will be encoded:

$${\tilde{x}}^T=x^T+\mathbf{p}$$

(5)

3.4. Domain Prompt Updating

We follow prior works [10,34,64] to perform exponential moving average (EMA) [65] parameter updates between teacher–student networks and the prompt module, which is shown in Figure 3. We believe that the knowledge in the prompt module should be temporally updated, and the degree depends on the current mean entropy value. If it is small, the prompt should be relatively stable, and more weights from previous iterations should be reserved. The parameters ${\theta }_{\mathcal{T}}^t$ of teacher model are updated by ${\theta }_{\mathcal{S}}^t$ of the student model by Equation (6):

$${\theta }_{\mathcal{T}}^{t+1}=\alpha {\theta }_{\mathcal{T}}^t+(1-\alpha ){\theta }_{\mathcal{S}}^t$$

(6)

For the prompt module E, we hope it can accumulate temporal information and not be limited to encode the domain information of the current image; thus, more domain-invariant features can be encoded as well as the relationship underlying the uncertainty map $H\left(\widehat{y}\right)$ and the original image $x^T$. Therefore, we follow SVDP [64] to update the prompt module based on the image-level uncertainty. We assume that if the uncertainty $\pi \left(x^T\right)$ is low, there is a higher probability of accurate predictions, and thus minimal modifications are needed for the prompt module and vice versa. Here, we use Equation (8) to define the EMA hyper-parameter in Equation (7), which is also shown in Figure 3.

$$E_t=\lambda E_{t-1}+(1-\lambda )E_t$$

(7)

$$\lambda =1-\left(\pi \left(x^T\right)\right)\times 0.01$$

(8)

3.5. Loss Function

Like previous TTA methods [32,64], we utilize the teacher model $f_{{\theta }_{\mathcal{T}}}$ to generate pseudo-labels ${\widehat{y}}^T$ which are then refined through test-time augmentation [23] and confidence filtering. Subsequently, we employ the pixel-wise cross-entropy consistency loss as the objective function for optimization.

$${\mathcal{L}}_{con}\left({\tilde{x}}^T\right)=-\sum _{w,h}^{W,H}\sum _{c=1}^C{\widehat{y}}^T(w,h,c)log{\tilde{y}}^T(w,h,c)$$

(9)

where C means the number of categories and ${\tilde{y}}^T$ is the output of the student model $f_{{\theta }_{\mathcal{S}}}$. When applying test-time augmentation, this function will execute multiple times to complete the consistency training between the original image and several augmented images.

4. Results

4.1. Task Settings

4.1.1. TTA and CTTA

Both tasks aim to make the pretrained model on the source domain adapt the unseen target domains. During adaptation, the source domain is inaccessible, and the data from the target domain can only be accessed once. The basic setting of continual test-time adaptation (CTTA) is similar to TTA, but it involves a continuously changing target domain sequence, which introduces additional challenges. For facilitating reading, the design of all the experiments is shown in Figure 4.

4.1.2. Cityscapes-to-ACDC

In all the TTA scenarios, we use Cityscapes [43] as the source domain to pretrain the model. ACDC [9] is a semantic segmentation benchmark specifically designed for cross-domain learning tasks. It comprises four challenging conditions: fog, rain, snow, and night. For the TTA task, our approach will make each pretrained model individually adapt to each scene. In case of CTTA, we follow previous work [32,64] and simulate real-world variations by repeating the same sequence of target domains (Fog→Night→Rain→Snow) multiple times. As the ACDC dataset is constructed by extracting video frames, it inherently possesses the attribute of temporal continuity.

4.1.3. Citsycapes-to-(Foggy and Rainy) Citsycapes

These two datasets [7] apply synthetic fog and rain to Cityscapes [43] images. Compared to ACDC [9], Foggy and Rainy Cityscapes are more difficult due to the larger density of adverse condition factors. Following the previously mentioned TTA configuration, we adapt the model to each scene individually.

4.2. Dataset and Metrics

The datasets are listed in

Table 1. The datasets used.

Dataset	Scenario	Labelled Images	Number of Label Used	Experiment Purpose
Cityscapes	Autonomous Driving in clear scene	2975(train)/500(val)	2975(for train source)	not involved in testing
Foggy Cityscapes	Autonomous Driving in synthetic foggy scene	2975(train)/500(val)	0(TTA)/2975(for evaluation)	Test the performance of TTA
Rainy Cityscapes	Autonomous Driving in synthetic rainy scene	2975(train)/500(val)	0(TTA)/2975(for evaluation)	Test the performance of TTA
ACDC-Fog	Autonomous Driving in real foggy scene	400(train)/100(val)	0(TTA)/400(for evaluation)	Test the performance of TTA
ACDC-Rain	Autonomous Driving in real rainy scene	400(train)/100(val)	0(TTA)/400(for evaluation)	Test the performance of TTA
ACDC-Night	Autonomous Driving in real nighttime scene	400(train)/106(val)	0(TTA)/400(for evaluation)	Test the performance of TTA
ACDC-Snow	Autonomous Driving in real snowy scene	400(train)/100(val)	0(TTA)/400(for evaluation)	Test the performance of TTA

. Cityscapes is only used to train the source domain model and does not participate in the adaptation process. The remaining datasets serve as benchmarks for evaluating the performance. TTA tasks differ from other tasks in that all images in its training set will be passed in once as “test data”, so the model needs to be evaluated on its training set images. The Foggy and Rainy Cityscapes dataset contained images with various levels of synthetic adverse conditions, and only the lightest 2975 training images were selected for evaluation. The ACDC dataset is divided into subsets of four weather types, each of which includes about 400 and 100 labeled images in the training and validation set, respectively. We use the mIoU (mean Intersection over Union) and mAcc (mean accuracy) as the evaluation metrics. Assuming there are a total of $k+1$ classes, in which 1 is the background class, and $p_{ij}$ represents the number of pixels labeled as class i and predicted as class j; then, mIoU and mAcc can be calculated as follows:

$$\mathrm{mIoU}=\frac{1}{k+1}\sum _{i=0}^k\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}}$$

(10)

$$\mathrm{mAcc}=\frac{1}{k+1}\sum _{i=0}^k\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}}$$

(11)

4.3. Experiment Workflow

In Figure 4, we show the idea of the whole experiment. In order to explore the USP proposed in this paper, we divided the experiment into two parts. The first part includes the Verification experiment, which compares the performance of our USP with other methods, including the performance comparison results under three experimental settings. This section corresponds to

Table 2. Performance comparison of Cityscapes-to-ACDC TTA. We use Cityscape as the source domain and ACDC as the four target domains in this setting. Source domain data are unavailable, while target domain data are only accessed once during testing. The best performance data is shown in bold.

Test-Time Adaptation		Source2Fog		Source2Night		Source2Rain		Source2Snow		Mean-mIoU↑
Method	REF	mIoU↑	mAcc↑	mIoU↑	mAcc↑	mIoU↑	mAcc↑	mIoU↑	mAcc↑
ERFNet	TITS2018 [67]	61.3	71.1	28.4	41.6	47.6	61.0	51.8	62.7	47.3
IAL	TITS2019 [68]	64.0	74.8	34.5	46.3	52.8	67.2	54.0	66.4	51.3
SegFormer	NeurIPS2021 [57]	69.1	79.4	40.3	55.6	59.7	74.4	57.8	69.9	56.7
TENT	ICLR2021 [28]	69.0	79.5	40.3	55.5	59.9	73.2	57.7	69.7	56.7
CoTTA	CVPR2022 [32]	70.9	80.2	41.2	55.5	62.6	75.4	59.8	70.7	58.6
MLFNet	TIV2023 [69]	68.4	77.3	39.2	54.1	60.8	72.3	56.3	68.2	56.2
EIE	TITS2023 [70]	70.2	79.1	40.5	55.0	60.2	72.1	58.6	70.3	57.4
DePT	ICLR2023 [41]	71.0	80.2	40.9	55.8	61.3	74.4	59.5	70.0	58.2
VDP	AAAI2023 [40]	70.9	80.3	41.2	55.6	62.3	75.5	59.7	70.7	58.5
SVDP	ICCV2023 [64]	72.5	81.4	45.3	58.9	65.7	76.7	62.2	72.4	61.4
USP	Ours	73.2	82.1	44.9	59.4	66.4	77.0	63.2	72.8	61.9

,

Table 3. Performance comparison for Cityscape-to-ACDC CTTA. We take the Cityscape as the source domain and ACDC as the fourtarget domains. During testing, we sequentially evaluate the four target domains multiple times. Mean is the average score of mIoU for all times. Gain refers to the improvement achieved by the method compared to the source model. The best performance data is shown in bold.

Time		t————————————————————————————————>
Round		1					2					3					Mean	Gain
Method	REF	Fog	Night	Rain	Snow	Mean	Fog	Night	Rain	Snow	Mean	Fog	Night	Rain	Snow	Mean
ERFNet	TITS2018 [67]	61.2	28.4	47.5	51.8	47.2	61.2	28.4	47.5	51.8	47.2	61.2	28.4	47.5	51.8	47.2	47.2	-
IAL	TITS2019 [68]	64.0	34.4	52.8	54.0	51.3	64.0	34.4	52.8	54.0	51.3	64.0	34.4	52.8	54.0	51.3	51.3	-
MLFNet	TIV2023 [69]	68.4	39.3	60.9	56.3	56.2	68.4	39.3	60.9	56.3	56.2	68.4	39.3	60.9	56.3	56.2	56.2	-
EIE	TITS2023 [70]	70.1	40.3	60.3	58.5	57.3	70.1	40.3	60.3	58.5	57.3	70.1	40.3	60.3	58.5	57.3	57.3	-
SegFormer	NeurIPS2021 [57]	69.1	40.3	59.7	57.8	56.7	69.1	40.3	59.7	57.8	56.7	69.1	40.3	59.7	57.8	56.7	56.7	-
TENT	ICLR2021 [28]	69.0	40.2	60.1	57.3	56.7	68.3	39.0	60.1	56.3	55.9	67.5	37.8	59.6	55.0	55.0	55.7	-1.0
CoTTA	CVPR2022 [32]	70.9	41.2	62.4	59.7	58.6	70.9	41.1	62.6	59.7	58.6	70.9	41.0	62.7	59.7	58.6	58.6	+1.9
DePT	ICLR2023 [41]	71.0	40.8	58.2	56.8	56.5	68.2	40.0	55.4	53.7	54.3	66.4	38.0	47.3	47.2	49.7	53.4	-3.3
VDP	AAAI2023 [40]	70.5	41.1	62.1	59.5	58.3	70.4	41.1	62.2	59.4	58.2	70.4	41.0	62.2	59.4	58.2	58.2	+1.5
SVDP	ICCV2023 [64]	72.5	45.9	67.0	64.1	62.4	72.2	44.8	67.3	64.1	62.1	72.0	44.5	67.6	64.2	62.1	62.2	+5.5
USP	Ours	74.2	44.6	67.4	64.5	62.7	74.1	44.6	67.3	64.2	62.5	74.1	44.6	67.0	63.8	62.3	62.5	+5.8

,

Table 4. Comparison of performance between models trained with USP and without USP on the four target domains. The best performance data is shown in bold.

		Sky	Road	Build	Veget.	Train	Car	Terrain	Truck	S.walk	Rider	Tr.Sing	Tr.Light	Bus	M.Bike	Wall	Person	Pole	Bike	Fence	mIoU
S2Fog	w/o USP	98.1	90.6	88.3	84.3	82.6	89.4	73.5	77.9	80.4	76.9	79.3	69.4	78.6	55.8	60.1	59.9	49.3	30.3	23.3	70.9
	w/ USP	98.8	93.5	87.2	81.4	88.9	92.6	77.4	80.1	79.5	81.2	80.1	75.2	77.1	56.9	63.4	60.2	44.7	40.8	32.5	73.2
S2Night	w/o USP	37.9	76.4	68.3	65.7	32.9	72.1	38.4	15.3	50.3	20.3	56.7	35.4	42.9	26.4	6.60	54.4	38.3	33.8	10.2	41.2
	w/ USP	44.6	75.1	79.2	48.3	50.3	68.9	41.2	26.4	50.1	25.2	37.0	15.8	40.3	31.5	29.4	55.9	37.1	47.6	18.3	44.9
S2Rain	w/o USP	96.4	83.1	85.4	92.6	76.5	84.3	41.8	65.9	45.4	22.1	67.3	62.4	58.7	49.7	44.7	57.7	52.1	56.3	47.5	62.6
	w/ USP	97.9	88.2	90.4	91.0	79.4	88.9	43.0	61.0	55.4	45.3	62.1	62.4	82.3	42.8	60.3	58.9	51.8	57.0	41.0	66.3
S2Snow	w/o USP	94.7	81.3	83.7	89.4	78.3	87.8	10.3	54.9	49.8	43.1	64.2	65.4	51.9	37.5	35.3	64.1	47.0	57.1	40.8	59.8
	w/ USP	96.9	84.1	84.2	85.3	87.1	88.9	25.6	53.5	37.4	55.3	62.4	70.3	74.1	39.2	28.5	61.6	51.3	61.7	53.2	63.2

and

Table 5. Cityscapes to (Foggy and Rainy) Cityscapes TTA performance comparison. The best performance data is shown in bold.

Foggy	SegFormer	TENT	CoTTA	DePT	VDP	SVDP	USP
mIoU	69.2	69.3	72.1	71.9	71.8	74.5	76.2
mAcc	79.1	79.0	79.4	80.2	80.0	82.3	84.3
Rainy	SegFormer	TENT	CoTTA	DePT	VDP	SVDP	USP
mIoU	58.4	58.7	62.4	61.2	63.7	65.3	65.3
mAcc	71.5	71.4	74.0	72.4	73.1	75.6	77.7

and Figure 5. The second part includes the Exploration experiment, which mainly explores the function of each subcomponent (

Table 6. Ablation: ✓ indicates that the component is being used in the current experiment. Contribution of each component.

	TS 1	USP 2	DPU 3	P-Drop 4	mIoU	mAcc
$E{x}_1$					69.1	79.4
$E{x}_2$	✓				69.4	79.6
$E{x}_3$	✓	✓			72.4	81.2
$E{x}_4$	✓	✓	✓		72.9	81.2
$E{x}_5$	✓	✓		✓	73.0	81.8
$E{x}_6$	✓	✓	✓	✓	73.2	82.1

¹ teacher–student framework. ² uncertainty-based prompt. ³ domain prompt updating. ⁴ prompt dropout.

), and the influence of hyper-parameters (Figure 6 and Figure 7) and prompt module structure on the performance (

Table 7. The effect of different prompt module architecture. We design four prompt modules with different number of parameters.

Prompt Module City-to-ACDC(fog) TTA	Architecture
	Prompt|S	Prompt|M	Prompt|L	Prompt|H
Encoder	$\left[\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 32\end{array}\right]$	$\left[\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 32\\ \mathrm{Conv}& 3\times 3& 64\end{array}\right]$	$\left[\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 32\\ \mathrm{Conv}& 3\times 3& 64\\ \mathrm{Conv}& 3\times 3& 96\end{array}\right]$	$\left[\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 32\\ \mathrm{Conv}& 3\times 3& 64\\ \mathrm{Conv}& 3\times 3& 128\end{array}\right]$
Decoder	$\left[\begin{array}{c}\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 32\end{array}\\ \mathrm{Dropout}\\ \mathrm{Sigmoid}\\ \mathrm{Re}\mathrm{LU}\end{array}\right]$	$\left[\begin{array}{c}\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 32\\ \mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 3\end{array}\\ \mathrm{Dropout}\\ \mathrm{Sigmoid}\\ \mathrm{Re}\mathrm{LU}\end{array}\right]$	$\left[\begin{array}{c}\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 64\\ \mathrm{Conv}& 3\times 3& 32\\ \mathrm{Conv}& 3\times 3& 8\\ \mathrm{Conv}& 3\times 3& 3\end{array}\\ \mathrm{Dropout}\\ \mathrm{Sigmoid}\\ \mathrm{Re}\mathrm{LU}\end{array}\right]$	$\left[\begin{array}{c}\begin{array}{ccc}\mathrm{Conv}& 3\times 3& 96\\ \mathrm{Conv}& 3\times 3& 64\\ \mathrm{Conv}& 3\times 3& 32\\ \mathrm{Conv}& 3\times 3& 3\end{array}\\ \mathrm{Dropout}\\ \mathrm{Sigmoid}\\ \mathrm{Re}\mathrm{LU}\end{array}\right]$
mIoU	73.9	73.2	73.2	73.8
mAcc	81.4	82.1	82.0	81.8

). We then highlight the visualizations of the prompt module (Figure 8 and Figure 9), show the influence of augmentation (

Table 8. Impact of various data augmentation methods on TTA performance (evaluated by mIoU).

	Augmentation					TTA
Configuation	M-S 1	R.H 2	R.B 3	R.S 4	R.C 5	Fog	Night	Rain	Snow
	✓					70.9	41.2	62.6	59.8
		✓				71.4	41.9	63.0	61.1
			✓			71.2	42.3	62.8	60.3
				✓		71.5	42.1	63.1	61.5
					✓	71.8	42.0	62.9	60.7
	✓	✓	✓	✓	✓	73.2	44.9	66.4	63.2

¹ multi-scale resizing. ² random hue. ³ random brightness. ⁴ random saturation. ⁵ random contrast.

and Figure 10) and explain the reasons for not using the existing large-scale segmentation model (Figure 11 and

Table 9. Comparison of performance and running speed between SegFormer-based models and SAM-based models.

	SegFormer-Based	SAM-PT 1	SAM-FT 2
Memory (Bs = 1, GB)	6–8	15–20
Speed (FPS)	6–7	1–2
mIoU (City|40,000 3)	80.2	18.3	16.4

¹ prompt tune SAM by VPT [39]. ² fine-tune SAM by traditional ways. ³ after training 40,000 iterations on Cityscapes [43] dataset.

). Finally, we show some drawbacks of our methods (Figure 12) and analyze the possible reasons. All acronyms are shown in Nomenclature section.

4.4. Implementation Details

We followed the basic implementation details in [32,64] to set up our semantic segmentation TTA experiment based on a segmentation framework [71]. Specifically, we used the SegFormer-B5 [57] model pretrained on Cityscapes [43] as our source model. In terms of data, we down-sampled the 1920 × 1080 images to 960 × 540 as input. We also used the Adam optimizer [72] with (${\beta }_1$, ${\beta }_2$) = (0.9, 0.999), set the learning rate to $3\times {10}^{-4}$ and set the batch size to 1. During the TTA process, we employed the same test-time augmentation strategy as SVDP [64]. The prompt module is designed with an Encoder–Decoder architecture consisting of eight convolutional layers (will be explained in detail in Section 5.2), producing neural prompt points with three channels as output. We set the hyper-parameters: EMA update coefficient $\alpha =0.99$ (in Equation (6)), randomly dropout probability $p=0.5$ (in Equation (3)) and high entropy threshold $\tau =0.85$ (in Equation (4)). The ablation study will be shown in Section 5.3. All experiments were conducted on NVIDIA Tesla V100 GPUs.

4.5. Performance Comparison

4.5.1. Cityscapes-to-ACDC TTA

We evaluate our TTA method on the four individual adverse-condition subdatasets of ACDC [9], as shown in Table 2. The original ERFNet [67], IAL [68], MLFNet [69], EIE [70] and SegFormer [57] were only trained on the source domain and exhibited its inherent generalization across the four scenarios. The above algorithms, which are already suitable for intelligent vehicles, show a performance decline in each of these adverse scenarios. TENT [28] employed a relatively simple domain alignment approach, leading to some degree of under-fitting. CoTTA [32] achieved performance improvement due to the strongly augmented contrast learning during testing. Compared with CoTTA, our method achieved a 2.3% and 1.9% increase in mIoU and mAcc under Fog weather, indicating our method’s stronger ability to handle the domain gap. In night, rain, and snow conditions, our method also outperformed CoTTA by 3.7%, 3.8% and 3.4%, respectively. Although SDVP [64] is a formidable competitor in this study, it had a slight deficiency in the night scene due to the entropy distribution not being as accurate as in the day scene, especially sky and vegetable categories.

4.5.2. Citsycapes-to-(Foggy and Rainy) Citsycapes TTA

We also evaluate our method on more challenging Foggy and Rainy Cityscapes [7], as shown in Table 5. We no longer compare ERFNet [67] and similar algorithms; only the strongest SegFormer is represented. We observe a similar trend to Cityscapes-to-ACDC TTA. Our approach significantly outperforms the baseline method CoTTA by 4.1% and 3.1% in Foggy and Rainy scenes, respectively, demonstrating its ability to generalize well in rainy and snowy scenes.

4.5.3. Cityscapes-to-ACDC CTTA

Finally, we repeate the four adverse-condition scenarios of ACDC multiple times to simulate continuous change and evaluate the forgetting resistance, as shown in Table 3. In the three rounds, “Source” freezes the original ERFNet [67], IAL [68], MLFNet [69], EIE [70] and SegFormer [57] pretrained weight, so the performance remained consistent. TENT [28] and DePT [41] showed noticeable degradation because their parameters were continuously influenced and perturbed by the new data. It is worth noting that the structure of DePT [41] is not very suitable for the dense prediction task of TTA because the memory pool cannot meet the requirements. With the benefit of stochastic parameter restoration, CoTTA [32] maintained stable performance without obvious degradation. VDP [40], which completely froze the original model and only modified the prompt, also maintained good performance. However, it was not specifically designed for dense prediction tasks, so its performance was not as good as CoTTA. SVDP [64] has good performance by sparse prompts, but it also experienced obvious degradation (such as “fog”, mIoU from 72.5 to 72.0). On one hand, our method improved by 5.6% over the CoTTA baseline during the three adaptation rounds. On the other hand, our method has surpassed SVDP comprehensively (including night scenes) from the third adaptation round, demonstrating its good adaptability and stability.

5. Discussion

5.1. Effectiveness of Each Component

In Table 6, we design six experiments to explore the contributions of each component. They are conducted in the Cityscapes-to-ACDC (fog) TTA scenario. TS, USP, DPU, and P-Drop represent the adoption of teacher–student architecture [34], uncertainty-based prompt, domain prompt updating, and prompt dropout, respectively. $E{x}_1$ does not employ any TTA strategy, meaning it uses the pretrained model of SegFormer. $E{x}_2$ utilizes the teacher–student mode, which refers to weight-averaged teacher [34]. $E{x}_3$ adds the uncertainty-based prompt on top of $E{x}_2$, and $E{x}_4$ includes domain prompt updating. $E{x}_5$ randomly applies dropout to USP. $E{x}_6$ represents the complete configuration. It can be seen that after the adoption of USP, the model’s performance improves the most, reaching up to 3.0%, compared to using only TS. Furthermore, the model’s performance increases by 0.5% and 0.1% when DPU and P-Drop are employed. This indicates that USP, as the core innovation, effectively utilizes uncertainty to guide the model toward better generalization. Since the ACDC [9] dataset exhibits a relatively uniform distribution with smooth change, the uncertainty within the same subset fluctuates within a certain range. Therefore, the knowledge provided by the previous prompt module has limited impact. After all configurations are added, no interference between modules happened, resulting in optimal performance.

5.2. How Does Prompt Architecture Affect the Performance?

We have explored the effect of prompt’s architecture. The experiments were conducted on the Cityscapes-to-ACDC (fog) TTA dataset. We have also observed similar phenomena in other scenes. The prompt module mainly includes convolutional layers and is divided into four architectures: small, medium, large, and huge, based on the number of parameters. The meanings in Table 7 represent the size of the convolutional kernel and the number of output channels. The prompt module will process the entropy map and target image to output three-channel images that can be compatible to input. As shown in Table 7, the best results are achieved by Prompt_M, which only contains 0.04 M parameters, indicating that prompt generation can be accomplished using parameters by adding 1–2%. The table also demonstrates that more parameters do not necessarily lead to better performance. Excessive parameters may result in over-fitting to the image information. Additionally, due to the weak gradient propagation, it is not advisable to use a deep architecture for the prompt module.

5.3. How Do the Hyper-Parameters Affect the Performance

In our USP framework, there are two hyper-parameters, which are included in Equations (3) and (4): the probability p for randomly dropout prompts and the probability $\tau$ for retaining prompt features. Both these parameters have a certain peak-like property, meaning that being set too small will affect the regularization, while be set too large can render the prompt module almost ineffective. We conducted experiments on the Cityscapes-to-ACDC (night) dataset, as shown in Figure 6 and Figure 7, to explore the effects of these two hyper-parameters. For p, the range is $(0,1)$. For $\tau$, we considered $(0.5,1)$ as a reasonable range based on statistical analysis of tensor values. Firstly, we fix $\tau$ at 0.75 and test different p values. We found that $p=0.5$ yielded the best results. Similarly, by fixing $p=0.5$ and testing different $\tau$ values, we found that $\tau =0.85$ yielded the best results. When $p\to 0$ or $p\to 1$, the performance of the model tends to degrade toward the baseline CoTTA [32].

5.4. The Visual Effect of Prompt

In Figure 8, we illustrate the visual effect of the prompt. Since the prompt module outputs a three-channel image that is difficult to distinguish after merging with the original image, we applied some enhancement methods for visual representation. Here, the visual domain prompt mostly preserves the distribution of entropy, placing some cues in regions with higher uncertainty. However, the prompt module also produced some different interpretations, such as the parts framed by the yellow boxes in the Figure 8. We believe that these different outputs are the combined integration of entropy and image information, which will help to produce more comprehensive judgement. For example, in the foggy scene (1st row in Figure 8), the entropy is higher in the sky area, but the prompt module considers it more important to focus on the ground after learning. Similarly, in the rainy image of the second row, the entropy is high in the lawn area, but the prompt module deems it unnecessary to pay much attention there, whereas in the snowy image of the fourth row, the prompt module considers the nearby wall surface to be more critical. Thus, the added prompt tries to introduce regularization to specific areas (with a larger domain gap). In SVDP [64], this regularization is more sparse, resulting in there not being enough cues. In VDP [40], the cues are too dense with their placement, and their meaning is unclear. Our proposed USP applies a continuous and dense prompt, allowing the model to make smoother predictions with considering contextual information. On the other hand, the role of the prompt is somewhat similar to masked image modeling [58]. The teacher receives the complete target image as input; hence, the generated pseudo-labels can be considered complete. By adding the prompt, the image is perturbed (like being masked), and then the student network needs to learn how to “restore” the original information, thereby enhancing robustness.

5.5. Attention and Specific Performance Affected by USP

To further explore the impact of entropy self-prompting, we added a decoding head to the segmentor for collecting attention information. As is shown in Figure 9, we test two models (one that had undergone USP and one without USP) and display their predicted entropy with attention values. The higher the attention value, the deeper the color. We can see that the model without USP training outputs higher prediction entropy and experiences more dramatic attention changes, reflecting the model’s sensitivity to textures, shapes, and other information (which are easily affected by the domain gap) in the image. On the other hand, the model has lower entropy with the attention being more evenly distributed after the USP training, indicating the model has good adaptability to various parts, thereby suggesting better generalization.

In Table 4, we present the recognition performance of models trained with and without USP across four different scenarios for each category. Corresponding to the TTA section in Table 2, w/o USP refers to the baseline method COTTA [32]. We found that in foggy conditions, the “road” and “train” categories achieved accuracy rates of 93.5% and 88.9%, respectively, which are 10.5% and 8.2% higher than the baseline COTTA on the “Bike” and “Fence” categories. In the night scene, the improvement was slightly less, with the “Building”, “Terrain”, and “Truck” categories reaching 79.2%, 41.2%, and 26.4%, respectively. Some declines appears, indicating that nighttime poses significant interference, which is also reflected in other TTA methods. In the rainy condition, our method effectively overcomes the presence of water on the ground, achieving recognition rates of 88.2% for the “road” category, 90.4% for the “building” category, and 60.3% for the “wall” category, which confirms the effectiveness in adjusting the model’s attention. In the snowy condition, our method also achieved improvements in some difficult categories (such as “train” and “bus”). Therefore, our approach is adaptable to challenging categories in the test scenarios.

5.6. The Impact of Augmentation

We use five data augmentation techniques to generate test images at different scales for consistency learning in Section 3.5, which assists the model to better adapt the domain gap. The first method is multi-scale image resizing, where we set six image scaling ratios of $[0.5,0.75,1.0,1.25,1.5,1.75]$. At each scale, we apply four types of transformation with a probability of 0.5, namely random brightness, random contrast, random saturation, and random hue. In Figure 10, we demonstrate some augmentation examples. It is shown that some augmentation can even simulate domain gaps, which may effectively help the model’s learning.

In Table 8, we analyzed the effects of different data augmentation methods. Permuting and combining five different data augmentation methods is excessive and unnecessary. Thus, we have only conducted validations on the individual effects and their combined results. The original CoTTA [32] only employed multi-scale scaling. We achieve improvements by applying all the different augmentation schemes, indicating that data augmentation methods indeed help the model achieve better adaptability.

5.7. The Case against Prompt-Tuning Large Models

Nowadays, large-scale models for computer vision (CV) are emerging one after another. The prominent models in the segmentation field are the Segment Anything Model (SAM) [37] and Painter [38] (or SegGPT [73]). SAM benefits from its extremely large training dataset SA-1B, which leads to astonishing segmentation performance. Painter, on the other hand, effectively utilizes in-context learning [36] and realizes high-level tasks (including semantic segmentation, instance segmentation, depth estimation, and pose estimation) as well as low-level tasks (rain removal, denoising, low-light enhancement) within one model. Painter demonstrates the potential of in-context learning, and its task prompt provides valuable insights for unifying different CV tasks in the future.

However, SAM’s segmentation results lack semantic information, and its performance is not strong enough in adverse-condition environments [74]. Painter [38] suffers from severe over-fitting. We attempted to use the semantic segmentation labels from the ACDC dataset as the task prompt, but Painter produced an output for the depth estimation instead. Therefore, it is challenging for large-scale models to surpass those specialized domain models in a short period. Moreover, transferring large-scale models to proprietary task label systems is both time consuming and difficult. Taking SAM as an example, it is trained using the ViT [75] series framework (because ViT is naturally suitable for pretraining in masked image modeling [58]). We tried both fine-tuning and prompt tuning [39] for adapting to Citsycapes, but it was difficult (it only reached 18% mIoU after 40,000 iterations, while SegFormer [57] easily achieves 75%+), and SAM’s fine-tuning was easily bottlenecked due to the limited volume of Cityscapes. This indicates that the built-in segmentation knowledge in SAM does not help the model smoothly migrate to the current high-level domain. On the other hand, ViT-based models are inferior to SegFormer in terms of speed and memory consumption (as shown in Table 9). SegFormer only requires 6–8 GB of memory for testing, while ViT-based models require 15–20 GB (batch_size = 1). Additionally, SegFormer-based models significantly outperform ViT-based models in testing speed (6–7 FPS vs. 1–2 FPS). Therefore, we have decided to use a SegFormer-based model instead of fine-tuning (or prompt tuning) a large-scale model.

5.8. Limitations

Although the USP method proposed in this paper achieves good performance in the adaptation task during testing, there are also some failure cases. The most typical example of them comes in the dark. As shown in Figure 12, we give the test results from the first round to the third round of adaptation. It can be seen that the test results of these three rounds gradually deteriorate, and the overall entropy distribution becomes worse and worse. This is because the segmentation of the night scene has already brought higher entropy even in the initial state (T = 0), while USP is directly dependent on entropy to explore the domain gap, which leads to the inevitable error accumulation in the exploration process of the model. This problem is also due to the inherent difficulty of the semantic segmentation during nighttime, needing to be further processed.

6. Conclusions

We propose a test-time adaptation framework based on uncertainty, named USP, which utilizes a teacher–student paradigm along with separating the consistency learning process to achieve better robustness. USP employs entropy as a measure of uncertainty to capture areas within consecutive images that exhibit domain gaps, and then self-prompts the model, guiding the latter to explore new knowledge. In the prompt process, we designed a prompt module and a random dropout strategy to mitigate model over-fitting and catastrophic forgetting. To capture temporal information, we also devised an uncertainty-based prompt module updating method that relies on total entropy.

Experimental results demonstrate that the prompt module can effectively integrate information from the entropy and raw images, making reasonable adjustments to the model’s attention. Our approach outperforms current TTA algorithms on mainstream datasets like ACDC and Foggy and Rainy Cityscapes. Ablation studies confirm that the teacher–student module, uncertainty self-prompt module, domain prompt update strategy, and random dropout strategy all contribute to the model’s robustness. We further explored the hyper-parameters of dropout probability and threshold through ablation experiments to ascertain their optimal values. In addition, we analyzed the role of test-time data augmentation strategies, showing that data augmentation could simulate real domain discrepancies to some extent. Lastly, we compared our method with currently popular large-scale pretrained semantic segmentation networks, elucidating the reasons for not fine-tuning these models. Our method is capable of mining information from unlabeled data and combating forgetting, exhibiting good robustness and adaptability. In the future research, we will pay more attention to the knowledge related to large models and better utilize prompt techniques in TTA.

There are still some limitations in this study. Our research is still within the TTA framework, requiring complex test-time augmentation operation, which greatly slows down the running speed. Our method is also less effective when dealing with scenes with high initial entropy, such as night. In future studies, we will try to improve the running speed of the algorithm.