Exploring Universal Domain Adaptation with CLIP Models: A Calibration Method

Abstract

CLIP models have shown their impressive learning and transfer capabilities in a wide range of visual tasks. It is, however, interesting that these foundation models have not been fully explored for Universal Domain Adaptation (UniDA). In this paper, we make comprehensive empirical studies of state-of-the-art UniDA methods using these foundation models. We first demonstrate that although the foundation models greatly improve the performance of the baseline method (which trains the models on the source data alone), existing UniDA methods struggle to improve over the baseline. This suggests that new research efforts are necessary for UniDA using these foundation models. Finally, we observe that calibration of CLIP models plays a key role in UniDA. To this end, we propose a very simple calibration method via automatic temperature scaling, which significantly enhances the baseline’s out-of-class detection capability. We show that a single learned temperature outperforms previous approaches in most benchmark tasks when adapting from CLIP models, excelling in evaluation metrics including H-score and a newly proposed Universal Classification Rate (UCR) metric. We hope that our investigation and the proposed simple framework can serve as a strong baseline to facilitate future studies in this field.

1. Introduction

A foundational goal of machine vision is to develop a model that can be applied to data from different distributions. With the emergence of many large-scale pre-trained models such as CLIP [1], ALIGN [2], and DINOv2 [3], significant progress has been made recently toward achieving this goal. These “foundation models” [4] often exhibit significantly greater robustness to various benchmark distribution shifts compared to standardly trained models. For instance, in image classification, while both CLIP and a standard ImageNet-pretrained model attain an accuracy of 76% on ImageNet, CLIP exhibits notable enhancements, with an accuracy increase of 6% on ImageNetV2 and 35% on ImageNet Sketch [1]. Due to the powerful capabilities of these foundation models, techniques for applying them to downstream applications are increasingly important. Indeed, the research community has spent significant effort over the past few years on improving the fine-tuning of these models for various downstream tasks, including few-shot classification [5], out-of-distribution (OOD) detection [6], and OOD generalization [7,8], among others.

Interestingly, Universal Domain Adaptation (UniDA) [9]—a practical setting that aims to adapt a model to a specific target domain without restrictions on the label sets—has not been thoroughly explored to date using powerful foundation models. This paper aims to fill this gap by initially assessing the performance of state-of-the-art UniDA methods when applied to foundation models such as CLIP. Through comprehensive experiments, we present several interesting findings. As expected, all methods achieved substantial improvements over their prior results that relied on ImageNet pre-trained models. However, the performance gap between the Source-Only (SO) baseline and the state-of-the-art (SOTA) methods has notably narrowed, rendering them largely comparable across various benchmark tasks. These findings suggest that new research efforts are necessary for UniDA in the context of foundation models.

Finally, we recognize the pivotal role that the calibration of CLIP foundation models plays in UniDA. To this end, we introduce a simple calibration method by learning a single temperature parameter using source data. Our method, which requires no additional model training, significantly enhances the baseline’s capability to detect out-of-class instances. Despite its simplicity, our method demonstrates exceptional robustness and efficacy across various task scenarios, including open-partial, open, closed, and partial UniDA settings. It excels according to both the established H-score metric and the novel UCR metric. We hope that this straightforward approach can serve as a solid baseline for future research on UniDA using CLIP models.

The main contributions of this paper are summarized as follows:

• To the best of our knowledge, we are the first to tackle the UniDA problem and conduct a comprehensive study of existing methods when applied to CLIP models. Our findings underscore the urgent need for further research in UniDA using these powerful foundation models.
• We propose a straightforward calibration method for UniDA, establishing a new baseline for adaptation from CLIP models. Our approach incorporates a self-calibration technique based on automatic temperature scaling, making it parameter-free and robust across various task scenarios.
• We introduce a novel evaluation metric for UniDA, the Universal Classification Rate (UCR), which is insensitive to thresholds and class ratios. Additionally, to facilitate rigorous and replicable experimentation in UniDA, we have developed and made publicly available the UniOOD framework. UniOOD simplifies the incorporation of new datasets and algorithms with only a few lines of code, thereby ensuring fairer comparisons between different methods.

2. Related Works

Domain Adaptation (DA). A cornerstone of domain adaptation is marginal distribution alignment, commonly achieved through explicit discrepancy measures (e.g., MMD [10], covariance alignment [11]) or adversarial training, as exemplified by the Domain Adversarial Neural Network (DANN) [12]. Subsequent adversarial methods have sought to achieve finer conditional alignment by incorporating techniques like pseudo-labeling [13] or dual-classifier discriminators [14,15]. More recently, Adversarial Self-Training (AST) [16] was introduced, demonstrating notable gains in both adversarial and clean accuracy for gradual DA tasks. The success of Transformers in various tasks has also inspired their use in DA. For instance, the WinWin Transformer (WinTR) [17] exploits domain-specific and invariant knowledge through target pseudo-label refinement and contrastive learning. CDTrans [18] employs a triple-branch architecture with a cross-attention mechanism to align cross-domain image pairs via patch similarity. Building on an adversarial framework, SSRT [19] utilizes perturbed target predictions for safe self-refinement. Recently, DoT [20] further learned locality consistency and reduced generalization error through automated domain-level attention. Despite these advances, most existing DA methods are designed for and evaluated in the closed-set category setting.

Universal Domain Adaptation. Different from the traditional DA problem, which assumes all labels in the target domain are identical to the source domain, UniDA [9] assumes that there is no prior knowledge about the label relationship between source and target domains. Due to the existence of labels shift in UniDA, classical DA methods of adversarial adaptation such as DANN [12] often suffer from negative transfer. To address this problem, UAN [9] and CMU [21] use sample-level uncertainty criteria to assign weights for each sample before adversarial alignment. In addition to adversarial adaptation, self-training or self-supervised-based methods usually have better performance due to the exploiting of discriminative representation in the target domain. Among these, DANCE [22] uses self-supervised neighborhood clustering to learn the target data structure; DCC [23] exploits cross-domain consensus knowledge to discover discriminative clusters of both domains; MATHS [24] designs a contrastive learning scheme to nearest neighbors for feature alignment; OVANet [25] proposes to train a one-vs-all classifier for each class and applies entropy minimization to target samples during adaptation; and more recently, UniOT [26] uses optimal transport criteria to select more confident clusters to target samples for self-training. However, all of these methods are evaluated solely using models pre-trained in ImageNet. In this paper, we compare against the most state-of-the-art methods under the foundation models such as CLIP. We show that there exists a strong baseline that can be competitive with or outperform the more complex methods listed above when using these foundation models.

Domain Generalization (DG). Unlike DA, DG assumes no access to target domain data during training. To enhance model robustness, DG methods typically employ strategies such as data augmentation at the image level [27] or feature level [28], remove spurious correlations via stable learning [29], or leverage the inherent inductive bias of neural networks [30,31]. However, many of these methods are empirically driven and their effectiveness is often validated on specific benchmarks. Recent empirical studies [32,33] have raised concerns about the actual efficacy of numerous DG algorithms, suggesting that current benchmark-based evaluations may be inadequate. In contrast to typical DG approaches, our work deliberately refrains from applying any specialized DG techniques (e.g., data augmentation). This design choice allows us to isolate and investigate the true performance of UniDA methods when integrated with CLIP models.

Adaptation of CLIP Models. The exceptional performance of foundation models in traditional vision tasks has led to a growing interest in developing more effective adaptive methods. In addition to adopting linear probing [34], full fine-tuning [35], or zero shot [1] to the backbone models, many new strategies or methods have been proposed. For example, prompt learning based methods [36,37,38] propose to learn better prompts under the language-vision models. CLIP-Adapter [39] and Tip-Adapter [40] are going to construct additional light models for efficient fine-tuning while freezing the backbone models. Surgical fine-tuning [8] suggests selective fine-tuning of a subset of layers based on different types of distribution shift. WiSE-FT [7] proposes to enhance the model robustness by integrating the zero-shot model and the fine-tuning model. And more recently, cross-model adaptation [5] shows the most powerful few-shot ability to CLIP based models by incorporating multi-modalities as training samples for ensemble training. In this paper, different from all these methods that aim to adapt models for closed-set classification task, we exploit to adapt for UniDA problem. We also show how effective would be if these representative methods are directly applied for the UniDA tasks.

Related Subfields. UniDA is also closely related to Open-Set Recognition (OSR) [41] and OOD detection [42]. OSR extends the closed-set classification to a more realistic open-set classification, where test samples may come from domains of unknown classes. This setting is very similar to UniDA but it assumes that there exists no domain shift and that one can not access the target domain during training. OOD detection, on the other hand, focuses on detecting the out-of-class samples only. In theory, a recent work of [43] unifies OSR and OOD detection into the same framework and shows that the loss criterion must be carefully designed otherwise it may face an intractable learning problem. In this paper, we are inspired by these works and introduce a similar evaluation metric of UCR for UniDA. Although many methods of OSR and OOD detection have been proposed during the past few years, a recent empirical study by Vaze et al. [44] shows that a good closed-set classifier can be competitive with or even superior to previous complex methods. These findings align with our results on UniDA under the foundation models.

3. Problem Formulation

In UniDA, we are provided with a source domain dataset ${\mathcal{D}}^s={\left\{({\mathbf{x}}_i^s,y_i^s)\right\}}_{i=1}^{n_s}$ consisting of $n_s$ samples, where the i-th sample ${\mathbf{x}}_i^s\in {\mathbb{R}}^d$ is a d dimensional vector and $y_i^s\in {\mathcal{Y}}^s$ is the associated label. Additionally, we have a target domain dataset ${\mathcal{D}}^t={\left\{\left({\mathbf{x}}_i^t\right)\right\}}_{i=1}^{n_t}$, which contains $n_t$ unlabeled samples from the same d-dimensional space. Samples in the source and target domains are drawn from their respective distributions, ${\mathcal{D}}^s\sim {\pi }_s(X^s,Y^s)$ and ${\mathcal{D}}^t\sim {\pi }_t(X^t,Y^t)$. We represent the collection of labels in the source domain as ${\mathcal{Y}}^s$ and in the target domain as ${\mathcal{Y}}^t$. Let ${\mathcal{Y}}^{st}={\mathcal{Y}}^s\cap {\mathcal{Y}}^t$ be the domain-shared label set and ${\mathcal{Y}}^{t/s}={\mathcal{Y}}^t\setminus {\mathcal{Y}}^s$ be the target-private label set. Similarly, ${\mathcal{Y}}^{s/t}$ is the set of source-private labels. In UniDA, we make no assumptions about ${\mathcal{Y}}^t$. Hence, ${\mathcal{Y}}^{st}$ and ${\mathcal{Y}}^{t/s}$ are also unknown. For convenience, we refer to target samples belonging to ${\mathcal{Y}}^{st}$ (known classes) as in-of-class samples ${\mathcal{D}}_{in}^t$ and those belonging to ${\mathcal{Y}}^{t/s}$ (unknown classes) as out-of-class samples ${\mathcal{D}}_{out}^t$.

The learning task of UniDA can be converted as two subtasks of in-of-class discrimination and out-of-class detection. Such objectives could be implemented by a unified framework as: (1) learning a scoring function $s:{\mathbb{R}}^d\to \mathbb{R}$ for out-of-class detection and (2) learning a classifier $f:{\mathbb{R}}^d\to {\mathbb{R}}^{|{\mathcal{Y}}^s|}$ for in-of-class discrimination. The scoring function s assigns a score to each sample, which reflects the uncertainty level regarding it being an out-of-class sample. A higher score indicates a higher likelihood of belonging to the in-of-class category. UniDA methods require a threshold value for the scoring function s to distinguish between out-of-class and in-of-class samples. This threshold can either be learned automatically or set manually.

Typically, the learning classifier $f=h\circ \varphi$ comprises a feature extractor $\varphi$ and a classifier head h. Prior research in UniDA primarily focuses on fine-tuning $\varphi$ using ImageNet pre-trained backbones. In our study, we aim to explore the training of f and the scoring function s using foundation models such as CLIP backbones.

4. Proposed Method

Given a classifier $f:{\mathbb{R}}^d\to {\mathbb{R}}^{|{\mathcal{Y}}^s|}$ before softmax layer, the output probability of each target instance $x^t$ is:

$$p\left(x^t\right)=\sigma (f\left(x^t\right)/\tau ),$$

(1)

where $\sigma$ is the softmax function and $\tau$ is the scaling temperature. In UniDA, since we do not have a validation set and lack prior knowledge about target categories, we do not treat $\tau$ as a fixed hyperparameter but aim to automatically learn such a temperature $\tau$ for each specific UniDA task.

4.1. Motivations

Why scaling the logits? In UniDA tasks, however, the objective is not solely closed-set classification but also demands effective out-of-class detection. We argue that a well-calibrated model plays a pivotal role in achieving this goal. To gain a clearer perspective on our argument, we illustrate two reliability diagrams [45] comparing the CLIP zero-shot model before and after calibration in Figure 1. As depicted in the figure, without logit scaling, the CLIP zero-shot method tends to classify most samples with low confidence, even if they are classified correctly. This would readily lead to misidentifying the majority of in-of-class samples as out-of-class ones, causing a decrease in in-of-class classification performance. After calibration through temperature scaling, significantly improved confidence estimates can be observed, resulting in a more trustworthy prediction system. Therefore, we scale the logits to ensure the model’s proper calibration and enhance its performance in both out-of-class detection and in-of-class discrimination [46].

4.2. Learning Temperature Scaling by Source Confidence Calibration

Different from previous calibration methods [46,47,48] that were proposed for closed-set classification, confidence calibration by temperature scaling faces a challenge for UniDA tasks since we do not have prior knowledge about the target categories. To address this challenge, we propose to learn using the source data. We evenly divide the source data into two parts by class. The first part of the samples is treated as in-of-class samples for IID (In-Identical-Distribution) calibration, while the second part of the samples is treated as out-of-class samples for OOD calibration.

IID calibration. Given a ground truth joint distribution ${\pi }_{in}(X,Y)={\pi }_{in}\left(Y\right|X){\pi }_{in}\left(X\right)$, the expected calibration error (ECE) for a prediction model is defined as

$${\mathbb{E}}_{\widehat{P}}\left[\right|\mathbb{P}(\widehat{Y}=Y|\widehat{P}=p)-p\left|\right],$$

(2)

where $\widehat{Y}$ is a class prediction and $\widehat{P}$ is its associated confidence, i.e., probability of correctness. ECE could be approximated by partitioning predictions into K equally-spaced bins (similar to the reliability diagrams, see Figure 1) and taking a weight average of the bins’ accuracy/confidence difference [47]. Specifically,

$${\mathrm{ECE}}_{in}=\sum _{k=1}^K{\displaystyle \frac{|B_k|}{n_{in}}}|\mathrm{acc}\left(B_k\right)-\mathrm{conf}\left(B_k\right)|,$$

(3)

where $n_{in}$ is the number of in-of-class samples.

OOD calibration. For out-of-class samples ${\left\{x_i\right\}}_{i=1}^{n_{out}}$, which do not belong to any specific predicted category, our objective is to maintain a uniform distribution of their output class probabilities:

$${\mathrm{ECE}}_{out}={\displaystyle \frac{1}{n_{out}}}\sum _{i=1}^{n_{out}}|\mathrm{conf}\left(x_i\right)-{\displaystyle \frac{1}{N}}|,$$

(4)

where N is the number of in-of-class categories.

Negative log likelihood. As a standard measure of probabilistic model’s quality [49], negative log likelihood (NLL) is widely used in the context of deep learning, which is also known as the cross entropy loss. Given the known ground truth of in-of-class samples and a prediction probabilistic model ${\widehat{\pi }}_{in}\left(Y\right|X)$, NLL is defined as:

$${\mathrm{NLL}}_{in}=-\sum _{i=1}^{n_{in}}log\left({\widehat{\pi }}_{in}\left(y_i\right|x_i)\right)$$

(5)

${\mathrm{NLL}}_{in}$ is minimized if and only if ${\widehat{\pi }}_{in}\left(Y\right|X)$ recovers the ground truth conditional distribution ${\pi }_{in}\left(Y\right|X)$.

Our overall objective of learning temperature scaling is then written as

$${\tau }_{opt}=arg\underset{\tau }{min}{\mathrm{ECE}}_{in}+{\mathrm{ECE}}_{out}+{\mathrm{NLL}}_{in}.$$

(6)

5. Evaluation and Discussion

In the context of UniDA, the dual objective includes the task of accurately classifying samples from the correct classes ${\mathcal{Y}}^{st}$ while simultaneously rejecting samples from unknown classes ${\mathcal{Y}}^{t/s}$. This dual objective introduces additional complexity to the evaluation of UniDA methods.

5.1. Hard Out-of-Class Detection Criteria

The H-score metric is introduced with the aim of achieving a balance between the significance of detecting samples outside the class and accurately classifying in-of-class samples, as described in Fu et al.’s work [21]. H-score includes all unknown classes as a superclass and calculates the harmonic mean of the accuracy of the average classes in known classes (acc_in) and the accuracy in the superclass (acc_out), that is, the H-score = 2·acc_in·acc_out/(acc_in + acc_out). While this metric is commonly employed in the UniDA research community, it is important to note that it exhibits a noticeable bias toward the ratios between the quantities of in-of-class and out-of-class samples. Due to the lack of prior knowledge to target data, these ratios may vary in different tasks. It is usually impossible to handle all tasks of different ratios in order to have a fair evaluation between different methods.

It is worth noting that in scenarios where the target data lacks unknown classes, the H-score metric loses its applicability and degenerates into acc_in.

5.2. Soft Out-of-Class Detection Criteria

The criterias mentioned above require us to classify a sample as either out-of-class or in-of-class, which means that we have to set a threshold for out-of-class detection. In this paper, we are motivated from the field of open set recognition (Open Set Classification Rate (OSCR) [50] and the Detection and Identification Rate (DIR) [51]) and introduce a new UniDA evaluation metric, which is threshold- and ratio-free. However, unlike the OSR task, which assumes the absence of source private classes and the presence of target private classes, UniDA is more flexible and does not impose such strict constraints. Therefore, we adapt these metrics to accommodate various UniDA scenarios, introducing a new metric called Universal Classification Rate (UCR).

To calculate UCR, we compute a pair of Correct Classification Rate (CCR) and False Positive Rate (FPR) by varying the scoring threshold $\theta$. CCR assesses the proportion of correctly classified in-of-class samples from ${\mathcal{D}}_{in}^t$, and FPR quantifies the fraction of out-of-class samples from ${\mathcal{D}}_{out}^t$ that are incorrectly detected.

$$\begin{array}{cc}\hfill & \mathrm{CCR}\left(\theta \right)={\displaystyle \frac{\left|\right\{\mathbf{x}|\mathbf{x}\in {\mathcal{D}}_{in}^t\wedge f\left(\mathbf{x}\right)=\mathrm{label}\left(\mathbf{x}\right)\wedge s\left(\mathbf{x}\right)>\theta \}|}{|{\mathcal{D}}_{in}^t|}}\hfill \\ \hfill & \mathrm{FPR}\left(\theta \right)={\displaystyle \frac{\left|\right\{\mathbf{x}|\mathbf{x}\in {\mathcal{D}}_{out}^t\wedge s\left(\mathbf{x}\right)>\theta \}|}{|{\mathcal{D}}_{out}^t|}}.\hfill \end{array}$$

(7)

Then, the UCR is calculated as

$$\mathrm{UCR}=\left\{\begin{array}{c}\mathrm{Area}\mathrm{Under}\mathrm{the}(\mathrm{CCR}\mathrm{vs}\mathrm{FPR})\mathrm{Curve},\mathrm{if}|{\mathcal{D}}_{out}^t|>0\hfill \\ \mathrm{CCR}(-\infty ),\mathrm{if}|{\mathcal{D}}_{out}^t|=0\hfill \end{array}\right.$$

(8)

where, CCR($-\infty$) is identical to the closed-set classification accuracy on ${\mathcal{D}}_{in}^t$. Figure 2 shows an example illustration to the (CCR vs FPR) curve on VisDA task under the (6/3) setting. The distinction between UCR and AUROC lies in the replacement of the true positive rate (TPR) with the correct classification rate (CCR) in UCR. In contrast to previous evaluation metrics, UCR does not rely on thresholds or ratios, making it an additional criterion that does not consider the threshold effects.

6. Empirical Results

6.1. UniDA Methods Review

We briefly review some representative state-of-the-art (SOTA) methods for comparison: DANCE [22], OVANet [25], and UniOT [26]. In addition to these methods, we consider the Source Only (SO) method as a baseline that involves standard cross-entropy loss training on the source data alone. In inference, the softmax classifier f is employed for predictions, and the scoring function s is constructed based on the entropy of the softmax output probabilities, following the approach used in DANCE. For a concise overview of these methods, see

Table 1. A brief introduction of different methods. Previous approaches are categorized into three groups: SO, the baseline method that trains models solely on source data; DANCE [22], OVANet [25], and UniOT [26], state-of-the-art methods designed explicitly for the UniDA task; and WiSE-FT [7], CLIP cross-model [5], and CLIP zero-shot [1], three SOTA methods for CLIP model adaptation. In this paper, we introduce UniDA calibration, which is different from previous calibration method [48] that is designed for closed-set classification, as detailed in Section 4.

Method Type	Methods	Source	Target	Classifier	Scoring Rule	Threshold Value
Baseline	Source Only (SO)	✓	✗	softmax	negative entropy	$-log\left(|{\mathcal{Y}}^s|\right)/2$
UniDA SOTAs	DANCE [22]	✓	✓	softmax	negative entropy	$-log\left(|{\mathcal{Y}}^s|\right)/2$
	OVANet [25]	✓	✓	softmax	binary softmax prob.	1/2
	UniOT [26]	✓	✓	OT	maximum OT mass	$1/(n^t+T)$
CLIP adaptations	WiSE-FT [7]	✓	✗	softmax	negative entropy	$-log\left(|{\mathcal{Y}}^s|\right)/2$
	CLIP cross-model [5]	✓	✗	softmax	negative entropy	$-log\left(|{\mathcal{Y}}^s|\right)/2$
	CLIP zero-shot [1]	✗	✗	NN	maximum logit	-
Calibration	CLIP calibration [48]	✓ *	✗	softmax	negative entropy	$-log\left(|{\mathcal{Y}}^s|\right)/2$
	Our calibration	✓ *	✗	softmax	negative entropy	$-log\left(|{\mathcal{Y}}^s|\right)/2$

* Note that these methods use the source data for confidence calibration, not for training. The symbols ✓ and ✗ denote the utilization and non-utilization, respectively, of corresponding domain data for model training. “prob.” is an abbreviation for probability.

.

6.2. Datasets and Experimental Setup

Dataset. We train the above methods on the standard benchmark datasets for UniDA: Office [52], OfficeHome (OH) [53], VisDA (VD) [54], and DomainNet (DN) [55]. Office has 31 categories and three domains: Amazon (A), DSLR (D), and Webcam (W). OfficeHome contains 65 categories and four domains: Art (A), Clipart (C), Product (P), and Real-World (R) images. VisDA is a synthetic-to-real dataset with 12 categories in total. DomainNet is the largest dataset, including 345 categories and six domains, where three domains—Painting (P), Real (R), and Sketch (S)—are used in experiments following previous work [25,26]. For each dataset, we further split the total categories into three disjoint parts—common categories ${\mathcal{Y}}^{st}$, source private categories ${\mathcal{Y}}^{s/t}$, and target private categories ${\mathcal{Y}}^{t/s}$—to consist of the source and target domains. For a more comprehensive study, we assign each dataset with four different class splits: open-partial, open, closed, partial, following [22]. The different classes splits result in different running tasks, and each split setting, denoted ($|{\mathcal{Y}}^{st}|/|{\mathcal{Y}}^{s/t}|$), is shown in each table. Detail information about these four datasets and the class-split settings are presented in the Appendix A.

Implementation. For fair comparison between different methods, we implement UniOOD, a code framework to streamline rigorous and reproducible experiments in UniDA. By using the UniOOD framework, all methods are run under the same learning setting. For all methods requiring training in our experiments, including the SO baseline and others like DANCE and OVANet, we adopted a consistent linear probing protocol. Specifically, the visual backbone (e.g., CLIP) was kept frozen, and we only trained a linear classifier (a single fully-connected layer followed by softmax) on top of the frozen features. The initial learning rate is set to 0.01 for all new layers and 0.001 for pretrained backbone if it is fine-tuned and decays using the cosine schedule rule with a warm-up of 50 iterations. We use the SGD optimizer with momentum 0.9 and the batch size is set to 32 for each domain. The number of training iterations are set to 5000, 10,000, or 20,000 based on the scale of the training data, which is detailed in Appendix A. We report results of the last checkpoint due to the absence of validation data and average them among three random runs. Due to space constraints, we provide the average results for each split setting, while the detailed results for individual tasks can be found in the Appendix C. The hyperparameters for the previous methods follow their official codes. We do not use any data augmentation during training for fair comparison to different methods, which may be different from previous works.

Evaluation metrics. We use H-score and UCR as the evaluation metrics. H-score is a widely used metric for UniDA, and the UCR is introduced in this paper, as detailed in Section 5.

6.3. Comparison with SOTA UniDA Methods

Table 2. Comparison results of H-score between existing methods and the proposed method using the same CLIP backbone (ViT-L) in four UniDA settings (open-partial, open, closed, and partial scenarios respectively). The best results are highlighted in bold, while the second best results are highlighted in blue, and the data indicating significantly poor results ($<\approx$10% to the best) are marked in red.

Methods	Office				OfficeHome				VisDA				DomainNet				Avg
	(10/10)	(10/0)	(31/0)	(10/21)	(10/5)	(15/0)	(65/0)	(25/40)	(6/3)	(6/0)	(12/0)	(6/6)	(150/50)	(150/0)	(345/0)	(150/195)
SO	91.98	91.87	80.22	89.79	84.52	82.05	58.12	58.31	69.85	75.79	55.31	57.19	61.49	65.63	38.27	35.88	68.52
DANCE [22]	94.7	96.09	75.76	66.83	89.01	83.95	55.42	46.63	71.9	74.3	58.08	49.5	60.53	65.24	37.56	30.92	66.03
OVANet [25]	93.36	91.16	74.64	87.53	85.42	80.29	64.65	65.92	59.47	39.27	43.55	42.58	70.7	72.4	57.22	55.86	67.75
UniOT [26]	92.32	96.48	59.95	41.31	89.45	86.64	59.27	43.6	79.1	83.08	71.62	62.03	71.42	73.21	63.72	55.18	70.52
WiSE-FT [7]	82.34	94.07	47.87	53.57	79.37	73.44	13.64	16.56	62.68	72.21	30.05	27.4	3.74	7.92	0.3	0.29	41.59
CLIP cross-model [5]	93.04	93.59	83.21	92.55	86.2	84.26	62.65	63.14	77.69	81.66	62.08	67.98	61.98	67.1	36.2	34.06	71.71
w/o calibration ($\tau =1$)	0.0	0.07	0.0	0.0	0.17	1.31	0.0	0.0	0.0	0.05	0.0	0.0	0.0	0.0	0.0	0.0	0.1
CLIP calibration [48]	35.18	38.72	87.53	96.84	47.02	46.9	87.01	87.1	74.49	46.53	82.2	87.87	58.57	58.78	77.57	78.28	68.16
Our calibration	86.74	91.89	83.8	94.39	86.4	84.77	80.58	80.81	84.74	82.26	76.08	83.12	72.37	74.22	74.93	75.22	82.02

and

Table 3. Comparison results of UCR between existing methods and the proposed method using the same CLIP backbone (ViT-L) in four UniDA settings (open-partial, open, closed, and partial scenarios respectively). The best results are highlighted in bold, while the second best results are highlighted in blue, and the data indicating significantly poor results ($<\approx$10% to the best) are marked in red.

Methods	Office				OfficeHome				VisDA				DomainNet				Avg
	(10/10)	(10/0)	(31/0)	(10/21)	(10/5)	(15/0)	(65/0)	(25/40)	(6/3)	(6/0)	(12/0)	(6/6)	(150/50)	(150/0)	(345/0)	(150/195)
SO	93.98	94.95	91.44	96.99	86.89	84.53	83.55	84.44	63.46	71.17	76.66	79.51	63.19	66.01	71.26	70.78	79.93
DANCE [22]	95.17	97.09	87.69	81.1	90.33	86.76	81.74	75.63	57.78	63.45	67.86	56.4	64.88	68.37	71.66	67.6	75.84
OVANet [25]	95.36	95.8	91.4	96.84	88.18	85.72	83.52	84.37	68.57	66.45	76.68	79.58	64.3	66.94	71.37	70.94	80.38
UniOT [26]	90.62	97.2	92.14	55.94	88.85	85.59	85.5	63.61	72.22	78.8	84.79	74.78	62.88	67.33	73.85	67.99	77.63
WiSE-FT [7]	95.27	96.33	92.3	97.57	90.77	89.28	87.28	88.44	70.83	77.88	81.45	84.43	68.72	71.66	75.74	75.77	83.98
CLIP cross-model [5]	95.38	96.18	93.24	97.58	89.71	87.82	86.95	87.97	73.22	79.06	81.15	83.76	68.81	71.53	75.57	75.55	83.97
CLIP zero-shot [1]	90.1	97.68	87.69	96.61	90.21	89.67	89.08	89.43	78.6	82.86	87.56	88.1	70.78	73.34	79.48	79.87	85.69
w/o calibration ($\tau =1$)	92.46	97.75	87.68	96.61	92.91	91.71	89.08	89.41	80.9	85.75	87.56	88.1	69.2	72.81	79.48	79.88	86.33
CLIP calibration [48]	90.88	97.55	87.69	96.61	92.24	90.59	89.08	89.43	82.8	86.32	87.56	88.1	74.57	76.88	79.49	79.87	86.85
Our calibration	93.39	97.91	87.69	96.61	93.02	91.73	89.08	89.43	82.38	86.37	87.56	88.1	74.86	77.21	79.49	79.87	87.17

present the comparative results between our method and existing state-of-the-art (SOTA) UniDA approaches in two distinct evaluation metrics. The results cover four distinct UniDA settings, namely open-partial, open, closed, and partial settings, denoted as different class splits represented as ($|{\mathcal{Y}}^{st}|/|{\mathcal{Y}}^{s/t}|$) within the table. Regarding the H-score metric, it is evident that our method and the leading state-of-the-art approach are on par with each other in the open-partial and open settings. However, our method demonstrates significant enhancements in all closed and partial settings, achieving improvements that exceed 10% in five out of eight tasks. In terms of the UCR metric, our method significantly outperforms state-of-the-art UniDA methods in three out of four datasets: OfficeHome, VisDA, and DomainNet, across all four settings. In general, our method exhibits superior robustness across various settings and establishes a new state-of-the-art on UniDA benchmarks, excelling in both the H-score metric and the UCR metric.

6.4. Comparison with SOTA CLIP-Adaptation Methods

Recall that our focus is on developing a UniDA method based on foundation models like CLIP. Therefore, we also provide comparisons with some state-of-the-art (SOTA) adaptation methods that leverage CLIP models, even though they were not originally designed for the UniDA task. These methods include CLIP zero-shot (baseline) [1], WiSE-FT [7], and CLIP cross-model [5]. WiSE-FT is a new fine-tuning method to improve robustness by ensembling the weights of the zero-shot and fine-tuned models. CLIP cross-model is a recent study introduced by Lin et al. [5], which has demonstrated the most remarkable few-shot capability to date by leveraging cross-model information. However, as all these methods can not directly be used for UniDA, we construct a scoring function s following the SO method except for CLIP zero-shot, as illustrated in Table 1. While these methods have shown remarkable enhancements in closed-set robustness benchmarks such as ImageNet, they frequently exhibit lower performance than the SOTA UniDA methods when evaluated using the H-score metric on UniDA benchmarks, as shown in Table 2. However, it should be noted that all these adaptation methods consistently outperform the SOTA UniDA methods when considering the UCR metric, as illustrated in Table 3. Our method maintains its position as the most powerful performer in terms of both H-score and UCR evaluation metrics.

The CLIP calibration method [48] that was proposed for closed-set classification, is exactly equal to our IID calibration when using source data as the auxiliary dataset, which is a component of our overall calibration. The results in Table 2 show that while CLIP calibration [48] slightly outperforms our calibration in closed-set/partial scenarios, it performs significantly worse in open-partial/open-set scenarios. Since we do not have prior knowledge about which scenario to consider, our calibration method is more robust and suitable for UniDA tasks.

6.5. Analysis and Ablation Study

Temperature scaling is necessary. To demonstrate the effectiveness of our temperature scaling, we report the results of the CLIP model when setting $\tau =1$, as shown in Table 2. It is apparent that without temperature scaling, CLIP model fails to distinguish samples between in-of-class and out-of-class categories, leading to nearly zero performance on the H-score metric. This demonstrates the necessity of temperature scaling and the superiority of our self-calibration method.

Each calibration loss plays a key role. We conducted ablation studies to assess the significance of each calibration loss component in our method, namely ECE_in, ECE_out, and NLL_in, corresponding to IID calibration, OOD calibration, and NLL calibration, respectively. The results of this analysis are presented in

Table 4. Ablation studies. The best results are highlighted in bold, while data indicating significantly poor results ($<\approx$10% to the best) are marked in red. It exhibits enhanced stability and robustness across diverse task settings when incorporating all calibration losses.

Methods	Office				OfficeHome				VisDA				DomainNet				Avg
	(10/10)	(10/0)	(31/0)	(10/21)	(10/5)	(15/0)	(65/0)	(25/40)	(6/3)	(6/0)	(12/0)	(6/6)	(150/50)	(150/0)	(345/0)	(150/195)
H-score
Only IID calibration	35.18	38.72	87.53	96.84	47.02	46.9	87.01	87.1	74.49	46.53	82.2	87.87	58.57	58.78	77.57	78.28	68.16
Only NLL calibration	36.7	66.24	87.01	96.48	53.41	53.27	87.33	87.38	72.45	45.81	85.52	89.84	58.24	59.21	77.59	78.3	70.92
Only OOD calibration	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
w/o IID calibration	90.11	76.44	77.8	89.3	87.61	84.68	72.46	73.4	82.33	58.11	58.03	62.25	77.38	79.01	70.63	70.17	75.61
w/o NLL calibration	88.63	73.38	75.33	86.84	87.39	84.03	66.58	68.01	78.85	54.6	54.96	58.32	77.38	79.14	67.66	66.66	72.98
w/o OOD calibration	35.36	55.43	87.01	96.48	56.4	56.23	87.37	87.42	72.69	45.53	82.3	87.99	59.59	58.65	77.59	78.28	70.27
Ours	86.74	91.89	83.8	94.39	86.4	84.77	80.58	80.81	84.74	82.26	76.08	83.12	72.37	74.22	74.93	75.22	82.02
UCR
Only IID calibration	90.88	97.55	87.69	96.61	92.24	90.59	89.08	89.43	82.8	86.32	87.56	88.1	74.57	76.88	79.49	79.87	86.85
Only NLL calibration	90.52	97.7	87.69	96.61	92.39	90.78	89.08	89.43	82.83	86.31	87.56	88.1	74.57	76.89	79.49	79.87	86.86
Only OOD calibration	86.98	96.86	87.69	96.61	91.69	90.49	89.08	89.43	78.91	84.55	87.56	88.1	57.83	62.48	79.49	79.87	84.23
w/o IID calibration	93.51	97.88	87.69	96.61	93.07	91.84	89.08	89.43	82.2	86.16	87.56	88.1	74.92	77.26	79.49	79.87	87.17
w/o NLL calibration	93.51	97.88	87.69	96.61	93.08	91.85	89.08	89.43	82.1	86.15	87.56	88.1	74.9	77.25	79.49	79.87	87.16
w/o OOD calibration	90.38	97.65	87.69	96.61	92.45	90.87	89.08	89.43	82.83	86.31	87.56	88.1	74.6	76.87	79.49	79.87	86.86
Ours	93.39	97.91	87.69	96.61	93.02	91.73	89.08	89.43	82.38	86.37	87.56	88.1	74.86	77.21	79.49	79.87	87.17

. It is evident that the absence of either IID or NLL calibration leads to a substantial decrease in performance in the closed and partial settings. Conversely, the lack of OOD calibration affects the results in the open-partial and open settings. In summary, each calibration loss contributes significantly to the calibration process.

Simply setting a fixed temperature scaling is not optimal. To validate the effectiveness of the proposed self-calibration method, we present the results of our approach with fixed temperature scaling using various values, as depicted in

Table 5. Comparison to that of fixed temperature scaling with various values. The best results are highlighted in bold, while data indicating significantly poor results ($<\approx$10% to the best) are marked in red. Setting different values of $\tau$ has little impact to the UCR, but it significantly affects the H-score. The results illustrate that fixing a scaling value is sensitive to various tasks, while our method is robust across diverse settings. This demonstrates the efficacy of the proposed self-calibration method. Given the absence of a validation set in UniDA, our method offers a robust solution for adapting to diverse tasks.

Methods	Office				OfficeHome				VisDA				DomainNet				Avg
	(10/10)	(10/0)	(31/0)	(10/21)	(10/5)	(15/0)	(65/0)	(25/40)	(6/3)	(6/0)	(12/0)	(6/6)	(150/50)	(150/0)	(345/0)	(150/195)
H-score
$\tau =$ 0.1	17.79	27.94	87.53	96.84	20.31	20.3	88.35	88.41	32.13	41.24	89.6	92.06	4.55	6.13	79.34	80.1	54.54
$\tau =$ 0.2	62.28	71.5	87.27	96.69	60.49	60.33	86.75	86.9	67.96	71.96	84.37	89.21	62.49	63.83	77.47	78.17	75.48
$\tau =$ 0.3	83.28	90.45	84.7	95.16	81.09	80.49	80.06	80.24	83.41	86.6	75.01	82.37	77.54	79.58	63.02	61.75	80.3
$\tau =$ 0.4	90.45	95.51	75.13	88.0	88.95	87.09	62.12	64.09	83.03	87.32	62.15	67.33	56.51	63.38	26.64	26.75	70.28
$\tau =$ 0.5	80.04	93.82	50.79	55.82	88.18	84.77	32.85	38.97	68.3	80.4	39.98	40.39	15.82	24.23	2.61	3.04	50.0
$\tau =$ 0.6	59.36	81.49	24.95	25.91	78.42	73.0	7.9	12.97	39.74	63.18	14.31	12.63	1.61	3.1	0.1	0.14	31.18
$\tau =$ 0.7	33.69	64.05	6.76	3.35	57.83	50.18	1.07	2.26	10.95	34.87	3.2	1.47	0.05	0.16	0.0	0.01	16.87
$\tau =$ 0.8	2.07	41.75	0.38	0.0	26.6	22.82	0.13	0.31	1.15	8.61	0.07	0.06	0.01	0.01	0.0	0.0	6.5
$\tau =$ 0.9	0.0	8.68	0.03	0.0	4.64	5.09	0.0	0.02	0.0	1.38	0.0	0.0	0.0	0.0	0.0	0.0	1.24
$\tau =$ 1.0	0.0	0.07	0.0	0.0	0.17	1.31	0.0	0.0	0.0	0.05	0.0	0.0	0.0	0.0	0.0	0.0	0.1
Ours	86.74	91.89	83.8	94.39	86.4	84.77	80.58	80.81	84.74	82.26	76.08	83.12	72.37	74.22	74.93	75.22	82.02
UCR
$\tau =$ 0.1	90.14	97.49	87.69	96.61	91.72	89.85	89.08	89.43	82.98	86.23	87.56	88.1	72.3	74.72	79.49	79.87	86.45
$\tau =$ 0.2	92.22	97.73	87.69	96.61	92.5	90.94	89.08	89.43	82.89	86.61	87.56	88.1	74.67	77.0	79.49	79.87	87.02
$\tau =$ 0.3	93.26	97.86	87.69	96.61	92.92	91.56	89.08	89.43	82.57	86.63	87.56	88.1	74.86	77.23	79.49	79.87	87.17
$\tau =$ 0.4	93.52	97.9	87.69	96.61	93.06	91.8	89.08	89.43	82.23	86.5	87.56	88.1	74.4	76.89	79.49	79.87	87.13
$\tau =$ 0.5	93.47	97.91	87.69	96.61	93.1	91.88	89.08	89.43	81.93	86.34	87.56	88.1	73.7	76.35	79.49	79.87	87.03
$\tau =$ 0.6	93.31	97.89	87.69	96.61	93.08	91.88	89.08	89.43	81.66	86.19	87.56	88.1	72.87	75.7	79.49	79.87	86.9
$\tau =$ 0.7	93.09	97.86	87.69	96.61	93.05	91.86	89.08	89.43	81.43	86.06	87.56	88.1	71.96	74.99	79.49	79.87	86.76
$\tau =$ 0.8	92.91	97.83	87.69	96.61	93.0	91.82	89.08	89.43	81.23	85.94	87.56	88.1	71.03	74.26	79.49	79.87	86.62
$\tau =$ 0.9	92.69	97.79	87.69	96.61	92.96	91.77	89.08	89.43	81.06	85.84	87.56	88.1	70.11	73.53	79.49	79.87	86.47
$\tau =$ 1.0	92.46	97.75	87.69	96.61	92.91	91.71	89.08	89.43	80.9	85.75	87.56	88.1	69.2	72.81	79.49	79.87	86.33
Ours	93.39	97.91	87.69	96.61	93.02	91.73	89.08	89.43	82.38	86.37	87.56	88.1	74.86	77.21	79.49	79.87	87.17

. While the UCR remains consistently stable, the H-score is noticeably impacted by different temperature scaling settings. It is evident that maintaining a fixed temperature scaling is not the optimal solution due to its sensitivity to diverse tasks. Across all tasks and settings, our self-calibration method with automatic temperature scaling proves to be adaptive, showcasing its efficacy. Furthermore, our method, without requiring target validation, could be applied more practically.

7. Conclusions, Limitations and Future Work

In this paper, we establish a new, simple, and powerful baseline for UniDA using foundation models. Through comprehensive experiments, we first demonstrate that existing state-of-the-art UniDA methods face significant limitations when applied to foundation models like CLIP. Our key finding reveals that a properly calibrated CLIP model alone consistently outperforms these complex methods across various UniDA scenarios. This leads to our main contribution: a straightforward confidence calibration approach that sets a new state-of-the-art for UniDA with foundation models. The dramatic performance improvements achieved by our minimalistic method underscore that previous methodological complexities may be unnecessary when leveraging foundation models. We unequivocally position our calibrated CLIP as the new strong baseline for future UniDA research, hoping to steer the field toward more efficient and effective use of foundation models rather than increasingly intricate algorithms.

A limitation of this work is that we primarily explore the use of frozen foundation models, as our initial experiments indicated that full fine-tuning of these models led to subpar performance (see Appendix B). Our study thus focuses on leveraging their robust representations as-is, rather than adapting them through extensive retraining. However, recent studies have introduced new techniques for improving fine-tuning with these models, such as the fine-tuning pre-trained methods [56] and surgical fine-tuning [8]. We did not explore these techniques due to the substantial computational resources required, leaving this as a potential avenue for future research.