Progressive Optimization of Target Distribution and Effective Refinement of Hard Samples for Source-Free Domain Adaptation

Abstract

In recent years, the research on source-free domain adaptation has received increasing attention and has achieved considerable progress. It can overcome the dependence on source domain data and obtain a target domain model with robust performance on the target domain only by using the source domain model and unlabeled target domain data. However, existing studies typically handle the target domain distribution in a relatively coarse manner and are consistently susceptible to model noise interference. Therefore, we propose a progressive optimization strategy for the target domain distribution, including two parts: inter-category and intra-category. Regarding inter-category, we decide whether to separate category pairs based on the degree of discrepancy between them. Regarding intra-category, we consider whether to aggregate sample pairs based on whether their pairwise similarity—among samples assigned to the same predicted category—is sufficiently high. And as the training progresses, all data will be optimized. Additionally, for some difficult-to-distinguish categories, we propose a screening strategy that fuses source domain and target domain knowledge. We also further optimized the samples belonging to these categories. Our results on three image datasets demonstrate the effectiveness of our method.

1. Introduction

Deep neural networks have achieved remarkable success in various artificial intelligence tasks, including object recognition, object detection, and image segmentation. However, their performance heavily depends on large amounts of labeled training data. When deploying a trained model to a new domain—even one with the same category set—performance often degrades noticeably. To address this challenge, unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. A key limitation of UDA is its reliance on access to source data during the adaptation process, which may be restricted due to privacy policies or storage constraints. Source-free domain adaptation (SFDA) overcomes this obstacle by performing adaptation using only a pre-trained source model and unlabeled target data, eliminating the need for original source data.

Substantial advancements have been achieved in source-free domain adaptation research, which can broadly be categorized into three primary research directions. The first one is the pseudo-labeling strategy, which mainly assigns labels to target domain data via cluster centers or neighboring samples, thereby facilitating the model training process [1,2,3,4,5,6,7,8]. The second one is the contrastive learning strategy, which aims to minimize the discrepancy between positive sample pairs while maximizing that between negative sample pairs [9,10,11,12]. The third one is based on diversity and discriminability, with its objective being the optimization of target domain data distribution [3,4,11,13,14,15,16].

Existing studies on the optimization of the target domain distribution remain relatively coarse-grained and are often susceptible to noise in the model outputs. With the intention of alleviating this problem, we optimize the inter-category and intra-category distributions simultaneously. Specifically, we aim to separate different categories while compressing samples belonging to the same category. Moreover, taking into account the presence of model prediction noise, it will have a negative impact on the process of optimizing the distribution. That is, it will erroneously bring samples of different categories closer and samples of the same category further apart. Therefore, we adopt a progressive strategy for distribution optimization. We prioritize the separation of category pairs with significant differences and the aggregation of sample pairs of the same category with high similarity. This is carried out to prevent the inappropriate treatment of certain easily confused categories during the early stage of training. As the training progresses and the accuracy of the model output improves, we incorporate an increasing amount of data for processing. This helps to mitigate the interference of model prediction noise on domain adaptation.

In addition, in the target domain, for some categories with similar appearances, samples belonging to these categories are prone to being classified into incorrect but similar categories. This poses a certain level of difficulty for the model to achieve accurate classification. Therefore, we conduct additional optimization on these samples. Herein, we first screen out these indistinguishable similar categories, terming them hard categories; subsequently, we perform a further analysis and refinement on the samples affiliated with these categories, designating them as hard samples. To identify these hard categories, it is necessary to first count the category pairs with high similarity in the target domain. However, the category relationships in the target domain obtained from the target model during the early stage of training are prone to noise. To this end, we draw on the category relationships in the source domain, which can be obtained through the source classifier. Since the source model has been trained with labeled data, its category relationships can serve as a highly valuable reference for those in the target domain. Consequently, in order to alleviate the noise of the target model and the shift between the source domain and the target domain, we integrate the accurate category relationships of the source domain with the knowledge of the target domain. This approach aims to construct an improved category relationship that matches the actual data style of the target domain. Based on this, we can identify similar category pairs and then select the corresponding hard samples. We perform data augmentation on these samples to obtain more category information about the hard samples. A novel Gini confidence-based weighting strategy is adopted for these augmented samples to guide the training of the target model.

In conclusion, our contribution is mainly divided into the following three points. (1) We propose a progressive optimization framework for refining the data distribution in the target domain. (2) We introduce a novel strategy for screening hard categories in the target domain. (3) We present a novel weighting strategy for augmented samples to help correct the target model’s predictions on hard samples.

The remainder of this paper is organized as follows. Section 2 provides a comprehensive review of the existing literature in the field of domain adaptation. Section 3 elaborates on the implementation details of the proposed method, including the optimization of inter-category and intra-category distributions, as well as the refinement of hard samples. Section 4 presents the experimental results of the method on three datasets, along with in-depth discussion and analysis. Section 5 summarizes the key contributions of this study and outlines potential directions for future improvements.

2. Related Work

Domain adaptation aims to transfer knowledge from a source domain to a target domain with distribution discrepancy. Depending on whether the source data are accessible during adaptation, existing methods can be broadly categorized into unsupervised domain adaptation (UDA) and source-free domain adaptation (SFDA). UDA assumes that both labeled source data and unlabeled target data are available during training, while SFDA only relies on a pre-trained source model and unlabeled target data due to privacy or storage constraints. In this section, we first review representative UDA methods and then summarize recent advances in SFDA.

2.1. Unsupervised Domain Adaptation

Recent progress in unsupervised domain adaptation (UDA) has been significantly driven by models that leverage source-domain information. These methodologies can be broadly categorized into three distinct strategic frameworks.

The first approach centers on pseudo-labeling techniques, where core strategies encompass feature clustering, model-based prediction, and generative adversarial networks (GANs). PRPL [17] generates reliable target labels by selecting high-quality pre-trained features and adopting recurrent pseudo-labeling. PFAN [18] effectively screens reliable pseudo-labeled samples and mitigates the bias induced by erroneous labels via the alternating iterations of EHTS and APA. Representative studies also encompass [19,20,21,22]. Despite their effectiveness, their performance is highly sensitive to noisy pseudo labels and error accumulation.

The second paradigm focuses on distance-based alignments, aiming to bridge the domain gap by minimizing metrics that measure distributional divergence. Ref. [23] minimizes the mean and covariance discrepancies of features between the source domain and the target domain. The main works are [24,25,26,27,28,29]. However, most discrepancy-based methods focus on marginal alignment and may overlook class-conditional mismatches, leading to suboptimal semantic alignment.

The third strategy leverages adversarial learning to foster domain-invariant representations, typically achieved via feature-level alignment or data generation. DDAT [30] reduces the domain gap through dual-layer adversarial domain adaptation. Prominent studies within this direction also include [31,32,33,34,35]. Nevertheless, adversarial training is often unstable and may suffer from mode collapse or insufficient category-level alignment.

Despite their impressive results, these approaches rely heavily on the availability of source data—a requirement that is often impractical due to data privacy or storage constraints. Consequently, source-free domain adaptation has emerged as a more viable alternative for real-world deployment.

2.2. Source-Free Domain Adaptation

Due to the unavailability of source domain data, source-free domain adaptation (SFDA) poses a more significant challenge than conventional unsupervised domain adaptation, as it relies solely on a pre-trained model from the source domain and unlabeled data from the target domain. To tackle this challenge, three primary routes have been proposed.

The first one is the strategy based on pseudo-labels. It mainly assigns labels to target domain data through cluster centers or neighbor samples. Specifically, SHOT [3] updates the pseudo-labels of samples by iteratively refining the cluster centers, whereas NRC [4] facilitates more stable learning by analyzing distinct types of neighbors associated with each sample. The relevant studies also include [1,2,5,6,7,8]. However, its performance largely depends on the initial prediction quality and is sensitive to noisy labels.

The second one uses contrastive learning strategy. It primarily seeks to minimize the discrepancy between positive sample pairs while maximizing that between negative sample pairs. HCL [12] proposes a historical contrastive learning framework that encompasses both instance-level and category-level. The main works are [9,10,11]. Nevertheless, its effectiveness relies on reliable pair construction, which may be compromised under severe domain shifts.

The third one is the strategy based on diversity and discriminability. Its aim is to optimize the distribution of the target domain data. For instance, AaD [13] posits that locally similar features should yield consistent predictions (attraction), whereas dissimilar features should produce divergent predictions (dispersion). Certain methods are also considered to have employed this route, including [3,4,11,14,15,16]. However, these methods typically rely on global distribution regularization and may overlook the negative impact of prediction noise.

Currently, there is still relatively limited and rather rudimentary research on the distribution optimization of the target domain in the area of source-free domain adaptation. Considering that the process of domain adaptation is often disrupted by prediction noise, we propose a progressive optimization approach for the distribution of the target domain to mitigate the negative impacts brought about by prediction noise. Specifically, we conduct optimizations from both inter-category and intra-category perspectives. Secondly, there are always some hard samples in the target domain, which are difficult to be accurately classified because their characteristics align closely with those of multiple categories. This is often due to the small differences between several categories. Therefore, we consider adopting a novel strategy to filter out similar categories and optimize the samples belonging to these categories to help them be accurately classified into the correct categories.

3. Method

3.1. Preliminaries

In the source-free domain adaptation setting, a pre-trained source model—consisting of a feature extractor and a classifier—is initially trained using labeled data from the source domain. The objective is to adapt this model to the target domain by leveraging the unlabeled target data ${\left\{x_t^i\right\}}_{i=1}^{N_t}$, where $N_t$ denotes the number of samples in the target domain, through a knowledge transfer mechanism. Our approach adopts the mean-teacher framework, which includes a teacher model and a student model, both initialized using the parameters of the source model. The feature extractor and classifier of the teacher model are represented as $f_{tc}$ and $g_{tc}$, respectively, while those of the student model are denoted as $f_{st}$ and $g_{st}$. For an input sample $x^i$, the student model first generates a feature representation $z_{st}^i=f_{st}\left(x^i\right)\in {\mathbb{R}}^d$, with d being the dimensionality of the feature space. This feature is then passed through the classifier to produce a probability distribution, $p_{st}^i=\delta \left(g_{st}\left(z_{st}^i\right)\right)\in {\mathbb{R}}^K$, where $\delta$ represents the softmax function, and K is the number of categories. The final predicted label is subsequently obtained as ${\widehat{y}}_{st}^i=argmax\left(p_{st}^i\right)$), where $argmax(·)$ returns the index corresponding to the highest probability in the output vector. $st$ is the abbreviation for “student”, and $tc$ is the abbreviation for “teacher”. Ultimately, the student model serves as the final adapted target model.

3.2. Overview

The proposed methodology consists of two components: the progressive distribution optimization module and the hard sample optimization module.

To alleviate the interference caused by prediction noise in the early training phase, we introduce a progressive strategy to refine the target domain distribution by dispersing the inter-category distribution and compressing the intra-category distribution. Specifically, inter-category separation is enhanced by prioritizing category pairs with high disparity, while intra-category compactness is improved by focusing on sample pairs of the same category that exhibit high similarity. The purpose of this phased optimization mechanism is to enable robust adaptation under conditions of unreliable predictions. As illustrated in Figure 1.

Figure 1. The framework of progressive optimization of inter-category and intra-category distributions in the target domain. The red double arrows denote an increase in distance, whereas the black lines denote a decrease in distance. The dark arrows denote newly added optimizations in the current iteration, while the light arrows represent optimizations implemented in previous iterations.

Furthermore, due to high visual similarity among certain categories, samples from these categories are prone to misclassification. To mitigate this issue, we identify easily confused categories through knowledge fusion between the source and target domains. We then apply data augmentation to samples within these categories. For augmented samples, a novel weighting strategy is employed to guide and rectify the target model’s predictions. As presented in Figure 2.

Figure 2. The framework for screening hard categories (a) and optimizing hard samples (b). $C_H$ denotes the set of hard categories. $p_{st}^i$ represents the prediction result of the student model. $p_{tc,a}^i$ is the improved prediction generated by the teacher model.

3.3. Progressively Optimize Distributions

In the target domain, the data distribution directly affects the performance of the target model. It can be argued that, when inter-category distributions exhibit high distinguishability and intra-category samples demonstrate strong compactness, the classifier is better able to derive clear and accurate decision boundaries for target domain data. Consequently, optimizing the data distribution of the target domain is of great necessity. Nevertheless, owing to the presence of domain shift, particularly during the early phase of training, the predictions of the target model regarding the target data frequently exhibit noise and lack accuracy. Relying solely on these predictions to partition the target domain data may weaken the effectiveness of domain adaptation. Consequently, we implement a progressive strategy to optimize the data distribution within the target domain. This strategy encompasses two key aspects: enhancing inter-category separation and promoting intra-category compactness.

3.3.1. Inter-Category Optimization

To optimize across categories, it is first necessary to obtain the representation of each category. Suppose that each category within a batch is represented by its average prediction, and the representation for category k is

$${\mathit{P}}^k={\displaystyle \frac{{\displaystyle \sum _{i=1}^{N_b}}I({\widehat{y}}_{st}^i=k)·p_{st}^i}{{\displaystyle \sum _{i=1}^{N_b}}I({\widehat{y}}_{st}^i=k)}}\in {\mathbb{R}}^K,$$

(1)

where $p_{st}^i\in {\mathbb{R}}^K$ and ${\widehat{y}}_{st}^i$ denote the prediction vector and predicted label of the student model for sample i, respectively; $st$ is the abbreviation for “student”, K represents the number of categories in the target domain, $N_b$ refers to the mini-batch size, and  $I(·)$ is the indicator function.

In the early stage of training, the presence of prediction noise may introduce biases into the derived category representations—particularly for easily confused categories. Categories with substantial disparities can be separated with greater accuracy, whereas accurately distinguishing between similar categories presents a more pronounced challenge. Specifically, the classification boundaries for the former are better defined, while those for the latter remain relatively ambiguous. The undifferentiated dispersion of categories can give rise to negative transfer. Therefore, we initially focus on processing category pairs with more distinct differences. As training progresses, category representations gradually become more accurate, and then more categories are processed step by step, until all categories are optimized.

Specifically, the similarity between categories is represented as $sim(a,b)=a·b^T$, where both a and b denote category representations.

In each training iteration, we set a threshold, $h_1$, to determine whether to process a particular category. Specifically,

$$h_1={\displaystyle \frac{t}{t_{max}}}·M_{max},$$

(2)

where t denotes the current number of training iterations, $t_{max}$ represents the maximum number of training iterations, and ${\displaystyle \frac{t}{t_{max}}}\in (0,1]$. $M_{max}$ stands for the maximum similarity among all category pairs, serving to achieve a correspondence between training time and similarity, and it is updated throughout the training process.

Remark: The rationale for adopting a linear threshold schedule is twofold: (1) Training stability: As training progresses, the target model’s accuracy improves relatively steadily. A linear schedule smoothly and evenly relaxes the threshold, allowing for a consistent, gradual inclusion of category pairs into the optimization process. This avoids sudden, drastic shifts in the loss landscape that could destabilize training. (2) Simplicity and robustness: While non-linear schedules could theoretically be applied, a linear schedule is intuitive and effectively achieves progressive optimization without introducing additional complex hyperparameters that would require heavy tuning for different datasets.

When the divergence between two categories is relatively large, that is, the similarity does not exceed the threshold $h_1$, we increase the distance between them. Category pairs with a similarity greater than the threshold will not be processed. This is to avoid mistakenly moving certain samples away from their own categories.

$$U(A,B)=\left\{\begin{array}{cc}1\hfill & \hfill \mathrm{if}sim(P^A,P^B)\le h_1\\ 0\hfill & \hfill \mathrm{if}sim(P^A,P^B)>h_1\end{array},\right.$$

(3)

when $U(A,B)=1$, we perform optimization on categories A and B.

In summary, we optimize the relationship between categories by reducing the statistics of similarity among the categories that meet the conditions, i.e.,

$$Z_{inter}={\displaystyle \frac{{\sum }_{A=1}^K{\sum }_{B=1,B\ne A}^{K}U(A,B)sim(P^A,P^B)}{{\sum }_{A=1}^K{\sum }_{B=1,B\ne A}^{K}U(A,B)}}.$$

(4)

In a progressive manner, we increase the distances between different categories; that is, reducing their similarity and thereby enhancing their distinguishability.

3.3.2. Intra-Category Optimization

Our objective is to minimize the distance among samples of the same category in the target domain, thereby enhancing the compactness of their distribution. However, certain samples—despite being predicted as belonging to the same category—may in fact originate from different categories. Minimizing their distances without careful consideration could also induce negative transfer. To reduce this risk, we prioritize optimizing those sample pairs that are not only predicted to be of the same category but also have a high degree of similarity, thereby mitigating the impact of incorrect predictions. As the classifier’s accuracy improves during training, an increasing number of same-category samples are progressively incorporated into the optimization process.

Analogous to the inter-category training, in this case, we establish a threshold, $h_2$, for the optimization of intra-category samples. Specifically,

$$h_2=(1-{\displaystyle \frac{t}{t_{max}}})·N_{max},$$

(5)

where $N_{max}$ denotes the maximum value of the similarities of all sample pairs and is updated iteratively throughout the training process. The rationale for employing linear threshold scheduling herein aligns with that for threshold $h_1$.

When two samples are predicted to be of the same category and their similarity is not less than the threshold $h_2$, we optimize their relationship. When a sample pair is predicted to be of different categories or their similarity is less than the threshold, no processing is carried out on them to prevent negative optimization.

$$V(a,b)=\left\{\begin{array}{ccc}1\hfill & \hfill \mathrm{if}sim(p^a,p^b)\ge h_2& \mathrm{and}{\widehat{y}}_{st}^a={\widehat{y}}_{st}^b\hfill \\ 0\hfill & \hfill \mathrm{if}sim(p^a,p^b)<h_2& \mathrm{or}{\widehat{y}}_{st}^a\ne {\widehat{y}}_{st}^b\hfill \end{array},\right.$$

(6)

when $V(a,b)=1$, we perform optimization on samples a and b.

In conclusion, we enhance the similarity of intra-category sample pairs that satisfy the conditions to optimize their relationships.

$$Z_{intra}={\displaystyle \frac{{\sum }_{a=1}^{N_b}{\sum }_{b=1,b\ne a}^{N_b}V(a,b)sim(p_{st}^a,p_{st}^b)}{{\sum }_{a=1}^{N_b}{\sum }_{b=1,b\ne a}^{N_b}V(a,b)}}.$$

(7)

This is implemented in a progressive fashion to decrease the distances between intra-category samples, that is, increasing their similarity and thereby rendering their distribution more compact.

Consequently, the overall optimization function with respect to the inter-category and intra-category distributions of the target domain is

$$L_{ii}={\displaystyle \frac{Z_{inter}+e}{Z_{intra}+e}},$$

(8)

where $e={10}^{-8}$, which is employed to address the cases where the numerator or the denominator is zero.

3.4. Optimize Hard Samples

In the target domain, certain samples are misclassified by the classifier due to the high similarity between their true categories and the assigned incorrect categories. This inherent similarity introduces ambiguity in classification, making it challenging for the model to produce accurate predictions. Thus, addressing this issue is imperative, as it also contributes to the aforementioned optimization of the target domain distribution.

3.4.1. Screening Hard Categories and Samples

It is necessary to first identify category pairs with high similarity by systematically computing the pairwise similarities across all categories. However, in the early stages of training, the category relationships of the target domain obtained are subject to noise. In contrast, the source domain model is trained on the labeled source domain, during which it acquires accurate prior category information about the source domain. This information serves as a crucial clue for the target domain. Despite the existence of domain shift between the source domain and the target domain, the category relationships still exhibit relevance. For example, cats and tigers share a relatively high degree of similarity in both domains, while cats and tables maintain a relatively low similarity. Therefore, the category relationships in the source domain hold significant reference value for the unlabeled target domain. Consequently, after deriving the noisy category relationships of the target domain from the student model, their confidence levels are analyzed. We retain parts that are certain; for uncertain parts, we use the category relationships of the source domain for compensation. As the training proceeds, the accuracy of the target model improves, and the confidence levels of the category relationships in the target domain also increase steadily.

Here, the average Gini confidence of the student model’s prediction results for the target data is employed to represent the confidence level of the target domain category relationships. The Gini confidence of the prediction results of sample i obtained from the student model is denoted as

$$C\left(p_{st}^i\right)=1-G\left(p_{st}^i\right),$$

(9)

where $G\left(p_{st}^i\right)$ is the Gini impurity of $p_{st}^i$, which is specifically represented as $G\left(p_{st}^i\right)=1-{\sum }_{j=1}^K{\left(p_{st,j}^i\right)}^2$, and j represents the index within the $p_{st}^i$.

Thus, the confidence of the category relationships in the target domain is denoted as

$$\overline{C}\left(p_{st}\right)={\displaystyle \frac{1}{N_t}}\sum _{i=1}^{N_t}C\left(p_{st}^i\right).$$

(10)

Suppose that $R_{st}$ denotes the category relationship derived by the student model, which encapsulates domain-specific characteristics of the target distribution, i.e.,

$$R_{st}=cos(P_{st},P_{st}^T)\in {\mathbb{R}}^{K\times K},$$

(11)

where $cos(·,·)$ denotes the cosine similarity function, and $P_{st}\in {\mathbb{R}}^{K\times K}$ represents the average predictions of the K categories in the target domain output by the student model. Suppose $R_s$ denotes the category relationship of the source domain, and its calculation method is

$$R_s=cos(W,W^T)\in {\mathbb{R}}^{K\times K},$$

(12)

where $W\in {\mathbb{R}}^{K\times d}$ denotes the weight of the source classifier.

By integrating $R_s$ with $R_{st}$, an improved category relationship is obtained, which effectively preserves the prior knowledge of the source domain while achieving better alignment with the data style of the target domain. And the parts of the student model that are determined are retained, while the uncertain parts are supplemented through the category relationships in the source domain. The improved category relationship is specifically represented as

$$R_I=(1-\overline{C}\left(p_{st}\right))·R_s+\overline{C}\left(p_{st}\right)·R_{st}.$$

(13)

Remark: Despite the presence of a domain shift between the source and target domains, the category relationship of the source domain—serving as a supplement to the uncertain component of the target domain category relationship—typically provides valuable reference information for the latter. In extreme scenarios where the source and target domains differ drastically, the source domain category relationship only replaces the uncertain part of the target domain category relationship. Even without such supplementation, this uncertain component would inherently be noise. Thus, even under significant domain shift, the negative impact it introduces to the improved category relationship and overall training remains relatively minor. On the whole, the source domain category relationship primarily plays a facilitative role.

After obtaining the improved category relationship $R_I$, we identify the s pairs of categories with the highest similarity from the category relationship matrix, that is, ${\mathrm{top}}_s\left(R_I\right)$. These categories are considered the most difficult to distinguish in the target domain, namely hard categories. Then, we select the samples assigned to these categories by the student classifier and further optimize these samples, which are identified as hard samples.

3.4.2. Optimize Hard Samples

We conduct multiple data augmentation on hard samples to capture more abundant category information. Specifically, a simple yet effective data augmentation strategy is employed: L perturbations are sampled from the Gaussian distribution $\mathcal{N}(0,0.1)$, and each is added to the original data, generating L augmented samples that are structurally distinct from yet semantically consistent with the original data. Subsequently, the teacher model is employed to predict on these L new data. After calculating the weighted average, we obtain $p_{tc,a}$, which is used to align the results of the student model. The detailed operation is as follows.

Suppose $x_h$ is a hard sample belonging to the hard category, and $x_a^l$ ($l\in (1,L)$) is the l-th augmented sample generated for $x_h$. The prediction result of the teacher model for $x_a^l$ is $p_{tc,a}^l=\delta \left(g_{tc}\left(f_{tc}\left(x_a^l\right)\right)\right)\in {\mathbb{R}}^K$, where $tc$ is the abbreviation for “teacher”.

The improved prediction $p_{tc,a}$ for sample $x_h$ is derived by applying weighted aggregation to the prediction results of L augmented samples, i.e., 

$$p_{tc,a}=\sum _{l=1}^L{w}^l{p}_{tc,a}^l\in {\mathbb{R}}^K,$$

(14)

where $w^l$ denotes the weight assigned to the l-th augmented sample. And 

$$w^l={\displaystyle \frac{C\left(p_{tc,a}^l\right)}{{\sum }_{i=1}^{L}C\left(p_{tc,a}^i\right)}},$$

(15)

and $C(·)$ denotes the Gini confidence of the vector inside the parentheses.

The improved prediction $p_{tc,a}$ is employed to calibrate the outcomes of the student model. Suppose that the number of hard samples in the target domain is n, and the corresponding loss for these hard samples is

$$L_{hard}=\sum _{i=1}^{n}KL(p_{st}^i\Vert p_{tc,a}^i),$$

(16)

where $KL(·\Vert ·)$ denotes the Kullback–Leibler (KL) divergence function.

3.5. Overall Loss

Our approach comprises two distinct components: (1) Progressive Optimization Distribution; (2) Optimization of Hard Samples. Therefore, it is denominated as POSTAL. The overall loss function of the target model is

$$L_{POSTAL}=L_{baseline}+\alpha L_{ii}+\beta L_{hard},$$

(17)

where $\alpha$ and $\beta$ are trade-off hyperparameters. $L_{baseline}$ is the loss of baseline (i.e., SHOT or AaD).

Updating the teacher model parameters. Upon the completion of each iteration, the student model ${\theta }_{st}$ is updated via backpropagation. In contrast, the teacher model ${\theta }_{tc}$ performs a momentum-based update from the student model using the Exponential Moving Average (EMA) method, i.e., 

$${\theta }_{tc}^t=\sigma {\theta }_{tc}^{t-1}+(1-\sigma ){\theta }_{st}^t,$$

(18)

where ${\theta }_{tc}$ and ${\theta }_{st}$ are the parameters of the teacher and student models, respectively, and t denotes the training time. $\sigma$ is a hyperparameter set to 0.9.

3.6. Algorithm

Algorithm 1 describes the process of our method. The input comprises unlabeled target domain data ${\mathcal{X}}_t={\left\{x_t\right\}}_{i=1}^{N_t}$ and the maximum number of training iterations $t_{max}$. Line 1 initializes the teacher and student models. Lines 2–11 outline the training iteration process of the algorithm: Lines 3–5 detail the procedure for computing the loss $L_{ii}$; Lines 6–9 elaborate on the process of deriving the loss $L_{hard}$; Line 10 specifies the training and update mechanism for the student model; Line 11 describes the parameter update strategy for the teacher model. The final output is the target domain model (i.e., the student model), which consists of a feature extractor, $f_{st}$, and a classifier, $g_{st}$.

Algorithm 1: POSTAL algorithm for closed-set SFDA

4. Experiments

4.1. Settings

Datasets. We conducted image classification experiments on three benchmark datasets: Office-31 [36], ImageCLEF [37] and DomainNet [38]. Office-31 contains three domains—Amazon (A), DSLR (D) and Webcam (W)—with 2817, 498, and 795 images, respectively, totaling 31 categories. ImageCLEF contains three domains: Caltech-256 (C), ImageNet ILSVRC 2012 (I) and Pascal VOC 2012 (P). Each domain consists of 12 categories, with 50 images contained in each categories. DomainNet contains images from six different domains, amounting to roughly 600,000 images in total. To evaluate the effectiveness of the proposed method, we conduct a series of experiments on seven domain adaptation tasks.

Implementation Details. To ensure a fair comparison with existing approaches, we employ a ResNet-50 as the backbone network across all datasets. Following the setup of SHOT [3] and AaD [13], the model adopts an identical architecture featuring a fully connected layer, followed by Batch Normalization [39], and another fully connected layer with weight normalization [40]. For optimization, SGD is used with a momentum value of 0.9 and a batch size of 64 on all datasets. The initial learning rate is set to $1\times {10}^{-3}$ for all layers, while the last two fully connected layers use a higher learning rate of $1\times {10}^{-2}$. For the Office-31 and ImageCLEF datasets, we conducted training for 50 epochs; for the DomainNet dataset, the training was performed for 30 epochs.

The learning rate $\xi$ is adjusted dynamically according to the formula $\xi ={\xi }_0·{(1+10·{\displaystyle \frac{t}{t_{max}}})}^{-\tau }$, consistent with the methodologies in [41,42]. Here, the hyperparameter $\tau$ governs the decay rate and is fixed at 1. For all our experiments, we set $\alpha =1$, $\beta =0.1$ and $L=12$.

All experiments were conducted on a workstation equipped with an NVIDIA RTX 3090 GPU. The proposed model was implemented using the PyTorch (torch 2.4.1) deep learning framework. The training process was performed with CUDA (12.2) acceleration, and common scientific computing libraries including NumPy (1.23.2) and SciPy (1.10.1) were used for data processing.

Competitors. To prove the effectiveness of our method, we compare POSTAL with the state-of-the-art SFDA methods, including SHOT [3], NRC [4], G-SFDA [43], TPDS [44], C-SFDA [11], UI-SFDA [14], CADRN [45], NVC-ELR [8], DPL [7], OCRCI [5], PFC [6] and AaD [13].

4.2. Results of Object Recognition

Results on Office-31. The comparison result on Office-31 dataset between our method and other SFDA methods is shown in Table 1. Our method achieves an accuracy of 91.5%. A further boost in average accuracy is observed with our approach, which surpasses AaD and SHOT by 1.6% and 1.4%, respectively. The accuracy across all six tasks are not lower than the original ones. Particularly on the A→W task, POSTAL surpasses AaD with a significant 4.3% increase in accuracy.

Table 1. Accuracy (%) on Office-31 for source-free domain adaptation. “SF” stands for source-free. The best results are highlighted in bold and the second-best results are highlighted in underline.

Method	SF	A→D	A→W	D→A	D→W	W→A	W→D	Avg.
C-SFDA [11]	✓	96.2	93.9	77.3	98.8	77.9	99.7	90.5
UI-SFDA [14]	✓	95.3	94.2	76.4	98.3	77.5	99.9	90.3
CADRN [45]	✓	95.6	94.7	76.9	98.6	77.7	100.0	90.6
DPL [7]	✓	97.0	93.5	77.0	98.6	78.0	100.0	90.7
NVC-ELR [8]	✓	95.8	91.7	77.2	98.9	77.2	100.0	90.1
SHOT [3]	✓	94.0	90.1	74.7	98.4	74.3	99.9	88.6
POSTAL + SHOT	✓	95.8	93.2	75.6	99.2	76.0	100.0	90.0 (+1.4)
AaD [13]	✓	96.4	92.1	75.0	99.1	76.5	100.0	89.9
POSTAL + AaD	✓	98.2	96.4	77.1	99.1	78.0	100.0	91.5 (+1.6)

Results on ImageCLEF. The result on ImageCLEF dataset is shown in Table 2. We achieve an accuracy of 90.9%. Our method outperforms AaD and SHOT by margins of 1.1% and 1.0% in average accuracy, respectively, demonstrating a clear performance gain over these baselines. Notably, our method maintains performance consistency, with accuracies across all six tasks exceeding those of the original. Specifically, for the task of I→P, our accuracy is 1.5% higher than that of AaD.

Table 2. Accuracy (%) on ImageCLEF for source-free domain adaptation. “SF” stands for source-free. The best results are highlighted in bold and the second-best results are highlighted in underline.

Method	SF	C→I	C→P	I→C	I→P	P→C	P→I	Avg.
G-SFDA [43]	✓	92.0	79.5	96.7	79.2	95.5	92.5	89.2
NRC [4]	✓	93.3	78.0	97.2	79.2	96.7	92.3	89.4
OCRCI [5]	✓	94.3	80.0	97.5	80.8	97.7	93.7	90.7
SHOT [3]	✓	92.3	79.5	95.8	78.7	95.3	92.7	89.1
POSTAL + SHOT	✓	93.2	80.3	97.5	79.2	96.7	93.5	90.1 (+1.0)
AaD [13]	✓	93.8	78.7	96.8	79.5	96.7	93.2	89.8
POSTAL + AaD	✓	94.7	79.7	97.8	81.0	97.8	94.3	90.9 (+1.1)

Results on DomainNet. Our methods yields competitive results on DomainNet dataset compared with other methods, which is shown in Table 3. Our method achieves an accuracy of 71.0%. Relative to AaD and SHOT, our proposed technique yields an additional 1.4% improvement in terms of average classification accuracy. And the accuracy on all seven tasks is no less than the original. In the specific P→C task, our model significantly outperforms SHOT, delivering a substantial 4.3% margin in terms of accuracy.

Table 3. Accuracy (%) on DomainNet for source-free domain adaptation. “SF” stands for source-free. The best results are highlighted in bold and the second-best results are highlighted in underline.

Method	SF	R→C	R→P	P→C	C→S	S→P	R→S	P→R	Avg.
C-SFDA [11]	✓	70.8	71.1	68.5	62.1	67.4	62.7	80.4	69.0
TPDS [44]	✓	66.4	67.0	65.6	59.8	64.3	61.5	79.0	66.2
CADRN [45]	✓	69.2	68.8	70.3	61.6	66.7	60.7	81.2	68.4
PFC [6]	✓	71.9	70.3	72.7	62.3	68.5	61.0	83.1	70.0
SHOT [3]	✓	67.7	68.4	66.9	60.1	66.1	59.9	80.8	67.1
POSTAL + SHOT	✓	69.7	68.7	71.2	62.2	66.3	59.9	81.4	68.5 (+1.4)
AaD [13]	✓	71.0	70.3	71.8	62.5	69.0	60.4	82.3	69.6
POSTAL + AaD	✓	73.2	71.2	73.0	63.9	69.8	62.3	83.3	71.0 (+1.4)

From Table 1, Table 2 and Table 3, the following three conclusions can be drawn. First, when our method is integrated with the baseline methods, it yields a significant improvement in model accuracy over the original performance. This is attributed to two key aspects: our progressive optimization of inter-category and intra-category distributions in the target domain mitigates the interference of noise on distribution optimization, while our hard sample refinement strategy reduces the negative impact of such samples on model accuracy. Additionally, this integration validates the plug-and-play nature of our method. Second, our method exhibits favorable performance across the three diverse datasets, indicating its compatibility and generalization capability to different datasets. Third, despite the fact that the source-free domain adaptation process does not access the source domain data, promising results are still achieved on the target domain—this holds considerable application value for scenarios where access to source domain data is constrained.

4.3. Model Analysis

Parameter sensitivity. We conduct a sensitivity analysis to evaluate the impact of hyperparameters $\alpha$ and $\beta$ within our framework. Specifically, $\alpha$ and $\beta$ regulate the trade-off weights for the losses $L_{ii}$ and $L_{hard}$, respectively. The resulting performance on the Office-31 D→A task is illustrated in Figure 3.

Figure 3. Parametric sensitivity analysis of $\alpha$ and $\beta$.

We vary $\alpha$ from 0.01 to 1000 while keeping $\beta$ fixed at 0.1 (Figure 3a) and similarly range $\beta$ from 0.001 to 10 with $\alpha$ fixed at 1 (Figure 3b). As illustrated, the model attains its peak accuracy at the configuration of $\alpha =1$ and $\beta =0.1$. We employed identical hyperparameter configurations across three distinct datasets, thereby demonstrating robust cross-dataset generalizability. This allows the default hyperparameters to be reliably transferred to novel target domains, obviating the need for validation data.

Ablation study. To gain deeper insights into the contribution of various modules, we evaluate two architectural variants of our framework on the DomainNet dataset. The results, conducted under the SFDA setting, are summarized in Table 4. Specifically, we remove $L_{ii}$ and $L_{hard}$ from the full model, respectively. $L_{AaD}$ is the loss term of the baseline method AaD. Upon removing the loss $L_{ii}$, the accuracy of the model stands at 70.6%. After removing the loss $L_{hard}$, the model’s accuracy is 70.7%. The accuracy reaches its zenith at 71.0% upon the fusion of both components. This evidence confirms that each constituent contributes positively to the adaptation process, with their joint implementation being indispensable for optimal performance.

Table 4. Ablation study on DomainNet dataset. The w/o is short for without.

Method	R→C	R→P	P→C	C→S	S→P	R→S	P→R	Avg.
$L_{AaD}$	71.0	70.3	71.8	62.5	69.0	60.4	82.3	69.6
$\mathit{w}/\mathit{o}{L}_{ii}$	72.7	70.9	72.9	63.8	69.3	61.7	83.0	70.6
$\mathit{w}/\mathit{o}{L}_{Hard}$	73.1	71.1	72.8	63.4	69.5	61.8	82.9	70.7
Full Model	73.2	71.2	73.0	63.9	69.8	62.3	83.3	71.0

Accuracy analysis of hard categories. Figure 4 presents our analysis of the accuracy for hard categories in the W→A task on the Office-31 dataset. The orange line denotes the variation of model accuracy during training. The green line represents the accuracy of samples from hard categories identified via our strategy. The blue line indicates the prediction accuracy of samples from remaining categories after excluding hard categories from the full dataset. Key observations are as follows: The accuracy of samples in hard categories is substantially lower than the overall dataset accuracy, with a gap of approximately 20%. After removing samples of hard categories, the overall dataset accuracy improves significantly by around 6%. Thus, our method effectively identifies hard categories in the dataset, and optimizing these categories is critical for enhancing the overall prediction accuracy.

Figure 4. Accuracy analysis of hard categories.

Correcting predictions of hard samples. Table 5 shows examples of our method correcting the prediction results of hard samples. Empirical observation reveals that the original predicted categories exhibit high similarity to the corrected ones for these instances. Our method can effectively correct the original wrong predictions of the target model. This indicates the effectiveness of our method for correcting the prediction results of hard samples.

Table 5. Analysis of corrections for hard samples.

	1	2	3	4	5	6
Hard Sample
Original Prediction (Wrong)	flower	bear	swan	peas	dolphin	laptop
After Optimization (Correct)	onion	panda	bird	asparagus	penguin	computer

Analysis of the number of hard categories. Figure 5 presents an analysis of the impact of the number of hard category pairs on model performance in the C→S task of the DomainNet dataset. The number of hard category pairs is selected proportionally based on the total number of categories in the dataset. Results indicate that the model’s accuracy consistently remains above 63.6%. Specifically, when the number of hard category pairs is set to 25 (calculated as 126 × 0.2), the model achieves an accuracy of 63.9%, corresponding to a range of [26, 50] individual hard categories (approximately 20–40% of the total category number). When the number of hard category pairs is 63 (calculated as 126 × 0.5), the model accuracy reaches 64.0%, corresponding to a range of [64, 126] individual hard categories (approximately 51–100% of the total category number). Upon comprehensive consideration of computational cost and accuracy, 25 is selected as the number of hard category pairs. For all datasets, the number of hard category pairs is determined using the same protocol: multiplying the total number of categories in the dataset by 0.2 and rounding down. Specifically, Office-31 yields 6 (31 × 0.2), ImageCLEF yields 2 (12 × 0.2), and DomainNet yields 25 (126 × 0.2).

Figure 5. Analysis of the number of hard categories.

Analysis of data augmentation frequency. Figure 6 shows an analysis of the number of data augmentations for hard samples on the W→A task of the office-31 dataset. As illustrated in the results, the model’s accuracy plateaus above 77.8% once the number of data augmentations L reaches 10. The peak performance of 78.0% is attained at $L=12$. Consequently, we fix $L=12$ as the default augmentation frequency for hard samples in our experiments.

Figure 6. Analysis of the number of data augmentations for hard samples.

Comparison of running time and storage space. Table 6 presents a comparative analysis of the runtime and storage consumption between our proposed method POSTAL and other baseline methods. Our method POSTAL leverages AaD as the embedding baseline. It introduces only marginal increases in overhead. However, its resource consumption remains substantially lower than that of C-SFDA. Furthermore, POSTAL achieves superior accuracy compared to both AaD and C-SFDA. These results confirm that our method strikes a favorable balance between performance and resource consumption, making it suitable for practical use.

Table 6. Comparison of running time (s/epoch) and storage space (MB).

Method	A→D	A→W	D→A	D→W	W→A	W→D	Memory	Acc
C-SFDA	598.4	770.0	2492.3	776.3	2473.8	591.9	28,689	90.5
AaD	4.7	3.5	9.3	3.5	9.5	4.7	9458	89.9
Ours	7.2	9.3	34.4	9.5	32.4	7.5	9931	91.5

Training stability. Figure 7 illustrates the evolution of accuracy on the Office-31 dataset throughout the training process. It is evident that the performance exhibits a steady upward trend before reaching an asymptotic plateau, thereby substantiating the stability and convergence of our proposed method.

Figure 7. Training stability analysis on Office-31.

Feature visualization. To qualitatively assess the learned representations, we utilize t-SNE [46] to visualize the feature distributions for the P→C task on DomainNet. Figure 8 compares the source model, SHOT, AaD, and our proposed method. As illustrated, our approach yields significantly more compact clusters with superior inter-category separability compared to the baselines, thereby validating its effectiveness.

Figure 8. Feature visualizations of task P→C on DomainNet dataset by t-SNE [46].

4.4. Discussion

Our method enhances the performance of source-free domain adaptation by progressively optimizing inter-category and intra-category distributions, as well as refining predictions for hard samples. The experimental results presented above validate the effectiveness of our approach. However, the optimization of hard samples in our method still has limitations: while data augmentation can capture richer category information for hard samples, it remains insufficient for mining category information from samples with extremely ambiguous semantic cues, potentially failing to accurately capture their true category attributes. The work by [47]—a study focusing on few-shot learning—offers a promising solution: by analyzing feature maps across multiple scales, it synchronously captures high-level semantic information and low-level texture details, thereby enabling more accurate and comprehensive extraction of category information for hard samples. This strategy could also facilitate the optimization of both inter-category and intra-category distributions. This constitutes a key direction for our future work.

5. Conclusions

In this paper, we propose to tackle the source-free domain adaptation problem by leveraging progressive distribution optimization and novel correction of hard samples. Specifically, we propose a progressive strategy to optimize both inter-category and intra-category data distributions in the target domain, aiming to mitigate the adverse effects of model prediction noise. This enables more efficacious separation of distinct categories and tighter clustering of samples within the same category. Furthermore, we identify hard categories by incorporating knowledge from both source and target domains, and enhance the optimization of hard samples within these hard categories via data augmentation and Gini confidence weighting. Experimental results demonstrate the effectiveness of the proposed approach. However, there remains substantial room for improvement in the optimization of hard samples, which constitutes a key direction for our future research.