PHASE: Progressive Hierarchical Adaptation for Sample-Efficient Rebalancing in Long-Tail Classification

Abstract

When trained on long-tailed distributions, deep neural networks often suffer performance degradation and model bias due to the dominance of head classes. Existing reweighting and sampling strategies have significant limitations, such as reliance on fixed heuristics and inability to adapt to dynamic sample difficulty and class imbalance. Additionally, they fail to integrate sample-level granularity with class-level balance, further inadequately addressing the global imbalance issue. Motivated by these challenges, we introduce the Progressive Hierarchical Adaptation for Sample-Efficient rebalancing (PHASE) training framework, which employs a double-layer tuning paradigm to optimize performance under long-tailed distributions. Specifically, the double-layer tuning paradigm adopted by PHASE runs as follows: (1) an early-stage difficulty-aware mechanism targets those difficult-to-classify samples to guide representation learning; and (2) a later-stage multi-scale reweighting strategy integrates class distribution statistics with sample characteristics. This method ensures fine-grained adaptability and global balance, and thus outperforming those static or localized techniques. Extensive experiments on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets demonstrate that PHASE can significantly improve the accuracy of tail classes without degenerating head class performance, and acquire state-of-the-art classification results. PHASE provides a novel paradigm for long-tailed image recognition.

1. Introduction

In recent years, deep neural networks have made significant advancements in various computer vision tasks, such as image classification, object detection, and semantic segmentation. These advancements have been largely attributed to the availability of meticulously annotated [1,2,3] and well-balanced benchmarks like ImageNet [4], COCO [5], and Places [6]. These datasets play a crucial role as they provide a solid training foundation with evenly distributed classes and ample samples, facilitating the learning of comprehensive and transferable feature representations by the models. However, real-world data typically exhibit a long-tailed distribution, where a small number of classes (head classes) contain the majority of samples, leaving the remaining classes (tail classes) significantly under-represented. This class imbalance leads to a bias in training, where standard models tend to prioritize learning the dominant head classes, resulting in skewed decision boundaries and declined performance on the under-sampled tail classes. This bias poses a particular challenge in critical domains like medical diagnosis [7] and autonomous driving [8], where misclassification or neglect of rare yet crucial categories can have severe consequences. To tackle this issue, researchers have explored various strategies, including advanced data augmentation methods, class-aware resampling techniques, and specific loss function designs. The objective of these approaches is to enhance recognition rates for minority classes while maintaining overall accuracy levels.

The most commonly utilized technique for addressing class imbalance in machine learning is resampling, which involves adjusting the class distributions in training data through either under-sampling the majority classes or over-sampling the minority classes. Despite its popularity, resampling has notable limitations: under-sampling can diminish the diversity and generalization capabilities of the majority classes, while over-sampling may result in overfitting the minority classes [9].

In contrast to resampling, reweighting presents a more direct and computationally efficient solution by assigning varying weights to the training loss of each class, thereby adjusting their relative importance during training. Traditional reweighting methods typically employ fixed weight allocation strategies based on the sample sizes of each class [10]. However, these static weights do not meet the dynamic training requirements of learning models, thereby constraining the adaptability and generalization of the model.

Motivated by aforementioned issue, we present a novel dynamic reweighting framework called Progressive Hierarchical Adaptation for Sample Effective rebalancing (PHASE) which is specifically designed to meet the challenges associated with long-tailed data distributions. PHASE adaptively tunes the weights of both head and tail classes throughout the entire training process for ensuring that each class is sufficiently represented. By dynamically balancing the contributions of both majority and minority classes, PHASE effectively lowers the inherent biases in long-tailed datasets, further enhancing overall performance and robustness of the learning model. Extensive experiments on three benchmark long-tailed image recognition datasets demonstrate that the proposed PHASE framework significantly improves recognition accuracy for tail classes, and meanwhile, improves the generalization of the overall model. It successfully overcomes the limitations of traditional resampling and static reweighting techniques, and provides a robust and extended solution for treating long-tailed data learning issues.

Three main contributions of this study can be summarized as follows:

(1)

By combining sample-level difficulty tuning with class-level weight allocation, PHASE systematically reconfigures optimization strategies tailored for long-tailed scenarios. Theoretically, PHASE achieves dynamic adaptation of local weights and equilibrium of global weights, creating a multi-stage, multi-scale adaptive optimization system that enhances learning model performance on long-tailed datasets.

(2)

In the initial phase, PHASE incorporates a dynamic tuning factor utilizing sample prediction probabilities to establish a Difficulty-Aware Weighting (DAW) framework. This framework enables the adjustment of the learning impact of individual samples, enhancing the model’s sensitivity to the intricate sample distributions of tail classes. It actively redirects the trajectory of representation learning, further constructing a more robust feature space. In the second phase, a Multi-Scale Reweighting (MSR) approach is introduced. MSR integrates statistical patterns obtained from class distributions with the intrinsic data characteristics of each class. This approach effectively merges local and global class weight assignments, dynamically balancing the optimization paths of head and tail classes. It alleviates the adverse effects of weight competition-induced negative transfer between classes.

(3)

Extensive experiments are performed on three commonly utilized long-tailed datasets: CIFAR10-LT, CIFAR100-LT, and ImageNet-LT. The results indicate that PHASE outperforms existing state-of-the-art long-tailed reweighting approaches in terms of both overall classification accuracy and tail class recognition accuracy. Moreover, when integrated with data augmentation methods, PHASE achieves better or comparable accuracy than/to those of the current state-of-the-art techniques, indicating its efficacy and superiority in addressing tasks involving imbalanced data distributions.

The rest of this paper is organized as follows. Section 2 briefly reviews some previous work closely associated with this study, including reweighting methods and decoupling representation techniques. In Section 3, we describe the proposed method in detail. Section 4 provides the experimental results and the corresponding analysis to verify the effectiveness and superiority of the proposed method. Finally, Section 5 concludes this study.

2. Related Work

2.1. Reweighting

Reweighting methods generally involve two different types: loss weighting and logit adjustment. Loss weighting balances the model’s focus between head and tail classes by assigning different loss weights to samples belonging to different classes. Such methods include Class-Balance Loss [11], which uses the number of samples as weights, and Focal Loss [12], which emphasizes those difficult-to-learn samples by adjusting weights based on prediction probabilities. Logit adjustment techniques [13], on the other hand, allocate larger margins to tail classes, further alleviating imbalance. Additionally, methods such as Label-Distribution-Aware Margin (LDAM) loss [14], which enforces class-dependent margins, and Vector-Scaling (VS) loss [15], which combines additive and multiplicative logit adjustments, have also been proposed. These reweighting strategies effectively reduce bias towards head classes and improve classification performance of tail classes.

2.2. Decoupling Representation

Recent advancements in long-tailed visual recognition have emphasized using decoupling representation learning to enhance performance. The first method in such type, LDAM-DRW [14], adopts a two-phase training strategy, in which the first phase concentrates on learning robust feature representations, and the second phase employs delayed reweighting to refine decision boundaries. This approach significantly improves long-tailed prediction accuracy, although its underlying mechanisms could not be fully understood at that time. The foundational work on decoupled training [16] introduced this double-stage framework, and explained how it is different from traditional end-to-end paradigms. In the initial phase, various sampling strategies are empirically tried to search the most effective one for representation learning. In the second phase, the feature extractor is first fixed, and then different training strategies are evaluated and compared. Results indicate that the best choice is to use random sampling for optimizing representation learning and classifier re-adjustment as the training strategy, as this combination can significantly improve long-tailed recognition performance.

Based on these findings, MiSLAS [17] has been proposed. It adopts data augmentation by mix-up techniques in the first phase to strengthen representation learning, and employs a label-aware smoothing strategy in the second phase to improve classifier generalization. The improvements on both stages help to yield superior classification performance on long-tailed datasets. More recent studies have further refined the classifier training phase to alleviate specific challenges induced by long-tailed distributions. For example, SimCal [18] proposed a double-layer class balancing sampling strategy for dealing with long-tailed instance segmentation issue. It is helpful for calibrating the classification head and reinforcing the classifier training procedure.

These strategies have consistently shown superior classification performance on the aforementioned long-tailed datasets. To more clearly situate our proposed PHASE framework within the existing landscape and highlight its unique contributions, we provide a comparative summary of main characteristics between PHASE and several representative state-of-the-art methods [12,14,19,20] in

Table 1. A comparative summary of main characteristics between PHASE and several representative state-of-the-art methods.

Method	Training Paradigm	Focus on Stage 1	Focus on Stage 2	Sample-Level Difficulty Awareness?	Intra-Class Characteristic Consideration?
Focal Loss [12]	Single-stage	N/A	N/A	Yes (via modulating factor)	No
LDAM-DRW [14]	Two-stage	Minimize class-adjusted margin-based loss	Reweight by class frequency	Indirectly (via margin)	No
AREA [19]	Two-stage	Standard cross-entropy loss	Reweight by “effective area” of classes	No	Yes (via class feature distribution)
SURE [20] (conceptually)	Single-stage	N/A	N/A	Yes (via loss reweighting)	Implicitly (via robust optimization)
PHASE (Ours)	Two-stage	Difficulty-Aware Weighting (DAW)	Multi-Scale Reweighting (MSR)	Yes (explicitly via DAW)	Yes (explicitly via MSR)

.

As illustrated in Table 1, while existing methods often focus on a single aspect—such as sample difficulty (Focal Loss [12], SURE [20]), class-level margins (LDAM-DRW [14]), or intra-class distribution (AREA [19])—the PHASE framework is uniquely designed to systematically and explicitly integrate these complementary strategies in a progressive manner.

Unlike LDAM-DRW [14], which applies a static margin and delayed reweighting, PHASE introduces a dynamic, sample-level difficulty-aware weighting in the first stage to guide a more robust representation learning.

Unlike AREA [19], which uses a fixed feature distribution for reweighting in the second stage, PHASE proposes a multi-scale reweighting strategy that combines global class imbalance statistics with local, dynamic intra-class characteristics.

This progressive hierarchical adaptation—first refining features by sample difficulty, then refining the classifier by both class and intra-class balance—constitutes the novel contribution of PHASE, enabling a more fine-grained and effective handling of long-tailed distributions.

3. Methods

In this section, we introduce PHASE that is a novel approach designed to address long-tailed distributions issue in visual recognition tasks. Specifically, PHASE comprises two core components: DAW and MSR. Section 3.1 describes the primary challenges of long-tailed learning, followed by detailed descriptions of DAW in Section 3.2 and MSR in Section 3.3.

3.1. Preliminaries

The most critical challenge in long-tailed learning lies in effectively learning under inherently imbalanced class distributions. Consider a training set $D=\{\left(x_i,y_i\right){\}}_{i = 1}^N$ composed of $N$ samples, where each sample $x_i$ associates with a label $y_i\in \{1,2,\dots ,C\}$. If the dataset exhibits a highly imbalanced class distribution $\left(N_1,N_2,\dots ,N_C\right)$, where $N_j$ denotes the number of training instances in the $j$-th class, and typically the classes with fewer samples are less represented than those with more samples, it is considered a long-tailed dataset. This imbalance poses significant challenges to conventional training paradigms. The large discrepancy in the number of samples across different classes leads to representation biases, making learning models focus more on the head classes, but neglect the tail classes. To address the core challenges in long-tailed learning outlined above, we propose the PHASE framework. It is built on a progressive hierarchical adaptation paradigm, whose core idea is to decouple and sequentially optimize two critical aspects: robust feature representation learning and balanced classifier calibration.

3.2. Difficulty-Aware Weighting Framework

We propose DAW framework that prioritizes those challenging instances during representation learning. Inspired by asymmetric focal-like paradigms [21], the DAW framework reduces the weights of easily classified samples and meanwhile, emphasizes those more difficult ones, thereby providing valuable information for later training. By adaptively tuning weights based on sample recognition difficulty level, it lays a solid foundation for further weight optimization at second stage, as illustrated in Figure 1.

The model predicts a probability distribution $p\left(x_i\right)={\left[p_1\left(x_i\right),p_2\left(x_i\right),\dots ,p_c\left(x_i\right)\right]}^{\mathrm{\top }}$ satisfying the condition of ${\sum }_i{p}_j\left(x_i\right)=1$, where $p_j\left(x_i\right)$ denotes the predicted probability that $x_i$ belongs to the class $j$. Then it can be used to calculate the following asymmetric loss (‘ASL’):

$$L_{\mathrm{ASL}}=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\left[w_i^{+}\log p_{y_i}\left(x_i\right)+{\sum }_{j\ne y_i}{w}_{i,j}^{-}\log \left(1-p_j\left(x_i\right)\right)\right]$$

(1)

where $w_i^{+}$ and $\left(w_{i,j}^{-}\right)$ represent dynamic weights for positive and negative samples, respectively. These weights can be defined as:

$$w_i^{+}={\left(1-p_{y_i}\left(x_i\right)\right)}^{{\mathsf{\gamma }}^{+}}, w_{i,j}^{-}={\left(p_j\left(x_i\right)\right)}^{{\mathsf{\gamma }}^{-}}$$

(2)

where ${\mathsf{\gamma }}^{+}$ and ${\mathsf{\gamma }}^{-}$ are hyper-parameters which are used to control the degree of gradient decay. The positive sample weight $w_i^{+}$ decreases with the increase in $p_{y_i}\left(x_i\right)$, which can be used to lower the contribution of easily classified samples, while the negative sample weight $w_{i,j}^{-}$ reduces for small $p_j\left(x_i\right)$, which can be used to avoid excessive penalties on well-separated negative samples. To unify the gradient tuning mechanism for both positive and negative samples, we introduce a weight correction function $A\left(x_i,j\right)$ as follows:

$$A\left(x_i,j\right)={\left(1-p_j\left(x_i\right)\right)}^{{\mathsf{\gamma }}^{+}\cdot \mathcal{I}\left(j = y_i\right)+{\mathsf{\gamma }}^{-}\cdot \mathcal{I}\left(j \ne y_i\right)}$$

(3)

where $\mathcal{I}\left(\cdot \right)$ is an indicator function that is used to distinguish positive samples $\left(j=y_i\right)$ and negative samples $\left(j\ne y_i\right)$. With $A\left(x_i,j\right)$, the ASL Loss can be rewritten as:

$$L_{\mathrm{ASL}}=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\sum }_{j=1}^{C}A\left(x_i,j\right)\cdot t_{i,j}\cdot \log p_j\left(x_i\right)$$

(4)

where $t_{i,j}$ is the one-hot label distribution of $x_i$. To further improve the model’s robustness to tail-class samples and label noise, we also adopt the label smoothing technique to transform the one-hot label $t_{i,j}$ into a smoothed distribution $t_{i,j}^{\prime }$ as follows:

$$t_{i,j}^{\prime }=\left\{\begin{array}{c}1-ϵ+{\displaystyle \frac{ϵ}{C}}, \mathrm{if} j=y_i\\ {\displaystyle \frac{\mathsf{ϵ}}{C}}, \mathrm{if} j\ne y_i\end{array}\right.$$

(5)

where $\mathsf{ϵ}\in \left[\mathrm{0,1}\right]$ is the smoothing parameter that is used to balance the probabilities between the real class and other classes. By incorporating label smoothing, the final loss function can be expressed as:

$$L_{final}=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\sum }_{j=1}^{C}A\left(x_i,j\right)\cdot t_{i,j}^{\prime }\cdot \log p_j\left(x_i\right)$$

(6)

The optimization objective for the first stage is then defined as:

$${\Theta }^{\left(1\right)}=arg\underset{\Theta }{min}{\mathbb{E}}_{\left(x,y\right)\sim D_{train}^{\left(1\right)}}\left[L_{final}\left(x,y;\Theta \right)\right]$$

(7)

where $D_{\mathit{train}}^{\left(1\right)}$ represents the training dataset used at the first stage. After finishing the training of the first stage, the model parameters ${\Theta }^{\left(1\right)}$ have been optimized, and would be then fixed to provide stable feature representations for the second stage.

The complete procedure of the proposed ASL-based Difficulty-Aware Weighting (DAW) is summarized in Algorithm 1.

Algorithm 1: DAW

Input: Training set $D_{\mathrm{train}}^{\left(1\right)}=\{\left(x_i,y_i\right){\}}_{i=1}^N$, the number of classes C, the positive and negative focusing parameters γ+, γ−, the label smoothing parameter ϵ

Output: Optimized feature extractor parameters ${\Theta }^{\left(1\right)}$

Procedure:

1. Initialize the model with γ+, γ− and ϵ;

2. Define the log-softmax function for output normalization;

3. For each training instance (xi,yi)      (a) Calculate predicted class probabilities p(xi) using softmax;      (b) Generate one-hot label vector ti,j;      (c) Calculate asymmetric weights A(xi,j) as defined in Equation (3);      (d) Apply label smoothing to obtain smoothed labels $t_{i,j}^{\prime }$ according to Equation (5);

4. Aggregate the instance-wise losses to obtain the final loss Lfinal;

5. Optimize model parameters ${\Theta }^{\left(1\right)}$ by minimizing the expected loss on $D_{\mathit{train}}^{\left(1\right)}$ as defined in Equation (7).

3.3. Multi-Scale Reweighting

In long-tailed distributions, relying solely on static reweighting based on sample quantities always fails to adequately capture the complex intra-class feature distributions. To address this issue, we propose a Multi-Scale Reweighting (MSR) method and conduct it at the second stage of training. Specifically, MSR combines global weights tuning based on sample quantities with local dynamic weights tuning based on the effective area [19], which is illustrated in Figure 2.

Both tunings are conducted at the class level to dynamically determine the integrated weight of each sample for enhancing the learning quality of tail classes. First, to address the imbalance issue existing in sample quantities among classes, we define a global tuning weight $w_j^{\left(n\right)}$ to balance the contributions of different classes. Let $N_j$ denote the number of samples in class $j$, and $C$ represent the number of classes. The global tuning weight can be defined as follows:

$$w_j^{\left(n\right)}={\left({\displaystyle \frac{N_j}{{\sum }_{c=1}^C{N}_c}}\right)}^{-\gamma }$$

(8)

where $\gamma >0$ is a hyperparameter that is used to control the sensitivity of the weight distribution to sample quantities. This formulation is inspired by the concept of effective number of samples [11]. The parameter $\gamma$ governs the trade-off between alleviating head-class bias and maintaining training stability. As $\gamma \to 0$, $w_j^{\left(n\right)}\to 1$, the loss reduces to a standard balanced loss, treating all classes equally but failing to address the imbalance. As $\gamma \to \mathrm{\infty }$, the weight for the smallest class becomes excessively large, which may lead to overfitting on tail classes and instability. Thus, an appropriate $\gamma$ is chosen to smoothly amplify the gradient contributions of tail classes during backpropagation without causing divergence. Here, classes with fewer samples (i.e., tail classes) are generally assigned larger weights to amplify their contributions to the overall optimization. Furthermore, the global weight should be normalized as:

$${\bar{w}}_j^{\left(n\right)}={\displaystyle \frac{w_j^{\left(n\right)}}{\sum _{c=1}^C{w}_c^{\left(n\right)}}}\cdot C$$

(9)

Although the global tuning alleviates inter-class imbalance, it cannot fully capture the diversity within each class. To model the discrepancy among intra-class distributions, we further introduce a local dynamic tuning mechanism based on the similarity among samples within each class. Using the feature extractor that has been fixed at the first stage, we compute the feature representation of a sample $x_i$, denoted as $\phi \left(x_i\right)\in R^d$. For a specific class $j$, the class feature center ${\mu }_j$ is calculated by:

$${\mu }_j={\displaystyle \frac{1}{N_j}}{\sum }_{x_i\in {\mathcal{J}}_{\mathcal{j}}}\phi \left(x_i\right)$$

(10)

where ${\mathcal{J}}_{\mathcal{j}}=\{i\mid y_i=j\}$ is the set of indices for indicating which samples belong to the class $j$. Based on the feature deviation $\Delta \phi \left(x_i\right)=\phi \left(x_i\right)-{\mu }_j$, the similarity between a sample $x_i$ and another sample $x_k$ within the same class can be calculated as follows:

$$r_{ik}={\displaystyle \frac{\Delta \phi {\left(x_i\right)}^{\top }\Delta \phi \left(x_k\right)}{\left|\Delta \phi \left(x_i\right)\right|\cdot \left|\Delta \phi \left(x_k\right)\right|}}$$

(11)

where $\parallel \cdot \parallel$ denotes the Euclidean norm. By using these similarity scores, the local dynamic tuning weight for the sample $x_i$ is defined as:

$$w_j^{\left(e\right)}\left(x_i\right)={\left({\displaystyle \frac{1}{N_j}}{\sum }_{k\in {\mathcal{J}}_{\mathcal{j}}}{r}_{ik}\right)}^{-1}$$

(12)

This equation highlights those samples which deviate more significantly from the class center or have weaker relationships with other samples, making the model better handle the diverse feature distribution within each class. To combine global tuning and local dynamic tuning, we define the integrated class-level weight $w_j\left(x_i\right)$ as:

$$w_j\left(x_i\right)=\alpha \cdot {\bar{w}}_j^{\left(n\right)}+\left(1-\alpha \right)\cdot w_j^{\left(e\right)}\left(x_i\right)$$

(13)

where $\alpha \in \left[\mathrm{0,1}\right]$ is an tradeoff factor that controls the relative contributions of global and local weight tunings. The convex combination in Equation (13) is crucial for stability. The factor $\alpha$ balances trust between global class-level statistics and local sample-level characteristics. Setting $\alpha$ = 1 relies solely on robust global statistics but may ignore intra-class diversity. Setting $\alpha$ = 0 focuses entirely on intra-class similarity, which might be sensitive to feature noise. A value $0<\alpha <1$ harnesses the complementary strengths of both. Importantly, this formulation ensures the integrated weight $w_j\left(x_i\right)$ is bounded, i.e., $\mathrm{m}\mathrm{i}\mathrm{n}\left(w_j^{\left(n\right)},w_j^{\left(e\right)}\right)\le w_j\left(x_i\right)\le \mathrm{m}\mathrm{a}\mathrm{x}\left(w_j^{\left(n\right)},w_j^{\left(e\right)}\right)$, preventing gradient explosion or vanishing that could arise from uncontrolled weight scaling, thereby enhancing training stability. At the second stage of training, the model uses the integrated weight $w_j\left(x_i\right)$ to optimize the classifier. The objective function is defined as:

$${\Theta }^{\left(2\right)}=\arg \underset{\Theta }{\min }{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{w}_{y_i}\left(x_i\right)\cdot L_{\mathrm{CE}}\left(x_i,y_i;\Theta \right)$$

(14)

where $L_{\mathrm{CE}}\left(x_i,y_i;\Theta \right)$ represents the cross-entropy loss. The loss function in Equation (14) is a weighted empirical risk minimization. Since the feature extractor is fixed in stage two, the weights $w_{y_i}\left(x_i\right)>0$ are positive and bounded scalars determined prior to classifier training. Given that the cross-entropy loss $L_{CE}$ is smooth and differentiable with respect to the classifier parameters $\theta$, the gradient of $L_{MSR}$ remains Lipschitz continuous. Therefore, when optimized using standard stochastic gradient descent (SGD) with a suitable learning rate schedule (as in Section 4.1), the training process is expected to converge to a stationary point [22,23], consistent with the convergence guarantees of the underlying optimizer. The role of MSR is to reshape the loss landscape for more balanced learning without altering these fundamental convergence properties. By integrating the global weight tuning based on sample quantity with the local weight tuning based on intra-class similarity, MSR can effectively enhance the learning quality of tail classes, and meanwhile preserve the diversity within class distributions, helping achieve a more balanced optimization for long-tailed distributions.

The procedure of the proposed MSR algorithm is summarized in Algorithm 2.

Algorithm 2: MSR

Input: Training set $\mathcal{D}=\{\left(x_i,y_i\right){\}}_{i = 1}^N$; the sample counts of each class $\{N_j{\}}_{j = 1}^C$; the feature extractor $\phi \left(\cdot \right)$; the hyper-parameter γ > 0; the tradeoff factor $\alpha \in \left[\mathrm{0,1}\right]$

Output: The integrated class-level weights $\{w_j\left(x_i\right){\}}_{i = 1}^N$

Procedure: Calculate the global tuning weight $w_j^{\left(n\right)}$ for each class $j\in \{\mathrm{1,2},\dots ,C\}$ by using Equation (8) and normalize it via Equation (9); Obtain the fixed feature representations $\phi \left(x_i\right)$ for all training samples by using the pre-trained feature extractor; For each class j, calculate its class center μj by using Equation (10); For each instance xi with class label yi = j: (a) Compute the feature deviation $\Delta \phi \left(x_i\right)=\phi \left(x_i\right)-{\mu }_j$ (b) For each sample xk in the same class, calculate their cosine similarity rik by using Equation (11); (c) Calculate local dynamic weight $w_j^e\left(x_i\right)$ by using Equation (12); Calculate the integrated weight wj(xi) by using Equation (13); Use wj(xi) to reweight cross-entropy loss during the second-stage classifier training via Equation (14); Output final sample weights $\{w_j\left(x_i\right){\}}_{i = 1}^N$.

4. Experiments

In this section, we conducted a comprehensive evaluation of the performance of the proposed PHASE framework on three widely used long-tailed image classification benchmark datasets: CIFAR10-LT, CIFAR100-LT, and ImageNet-LT. These datasets exhibit varying degrees of class imbalance and serve as standard benchmarks for assessing the robustness and generalization capabilities of long-tailed learning approaches. To gain a deeper understanding of the specific contributions of each component within the PHASE framework, we also performed extensive ablation studies. These analyses enable us to independently assess the impact of individual modules and investigate the interactions between different components, offering valuable insights into the internal mechanisms and design decisions of the PHASE framework. Furthermore, we conducted comparative evaluations against several state-of-the-art long-tailed learning methods. The comparative results show the efficacy of our proposed approach across datasets with diverse levels of class imbalance, thereby confirming the superior performance of PHASE in addressing real-world long-tailed recognition tasks.

4.1. Experiment Settings

To assess our proposed approach thoroughly, we initially generated various long-tailed versions from both CIFAR-10 and CIFAR-100 datasets using an exponential decay sampling method [24] applied to the original balanced datasets. An imbalance factor (IF $={\displaystyle \frac{N_{\max }}{N_{\min }}}$) was defined, in where $N_{\max }$ and $N_{\min }$ represent the sample counts in the most and least represented classes, respectively. By adjusting the IF, we could precisely manage the degree of imbalance in the resulting dataset. Subsequently, CIFAR10-LT (Figure 3a) and CIFAR100-LT (Figure 3b) each produced five training subsets with varying imbalance ratios in $\{200,100,50,20,10\}$. Furthermore, we utilized ImageNet-LT (Figure 3c), a subset of the ImageNet dataset comprising 200 classes and 100,000 images resized to 64 $\times$ 64 pixels. ImageNet-LT demonstrates an imbalance factor of 256, providing a more demanding benchmark for assessing the resilience and generalization capabilities of the proposed method. Detailed information regarding these long-tailed datasets is available in

Table 2. The detailed information of the used long-tailed datasets.

Dataset	Imbalance Ratio	Number of Training Samples	Number of Testing Samples	Number of Classes	Size of Max Class	Size of Min Class
CIFAR10-LT	200, 100, 50, 20, 10	11,203~50,000	10,000	10	5000	25~500
CIFAR100-LT	200, 100, 50, 20, 10	9502~50,000	10,000	100	500	2~500
ImageNet-LT	256	115,846	50,000	1000	1280	5

.

In our experiments, all models were trained using the PyTorch (v2.0.1) framework [22] on a GeForce RTX 3090 GPU(NVIDIA Corporation, Santa Clara, CA, USA). We utilized the ResNet-32 architecture as the backbone for training on both CIFAR10-LT and CIFAR100-LT datasets, following the configuration specified in [19]. The models were trained for 200 epochs using standard SGD with a momentum of 0.9 and weight decay $5\times {10}^{-4}$. The initial learning rate was 0.1 with a linear warmup schedule [25], which was subsequently reduced to $1\times {10}^{-3}$ at epoch 160 and further decreased to $1\times {10}^{-5}$ at epoch 180 to facilitate convergence. For experiments on the ImageNet-LT dataset, we employed ResNet-50 as the backbone, with an initial learning rate of 0.1 and weight decay of $2\times {10}^{-4}$. To ensure fair comparisons with baseline methods, consistent data augmentation techniques such as random horizontal flipping and random cropping were applied to all models.

4.2. Long-Tailed Benchmark Results

To validate the effectiveness and superiority of our proposed method, we conducted comprehensive comparative experiments on multiple long-tailed image recognition datasets. Given the rapid advancements in this field, we selected various state-of-the-art methods for comparison, including single-stage reweighting methods and double-stage decoupled representation approaches. For instance, the Class-Balanced Loss method [11] adjusts the loss function by inversely weighting it based on the effective number of samples, thus ensuring a balanced training influence across different classes. In contrast, methods like LDAM-DRW [14] and AREA [19] employ a double-stage training approach, which consists of initial training followed by subsequent rebalancing, aiming to improve classification performance on long-tailed datasets. These strategies have consistently shown superior classification performance on the aforementioned long-tailed datasets.

It is notable that methods utilizing double-stage decoupled representation strategies exhibit a closer association with our PHASE approach. We conducted extensive experiments on the CIFAR10-LT and CIFAR100-LT datasets to assess the performance of our method, PHASE, in comparison to several state-of-the-art baseline approaches [11,12,14,19,26,27], across five different imbalance factors (10, 20, 50, 100, 200).

Table 3. Comparison results on CIFAR10/100-LT datasets. Top-1 accuracy (%) are reported, with the highest accuracy on each dataset have been highlighted in bold.

Dataset	CIFAR-10					CIFAR-100
IF	200	100	50	20	10	200	100	50	20	10
Cross-Entropy	65.68	70.36	74.81	82.23	86.39	34.84	38.32	43.85	51.14	55.71
Focal [12]	65.29	70.38	76.71	82.76	86.66	35.62	38.41	44.32	51.95	55.78
LDAM [14]	-	73.35	-	-	86.96	-	39.6	-	-	56.91
L2RW [26]	66.25	72.23	76.45	81.35	82.12	33.00	38.90	43.17	50.75	52.12
CB [11]	68.89	74.57	79.27	84.36	87.49	36.23	39.60	45.32	52.59	57.99
LDAM-DRW [14]	73.52	77.03	81.03	-	88.16	38.91	42.04	47.62	-	57.99
IB [27]	73.96	78.26	81.70	85.80	88.25	37.31	42.14	46.22	52.63	57.13
AREA [19]	74.99	78.88	82.68	85.99	88.71	43.85	48.83	51.77	57.02	60.77
Our methods
DAW
+DRW	74.81	78.73	82.53	86.13	88.94	44.46	48.96	52.52	57.67	61.10
+AREA	75.70	79.32	83.08	86.19	88.86	44.52	48.97	52.60	57.73	61.11
+MRS	75.93	79.37	83.18	86.30	89.04	44.84	49.07	52.79	57.76	61.23

displays the Top-1 accuracy of each approach on these datasets. The results of our experiments demonstrate that class-based reweighting techniques, such as DRW [14] and AREA [19], notably improve model accuracy regardless of the degree of class imbalance when combined with DAW. Particularly, integrating DAW with DRW and/or AREA yields substantial enhancements in accuracy, underscoring the efficacy of the proposed DAW component.

Also, we observe that integrating DAW with MSR in our proposed PHASE method yields additional performance enhancements. In comparison to the leading reweighting method AREA, PHASE demonstrates accuracy improvements of {0.94%, 0.49%, 0.96%, 0.31%, 0.33%} on the CIFAR10-LT dataset and {0.99%, 0.24%, 1.02%, 0.74%, 0.46%} on the CIFAR100-LT dataset.

The consistent improvements observed in both datasets with all imbalance levels clearly verify that PHASE is superior to existing reweighting strategies. Besides enhancing overall accuracy, the detailed class-wise performance depicted in Figure 4 demonstrates that PHASE significantly improves the recognition accuracy of tail classes, which are often underrepresented in long-tailed distributions. It verifies the effectiveness of PHASE in addressing class imbalance and facilitating balanced feature learning, indicating that it is particularly suitable for long-tailed visual recognition tasks.

Next, an extensive assessment of PHASE was conducted to compare its performance with current state-of-the-art baseline methods on the ImageNet-LT dataset. The results in

Table 4. Comparison results of various methods on ImageNet-LT dataset, in where the highest accuracy has been highlighted in bold.

Method	Top-1 Accuracy
Cross-Entropy	38.88
Focal Loss [12]	30.50
Class-Balanced Loss [10]	40.85
LDAM [14]	41.86
LDAM-DRW [14]	45.74
Decoupling [16]	47.30
VS + TLA + ADRW [29]	48.21
PHASE	49.96

demonstrate that PHASE achieves a Top-1 accuracy of 49.96% on the ImageNet-LT dataset, outperforming all existing reweighting techniques. To comprehensively evaluate the impact of PHASE on long-tailed distributions, we implemented a categorization approach based on the number of training samples as suggested in [28]. Specifically, classes in the ImageNet-LT dataset were categorized into three groups: head classes (more than 100 samples per class), medium classes (20 to 100 samples per class), and tail classes (fewer than 20 samples per class). Subsequently, the average accuracy was calculated for each group to assess whether the PHASE model can specifically enhance the performance of tail classes.

The results presented in Figure 5 demonstrate that PHASE consistently outperforms the double-stage method VS + TLA + ADRW [29] by 1.14%, 4.22%, and 6.78% in terms of Top-1 accuracy for head, medium, and tail classes, respectively. Additionally, PHASE shows significant superiority over the CE baseline by 16.10% and 27.74% in terms of Top-1 accuracy for medium and tail classes, with a minor decrease of 4.36% for head classes. These results suggest that PHASE effectively enhances the performance of tail and medium classes without compromising the performance of head classes, indicating its efficacy and superiority in managing long-tailed distribution data.

4.3. Ablation Study

To rigorously assess the effectiveness of the proposed PHASE method, we conducted a series of thorough ablation experiments using the CIFAR100-LT dataset with an imbalance factor of 200. These experiments were carefully designed to systematically evaluate the individual contributions of each component within the PHASE framework. By analyzing each component both independently and in combination, the ablation study aims to deepen our understanding of the specific functional roles and interactions between DAW and MSR in addressing the class imbalance issue. This method enables us to not only determine the impact of each module on performance enhancement but also to explore how their integration results in synergistic benefits that surpass the capabilities of each component in isolation. The outcomes of this analysis further confirm the robustness, modularity, and efficacy of the PHASE framework, indicating its relevance to long-tailed visual recognition tasks characterized by highly skewed class distributions.

In Equation (13), we introduced a hyper-parameter, denoted as $\alpha$, to balance the tradeoff between two factors, $w_j^{\left(n\right)}$ and $w_j^{\left(e\right)}$. To assess the impact of $\alpha$ on model performance, we present the accuracy–weight variation curve in Figure 6. In this curve, $\alpha$ varies from 0 to 1 with an increment of 0.1. The results depicted in Figure 6 indicate that the performance initially improves with an increase in $\alpha$. However, once it reaches an optimal value, the performance starts to decline. This observation aligns with our expectation that both weights are crucial for improving the performance of the model. Furthermore, we find that an optimal $\alpha$ setting can significantly improve the performance of the model. Notably, when $\alpha$ is set to 0.8, it enhances the performance by 0.15% and 0.32% compared to using $w_j^{\left(n\right)}$ and $w_j^{\left(e\right)}$ independently, respectively.

As previously outlined, the PHASE framework comprises two main modules: DAW and the MSR. To assess the efficacy of these two components individually and in combination, we conducted an ablation study using the CIFAR100-LT dataset with an imbalance factor of 200. The results, detailed in

Table 5. Ablation results of two main components of PHASE, where the best classification accuracy (%) on CIFAR100-LT (IF = 200) dataset has been highlighted in bold. Specifically, √ and × denote the corresponding component is involved or not, respectively.

DAW	MSR	Accuracy (%)
×	×	34.79
√	×	37.93
×	√	44.02
√	√	44.84

, indicate that both DAW and MSR, when employed separately, significantly enhance classification performance compared to the baseline approach. Furthermore, the integration of these two components within the PHASE framework leads to additional performance improvements, highlighting a synergistic relationship between the two strategies. These results not only confirm the effectiveness of each component in mitigating class imbalance challenges, but also verify the overall robustness of the unified PHASE framework in addressing long-tailed visual recognition tasks.

4.4. Comparison with SOTA Methods

In our initial experiments, we utilized a ResNet-32 architecture with 16 filters in the initial convolutional layer. However, recent state-of-the-art (SOTA) methods [20,30] suggest that employing deeper ResNet-32 architectures with 64 filters in the initial layer enhances classification performance. To ensure a fair comparison with these SOTA methods and fully maximize the potential of our proposed model, we also implemented a ResNet-32 architecture with the same number of channels. Furthermore, we integrated the Sharpness-Aware Minimization (SAM) technique [23] to enhance the optimization capability for tail classes. This technique assists in steering clear of saddle points and reaching flatter minima, thereby improving the overall performance [31].

In addition to the optimization techniques mentioned above, we also implemented various advanced image augmentation techniques [32], such as random cropping, horizontal flipping, and color jittering. These techniques aim to enhance input diversity, thereby enhancing the generalization ability of learning models in practical settings. We still conducted experiments on CIFAR100-LT dataset, with IF in range of {10, 50, 100}, respectively. The results presented in

Table 6. Compared results with current SOTA methods on CIFAR100-LT dataset with IF in range of {10, 50, 100}, in where the highest accuracy on each IF setting has been highlighted in bold.

Method	IF = 100	IF = 50	IF = 10
GML [33]	54.00	58.10	67.00
DeiT-LT [34]	55.60	60.50	-
MDCS [35]	56.10	60.10	-
GLMC [30]	57.11	62.32	72.33
SURE [20]	57.34	63.13	73.24
PHASE	58.00	62.51	72.44

demonstrate that the enhanced PHASE framework can achieve classification accuracy that is equal to or better than those state-of-the-art (SOTA) methods [20,30,33,34,35], thus confirming its efficacy and superiority. Specifically, PHASE performs best on CIFAR100-LT with IF = 100, and takes the second places on two other IF settings (only performs a little worse than SURE [32]). The results show that PHASE is robust enough as it performs excellent even if the class imbalance level is excessively high.

5. Conclusions

In this paper, we introduced PHASE, a novel progressive hierarchical adaptation framework for sample-efficient rebalancing in long-tailed classification. PHASE is designed around a two-stage tuning paradigm that systematically addresses both sample-level difficulty and class-level imbalance.

The effectiveness of PHASE is strongly demonstrated by our extensive experimental results. On CIFAR-100-LT with a high imbalance factor (IF) of 200, PHASE achieved a top-1 accuracy of 44.84%, significantly outperforming the state-of-the-art AREA method (43.85%). More notably, on the large-scale ImageNet-LT dataset, PHASE reached a top-1 accuracy of 49.96%, surpassing the previous best reweighting techniques and demonstrating a substantial improvement over the CE baseline (38.88%). A detailed analysis revealed that these gains were particularly pronounced for medium and tail classes, confirming PHASE’s capability to alleviate model bias without compromising head class performance. The consistent superiority of PHASE across multiple benchmarks and imbalance ratios underscores the validity of its core design principle: decoupling difficulty-aware representation learning from multi-scale classifier rebalancing.

These findings highlight that adaptive, hierarchical tuning is a highly promising direction for long-tailed visual recognition. The PHASE framework provides a robust and effective baseline for future research in this area.

In future work, we plan to explore more efficient approximations for the MSR computation to enhance scalability. Furthermore, we will prioritize investigating the application of the PHASE paradigm to more complex and noisy real-world scenarios, as well as other challenging vision tasks suffering from data imbalance, such as object detection and semantic segmentation.