Dynamic Topic-Based Hierarchical Prompt Learning for Multi-Label Image Classification

Abstract

Flat label supervision often constrains multi-label image classification, as it struggles to fully capture inherent label dependencies. It provides limited guidance to the hierarchical features that naturally emerge in Vision Transformers. To address this structural misalignment, we propose Dynamic Topic-based Hierarchical Prompt Learning (DyT-HPL). Instead of relying on predefined and fixed label graphs, DyT-HPL utilizes offline hierarchical clustering to construct multi-granularity semantic priors, from which hierarchical prompts are dynamically retrieved. A frozen visual query branch generates stable semantic queries, which are then used to retrieve discrete prompts from constructed coarse-mid-fine prompt pools. These hierarchical prompts are adaptively injected into different network depths, ensuring that semantic guidance with different abstraction levels is introduced at the most suitable architectural stages. To maintain stable routing and prevent prompt mode collapse, we jointly optimize the architecture with asymmetric classification, surrogate matching, and intra-pool diversity losses. This tripartite design promotes a diverse prompt space and isolates routing updates from final predictions. Comprehensive experiments on MS-COCO, NUS-WIDE, and Corel5k demonstrate that DyT-HPL achieves consistent and favorable performance across diverse settings, highlighting the value of hierarchical semantic guidance with different abstraction levels.

1. Introduction

Multi-label image classification (MLC) is a basic task in computer vision. One image may contain several semantic concepts at the same time [1]. Unlike single-label recognition, MLC must capture not only object appearance but also co-occurrence patterns and semantic relations among labels. This setting appears in image retrieval, scene understanding, and large-scale content management [2]. In most datasets, supervision is given as a flat multi-hot label vector. Such supervision offers only limited guidance for the hierarchical features learned by deep models.

Early studies modeled label correlations with convolutional backbones plus recurrent modules [3] or graph reasoning [4]. Vision Transformers [5] improved the modeling of global feature interactions [6]. Even so, many recent prompt-based methods [7,8] still treat labels as mostly independent. The hierarchical structure hidden in large label sets remains underused. When flat label semantics are matched to multi-level visual features through one path, inconsistencies can appear, especially across network depths.

Our method follows the hierarchy already present in the model. Recent studies on Vision Transformer dynamics suggest that representations from different depths exhibit different levels of semantic abstraction, rather than forming a strict shallow-local versus deep-global dichotomy [9,10,11]. Based on this property, we build a framework with a trainable Vision Transformer and an auxiliary frozen query branch. The frozen branch provides a stable reference and reduces gradient interference. We do not use predefined label graphs. Instead, we extract hierarchical semantic priors from the label space through offline clustering [12]. These priors guide prompt retrieval from separate pools [13,14]. The retrieved prompts are injected into different stages of the backbone. Accordingly, we introduce semantic guidance with different abstraction levels at different architectural stages, using broader semantic cues for global context establishment and less abstract intermediate semantic cues for later refinement. This improves the match between semantic guidance and visual features.

This work makes three main contributions:

• We propose DyT-HPL, a dual-branch framework that combines a frozen query branch with offline clustering priors and ViT hierarchies. It improves classification performance with little trainable-parameter overhead.
• We design a transient prompt injection strategy. Layer-specific prompts are injected only when needed and removed right after interaction. This keeps multi-scale semantic alignment efficient and avoids unnecessary sequence growth.
• We formulate a tripartite optimization objective integrating asymmetric classification, surrogate matching, and intra-pool diversity constraints. This joint formulation stabilizes dynamic routing by explicitly mitigating prompt mode collapse and completely decoupling routing updates from the prediction pathway.

2. Related Work

2.1. Multi-Label Image Classification

Modeling label dependency has long been a central idea in multi-label image classification. Many early methods combined convolutional backbones with graph modules or recurrent structures to model label correlations [1,3,4]. Recent methods use more refined feature extraction schemes. GKGNet [15], for instance, models local-to-global relations through neighbor grouping. SADCL [16] uses contrastive learning to separate highly overlapping categories. Even with these advances, static or hand-crafted label topologies remain a weakness. Correlations learned during training are often detached from the hierarchy of visual features. This mismatch makes fine-grained semantic alignment harder.

2.2. Transformer-Based Multi-Label Learning

Vision Transformers have become a strong choice for MLC because self-attention models global context well [2,5,6]. Representative examples include ML-Decoder [17], which improves the fusion of label embeddings through token refinement and cross-attention. Yet the semantic structure of multi-label data remains difficult to capture. A common reason is that many current methods still rely on flat representations. Once complex label semantics are compressed into one representation space or one cue, the natural coarse-to-fine hierarchy is blurred. A more suitable design should provide hierarchical guidance and let it interact with the Transformer at different depths.

2.3. Prompt Learning and Semantic Guidance

Visual Prompt Tuning (VPT) [14] shows that a frozen backbone can adapt to a new domain with only a small number of trainable parameters. Later methods such as GateVPT [18] and E²VPT [19] further improve prompt routing for complex visual inputs. At the same time, many studies show that external priors or prototypes can also help representation learning [20,21].

Bringing these ideas together is still not easy. Many prompt-based classifiers use static prompts or confine prompts to isolated feature spaces. Once discrete dynamic routing is introduced, mode collapse often appears and the model keeps selecting the same small group of prompts [22,23]. Hand-crafted label graphs add another limitation. They usually miss the natural multi-granularity structure inside complex datasets.

Related ideas have also appeared in other vision tasks, where complementary modules are used to exploit structured priors beyond a single supervision pathway. For example, Dong et al. [24] combine temporal distribution learning and spatial correlation refinement for universal moving object segmentation. Although their task setting and technical design differ substantially from ours, their results further support the value of incorporating structured priors through complementary branches. DyT-HPL addresses these issues by replacing fixed label graphs with offline hierarchical clustering [12]. The resulting semantic priors initialize hierarchical prompt pools.

2.4. Representation Dynamics in Vision Transformers

Recent studies suggest that Vision Transformer representations at different depths differ mainly in semantic abstraction, rather than following a strict shallow-local versus deep-global dichotomy [10,11,25].

DyT-HPL is built around this depth-wise change. We do not impose a single flat semantic structure across all layers. Instead, we derive semantic queries from selected intermediate and deep layers. All queries are extracted from CLS tokens and therefore remain globally contextualized semantic representations, while differing in abstraction level and functional role. The final-layer query is used to retrieve broad semantic guidance, whereas the intermediate-layer queries provide complementary and later-stage semantic refinement. As a result, our dynamic prompt injection stays closer to the network’s feature hierarchy and reduces the mismatch caused by flat label mapping.

Furthermore, this dynamic hierarchical strategy aligns with the recent paradigm shift towards Vision-Language Models and adaptive visual prompt tuning for multi-label recognition [26,27]. While these contemporary approaches have demonstrated the efficacy of prompt-based semantic alignment, DyT-HPL uniquely extends this success by mitigating prompt mode collapse through depth-aware hierarchical routing, distinguishing it from recent adaptive prompting methods [28].

3. Methods

3.1. Problem Formulation

Given an input image $x\in {\mathbb{R}}^{H\times W\times 3}$ and a label space with C categories, multi-label image classification aims to predict a binary target vector $y\in {\{0,1\}}^C$. Each element indicates whether the corresponding category is present in the image. Let the model output be logits $z\in {\mathbb{R}}^C$. The task is to learn a mapping from the image to these logits:

$$f:x\to z.$$

(1)

The logits were converted into multi-label decisions and optimized with a multi-label classification loss. In DyT-HPL, this mapping was further improved by hierarchical prompt retrieval and injection-guided by stable semantic queries.

3.2. Overall Architecture

Before online feature propagation starts, DyT-HPL performs offline hierarchical label clustering directly on the dataset label space. We do not rely on external text encoders. Instead, we use label distributions and label co-occurrence statistics from the training set to build the hierarchy. This turns flat labels into a structured topic tree with coarse, mid, and fine levels.

These topic centers define the structure of the prompt pools and reflect natural semantic groupings. They also change the way the model is trained. The Vision Transformer is no longer asked to learn from isolated visual features alone. Statistical priors provide an extra semantic signal, which helps relieve label sparsity and semantic interference.

DyT-HPL has two coupled branches: a frozen query branch and a trainable main Vision Transformer. The frozen branch outputs three query vectors, $q_{\mathrm{coarse}}$, $q_{\mathrm{mid}}$, and $q_{\mathrm{fine}}$, from different semantic depths. These queries match against three prompt pools and retrieve targeted prompts. The retrieved prompts were injected into the shallow, middle, and deep stages of the main ViT. The final features were sent to the classifier head for prediction.

The complete data flow is summarized as follows: image → frozen query ViT → $q_{\mathrm{coarse}},q_{\mathrm{mid}},q_{\mathrm{fine}}$ → prompt pools → main ViT → classifier head → losses.

3.3. Offline Hierarchical Clustering and Prior Initialization

To provide semantic guidance at different granularities, DyT-HPL initializes its prompt space with offline hierarchical clustering based on label co-occurrence. This avoids random prompt initialization and unstructured external vectors. The model starts from a more meaningful semantic prior, and online inference does not incur extra cost from this step.

We computed the label co-occurrence matrix $M\in {\mathbb{R}}^{C\times C}$ from the training annotations, where C is the number of categories. Each entry $M_{ij}$ denotes the conditional probability that category j appears when category i is present. We converted this co-occurrence similarity into a distance metric $D\in {\mathbb{R}}^{C\times C}$ so that standard clustering algorithms can be applied:

$$D_{ij}=1-{\displaystyle \frac{M_{ij}+M_{ji}}{2}}.$$

(2)

This symmetric matrix gives smaller distances to category pairs that often appear together.

We applied Agglomerative Hierarchical Clustering with average linkage to build a semantic taxonomy tree. Distance thresholds at different heights of the dendrogram split the categories into Fine, Middle, and Coarse topics. The leaves correspond to specific classes. These classes merge into mid-level topics and later into coarse macro-topics. As shown in the top-left of Figure 1, these topic clusters were used to initialize the keys of the prompt pools.

Figure 1. The overall architecture of the proposed DyT-HPL framework: (Top) Offline initialization: a label co-occurrence matrix is used for hierarchical clustering to initialize the prompt space, with an intra-pool diversity loss (${\mathcal{L}}_{\mathrm{div}}$) applied to encourage diverse prompt keys. In parallel, a frozen ViT branch extracts multi-granularity semantic queries ($q_{\mathrm{coarse}}$, $q_{\mathrm{mid}}$, $q_{\mathrm{fine}}$), which are supervised by a matching loss (${\mathcal{L}}_{\mathrm{match}}$). (Bottom) Online inference and interaction: the retrieved prompts are injected into the trainable main Vision Transformer. The final-layer query provides broad semantic guidance at early stages, while intermediate-layer queries provide more detailed semantic guidance at later stages. Within each injection stage, visual tokens and prompts are concatenated (Ⓒ) for interaction, after which the prompt tokens are removed (Ⓢ) to limit computational overhead. The entire network is optimized by ${\mathcal{L}}_{\mathrm{total}}$, including the final asymmetric loss (${\mathcal{L}}_{\mathrm{ASL}}$).

Based on this initialization, we used a progressive capacity and retrieval strategy. The shallow pool has a smaller capacity and uses top-1 retrieval for broad guidance. The middle and deep pools use larger capacities and top-k ($k>1$) retrieval to capture finer semantics. This coarse-to-fine setup follows the feature hierarchy of the backbone.

3.4. Frozen Query Branch

We used a frozen query branch to keep semantic queries separate from downstream gradient updates. This branch is an ImageNet-pretrained ViT [5] with all parameters fixed. It extracts query features from different depths:

$$q_{\mathrm{coarse}},q_{\mathrm{mid}},q_{\mathrm{fine}}=g_{\mathrm{frozen}}\left(x\right),$$

where $g_{\mathrm{frozen}}(·)$ denotes the auxiliary branch. Following the feature dynamics discussed in Section 2, we sampled queries from specific stages. The intermediate queries ($q_{\mathrm{mid}}$ and $q_{\mathrm{fine}}$) were taken from the CLS tokens of Layers 4 and 9, respectively. Here, the terms “mid” and “fine” refer to their functional roles at different stages of prompt injection in the main network, rather than their absolute extraction depths: the Layer-4 query provides complementary semantic cues at an intermediate stage, while the Layer-9 query was used at a later stage for more refined semantic guidance. The coarse query ($q_{\mathrm{coarse}}$) was taken from the final normalized class token at Layer 12, which provides the most abstract and globally contextualized representation. Because this branch stays frozen during training, prompt retrieval was performed in a stable reference space. Notably, $q_{\mathrm{mid}}$ and $q_{\mathrm{fine}}$ were extracted from intermediate CLS tokens, rather than from raw shallow patch features. Consequently, they represent less abstract yet still globally contextualized semantic queries relative to the final-layer CLS representation, rather than purely local low-level details.

3.5. Dynamic Prompt Retrieval and Hierarchical Injection

To retrieve relevant semantic prompts, we computed the cosine similarity between a query vector and the learnable keys in a prompt pool. Let $K\in {\mathbb{R}}^{M\times d}$ be the prompt-key matrix, where M is the pool size and d is the embedding dimension. We apply L2 normalization to both inputs and obtain similarity scores:

$$s=\widehat{q}{\widehat{K}}^{\top },$$

(3)

where $\widehat{q}$ and $\widehat{K}$ are the normalized tensors. From these scores, we selected the top-k matches and form the index set $\mathcal{I}$:

$$\mathcal{I}=\mathrm{TopK}(s,k).$$

(4)

The corresponding prompt values were concatenated along the sequence dimension to form the retrieved prompt block:

$$P_{\mathrm{retrieved}}^{\left(h\right)}=\mathrm{Concat}\left(\{P_i\mid i\in \mathcal{I}\}\right)\in {\mathbb{R}}^{L_p\times d},$$

(5)

where $P_i$ denotes the i-th prompt value token and $L_p$ defines the total length of the newly formed sequence.

The discrete TopK operation creates a training problem because it blocks gradient flow to the prompt keys. We addressed this by separating the optimization paths. The retrieved prompt values $\left\{P_i\right\}$ interact with visual tokens in self-attention and receive gradients from the main classification loss. The prompt keys K were updated through an independent surrogate matching loss (${\mathcal{L}}_{\mathrm{match}}$, detailed in Section 3.6). This gives the keys a stable and differentiable update path.

We injected the retrieved tokens $P_{\mathrm{retrieved}}^{\left(h\right)}$ directly into the main Vision Transformer. Let $X^{\left(l\right)}\in {\mathbb{R}}^{N\times d}$ be the visual tokens entering the l-th block, where N includes the image patches and the class token. We concatenate prompts and visual tokens to form an augmented input:

$${\tilde{X}}_{\mathrm{in}}^{\left(l\right)}=\left[X^{\left(l\right)}\Vert P_{\mathrm{retrieved}}^{\left(h\right)}\right]\in {\mathbb{R}}^{(N+L_p)\times d},$$

(6)

where $[·\Vert ·]$ denotes concatenation.

We use different positional treatments for the two token types. Visual tokens $X^{\left(l\right)}$ keep their spatial embeddings so that image topology is preserved. Retrieved prompts $P_{\mathrm{retrieved}}^{\left(h\right)}$ do not use positional encoding. They represent unordered semantic concepts rather than physical locations, so permutation invariance is more suitable.

The combined sequence is fed into a standard Transformer block, where multi-head self-attention lets visual features absorb the injected semantic priors:

$${\tilde{X}}_{\mathrm{out}}^{\left(l\right)}=\mathrm{TransformerBlock}({\tilde{X}}_{\mathrm{in}}^{\left(l\right)}).$$

(7)

Keeping these extra tokens through the whole network would lengthen the sequence and may introduce semantic cross-talk. We use a transient injection strategy. Once the attention step ends, the prompts are removed, and only the enriched visual tokens are passed to deeper layers:

$$X^{(l+1)}={\tilde{X}}_{\mathrm{out}}^{\left(l\right)}[1:N,:],$$

(8)

As shown in Figure 1 (bottom), this “use-and-discard” design keeps computation efficient and avoids unnecessary disturbance to the visual stream.

The injection locations are also chosen deliberately. This forms our cross-depth injection strategy. The coarse pool, retrieved by the final-layer query, is injected into the shallow block ($l=0$) to provide an early semantic bias. The intermediate and fine pools are injected into the middle ($l=5$) and deep ($l=10$) blocks to provide stage-specific semantic guidance and later-stage refinement, respectively.

This modular design also makes ablation analysis easier. By turning specific pools on or off, such as shallow-only injection versus the full hierarchy, we can directly observe how injection depth affects final performance.

3.6. Objective Function

The total objective of DyT-HPL has three parts: a multi-label classification loss, a surrogate matching loss for routing, and an intra-pool diversity term:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{ASL}}+{\lambda }_1{\mathcal{L}}_{\mathrm{match}}+{\lambda }_2{\mathcal{L}}_{\mathrm{div}},$$

(9)

where ${\lambda }_1$ and ${\lambda }_2$ balance semantic guidance and prompt-pool regularization during training.

Multi-Label Classification Loss (${\mathcal{L}}_{\mathrm{ASL}}$). Multi-label datasets usually contain many more negative than positive samples. Recent work has explored dynamic logit adjustment and re-weighting strategies for long-tailed multi-label learning [29]. In our model, we handled this imbalance with Asymmetric Loss (ASL) [30] (additional formulations are provided in Appendix B). Let $y\in {\{0,1\}}^C$ be the multi-hot ground truth and $p\in {[0,1]}^C$ the predicted probabilities obtained from a Sigmoid $\sigma (·)$ over logits $z_c$. The classification term is

$${\mathcal{L}}_{\mathrm{ASL}}={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}\left[-y_c{\mathcal{L}}_{+}^{\left(c\right)}-(1-y_c){\mathcal{L}}_{-}^{\left(c\right)}\right],$$

(10)

where the positive and negative parts are weighted by asymmetric focusing parameters ${\gamma }_{+}$ and ${\gamma }_{-}$:

$${\mathcal{L}}_{+}^{\left(c\right)}={(1-p_c)}^{{\gamma }_{+}}log\left(p_c\right),$$

(11)

$${\mathcal{L}}_{-}^{\left(c\right)}={\left(p_m\right)}^{{\gamma }_{-}}log(1-p_m).$$

(12)

The threshold margin $m\ge 0$ in $p_m=max(0,p_c-m)$ filters out easy negative samples. This reduces their gradients and lets the model focus more on rare positives and hard negatives.

Surrogate Matching Loss (${\mathcal{L}}_{\mathrm{match}}$). Prompt retrieval also needs explicit supervision. We used a continuous surrogate matching loss to align each query with its selected keys. Let the prompt-key matrix be $K=\{k_1,k_2,\dots ,k_M\}$. For a visual-semantic query $q\in \{q_{\mathrm{coarse}},q_{\mathrm{mid}},q_{\mathrm{fine}}\}$, we penalized the spherical distance between q and its top-k retrieved keys, indexed by ${\mathsf{\Omega }}_k\left(q\right)$:

$${\mathcal{L}}_{\mathrm{match}}=\sum _{j\in {\mathsf{\Omega }}_k\left(q\right)}\left(1-{\displaystyle \frac{q^{\top }{k}_j}{{\parallel q\parallel }_2{\parallel k_j\parallel }_2}}\right).$$

(13)

Since discrete routing is non-differentiable, this loss serves as a proxy objective. Because the query branch is frozen, ${\mathcal{L}}_{\mathrm{match}}$ updates only the selected keys, encouraging them to align with the stable query space induced by the target data. This design improves routing stability and reduces gradient interference, although it may limit task-specific adaptation when the pretrained query space is not fully aligned with the target domain.

Intra-Pool Diversity Loss (${\mathcal{L}}_{\mathrm{div}}$). Without regularization, dynamic routing often overuses a small subset of prompts, which leads to mode collapse [23,31]. We addressed this with an intra-pool diversity penalty. For a pool of size M, we computed the mean squared cosine similarity over all unique non-diagonal key pairs:

$${\mathcal{L}}_{\mathrm{div}}={\displaystyle \frac{2}{M(M-1)}}{\displaystyle \sum _{i=1}^{M-1}}{\displaystyle \sum _{j=i+1}^M}{\left({\displaystyle \frac{k_i^{\top }{k}_j}{{\parallel k_i\parallel }_2{\parallel k_j\parallel }_2}}\right)}^2.$$

(14)

Driving this similarity toward zero pushes the keys apart in latent space. This keeps the prompt library diverse instead of letting it collapse into redundant topics.

Although ${\mathcal{L}}_{\mathrm{div}}$ has an $O\left(M^2\right)$ complexity, its practical overhead is negligible in our setting because the prompt pools remain small and the computation is fully vectorized on GPU. Even for the largest pool, the number of pairwise similarities is far smaller than the pairwise interactions in a single standard ViT self-attention map. If larger pools are considered in future extensions, the cost can be reduced by sampling a subset of keys during training.

4. Experiments and Results

4.1. Experimental Setup

Datasets. We evaluate DyT-HPL on three widely used multi-label image classification benchmarks: MS-COCO [32], NUS-WIDE [33], and Corel5k [34] (summarized in Table 1).

• MS-COCO [32]: We use the standard split with 80 semantic categories. The dataset contains 122,218 images in total, including 82,081 for training and 40,137 for testing.
• NUS-WIDE [33]: This dataset contains real-world web noise and diverse annotations. We use its 81 visual concepts, with 161,789 images for training and 107,859 for testing.
• Corel5k [34]: This dataset is small and highly sparse in labels, so it is useful for testing recall on rare concepts. It contains 4999 images and 260 fine-grained categories. We follow the official split and use 4500 images for training and 499 for testing.

Table 1. Dataset statistics and evaluation splits used in our experiments.

Dataset	Classes	Training Images	Testing Images
MS-COCO	80	82,081	40,137
NUS-WIDE	81	161,789	107,859
Corel5k	260	4500	499

Table 1. Dataset statistics and evaluation splits used in our experiments.

Dataset	Classes	Training Images	Testing Images
MS-COCO	80	82,081	40,137
NUS-WIDE	81	161,789	107,859
Corel5k	260	4500	499

Evaluation Metrics. We use mean Average Precision (mAP) [1] as the main evaluation metric. We also report overall precision (OP), recall (OR), and F1-score (OF1), together with their class-wise versions (CP, CR, CF1) [35]. All metrics follow standard definitions. Detailed formulas are given in the Appendix A.

Implementation Details. DyT-HPL uses a ViT-Base backbone [5] initialized with ImageNet weights. The patch size is 16 and the embedding dimension is 768. The frozen query branch has the same architecture. We use three prompt pools for the shallow, middle, and deep stages, with sizes of 5, 10, and 20. The prompt length is 5, and the top-k values are $k=1,2,3$. For Corel5k, we use the standard train/test split. OR and OF1 are computed with a fixed threshold of 0.5, and mAP is averaged over valid classes. Baseline results are taken from the original publications unless otherwise specified.

Following standard ViT training recipes [36], all images are resized to $224\times 224$ and augmented with RandAugment and Cutout. We train the model for 50 epochs using AdamW, with weight decay set to 0.05 and a One-Cycle learning rate schedule that peaks at $1\times {10}^{-4}$. Batch sizes are 64 for MS-COCO, 128 for NUS-WIDE, and 32 for Corel5k. We also use an Exponential Moving Average (EMA) with decay of 0.9997 to stabilize training. The final objective includes the asymmetric classification loss [30], the surrogate matching loss (${\lambda }_1=0.5$), and the diversity term (${\lambda }_2=0.2$).

4.2. Comparison Methods

We compare DyT-HPL with three groups of baselines: conventional deep models based on CNNs plus recurrent or graph modules, standard Transformer-based methods, and recent prompt-tuning or semantic-guided approaches.

4.3. Main Results and Comparisons

We now compare DyT-HPL with representative methods on MS-COCO, NUS-WIDE, and Corel5k.

Performance on MS-COCO. DyT-HPL reaches 83.5% mAP at $224\times 224$ resolution (Table 2). This is 3.1% higher than Query2Label [6] and 2.4% higher than ML-Decoder [17]. Compared with recent prompt-tuning methods such as E²VPT [19], our model shows a better precision–recall balance. E²VPT has higher recall (75.7% CR), but DyT-HPL gives higher precision (81.8% CP) and achieves the best CF1 (76.4%) and OF1 (78.5%). These results suggest that hierarchical prompt injection helps model complex semantic dependencies.

Table 2. Performance comparison with representative methods on the MS-COCO dataset. All models are evaluated at a unified $224\times 224$ input resolution. The best results in each column are in bold, and the second best are underlined.

Method	mAP	CF1	CP	CR	OF1	OP	OR
CNN-RNN [3]	61.2	60.4	66.0	55.6	69.2	84.0	66.4
RDAR [37]	73.4	67.4	79.1	58.7	72.0	84.0	63.0
KD-WSD [38]	74.6	66.8	72.7	-	-	-	-
ResNet-SRN-att [39]	76.1	70.0	81.2	63.3	75.0	84.1	67.7
ResNet-SRN [39]	77.1	71.2	81.6	65.4	75.8	82.7	69.9
ResNet-101 [40]	78.3	69.5	80.8	63.4	74.4	82.2	68.0
Query2Label [6]	80.4	72.7	81.9	65.6	-	-	-
IDA-SwinS [15]	80.6	71.8	81.1	64.5	73.8	65.8	65.8
GateVPT [18]	80.8	74.2	74.8	74.5	76.9	77.8	77.3
VPT [14]	81.0	74.5	75.1	74.8	77.3	78.1	77.7
ML-Decoder [17]	81.1	74.5	83.0	67.6	-	-	-
E2VPT [19]	81.9	75.4	76.0	75.7	77.9	78.8	78.3
DyT-HPL (Ours)	83.5	76.4	81.8	71.6	78.5	84.3	73.4

Performance on NUS-WIDE. NUS-WIDE contains noisy web images, so semantic discrimination is more difficult. DyT-HPL achieves 66.3% mAP (Table 3). This is 4.9% higher than MS-CMA [41] and 1.3% higher than Q2L-R101 [6]. It also matches the best mAP reported by Q2L-TResL [6], while giving a slightly higher CF1-score (64.3% vs. 64.0%). These results suggest that the proposed framework remains competitive under label noise, with slight improvements in CF1 and OF1.

Table 3. Comparison of our method with representative models on the NUS-WIDE dataset. All results are reported at the input resolution of $224\times 224$. The best results are in bold, and the second best are underlined.

Method	mAP	CF1	OF1
ResNet-101 [40]	59.8	55.7	72.5
MS-CMA [41]	61.4	60.5	73.8
ResNet-SRN [39]	62.0	58.5	73.4
ICME [42]	62.8	60.7	74.1
TresNet [43]	63.1	61.7	74.6
CCD-R101 [44]	64.2	61.8	74.6
Q2L-R101 [6]	65.0	63.1	75.0
SADCL [16]	65.9	63.0	75.0
Q2L-TResL [6]	66.3	64.0	75.0
DyT-HPL (Ours)	66.3	64.3	75.2

Performance on Corel5k. Corel5k has sparse annotations and fine-grained labels, so recall is especially hard to improve. DyT-HPL achieves the best overall results among the compared methods, reaching 80.2% Overall Recall (OR) and 58.8% OF1 (Table 4). Given the backbone differences among methods, especially versus earlier CNN- or kernel-based models, part of this large margin may also benefit from ViT pretraining. Therefore, these results mainly support the effectiveness of the overall framework in sparse-label settings.

Table 4. Comparison of our method with representative models in terms of OR, OP, and OF1 on the Corel5k dataset. All results are reported at the input resolution of $224\times 224$.

Method	OR	OP	OF1
FT DMN + SVM	38.1	23.4	28.9
CNN-RNN [3]	41.3	32.0	36.0
3-layer DKN + SVM	43.2	25.6	32.1
DCKN [45]	44.1	33.4	38.1
LNR + 2PKNN [46]	46.1	44.2	44.9
Q2L-TResL [6]	48.1	43.5	45.7
DMCKN-Cvt [47]	49.1	45.2	47.0
DyT-HPL (Ours)	80.2	45.7	58.8

4.4. Ablation Studies

To examine the main components of DyT-HPL, we conduct targeted ablation studies on the benchmark datasets:

• Hierarchical Injection (MS-COCO): We compare the standard ViT (Baseline) with three prompt settings: Single-Shallow, Single-Deep, and the Full coarse-to-fine hierarchy, to isolate the role of multi-scale alignment.
• Generalization (NUS-WIDE & Corel5k): We compare the Baseline and the Full model to test whether the gains remain consistent under different data distributions.
• Effect of Model Components (Corel5k): We further ablate the two auxiliary losses and the frozen query branch to verify the contribution of each core design component.
• Sensitivity to Hierarchical Granularity (Corel5k): We vary the coarse/mid/fine prompt-pool sizes around the default setting to examine the robustness of DyT-HPL to moderate changes in the offline hierarchical structure.
• Statistical Significance (Corel5k): We employed both the parametric paired t-test and the non-parametric Wilcoxon signed-rank test [48,49] to rigorously evaluate whether our performance improvements are statistically significant.

Hierarchical Injection Strategies. On MS-COCO, using prompts only in shallow blocks or only in deep blocks gives limited gains (Table 5). In contrast, the full cross-depth hierarchy performs best and achieves the lowest Hamming loss and One-error. This suggests that distributing semantic guidance across multiple stages is more effective than relying on a single fixed injection level, although the current strategy remains manually specified rather than dynamically learned.

Table 5. Ablation study on prompt injection strategies evaluated on the MS-COCO dataset. The downward arrow (↓) indicates that a smaller value corresponds to better performance.

Setting	mAP	Hamming Loss ↓	One-Error ↓
ViT Baseline	81.7	5.2	3.5
Single-Shallow Injection	83.3	3.6	3.2
Single-Deep Injection	82.9	3.8	3.4
DyT-HPL (Full)	83.5	3.5	3.1

Generalization Across Datasets. We also compare the ViT baseline and the full model on NUS-WIDE and Corel5k. Table 6 shows that DyT-HPL improves over the baseline on both datasets. The larger gain on Corel5k suggests that offline hierarchical clustering is especially useful when labels are sparse.

Table 6. Performance generalization: Baseline vs. Full DyT-HPL across NUS-WIDE and Corel5k datasets.

Dataset	Method	mAP
NUS-WIDE	ViT Baseline	62.8
	DyT-HPL (Full)	66.3
Corel5k	ViT Baseline	60.6
	DyT-HPL (Full)	63.7

Effect of Model Components. We further ablate the core components of DyT-HPL on Corel5k. As shown in Table 7, removing either ${\mathcal{L}}_{\mathrm{div}}$ or ${\mathcal{L}}_{\mathrm{match}}$ reduces performance, and removing both causes a larger drop, indicating that the two losses are complementary. Replacing the frozen query branch with a trainable one further reduces the mAP from 63.72% to 63.07% and the OF1 from 58.80% to 55.60%, confirming the benefit of a stable frozen query space.

Table 7. Ablation study on the core components of DyT-HPL on the Corel5k dataset.

Variant	mAP (%)	OF1 (%)
DyT-HPL (Full)	63.72	58.80
w/o ${\mathcal{L}}_{\mathrm{div}}$	63.36	55.20
w/o ${\mathcal{L}}_{\mathrm{match}}$	63.12	55.20
w/o both	62.53	54.70
Trainable Query Branch	63.07	55.60

Sensitivity to Hierarchical Granularity. We vary the coarse/mid/fine prompt-pool sizes around the default setting on Corel5k to assess the sensitivity of DyT-HPL to the offline hierarchical structure. As shown in Table 8, the model remains stable near the default configuration, while overly small or large pool allocations lead to moderate degradation. This suggests that DyT-HPL is not tied to a single fragile clustering choice, but still benefits from a reasonable hierarchical granularity.

Table 8. Sensitivity analysis of hierarchical prompt-pool granularity on the Corel5k dataset.

Pool Size Setting (Coarse/Mid/Fine)	mAP (%)	OF1 (%)
3/6/12	63.44	52.0
4/8/16	63.32	56.9
5/10/20	63.72	58.8
6/12/24	63.71	58.1
7/14/28	62.21	55.3

Statistical Significance. On Corel5k, we further test the performance gain over the baseline with the Paired t-test [48] ($p=0.028$) and the Wilcoxon Signed-Rank Test [49] ($p=0.022$). Both results satisfy $p<0.05$, which indicates that the improvement is statistically significant.

4.5. Qualitative Analysis and Visualization

Figure 2 puts DyT-HPL’s multi-granularity alignment to a direct visual test. Using the deeply hierarchical Corel5k dataset [34] as our testbed, we unpack three representative prediction scenarios.

Figure 2. Qualitative results of DyT-HPL on the Corel5k dataset. GT indicates the ground-truth labels, and Pred indicates the predicted labels. (a,b) illustrate successful cases of multi-granularity semantic prediction, while (c) shows a failure case of “semantic over-extension”, where the model predicts additional scene-related concepts.

When faced with wide semantic divides (Figure 2a), the framework seamlessly bridges the gap. It locks onto the sweeping background concept (sky) while simultaneously pinpointing hyper-specific foreground objects (jet, plane).

Figure 2b pushes this further by testing associative taxonomy. Corel5k is notorious for burying labels within complex hierarchical contexts. In this image, the raw pixels simply show a tiger in tall grass. Yet, the ground truth demands both the biological hypernym (cat) and the broader habitat (forest). DyT-HPL nails every single label. Rather than just skimming surface-level appearance cues, the model actively penetrates the underlying semantic taxonomy and contextual co-occurrences.

For completeness, we also present a typical failure case in Figure 2c. While the model correctly retrieves the ground-truth labels, it also outputs several false-positive concepts (e.g., boats, beach, sunset). This phenomenon, which we refer to as “semantic over-extension,” may occur because the prompt learning mechanism captures scene-level semantic correlations that are not fully constrained by the visual evidence in the image. Introducing stronger spatial attention constraints may help reduce this effect in future work.

4.6. Efficiency Analysis

In this subsection, we analyze the computational overhead and parameter efficiency of the proposed DyT-HPL framework. Table 9 reports the total, trainable, and frozen parameter counts, FLOPs, and inference latency under the same input resolution. Compared with the vanilla ViT baseline, DyT-HPL increases the total parameters from 86.00 M to 171.96 M, the FLOPs from 16.85 G to 33.92 G, and the latency from 3.19 to 6.52 ms/image, while adding only 0.16 M trainable parameters. This shows that the prompt pools are lightweight and that the main overhead comes from the auxiliary frozen query branch.

Table 9. Efficiency comparison between DyT-HPL and the ViT baseline under $224\times 224$ resolution.

Method	Total (M)	Train. (M)	Frozen (M)	FLOPs (G)	Latency (ms)
ViT Baseline	86.00	86.00	0.00	16.85	3.19
DyT-HPL	171.96	86.16	85.80	33.92	6.52
Increase	+85.96	+0.16	+85.80	+17.07	+3.33
Relative (%)	(+99.9%)	(+0.19%)	(–)	(+101.3%)	(+104.4%)

5. Discussion

DyT-HPL relies on three closely related components. The frozen query branch gives a stable reference space and separates prompt retrieval from continuous backbone updates. The clustered prompt pools provide semantic guidance step by step. Different Transformer depths receive different levels of semantic cues. A diversity penalty keeps the prompt keys apart and prevents redundant patterns from dominating the pool.

The experiments suggest that this design is useful for complex semantic scenes. Visual understanding often requires both global context and local detail. Multi-stage guidance helps connect these two levels. This is especially important on datasets such as Corel5k [34], where labels are sparse and entangled. In such cases, DyT-HPL uses statistical priors to strengthen the Vision Transformer and partly compensate for the limits of standard self-attention.

Limitations: The framework still depends on good initialization. Since the prompt hierarchy is built from offline label co-occurrence clustering and kept fixed during training, noisy or misaligned semantic groups may weaken downstream guidance. The gain also depends on the dataset scale and label density. In addition, the current prompt injection points are fixed at shallow, middle, and deep blocks, which may not be optimal for all inputs or architectures. Future work will explore more adaptive hierarchy construction and learnable prompt injection mechanisms.

6. Conclusions

This paper presents DyT-HPL for multi-label image classification. The framework combines offline hierarchical clustering [12], a frozen query branch, and dynamic prompt retrieval. Under flat label supervision, it improves visual-semantic alignment at multiple granularities. Coarse-to-fine prompts are injected into different depths of the Vision Transformer [5], so the model can use broad semantic cues and still refine local details.

The optimization objective combines asymmetric classification loss [30], surrogate matching loss, and intra-pool diversity loss.

Experiments on MS-COCO [32], NUS-WIDE [33], and Corel5k [34] support this design. DyT-HPL achieves competitive performance while adding only a small number of trainable parameters; the additional inference cost mainly comes from the frozen query branch. The ablation results and statistical tests also clarify the role of each core component.

Future work will examine stronger semantic initialization, dynamic prompt length scaling, and more adaptive injection strategies. These directions may further improve representation learning for large-scale multi-label visual tasks.

Appendix A.1. Class-Wise Metrics (Macro-Averaging)

The class-wise metrics evaluate the performance of each category independently and compute the unweighted mean across all categories. They are defined as follows:

$$CP={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}{\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c}}$$

(A1)

$$CR={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}{\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}}$$

(A2)

$$CF1={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}{\displaystyle \frac{2\times P_c\times R_c}{P_c+R_c}}$$

(A3)

where $P_c$ and $R_c$ are the precision and recall for the c-th category, respectively.

Appendix A.2. Overall Metrics (Micro-Averaging)

The overall metrics aggregate the contributions of all classes to compute average metrics, which is particularly useful for evaluating performance on datasets with class imbalance:

$$OP={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^C(T{P}_c+F{P}_c)}}$$

(A4)

$$OR={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^C(T{P}_c+F{N}_c)}}$$

(A5)

$$OF1={\displaystyle \frac{2\times OP\times OR}{OP+OR}}$$

(A6)

Appendix A.3. Mean Average Precision (mAP)

The Average Precision (AP) for a specific class c summarizes the shape of the precision–recall curve and is defined as the integral of precision $p_c\left(r\right)$ over all recall levels r:

$$A{P}_c={\int }_0^1{p}_c\left(r\right)dr$$

(A7)

In practice, this is computed as the weighted sum of precisions at each threshold, with the weight being the increase in recall from the previous threshold. The mean Average Precision (mAP) is the mean of the AP scores across all C categories:

$$mAP={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}A{P}_c$$

(A8)