Dependency-Aware Entity–Attribute Relationship Learning for Text-Based Person Search

Abstract

Text-based person search (TPS), a critical technology for security and surveillance, aims to retrieve target individuals from image galleries using textual descriptions. The existing methods face two challenges: (1) ambiguous attribute–noun association (AANA), where syntactic ambiguities lead to incorrect associations between attributes and the intended nouns; and (2) textual noise and relevance imbalance (TNRI), where irrelevant or non-discriminative tokens (e.g., ‘wearing’) reduce the saliency of critical visual attributes in the textual description. To address these aspects, we propose the dependency-aware entity–attribute alignment network (DEAAN), a novel framework that explicitly tackles AANA through dependency-guided attention and TNRI via adaptive token filtering. The DEAAN introduces two modules: (1) dependency-assisted implicit reasoning (DAIR) to resolve AANA through syntactic parsing, and (2) relevance-adaptive token selection (RATS) to suppress TNRI by learning token saliency. Experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid demonstrate state-of-the-art performance, with the DEAAN achieving a Rank-1 accuracy of 76.71% and an mAP of 69.07% on CUHK-PEDES, surpassing RDE by 0.77% in Rank-1 and 1.51% in mAP. Ablation studies reveal that DAIR and RATS individually improve Rank-1 by 2.54% and 3.42%, while their combination elevates the performance by 6.35%, validating their synergy. This work bridges structured linguistic analysis with adaptive feature selection, demonstrating practical robustness in surveillance-oriented TPS scenarios.

1. Introduction

Text-based person search (TPS) [1,2,3,4,5,6,7,8] has revolutionized cross-modal retrieval by enabling targeted individual identification from large-scale image databases through natural language queries. This technology holds transformative potential for public security and intelligent surveillance systems [9,10], in which precise person retrieval from textual descriptions is critical for real-world applications like suspect tracking and missing person identification.

The existing approaches primarily focus on two technical paradigms to bridge the visual–textual gap: (i) global-matching methods [11,12,13], which extract holistic features via contrastive learning, and (ii) local-matching methods [14,15,16,17], which align textual entities to visual body regions for fine-grained matching. Recent advancements [1,18,19,20] further integrate pre-trained vision–language models, such as CLIP [21], BLIP [22], and ALBEF [23], to enhance feature representation. However, these methods predominantly operate on limited-scale datasets (e.g., CUHK-PEDES [2], ICFG-PEDES [24], and RSTPReid [25] with approximately 40,000 images), thus limiting their capacity to handle complex linguistic variations in real-world scenarios. Researchers in other fields also pay great attention to and utilize machine learning to solve this problem. For instance, entity alignment in multi-lingual, temporal, and probabilistic knowledge graphs has been explored in dynamic contexts, such as weather forecasting and medical diagnosis [26], emphasizing the need for reasoning over uncertain and structurally complex information. Similarly, tweet prediction models in social media utilize machine learning to infer semantic and contextual nuances in short informal text sequences [27].

In real-world surveillance and public safety scenarios, the accuracy of text-based person search directly influences the timeliness and reliability of identity verification tasks. Descriptions collected from eyewitnesses or field operators often contain ambiguous or redundant language, which significantly challenges the current retrieval systems. Specifically, two fundamental linguistic challenges remain under-addressed in the current TPS systems. First, the ambiguous attribute–noun association (AANA) problem arises from syntactic complexity in natural language. As illustrated in Figure 1, the description “a yellow shirt, black and loose fitting pants” may erroneously associate “black” with “shirt” owing to parsing ambiguities, substantially reducing the retrieval accuracy. Although recent work [1] attempts cross-modal alignment, it overlooks the structured linguistic analysis critical to addressing these challenges. Second, textual noise and relevance imbalance (TNRI) occurs when non-discriminative tokens (e.g., “wearing”) dominate text representations, obscuring critical visual attributes. Addressing issues such as attribute–noun ambiguity and irrelevant token interference is essential for improving the reliability of these systems in high-stakes environments like criminal investigations or emergency response.

To address these limitations, we propose the dependency-aware entity–attribute alignment network (DEAAN), a novel framework integrating grammatical analysis with adaptive feature selection. Our key innovations include the following:

• Dependency-Assisted Implicit Reasoning (DAIR): A syntactic parsing module to resolve AANA by reconstructing the grammatical dependency between attributes and nouns using dependency trees, as shown in Figure 1.
• Relevance-Adaptive Token Selection (RATS): A saliency-driven mechanism to suppress TNRI by dynamically filtering out irrelevant tokens and amplifying discriminative features.

Extensive experiments validate the effectiveness of the DEAAN across multiple TPS benchmarks. In particular, ablation studies demonstrate that our two key modules (DAIR and RATS) not only function effectively in isolation but also work synergistically to significantly improve performance. Our work advances TPS by integrating linguistic structural analysis with adaptive feature learning, setting new directions for robust text–visual alignment. In the next section, we review the related work on TPS and attention mechanisms incorporating syntactic information.

2. Related Work

In this section, we first discuss the general challenges in TPS, followed by an overview of global and local matching techniques, and then we explore methods incorporating syntactic information into attention mechanisms.

2.1. Text-Based Person Search

Text-based person search (TPS) is a novel and challenging task that aims to match a person image with a given natural language description [2,4,18,28,29,30,31,32,33]. The existing TPS methods could be roughly classified into two groups according to their alignment levels, i.e., global-matching methods [11,25,34] and local-matching methods [16,17,35]. The former try to learn cross-modal embeddings in a common latent space by employing textual and visual backbones with a matching loss (e.g., CMPM/C loss [12] and Triplet Ranking loss [36]) for TPS. However, these methods mainly focus on global features while ignoring the fine-grained interactions between local features, which limits their performance improvement. To achieve fine-grained interactions, some of the latter methods explore explicit local alignments between body regions and textual entities for more refined alignments. However, these methods require more computational resources due to the complex local-level associations. Recently, inspired and benefited from vision–language pre-training models [37], some methods [1,21,38] expect to use the learned rich alignment knowledge of pre-trained models for local- or global alignments.

CLIP offers strong zero-shot alignment but lacks task-specific grounding; BLIP improves grounded generation via caption pre-training; ALBEF introduces momentum distillation for more stable multimodal learning. Although these methods achieve promising performance, they often fail to capture fine-grained attribute–entity relations, particularly when facing ambiguous or complex syntax structures.

Although TPS is primarily framed as a supervised task that leverages annotated image–text identity pairs, recent work [39,40] has shown the potential benefits of unsupervised learning as an auxiliary strategy. For instance, Sinaga and Yang [40] proposed a globally collaborative multi-view k-means clustering method (G-CoMVKM) that balances local view-specific features with global alignment via entropy-regularized dimensionality reduction. Recent TPS works [1,18,24,41,42,43,44] have followed this trend. They enhance robustness to noise, or adapt to unlabeled surveillance data. At the global and local scales, SSAN [24] embeds visual features and text features into potential common spaces and brings similar graphic and text pairs closer to assist in its supervised learning. Another similar process is RASA [18]. SRCF [41] first performs unsupervised segmentation of text features and the foreground and background of visual features to achieve noise filtering and then conducts supervised learning on the obtained clean features. IRRA [1] proposes Similarity Distribution Matching (SDM), which minimizes the KL divergence between image–text similarity score distributions and image–text matching distributions. It can effectively enlarge the variance between non-matching pairs and the correlation between matching pairs.

2.2. Attention Using Syntactic Information

An attention mechanism is crucial in the field of natural language processing, helping models to focus on key parts of input. However, a traditional attention mechanism [45] often does not fully consider the syntactic structure of language, which can be a shortcoming in tasks requiring in-depth understanding of language relationships. Thus, researchers have explored a variety of ways to incorporate syntactic information into attention mechanisms. We divide these approaches into two main categories: syntactic-directed attention and syntactic-fused attention.

Those methods [46,47,48,49] based on syntactic-directed attention basically operate between features according to the dependency parsing tree of the sentence. Among them, SynGen [47] aligns attention maps with syntactic bindings between entities and attributes to improve the faithfulness of text-to-image generation. Duan [48] proposed a syntax-aware data augmentation strategy that adjusts word replacement probabilities based on syntactic roles, improving translation quality and structural coherence.

The syntactic-fused attention approach deeply imparts syntactic information into the process of attention calculation so that the model’s attention allocation is closely integrated with the syntactic structure. Specifically, such methods [50,51,52,53] encode information such as dependencies (or syntactic distances) into a dependency mask (or weight) matrix, which is then used as syntactic knowledge and fused with the attention map. Bugliarello [50] integrated a parameter-free dependency-aware self-attention mechanism into the transformer architecture to enhance machine translation, especially in long or low-resource sentences. Li [51] developed a syntax-aware local attention mechanism for BERT, allowing the model to focus more effectively on syntactically relevant words, thus improving sentence-level tasks.

In addition, we adopt the SpaCy [54] parser. The SpaCy dependency parser is a widely used tool for syntactic analysis that parses input text into a dependency tree, where each node corresponds to a word and the directed edges represent grammatical relationships (e.g., subject–object and adjectival modifier). It provides a solid foundation for the DAIR module, allowing the DEAAN framework to resolve ambiguous attribute–noun associations by leveraging the syntactic structure of the sentence.

3. Methodology

In this section, we introduce the dependency-aware entity–attribute alignment network (DEAAN), illustrated in Figure 2. The methodology is divided into three main components: (1) dual-stream feature extraction, (2) dependency-aware alignment, and (3) relevance-adaptive token selection.

3.1. Variable and Abbreviation Definitions

All variable and abbreviation definitions used in this section are summarized in

Table 1. Variable definitions.

Variable	Definition
$\mathcal{V}$	Image set
$\mathcal{T}$	Text set
$I_i$	The $i\mathrm{th}$ image of $\mathcal{V}$
$T_i$	The $i\mathrm{th}$ image of $\mathcal{T}$
$\widehat{T_i}$	The $i\mathrm{th}$ masked text of $\mathcal{T}$
$f_i^v$	The visual features about $I_i$ extracted through image encoder
$f_i^t$	The textual features about $T_i$ extracted through text encoder
$\mathcal{D}$	Dependency matrix
$\mathbf{G}$	Attention guidance after Gaussian smoothing of $\mathcal{D}$
$\mathcal{M}$	The index set of masked textual tokens
${\mathcal{V}}_{vocab}$	Vocabulary
${\mathbf{A}}_i^v$	The attention map of $I_i$ from the last transformer block of image encoder
${\mathbf{A}}_i^t$	The attention map of $T_i$ from the last transformer block of text encoder
${\mathbf{K}}_i^v$	The index set of the top-$\kappa$-selected local visual tokens
${\mathbf{K}}_i^t$	The index set of the top-$\kappa$-selected local textual tokens
${\mathbf{v}}_{ts}^i$	Visual embedding after token selection
${\mathbf{t}}_{ts}^i$	Textual embedding after token selection

and 

Table 2. Abbreviation definitions.

Abbreviation	Definition
TPS	Text-based Person Search
DEAAN	Dependency-Aware Entity–Attribute Alignment Network
DMC	Dependency Mask Calculation
AANA	Ambiguous Attribute–Noun Association
TNRI	Textual Noise and Relevance Imbalance
DAIR	Dependency-Assisted Implicit Reasoning
RATS	Relevance-Adaptive Token Selection
DISA	Dependency Intervention Self-Attention
BITC	Basic Image–Text Contrast
TSITC	Token-Selected Image–Text Contrast
RE loss	Relevance Enhancement loss

to avoid ambiguity and ensure a comprehensive understanding of the DEAAN framework.

3.2. Feature Extraction Encoder

Our framework adopts a dual-stream encoder to extract discriminative features from both visual and textual modalities, building upon the CLIP architecture.

Visual Feature Extraction. Given an input image $I_i\in \mathcal{V}$, the image encoder (IE) of CLIP based on a transformer architecture extracts an image feature. Specifically, the image is partitioned into N fixed-size patches, each projected into a high-dimensional vector. These vectors are processed through multiple transformer layers to encode N features as follows:

$$f_i^v=\mathrm{IE}\left(I_i\right)={\{v_{\mathrm{cls}}^i,v_1^i,\dots ,v_N^i\}}^{\top }\in {\mathbb{R}}^{(N+1)\times d},$$

(1)

where $v_{\mathrm{cls}}^i$, as a class token, represents the global image representation.

Textual Feature Extraction. For an input text description $T_i\in \mathcal{T}$, we first tokenize it using Byte-Pair Encoding (BPE) [55] with a vocabulary size of 49,152. The token sequence, added with two special tokens (i.e., start of sequence [SOS] and end of sequence [EOS]), is encoded by a transformer-based text encoder (TE) as follows:

$$f_i^t=\mathrm{TE}\left(T_i\right)={\{t_{\mathrm{s}}^i,t_1^i,\dots ,t_L^i,t_e^i\}}^{\top }\in {\mathbb{R}}^{(L+2)\times d},$$

(2)

where $t_{\mathrm{s}}^i$ and $t_{\mathrm{e}}^i$ are the features of [SOS] and [EOS] tokens, and $t_{\mathrm{s}}^i$, as a class token, captures global textual semantics. L is the number of the word-level local features.

3.3. Dependency Mask Calculation

This subsection details our syntactic dependency integration strategy through four key steps:

Step 1: Dependency Tree Construction. Using SpaCy dependency parser, we first parse the input text $T_i$ into a syntactic dependency tree. Each node represents a word, with directed edges indicating grammatical relationships (e.g., nominal subject and adjectival modifier).

Step 2: Word-to-Word Dependency. Let $\mathcal{P}\left(\epsilon \right)$ denote the dependency distance (i.e., the number of arrows) between each word and the word $\epsilon$ in the dependency tree. We construct a dependency distance matrix $\mathcal{D}\in {\mathbb{R}}^{L\times L}$, where each element is as follows:

$$d_{\epsilon ,j}=\left\{\begin{array}{cc}\mathrm{dependency}\mathrm{distance}\mathrm{between}\mathrm{word}\epsilon \mathrm{and}\mathrm{word}j,\hfill & j=1,\dots ,L\mathrm{and}j\ne \epsilon \hfill \\ 0,\hfill & j=\epsilon \hfill \end{array}\right..$$

(3)

Step 3: Gaussian Mask Regularization. We transform discrete distances into continuous attention guidance through Gaussian smoothing:

$${\mathbf{G}}_{\epsilon ,j}={\displaystyle \frac{1}{\sigma \sqrt{2\pi }}}exp\left(-{\displaystyle \frac{{\left(d_{\epsilon ,j}\right)}^2}{2{\sigma }^2}}\right),$$

(4)

where $\sigma =1.2$ controls spatial decay. The zero-centered Gaussian ($\mu =0$) prioritizes the word itself. As shown in Figure 3, an example of the word $\epsilon$ “The” and the word j “white” is highlighted in red.

3.4. Dependency-Assisted Implicit Reasoning

This subsection outlines a novel module that enhances transformer-based models by integrating grammatical dependencies and visual context, detailing the dependency intervention self-attention mechanism and Masked Language Modeling with visual interaction.

3.4.1. Dependency Intervention Self-Attention

We propose a syntax-enhanced attention mechanism that injects grammatical dependencies into the transformer architecture. Given the standard self-attention formulation,

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right)V.$$

(5)

Dependency intervention self-attention (DISA) integrates the dependency mask $\mathbf{G}$ (from Equation (4)) through additive intervention:

$${\mathrm{Attention}}_{\mathrm{DISA}}(Q,K,V,\mathbf{G})=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}+\lambda \mathbf{G}\right)V,$$

(6)

where $\lambda$ is a hyperparameter that controls the influence of the dependency information on the attention scores. Q, K, and V are the query, key, and value matrices, respectively. d is the dimension of the key vectors, serving as a scaling factor to stabilize the gradients.

3.4.2. Masked Language Modeling with Visual Context

We utilize MLM to predict masked textual tokens not only by the rest of unmasked textual tokens but also by the visual tokens. First, 15% of text tokens are randomly replaced with [MASK] [1], defined as

$${\widehat{T}}_i=\{{\widehat{t}}_s^i,{\widehat{t}}_1^i,\dots ,\underset{{\mathcal{M}}_k}{\underbrace{\left[\mathrm{MASK}\right]}},\dots ,{\widehat{t}}_L^i,{\widehat{t}}_e^i\}.$$

(7)

The masked text sequence interacts with visual features via cross-modal attention. The output of the interaction is fed to a special transformer architecture, defined as ${\mathrm{Transformer}}_{\mathrm{DISA}}$, whose self-attention mechanism is refined to our proposed ${\mathrm{Attention}}_{\mathrm{DISA}}$.

$${\left\{h_j\right\}}_{j=1}^L={\mathrm{Transformer}}_{\mathrm{DISA}}\left(\mathrm{MCA}(\mathrm{LN}(\underset{Q}{\underbrace{\mathrm{TE}\left({\widehat{T}}_i\right)}},\underset{K,V}{\underbrace{\mathrm{IE}\left(I_i\right)}})\right),$$

(8)

where $LN(·)$ and $MCA(·)$ denote the layer-normalization and the multi-head cross-attention, respectively. For masked position ${\{h_j:j\in \mathcal{M}\}}_{j=1}^L$, we use a Multi-Layer Perception (MLP) classifier to predict corresponding probability with original tokens ${\left\{p_k^j\right\}}_{k=1}^{\left|{\mathcal{V}}_{vocab}\right|}=MLP\left(h_j\right)$. The implicit alignment loss combines language reconstruction with identity-aware balancing as follows:

$${\mathcal{L}}_{\mathrm{cmia}}=-{\displaystyle \frac{1}{\left|\mathcal{M}\right||{\mathcal{V}}_{vocab}|}}\sum _{j\in \mathcal{M}}\sum _{k\in {\mathcal{V}}_{vocab}}{y}_k^{j}log{\displaystyle \frac{{\mathcal{F}}_{id}^i·exp\left(p_k^j\right)}{{\sum }_{l=1}^{|{\mathcal{V}}_{vocab}|}exp\left(p_l^j\right)}},$$

(9)

where $\mathcal{M}$ denotes the index set of masked text tokens and $|{\mathcal{V}}_{vocab}|$ is the size of vocabulary ${\mathcal{V}}_{vocab}$. ${\mathcal{F}}_{id}^i$ is the class weight for identity label; rare classes can be assigned higher weights in the statistical process.

3.5. Relevance-Adaptive Token Selection

This subsection introduces a RATS module that enhances image–text alignment by leveraging basic and token-selected contrastive learning, coupled with a novel relevance enhancement loss to capture fine-grained cross-modal correspondences.

3.5.1. Basic Image–Text Contrast

For any image–text pair $(I_i,T_j)$, we can directly use the global features of [CLS] and [SOS] tokens to compute similarity as basic image–text contrast (BITC) by the cosine similarity as follows:

$$s\left({\mathbf{v}}_{cls}^i,{\mathbf{t}}_s^j\right)={\displaystyle \frac{{\mathbf{v}}_{cls}^{i\top }{\mathbf{t}}_s^j}{\parallel {\mathbf{v}}_{cls}^i\parallel \parallel {\mathbf{t}}_s^j\parallel }}.$$

(10)

However, optimizing the BITC similarities alone may not capture the fine-grained interactions between two modalities, which will limit performance improvement. To address this issue, in Section 3.5.2, we exploit the local features of informative tokens to learn more discriminative embedding representations, thus mining the fine-grained correspondences.

3.5.2. Token-Selected Image–Text Contrast

Significant Token Selection. We select informative tokens based on the correlation between local tokens and class token [4,56]. Specifically, for image $I_i$ and text $T_i$, the attention maps ${\mathbf{A}}_i^v\in {\mathbb{R}}^{(1+N)\times (1+N)}$ and ${\mathbf{A}}_i^t\in {\mathbb{R}}^{(2+L)\times (2+L)}$ are extracted from the last transformer block of image encoder and text encoder, respectively. Then, we select top-$\kappa$ tokens using first-row attention weights (class-to-local correlation) and record them as the set of indices ${\mathbf{K}}_i$ for the selected local tokens as follows:

$$\begin{array}{cc}\hfill {\mathbf{K}}_i^v& =\mathrm{Top}\text{-}\mathrm{k}({\mathbf{A}}_i^v[0,1:N+1],⌊\kappa N⌋),\hfill \\ \hfill {\mathbf{K}}_i^t& =\mathrm{Top}\text{-}\mathrm{k}({\mathbf{A}}_i^t[0,1:L+1],min(⌊\kappa L_m⌋,L)),\hfill \end{array}$$

(11)

where $\kappa$ controls selection ratio, $⌊·⌋$ denotes round down, and $L_m$ is the maximum input sequence length of $f_i^t$.

Cross-Modal Feature Aggregation. We perform an embedding transformation on these selected token features to obtain subtle representations. Specifically, taking image $I_i$ as an example, ${\widehat{f}}_i^v$ is obtained by selecting the local tokens of $f_i^v$ according to the index set ${\mathbf{K}}_i^t$. Then, transform selected tokens into compact representations by an embedding module like the residual block [57].

$$\begin{array}{cc}\hfill {\widehat{f}}_i^v& =\mathrm{LN}\left({\{v_j^i:j\in {\mathbf{K}}_i^v\}}_{j=1}^L\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathbf{v}}_{\mathrm{ts}}^i& =\mathrm{MaxPool}\left(\mathrm{MLP}\left({\widehat{f}}_i^v\right)+\mathrm{FC}\left({\widehat{f}}_i^v\right)\right),\hfill \end{array}$$

(13)

where $\mathrm{MaxPool}(·)$ is the max-pooling function; $\mathrm{FC}(·)$ is a linear layer. The same operation can be applied to $T_i$ to obtain ${\mathbf{t}}_{ts}^i$. Finally, we compute the cosine similarity between ${\mathbf{v}}_{\mathrm{ts}}^i$ and ${\mathbf{t}}_{\mathrm{ts}}^i$.

3.5.3. Relevance Enhancement Loss

By clustering in a shared semantic space, direct semantic interference between different identities can be avoided. Based on the ITC [37] loss, we introduce a novel relevance enhancement (RE) loss to assist BITC and TSITC from a joint constraint.

For BITC, we define bidirectional contrastive objectives as ${\mathcal{L}}_{\mathrm{BITC}}=({\mathcal{L}}_{\mathrm{I}2\mathrm{T}}+{\mathcal{L}}_{\mathrm{T}2\mathrm{I}})/2$. The image-to-text loss formulated as

$$\begin{array}{cc}\hfill {\mathcal{P}}_{id}^i:& ={\mathcal{F}}_{id}^i·{\displaystyle \frac{C_{\mathrm{batch}}\left(i\right)+\alpha }{C_{\mathrm{batch}}}},\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{I}2\mathrm{T}}& =-\sum _{i=1}^{C_{\mathrm{batch}}}{\mathcal{P}}_{id}^i·log{\displaystyle \frac{exp(s({\mathbf{v}}_{cls}^i,{\mathbf{t}}_s^i)/\tau )}{{\sum }_{j=1}^{C_{\mathrm{batch}}}exp(s({\mathbf{v}}_{cls}^i,{\mathbf{t}}_s^j)/\tau )}},\hfill \end{array}$$

(15)

where ${\mathcal{P}}_{id}^i$ is the identity frequency weight, updated according to the identity label distribution of the current batch at each training step. ${\mathcal{F}}_{id}^i$ is mentioned in Equation (9); $\alpha$ is the smoothing coefficient, which is set to 0.1. $C_{\mathrm{batch}}$ is the count of samples, and $C_{\mathrm{batch}}\left(i\right)$ denotes the count of person identity i in current batch. The calculation of text-to-image ${\mathcal{L}}_{\mathrm{T}2\mathrm{I}}$ is the same as Equation (14):

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{T}2\mathrm{I}}& =-\sum _{i=1}^{C_{\mathrm{batch}}}{\mathcal{P}}_{id}^i·log{\displaystyle \frac{exp(s({\mathbf{v}}_{cls}^i,{\mathbf{t}}_s^i)/\tau )}{{\sum }_{j=1}^{C_{\mathrm{batch}}}exp(s({\mathbf{v}}_{cls}^j,{\mathbf{t}}_s^i)/\tau )}}.\hfill \end{array}$$

(16)

For TSITC, given ${\mathbf{v}}_{ts}^i$ and ${\mathbf{t}}_{ts}^i$, ${\mathcal{L}}_{\mathrm{TSITC}}$ can be obtained by the calculation procedure of Equations (14) and (16). The overall model loss ${\mathcal{L}}_{total}$ is formulated as a weighted sum of three key loss terms as follows:

$$\begin{array}{ccc}\hfill {\mathcal{L}}_{total}& =& {\mathcal{L}}_{id}+{\gamma }_1·{\mathcal{L}}_{cmia}+{\gamma }_2·{\mathcal{L}}_{bitc}+{\gamma }_3·{\mathcal{L}}_{tsitc}\hfill \end{array}$$

(17)

where ${\mathcal{L}}_{id}$ is commonly utilized ID loss [34]. ${\gamma }_1$, ${\gamma }_2$, and ${\gamma }_3$ are hyperparameters that balance the contribution of each loss, and we empirically set them to 1, 0.5, and 0.5, respectively.

4. Experimental Setup and Evaluation

In this section, we describe the experimental setup, including the datasets used for evaluation, the experimental settings, and the evaluation metrics. We then present the results of our experiments to assess the performance of the proposed model.

4.1. Datasets

In the experiments, we use the CHUK-PEDES [2], ICFGPEDES [24], and RSTPReid [25] datasets to evaluate our RDE. We split the datasets into training, validation, and test sets, wherein the ICFG-PEDES dataset only has training and validation sets. More details are provided in Appendix A.5.

4.2. Experimental Settings

Evaluation Metrics. We utilize the popular Rank-k metrics ($k=1,5,10$) as our principal assessment measures. Rank-k reports the probability of finding at least one matching person image within the top-k candidate list when given a textual description as a query. To ensure a holistic appraisal, we additionally incorporate mean average precision (mAP) as supplementary criteria for evaluating retrieval efficacy. A superior Rank-k value, alongside elevated mAP scores, denote more proficient system performance.

Implementation Details. As mentioned earlier, we adopt the pre-trained model CLIP [37] for our modality-specific encoders. The dependency-aware entity–attribute alignment network (DEAAN) method uses a pre-trained image encoder, i.e., CLIP-ViTB/16, a pre-trained text encoder, i.e., CILP text transformer, and a random-initialized multimodal interaction encoder applied to DAIR module. During training, we introduce data augmentations to increase the diversity of the training data. Specifically, we utilize random horizontal flipping, random crop with padding, and random erasing to augment the training images. For training texts, we employ random masking, replacement, and removal for the word tokens as the data augmentation. Moreover, the input size of images is $384\times 128$, and the maximum length of input word tokens is set to 77. We employ the Adam [58] optimizer to train our model for 60 epochs with a cosine learning rate decay strategy. The initial learning rate is $1\times {10}^{-5}$ for the original model parameters of CLIP, and the initial one for the network parameters of DAIR is initialized to $1\times {10}^{-3}$. The batch size is 64. We adopt an early training process with a gradually increasing learning rate. More detailed implementation settings can be found in Appendix A.

4.3. Comparison with State-of-the-Art Methods

Table 3. Results of state-of-the-art models compared with DEAAN (ours) on CUHK-PEDES, ICFG-PEDES, and RSTPReid.

Methods	Ref.	CUHK-PEDES				ICFG-PEDES				RSTPReid
		R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP
SSAN [24]	arXiv21	61.37	80.15	86.73	-	54.23	72.63	79.53	-	43.50	67.80	77.15	-
TIPCB [59]	Neuro22	63.63	82.82	89.01	-	-	-	-	-	-	-	-	-
SRCF [41]	ECCV22	64.04	82.99	88.81	-	57.18	75.01	81.49	-	-	-	-	-
UniPT [60]	ICCV23	68.50	84.67	90.38	-	60.09	76.19	82.46	-	51.85	74.85	82.85	-
CFine [21]	TIP23	69.57	85.93	91.15	-	60.83	76.55	82.42	-	50.55	72.50	81.60	-
VGSG [61]	TIP23	71.38	86.75	91.86	67.91	63.05	78.43	84.36	-	-	-	-	-
IRRA [1]	CVPR23	73.38	89.93	93.71	66.13	63.46	80.25	85.82	38.06	60.20	81.30	88.20	47.17
TBPS-CILP [29]	AAAI24	73.54	88.19	92.35	65.38	65.05	80.34	85.47	39.83	61.95	83.55	88.75	48.26
RDE [4]	CVPR24	75.94	90.14	94.12	67.56	67.68	82.47	87.36	40.06	65.35	83.95	89.90	50.88
DEAAN (BiLSTM)		63.27	81.54	87.25	58.17	55.97	74.92	80.75	31.30	45.36	68.34	78.49	37.24
DEAAN (RAN)		67.13	85.20	90.41	60.85	58.93	77.30	83.38	34.86	50.83	72.30	82.46	42.04
Baseline (CLIP+MLM)		70.36	86.40	91.13	61.84	62.79	78.27	84.85	36.46	57.68	80.05	88.82	45.28
DEAAN (Ours)		76.71	90.37	94.56	69.07	67.73	82.12	86.59	41.42	65.12	84.44	90.17	50.54

presents a comprehensive comparison of the proposed model with state-of-the-art methods using Rank-1 (R@1), Rank-5 (R@5), and Rank-10 (R@10) metrics on three widely recognized datasets: CUHK-PEDES, ICFG-PEDES, and RSTPReid.

Performance Comparisons on CUHK-PEDES. We evaluate DEAAN on the widely used CUHK-PEDES benchmark. As detailed in performance metrics, DEAAN achieves a Rank-1 accuracy of 76.71%, surpassing the previous state-of-the-art method, RDE, which records 75.94%—a notable improvement of +0.77%. In terms of mAP, DEAAN reaches 69.07%, outperforming IRRA at 66.13%, TBPS-CILP at 65.38%, and RDE at 67.56% by 2.94%, 3.69%, and 1.51%, respectively. Higher-rank metrics further highlight DEAAN’s strength, with Rank-5 and Rank-10 accuracies of 90.37% and 94.56%, compared to RDE at 90.14% and 94.12%. This consistent outperformance across all metrics emphasizes DEAAN’s enhanced retrieval precision and robustness on this dataset. The growing reliance on transformer-based backbones for TPS remains evident, underscoring the demand for powerful feature extraction in achieving these gains.

Performance Comparisons on ICFG-PEDES. Results on the ICFG-PEDES dataset showcase DEAAN’s competitive performance. It achieves a Rank-1 accuracy of 67.73%, slightly ahead of RDE at 67.68% by +0.05%, and an mAP of 41.42%, improving over RDE at 40.06% by +1.36%. These results affirm DEAAN’s leading position, even on a dataset with greater scope and complexity, although modest improvements suggest room for further optimization in challenging retrieval scenarios.

Performance Comparisons on RSTPReid. On the newer RSTPReid dataset, DEAAN demonstrates competitive performance against state-of-the-art methods. These results highlight DEAAN’s strengths in broader retrieval (Rank-5 and Rank-10) and overall ranking quality (mAP) relative to prior methods, although it narrowly lags behind RDE in top-1 precision and mAP. This mixed performance underscores DEAAN’s adaptability across diverse camera perspectives and identities, positioning it as a robust yet balanced solution for complex retrieval tasks on RSTPReid.

Performance Comparisons on Different Text Backbones. On CUHK-PEDES, BiLSTM and RAN achieve Rank-1 of 63.27% and 67.13%, respectively, which are significantly lower than DEAAN (76.71%). Similar trends are observed on ICFG-PEDES and RSTPReid, confirming DEAAN’s consistent superiority over both non-transformer and transformer baselines.

4.4. Ablation Study

To fully demonstrate the impact of different components in DEAAN, we conducted a comprehensive empirical analysis on two public datasets (i.e., CUHK-PEDES and ICFG-PEDES). The Rank-1, Rank-5, and Rank-10 accuracies (%) are reported in

Table 4. Ablation studies on the CHUK-PEDES and ICFG-PEDES datasets.

No.	Methods	Components			CUHK-PEDES			ICFG-PEDES
		DAIR	RATSw/BITC	RATSw/TSITC	R@1	R@5	R@10	R@1	R@5	R@10
0	Baseline (CILP+MLM)				70.36	86.40	91.13	62.79	78.27	84.85
1	+DAIR	√			72.90	87.96	92.53	64.77	79.68	84.65
2	+RATSw/BITC		√		72.27	87.50	92.16	64.91	79.58	85.07
3	+RATSw/TSITC			√	73.78	87.79	92.33	65.52	80.83	85.94
4	+RATS		√	√	74.49	88.93	93.36	66.60	81.94	86.33
5	+DAIR+RATSw/BITC	√	√		74.81	89.18	93.53	66.85	81.79	86.27
6	+DAIR+RATSw/TSITC	√		√	75.12	89.38	94.10	67.21	81.74	86.38
7	DEAAN	√	√	√	76.71	90.37	94.56	67.73	82.12	86.59

. The baseline (No. 0) builds on CLIP with Masked Language Modeling (MLM).

DAIR learns relations between attributes and nouns through dependency intervening, which can be easily integrated with other transformer-based methods to facilitate fine-grained attribute–noun correspondence. The efficacy of DAIR is revealed via the experimental results of No. 0 vs. No. 1, No. 2 vs. No. 5, No. 3 vs. No. 6, and No. 4 vs. No. 7. Merely adding the DAIR to baseline improves the Rank-1 accuracy by 2.54% and 1.98% on CUHK-PEDES and ICFG-PEDES datasets, respectively.

To demonstrate the effectiveness of our proposed RATS module, we compare it with the baseline on two public datasets. First, basic image–text contrast (BITC) greatly improves the retrieval accuracy (No. 0 vs. No. 2 and No. 1 vs. No. 5). After local token selecting through token-selected image–text contrast (TSITC), the retrieval accuracy is further improved (No. 2 vs. No. 4). It can also be noted that the improvement in BITC is small compared to TSITC (No. 2 vs. No. 3). This shows that there is some noise information interference in BITC image–text matching, which affects the retrieval.

4.5. Parametric Analysis

To study the impact of different hyperparameter settings on performance, we perform sensitivity analyses for key hyperparameters on the CHUK-PEDES, ICFG-PEDES, and RSTPReid datasets. The key parameters analyzed include the hyperparameter $\lambda$ in the dependency intervention self-attention (DISA) mechanism and the selection ratio $\kappa$ in the RATS module.

Hyperparameter $\lambda$ (from Equation (6)) Analysis. The hyperparameter $\lambda$ controls the influence of syntactic dependency information in DISA. As shown in

Table 5. Effect of $\lambda$ on performance metrics regarding CUHK-PEDES, ICFG-PEDES, and RSTPReid.

No.	$\mathit{\lambda }$	CUHK-PEDES				ICFG-PEDES				RSTPReid
		R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP
1	0.1	72.25	87.53	91.82	61.75	65.23	80.05	82.75	37.73	60.50	79.28	85.15	46.83
2	0.2	75.87	90.22	94.06	67.71	67.03	81.31	85.36	38.90	63.63	82.49	86.70	49.28
3	0.3	76.71	90.37	94.56	69.07	67.73	82.12	86.59	41.42	65.12	84.44	90.17	50.54
4	0.4	76.65	90.40	94.37	68.87	67.44	81.92	86.46	41.23	64.87	84.25	89.85	50.82
5	0.5	76.31	89.89	94.13	69.02	67.26	81.55	86.22	41.03	64.70	84.00	89.40	50.73
6	0.6	76.27	89.73	94.06	68.76	66.65	81.43	85.56	40.51	64.79	83.61	89.76	50.30
7	0.7	75.85	88.93	93.75	68.19	66.46	81.75	85.30	39.83	64.35	83.30	89.20	49.91
8	0.8	75.39	88.58	93.24	68.32	65.99	81.29	85.62	39.69	64.12	83.08	88.95	49.67
9	0.9	75.42	88.85	93.38	68.48	66.10	80.93	84.37	39.36	63.85	83.29	88.59	49.04
10	1.0	75.10	88.68	92.10	68.25	66.05	80.86	84.24	39.51	63.72	82.65	88.32	48.66

and Figure 4, performance peaks at $\lambda =0.3$, achieving the highest Rank-1 accuracy of 76.71% on CUHK-PEDES. Smaller values (e.g., $\lambda =0.1, 0.2$) underutilize syntactic cues, while larger values (e.g., $\lambda \ge 0.7$) overly bias the attention mechanism, suppressing critical semantic information. These results indicate that moderate integration of dependency structure yields the best balance for disambiguating attribute–noun associations.

Hyperparameter $\mathit{\kappa }$ (from Equation (11)) Analysis. In the RATS module, $\kappa$ determines the proportion of top informative tokens selected for fine-grained alignment. As shown in

Table 6. Effect of $\kappa$ on performance metrics regarding CUHK-PEDES, ICFG-PEDES, and RSTPReid.

No.	$\mathit{\kappa }$	CUHK-PEDES				ICFG-PEDES				RSTPReid
		R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP
1	0.1	73.36	88.52	93.30	66.25	63.75	77.84	81.36	36.93	60.48	81.89	84.92	45.33
2	0.2	75.79	89.33	93.95	68.54	65.33	80.90	86.70	40.29	64.78	83.55	88.38	49.58
3	0.3	76.50	90.03	94.19	68.86	67.39	82.36	86.91	41.36	65.00	84.62	89.87	50.17
4	0.4	76.71	90.37	94.56	69.07	67.73	82.12	86.59	41.42	65.12	84.44	90.17	50.54
5	0.5	76.35	90.15	94.37	68.95	66.97	81.71	86.07	41.28	64.67	83.83	88.84	50.36
6	0.6	75.62	90.00	93.84	68.35	66.47	81.36	85.66	41.10	64.86	83.80	89.43	49.97
7	0.7	75.05	89.10	93.96	68.22	66.33	81.70	85.59	40.61	64.31	83.53	88.89	49.59
8	0.8	74.85	89.37	93.62	68.05	66.08	81.38	85.75	40.67	64.35	83.17	88.27	48.25
9	1.0	74.76	89.15	93.68	67.93	65.42	80.98	85.12	40.36	64.28	83.35	87.68	49.64

and Figure 4, Rank-1 accuracy peaks at 76.71% when $\kappa =0.4$, with both smaller and larger values leading to degraded performance. A low $\kappa$ risks missing key features, while a high $\kappa$ introduces noise. Thus, $\kappa =0.4$ offers an optimal trade-off between token saliency and noise suppression.

Similarly, we conducted parameter experiments with $\lambda$ and $\kappa$ on the ICFG-PEDES and RSTPReid datasets. The results show that still, when $\lambda =0.3$ and $\kappa =0.4$, DEAAN reaches the optimum. This indicates that $\lambda$ and $\kappa$ are not dataset-sensitive.

4.6. Qualitative Results

This subsection presents a qualitative analysis of DEAAN’s performance, focusing on retrieval results and visual attention comparisons against the baseline, highlighting the effectiveness of the DAIR and RATS modules in improving retrieval accuracy and semantic understanding across complex language structures and visual attributes.

4.6.1. Retrieval Result Analysis

Figure 5 compares the top-10 retrieval results from the baseline (from Table 4) and our proposed DEAAN. As the figure shows, DEAAN achieves much more accurate retrieval results and obtains accurate retrieval results when baseline fails to retrieve them. In the case of text query with more complex language structure, our method can still maintain good retrieval results. This is mainly due to the robustness of the dependency-assisted implicit reasoning (DAIR) module in analyzing complex language structures.

In addition, we have observed that DEAAN retrieval results often have critical information. This shows that the retention of local token selectivity in the relevance-adaptive token selection (RATS) module ensures that the search space is biased towards the correct match of the positive sample. These components together contribute to superior performance in Rank-k and mAP, demonstrating the model’s ability to handle both positive and negative samples effectively.

4.6.2. Visual Analysis

In Figure 6, several visual comparisons between the baseline (from Table 4) and DEAAN are presented. Specifically, Grad-CAM algorithm [62] was employed to extract attention maps from the cross-attention, each corresponding to the attention of a single word in the whole person image. In the figure, some attribute chunks were chosen to highlight the effect of the dependency structures.

It can be intuitively observed from the figure that baseline’s understanding of multiple attribute–noun combinations in the text (such as “blue pants”, “brown boots”, etc.) has obvious ambiguity and positioning bias. Its heat map shows that it is difficult to accurately focus on a specific visual area in the description text.

In DEAAN, dependency structure (DAIR module) is used to clearly distinguish and accurately locate the target region corresponding to each attribute. This fine-grained semantic understanding and visual attention not only improve the accuracy of the model’s feature capture but also significantly enhance the robustness of the model’s description of complex languages.

In Figure 7, we compare the syntax attention and the ordinary attention using a heat-map of attention scores. The heatmap excludes punctuation, [CLS] and [SEP] tokens to establish a clearer correlation among the other tokens. We observe that the DISA exhibits the capacity to correctly recognize associations between attributes and nouns. For example, after incorporating syntactic information, attention scores between “t-shirt” and “blue” change from high to low, and attention scores between “pants” and “blue” change from low to high. This finding provides a compelling explanation for the effectiveness of DISA.

5. Conclusions

In this study, we introduced the dependency-aware entity–attribute alignment network (DEAAN), a novel framework tackling ambiguous attribute–noun association (AANA) and textual noise and relevance imbalance (TNRI) in text-based person search (TPS). By integrating dependency-assisted implicit reasoning (DAIR) and relevance-adaptive token selection (RATS), the DEAAN combines syntactic parsing with adaptive token filtering to enhance text–visual alignment, achieving state-of-the-art performance. The DEAAN bridges structured linguistic analysis with adaptive feature selection, advancing robust cross-modal retrieval. Its core, DAIR, which resolves attribute–noun ambiguities, and RATS, which suppresses noise, work synergistically to improve alignment precision.

Extensive experiments demonstrate that the DEAAN achieves state-of-the-art results, notably reaching 76.71% Rank-1 and 69.07% mAP on CUHK-PEDES, outperforming the previous best by +0.77% and +1.51%, respectively. The DEAAN also maintains superior performance on ICFG-PEDES and RSTPReid, validating its generalization across diverse datasets. These results confirm that integrating syntactic dependency parsing with adaptive token filtering significantly enhances text–visual alignment accuracy in TPS. Overall, this sets a new direction for TPS, offering a framework that is adaptable to broader cross-modal tasks.

We summarize the key advantages of the DEAAN: (1) the DAIR module significantly improves attribute–noun association in ambiguous expressions through dependency-enhanced attention, and (2) the RATS module enables discriminative token selection, enhancing local visual–textual alignment while suppressing noise. Nonetheless, the DEAAN also has limitations. Its reliance on syntactic parsers may introduce errors, especially in noisy or colloquial texts. Additionally, the marginal performance gains on ICFG-PEDES indicate room for improvement in diverse-data scenarios.

In future work, we will include a detailed discussion of the DEAAN’s theoretical scalability, highlighting its modular design, CLIP’s generalization, and optimization strategies (e.g., lightweight parsers and distributed training). We will also consider parser-agnostic modeling and robustness enhancements under real-world surveillance inputs. Furthermore, ensemble learning approaches have demonstrated success in customer churn prediction by modeling user behavior under high-dimensional sparse features [63], providing methodological inspiration for future extensions of the DEAAN in real-world retrieval or recommendation systems.