PMG—Pyramidal Multi-Granular Matching for Text-Based Person Re-Identification

Abstract

Given a textual query, text-based person re-identification is supposed to search for the targeted pedestrian images from a large-scale visual database. Due to the inherent heterogeneity between different modalities, it is challenging to measure the cross-modal affinity between visual and textual data. Existing works typically employ single-granular methods to extract local features and align image regions with relevant words/phrases. Nevertheless, the limited robustness of single-granular methods cannot adapt to the imprecision and variances of visual and textual features, which are usually influenced by the background clutter, position transformation, posture diversity, and occlusion in surveillance videos, thereby leading to the deterioration of cross-modal matching accuracy. In this paper, we propose a Pyramidal Multi-Granular matching network (PMG) that incorporates a gradual transition process between the coarsest global information and the finest local information by a coarse-to-fine pyramidal method for multi-granular cross-modal features extraction and affinities learning. For each body part of a pedestrian, PMG is adequate in ensuring the integrity of local information while minimizing the surrounding interference signals at a certain scale and can adapt to capture discriminative signals of different body parts and achieve semantically alignment between image strips with relevant textual descriptions, thus suppressing the variances of feature extraction and improving the robustness of feature matching. Comprehensive experiments are conducted on the CUHK-PEDES and RSTPReid datasets to validate the effectiveness of the proposed method and results show that PMG outperforms state-of-the-art (SOTA) methods significantly and yields competitive accuracy of cross-modal retrieval.

1. Introduction

Person re-identification (Re-ID) aims to retrieve a query person in a large image pool and is nowadays an extremely valuable task for its use in video surveillance and activity analysis [1,2,3,4,5]. With the proliferation of surveillance cameras in urban areas, massive original video data containing various events and persons are generated every second. Obviously, it is impractical to manually search for the corresponding persons in such large-scale videos, which is extremely time-consuming and boring. Thus, automatic retrieval methods are in urgent need to handle this task more efficiently. According to the query modality, existing methods can be mainly classified into the ones with an image-based query, attribute-based query, and text-based query. The major limitation of image-based person Re-ID methods is the requirement for at least one high-quality image of the queried person, a condition difficult to satisfy on many occasions. As to attribute-based methods, due to the limited descriptive capability of attributes, many candidate images with similar attributes are often matched for a query given as a set of predefined person attributes [6]. Nevertheless, text-based methods utilize verbal descriptions as queries for person search and can provide much more detailed information about the queried person. Moreover, a verbal description of the suspect may be the only accessible information in many real-life scenes. Although text-based person Re-ID has the above advantages and has been studied from various perspectives, the inherent ambiguity of natural language and the huge modality gap make it still a challenging task that needs to be further addressed.

The central problem of text-based person Re-ID lies in properly bridging the gap between visual modality and textual modality. Technically speaking, “bridging the gap” relies on a two-stage process: cross-modal feature extraction and cross-modal similarity learning [7]. In fact, how to effectively extract visual and textual features for the upcoming cross-modal similarities measuring under potential semantic relevance between modalities is the main challenge faced in the text-based person Re-ID task [8]. Nevertheless, due to the inevitable background clutter, position transformation, posture diversity, and occlusion in surveillance videos, visual features extracted from multiple images of the same pedestrian may have significant differences, even when a detection model is initially utilized to delimit the region of the targeted pedestrian in an image by generating a bounding box. Meanwhile, the ambiguity of natural language determines that textual descriptions of the same image may vary dramatically, which also leads to deviations in extracted textual features. The imprecision and variances of visual and textual features make it difficult to identify the same person and distinguish different persons, diminishing the cross-modal matching accuracy. Therefore, robust visual and textual feature descriptors used to effectively capture the discrimination of persons need to be well designed.

Nowadays, almost all text-based person Re-ID methods are striving to alleviate the adverse effect caused by the aforementioned problems and extract discriminative features with small inter-class and large intra-class variations as much as possible [9]. These methods typically employ deep neural networks to obtain global and local feature representations. Li et al. [10] introduced the concept of text-based person Re-ID in their initial work. They also proposed the use of VGG-16, a deep convolutional neural network model, to extract global visual features for this task. Inspired by this work, Chen et al. [11] devised an efficient patch-word matching model to accurately capture the local matching details between visual and textual data, by computing the affinity of the best matching patch of an image toward a word. On the other hand, Jing et al. [8] incorporated pose information to aid in localizing the discriminative body parts for effective local visual feature extraction. However, utilizing prior knowledge like poses can suffer from the inaccuracy of human pose estimation and great computation burden. In the study conducted by Niu et al. [12], a Multi-granularity Image-text Alignments (MIA) model was proposed, by which a feature map is horizontally cropped into a group of non-overlapping image stripes for local visual feature extraction. Following this work, Ding et al. [13] proposed a Word Attention Module (WAM) to acquire correlations between words and image stripes, thus extracting semantically aligned part-level features. Briefly, existing works typically utilize partitioning methods to divide pedestrian images into multiple strips to capture more discriminative local visual features, and employ attention mechanisms for better matching of image strips with relevant textual descriptions.

Although existing methods have achieved significant progress in cross-modal retrieval, there still exist some limitations. Taking the SOTA method SSAN [13] as an example, it is difficult to determine how many stripes an image should be partitioned into. In fact, the coarser the image slicing, the more difficult it is to capture detailed information, while the finer the slicing, the more likely it is to compromise the integrity of local features. Intuitively, diverse partitioning scales are preferred, ensuring that for each body part of a pedestrian, there always exists an appropriate scale that allows an effective extraction of local features from a certain image strip and accurate alignment with relevant words/phrases at this scale. Figure 1 illustrates this concept, at scale 4, the weakened signal of “glasses” is greatly strengthened by minimizing the surrounding interference of excessive background while the integrity is well-preserved, which facilitates extracting discriminative local features and aligning with relevant words/phrases highlighted by green rectangles.

Inspired by this idea, we propose a Pyramidal Multi-Granular matching network (PMG) with an overall architecture illustrated in Figure 2. PMG extracts eight representations with different granularities, including a global visual representation ($V_g$), a global textual representation ($T_g$), three local visual representations ($V_{l2}$, $V_{l4}$, $V_{l6}$), and three local textual representations ($T_{l2}$, $T_{l4}$, $T_{l6}$), and then four different levels of cross-modal similarities between visual and textual features, i.e., $V_g$-$T_g$, $V_{l2}$-$T_{l2}$, $V_{l4}$-$T_{l4}$ and $V_{l6}$-$T_{l6}$ are calculated to better match visual and textual information. To be more specific, PMG horizontally partitions feature maps of the visual backbone into multiple non-overlapping stripes with various pyramidal scales and employs a Word Attention Module to acquire correlations between words and image stripes, then extracts modality features with different scales independently, which can incorporate the gradual transition process between the coarsest global information and the finest local information. After training the combination of multi-granular visual and textual representations, PMG obtains the final cross-modal similarity by fusing the multi-granular cross-modal similarities. In contrast with existing methods which extract local features with a single granularity for matching, our work employs a pyramidal method for the extraction and matching of features with four different granularities. Obviously, multiple granularities can better adapt to the extraction and matching of different features, even for pedestrian images under intensive changes, such as pose, viewpoint, etc., thus suppressing the variances of feature extraction and improving the robustness of feature matching. The main contribution of this work is threefold:

• A coarse-to-fine pyramidal matching method is employed to handle the problem of ineffective local feature extraction from surveillance videos plagued by arbitrariness in existing single-granular methods.
• A Pyramidal Multi-Granular matching network (PMG) is proposed to learn multi-granular cross-modal affinities.
• Comprehensive experiments are conducted on the CUHK-PEDES and RSTPReid datasets, which indicate that PMG outperforms other previous methods significantly and achieves state-of-the-art performance.

2. Related Works

2.1. Person Re-Identification

Person Re-ID has drawn remarkable attention due to its applicability and research significance. In the early days, hand-crafted person Re-ID systems were predominantly reported, which focused on manually extracting features such as color and texture, and learning better similarity measures by using classic supervised or unsupervised metric learning algorithms. However, traditional methods have certain limitations and are difficult to handle the different poses, backgrounds, lighting, scales, and other issues of pedestrians captured by different cameras. Since deep learning technology was first introduced into this task by Yi et al. [14], person Re-ID methods based on deep learning have achieved great success over the last few years. Zheng et al. [15] designed a unified framework named DG-Net to enable end-to-end interaction between the generative module and discriminative module for more robust and accurate Re-ID learning. Liu et al. [16] presented a two-branch Deep Joint Learning (DJL) network to enhance the discriminative capability of local and global visual features, and hence a hierarchical feature aggregation mechanism was proposed to aggregate the learned hybrid features. To globally learn the attention for each feature node via a global view of feature relations, Zhang et al. [17] proposed a Relation-Aware Global Attention (RGA) network to excavate the global structural information, thus enhancing the capability of image feature representation. Li et al. [18] managed to propose a lightweight network CDNet to reduce computational resources and time costs of person Re-ID via a novel search space called Combined Depth Space and a new search strategy called Top-k Sample Search. Bak et al. [19] proposed a one-shot learning method that decomposes the Re-ID metric into different components to reduce the amount of training data required.

2.2. Text-Based Person Re-Identification

Recently, due to the rapid development of natural language processing (NLP) technology, person Re-ID based on text description has gradually become a focus of attention. Text-based person Re-ID can find the corresponding target person according to the given text query. This combination of text description and image not only improves the accuracy of pedestrian recognition but also realizes cross-modal person Re-ID. In recent years, the task of text-based person Re-ID has been studied from many perspectives. Li et al. [10] first put forward this task and further employed a deep neural network GNA-RNN for person search, which can extract the global visual feature and estimate the affinity between a query text and a person image. Following their work, an efficient patch-word matching model was proposed by Chen et al. [11], with the aim of accurately exploiting the cross-modal local matching details between image patches and text words. Liu et al. [20] proposed a novel deep adversarial graph attention convolution network (A-GANet) in which an adversarial learning module is developed to learn a joint textual-visual feature space for cross-modality matching. Sarafianos et al. [21] proposed a Text-Image Modality Adversarial Matching (TIMAM) approach which learns modality-invariant feature representations using adversarial and cross-modal matching objectives. In addition, BERT, a publicly available language model is first applied for extracting word embeddings in this work. Aggarwal et al. [22] proposed a novel framework to learn common representations for images and text, such that semantics are explicitly preserved in the features. More recently, both the global and local cross-modal information are usually taken into consideration together by many works for further improvement in retrieval accuracy. Jing et al. [8] employed pose information to benefit localizing the discriminative regions, and then two alignment networks were proposed to learn the latent image-text alignments at global and local scales, respectively. Hao et al. [23] presented a novel Modality Confusion Learning Network (MCLNet), with the aim of confusing two modalities during optimization to focus solely on modality-irrelevant information. A novel cross-modal alignments model named MIA was proposed by Niu et al. [12]. By horizontally splitting the feature map into non-overlapping strips, MIA extracts fine-grained image features, and then the cross-modal similarities are evaluated from multi-granular image-text alignments via a cross-modal attention mechanism. A NAFS model was proposed by Gao et al. [24], in which both the visual and textual description are decomposed at three scales, and then a contextual non-local attention mechanism is employed to discover latent alignments. With a Gumbel attention module, a hierarchical adaptive matching model was introduced by Zheng et al. [25] to tackle the problem of matching redundancy when aligning image regions and words/phrases. Wang et al. [26] proposed a Visual-Textual Attributes Alignment (ViTAA) model, which warps both visual and textual attributes into multiple categories and then learns to disentangle the feature space of a person into subspaces corresponding to attributes using a light auxiliary attribute segmentation layer. A DSSL model was proposed by Zhu et al. [27] along with a new RSTPReid dataset. In order to obtain higher retrieval accuracy, DSSL employed a surroundings–person separation mechanism to effectively excavate person information and adopted diverse alignment paradigms to adequately utilize multi-modal and multi-granular information. Ding et al. [13] proposed the SSAN method to extract semantically aligned part-level features for the multi-modal data and a compound ranking loss is also employed to optimize global and part feature learning.

For a more comprehensive review of research works in the field of text-based person Re-ID, we summarize the related literature reviews according to their optimization components and local cross-modal alignments in

Table 1. Summary of related literature reviews.

Method	Network Architecture	Attention Mechanism	Aiding Information	Local Feature Alignments
SUM [7]	✓			-
PMA [8]		✓	✓	$V_L^6$-$T_L^p$
CAIBC [9]	✓		✓	$V_L^3$-$T_L^w$
GNA-RNN [10]	✓			-
PWM-ATH [11]		✓		-
MIA [12]	✓	✓	✓	$V_L^6$-$T_L^w$
SSAN [13]		✓		$V_L^3$- $T_L^p$
A-GANet [20]	✓	✓		-
TIMAM [21]	✓			-
CMAAM [22]	✓			$V_L^2$-$T_L^p$
MCLNet [23]	✓			-
NAFS [24]	✓	✓		$V_L^3$-$T_L^{p,w}$
HGAN [25]		✓		-
ViTAA [26]			✓	$V_L^5$-$T_L^p$
DSSL [27]		✓		$V_L^6$-$T_L^{p,w}$

, where “✓” indicates the integration of this component, $V_L^k$ denotes the number of image strips/parts partitioned for local cross-modal alignments, and $T_L^p/T_L^s/$ denotes that the textual description is split into phrase/words.

3. Proposed Method

In this section, we describe the proposed Pyramidal Multi-Granular matching network (PMG) in detail. First, we introduce the process of extracting pyramidal multi-granular visual and textual representations. Then, a coarse-to-fine pyramidal matching method is presented to learn the image-text similarities. Finally, a compound Re-ID loss and a two-stage training strategy are adopted for the training of PMG. The algorithm framework of PMG is shown in Figure 3.

3.1. Pyramidal Multi-Granular Feature Extraction

PMG extracts eight representations with different granularities, including global visual representation ($V_g$), local${}_2$ visual representation ($V_{l2}$), local${}_4$ visual representation ($V_{l4}$), local${}_6$ visual representation ($V_{l6}$), global textual representation ($T_g$), local${}_2$ textual representation ($T_{l2}$), local${}_4$ visual representation ($T_{l4}$) and local${}_6$ visual representation ($T_{l6}$). Correspondingly, a visual pyramidal module and a textual pyramidal module are utilized to split the representation extraction into multiple parallel branches for multi-granular visual and textual feature representations, respectively.

3.1.1. Visual Feature Extraction

In order to extract global and local visual features from a given image I, a ResNet-50 backbone that has been pretrained on the ImageNet dataset is first employed to generate the shared visual feature map $\psi \left(I\right)\in {\mathbb{R}}^{w\times h\times c}$ with width w, height h and channels c. Considering that horizontally partitioning $\psi \left(I\right)$ into a fixed number of non-overlapping stripes may suffer from the misaligned bounding box problem as discussed in the Introduction section, we adopt a coarse-to-fine pyramidal matching method to obtain visual representations with various pyramidal scales.

Specifically, PMG introduces four different pyramidal scales by horizontally partitioning $\psi \left(I\right)$ into 1, 2, 4, or 6 spatial bins, respectively. Accordingly, the feature map is denoted as ${\psi }_K\left(I\right)\in {\mathbb{R}}^{w\times h_K\times c}$, where K = 1, 2, 4, 6 and $h_K=h/K$ means $\psi \left(I\right)$ is horizontally portioned by K uniform parts. For each ${\psi }_K\left(I\right)$, the partitioning operation is carried out by an average pooling module

$$AvgPoolin{g}_K=\frac{1}{h_K\times w}\sum _{i=1}^{h_K}\sum _{j=1}^w{\psi }_K^{[1:K]}{\left(I\right)}_{i,j,1:c},$$

(1)

and a maximum pooling module

$$MaxPoolin{g}_K=\underset{1\le i\le h_K,1\le j\le w}{max}{\psi }_K^{[1:K]}{\left(I\right)}_{i,j,1:c},$$

(2)

hence the visual pyramidal module is defined as follows:

$${\mathsf{\Phi }}_K\left(I\right)=AvgPoolin{g}_K\left(I\right)+MaxPoolin{g}_K\left(I\right),$$

(3)

where $K\in \{1,2,4,6\}$ denotes the pyramidal level and ${\mathsf{\Phi }}_K\left(I\right)\in {\mathbb{R}}^{K\times 1\times c}$. This fused pooling module takes advantage of the down-sampling characteristic of the average pooling layer to reduce computation complexity, as well as the capability for capturing discriminative signals of the maximum pooling layer, which may be weakened when they are surrounded by signals relatively poor (e.g., background).

Then, the four different pyramidal scales are separately processed to obtain the final visual representations. To extract the global visual representation $V_g\in {\mathbb{R}}^P$, we reshape ${\mathsf{\Phi }}_1\left(I\right)\in {\mathbb{R}}^{1\times 1\times c}$ to a c-dimensional vector and then pass it through a batch normalization (BN) layer followed by a 1 × 1 convolutional layer consisting of P kernels. As for local visual representations of $V_{l2}$, $V_{l4}$ and $V_{l6}$, we first create three different $K\times 1\times c$$(K=2,4,6)$ processing structures, in which a 1 × 1 convolutional layer is, respectively, adopted to reduce the dimensionality from c to $c/2$. Then, these K vectors with the dimensionality of $c/2$ are separately passed through a multi-layer perception containing a batch normalization layer and two fully-connected (FC) layers with a ReLU layer between them to obtain the local visual-medium vector, which follows:

$$v_{lK}^k=W_{V_K}^k\mathrm{ReLU}\left({\tilde{W}}_{V_K}^k\mathrm{BN}\left({\tilde{\Phi }}_K^k\left(I\right)\right)\right),$$

(4)

where ${\tilde{\Phi }}_K^k\left(I\right)$ is the k-th vector of the modified visual feature map with reduced dimensionality of c/2, ${\tilde{W}}_{V_K}^k\in {\mathbb{R}}^{c\times \frac{c}{2}}$, $W_{V_K}^k\in {\mathbb{R}}^{P\times c}$, and $v_{lK}^k\in {\mathbb{R}}^P(K\in \{2,4,6\},k\in \{1,...,K\})$ is the visual representation of the k-th local strip on scale K. As a result, the final local visual representation $V_{lK}=\{v_{lK}^1,v_{lK}^2,...,v_{lK}^K\}$ can be formed by concatenating $v_{lK}^k$, and thus there are total ${\displaystyle \sum _{K\in \{2,4,6\}}}K=12$ P-dimensional vectors to represent the local visual features.

3.1.2. Textual Feature Extraction

To extract global and local textual representations, we begin by utilizing a pretrained BERT language model to embed each word within the textual description. Specifically, given a description of length n, we build an embedding matrix $W_e\in {\mathbb{R}}^{m\times n}$ and then obtain the word embedding for each word by $x_i=W_e\times w_i$, where $w_i\in {\mathbb{R}}^n$ is the i-th word in the description, and $x_i\in {\mathbb{R}}^m$ denotes $w_i$ is embedded into a vector with dimensionality of m. After that, we feed the embedded words through a bi-directional gated recurrent unit (Bi-GRU) to capture the relationship among them, which follows

$$\overrightarrow{h_i}=\overrightarrow{GRU}(x_i,\overrightarrow{h_{i-1}}),1\le i\le n$$

(5)

$$\overleftarrow{h_i}=\overleftarrow{GRU}(x_i,\overleftarrow{h_{i-1}}),1\le i\le n$$

(6)

where $\overrightarrow{h_i},\overleftarrow{h_i}\in {\mathbb{R}}^c$ and represent the forward and backward hidden states of the i-th word, respectively.

Then, we define the representation of the i-th word as the average of the i-th last hidden states from both forward and backward GRUs, namely $e_i=(\overrightarrow{h_i}+\overleftarrow{h_i})/2$, where $e_i\in {\mathbb{R}}^c(1\le i\le n)$. To generate the representation E for the entire textual description, we concatenate all n word representations together as $E=\{e_1,e_2,...,e_n\}$.

After obtaining the textual feature map E, we carry out a row-wise max pooling (RMP) operation on E and then obtain the global textual representation $T_g$ through a $1\times 1$ convolutional layer as follows

$$T_g=W_{T_g}\times softmax\left(\underset{c}{max}E\right),$$

(7)

where $W_{T_g}\in {\mathbb{R}}^{P\times n}$ and $T_g\in {\mathbb{R}}^P$. Next, we utilize a textual pyramidal module to obtain local textual representations according to the word-part correspondences, which expends the part-level textual feature extraction method in SSAN [13] to various pyramidal scales corresponding to the K-partitioned feature map ${\psi }_K\left(I\right)(K=2,4,6)$. We first predict the probability that the i-th word belongs to the k-th strip by $s_K^{i,k}=\sigma \left({\omega }_K^k{e}_i\right)$, where $\sigma$ denotes the Sigmoid function and ${\omega }_K^k\in {\mathbb{R}}^{1\times c}$ stands for a linear transformation operation. Then, we modify the textual feature map $E={\left\{e_i\right\}}_{i=1}^n$ as $E_K^k={\left\{s_K^{i,k}{e}_i\right\}}_{i=1}^n$ to represent the textual description for the k-th local strip. Similar to the process of obtaining $T_g$, each modified textual description $E_K^k$ is passed through an RMP layer followed by a $1\times 1$ convolutional layer to obtain $t_{lK}^k$ as

$$t_{lK}^k=W_{T_K}^k\times softmax\left(\underset{c}{max}{E}_K^k\right),$$

(8)

where $W_{T_K}^k\in {\mathbb{R}}^{P\times n}$, and $t_{lK}^k\in {\mathbb{R}}^P$ represents the textual features for the k-th local strip. Finally, we stack $t_{lK}^k(1\le k\le K)$ as $T_{lK}=\{t_{lK}^1,t_{lK}^2,...,t_{lK}^K\}$ to obtain the final local textual representation with a given pyramidal scale K.

3.2. Image-Text Matching

After obtaining a total of eight representations of the diverse modal and granularity, four different cross-modal similarities of global and local features are proposed to, respectively, match visual and textual information with different granularities.

For each of the four cross-modal combinations, i.e., $V_g$-$T_g$, $V_{l2}$-$T_{l2}$, $V_{l4}$-$T_{l4}$ and $V_{l6}$-$T_{l6}$, the modified cosine metric is adopted to evaluate the similarity with relevant representations by

$${\mathrm{simi}}_g=\frac{V_g^T·T_g}{\left|\right|V_g\left|\right|\times \left|\right|T_g\left|\right|},$$

(9)

$${\mathrm{simi}}_{l2}=\frac{1}{2}\sum _{k=1}^2(\frac{{\left(v_{l2}^k\right)}^T·t_{l2}^k}{\left|\right|v_{l2}^k\left|\right|\times \left|\right|t_{l2}^k\left|\right|}),$$

(10)

$${\mathrm{simi}}_{l4}=\frac{1}{4}\sum _{k=1}^4(\frac{{\left(v_{l4}^k\right)}^T·t_{l4}^k}{\left|\right|v_{l4}^k\left|\right|\times \left|\right|t_{l4}^k\left|\right|}),$$

(11)

$${\mathrm{simi}}_{l6}=\frac{1}{6}\sum _{k=1}^6(\frac{{\left(v_{l6}^k\right)}^T·t_{l6}^k}{\left|\right|v_{l6}^k\left|\right|\times \left|\right|t_{l6}^k\left|\right|}),$$

(12)

where $\left|\right|·\left|\right|$ denotes the 2-norm of the vector, and the cross-modal similarity sim${}_{lK}$$(K=2,4,6)$ is calculated by averaging the K part-level feature similarities of the image-text pair with a given pyramidal scale K.

Then, the final cross-modal similarity is fused by the above four similarities as follows:

$${\mathrm{simi}}_{PMGNet}={\mathrm{simi}}_g+\lambda ·({\mathrm{simi}}_{l2}+{\mathrm{simi}}_{l4}+{\mathrm{simi}}_{l6}).$$

(13)

where $\lambda$ is the fusion coefficient and set to 0.5 in the testing stage, indicating that global and local cross-modal similarities are equally important to the final image-text matching.

3.3. Loss Functions and Training Strategy

For the training of PMG, we adopt a compound Re-ID loss, which uses both identification (ID) loss with loose constraints and triplet ranking loss with hard mining. Furthermore, a two-state training strategy is employed.

During Stage-I, the visual backbone parameters remain fixed, while the remaining components of PMG are optimized using the ID loss. This process involves grouping individuals into distinct clusters with the guidance of their IDs. As global representations can provide more comprehensive information for this classification process, only the two global representations, namely $V_g$ and $T_g$, are utilized here. We first build a shared transformation matrix $W_{\mathrm{ID}}\in {\mathbb{R}}^{N\times P}$ via a bias-free fully-connected layer, where N represents the total number of distinct individuals presented in the training set, and then calculate the two proposed ID losses ${\mathcal{L}}_{\mathrm{ID}}^V$ and ${\mathcal{L}}_{\mathrm{ID}}^T$ for visual and textual representations as

$${\mathcal{L}}_{\mathrm{ID}}^V=-\frac{1}{N}\sum _{i=1}^N\log \left(softmax(W_{\mathrm{ID}}^i·V_g)\right),$$

(14)

$${\mathcal{L}}_{\mathrm{ID}}^T=-\frac{1}{N}\sum _{i=1}^N\log \left(softmax(W_{\mathrm{ID}}^i·T_g)\right),$$

(15)

where $W_{\mathrm{ID}}^i\in {\mathbb{R}}^{1\times P}$ is the i-th row vector of $W_{\mathrm{ID}}$. Here, the transformation matrix $W_{\mathrm{ID}}$ is shared so as to map multi-modal ID representations into the same latent space. The integrated loss function for Stage-I is

$${\mathcal{L}}_{\mathrm{stageI}}={\mathcal{L}}_{\mathrm{ID}}^V+{\mathcal{L}}_{\mathrm{ID}}^T.$$

(16)

In Stage-II, all parameters of PMG including the visual backbone are together fine-tuned. In addition to the ID loss proposed in Stage-I, the popular triplet ranking loss is also used to ensure fine-grained visual-textual matching, which follows:

$$\begin{array}{cc}{\mathcal{L}}_{\mathrm{rank}}^K=\sum _{\widehat{T_K}}\max \{\alpha -\cos (V_K,T_K)\hfill & +\cos (V_K,\widehat{T_K}),0\}\hfill \\ & +\sum _{\widehat{V_K}}\max \{\alpha -\cos (V_K,T_K)+\cos (\widehat{V_K},T_K),0\},\hfill \end{array}$$

(17)

where ${\mathcal{L}}_{\mathrm{rank}}^K\in \{{\mathcal{L}}_{\mathrm{rank}}^g,{\mathcal{L}}_{\mathrm{rank}}^{l2},{\mathcal{L}}_{\mathrm{rank}}^{l4},{\mathcal{L}}_{\mathrm{rank}}^{l6}\}$ denotes the ranking loss of visual-textual matching on scale K. Similarly, visual representation $V_K$ can correspond to $V_g$, $V_{l2}$, $V_{l4}$, or $V_{l6}$, and textual representation $T_K$ can be $T_g$, $T_{l2}$, $T_{l4}$, or $T_{l6}$ according to ${\mathcal{L}}_{\mathrm{rank}}^K$. Here, $(V_K,T_K)$ represents a matching visual-textual pair, while $(V_K,\widehat{T_K})$ and $(\widehat{V_k},T_k)$ represent mismatched pairs. The margin $\alpha$ is applied to ensure the similarity score of matched pairs is larger than mismatched ones in a mini-batch. The joint loss function in Stage-II is

$${\mathcal{L}}_{\mathrm{stageII}}={\mathcal{L}}_{\mathrm{ID}}^V+{\mathcal{L}}_{\mathrm{ID}}^T+{\mathcal{L}}_{\mathrm{rank}}^g+{\mathcal{L}}_{\mathrm{rank}}^{l2}+{\mathcal{L}}_{\mathrm{rank}}^{l4}+{\mathcal{L}}_{\mathrm{rank}}^{l6},$$

(18)

4. Experiment Details

4.1. Experiment Setup

4.1.1. Experiment Preparations

In our experiments, we utilize Python environment 3.7.2, Anaconda3, Pycharm 2019.3, PyTorch 1.5.0, CUDA 10.2, cuDNN 7.65, and Windows 10 Professional OS. The experiments are executed on a Lenovo ThinkStation P520C workstation equipped with 128 GB DDR4 RAM, an Intel Xeon W-2245 processor, and an NVIDIA RTX A5000 GPU.

4.1.2. Dataset and Evaluation Metrics

To evaluate the performance of the proposed PMG method, we conduct extensive experiments on two text-based person Re-ID datasets: CUHK-PEDES [10] and RSTPRedi [27]. CUHK-PEDES is currently the most popular dataset used for text-to-image person Re-ID and has become the de facto benchmark for text-based person retrieval tasks. Compared with CUHK-PEDES, the newly conducted RSTPRedi is more adaptable to real application scenarios in that images were caught by different cameras with complex scene transformations and backgrounds in various periods of time [27]. (1) CUHK-PEDES: We follow the official data split paradigm [10] for the training, validating, and testing of PMG. Specifically, the training set consists of 34,054 images representing 11,003 individuals, along with 68,126 query sentences. The validation set comprises 3078 images, 1000 individuals and 6158 textual descriptions. The testing set contains 3074 images, 1000 individuals and 6156 descriptions. (2) RSTPReid: The RSTPReid dataset used in this study consists of 20,505 images and 41,010 descriptions showcasing a total of 4101 individuals. Following the data split approach in [27], the whole dataset is divided such that 3701 identities are assigned to the training set, 200 identities to the validation set, and 200 identities to the testing set. For each identity in all sets, 5 images along with 10 annotated textual descriptions are attached to represent the corresponding identity.

We evaluate the performance using Top-1, Top-5, and Top-10 accuracies, which are the most popular and crucial metrics in person Re-ID. Given a query sentence, all images in the test set are ranked by their similarities with the query text, and Top-k images with the highest scores are selected as candidates. If the Top-k candidate images contain at least one image of the targeted pedestrian, the query is called to be successful. Then, the Top-k accuracy is calculated as the ratio of successful queries to the total number of test queries. Figure 4 shows an example of Top-5 text-based person Re-ID results.

4.1.3. Implementation Details

In our experiments, the representation dimension of the feature space P is set to 1024 following [7], in which an ablation experiment has been conducted on CUHK-PEDES for the best setting of P. As to the dimension of embedded word vectors c, we maintain the existing setting of WAM [13] to ensure consistency by keeping c at 768. The Natural Language ToolKit (NLTK) is employed to extract the noun phrases from each query sentence by syntactic analysis, word segmentation, and part-of-speech tagging. The number of phrases n is kept flexible with an upper bound of 26 based on the length of query sentences. A pretrained BERT language model [28] with initialized fixed parameters is also used to handle the textual input.

For fair comparisons with previous works, we choose the pretrained ResNet-50 as our visual CNN backbone, whose parameters are initialized with weights pretrained on the ImageNet classification task. All input images are resized to 384 × 128 × 3 resolution, and random horizontal flipping is applied for data augmentation before feeding into the backbone [12,13], and we extract the feature maps before the average pooling layer of ResNet-50, whose size is 24 × 8 × 2048.

An Adaptive Moment Estimation (ADAM) optimizer is also utilized to optimize the learning strategy of PMG, with a training batch size of 64, which is a moderate batch size adapted to the processing capability of our hardware platform and widely employed by many research works. In training Stage-I, the learning rate is set to $1\times {10}^{-3}$ for an iteration of 10 epochs with all parameters in the visual backbone fixed. And then in Stage-II, the learning rate is first initialized to $2\times {10}^{-4}$ and PMG is further trained for 50 epochs. The settings of learning rate and training epoch are determined by ablation analysis, as shown in

Table 2. Ablation analysis of learning rate of Stage-I.

Epoch	ID_Loss ($1\times {10}^{-2}$)	ID_Loss ($1\times {10}^{-3}$)	ID_Loss ($1\times {10}^{-4}$)
1	56.04%	56.04%	56.04%
5	27.94%	24.02%	27.80%
10	17.01%	12.10%	17.49%

,

Table 3. Ablation analysis of learning rate of Stage-II.

Epoch	ID_Loss ($2\times {10}^{-2}$)	ID_Loss ($2\times {10}^{-4}$)	ID_Loss ($2\times {10}^{-5}$)
20	6.24%	8.47%	9.57%
30	5.43%	4.32%	5.80%
40	4.36%	3.50%	4.14%

and

Table 4. Ablation analysis of trained epoch.

Epoch	Top-1	Top-5	Top-10
40	62.15%	80.86%	87.04%
50	64.59%	83.19%	89.12%
60	63.94%	82.87%	88.91%

. In the training stage, the margin of triplet ranking loss $\alpha$ is used to enhance the discrimination between positive and negative pairs, and we set $\alpha$ to 0.2 following almost all research works [7,8,13,27,29].

4.2. Comparison with SOTA

The performance comparisons on the CUHK-PEDES and RSTPReid datasets are presented in

Table 5. Comparison with SOTA on CUHK-PEDES.

Method	Top-1	Top-5	Top-10
CNN-RNN [30]	8.07	-	32.47
Neural Talk [31]	13.66	-	41.72
GNA-RNN [10]	19.05	-	53.64
IATV [32]	25.94	-	60.48
PWM-ATH [11]	27.14	49.45	61.02
Dual Path [33]	44.40	66.26	75.07
GLA [34]	43.58	66.93	76.26
CMPM-CMPC [35]	49.37	71.69	79.27
MIA [12]	53.10	75.00	82.90
A-GANet [20]	53.14	74.03	81.95
PMA [8]	54.12	75.45	82.97
TIMAM [21]	54.51	77.56	84.78
ViTAA [26]	55.97	75.84	83.52
CMAAM [22]	56.68	77.18	84.86
HGAN [25]	59.00	79.49	86.62
NAFS [24]	59.94	79.86	86.70
DSSL [27]	59.98	80.41	87.56
SSAN [13]	61.37	80.15	86.73
PMG (ours)	62.33	81.32	87.26
PMG + BERT (ours)	64.59	83.19	89.12

and

Table 6. Comparison with SOTA on RSTPReid.

Method	Top-1	Top-5	Top-10
IMG-Net [29]	37.60	61.15	73.55
AMEN [36]	38.45	62.40	73.80
DSSL [27]	39.05	62.60	73.95
CAIBC [9]	47.35	69.55	79.00
PMG (ours)	46.60	70.85	79.55
PMG + BERT (ours)	48.85	72.65	81.30

, respectively. For more visually appealing results, graphical comparisons are also presented in Figure 5 and Figure 6. It is noted that considering many previous methods do not utilize BERT, a powerful pretrained language model which can improve the performance, we validate the performance of PMG both with and without BERT for the fairness of experimental comparisons.

As shown in Table 5 and Figure 5, the PMG model outperforms other methods by considerable margins in terms of Top-1, Top-5, and Top-10 metrics. For instance, compared with SSAN, which is the SOTA method using both global and local features, PMG+BERT(ours) gains 3.22%, 3.04%, and 2.39% performance improvement in terms of Top-1, Top-5, and Top-10 metrics, respectively. This is a significant performance gain in text-based person Re-ID which has reached a technological bottleneck in terms of performance improvement in the past few years and indicates the effectiveness of our proposed method on the CUHK-PEDES dataset. Instead of utilizing methods that optimize the cross-modal matching from various aspects, such as network architecture, local representation extraction, loss function, and aiding discriminative information, PMG employs a simple coarse-to-fine pyramidal matching method to extract local visual and textual features with different granularities, thus to suppress feature variances and lead to more robust representation and better cross-modal matching accuracy. Additionally, it can be observed that thanks to the inherent capability of BERT for retaining correlations between noun phrases and the integrity of textual features, PMG achieves better performance when combined with BERT, which further optimizes the extraction of fine-grained local textual features.

To further validate the effectiveness of PMG, we also compare it with some other methods proposed in our previous works, including IMG-Net [29], AMEN [36], DSSL [27] and the latest CAIBC [9] on the RSTPReid datasets, and achieve similar results as shown in Table 6 and Figure 6. It can be observed that by utilizing the proposed multi-granular method, PMG+BERT(ours) outperforms CAIBC by 1.50%, 3.10%, and 2.30% on RSTPReid under the Top-1/5/10 accuracies, respectively. It is noted that, although the CAIBC method also utilizes the pretrained BERT model, PMG+BERT still shows better performance, indicating the superiority of PMG under the same condition. On the other hand, we can observe that PMG without BERT is inferior under the Top-1 metric while is superior under the Top-5 and Top-10 metrics, which indicates that even if PMG does not utilize BERT, it is still superior to CAIBC in most cases.

4.3. Ablation Study

Comprehensive ablation studies are carried out to further analyze components of the PMG model. As shown in

Table 7. Ablation Analysis of PMG on CUHK-PEDES.

Method	Top-1	Top-5	Top-10
PMG (001)	62.02	81.82	87.04
PMG (010)	62.18	81.93	87.17
PMG (011)	63.52	82.16	88.73
PMG (100)	61.08	79.98	86.65
PMG (101)	64.04	82.95	88.94
PMG (110)	64.20	83.13	89.08
PMG (111)	64.59	83.19	89.21

and

Table 8. Ablation Analysis of Pooling Methods on CUHK-PEDES.

Method	Top-1	Top-5	Top-10
PMG-AvgPooling	63.47	83.02	89.10
PMG-MaxPooling	63.66	83.15	89.13
PMG-FusedPooling	64.59	83.19	89.21

, the digit ’0’ in the parentheses means the corresponding pyramidal scale is not utilized while ’1’ means that is utilized. From left to right, the three digits denote pyramidal scales 2, 4, and 6, respectively. Several examples of Top-5 retrieval results are shown in Figure 4.

4.3.1. Pyramidal Scales

As shown in Table 7 and Figure 7, PMG achieves the best performance with all three pyramidal scales (denoted by “111”), and the worst performance with only one scale (“001” or “010” or “100”). Intuitively, the model solely utilizing scale 2 instead of scale 3 reaches the lowest accuracy on account that the partitioning is too rough to provide detailed information. Nevertheless, the model achieves the highest accuracy when solely utilizing scale 4 instead of scale 6, which indicates that the image feature maps need not be partitioned as fine as possible, and too many strips may compromise the integrity of visual features. Not unexpected, when two or all of the three scales are employed together, varied pyramidal scales can complement each other and help to capture more cross-modal matching details, thereby leading to superior accuracy performance. Compared to solely utilizing a single scale, PMG utilizing all three scales brings an average accuracy improvement of 2.8%, 2.0%, and 2.3% under the Top-1/5/10 accuracies, respectively. This demonstrates the advantages of the multi-scale approach and confirms that methods utilizing multi-granular representation extraction and matching outperform the ones utilizing a single granularity.

4.3.2. Fused Pooling Method

Table 8 shows the comparison of different pooling methods. We find that the model with the maximum pooling method performs slightly better than the one with the average pooling method. This is reasonable as the average pooling method takes all contextual information into consideration while down-sampling, it may not perform properly if the discriminative signal is surrounded by unrelated signals. In contrast, the maximum pooling method catches the most salient signals for a local view. By fusing the two methods together to take advantage of both contextual information and the most salient signals, PMG achieves a result better than models with either of them.

5. Conclusions

In this paper, we propose a Pyramidal Multi-Granular matching network (PMG) for the text-based person Re-ID task. PMG utilizes a coarse-to-fine pyramidal matching method to learn the cross-modal affinities by extracting and aligning multi-granular visual and textual features. The introduction of the novel gradual transition process between the most coarse global information and the finest local information can suppress the variances of feature extraction and improve the robustness of feature matching. Hence, the problem of imprecision and variances of visual and textual features commonly existing in methods utilizing strict image partitioning strategy with a single granularity, is effectively alleviated. Comprehensive experiments are conducted both on the CUHK-PEDES and RSTPReid datasets, and results clearly indicate that PMG surpasses state-of-the-art methods by a significant margin and yields competitive accuracy of text-based cross-modal retrieval.

The main advantage of PMG lies in its ability to achieve additional performance gains by simply incorporating multiple parallel processing branches. This versatility allows PMG to serve as a flexible network framework, easily integrated with other optimization components to enhance cross-modal retrieval accuracy. However, a notable drawback of PMG is its relatively low computational efficiency. For instance, when utilizing three partitioning scales (i.e., 2, 4, and 6), PMG divides an image feature map into a total of 12 strips. Although the most discriminative feature of a body part can be extracted from a specific image strip, computations are still conducted on the remaining 11 strips, leading to a considerable amount of unnecessary computation. To enhance performance further, we explore the use of a deep neural network to identify unimportant image strips, retaining only the one most conducive to capturing essential features.