A Patch Information Supplement Transformer for Person Re-Identification

Abstract

Extracting fine-grained features from person images has proven crucial in person re-identification (re-ID). Although the research of convolutional neural networks (CNN) has been very successful in person re-ID, due to the small receptive field and downsampling operation, the existing CNNs cannot solve the problem of information loss. The multi-head attention modules in transformer can solve the above problems well. However, since dicing operations destroy the spatial correlation between patches, transformer still loses some local features. in this paper, we propose the scheme of the patch information supplement transformer (PIT) to extract fine-grained features in the dicing stage. Patch pyramid network (PPN) is introduced to solve the problem of local information loss. This is accomplished by dividing the image into different scales through the dicing operation and adding them together from top to bottom according to the pyramid structure. In addition, we insert a learnable identity information-embedding module (IDE) to reduce the feature bias of clothing and camera perspective. Experiments verify the superiority and effectiveness of PIT compared to state-of-the-art methods.

1. Introduction

Person re-identification [1] technology is a crucial but challenging task in the fields of multimedia and computer vision, which has been applied in various applications. The purpose of person re-identification is to associate a particular person quickly across different scenes and camera perspectives. The main challenge comes from the difficulty of accurately distinguishing a person with similar appearances. For example, as shown in Figure 1, the clothes of the two women are of the same colour, the postures of the two men have changed, and the occlusions cover many key pedestrian features. Therefore, extracting fine-grained features is crucial to person re-ID [2].

According to the authors in [3,4], CNN-based methods are the very intriguing in current research. However, the research methods based on CNN still face two challenges: (1) obtaining rich contextual information in global feature extraction is crucial for person re-ID [5]. Due to the convolution and the small receptive field, CNN methods only focus on local small feature information [6]. (2) Extracting detailed fine-grained features with detailed information in the person image is also important. However, the convolution and pooling operations of CNN reduce the resolution of the image, leading to feature information losses. This greatly affects the accuracy of fine-grained feature extraction, such as similar appearance [7].

The multi-head attention modules in transformer [8] and the operations of removing convolution and pooling can better solve the above-mentioned critical problems facing CNNs. The reasons are as follows. (1) The multi-head attention modules can capture the relationship between long distances and make the model concentrate more on the relationship between different regions. (2) Removing operations such as convolution and pooling to reduce feature loss allows the model to save more detailed feature information [9]. Combining the above advantages, we chose the transformer network as the baseline for person re-ID.

Although transformer has the above-mentioned significant advantages, it still has tremendous challenges in the face of person re-ID tasks, such as occlusion, lighting, camera perspective, pose diversity, etc. Therefore, considering the above problems, the transformer should be improved. Many attempts have been made to face this problem when researching person re-ID based upon the CNN. As for extracting fine-grained features, local part features [10,11,12] and auxiliary features [13,14] have been shown to be essential and effective. CNN-based methods extract features from a complete image, while the slicing operation in transformer cuts the image into independent patches, directly migrating CNN-based methods into transformer results in suboptimal recognition [15]. In addition, directly adding the CNN-based side semantic information module to the transformer network cannot improve the use of its encoding ability. Therefore, it is also a big challenge to embed the module based on the specific design of CNN into the transformer model.

We proposed a scheme of the PIT to solve the above-mentioned critical problems. Firstly, we proposed a PPN to solve the problem that the correlation between the image regions is destroyed due to the dicing operation. This divides the image into blocks of different scales by dicing, which are connected and added from top to bottom according to a pyramid structure for further fine-grained feature learning. The PPN structure can make the most miniature scale patch to obtain the feature information of other scale images to strengthen its fine-grained features. Therefore, the network can extract global perturbation-invariant and fine-grained features. To the best of our knowledge, the issue of transformer-based identity information encoding has not been investigated [16]. Different from utilizing semantic information embedding in CNN-based methods, we designed the IDE for Transformer, effectively integrating identity semantic information through learnable embeddings to reduce feature bias of peripheral information. For instance, the proposed IDE can solve the problem of matching similarity caused by similar clothing.

In summary, this is the first study to design a PPN structure and apply it to person re-ID. The main contributions of this paper are as follows:

• We designed a PPN structure and applied it with transformer for person re-ID, solving the global perturbation caused by spatial dimension segmentation and poor fine-grained features.
• We provided an approach to identity information embedding, encoding identity information through learnable embeddings. Thus, effectively addressing the problem of learned feature bias.
• The proposed PIT had better performance improvement in Market-1501 [17], Duke-MTMC [18], Occluded-Duke [19] and MSMT17 [20].

2. Related Work

2.1. Auxiliary Feature Representation Learning

The information interference is the most challenging problem in person re-ID [16]. Auxiliary feature representations have been adopted to cultivate consistent person re-ID to deal this problem. Some common methods are adopted to enhance auxiliary feature representation such as additional annotation information [13,14,21,22,23], timing information [24,25] and generated/augmented training samples [26]. Furthermore, some studies have fused representations from multiple levels [4,27,28], requiring extra methods, such as body-part detection [4,27] or attention mechanisms [28]. However, while most of these methods have been designed based on CNNs, the slicing operation in transformer cuts the image into independent patches, making most of these methods unsuitable for transformer for direct application [15]. Thus, an auxiliary feature representation method of the transformer for person re-ID is needed.

2.2. Visual Transformer

Transformer [9] was originally proposed and applied in the field of natural language processing (NLP) and has gradually become the mainstream method for NLP tasks. Recently, transformer has been shown to outperform traditional methods in many vision tasks, such as object detection [29], generative adversarial networks [30], action recognition [31]. Visual transformers (VIT) [8] are the first to transformers to be applied to image vision. They first divide the input image into image patches, then use linear projection to map it to a 1D vector and add learnable ‘class labels (CLS)’, and finally pass the merged 1D vector to the transformer encoder. TransReID [32] is the first application of VIT in the field of object re-ID. The patch shuffle operation and side information module were designed to verify the advantages of the transformer on the re-ID task. PAT [27] integrates transformer architectures into CNNs. PAT [27] designs part diversity part discriminability to diversify part discovery. However, its failed solve the problem of fine-grained feature loss caused by transformer dicing operations. Thus, it is still an urgent problem to obtain high correlations from scattered patch blocks in transformer. This paper aims to propose a transformer-based patch information supplementation method to address the problems in existing transformer fine-grained feature extraction schemes.

3. Methodology

In this section, we will present the designed techniques to improve the transformer baseline. As demonstrated in Figure 2, the proposed PIT consists of the baseline, a patch feature extract network PPN, and the semantic information encoder IDE. The first part is the baseline, we use the ViT model as the baseline for person re-identification task. The details will be introduced in Section 3.1. The second part is the PPN, which obtains fine-grained features that are invariant to global disturbances by solving the local feature information loss caused by dicing operations. The details of which are introduced in Section 3.2. The third part is the IDE, which aims to avoid interfering with the generation of information encoding. The same person should have the same identity information encoding. The details will be introduced in Section 3.3.

3.1. Baseline

As shown in Figure 3, input an image $x\in R^{C,H,W}$, while C, H, and W represent the channel size, height and width, respectively. We divide it into N fixed-size patches $\left\{x_p^i\mid i=1,2,\cdots ,N\right\}$ through the slicing operation. These patches are flattened into one-dimensional tensors and then projected onto a lower-dimensional space D using a linear transformation. The resulting feature vectors for each patch are used as inputs for the subsequent layers. In addition, a learnable embedding token denoted as $x_{cls}$ is pre-loaded into the input sequence, with its output serving as the global feature denoted by f. Learnable position embeddings are also incorporated to capture spatial information. The input sequence to the transformer layers can be represented as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Z_0=\left[x_{cls};F\left(x_p^1\right);F\left(x_p^2\right);\cdots ;F\left(x_p^N\right)\right]+P\end{array}\end{array}$$

(1)

where $Z_0$ is the input sequence embeddings of the baseline, and $P\in R^{\left(N+1\right)\times D}$ is the location information embedding. F is the linear mapping of the diced patches $x_p^i$ to D dimensions. Additionally, l transformer layers is used to learn feature representations.

Patch Slicing. Patch splitting refers to the process of dividing an input image into several small, equally-sized blocks called “patches”. We use a sliding window approach to divide an image of size $\mathit{H}$ × $\mathit{W}$ into N fixed-size patches. The stride is denoted as $\mathit{S}$, and the patch size is denoted as $\mathit{P}$.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill N=N_H\times N_W=\left[\frac{H+S-P}{S}\right]\times \left[\frac{W+S-P}{S}\right]\end{array}\end{array}$$

(2)

where $\left[·\right]$ is the floor function and $\mathit{S}$ is set same to $\mathit{P}$. $N_H$ and $N_W$ represent the number of splitting patches in height and width, respectively.

Position Embeddings. The image resolution of person re-ID is different from the original resolution of VIT images. The pre-trained position embeddings in ImageNet cannot be directly loaded into the baseline. Therefore, we introduce bilinear 2D interpolation to handle the input image resolution.

Supervised Learning. We use ID loss to optimize the baseline. ID loss is cross-entropy loss without label smoothing and can be expressed as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L(y,\widehat{y})=\sum _{i=1}^M-q_{i}log\left(p_i\right){\{}_{q_i=1,y=i}^{q_i=0,y\ne i}\end{array}\end{array}$$

(3)

where $\mathit{M}$ represents the total number of people, y represents the truth ID label, $q_i$ represents the target probability, and $p_i$ represents the ID prediction logits of class $\mathit{i}$. The smaller the cross-entropy loss, the closer the predicted result is to the true result.

3.2. Patch Pyramid Network

Although encoding the transformer provides a larger receptive field, the dicing operation reduces the correlation of the patch in the spatial dimension, leading to an inability to effectively extract fine-grained features.

As shown in Figure 4, the proposed PPN is a multi-scale patch fine-grained feature extraction network. Given an image $x\in R^{C,H,W}$, C, H, and W represent the channel size, height and width, respectively. Then, we split it into N fixed-sized patches $\left\{x_p^i\mid i=1,2,\cdots ,N\right\}$ by PPN. We explain the details of PPN below.

Based on convolution operations with four different kernel sizes (2 × 2, 4 × 4, 8 × 8, 16 × 16), x is divided into various patch sizes $\left\{P_0,P_1,P_2,P_3\right\}\in R^{C,H,W}$, which only alter the size of the patches. The size of $\mathit{H}$, and $\mathit{W}$ among the patches are expressed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill H_3& =2H_2=8H_1=16H_0\hfill \\ \hfill W_3& =2W_2=8W_1=16W_0\hfill \end{array}\end{array}$$

(4)

Then, $P_0,P_1,P_2,$ and $P_3$ are arranged in a pyramid structure, where the top of the pyramid is the largest patch scale $P_3$ and the bottom is the smallest $P_0$. Starting from the top of the pyramid, the downsampling operator $F_{2D}$ is applied to scale the patches. After downsampling, adjacent patches are summed together. The above two operations are repeated until the patches reach the smallest scale, which is the target patch. Accordingly, the object patches $x_p^i\in R^{C,H,W}$ can be obtained. Moreover, the proposed feature extraction module $F_E$ is used before each downsampling operation to obtain $\left\{P_1^{″},P_2^{″},P_3^{″}\right\}\in R^{C,H,W}$. It is worth noting that the minimum size of a patch does not perform feature extraction, having an impact on model accuracy due to the shallow feature extraction operation. Finally, the output $x_p^i$ can be deduced, given by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {P_3}^{″}& =F_E\left(P_3\right)\hfill \\ \hfill {P_2}^{″}& =F_E\left(F_{2D}\left({P_3}^{″}\right)+P_2\right)\hfill \\ \hfill {P_1}^{″}& =F_E\left(F_{2D}\left({P_2}^{″}\right)+P_1\right)\hfill \\ \hfill x_p^i& =F_{2D}\left({P_1}^{″}\right)+P_0\hfill \end{array}\end{array}$$

(5)

where $F_{2D}$ represents the downsampling operation with a convolution kernel of 3 × 3.

As shown in Figure 5, the extraction module consists of four parts: Input $P\in R^{C,H,W}$, AC module, AS module and output $P^{″}\in R^{C,H,W}$. The AC module is an attention mechanism module in the channel dimension, and the AS module is an attention mechanism module in the spatial dimension. The two modules work in parallel in the extraction module by being connected together. They, respectively, extract channel features and spatial features in the patch block, and then add them together to obtain the final fine-grained feature.

AC module. In the channel dimension, the AC module is an adaptive feature extraction module. Two 2D maps $P_M\in R^{C,1,1}$ and $P_A\in R^{C,1,1}$ can be generated by the parallel maximum-pooling $F_{Max}$ and average-pooling $F_{Avg}$, which resize the input P from $C,H,W$ to $C,1,1$. Then, in order to avoid overcomplicating the model, the obtained 2D feature maps are processed by the squeeze and permute operation $F_S$ to obtain the $P_{SM}\in R^{1,C}$ and $P_{SA}\in R^{1,C}$. After this, the 1D convolution module $F_{1D}$ with a kernel size of 3 × 3 is used to process $P_{SM}$ and $P_{SA}$ separately. The local cross-channel interaction can be realized, leading to $P_{CM}\in R^{1,C}$ and $P_{CA}\in R^{1,C}$. Then, both are added together to obtain $P_C\in R^{1,C}$. The $P_{sg}\in R^{1,C}$ is acquired by applying a sigmoid activation function $F_{sg}$ to $P_C$. Finally, the weight of the channel dimension $P_{weight}\in R^{C,1,1}$ is obtained after a dimensional upgrade operation $F_{us}$. The operation of the AC module can be deduced by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill P_{weight}=F_{us}\left[F_{sg}\left[F_{1D}\left(F_S\left(F_{Max}\left(p\right)\right)\right)+F_{1D}\left(F_S\left(F_{Avg}\left(P\right)\right)\right)\right]\right]\end{array}\end{array}$$

(6)

The adaptive channel feature $A_C\left(P\right)$ can be acquired by multiplying $P_{weight}$ by the input patch P, giving

$$\begin{array}{c}\hfill \begin{array}{c}\hfill A_C\left(P\right)=P_{weight}\otimes P\end{array}\end{array}$$

(7)

AS module. The AS module is an adaptive feature extraction module in the spatial dimension. Two 2D maps ${P_M}^{\prime }\in R^{1,H,W}$ and ${P_A}^{\prime }\in R^{1,H,W}$ can be generated through the adaptive maximum-pooling $F_M$ and adaptive average-pooling $F_A$, producing two 1*H*W feature maps from the input P. After this, the ${P_C}^{\prime }\in R^{2,H,W}$ is obtained by concatenating $P_M$ and $P_A$ through a concatenation operation ⊕. Then, ${P_{Cov}}^{\prime }\in R^{1,H,W}$ is obtained by a convolution $F_{2D}$ with a kernel size of 3 × 3 that turns the number of channels in $P_C$ to 1. The adaptive weight ${P_{weight}}^{\prime }$ of the spatial dimension is acquired by applying a sigmoid activation function $F_{sg}$ to the ${P_{Cov}}^{\prime }$. Accordingly, the operation of the AS module can be computed as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill P_{weight}^{\prime }=F_{sg}\left[F_{2D}\left(F_M\left(P\right)\oplus F_A\left(P\right)\right)\right]\end{array}\end{array}$$

(8)

Similarly, the adaptive spatial feature $A_S\left(P\right)$ can be defined as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill A_S\left(P\right)=P_{weight}^{\prime }\otimes P\end{array}\end{array}$$

(9)

Based on the AC and AS operation, the output $P^{″}$ of the extraction module can be computed as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill P^{″}=w_1\ast A_C\left(P\right)+w_2\ast A_S\left(P\right)\end{array}\end{array}$$

(10)

where $w_1$ and $w_2$ are the weights that change with the gradient whose sum is 1.

3.3. Identity Information Embedding Module

Despite fine-grained feature being obtained, the impact of clothing changes cannot be ignored. In other words, the model may not be able to discriminate different objects from the same angle due to the bias of clothing information. Therefore, the proposed IDE incorporates the identity information of people into the embedding representation to obtain robust features.

Similar to positional embeddings employing learnable embeddings to encode positional information, we insert learnable one-dimensional embeddings to preserve identity information. In particular, as shown in Figure 1, we insert the identity information embedding into the transformer encoder with the patch and position embeddings. Specifically, assuming that there is a total of M person IDs, we initialize the learnable identity information embedding as $G_{ID}\in R^{M\times D}$. If the ID of a person is k, the identity information embedding can be expressed as $G_{ID}\left(k\right)$. Unlike positional embeddings which vary between patches, identity information embeddings $G_{ID}\left(k\right)$ are the same for all patches of an image.

As the identity information embedding, patch embedding, and position embedding are all linearly mapped to a D-dimensional space, their input sequences can be directly added for information integration. The IDE can be written as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {Z_0}^{\prime }& =Z_0+\lambda G_{ID}\left(k\right)\hfill \end{array}\end{array}$$

(11)

where ${Z_0}^{\prime }$ is the input sequence after adding the identity information embedding, $Z_0$ is the original input sequence in the baseline, and $\lambda$ is the parameter of the identity information embedding. The transformer encoding layer can encode embeddings of different distribution types and add these features directly.

4. Experiments

4.1. Datasets

In order to demonstrate the effectiveness of our methods, the experiments were conducted on the three widely used holistic re-ID datasets (Market-1501 [17], Duke-MTMC [18], MSMT17 [20]), and one occluded re-ID dataset (Occluded-Duke [19]).

Table 1. Dataset statistics used in this article.

Dataset	#ID	#Cam	#Image	#Query	#Gallery
Market-1501	1501	6	32,668	3368	15,913
Duke-MTMC	1812	8	36,411	2228	17,661
MSMT17	4101	15	126,441	11,659	82,161
Occluded-Duke	1812	8	15,618	2210	17,661

provides a brief explanation of the datasets, explained in detail as follows.

Market-1501 [17] has a total of 32,668 images, consisting of 1501 identities under 6 cameras. It is split into a training set of 12,936 images of 751 identities and a test set of 19,732 images of 750 identities.

Duke-MTMC [18] has a total of 36,411 images from 1812 people captured by 8 cameras. The 16,522 images of 702 people are randomly selected as the training set, the 2228 remaining images are used as query images and the 17,661 gallery images are used as a test set.

MSMT17 [20] is the largest dataset with 4101 IDs, with 126,441 images captured from 15 cameras. The training set has 32,621 images of 1041 identities, and the test set has 93,820 images of 3060 identities. In the test set, the 11,659 images are randomly selected as query images.

Occluded-Duke [19] is a split of Duke-MTMC [18] that preserves occluded images and removes overlapping images. It contains 15,618 training images, 17,661 gallery images, and 2210 occlusion query images.

4.2. Experimental Strategy and Experimental Environment

The size of all images were adjusted to 256 × 128 for training and testing. As for the data augmentation, we employed random horizontal flipping, padding, random cropping and random erasing. The SGD optimizer was adopted with a momentum of 0.9 and a weight decay set as $1\times {10}^{-4}$. The initial learning rate and the batch size are set to 0.008 and 64, respectively.

Limited by experimental equipment, all experiments used a Nvidia TITANX GPU, which were trained on the PyTorch platform with PyTorch = 1.7.0, torchvision = 0.8.0, CUDA = 11.1 and CUDNN = 8.0.

Evaluation Protocols. Cumulative matching characteristic (CMC) curves and mean average precision (mAP) were adopted to evaluate the quality of different the re-ID models. The re-ranking was not taken to refine the matching results further in the experiments that are performed in a single query mode.

4.3. Results of Backbones

In this section, we compare the performances of the backbones. We chose several backbones, including SqueezeNet, ShuffleNet, DenseNet, ResNet50, ResNet101, SEResNet50, SEResNet101 and VIT-B/16, where VIT-B/16 represents the VIT baseline with patch size 16.

As shown in

Table 2. Comparison of different backbones. VIT-B/16 is the baseline model of this paper, referred to as baseline for short.

Backbone	Market-1501		Duke-MTMC		Occluded-Duke
	mAP	Rank-1	mAP	Rank-1	mAP	Rank-1
SqueezeNet	30.7	49.1	21	36.3	6.7	11.2
ShuffleNet	34.1	56.9	25.2	41.1	9.5	14.8
DenseNet	62.5	83.9	61.6	73.6	28.8	42.1
ResNet50	67.9	84.5	54.3	73	31.1	42.7
ResNet101	68.7	84.9	59.7	76.8	38.4	41
SEResNet50	73.7	88.4	60.1	77.6	32.7	42.8
SEResNet101	75.1	89.3	63	80.2	35.9	45.3
VIT-B/16	86.7	94.5	79.4	88.4	52.6	60.2

, there is a big gap in performance between the VIT-B/16- and CNN-based backbones. On the re-ID benchmark, the VIT-B/16 greatly outperforms the CNN-based backbones, the VIT-B/16 achieved at least 11.6% mAP and 8.2% rank-1 over SEResNet101. This suggests that the multi-head attention mechanism of transformer can obtain a larger receptive field. Thus, we choose the VIT-B/16 as the baseline.

4.4. Ablation Study of PPN

As shown in

Table 3. The ablation study of patch pyramid network on the MSMT17.

Backbone	${\mathit{P}}_0$	${\mathit{P}}_1$	${\mathit{P}}_2$	${\mathit{P}}_3$	Extraction Module	mAP	Rank-1	Rank-5	Rank-10
Baseline	√					60.9	81	90.2	92.9
+PPN	√	√			√	61.2	81.2	90.2	92.9
+PPN	√	√	√		√	61.9	81.8	90.3	93.0
+PPN	√	√	√	√		61.7	81.5	90.2	92.9
+PPN	√	√	√	√	√	62.5	82.1	90.5	93.1

, we evaluate the superiority of our proposed PPN. Compared to the baseline, the PPN improves by +1.6% mAP and +1.1% rank-1. Increasing number of P in the PPN, the performance of the PPN gradually increases, suggesting that adding larger patches to the PPN is helpful to extract spatial features in the dicing stage. Furthermore, by comparing the PPN and PPN (without the extraction module), we can find that the extraction module operation can help the model to improve by +0.8% mAP and +0.6% rank-1, suggesting that the extraction module in our PPN helps the model to effectively obtain fine-grained features. The visual attention map in Figure 6 shows that the model obtains more context-aware information and better fine-grained features through the PPN operation, improving the anti-interference ability of the model.

4.5. Ablation Study of IDE

Performance Analysis. As shown in

Table 4. Ablation study of IDE on the Market-1501.

Method	mAP	Rank-1	Rank-5	Rank-10
Baseline	86.7	94.5	98.2	99.0
Baseline + IDE	88.4	95.3	98.5	99.1

, we assess the utility of the proposed IDE. The experimental results show that our method achieves 1.7% mAP and 0.8% rank-1 improvement with slight improvments of the both rank-5 and rank-10 at extremely high performances, showing that the proposed IDE can work well on the baseline. The IDE can separate invariant ID features from semantic features, significant for person re-ID to eliminate interference.

Ablation study of $\lambda$. As demonstrated in Figure 7, we analysed the effect of parameter $\lambda$ on the IDE. Particularly, the baseline reached 86.7% mAP and 94.5% rank-1 with $\lambda$ set as 0. When $\lambda$ increases, the performance of the IDE improves by 88.4% mAP and 95.3% rank-1 ($\lambda$ = 2.0), indicating that our methods can learn invariant features well. With a further increase in $\lambda$, the feature embedding and position embedding gradually decrease to set the performance of the system.

4.6. Ablation Study of PIT

As shown in

Table 5. The ablation study of PIT on the MSMT17.

Backbone	PPN	IDE	mAP	Rank-1	Rank-5	Rank-10
Baseline	×	×	60.9	81	90.2	92.9
	√	×	62.5	82.1	90.5	93.1
	×	√	62.8	81.9	90.4	92.9
PIT	√	√	63.1	82.3	90.8	93.1

, compared to the baseline, our proposed PPN and IDE improve the performance by +1.6% mAP /+1.1% rank-1 and +1.9% mAP /+0.9% rank-1, respectively. The ablation studies illustrate that the proposed PIT achieves a better performance of 63.1% (+2.2%) mAP and 82.3% (+1.3%) rank-1 than the baseline. The experimental results demonstrate that the PPN and IDE complement each other in person re-ID.

4.7. Comparison with State-of-the-Art Approaches

As shown in

Table 6. Comparison with the state-of-the-art methods.

Methods	Market-1501		Duke-MTMC		MSMT17		Occluded-Duke
	mAP	Rank-1	mAP	Rank-1	mAP	Rank-1	mAP	Rank-1
PCB(2018) [4]	81.6	93.8	69.2	83.3	-	-	33.7	42.6
AlignedReID (2017) [12]	79.3	91.2	-	-	-	-	37.3	51.4
PGFA (2019) [19]	76.8	91.2	65.5	82.6	-	-	43.8	55.1
HONet(2020) [33]	84.9	94.2	75.6	86.9	-	-	52.3	62.8
ICE (2021) [34]	86.6	95.1	76.5	88.2	50.4	76.4	-	-
Transreid (2021) [32]	88.0	94.7	81.2	90.1	63.9	82.7	55.6	62.8
PAT (2021) [27]	88.0	95.4	78.2	88.8	-	-	53.6	64.5
AAformer (2021) [35]	87.7	95.4	80.0	90.1	63.2	83.6	58.2	67.0
DRL-Net (2022) [36]	86.9	94.7	76.6	88.1	55.3	78.4	50.8	65.0
SpCL (2022) [37]	76.7	90.3	68.8	82.9	26.8	53.7	-	-
FED (2022) [38]	85.9	94.8	78.0	89.4	-	-	55.8	67.4
ISE (2022) [39]	87.8	95.6	-	-	51.0	76.8	-	-
Baseline	86.7	94.5	79.4	88.4	60.9	81	52.6	60.2
Ours	88.8	95.6	81.3	90.4	63.1	82.3	54.8	61.7

, we compare our model with the state-of-the-art methods on three holistic benchmarks and one occluded benchmark to show the effectiveness of the proposed PIT. In addition, we also compare the results from the baseline.

Market1501. Compared with the state-of-the-art methods, the proposed PIT acquires outstanding performances. Specifically, when compared with other transformer-based networks [17,18,33,38], our method receives the best result of 88.8% (+0.8%) mAP. There is little difference in the results between these methods because the exploration of Market1501 is saturated.

Duke-MTMC. Although the proposed PIT performs close to the other methods, our model still derives first-rate performances, showing the robustness of the proposed methods. Furthermore, compared to the baseline, our method exhibits better results (+1.9% mAP/+2.0% rank-1), which validates our work on the Transformer baseline.

MSMT17. Being the largest person re-ID dataset means that robust feature extraction in complex environments is more difficult. Due to the use of a greater number of cameras to capture people in the MSMT17 dataset, there is a greater diversity of clothing and posture changes among the individuals, leading to a suboptimal recognition performance of the models. However, the proposed PIT proved its reliability with good experimental results, outperforming the ISE [39] when integrating identity information.

Occluded re-ID. From the experimental results, the introduced PIT still achieves a good score of 54.8% mAP in the face of severe occlusion and fewer features. This is because occlusions generate interfering features, resulting in a suboptimal recognition performance of the models. Additionally, compared with the PAT [27] with aligned body parts, our method achieved better results (+1.2% in front of PAT [27]) without aligning body parts.

4.8. The Matching Visualization Results

We compare the ranking results of the baseline and PIT in Figure 8, and the results indicate that PIT can effectively address the recognition inaccuracy caused by similar clothing, human pose variations, and partial occlusions. The identity information provides useful assistance to the model.

5. Conclusions

In this paper, we proposed a patch information supplement method (PIT) that can handle local information loss. By using the introduced PPN, PIT can solve decreasing spatial correlations caused by dicing operations and obtain the fine-grained features. Furthermore, we designed the IDE to eliminate the influence of clothing changes to separate ID-related features from the semantic features. Extensive experimental results for re-ID verify the superiority of PIT and the effectiveness of its components.

However, our model does not perform optimally on occluded re-ID data. It cannot offer a high generalization performance in a randomly occluded environment. We will concentrate on how to better utilize structural information to handle the interference of occlusions in future research.