Unsupervised Domain Adaptive Person Re-Identification Method Based on Transformer

Abstract

Person re-identification (ReID) is the problem of cross-camera target retrieval. The extraction of robust and discriminant features is the key factor in realizing the correct correlation of targets. A model based on convolutional neural networks (CNNs) can extract more robust image features. Still, it completes the extraction of images from local information to global information by continuously accumulating convolution layers. As a complex CNN, a vision transformer (ViT) captures global information from the beginning to extract more powerful features. This paper proposes an unsupervised domain adaptive person re-identification model (ViTReID) based on the vision transformer, taking the ViT model trained on ImageNet as the pre-training weight and a transformer encoder as the feature extraction network, which makes up for some defects of the CNN model. At the same time, the combined loss function of cross-entropy and triplet loss function combined with the center loss function is used to optimize the network; the person’s head is evaluated and trained as a local feature combined with the global feature of the whole body, focusing on the head, to enhance the head feature information. The experimental results show that ViTReID exceeds the baseline method (SSG) by 14% (Market1501 → MSMT17) in mean average precision (mAP). In MSMT17 → Market1501, ViTReID is 1.2% higher in rank-1 (R1) accuracy than a state-of-the-art method (SPCL); in PersonX → MSMT17, the mAP is 3.1% higher than that of the MMT-dbscan method, and in PersonX → Market1501, the mAP is 1.5% higher than that of the MMT-dbscan method.

1. Introduction

Person re-identification (ReID) is a prominent research topic in computer vision. It is widely regarded as a sub-problem of image retrieval. It mainly involves the use of computer vision technology to judge whether there is a specific person in the image or video, i.e., to retrieve the picture of a pedestrian across devices given a monitored pedestrian image. ReID technology can make up for the visual angle limitation of the current fixed cameras and can be combined with person detection and person tracking technology. It is widely used in intelligent security, video surveillance, and many other fields. Due to the different scenes in real life, ReID also faces different challenges, such as changes in pedestrian appearance brought on by various cameras, the loss of pedestrian features due to target occlusion, and diminished differentiation due to close clothing color matching between different pedestrians. For a long time, CNN-based methods have dominated ReID research [1,2,3,4,5].

The actual receptive field based on the CNN method is far smaller than the theoretical receptive field, which is not conducive to entirely using context information for feature capture. Although the receptive field can be expanded by continuously stacking deeper convolution layers, it will cause the model to be too bloated. The amount of calculation will increase sharply, contrary to the original intention of simplicity and practicality. The transformer network does not have this concern [6]. Its advantage is the use of the way of attention to capture the global context information, establish a long-distance dependence on the target, and extract more powerful features. In addition, the migration effect of the transformer is better than that of CNN, and the model pre-trained on large datasets can be easily migrated to small datasets.

The following three improvements are made in this paper to solve the problem of feature loss caused by target occlusion and the discrimination degradation caused by the similar clothing color of different pedestrians. (1) In Section 3.3, the transformer encoder replaces ResNet [7] as our feature extraction network for unsupervised adaptive ReID so that we can better combine the global information of the image and creatively use ViT [8] for unsupervised domain adaptive person re-identification. (2) In Section 3.2, to mine the potential similarity from the whole to the part, the grouping idea of the improved SSG method [3] is adopted, and the person’s head is trained as the local feature combined with the two branches of the person global feature. (3) In Section 3.4, to make the class more compact, the center loss function [9] proposed in face recognition is introduced, and the cross-entropy and triplet loss function [10] are combined as the combined loss function to update the network.

2. Related Work

At present, person re-identification has two primary directions: one is a supervised learning method with labeled data training, and the other is an unsupervised learning method without labeled data. Unsupervised learning is usually divided into sub-problems to study, such as the unsupervised domain adaptive model and the semi-supervised model.

In the supervised learning of person re-identification, the method based on local features extracts the feature of a region in the image. It uses multiple local features as the final feature [1,2,11,12,13]. The method based on video sequence uses a continuous series of person pictures for re-identification or a sequence search sequence to solve the problem [14,15,16]. The method based on GAN Network [17] uses GAN to generate picture samples and increase the data for training [18,19,20,21]. Because there is data bias between the labeled source domain and the unlabeled target domain, the ReID model trained on the source domain often performs poorly in the target domain. At the same time, due to the high acquisition cost of annotated large-scale datasets, research on person re-identification is gradually changing to an unsupervised domain adaptive method (UDA) [22].

Unlike unsupervised learning, which is not needed to label the source domain, the adaptive domain learning of person re-identification uses the labeled source domain data for training. In recent years, the unsupervised domain adaptive method of person re-identification is mainly the pseudo-label algorithm based on clustering. This algorithm’s main challenge is improving the accuracy of pseudo-labels and reducing the impact caused by noisy pseudo-labels. Representative algorithms such as SSG [3] use the potential similarity of unlabeled target domain data (from global to local), automatically construct multiple clusters from different feature groups, and assign labels to these independent clusters, and these labels are used as pseudo-labels to supervise training. This allocation and training process is performed repeatedly and alternately until the model converges. PAST [23] proposes a new self-training strategy, which gradually enhances the model’s ability by alternately using the conservative and promoting stages. In addition, to improve the reliability of the selected triplet samples, the triplet loss function [10] based on the ranking is introduced into the conservative stage; it is an unlabeled objective function based on the similarity between data pairs. MMT [4] uses the “synchronous average teaching” framework to optimize pseudo-tags. The core idea is to use more robust “soft” tags to optimize pseudo-tags online. AD-Cluster [24] uses GAN [21] to generate new sample data, inputs all data (new + old) into a CNN for feature extraction and re-clustering, and is iteratively optimized according to max–min (maximization of sample generation diversity and minimization of intra-class distance). SPCL [5] proposes the use of all source domain data and target domain data in training; the use of a hybrid memory model to treat all source domain classes, target domain clusters, and each cluster outlier instance of the target domain as equal categories for training supervision; and the use of momentum update to dynamically update and change the prototypes of these three categories in the hybrid memory model. MAR [25] puts forward a new idea of ReID applied to unsupervised learning: soft multilabel. Three weighted combinations of loss functions are designed to train the model based on soft multilabel. DG-Net++ [26] proposes a cross-domain and cyclic consistent image generation method, which decomposes the source image and target image by simulating three potential spaces (including the shared appearance space capturing ID-related features) through the corresponding encoder to jointly solve the problem of mutual promotion decoupling and domain adaptation. D-MMD [27] puts forward the UDA method in a different space: the D-MMD loss realized in the other areas, the distance aligned by D-MMD loss, and the optimization recognized by gradient descent on a relatively small batch size. There are also some studies based on the adversarial discriminative domain adaptation method, which explores the impact of domain-invariant features of pedestrian attributes (e.g., hair length, clothing style, and color) on person re-identification methods [28]. Some other studies propose a style transfer-based approach (STReID), which converts the source domain image to the style of the target domain image and trains the converted style data with the source domain data to obtain better performance across domains [29]. Some other studies proposed the camera penalty learning (CPL) method, which adds camera-ID-penalty to the traditional triplet loss to solve the pseudo-label noise caused by cross-camera target retrieval [30].

The transformer [6] emerged in the field of machine translation. It proposes an attention-based encoder–decoder architecture, which uses the attention mechanism to improve the model training speed and automatically capture the relative correlation at different positions of the input sequence. It can work in high parallel, and the training speed is breakneck. The potential of the transformer blooms everywhere in the field of natural language processing and shines brightly in the field of computer vision. The conformer [31] presents a hybrid network structure, which combines convolution operation and a self-attention mechanism to enhance feature representation. It relies on a feature coupling unit (FCU) to interactively integrate local and global feature representation at different resolutions. The ViT [8] indicates that the transformer has robust scalability (the more extensive the model, the better the effect). Further improving the number of parameters can bring better results; a pure transformer can also perform well directly as a feature extraction network for computer vision tasks. In the study of supervised person re-identification in 2021, TransReID [32] has proved its effectiveness as a transformer. It uses a transformer as the backbone network to extract features alone, rather than simply using a CNN (such as ResNet [7]) or CNN+ transformer [31] to extract features. TransReID obtains a state-of-the-art (SOTA) effect.

3. Proposed Method

Figure 1 shows the overall framework. Given the strong representation ability and feature extraction ability of the transformer’s attention mechanism, this paper proposes an unsupervised domain adaptive method (ViTReID) based on the transformer, composed of three parts. Firstly, the transformer network pre-trained on ImageNet [33] is used as the pre-training weight (this section is described in Section 3.1). Then the transformer encoder is used to replace the traditional CNN neural network as the feature extraction network. Finally, the network is updated by stochastic gradient descent (SGD), and the network is optimized by combining the loss function of triplet loss, center loss, and cross-entropy loss function. $X^s$ and $X^t$ are the training images of the source and target domains, respectively. The ViTReID network is trained in two branches (see Section 3.2): the local branch (head) and the global branch (body). Then the source and target domain data are passed through the feature extraction network (see Section 3.3) and the global average pooling (GAP) to obtain the feature vectors $f^s$ and $f^t$ in the source and target domains, respectively. GAP is a structured regularization of the network that prevents overfitting, incidentally using global information. Finally, the feature vectors are optimized by the fully connected layer and the loss functions in the source and target domains to obtain the final weight results (see Section 3.4). FC is the feature vector layer after full connection; it has the output size of 1 × 1 × 768 after feature extraction is output as 768 after the full connection.

3.1. Pre-Training Weight

For application in industrial scenes, the weight of the basic ViT model pre-trained on the large dataset ImageNet [33] is used as the pre-training weight. Because a transformer model pre-trained by large datasets can achieve better results, the backbone of the pre-training model adopts the transformer base model, and the picture size is set to 384 × 128, the embedding dimension is set to 768, and the attention header is set to 12. We use the SSG model of the ResNet50 backbone network as the baseline model and compare it with the ViTReID model. ViTReID adopts the standard paradigm of the transformer and fully follows the pre-training–fine-tuning process. It is trained on the extensive dataset ImageNet. When the bottom feature extraction reaches good performance, it is combined with the person re-identification task, retains the bottom feature extractor, and fine-tunes the upper layer. The training optimizer uses SGD, the learning rate is set to 3 × 10⁻⁴, the weight decay is set to 5 × 10⁻⁴, and the epochs are set to 120.

3.2. Head Branch Training

For local branch training, the feature grouping idea proposed by SSG [3] divides the person’s features into upper and lower parts. The features based on clothing will change with the external environment or monitoring equipment, resulting in the low accuracy of the extracted features. Using head feature training, the extracted facial features and facial features have better robustness and higher discrimination. Therefore, this paper will focus on head information as local features.

To construct the head branching algorithm for ViTReID, We first perform image serialization of the pedestrian image $x_{body}^i$ in the dataset by dividing the pedestrian image horizontally into two parts and taking only three-tenths $\frac{3H}{10}\times \frac{3W}{10}\times C$ of it as the head. Next, the tokens generated from the image sequences of the overall image $H\times W\times C$ and the head image $\frac{3H}{10}\times \frac{3W}{10}\times C$ are input to the ViT/B-384 backbone for feature extraction, as shown in Figure 2, to obtain the overall body feature map (denoted as $F_{body}^i\in {ℝ}^{H\times W\times C}$) and the low-slice head feature map (denoted as $F_{head}^i\in {ℝ}^{\frac{3H}{10}\times \frac{3W}{10}\times C}$). Then the global average pooling (GAP) operation is applied to the body feature map $F_{body}^i$ and the head feature map $F_{head}^i$ to obtain the $F_{body}^i$ and the two low feature vectors of $F_{head}^i$. We repeat the above steps for each image to generate the two sets of feature vectors as shown in Equation (1).

$$\left\{\begin{array}{c}{f}_{body}=\left\{f_{body}^1,\dots ,f_{body}^k,\dots ,f_{body}^N\right\}\\ f_{head}=\left\{f_{head}^1,\dots ,f_{head}^k,\dots ,f_{head}^N\right\}\end{array}\right.$$

(1)

3.3. Feature Extraction Network

In the feature extraction stage of the network, ViTReID uses a transformer encoder (ViT) as the feature extraction network to replace the traditional CNN or recurrent neural network (RNN) feature extraction network. The whole network structure is wholly composed of an attention mechanism. It consists of multiple self-attentions and MLPs. The structure of the feature extraction network is shown in Figure 2.

To make the network more suitable for person re-identification tasks, the transformer network structure is improved accordingly, and the lightweight transformer network structure is adopted to reduce the complexity of the network. The specific structure comprises a multi-head self-attention (MSA) mechanism and an MLP module. LayerNorm (LN) is applied before each module and residual connection is applied after each module. The MLP module contains two fully connected layers with the GELU nonlinear activation function. The implementation of the transformer encoder feature extraction network structure is shown in Equations (2)–(5).

$${\mathcal{Z}}_0=\left[X_{class};E_i\right]+E_{pos}$$

(2)

$${\mathcal{Z}}_{\ell }^{\prime }=MSA\left(LN\left({\mathcal{Z}}_{\ell -1}\right)\right)+{\mathcal{Z}}_{\ell -1}$$

(3)

$${\mathcal{Z}}_{\ell }=MLP\left(LN\left({\mathcal{Z}}_{\ell }^{\prime }\right)\right)+{\mathcal{Z}}_{\ell }^{\prime }$$

(4)

$$\mathcal{Y}=LN\left({\mathcal{Z}}_L^0\right)$$

(5)

The image is divided into N slices, and ${\mathcal{Z}}_0$ is the input sequence code. A learnable tag class $X_{class}$ is added before the input sequence. $E\in {ℝ}^{N\times D}\left(E_i=f\left(x_i\right),i=1,2\dots N\right)$ is obtained by mapping to the linear projection function f in the D dimension, $E_{pos}\in {ℝ}^{\left(N+1\right)\times D}$ is the location code, and $\ell =1\dots L$ is the layer of the transformer encoder. ${\mathcal{Z}}_{\ell }^{\prime }$ is the result of each slice after $\ell -1$ is the transformer encoder’s multi-head self-attention layer, ${\mathcal{Z}}_{\ell }$ is the result after $\ell$ transformer encoder’s MLP layer, and finally, ${\mathcal{Z}}_L^0$ is the result of the transformer encoder’s output. $\mathcal{Y}$ is the state of ${\mathcal{Z}}_L^0$ after LayerNorm operation.

The specific implementation of the network is as follows: First, embed 384 × 128 size picture $X$, and then encode positional encodings. In the even position, use the sine encoding. In the odd position, use the cosine encoding and output the position vector $X_{embedding}$ to retain location information. Next, perform linear mapping of $X_{embedding}$ to three matrix vectors Q(Query), K(Key), V(Value) passing through the self-attention module to obtain a weighted eigenvector $\mathcal{Z}$, namely Attention$\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)$, as shown in Equation (6).

$$\mathcal{Z}=\mathrm{Attention}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{softmax}\left(\frac{{\mathrm{QK}}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}\right)\mathrm{V}$$

(6)

The dimensions of the three matrix vectors are the same. ${\mathrm{d}}_{\mathrm{k}}$ represents the dimension of K and then calculates the softmax function to obtain the weight of this self-attention. To reduce the network’s burden, the attention mechanism’s head is set to 12, and the final output result is obtained as ${\mathcal{Z}}_{\ell }^{\prime }$ after circulating the multi-head attention layer. After that, it will be sent to the next module of the encoder, the MLP module. This full connection has two layers. The activation function of the first layer is GELU, and the second layer is a linear activation function, as shown in Equation (7).

$$\mathrm{MLP}=\mathrm{FFN}\left(\mathcal{Z}\right)=\mathrm{GELU}\left(\mathcal{Z}{\mathrm{W}}_1+{\mathrm{b}}_1\right){\mathrm{W}}_2+{\mathrm{b}}_2$$

(7)

where W is the weight and b is the offset. A feed-forward network, that is, the fully connected layer 1 → GELU → dropout → the fully connected layer 2 → dropout, is a standard MLP structure.

3.4. STC Combined Loss

To train better discriminant features, the combined cross-entropy loss function, triplet loss, and center loss are taken as the loss function of the whole network. The cross-entropy loss function is used for the final classification learning, and triplet loss and center loss are used for measurement learning.

Softmax + cross-entropy loss function is used as ID loss, as shown in Equation (8).

$${\mathcal{L}}_{\mathrm{s}}=\mathrm{CrossEntropy}=-\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\log \left(\mathrm{p}\left({\mathrm{y}}_{\mathrm{i}}{|\mathrm{x}}_{\mathrm{i}}\right)\right)$$

(8)

where n is the number of samples trained in each batch; $\mathrm{p}\left({\mathrm{y}}_{\mathrm{i}}{|\mathrm{x}}_{\mathrm{i}}\right)$ represents the input image ${\mathrm{x}}_{\mathrm{i}}$ and its category label ${\mathrm{y}}_{\mathrm{i}}$. Classified by softmax, ${\mathrm{x}}_{\mathrm{i}}$ is the prediction probability of class ${\mathrm{y}}_{\mathrm{i}}$. It is expressed by $\mathrm{p}\left({\mathrm{y}}_{\mathrm{i}}{|\mathrm{x}}_{\mathrm{i}}\right)$. The predicted probability distribution is the distribution learned by the trained model. To make the expected probability distribution as close as possible to the actual probability distribution, the cross-entropy loss function can be used to reduce the difference between the two probability distributions.

The triplet loss function is used to shorten the distance between the same IDs and expand the distance between different IDs. The clustering effect achieved by triplet loss is shown in Equation (9).

$$\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{a}},{\mathrm{x}}_{\mathrm{i}}^{\mathrm{p}}\right)+\mathsf{\alpha }\le \mathrm{d}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{a}},{\mathrm{x}}_{\mathrm{i}}^{\mathrm{n}}\right)$$

(9)

where d(·) represents the Euclidean distance between two vectors and α represents the margin between two vectors. To prevent $\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{a}},{\mathrm{x}}_{\mathrm{i}}^{\mathrm{p}}\right)$ = $\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{a}},{\mathrm{x}}_{\mathrm{i}}^{\mathrm{n}}\right)$ = 0, we minimize the triplet loss function to achieve this goal, as shown in Equation (10).

$$Triplet Loss={\left[\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{a}},{\mathrm{x}}_{\mathrm{i}}^{\mathrm{p}}\right)-\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{a}},{\mathrm{x}}_{\mathrm{i}}^{\mathrm{n}}\right)+\mathsf{\alpha }\right]}_{+}$$

(10)

The online training method is used. Different IDs of the same identity p are selected. Each ID contains k pictures, forming a mini-batch with $batch size=p\times k$. Thus, a complex or easy example can be selected to form loss in this mini-batch.

Therefore, the triplet loss function in this paper can be set as shown in Equation (11).

$${\mathcal{L}}_t=log\left[1+exp\left(\Vert f_a-f_p\Vert {}_2^2-\Vert f_a-f_n\Vert {}_2^2\right)\right]$$

(11)

where f is the global feature representation, <a, p, n> is a triplet, a is an anchor, p is a positive sample, n is a negative sample, p and a are samples of the same category, and n and a are samples of different categories. The effect is to pull in the distance of a and p and further the distance of a and n. Triplet loss is used to train samples with slight differences, and the similarity between samples is calculated by optimizing that the distance between the same samples a and p is smaller than the distance between different samples a and n. The same samples refer to people with the same identity (ID), and different samples are different people.

To improve the distinguishing ability of features, this paper focuses on supplementing the local features of the head and introduces the center loss function on the premise of maintaining the separability of data, further reducing the differences between classes, and further enhancing the face features while ensuring the global features. The center loss function is added to achieve this purpose, as shown in Equation (12).

$${\mathcal{L}}_c=\frac{\lambda }{2}{\displaystyle \sum }_{i=1}^m\Vert x_i-c_{y_i}\Vert {}_2^2$$

(12)

Two superparameters are introduced using the center loss function, α and $\lambda$. Here, $\lambda$ indicates the punishment intensity of center loss, and α controls the learning rate at center point $c_{y_i}$ of class. $c_{y_i}$ denotes the center of class $y_i$, and the center point $c_{y_i}$ of the intra-class will change with different features. Generally, the center point $c_{y_i}$ in the class will be updated in each mini-batch. For data with a complex distribution such as head data, it is more critical to maintain intra-class compactness while keeping the feature space inter-class separable. Because the intra-class variation of the same person is likely to be larger than the inter-class variation, only by keeping the intra-class compactness can we have a more robust decision result for those samples with a considerable intra-class variation. The center loss constrains the intra-class compactness condition.

Finally, ViTReID combines center loss, triplet loss, and cross-entropy loss function to form STC combined loss function, as shown in Equation (13).

$${\mathcal{L}}_{\mathrm{STC}}={\mathcal{L}}_{\mathrm{s}}+{\mathcal{L}}_{\mathrm{t}}+{\mathcal{L}}_{\mathrm{c}}$$

(13)

4. Experiments and Results

4.1. Datasets

The mainstream public datasets used for training and evaluation of person re-identification models include Market1501 [34], MSMT17 [18], and PersonX [35]. Market1501 and MSMT17 are commonly used datasets for person re-identification tasks. MSMT17 has the most image data and is more challenging. PersonX is a synthetic dataset, which is quite different from the domain of real data, and there will be much room for improvement in the algorithm. Our experiments are tested on public person re-identification datasets, including Market1501, MSMT17, and PersonX datasets.

Table 1. Experimental datasets.

Datasets	Train IDs	Train Images	Test IDs	Query Images	Cameras	Total Images
Market1501	751	12,936	750	3368	6	32,668
MSMT17	1041	32,621	3060	11,659	15	126,441
PersonX	410	9840	856	5136	6	45,792

gives detailed information corresponding to each dataset.

Six cameras were used in the Market1501 [34] dataset collection. The dataset contains 32,668 annotated bounding boxes with 1501 identities. Each annotated logo must appear in at least two cameras for a cross-camera search to be performed. The deformable part model (DPM) [36] is used as a person detector. Each recognition can have multiple images under each camera. During cross-camera search, each identity has multiple queries and multiple ground truths.

MSMT17 [18] is a new giant dataset closer to the actual scene, covering multiple scenes and periods. It adopts 15 camera networks on the campus, including 12 outdoor cameras and 3 indoor cameras. Four days with different weather conditions were selected in a month to collect the original surveillance video. Three hours of videos were collected every day, covering three time periods: morning, noon, and afternoon. Therefore, the total length of the original video is 180 h. Based on Faster RCNN [37] as a person detector, three manual taggers viewed the detected bounding box and marked person labels for two months. Finally, 126,441 bounding boxes of 4101 persons were obtained.

The PersonX [35] synthetic dataset is based on Unity and contains 1266 manual person models, including 547 women and 719 men, covering various types of clothing, walking postures, ages, body shapes, and skin colors. It can provide different backgrounds, character actions, clothing accessories, etc., for different modeling applications according to specific tasks. At the same time, this dataset has a viewpoint label. The dataset can reduce the dependence on manually collecting and labeling data. It can generate a large amount of data, which is cheaper, faster, and more accurate than manually marking data. It can generate data that are difficult to capture in the real world (such as visual content in traffic conflict areas). It can generate data (such as occlusion and edge conditions) that do not often occur in nature but are essential for training. It can reduce security and privacy issues.

4.2. Evaluation Protocol

Rank-k [38] and mAP are used as evaluating indicators in the experiment. The target re-identification task aims to find the target most similar to the query under different cameras. There are query sets and test sets (or gallery sets) in the dataset. When a vehicle image in the query set is input, the most similar vehicle image will be found in the gallery set. The processing method often uses the trained network to feed forward and extract the characteristics of the last FC layer. Its similarity with all images in the gallery set is calculated for the feature of the image in each query set.

Rank-k indicates the accuracy rate of the first k images sorted by the similarity that belongs to the same ID as the query image. We calculated the accuracy of rank-k (k = 1, 5, 10), i.e., that the images with the same label as the query image could be found in the first 1, 5, and 10 images of the sorted test set. The mAP is a commonly used evaluation index in target detection and multi-label image classification tasks. We take it as an essential standard to measure the quality of the ReID model.

4.3. Comparision with State-of-the-Art Methods

We compared the model with the unsupervised domain adaptive person re-identification method based on CNN. The experimental results are shown in

Table 2. Performance comparison between adaptive methods with unsupervised domain.

Methods	Market1501 → MSMT17				Methods	MSMT17 → Market1501
	mAP	R1	R5	R10		mAP	R1	R5	R10
PTGAN [18]	2.9	10.2	-	24.4	MAR [25]	40.0	67.7	81.9	-
ECN [39]	8.5	25.3	36.3	42.1	PAUL [41]	40.1	68.5	82.4	87.4
SSG [3]	13.2	31.6	-	49.6	CASCL [42]	35.5	65.4	80.6	86.2
MMCL [40]	15.1	40.8	51.8	56.7	D-MMD [27]	50.8	72.8	88.1	92.3
MMT [4]	22.9	49.2	63.1	68.8	DG-Net++ [26]	64.6	83.1	91.5	94.3
SPCL [5]	26.8	53.7	65.0	69.8	SPCL [5]	77.5	89.7	96.1	97.6
ViTReID	27.2	54.2	66.5	71.6	ViTReID	76.5	90.9	96.6	98.2

. It can be seen from the table that using the ViTReID model has increased the mAP by 14% and the rank-1 accuracy (R1) by 22.6% compared with the baseline method (SSG) on the datasets from Market1501 to MSMT17. On MSMT17 to Market1501, ViTReID is similar to SPCL in mAP, but R1 increases by 1.2% and the rank-5 accuracy (R5) and rank-10 accuracy (R10) increase by 0.5% and 0.6% respectively. This is because the ViTReID method proposed in this paper adopts the ViT-based feature extraction network and head branch training, which can help the model learn more global context information to establish a long-distance dependence on the target and make the model have better robustness and discriminant features.

At the same time, we also conducted experiments on the PersonX synthetic dataset with the Market1501 dataset and MSMT17 dataset in the source domain and target domain. The experimental results are shown in

Table 3. Experimental results of PersonX synthetic dataset to pedestrian real dataset.

Methods	PersonX → MSMT17				PersonX → Market1501
	mAP	R1	R5	R10	mAP	R1	R5	R10
MMT-dbscan [4]	17.7	39.1	52.6	58.5	71.0	86.5	94.8	97.0
ViTReID	20.8	46.8	59.1	64.6	72.5	87.3	95.8	97.9

. The results of MMT-dbscan [4] are obtained according to the author code implementation of MMT [4]. Due to the domain difference between synthetic data and real data, when PersonX is trained as the source domain, when migrating to independently and identically distributed datasets of MSMT17 and Market1501 target domains, the ViTReID method is more accurate than MMT-dbscan (as shown in Table 3), indicating that ViTReID is more generalizable. Compared with the MMT-dbscan method, ViTReID improves mAP by 3.1% and R1 by 7.7% on PersonX to MSMT17. On PersonX to Market1501, the mAP of the ViTReID method is increased by 1.5%, and R1 is also increased by 0.8%.

4.4. Ablation Study

To fully explore the impact of model parameters on our algorithm in different situations, several scenarios are set up to analyze the effect of varying parameter weights on performance. The performance of ViT after mapping and learning the target project in the same backbone network is verified. Specifically, ViTReID shows the performance change trend of the target item ViT/B-384 with the different parameters of the loss item (see

Table 4. Comparison of method performance with different parameter items (fine-tuned adds RandomGrayscale data enhancement method).

Methods	Market1501 → MSMT17				MSMT17 → Market1501
	mAP	R1	R5	R10	mAP	R1	R5	R10
Baseline (SSG) [3]	13.2	31.6	-	49.6	-	-	-	-
Baseline + VIT/B-384	23.8	51.6	63.5	68.8	76.2	89.3	96.2	97.7
Baseline + Head	25.4	49.0	61.4	67.9	74.8	89.6	96.0	97.5
Baseline + Head + Center Loss	26.2	49.8	62.8	68.1	75.2	90.2	96.3	97.9
ViTReID (full-finetune)	27.2	54.2	66.5	71.6	76.5	90.9	96.6	98.2

). The specific change trend of mAP with the increase in epochs is shown in Figure 3 and Figure 4.

Based on the experimental results, we first changed the feature extraction network of the baseline to cope with the variation of pedestrian features under different cameras and the partial feature loss due to target occlusion. Because ViTReID uses ViT with a more robust learning ability of long dependence as the backbone network, it is stronger than the model based on CNN. Therefore, it improves the mAP by nearly 10% on the datasets from Market1501 to MSMT17. It shows that the transformer can capture global features better than the CNN model. At the same time, we changed the baseline’s local branching structure to solve the differentiation degradation problem due to color approximation and feature approximation of different target clothes. The combined head features include the most representative discriminant features in pedestrian information, so they can also improve the mAP by nearly 2%. This shows that the head branch plays a significant role. Meanwhile, after adding the center loss function to the head branch, the mAP is improved by nearly 1% again, indicating that the center loss dramatically contributes to the more complex head feature extraction. On MSMT17 to Market1501, after using the ViT feature extraction network and head branch training, as well as fine-tuning steps, such as data enhancements, the results are also improved greatly.

As can be seen from Figure 3 and Figure 4, both from the large dataset (MSMT17) to the small dataset (Market1501) and from a small dataset to an extensive dataset, mAP increases significantly with the increase in the number of training rounds. However, the promotion range from the small dataset to the large dataset is more extensive; from the large dataset to the small dataset, the promotion range is smaller. It can be seen that using large datasets as the source domain or directly using large datasets for unsupervised training has good enforceability for the transformer model, and the results will be much better than those of the CNN model.

5. Conclusions

This paper proposes an unsupervised domain adaptive person re-identification method (ViTReID) based on the transformer. The grouping idea of the feature extraction network ViT and a head branch is introduced. The long-distance dependence can be constructed through the global context information obtained by the feature extraction network. The discriminative nature of local (head) features is improved by using the idea of local and global feature grouping. The above operations enhance the accuracy of the ViTReID network but also increase the complexity of the network, leading to offline training and inference only. Experiments show that ViTReID can achieve good results on several popular datasets. According to the advantages of the transformer’s parallel computing and its self-attention mechanism, it can produce a more interpretable model. At the same time, with the rapid development of image classification, target detection, and semantic segmentation in recent years, the transformer will have great potential in ReID tasks. It will become a mainstream research direction in the future. For example, person re-identification is applied in public safety, new retail, and intelligent transportation. Our future works are to improve the attention mechanism of the transformer network by MLP-like methods, using a better loss function and the Adam optimization algorithm, and further subdivide the head features to obtain a more efficient network.