Cross-Attention Guided Local Feature Enhanced Multi-Branch Network for Person Re-Identification

Abstract

The purpose of person re-identification is to identify and retrieve the same individual across different scenes, angles, and times. As an instance-level recognition problem, human re-identification relies on discriminative features, but is not determined by a single salient feature. This requires us to extract identity-specific features from multiple perspectives. In this paper, we propose a cross-attention guided local feature enhanced multi-branch network, which includes a global branch, a partial branch, and an attention channel branch. The network is guided by attention and jointly extracts local and global features to capture discriminative identity features from multiple aspects. To enable each branch to effectively mine identity information, we design modules dedicated to the function of each branch, making the branch network capture discriminative features. Finally, we conducted extensive testing on the Market1501 and CUHK03 datasets and achieved outstanding results.

1. Introduction

In recent years, with the in-depth development of deep learning [1], neural networks have shown astonishing progress in different directions. Similarly, this has promoted the rapid development of person re-identification. With the development of science and technology and the popularity of street cameras, person re-identification has been the focus of more attention and has been applied to safety precautions [2]. Although person re-identification has achieved impressive results in recent years, the task of matching individuals with different identities remains a significant challenge due to the complexity of the real world. There are still many hurdles to overcome in the field of person re-identification, including issues related to posture, privacy, cross-scene variations, and occlusions. These factors add layers of complexity to the task.

At the same time, since human activity is uncertain, this requires the ability to correctly match pedestrian images taken from different cameras. Some approaches propose monitoring [3], activity analysis [4], and people-tracking. Therefore, we need more discriminative image feature representation. This requires us to use various factors to improve discrimination between features. To address this problem, some methods [5,6] have focused on extracting global features through feature representation learning; Lin et al. [7] reweighted attributes by correcting predictions based on learned attribute dependencies and correlations. However, these methods ignore fine-grained features to a certain extent. In order to obtain more detailed feature representations, further research is required to add human body information to the network, that is, to predict human posture or human body parts. Some methods [8,9] employ semantic segmentation techniques for images focusing on human body analysis, utilizing human body analysis as an auxiliary extraction method. There are also methods [10,11] that use unsupervised methods to generate false tags from unlabeled samples, so that the network training does not rely on manual tags. In addition, there is another approach to locating different body parts on the human body, which involves estimating key points or utilizing human pose estimation techniques [12]. However, this auxiliary model is pre-trained on different domains, which means that the learning process have significant domain gaps and data bias problems, and that such labeling will incur additional computational costs. In addition, the scale of the training dataset for person ReID tasks is very limited, which leads to the fact that the network can only fit rough and obvious features in the process of global feature learning [13], and some non-significant key information is easily ignored.

To address the above issue and enable the network to extract more detailed feature representations, researchers have developed a number of local fine-grained feature representation methods [14]. Sun et al. [15] proposed a way to divide the feature space into several fringes and correct the outlier features in each region, so as to achieve a kind of local feature capture, and finally connect the features as the final representation. Zheng et al. [16] proposed a coarse-to-fine pyramid network to extract identification information from different spatial scales in a progressive manner. The contribution of global features in the task of person ReID is very large, but, if fine-grained features are ignored, the network will fall into local optimality. Therefore, we speculate that integrating a partial branch into the network will encourage the network to pay attention to global features while taking into account fine-grained features, and enhance the network’s identity discrimination. Previous works [13,17,18] have also demonstrated the benefits of multiple branches. Ref. [17] proposed the use of global and local branches to extract salient features and common features of images, respectively. However, this network is only a simple association of global branches and local branches, and the local branches are just the result of the hard segmentation of features.

Driven by the above factors, we propose a novel approach called the cross-attention guided local feature enhanced multi-branch network, which uses a multi-branch network guided by channel attention. First, we establish three branches in the network, namely the global branch, the local branch, and the attention branch. Similar to most methods, we employ the global branch to capture the overall characteristics of pedestrians, such as their clothing and body shape. However, what sets our model apart is the inclusion of a feature expansion layer within the global branch. This layer aids in more accurately extracting the overall features of individuals. However, we know that, in the real world, a person’s distinctive features cannot be determinant. In other words, salient features, such as clothing, are only a single attribute of a person, not the whole. Typically, we cannot determine a person’s identity based on their clothing, because two people with the same clothes cannot be identified as the same person. Therefore, if only global features are used, the network will inevitably focus on the clothes that have a greater attraction for identifying the identity. In other words, this will make the network more attentive to the clothes rather than the person. Obviously, this is not the result we want. To address this problem, we have added a local branch to the network. The local branch is more focused on the capture of fine-grained features. The local branch is utilized to focus on local information within the image, including fine-grained information such as the pose of the figure and the texture of the clothing. To assist in extracting crucial fine-grained features from images of people, we employ attention-guided channel features to explore discriminative information within the image.

2. Related Work

2.1. Holistic Feature Representation Learning

Person re-identification (ReID) is a technique [19] that aims to match images of people captured from non-overlapping camera views. This technology has gained popularity in the field of intelligent video analysis, is a primary task in many surveillance and security applications, and is receiving increasing attention in the field of computer vision. The goal of this technology is to identify target pedestrians from existing video sequences from possible sources and non-overlapping camera views. Existing ReID methods typically use metric learning methods [6,7] and deep learning methods [20,21,22]. Before deep learning dominated the ReID task, hand-crafted features were developed. Some methods [23,24] improve color and texture feature extraction by dividing the pedestrian image into stripes or triangle-shaped areas [25,26]. Later, deep learning methods were widely used. Liu et al. [27] focused on the comprehensive extraction of multi-scale features, thereby enriching the semantic information of the features. Li et al. [20] designed a multi-scale network to extract feature context information using multi-scale features. However, these ignore local fine-grained feature extraction [28].

2.2. Local Feature Learning in ReID

With the development of deep learning, the ability of feature extraction can better serve the extraction of fine features, which further promotes the application of partial features in re-identification. Zhao et al. [21] used a deep network to decompose the human body into several spatial regions, extract features, and then connect the features. Sun et al. [15] proposed dividing the overall feature into several stripes and learning the local features of each stripe region. Wang et al. [13] proposed a multi-branch network, which divides the network into a multi-branch network; some branches were used to learn local feature representation, thus enhancing the discrimination of the network. Park et al. [29] introduced a new relational module that considered the relationship between a single site and others, This method fully excavated the features between parts. The local feature could effectively extract partial-level feature information from pedestrians, but it was easily affected by changes in human posture and the occlusion of pedestrians or non-pedestrians. Therefore, we also used a combination of local and global, namely fine-grained local features and coarse-grained global features, which were combined to enhance the feature representation.

2.3. Attention in ReID

Using attention has achieved very good results in other fields, such as natural language processing [30], object detection [31], image segmentation, etc. Similarly, attention is superior when carrying out the ReID task, especially in capturing more discriminating features. The common way to use attention in ReID is to aggregate separate attention blocks into a deep convolutional ReID model. Li et al. [32] proposed a new harmonious attention CNN (HA-CNN) model, this model jointly learns pixel-level soft attention and region-level hard attention to solve the problems caused by poor boundary frame positioning and pixel-level noise information. Yan et al. [33] designed an internal attention network to search for information and discriminative regions in images of whole bodies or body parts, and to decompose attention into spatial attention and channel attention for enhancing the network’s focus on pedestrian features. However, attention is, to some extent, very rough and cannot model complex relationships between parts, which leads to the loss of fine-grained information. Chen et al. [34] proposed a method that uses high-order attention modules and a linear polynomial predictor to model the high-order semantics within attention. Zhang et al. [35] proposed a competent relationship-aware global attention module for efficient learning to recognize features. This module explores global scope relationships to mine structural information, in order to improve network authentication ability. Chen et al. [22] proposed using global attention and local attention to learn the features of attention perception, use global attention to decouple the background, and use local attention to decouple human body parts, which enhanced the robustness of the network. In this paper, we use channel-level attention to constrain feature channels to assist in extracting finer grained key human features.

3. Methods

Our proposed approach is an attention-directed global and local federated multi-branch network, as shown in Figure 1. Similar to [18], we use OSNet [36] as the backbone network for fine extraction of human features. It differs from ResNet50 [37], in that it has a smaller parameter count and is smaller than ResNet50. We divide OSNet into two parts; the first part is from the first block of OSNet to conv3_0, which we use as the backbone of the whole network. The other part is from conv3_1 to the last block of the network, which we use as a Block, and the subsequent three branch networks will have independent weights of this Block. Our overall frame diagram is shown in Figure 1. In the following subsections, we will introduce our proposed global branch, partial branch, and attention channel branch.

For our global branch, we have retained the global branch from a previous work [13], but have modified the maximum pooling layer to better enable the network to extract global features of the person. In other words, we have enhanced the discriminate power of the rough features extracted by the global branch. Therefore, we use a feature amplification module to replace the maximum pooling layer. After the pedestrian image is input into the backbone, when it reaches the layer 3_0 of the network, we obtain a feature vector $F_1$ with a dimension of 24 × 8 × 512. After obtaining the feature vector $F_1$, we feed it into the remaining layers of OSNet without shared weights, which enables us to obtain a 512-dimensional vector $F_1$. After that, we feed this vector into the proposed feature amplification module. Then the feature amplification module enhances the features, and we obtain a more significant 512-dimensional feature vector. It is worth noting that the feature enhancement layer only changes the internal attributes of the features, without changing the feature dimensions. Therefore, our $F_1$ is still 512-dimensional. In the next section, we will introduce the feature amplification module. On the other hand, similarly to [18], we added a dropout block in the global branch, which takes the initial tensor as input. The dropout module erases the horizontal regions with the highest activation, forcing the network to pay attention to the regions with lower discriminate power. This naturally increases the robustness of the resulting representation.

For the partial branch, like most methods, we divide the feature into N parts, but we find that hard segmentation of the feature, as conducted in previous works, does not provide much benefit for the network. This is because, for the semantic parts of the human body, if hard segmentation is directly applied, it will cause a rupture between parts, making the network ignore the connections between parts, and also damage the semantic features of the human body to some extent. Therefore, to avoid breaking the relationship between parts via the hard segmentation of the stripes, we propose a part fusion module (PFM). We feed the partial branch feature into the PFM as input, and obtain K fused partial features that have absorbed other feature information. In this way, each partial feature is no longer independent, but related to other parts. In Section 3.2, we will explore PFM in detail.

In the attention branch, the feature vector is pooled and decomposed into two 256-dimensional feature vectors, which we feed to the channel attention [38] to obtain more information worth attention. We then use a 1 × 1 convolution to expand the feature representation and obtain two feature vectors, namely ${\widehat{\mathrm{a}}}_1$ and ${\widehat{\mathrm{a}}}_2$.

Finally, we receive all the feature vectors obtained by the three branches, and we monitor these features with triple loss [39]. At the same time, we use BNNeck [40] to process these features. BNNeck is composed of batch normalized and fully connected layers. These feature representations processed by BNneck blocks are applied to cross-entropy loss monitoring.

3.1. Feature Amplification Module

Unlike other global feature networks, our global branch specifically focuses on highly discriminative features, such as clothing characteristics and body attributes, which are more apparent. Therefore, we constructed a feature amplification module (FAM). As illustrated in Figure 2, the input to the feature amplification module is the global feature vector $F_1$, which is subsequently fed into both the max pooling layer and the average pooling layer to obtain $g_{max}$ and $g_{avg}$, respectively.

$$g_{max}=maxpool\left(F_1\right),g_{avg}=avgpool\left(F_2\right)$$

(1)

The max pooling layer is utilized to extract the most salient parts of the features while filtering out background noise. On the other hand, performing average pooling on the features aims to preserve important information about the individuals, aiding us in obtaining smoother feature maps. Afterward, we concatenate the two features and feed them into a convolutional block to integrate the information from both sources, resulting in a more robust feature representation. To stay aligned with the objective of the global branch, which is to capture more salient features, we perform element-wise addition between the concatenated feature and $g_{max}$. Finally, we compress the resulting feature vector G, thus obtaining the optimized feature vector after going through the feature amplification module.

$$G=R_{down}\left(concat(g_{max},\mathrm{Conv}(g_{avg}+g_{max}))\right)$$

(2)

where Conv consists of 1 × 1 convolutional blocks followed by normalization and ReLU activation function. Concatenation represents the concatenation operation, and $R_{down}$ refers to the convolutional layer that reduces the number of channels.

3.2. Cross Fusion Module

In most local methods, we often observe that they simply divide the overall features into several parts, but these parts are independent of each other and lack interconnection. In our approach, we employ a cross-fusion technique to blend the information of each part’s feature representation with that of other parts. This allows us to establish connections between different parts, enabling a more comprehensive understanding of the overall feature representation. We utilize average pooling to divide the feature vector $F_2$ into three parts, namely $p_1$, $p_2$, and $p_3$. As shown in Figure 3, we take three local feature vectors as input. To combine the characteristic information of $p_1$ and $p_2$, we merge $p_1$ and $p_2$ to obtain a new feature vector $p_0$, as formulated below:

$$p_0={\displaystyle \frac{p_1+p_2}{2}},$$

(3)

Then, we perform cross-fusion on these vectors to achieve the effect of mutual connection. We concatenate the newly generated $p_0$ with $p_3$, and concatenate the newly generated $p_1$ with $p_2$, so we obtain three new vectors, namely ${\widehat{p}}_1$, ${\widehat{p}}_2$, and ${\widehat{p}}_3$. It is worth noting that our concatenation is achieved by using a cross-attention mechanism. The cross-attention mechanism can be represented as follows:

$$f_{cross}(X_i,X_j)=Softmax\left({\displaystyle \frac{Q_i{K}_j^T}{\sqrt{d_k}}}\right)V,$$

(4)

where $d_k$ is the scaling factor, $Q_i$ is the linear mapping for $X_i$, K is the linear mapping for $X_j$, and we set $X_i$ as V. In the proposed method, the cross-attention mechanism is applied to $p_1$ and $p_2$, $p_1$ is projected as Q, and $p_2$ is projected as K and V; thus, ${\widehat{p}}_1$ is obtained. Then, we switch $p_1$ and $p_2$ to perform the cross-attention mechanism and obtain ${\widehat{p}}_2$. Similarly, the cross-attention mechanism is performed on $p_3$ and $p_0$, with $p_3$ projected as Q and $p_0$ projected as K and V; thus, ${\widehat{p}}_3$ is obtained.

3.3. Training Loss

Given a probe image $I\in R^{H\times W\times C}$, where H, W, and C denote the height, width, and the number of channels, respectively. After obtaining the network output features, we use two kinds of losses to supervise the network, which are multi-similarity loss [41] (MS Loss) and ID Loss. Among them, we use cross entropy as ID Loss. As shown in Figure 1, we use the classification embedding calculated by each branch feature after BNNeck to calculate ID Loss. The feature before BNNeck is used to calculate MS Loss. Functionally, ID Loss can be presented as follows:

$$L_{ID}=-y_{i}log\left({\displaystyle \frac{exp\left(C_i{W}_i\right)}{{\displaystyle \sum _{j=1}^N}exp\left(C_j{W}_j\right)}}\right),$$

(5)

where $W_i$ is a linear projection matrix, $y_i$ represents the corresponding ground truth label, and N represents the total number of identities. Regarding MS loss, by calculating similarity pairs and mining information pairs, weighted selection is further applied to the pairs using similarity, thereby enabling pair mining. Functionally, MS Loss can be presented as follows:

$$\begin{array}{c}\begin{array}{c}{L}_{MS}={\displaystyle \frac{1}{m}}\sum _{i=1}^{m}log\left(1+\sum _k^{P}exp(S_{ik}-\mu )\right)\hfill \\ \hfill +log\left(1+\sum _k^{N}exp(S_{ik}-\mu )\right),\hfill \end{array}\end{array}$$

(6)

where m is the total number of identities, $\mu$ is a hyper parameter, P and N, respectively, refer to the number of positive samples and negative samples, $S_{ik}$ represents similarity, and the calculation method is as follows:

$$S_{ij}=<f\left(x_i\right|\theta ),f\left(x_j\right|\theta )>,$$

(7)

where $<·,·>$ represents the dot product. Therefore, during the training process, we define the overall objective function for optimization as follows:

$$L=\lambda L_{ID}+(1-\lambda )L_{MS},$$

(8)

where $\lambda$ is a hyperparameter. This parameter is used to balance the two losses, and, in this article, we set it to 0.5. In addition, we use the cosine annealing strategy as an optimization strategy for updating the learning rate. The cosine annealing strategy is commonly used in ReID networks, and it can better motivate the network compared to traditional step-wise learning rate scheduling, thereby improving the performance of the network. During the first 10 epochs, the learning rate increases from $6\times {10}^{-5}$ to $6\times {10}^{-4}$, and then in the remaining period, cosine decay is applied to reach a final learning rate of $6\times {10}^{-7}$.

4. Results

4.1. Datasets and Evaluation Metrics

During model training, the use of biased data may lead to algorithmic discrimination [42,43]. In this study, we have also considered fairness in both the dataset and evaluation metrics by selecting the currently widely recognized mainstream pedestrian re-identification datasets for training. To assess the reliability of the model, we adopt the cumulative matching characteristics (CMC) at Rank 1 and the mean average precision (mAP) to evaluate performance.

We evaluated our model using three datasets commonly used in pedestrian re-identification tasks, namely Market-1501 [44], CUHK03 [45] and DukeMTMC-reID [46]. Market-1501 is an image dataset for pedestrian re-identification, collected on the campus of Tsinghua University. The dataset contains 32,217 images of 1501 pedestrians captured by six cameras (including five high-definition cameras and one low-definition camera). This dataset has become one of the most commonly used datasets in the field of pedestrian re-identification. CUHK03 is a large-scale pedestrian re-identification dataset collected by the Chinese University of Hong Kong (CUHK). It contains images of 1467 different individuals captured by five pairs of cameras. The images were collected on the campus of CUHK, covering a variety of scenes and lighting conditions. We used the new training/testing protocol proposed in 2017, where the dataset is divided into 767 pedestrians in the training set and 700 pedestrians in the testing set. DukeMTMC-reID is an image dataset collected on the Duke University campus in 2017. It is composed of more than 36,000 images, covering 702 individuals with different identities, which provides abundant data for algorithms. Among them, the training set contains 16,522 pictures of 702 pedestrians, and the test set contains 70 pedestrians and 408 interfering pedestrians, totaling 17,661 pictures. The query set randomly selects one picture of each of the 702 pedestrians in the test set from each camera, with a total of 2228 pictures.

4.2. Implementation Details

We uniformly adjust the resolution of the input images to 384 × 128 pixels, and also adjust the image size to 105% of the original width and height. We perform data augmentation through random cropping and random horizontal flipping with a probability of 0.5. The model is trained on 120 epochs for Market-1501 and 180 epochs for CUHK03, with a batch size of 48. Each batch consists of eight samples, and each sample has six identities. The parameters are optimized using the Adam optimizer with a weight decay of $5\times {10}^{-4}$, ${\beta }_1=0.9$ and ${\beta }_2=0.999$. We use the PyTorch library in Python to implement the code for the network proposed in the paper, and utilize the ImageNet dataset to pre-train the backbone model. We use $\lambda$ to balance the overall loss of the entire network. In this paper, $\lambda$ is set to 0.5.

4.3. Comparison with State-of-the-Art Methods

In

Table 1. Comparison of the experimental results. The table shows the results of multiple methods on the different datasets, where results in bold underlined fonts are the best results, and results in bold-only fonts are the second best results.

Methods	Market1501		CHUK03-D		DukeMTMC-reID
	Rank 1	mAP	Rank 1	mAP	Rank 1	mAP
AlignedReID [50]	91.8	79.1	61.5	59.6	82.1	69.7
PCB [15]	92.4	77.3	61.3	54.2	81.9	65.3
HA-CNN [32]	91.2	75.7	41.7	38.6	80.5	63.8
Auto-ReID [51]	94.5	85.1	73.3	69.3	-	-
MHN [34]	95.1	85.0	71.7	65.4	89.1	77.2
OSNet [36]	94.8	84.9	72.3	67.8	88.6	73.5
DAAF-BoT [22]	95.1	87.9	64.9	63.1	87.9	77.9
GCP [29]	95.2	88.9	74.4	69.6	89.7	78.6
MGN [13]	95.7	86.9	66.8	66.0	88.7	78.4
NFormer [49]	94.7	91.1	79.0	76.4	90.6	85.7
TransReID [47]	95.2	88.9	75.1	72.9	90.7	82.0
AAformer [48]	95.4	88.0	78.1	77.2	90.1	80.9
ours	96.0	90.2	82.6	79.7	90.8	81.9

, we conducted comparative experiments by comparing our network with other methods. Our model achieved very impressive results on Market-1501 and CUHK03, both in terms of rank-1 accuracy and mAP. Regarding the results for Market-1501, as shown in Table 1, our method achieves the best mAP and a comparable Rank-1 accuracy among all the state-of-the-art competitors. Among them, TransReID [47] and AAformer [48] are ViT network-based methods. Our method achieved a Rank 1 accuracy of 1.3% higher than that of NFormer [49]. This greatly reflects the superiority of our method. Finally, we provide the visualization of the retrieval results, which is shown in Figure 4. The visualization results represent the probability that the top K graphs in the search results are correct, as well as the confidence level of the model’s performance.

4.4. Ablation Studies

Finally, in order to demonstrate the validity of each branch and its contribution to the network, we evaluated the benefits of introducing each branch and tested them, as shown in

Table 2. The table shows the ablation experiments we performed for the individual branches. In the table, "Global" represents the baseline network.

Branch	Market-1501		CUHK03-D		DukeMTMC-reID
	Rank-1	mAP	Rank-1	mAP	Rank-1	mAP
Global (G)	94.7	87.6	78.0	74.7	88.4	74.8
Part (P)	95.1	87.3	73.8	70.6	88.7	77.9
Attention (A)	92.8	84.5	71.5	67.2	85.0	71.7
G and P	95.8	90.1	81.6	79.1	90.6	80.8
G and A	94.6	88.3	78.5	75.2	88.7	78.0
P and A	95.0	88.6	75.3	73.4	89.5	79.3
All	96.0	90.2	82.6	79.7	90.8	81.9

.

In three different datasets, we conducted ablation experiments on the various branches of our network, and the results are shown in Table 2. It is worth noting that the baseline network is also the network that removes the attention branch and the local branch, namely the global network. It is evident that using all branches together leads to a noticeable improvement in the network’s performance. When training with only a single branch on different networks, it is apparent that each branch has different effects on different datasets. For example, on the Market-1501 dataset, only using the partial branch yields the best results, whereas, on CUHK03-D, only using the partial branch is not as effective as only using the global branch. It can be anticipated that jointly using these two branches might yield good results. As expected, the results in the lower half of the table show that the combined use of the global and partial branches leads to improvements across all three datasets, surpassing the performance of using a single branch.

In addition, we conducted experiments on the $\lambda$ parameter within the loss function. Varying values of $\lambda$ exhibited different effects on the training of the network, as illustrated in Figure 5. In order to compare the impact of different $\lambda$ values on network training, we conducted experiments on the network using three different values: 0.3, 0.5, and 0.7. From the experimental results, it can be seen that, on the CUHK03 dataset, the impact of $\lambda$ on performance is the most significant, and the best results are achieved when $\lambda$ is set to 0.3. However, on the Market1501 dataset, the opposite is true, and the best results are obtained when $\lambda$ is set to 0.7. Therefore, in this experiment, we choose $\lambda$ = 0.5 as our training parameter.

5. Discussion

In this paper, we propose an attention-driven holistic local-joint multi-branch network architecture. The different branches in our network are designed to capture specific types of features that we desire, resulting in a complementary approach. We conducted extensive experiments and obtained superior results to support the superiority of our network. Therefore, we believe that considering character features from multiple perspectives and designing multiple branches to extract features of different scales are effective in person re-identification.

However, it should be noted that the network method we propose has certain limitations. On the one hand, as it is a multi-branch network, it has relatively high requirements for computing resources and may not be able to operate efficiently in environments with low hardware configurations, which limits its wide application range to some extent. On the other hand, the data for model training is targeted at conventional person re-identification, and it is also rather dependent on the scale and quality of the training data. If there are biases or deficiencies in the data, it may affect the final recognition results in some person re-identification tasks under special scenarios.

Based on the experiments and analyses in this paper, there is still much work to be done. Firstly, we will continuously optimize the network architecture. We will attempt to reduce the dependence on computing resources, simplify the training process, and improve the training efficiency and generalization ability of the model by means of improving the algorithm structure and introducing new technical means. Secondly, in order to further enhance the generalization ability of the model, we will collect more image data of pedestrians in different scenarios and of different types to enrich the diversity of data, enabling the model to learn more comprehensive features and thus strengthening the pedestrian re-identification ability under various complex circumstances.