Dynamic Screening Strategy Based on Feature Graphs for UAV Object and Group Re-Identification

Abstract

In contemporary times, owing to the swift advancement of Unmanned Aerial Vehicles (UAVs), there is enormous potential for the use of UAVs to ensure public safety. Most research on capturing images by UAVs mainly focuses on object detection and tracking tasks, but few studies have focused on the UAV object re-identification task. In addition, in the real-world scenarios, objects frequently get together in groups. Therefore, re-identifying UAV objects and groups poses a significant challenge. In this paper, a novel dynamic screening strategy based on feature graphs framework is proposed for UAV object and group re-identification. Specifically, the graph-based feature matching module presented aims to enhance the transmission of group contextual information by using adjacent feature nodes. Additionally, a dynamic screening strategy designed attempts to prune the feature nodes that are not identified as the same group to reduce the impact of noise (other group members but not belonging to this group). Extensive experiments have been conducted on the Road Group, DukeMTMC Group and CUHK-SYSU-Group datasets to validate our framework, revealing superior performance compared to most methods. The Rank-1 on CUHK-SYSU-Group, Road Group and DukeMTMC Group datasets reaches 71.8%, 86.4% and 57.8%, respectively. Meanwhile, our method performance is explored on the UAV datasets of PRAI-1581 and Aerial Image, the infrared datasets of SYSU-MM01 and CM-Group and the NIR dataset of RBG-NIR Scene dataset; the unexpected findings demonstrate the robustness and wide applicability of our method.

1. Introduction

Object re-identification (re-ID) is a technique used to match objects observed across different cameras without overlaps, which involves some challenges, including issues related to low resolution, occlusion and changing poses and illuminations, etc. In the last decade, many single-object re-ID approaches have been proposed and achieved great success in both public safety and video surveillance [1,2,3,4,5,6,7]. However, there are few works focused on object re-identification in the context of Unmanned Aerial Vehicles (UAVs). The reason for this can be attributed to the time-consuming nature of annotating datasets for object re-identification in UAVs, which necessitates annotating both the positions of bounding boxes and assigning ID numbers to each object. In addition, objects frequently get together in groups in real-world scenarios.

Group re-ID focuses on aligning groups of objects observed from various camera viewpoints. Most existing methods utilize contextual information from neighbors to assist re-identify each group member within groups [8,9,10,11]. Utilizing group context information has demonstrated its effectiveness in mitigating ambiguity in single-object re-ID [10,12,13]. Therefore, the process of group re-ID can effectively be applied to person re-ID in groups. In comparison to single-object re-ID, group re-ID presents greater challenges because of the changes in group members and layout. The person as an object and group re-ID are shown in Figure 1a,b. The current group re-ID methods can be categorized into two types: feature aggregation-based [11,14] and crowd matching-based methods [15,16]. The first category of method aggregates the contextual information from neighbors to re-identifying each group members within groups. For example, Yan et al. [14] devised a multilevel attentional mechanism to capture intra- and inter-group contexts and suggested a novel uniform framework for group re-ID. However, these approaches neglect the significance of background features, which can be used to assist message transmission between group nodes. The second category of method usually adopts a single- or multi-object matching strategy and assigns important weights to objects to capture the dynamic changes within the group. For example, Lisanti et al. [16] presented multi-granularity representation for group images, which they combined with a dynamic weighting approach to obtain improved people matching. However, these methods cannot solve the problem of group members changes well.

To address the aforementioned challenges, a novel dynamic screening strategy based on a feature graphs framework is proposed for UAV object and group re-ID. In particular, the Pre-Processing module (PREP) designed aims to remove objects that do not belong to the current group, as shown in Figure 1c. Specifically, the group pairs are as input and the CNN model is employed to extract person and background features. In the PREP module, the person features are regarded as nodes to construct the context graphs using the k-nearest neighbor algorithm. First, if the graph is connected, the graph nodes are determined whether in the same group. When these nodes are not in the same group, the graph nodes and their connected edges are pruned. Otherwise, the original graph structure is retained without any processing. If the graph is not connected, the model selects a sub-graph and repeat the above steps until the nodes in the graph belong to the same group. Finally, the reconstructed graphs do not contain the persons that do not belong to the current group.

Here, the Graph-based Feature Matching (GFM) module presented attempts to enhance the transmission of group contextual information by using adjacent feature nodes. In this module, matched background features can be obtained using the method of feature matching. The background features are regarded as newly added nodes to the reconstructed graph. Inspired by the method of Multi-Attention Context Graph [14], the Multi-Objects Context Graph (MOCG) module designed tries to capture context messages of objects by adopting multi-level attention mechanism within and between graphs. It is noteworthy that accuracy of group re-ID has seen enhancement to some extent through continuous messages transmission between person nodes and new nodes, as well as continuous updates of nodes in the context graph.

The main contributions of this paper are as follows:

• The Graph-based Feature Matching module designed attempts to enhance the transmission of group contextual information by adding matched features for the first time.
• The Pre-Processing module presented aims to solve the challenge of group member changes in the group re-ID, which can remove group members that do not belong to the current group through pruning operations.
• The Multi-Objects Context Graph module proposed tries to employ multi-level attention mechanisms within and between graphs to capture contextual information.
• Our proposed framework is examined on several datasets, and the experimental results demonstrate that the model surpasses the most advanced approaches.

2. Related Work

2.1. Single Object Re-Identification

Recently, a lot of attention has been attracted by single object re-identification from the academic and industrial perspectives in computer vision [17,18]. It faces many challenges, such as the variety of views [19,20], poor image resolutions [3,21], illumination variations [22], unconstrained postures [23,24,25], occlusions [26], heterogeneous modes [27,28], complex camera environments [29], backdrop clutter and unreliable boundary box generation. Early studies primarily focused on manual body structure feature construction [30,31,32] or distance metric learning [33,34,35,36,37]. Single object re-identification has shown encouraging results on commonly utilized benchmarks [38,39,40]. However, few works are focused on Unmanned Aerial Vehicles (UAVs) object re-ID. In this work, our model are applied to the task of UAV person re-ID for the first time, demonstrating the strong generalization ability of our proposed model.

2.2. Group Re-Identification

In comparison to single person re-identification, group re-identification aims to evaluate the group images’ similarities and few works have been proposed [41,42,43]. For example, Cai et al. [8] earliest presented the covariance descriptor to match groups. To match the two patch sets, Zhu et al. [9] constructed group re-ID as a patch-matching task, and the patch was placed in the “Boosted Salience Channels”. To address the spatial displacement arrangement of individuals within the group, Zheng et al. [10] developed a visual descriptor and sparse feature coding approach. Then, Lisanti et al. [16] presented multi-granularity representation for group images, which they combined with a dynamic weighting approach to obtain improved people matching. With the widespread adoption of graph neural networks in recent years, Zhu et al. [11] developed a network to group analysis where context is represented by a graph. The model uses a novel technique to aggregate information from neighboring nodes using features derived from SKNNG, such as node entry degree and spatial interactions between nearby nodes (i.e., relative distance and direction). Yan et al. [14] devised a novel uniformed framework to capture group-level contexts with multilevel attentional mechanism. Nevertheless, the majority of the approaches mentioned cannot solve the changes of group members well. In this work, our proposed PREP module can effectively remove the object (person) who does not belong to this group.

2.3. Graph Neural Network

Graph Neural Networks (GNNs) aim to use graphs to represent information and extract features to replace conventional handwritten features [44,45,46]. GNNs have been employed in a multitude of visual tasks, including object detection, semantic segmentation, and solving visual question-answering problems. The fundamental concept involves creating node embeddings using neural networks through the aggregation of local information. In recent times, there has been a growing fascination with expanding convolution operations within GNNs. Kipf et al. [44] presented the Graph Convolutional Networks (GCNs), utilizing convolutional operations to gather messages from 1-step neighbourhoods surrounding each node. Velivckovic et al. [45] presented the Graph Attention Networks (GATs), employing a self-attention mechanism for the calculation of weights to gather data from neighboring nodes. Wu et al. [46] proposed the Graph Attention Model (GAM), which addresses the graph classification problem through adaptive access to the sequence of each significant node for information processing.

3. Method

3.1. Overview

Group re-ID faces significant challenges because it not only suffers from the same problems as single object re-ID, but also faces other challenges, such as variations in the number of group members. Therefore, the key to re-identifying a group is that the model can be capable of removing group members who do not belong to this group.

In response to the aforementioned challenge, a novel dynamic screening strategy based on feature graphs framework (DSFG) is proposed for group re-ID and the pipeline is shown in Figure 2. Overall, the DSFG framework can be viewed as a uniform approach with the capacity to (1) extracting the features of single person and background by CNN respectively, (2) removing members that do not belong to the group and (3) using background features to enhance the transmission of group contextual information to update individual features. Specifically, the group pairs are used as input to extract person and background features from each group. The person features are regarded as nodes, with the contextual graphs constructed using the k-nearest neighbor algorithm. The Pre-Processing module proposed attempts to remove the members who do not belong to current group. At the same time, the Graph-based Feature Matching module designed aims to enhance the transmission of group contextual information by using adjacent background semantic information. Subsequently, the background features are added as new nodes to the reconstructed graph. Finally, the Multi-Objects Context Graph (MOCG) module updates the node features in the contextual graph with both intra-group and inter-group information. The architectural details are clarified in the following subsections.

3.2. Graph-Based Feature Matching Module

Inspired by the SuperGlue [47] method, to exatract the features that do not contain a person, the areas within the person bounding boxes are first removed in each image, and then the background semantic information is obtained through the method of feature matching. Given two images A and B, each contains keypoint positions denoted as $\mathit{p}$ and their corresponding description visuals denoted as $\mathit{d}$, which are combined into local features ($\mathit{p}$, $\mathit{d}$). The i-th keypoint positions are composed of the coordinates (x, y) within images, along with a detection confidence score c, ${\mathit{p}}_i={(x,y,c)}_i$.

To incorporate keypoint i to high-dimensional vector, the original keypoint representation ${}^{\left(0\right)}\mathit{x}_i$ is mixed its description visual with its position through Multilayer Perceptron ($MLP$), as follows:

$${}^{\left(0\right)}\mathit{x}_i={\mathit{d}}_i+ML{P}_{enc}\left({\mathit{p}}_i\right).$$

(1)

The keypoints features are constructed to unified complete graph as the nodes. To facilitate information aggregation along edges within graphs, the message passing [48] combined with GNN can be calculated to aggregate representation at every level from other nodes. The intermediary representation for the keypoint i of image A in the ℓ-th layer presents ${}^{(\ell )}\mathit{x}_i^A$. And final message passing with all keypoints in A is updated:

$${}^{(\ell +1)}\mathit{x}_i^A={}^{(\ell )}\mathit{x}_i^A+MLP\left(\right[{}^{(\ell )}\mathit{x}_i^A\left|\right|{\mathit{m}}_{\epsilon \to i}\left]\right),$$

(2)

where $[·||·]$ indicates concatenation and ${\mathit{m}}_{\epsilon \to i}$ means the results of aggregating information from all other keypoints to keypoint i. The ultimate matching descriptors are obtained through linear projection:

$${\mathit{f}}_i^A=\mathbf{W}·{}^{\left(L\right)}\mathit{x}_i^A+\mathit{b},\forall i\in \mathcal{A},$$

(3)

and the keypoints in image B are in the same manner. The matching descriptors ${\mathit{f}}_i^A$ and ${\mathit{f}}_j^B$ are combined as the final matching descriptor to calculate the pairwise score:

$${\mathit{S}}_{i,j}=<{\mathit{f}}_i^A,{\mathit{f}}_j^B>\forall (i,j)\in \mathcal{A}\times \mathcal{B},$$

(4)

where $<·,·>$ is the inner product. The module has been trained using a supervised methodology and verified with the matches of ground truth $\mathcal{M}=\left\{\right(i,j)\subset \mathcal{A}\times \mathcal{B}\}$ where $I\subseteq \mathcal{A}$ and $J\subseteq \mathcal{B}$ as not matched when they are not near any reprojection. The background matching loss function $L^{match}$ is calculated as:

$$L^{match}=-\sum _{(i,j)\in \mathcal{M}}{log\overline{\mathit{P}}}_{i,j}-\sum _{i\in I}log{\overline{\mathit{P}}}_{i,N+1}-\sum _{j\in J}log{\overline{\mathit{P}}}_{M+1,j}.$$

(5)

Through the above steps, some rough background matching points are obtained in the two images. To obtain the unique background matching point, the following calculation strategy is designed by calculating smallest difference between the distance of all keypoints to a person in image A and the distance of matched keypoints to a person in image B:

$$s=min\left(\left|\frac{{\displaystyle \sum _{i=1}^{\mathrm{H}}}\sqrt{{(x_b^A-x_i^A)}^2+{(y_b^A-y_i^A)}^2}}{H}-\frac{{\displaystyle \sum _{j=1}^K}\sqrt{{(x_b^B-x_j^B)}^2+{(y_b^B-y_j^B)}^2}}{K}\right|\right),$$

(6)

where H and K represent the count of persons in images A and B. $(x_b^A,y_b^A)$ and $(x_b^B,y_b^B)$ represent the coordinates of background matching point b in images A and B, respectively. $(x_i^A,y_i^A)$ and $(x_j^B,y_j^B)$ represent the center coordinates of i-th and j-th bounding boxes of person in images A and B, respectively.

3.3. Pre-Processing Module

In this module, the group pairs are used as input. The CNN model is employed to extract person features, which can be regarded as nodes to construct a graph through k-nearest neighbor algorithm. In general, pedestrians are often divided into groups of three-persons, four-persons, five-persons or even more. Ukita et al. [49] demonstrated that all three-persons groups were judged as the same group. For other groups, there exist additional pedestrians that do not belong to this group, thereby introducing the noise for group re-ID. In order to remove additional pedestrians, a threshold f is designed to determine whether the current group members are in the same group.

$$f=\frac{N_p}{N_p+N_N},$$

(7)

where $N_p$ represents the count of persons and edges, $N_N$ represents the count of node pairs that are not in the same group. Since all three-persons groups are all judged to be the same group, $f=\frac{2}{2+1}\approx 66.6\%$, where $N_p$ denotes the edges of node 1 → node 2 and node 2 → node 3, $N_N$ denotes the nodes pairs of node 1 and node 3, as shown in Figure 3b. Therefore, when f < 66.6% they are judged as different groups, and f ≥ 66.6%, they are judged as the same group. If group members are not in the same group, a person is (or persons are) removed who do not belong to the group.

In this module, the edges of the graph are divided into strongly, moderately and weakly connected edges, as shown in Figure 3a. When two nodes choose each other as the nearest node, the edges between them are denoted as strongly connected edges. When a node is selected as the nearest node by some other nodes and also selects other nodes as the nearest node, the edge connecting the node to its nearest neighbor is denoted as a moderately connected edge, such as node 2 → node 3, where → denotes the edges of node 2 choosing node 3 as the nearest node through k-nearest algorithm. When a node selects another node as its nearest node but is not selected by other nodes, the connection between this node and its nearest node is called a weakly connected edge, such as node 1→ node 2 (the rightmost image in Figure 3a). The specific pruning process of the five-persons group is shown in Figure 3c. Specifically, for the five-persons group, $f=\frac{4}{4+3}\approx 57.1\%<66.6\%$ which is judged as different group, where $N_P$ = 4 represents the edges of node 1 → node 2, node 2 → node 3, node 3 → node 4, node 4 → node 5 and $N_N$ = 3 represents the node pairs of node 1 and node 2, node 1 and node 3, node 1 and node 4, which are not in the same group. Therefore, the weakly connected edges (node 1 → node 2 or node 4 → node 5) and their nodes need to be pruned to obtain the four-persons group. If the edge of node 4 → node 5 is longer than node 1 → node 2, the longer weakly connected edges and its node (yellow node 5) should be removed. Subsequently, the four-persons group continue to determine whether the group is the same group, $f=\frac{3}{3+2}\approx 60\%<66.6\%$ which is judged as different group, where $N_P$ = 3 represents the edges of node 1 → node 2, node 2 → node 3, node 3 → node 4 and $N_N$ = 2 represents the node pairs of node 1 and node 2, node 1 and node 3, which are not in the same group. Finally, the weakly connected edges and their node (red node 1) are pruned to obtain the three-persons group. In addition, the pruning process of all five-persons, four-persons, and three-persons groups are simulated respectively, as shown in Figure 3d.

3.4. Multi-Object Context Graph Module

After obtaining the reconstructed graph, the background features are added as nodes to the graph. In this module, a multi-level attention mechanism is adopted to capture intra-graph and inter-graphs contextual information between background and person features.

Each person feature ${\mathit{h}}_{si}$ and background feature ${\mathit{h}}_{rb}$ by CNN model can be divided into P parts as ${\mathit{h}}_{si}=[{\mathit{h}}_{si1},...,{\mathit{h}}_{siP}]$ and ${\mathit{h}}_{rb}=[{\mathit{h}}_{rb1},...,{\mathit{h}}_{rbP}]$, respectively. The descriptions of symbols are provided in

Table 1. Descriptions of main symbols mentioned in this paper.

Symbols	Description
$I_s$	s-th input group image
$N_s$	number of person in $I_s$
$G_s$	group context graph in $I_s$
t	the t-th layer
${\mathit{h}}_{sbp}^{\left(t\right)}$	the feature vector from the p-th part of background information b in $G_s$
${\mathit{h}}_{sbq}^{\left(t\right)}$	the feature vector from the q-th part of background information b in $G_s$
${\mathit{m}}_{sip}^{\left(t\right)}$	the message from the p-th intra-part in $G_s$
${\mathit{n}}_{sip}^{\left(t\right)}$	the message from the p-th inter-parts in $G_s$
${\mathit{\mu }}_{sip}^{\left(t\right)}$	the message from the p-th inter-graphs in $G_s$
${\mathit{h}}_s$	graph representation in $G_s$
${\mathit{h}}_r$	graph representation in $G_r$

.

3.4.1. The Relationship of Intra-Graph

The person and background features are divided into four parts. The correlation calculations of intra-graph can be divided into the same (intra-part) and different parts (inter-part) of the background features and person features respectively.

The intra-part correlation between ${\mathit{h}}_{sip}^{(t-1)}$ and ${\mathit{h}}_{sbp}^{(t-1)}$ can be calculated in image $I_s$. These features consist of features extracted from person node i and background node b, which correspond to the same part p. It is able to determine the important weights of intra-part messages transmitted from b to i:

$$e_{sibp}=\phi ({\mathbf{W}}_e^{(t-1)}{\mathit{h}}_{sip}^{(t-1)},{\mathbf{W}}_e^{(t-1)}{\mathit{h}}_{sbp}^{(t-1)}),$$

(8)

where $\phi$ represents a correlation measurement function and ${\mathbf{W}}_e^{(t-1)}$ represents a weight matrix. All important weights $e_{sibp}$ are normalized using the softmax function to calculate the attention weights:

$$a_{sibp}=softmax\left(e_{sibp}\right).$$

(9)

Finally, the update attention weights ${\mathit{o}}_{sibp}^{\left(t\right)}$ of the intra-part are obtained by merging the background nodes b to the person nodes i:

$${\mathit{o}}_{sibp}^{\left(t\right)}=a_{sibp}{\mathbf{W}}_e^{(t-1)}{\mathit{h}}_{sbp}^{(t-1)}.$$

(10)

The inter-part correlation calculation are defined in a similar manner. The inter-part messages between ${\mathit{h}}_{sip}^{(t-1)}$ and ${\mathit{h}}_{sbq}^{(t-1)}$ can be transferred from person node i and the background node b relating to distinct parts (p, q). The final update attention weights ${\mathit{r}}_{sib}^{\left(t\right)}$ of the inter-part are obtained by merging the background nodes b to the person nodes i:

$$e_{sib}^{pq}=\phi ({\mathbf{W}}_e^{(t-1)}{\mathit{h}}_{sip}^{(t-1)},{\mathbf{W}}_e^{(t-1)}{\mathit{h}}_{sbq}^{(t-1)}),$$

(11)

$$a_{sib}^{pq}=softmax\left(e_{sib}^{pq}\right)=\frac{exp\left(e_{sib}^{pq}\right)}{{\sum }_k^{(i,k)\in E_s}{\sum }_l^{l\ne p}exp\left(e_{sib}^{pl}\right)},$$

(12)

$${\mathit{r}}_{sib}^{\left(t\right)}=\sum _i^{(i,b)\in E_s}\sum _q^{q\ne p}{a}_{sib}^{pq}{\mathbf{W}}_e^{(t-1)}{\mathit{h}}_{sbq}^{(t-1)}.$$

(13)

After acquiring both intra-part message ${\mathit{o}}_{sibp}^{\left(t\right)}$ and inter-part message ${\mathit{r}}_{sib}^{\left(t\right)}$, both of them belong to the intra-graph messages. The attention mechanism within the graph is depicted in Figure 4, showing how it focuses on both the intra-part and inter-part.

3.4.2. The Relationship of Inter-Graph

The intra-graph context can be effectively modeled by intra-graph message passing, as mentioned above. However, it is essential to study the correlation and compute the similarity between groups for group re-ID. Intuitively, when two groups share the same ID, there is probable that certain correspondence exists among the individuals in both groups. Therefore, the higher the similarity of individual pairs, the higher the similarity at the group level. The important weights of inter-graph between ${\mathit{h}}_{si}^{(t-1)}$ and ${\mathit{h}}_{rb}^{(t-1)}$ in $G_s$,$G_r$ are computed as follows:

$$z_{ib}=\phi ({\mathbf{W}}_z^{(t-1)}{\mathit{h}}_{si}^{(t-1)},{\mathbf{W}}_z^{(t-1)}{\mathit{h}}_{rb}^{(t-1)}),$$

(14)

where $\phi$ denotes an inner product layer and ${\mathbf{W}}_z^{(t-1)}$ denotes a projection matrix. The calculation of attention weights for inter-graph messages transmitted from background node b in graph $G_r$ to person node i in graph $G_s$ is in the same manner:

$$w_{ib}=softmax\left(z_{ib}\right),$$

(15)

$${\mathit{f}}_{sip}^{\left(t\right)}=\sum _q{w}_{ib}{\mathbf{W}}_z^{(t-1)}{\mathit{h}}_{rbq}^{(t-1)}.$$

(16)

The above node features can be updated by Multi-Layer Perceptron (MLP) to aggregate features into a better feature space and fine-tune the model to better training:

$${\mathit{h}}_{sip}^{\left(t\right)}=MLP({\mathit{h}}_{sip}^{(t-1)},{\mathit{m}}_{sip}^{\left(t\right)},{\mathit{n}}_{sip}^{\left(t\right)},{\mathit{o}}_{sibp}^{\left(t\right)},{\mathit{r}}_{sib}^{\left(t\right)},{\mathit{\mu }}_{sip}^{\left(t\right)},{\mathit{f}}_{sip}^{\left(t\right)}).$$

(17)

3.4.3. Correspondence Learning

To improve the learning of group correspondence, the circle loss is employed to enforce feature similarity within the same group and promote the separation of different groups:

$$\begin{array}{ccccc}\hfill & \hfill & \hfill L_{\mathrm{g}}^{circle}& =log[1+\sum _{i=1}^K\sum _{j=1}^{L}exp\left(\gamma (a_s^i{\mathit{h}}_s^i-a_r^j{\mathit{h}}_r^j)\right)\hfill & \\ \hfill & \hfill & & =log[1+\sum _{j=1}^{L}exp\left(\gamma a_s^i{\mathit{h}}_s^i\right)\sum _{i=1}^{K}exp\left(\gamma a_r^j{\mathit{h}}_r^j\right),\hfill \end{array}$$

(18)

where $a_s^i$ and $a_r^j$ denote non-negative weighting factors of i-th and j-th persons in images $I_s$ and $I_r$ respectively, $\gamma$ denotes as a scale factor.

Moreover, when building global correspondence, it is crucial to take into account local person- and background-level information. The pair-wise loss can be adopted for person- and background-level corresponding learning:

$$L_p^{pair}=\sum _{i\in v_s,j\in v_r}\sum _{p}max\{0,m-y_p^{ij}(1-{∥{\mathit{h}}_{sip}-{\mathit{h}}_{rjp}∥}^2)\},$$

(19)

$$L_b^{pair}=\sum _{i\in v_s,b\in v_r}\sum _{b}max\{0,m-y_b^{ib}(1-{∥{\mathit{h}}_{sip}-{\mathit{h}}_{rbp}∥}^2)\},$$

(20)

where m denotes the margin and $y_p^{ij}$, $y_b^{ib}$ represent the pair labels. Finally, the value of forecast matrix $\mathit{S}$ and ground reality matrix ${\mathit{S}}^{gt}$ can be calculated through cross-entropy loss [50]:

$$L_{pce}=-\sum _{i\in v_s,j\in v_r}({\mathit{S}}_{i,j}^{gt}log{\mathit{S}}_{i,j}+(1-{\mathit{S}}_{i,j}^{gt}log(1-{\mathit{S}}_{i,j})),$$

(21)

where ${\mathit{S}}^{gt}\in R^{Ns\times Nr}$ denotes a binary matrix, ${\mathit{S}}_{i,j}^{gt}$ = 1 when i-th and j-th persons show the same identity. The total loss is calculated by combining all the aforementioned loss functions in a linear manner:

$$L=L^{match}+L_g^{circle}+L_p^{pair}+L_b^{pair}+L_{pce}.$$

(22)

4. Experimental Results

4.1. Datasets and Experimental Settings

Datasets. (1) the PRAI-1581 dataset [51] contains 39,461 person images from two UAVs flying at heights between 20 and 60 m above ground level and the resolution of person images is 30–50 pixels; (2) the Road Group dataset (RG) [10] contains 162 pairs of group images taken by two cameras and the resolution of group images is 89–302 pixels; (3) the DukeMTMC Group dataset (DG) [11] based on DukeMTMC dataset contains 177 pairs of group images captured by eight cameras and the resolution of group images is about 180–1034 pixels; (4) the CUHK-SYSU-Group dataset (CSG) [14] based on CUHK-SYSU dataset contains 3839 pairs of group images and the resolution of group images is 600 × 800 pixels; (5) CM-group dataset [52] contains visible and infrared 30,946 images of 427 groups and the resolution of group images is 2560 × 1440 pixels; (6) the SYSU-MM01 dataset [53] contains 45,863 visible and infrared person images and the resolution of person images is about 83–548 pixels; (7) the Aerial Image dataset (AID) [54] contains 10,000 aerial scene images, featuring 30 distinct classes with each class containing between 220 to 420 images at an altitude from about half a meter to about 8 m and the resolution of aerial images is 600 × 600 pixels; and (8) RGB-NIR Scene dataset [55] contains 477 images in nine classes in RBG and Near-infrarted(NIR) and the resolution of images is 1024 × 768 pixels. The PRAI-1581, CSG, RG, DG, AID datasets are visible images with a spectral resolution of 390 nm–780 nm, the CM-group, SYSU-MM01 datasets are visible-infrared images with a spectral resolution of 1 mm–750 nm, and RGB-NIR Scene dataset are RGB-NIR images with a spectral resolution of 1000 nm–1700 nm. In addition, the PRAI-1581 and AID datasets contain the messages of geometric and spatial resolution, while the other datasets only contain geometric resolution messages. Figure 5 and Figure 6 show the examples of images.

Experimental Settings. The datasets are divided into 60% training set and 40% testing set in a random manner. The model utilize ResNet50 as our backbone and train 300 epochs with 0.0001 initial learning rate. Each 100 epochs of training, the learning rate decreases by 10%. To facilitate implementation, virtual nodes are added to graphs with different numbers of nodes. The model employs two-layer GNNs and trains on a 2080ti GPU, which takes approximately 60 h to convergence on CSG dataset.

4.2. Comparison with Other Group Re-ID Methods

Because of absence of UAV group re-ID datasets, it is hard to perform our method on such datasets is challenging. However, the images of group re-ID datasets are similar to UAV person re-ID datasets, both from a top-down perspective, as shown in Figure 5a,b. Therefore, the several existing group re-ID methods are compared with our model on RG, DG and CSG datasets, including crowd matching-based and features aggregation-based methods, experimental results are shown in

Table 2. In comparison to several group re-ID methods. The best results are shown are in black boldface font.

Methods	CUHK-SYSU-Group				Road Group				DukeMTMC Group
	R1	R5	R10	R20	R1	R5	R10	R20	R1	R5	R10	R20
Crowd matching-based methods
CRRRO-BRO (BMVC2009) [10]	10.4	25.8	37.5	-	17.8	34.6	48.1	-	9.9	26.1	40.2	-
Covariance (ICPR2010) [8]	16.5	34.1	47.9	67.0	38.0	61.0	73.1	82.5	21.3	43.6	60.4	78.2
BSC+CM (ICIP2016) [9]	24.6	38.5	55.1	73.8	58.6	80.6	87.4	92.1	23.1	44.3	56.4	70.4
PREF (CVPR2017) [16]	19.2	36.4	51.8	70.7	43.0	68.7	77.9	85.2	22.3	44.3	58.5	74.4
LIMI (MM2018) [41]	-	-	-	-	72.3	90.6	94.1	-	47.4	68.1	77.3	-
MGR (TCYB2021) [12]	57.8	71.6	76.5	82.3	80.2	93.8	96.3	97.5	48.4	75.2	89.9	94.4
Features aggregation-based methods
DotGNN (MM2019) [42]	-	-	-	-	74.1	90.1	92.6	-	53.4	72.7	80.7	-
GCGNN (TMM2020) [11]	-	-	-	-	81.7	94.3	96.5	97.8	53.6	77.0	91.4	94.8
MACG(TPAMI2020) [14]	63.2	75.4	79.7	84.4	84.5	95.0	96.9	98.1	57.4	79.0	90.3	94.3
DotSCN(TCSVT2021) [15]	-	-	-	-	84.0	95.1	96.3	98.8	86.4	98.8	98.8	98.8
PRM (IEEEAccess2021) [43]	-	-	-	-	62.4	75.9	82.1	88.3	-	-	-	-
Ours	71.8	81.9	86.0	88.9	86.4	95.5	97.0	98.5	57.8	80.2	88.5	94.4

. It is readily apparent our approach reaches competitive results in all three datasets. More precisely, our approach reaches Rank-1 accuracies of 71.8%, 86.4% and 57.8% on the CSG, RG and DG datasets, respectively.

Comparison with crowd matching-based methods: Crowd matching-based methods adopt a single or multi-person matching strategy and assign important weights to people to capture the dynamic changes within the group. However, these methods (e.g., PREF [16], MGR [12]) cannot solve the challenge of varying group members well. Our novel approach demonstrates superior performance in comparison to all the methods evaluated. For example, compared to the MGR method, our proposed approach reaches a Rank-1 improvement of 14% on CSG dataset, a Rank-1 improvement of 6.2% on the RG dataset and a Rank-1 improvement of 9.4% on the DG dataset.

Comparison with features aggregation-based methods: Features aggregation-based methods aggregate the contextual information from neighbors to re-identify each individual within groups as well as the entire group. However, these methods (e.g., GCGNN [11], MACG [14]) ignore the importance of background semantic information for messages transmission. Our proposed method significantly outperforms most compared methods. For example, compared to MACG method, our method reaches a Rank-1 improvement of 8.6% on the CSG dataset and a Rank-1 improvement of 1.9% on the RG dataset.

4.3. Person Re-ID in Groups

Utilizing group context information has been demonstrated for single person re-ID [10,12,13]. In our method, the background information and screening strategy can be utilized to enhance group and individual correspondence learning, which can directly apply to single person re-ID. Our method is compared with the person re-ID model without group context information, such as our baseline CNN, Strong-Baseline [56], PCB [57] and OSNet [58], and with group context information, such as MGR [12], Context Graph (CG) [13] and MACG [14].

Table 3. Comparison of results for single person re-ID with the inclusion of group context information. ✓ denotes the model uses group context information.

Model	Group	Rank-1	Rank-5	Rank-10
Our CNN	-	63.5	74.2	79.9
Strong-Baseline [56]	-	63.1	80.3	84.3
PCB [57]	-	63.7	80.2	83.9
OSNet [58]	-	62.4	77.0	81.0
MGR [12]	✓	63.8	79.9	83.8
CG [13]	✓	62.7	78.4	82.6
MACG [14]	✓	65.6	80.5	84.6
DSFG	✓	68.2	81.2	84.9

shows the experimental results. In comparison to our baseline CNN, our model reaches a 4.7% increase in Rank-1 accuracy. Compared with MACG, our model reaches a 2.6% increase in Rank-1 accuracy. The above results reveal our model can effectively enhance the accuracy of single-person re-ID. The experimental results above illustrate that using group information not only helps in groups but also makes it easier to re-identify a single person. Therefore, the proposed DSFG framework can also re-identify a single person, even a UAV person.

To assess the generalization capability of our framework for UAV person re-ID, the model trained on CSG is directly applied to the UAV PRAI-1581 dataset, as shown in

Table 4. CSG dataset transfer results for PRAI-1581 dataset.

Model	CSG —> PRAI-1581
	Rank-1	Rank-5	Rank-10
Strong-Baseline [56]	46.10	49.23	52.34
PCB [57]	48.07	51.20	55.97
OSNet [58]	54.40	58.85	62.37
Ours	51.32	55.10	57.38

. It is easy to observe that our method can effectively improve the accuracy of UAV person re-ID. For example, compared with PCB, our model reaches a 3.25% increase in Rank-1 accuracy. Since PRAI-1581 dataset lacks group context, a single person need to be replicated to construct a group for building the context graph.

In addition, to explore the effectiveness of our model in infrared images, our models trained on the CSG and CM-Group datasets are also directly applied to the SYSU-MM01 dataset, respectively, as shown in

Table 5. CSG and CM-Group datasets transfer results for SYSU-MM01 dataset.

Model	CSG —> SYSU-MM01
	Rank-1	Rank-5	Rank-10
Strong-Baseline [56]	34.39	36.34	37.83
PCB [57]	36.54	38.47	40.61
OSNet [58]	40.18	43.64	45.59
Ours	38.12	41.22	43.67
	CM-Group —> SYSU-MM01
Ours	52.24	55.83	57.92

. Note that only the infrared images are trained on CM-Group datasets. It can be observed to find that the transfer results of the model trained using CM-Group to SYSU-MM01 dataset is much higher than the model trained using CSG. It is probably that there is less useful information in infrared images. Therefore, our model works separately for visible and infrared images, but it is best to transfer in the same modality.

4.4. Model Analysis

Backbone network. The impact of different backbone networks are analyzed on model performance. In this study, the ResNet50 backbone is replaced by ResNet18, ResNet34, ResNet101, DenseNet121 and DenseNet169. Figure 7a shows the experiment results. Significant performance decline is observed when utilizing shallow networks, especially in Rank-1. Furthermore, the utilization of ResNet101 only resulted in minor improvements when compared to ResNet50. These results highlight the substantial influence of distinct backbone networks on the performance with our model. Ultimately, it is clearly easy to observe ResNet50 strikes an optimal balance between model complexity and performance, making it a suitable choice for the backbone in our setup.

Body parts. The effects of various human body and background information partitions are evaluated. The results of various partitions are shown in Figure 7b. It is evident from the results that a P = 2 partition yields notably lower performance compared to other partitions, primarily due to its coarse segmentation. Conversely, the most favorable performance is attained when employing P = 4 partitions.

Node connection. In our framework, the context graphs are constructed with connecting all members in a group. Figure 7c shows the experiment results, where ’Neighbor = K’ means that an individual is exclusively linked to its K closest neighbors. The observed trend indicates a consistent improvement in performance as K increases in size, i.e., with denser connections. In the case of ‘Neighbor = 3’ and ‘Neighbor = 4’, the accuracy is already at its highest. In the case of ‘Neighbor = 5’, the accuracy is lower than ‘Neighbor = 3’ and ‘Neighbor = 4’. It is probably that most of groups the existing datasets are four-persons or five-persons. In addition, when constructing the context graph with empty nodes, the model may create redundancy to result in performance decrease. It can be observed that ‘Neighbor = 3’ and ‘Neighbor = 4’ graph configurations facilitate effective passing of messages in the group, thereby aiding model in acquiring contextual information and screening the other person not belonging to the current group.

GNN layers. Furthermore, the impact of utilizing different numbers of GNN layers (T = 1, …, 5) is analyzed. The experimental results are shown in Figure 7d. It is easy to find that the use of two-layer GNN results in improved performance compared to GNNs with other layer numbers. This improvement can be attributed to the fact that single-layer GNN may not provide adequate representational capacity, whereas GNNs with more layers may introduce excessive parameters, increasing the risk of overfitting. This observation could be linked to limited size of the dataset, and the need for further dataset expansion might require the utilization of deeper GNNs to improve performance.

4.5. The Visualization of the Results

Graph-based Feature Matching Module. To confirm the GFM module’s effectiveness, the image matching results are visualized on CSG dataset by using the original image and removing the person area. The visualization results are shown in Figure 8. It can be observed that keypoints matched in Figure 8b are much less than those in Figure 8a, effectively reducing the keypoints of pedestrians. It is more helpful for us to find additional useful features that do not contain the object (person).

In addition, the GFM module is performed to feature-match and re-identify the images of same class and different classes on AID and RGB-NIR Scene datasets, respectively. The visualization results are depicted in Figure 9 and Figure 10. In Figure 9, the first two lines are visible images matching on AID dataset. It can be observed that there are far more matching points in the same class of images than in different classes of images. The last two lines are NIR images matching on RGB-NIR Scene dataset. It can be found that our proposed module not only adapts to visible images, but also NIR images. In Figure 10, the first two lines are visible images re-ID of the same class on AID dataset, the last two lines are NIR images re-ID of the same class on the RGB-NIR Scene dataset. It is evident to observe that the GFM module can effectively retrieve the same class.

Pre-Processing Module. As the group moves, the members of the group will change. To solve the above issue, the PREP module presented aims to remove additional pedestrians. The experimental results of the pruning process in six-persons groups are shown in Figure 11. The $SSIM$ algorithm is utilized to determine the structural similarity between the pruned group and contrast group. It can be observed that the group similarity is higher after each pruning operation. In addition, the different structures are designed with five-persons groups and four-persons groups and the situations in which they are judged as the same group and different groups, as shown in Figure 12a,b. The experimental results indicate that compared with the contrast groups, groups judged as different groups exhibit lower similarity than groups judged as the same.

The visualization of group and UAV person re-ID results. The visualization results are depicted in Figure 13 and Figure 14, respectively. The images from the test set can be retrieved in the gallery set and it is easy to observe that persons sharing a similar appearance have an impact on the retrieval results, which may make it hard to retrieve the correct match with the model.

4.6. Ablation Study

Module Ablation. The ablation experiments are performed on three datasets to confirm the effectiveness of our proposed module, as shown in

Table 6. Examination of the components of our proposed approach on three datasets. Bold indicates the best performance.

	CSG	RG	DG
MACG	63.2	84.5	57.4
MOCG + GFM	64.9	84.4	44.7
MACG + PREP	65.2	84.9	57.8
MOCG + GFM + PREP	66.5	85.0	46.8
MOCG + GFM + PREP + CL	71.8	86.4	49.5

. The model regards the method of MACG as our baseline and proposes the MOCG module based on the GFM module to improve the MACG method. It can be understood that after using the GFM module to obtain background features, the MOCG method proposed needs to be used. It is easy to observe the images of CSG datasets exist overlapping areas under different cameras, so adding the GFM module can effectively improve the performance. In the RG dataset, almost every image has the same ground background information. Therefore, adding the GBFM module has an insignificant impact and it cannot be used as an effective way to associate two images. In the DG dataset, the same group of people will move to another scene with completely different background information (across background scenes). In this case, the background semantic information does not match and adding it may introduce noise to group re-ID, thus reducing the performance extensively. Therefore, the method of the GFM module still has limitations on some specific datasets. In addition, experimental results prove that the PREP module and circle loss [59] (CL) function have significant improvements on three datasets.

In addition, the distinct variants of the framework are examined on CSG dataset, such as Graph-based Feature Matching (GFM), Pre-processing (PREP) module and circle loss (CL) [59] function, as shown in

Table 7. Ablation study of various combination with our proposed modules on the CSG dataset. ✓ denotes the model chooses the various modules.

Modules	Base	Variants	Variants	Variants	Variants	Variants	Variants
GFM		✓			✓		✓
PREP			✓			✓	✓
CL				✓	✓	✓	✓
Rank-1	63.2	64.9	65.2	65.4	68.8	69.2	71.8

. The ultimate model performance can be determined by rearranging elements in diverse arrangements.

Distance ablation. To confirm the efficacy of utilizing Euclidean distance in calculating the distance between keypoints and person, the Euclidean distance is compared with Cosine and Manhattan, respectively, as shown in the

Table 8. Ablation study of various distance on CSG dataset. Bold indicates the best performance.

	Rank-1	Rank-5	Rank-10
Manhattan distance	55.2	62.3	66.9
Cosine distance	60.7	70.4	74.1
Euclidean distance	71.8	81.9	86.0

. It can be found that the Rank-1 of the Euclidean distance is higher than the others. It is probably that the relative distance between person and the same background matching points is unchanging, so the optimal background matching point can be calculated with person and background points by using Euclidean distance.

Loss Fuction Ablation. The impact of employing various loss functions on our model’s effectiveness is illustrated by employing cross-entropy loss, triple loss, and circle loss functions. The circle loss has two different parameters, i.e., 32 and 64. The experimental results are reported on CSG dataset in

Table 9. The experiment of circle loss replacing cross entropy loss or triple loss function on CSG dataset. Bold indicates the best performance.

	Rank-1	Rank-5	Rank-10	Rank-20
Ours	66.5	72.5	77.3	80.2
Ours + cross entropy loss	52.6	67.8	74.2	78.5
Ours + triple loss	68.4	75.6	82.3	86.2
Ours + circle loss_32	70.1	81.1	84.4	88.0
Ours + circle loss_64	71.8	81.9	86.0	88.9

. Our findings indicate that the utilization of circle loss function effectively minimizes feature distance within a class while simultaneously maximizing the distance between classes.

5. Discussion

In the group re-ID tasks, the model performs with high accuracy using our proposed algorithm on three datasets. And it can be directly applied to the UAV visible person re-ID PRAI-1581 dataset and infrared SUSY-MM01 dataset for the first time. However, our proposed algorithm needs too much time to train. This is probably because after new background features nodes are added to the graph, the model requires a lot of time to transfer context information between groups during the matching process. In addition, although graph structure has achieved great achievement in solving the issues of group re-ID, no solution based on spatio-temporal information has been found. It is probably that there is a lack of appropriate data. It can be argued that the spatio-temporal problem may effectively enhance not only the transmission of information between group members, but also the problem of member changes (i.e., the time when persons enter and exit the images). Finally, given the scarcity of datasets for UAV group re-ID, it is worthwhile to prioritize research efforts in the future.

6. Conclusions

In this paper, a novel framework is proposed to solve the challenge of group re-ID, containing three modules: Graph-based Feature Matching (GFM), Pre-Processing (PREP) and Multi-Object Centext Graph (MOCG). The GFM module enhances the transmission of group contextual information by adding background matched features for the first time. The PREP module solves the challenge of group member changes in the group re-ID, which can remove group members that do not belong to the current group through pruning operations. And the MOCG module aggregates the messages from background features to person features and updates the person and group features maximizely. Experiments show that outstanding results are obtained on three datasets for group re-ID, demonstrating the efficacy of the suggested methodology. The Rank-1 on CUHK-SYSU-Group, Road Group and DukeMTMC Group datasets reach 71.8%, 86.4% and 57.8%, respectively. In addition, the models trained on CSG and CM-Group datasets to the UAV visible person re-ID PRAI-1581, infrared SUSY-MM01 and RGB-NIR Scene datasets, which achieve certain results.