Multi-Rotor Drone-Based Thermal Target Tracking with Track Segment Association for Search and Rescue Missions

Abstract

Multi-rotor drones have expanded their range of applications, one of which being search and rescue (SAR) missions using infrared thermal imaging. This paper addresses thermal target tracking with track segment association (TSA) for SAR missions. Three types of associations including TSA are developed with an interacting multiple model (IMM) approach. During multiple-target tracking, tracks are initialized, maintained, and terminated. There are three different associations in track maintenance: measurement–track association, track–track association for tracks that exist at the same time (track association and fusion), and track–track association for tracks that exist at separate times (TSA). Measurement–track association selects the statistically nearest measurement and updates the track with the measurement through the IMM filter. Track association and fusion fuses redundant tracks for the same target that are spatially separated. TSA connects tracks that have become broken and separated over time. This process is accomplished through the selection of candidate track pairs, backward IMM filtering, association testing, and an assignment rule. In the experiments, a drone was equipped with an infrared thermal imaging camera, and two thermal videos were captured of three people in a non-visible environment. These three hikers were located close together and occluded by each other or other obstacles in the mountains. The drone was allowed to move arbitrarily. The tracking results were evaluated by the average total track life, average mean track life, and average track purity. The track segment association improved the average mean track life of each video by 99.8% and 250%, respectively.

1. Introduction

Multi-rotor drones have extended their application to diverse domains in commerce, military, and industrial areas [1,2,3,4]. They can cover large and hazardous areas quickly and provide real-time data when being controlled remotely or autonomously [5]. A multi-rotor drone’s hovering function at arbitrary altitude and space enable hard tasks to be performed simply and easily [6]. This operation is also cost-effective and requires low human resources [7]. Multi-rotor drones with standard cameras are often employed for detection [8,9] and target tracking [10,11,12,13,14,15,16,17,18,19].

Infrared thermal imaging detects infrared radiation emitted by warm objects and generates a gray-scale image. Thus, thermal imaging can be used effectively when visibility is low, and temperature sensing is important. However, this technology has limited imaging quality compared to visible-light images [20]. Various techniques have been developed for thermal target tracking [21].

In SAR missions, drone-based thermal imaging offers significant advantages in locating people in low-visibility conditions [22,23]. Other applications include agriculture [24,25], wildlife monitoring [26], non-invasive inspection [5], firefighting [27], and surveillance [28].

Drone-based people detection with thermal imaging was studied in Refs. [29,30,31]. Persons, animals, and cars were detected through the YOLO approach in Refs. [29,30]. A two-stage detection method recognized persons in Ref. [31]. Object detection and tracking on the sea surface was demonstrated using a thermal camera mounted on an unmanned aerial vehicle (UAV) in Ref. [32]. However, in Ref. [32], non-living objects were tracked by a fixed-wing drone. A multimodal fusion tracker that utilizes both RGB and thermal data was proposed in Ref. [33].

Multi-rotor drone-based people tracking with thermal imaging was performed in Ref. [34]. The YOLOv5 detection model was adopted, and the bounding box gating was proposed in Ref. [35]. However, in Ref. [35], the number of track breakages increased due to the high maneuvering of the target or platform.

A multiple-target tracker estimates the dynamic state of multiple targets based on observations. Observations often result in corrupted measurements due to noise, missed detections, and false alarms. In order to overcome the imperfection of observation, state estimators, such as Kalman filtering or the interacting multiple model (IMM) approach [36,37], are often adopted. A track can be viewed as a set of state estimates based on measurements. Therefore, it is important to associate measurements of the same targets with tracks. Measurement–track association and track association and fusion facilitate maintenance of the same measurement source in a track [38,39,40].

Tracks can be broken for various reasons, such as wrong measurement-to-track associations, long missed detections, or high maneuvering of the target or platform. In order to connect broken tracks, i.e., track segments, track segment association (TSA) was first proposed and applied to an airborne early warning system (AEW) scenario in Ref. [41].

In this paper, multi-rotor done-based thermal target tracking for SAR missions is addressed. There are three types of associations in track maintenance: measurement–track association, track–track association for tracks exist at the same time (track association and fusion), and track–track association for tracks exist at separate times (TSA). These three associations, along with track initialization and termination, aim to generate a unique, pure, and continuous track for each target.

Figure 1 shows the framework of multiple-target tracking following object detection. The detection results are input into the target tracker as measured positions. In this study, object detection was performed by the YOLOv5 model, the fifth iteration of a popular deep learning framework for object detection [42]. The same videos and YOLOv5x detection results from Ref. [35] were utilized since this study aims to develop a target tracker and compare it with the previous study.

Figure 1. Framework of multiple target tracking following object detection.

The contributions of this study are as follows: (1) TSA is further developed and clarified from Ref. [41]; the sequential steps of TSA are newly defined as the selection of candidate track pairs, backward filtering, association testing of forward and backward estimates, and an assignment rule. This technology significantly reduces the number of track breakages and increases track continuity. (2) The framework that combines three associations for track maintenance is completed with the IMM approach. The IMM filter shows robust performance on high-maneuvering targets [41]. It is used for both forward and backward filtering in this study. This framework is also shown to work effectively even when the platform moves rapidly in a complex background. The contributions above differentiate this study from the previous work [35], where only Kalman filtering was applied and track continuity was low due to track breakages.

In the experiments, a small drone-based thermal imaging camera captured three hikers at night. The hikers were closely located and heavily occluded by other people, trees, or leaves, and the drone could move quickly in any direction. The average total track life (TTL) of the two videos is observed to be 0.998 and 0.931, respectively. The average mean track life (MTL) of the two videos is 0.998 and 0.584, respectively. The average track purity (TP) of the two videos is obtained to be 1 and 0.982, respectively. The average TTL is improved by track association and fusion, and the average MTL is improved by TSA. The evaluation metrics, TTL, MTL, and TP are originally defined in Ref. [41] but slightly modified in Refs. [35,39,40].

The paper is structured as follows: Section 2 presents multiple-target tracking. The experiments are demonstrated in Section 3, and the discussion is in Section 4. Conclusions follow in Section 5.

2. Multiple-Target Tracking

Figure 2 shows the entire process of multiple-target tracking. This section describes each part in detail but briefly covers track initialization and track association and fusion.

Figure 2. Block diagram of multiple-target tracking.

2.1. System Modeling

Targets are described by the multi-mode dynamic equation, which follows the nearly constant velocity motion as

$${\mathit{x}}^t\left(k\right)=F\left(\tau \right){\mathit{x}}^t\left(k-1\right)+q\left(\tau \right){\mathit{v}}_j\left(k-1\right),\hspace{1em}\hspace{1em}j=1,\dots ,M,\hspace{1em}\hspace{1em}t=1,\dots ,N_T\left(k\right),$$

(1)

$$F(\tau )=\left[\begin{array}{c}1\tau 00\\ 0100\\ 001\tau \\ 0001\end{array}\right], q(\tau )=\left[\begin{array}{c}{\tau }^2/2 0\\ \tau \hspace{0.33em}0\\ 0 {\tau }^2/2\\ 0 \tau \end{array}\right],$$

(2)

where the state vector ${\mathit{x}}^t\left(k\right)=[x^t\left(k\right) v_x^t\left(k\right) y^t\left(k\right) v_y^t(k){]}^T$ comprises the positions, $x^t\left(k\right)$ and $y^t\left(k\right)$, and the velocities, $v_x^t\left(k\right)$ and $v_y^t$(k), in a two-dimensional plane; the superscript T denotes the transpose of the matrix; $\tau$ is the frame interval; and ${\mathit{v}}_j\left(k\right)$ is a process noise vector. It is assumed that the process noise follows the zero-mean white Gaussian distribution. The covariance matrix of the process noise vector is $Q_j=diag\left([{\sigma }_{jx}^2 {\sigma }_{jy}^2]\right)$. M is the mode number of the IMM filter. The target is modeled as a multi-state representing different modes in the IMM approach. $N_T\left(k\right)$ is the number of targets at frame k. Ideally, the number of targets $N_T\left(k\right)$ is equal to the number of generated tracks. The measurement equation represents observations as

$${\mathit{z}}^t(k)=\left[\begin{array}{c}{z}_x^t\left(k\right)\\ z_y^t\left(k\right)\end{array}\right]=H{\mathit{x}}^t(k)+\mathit{w}(k),$$

(3)

$$H=\left[\begin{array}{c}1000\\ 0010\end{array}\right],$$

(4)

where $\mathit{w}\left(k\right)$ is a measurement noise vector. It is also assumed to be zero-mean white Gaussian. Its covariance matrix is $R=diag\left(\left[r_x^2{ r}_y^2\right]\right)$.

2.2. Track Initialization

Two observed positions in the consecutive frames initialize a track. The two positions satisfy the following speed gating:

$${\displaystyle \frac{‖\left[z_x^t\left(k\right)-z_x^t\left(k-1\right) z_y^t\left(k\right)-z_y^t(k-1)\right]‖}{\Delta }}\le V_{max},$$

(5)

where $‖·‖$ denotes the l₂ norm, and V_max is the maximum target speed allowed for the initialization. The detailed initialization process is described in [38,39,40].

2.3. Measurement–Track Association with Forward Filtering

During measurement–track association, the IMM filter is adopted as a state estimator to handle multi-modal target states. The IMM filter consists of multi-mode interaction, mode-matched Kalman filtering, and track update.

2.3.1. Multi-Mode Interaction

The multi-mode interaction is essential in the IMM estimator; the state and covariance of target t for mode j at the previous frame $k-1$ are mixed to generate the interacted states, interacted covariances, and interacted mode probabilities as

$${\widehat{\mathit{x}}}_{0j}^t\left(k-1|k-1\right)={\sum }_{i=1}^M{\widehat{\mathit{x}}}_i^t\left(k-1|k-1\right){\mu }_{i|j}^t(k-1|k-1)$$

(6)

$$P_{0j}^t\left(k-1|k-1\right)={\sum }_{i=1}^M\left\{\begin{array}{c}{\mu }_{i|j}^t\left(k-1|k-1\right)\left\{\begin{array}{c}{P}_i^t\left(k-1|k-1\right)+\\ \left[{\widehat{\mathit{x}}}_i^t\left(k-1|k-1\right)-{\widehat{\mathit{x}}}_{0j}^t\left(k-1|k-1\right)\right]{\left[{\widehat{\mathit{x}}}_i^t\left(k-1|k-1\right)-{\widehat{\mathit{x}}}_{0j}^t\left(k-1|k-1\right)\right]}^T\end{array}\right\}\end{array}\right\},$$

(7)

$${\mu }_{i|j}^t\left(k-1|k-1\right)={\displaystyle \frac{p_{ij}{\mu }_i^t(k-1)}{\sum _{i=1}^M{p}_{ij}{\mu }_i^t(k-1)}}, i,j=1,\dots ,M,$$

(8)

where ${\widehat{\mathit{x}}}_i^t\left(k-1|k-1\right)$, $P_i^t\left(k-1|k-1\right)$, and ${\mu }_i^t\left(k-1\right)$ are, respectively, the state, covariance, and mode probability. The initial mode probabilities are set to 1/M. p_ij is the mode transition probability from mode i to mode j. The mode transition probability matrix is set to $\left[\begin{array}{cc}0.8& 0.2\\ 0.3& 0.7\end{array}\right]$.

2.3.2. Mode-Matched Kalman Filtering with the Nearest Neighbor Measurement

The state and covariance of target t at frame k are updated as

$${\widehat{\mathit{x}}}_j^t\left(k|k\right)={\widehat{\mathit{x}}}_j^t\left(k|k-1\right)+W_j^t\left(k\right){\mathit{\nu }}_{{\widehat{m}}_j^t\left(k\right)j}^t\left(k\right),$$

(9)

$$P_j^t(k|k)=P_j^t(k|k-1)-W_j^t(k)S_j^t(k)W_j^t(k{)}^T,$$

(10)

where ${\widehat{\mathit{x}}}_j^t(k|k-1)$ and $P_j^t(k|k-1)$, respectively, are the state and covariance prediction of target t as

$${\widehat{\mathit{x}}}_j^t\left(k|k-1\right)=F{\widehat{\mathit{x}}}_{0j}^t\left(k-1|k-1\right),$$

(11)

$$P_j^t (k|k-1)=F{P}_{0j}^t(k-1|k-1)F^T+q\left(\tau \right)Q_{j}q(\tau {)}^T.$$

(12)

The filter gain $W_j^t(k)$ in Equations (9) and (10) and the residual covariance $S_j^t(k)$ in Equation (10) are, respectively, obtained as

$$W_j^t(k)=P_j^t\left(k|k-1\right)H^T{S}_j^t(k{)}^{-1},$$

(13)

$$S_j^t\left(k\right)=H{P}_j^t\left(k|k-1\right)H^T+R.$$

(14)

The measurement residue ${\mathit{\nu }}_{mj}^t\left(k\right)$ in Equation (9) is

$${\mathit{\nu }}_{mj}^t\left(k\right)={\mathit{z}}_m\left(k\right)-H{\widehat{\mathit{x}}}_j^t\left(k|k-1\right),$$

(15)

where ${\mathit{z}}_m\left(k\right)$ is the m-th position measurement. The nearest measurement associated with the j-th mode filter of target t at frame k minimizes the following statistical distance squared:

$${\widehat{m}}_j^t\left(k\right)={argmin}_{m=1,\dots ,N\left(k\right)}‖{\mathit{\nu }}_{mj}^t{\left(k\right)}^T{\left[S_j^t\left(k\right)\right]}^{-1}{\mathit{\nu }}_{mj}^t(k)‖,$$

(16)

where N(k) is the number of measurements at frame k. The selected measurement must pass the following measurement gating and speed gating:

$${\mathit{\nu }}_{{\widehat{m}}_j^t\left(k\right)j}^t{\left(k\right)}^T{\left[S_j^t\left(k\right)\right]}^{-1}{\mathit{\nu }}_{{\widehat{m}}_j^t\left(k\right)j}^t\left(k\right)\le {\gamma }_m \& {\displaystyle \frac{‖{\mathit{z}}_{{\widehat{m}}_j^t\left(k\right)}\left(k\right)-\left[\begin{array}{c}{x}^t\left(k-1|k-1\right) \\ y^t(k-1|k-1)\end{array}\right]‖}{\tau }}\le S_{max},$$

(17)

where ${\gamma }_m$ is the threshold for the measurement gating, and $S_{max}$ is the maximum target speed. The measurement gating is equivalent to chi-square hypothesis testing of normalized innovation under the independent Gaussian assumption. After the measurement–track association, the remaining measurements move to the initialization step in Section 2.2. The state and covariance of the tracks for which the measurements are not associated are merely the predictions as

$${\widehat{\mathit{x}}}_j^t\left(k|k\right)={\widehat{\mathit{x}}}_j^t\left(k|k-1\right),$$

(18)

$$P_j^t\left(k|k\right)=P_j^t\left(k|k-1\right).$$

(19)

2.3.3. Track Update

At last, the state and covariance of target t at frame k are updated as

$${\widehat{\mathit{x}}}^t\left(k|k\right)=\sum _{j=1}^M{\widehat{\mathit{x}}}_j^t\left(k|k\right){\mu }_j^t\left(k\right),$$

(20)

$$P^t\left(k|k\right)=\sum _{j=1}^M{\mu }_j^t\left(k\right)\left\{P_j^t\left(k|k\right)+\left[{\widehat{\mathit{x}}}_j^t\left(k|k\right)-{\widehat{\mathit{x}}}^t\left(k|k\right)\right]{\left[{\widehat{\mathit{x}}}_j^t\left(k|k\right)-{\widehat{\mathit{x}}}^t\left(k|k\right)\right]}^T\right\}.$$

(21)

The mode probability ${\mu }_j^t\left(k\right)$ in the equations above is

$${\mu }_j^t\left(k\right)={\displaystyle \frac{{\mathsf{\Lambda }}_j^t\left(k\right)\sum _{i=1}^M{p}_{ij}{\mu }_i^t\left(k-1\right)}{\sum _{j=1}^M{\mathsf{\Lambda }}_j^t\left(k\right)\sum _{i=1}^M{p}_{ij}{\mu }_i^t\left(k-1\right)}},$$

(22)

$${\mathsf{\Lambda }}_j^t\left(k\right)=N\left(\mathbf{0};{\mathit{\nu }}_{{\widehat{m}}_j^t\left(k\right)j}^t\left(k\right),S_j^t\left(k\right)\right),$$

(23)

where N is the Gaussian probability distribution function. It should be noted that the likelihood function in Equation (23) is assumed to be constant if no measurement is associated, and the mode probability becomes

$${\mu }_j^t\left(k\right)={\sum }_{i=1}^M{p}_{ij}{\mu }_i^t\left(k-1\right).$$

(24)

2.4. Track–Track Association at the Same Time (Track Association and Fusion)

A track fusion method was developed to associate redundant tracks generated in a multi-sensor environment [43]. The same technology has been applied to reduce redundant tracks in cases of a single sensor [38]. It was further improved with the directional gating in Ref. [39] and track selection process in Ref. [40]. The detailed procedures for the track association and fusion can be found in Ref. [40]. This track association and fusion is proven to work well with the nearest-neighbor measurement–track association scheme by fusion of redundant tracks generated by multiple associations between a single measurement and multiple tracks [35].

2.5. Track–Track Association at Different Times (Track Segment Association)

In the previous subsection, track fusion occurs on tracks that exist simultaneously. However, TSA occurs on tracks that exist at separate times. TSA consists of the selection of candidate track pairs, backward filtering, association testing, and an assignment rule.

2.5.1. Candidate Track Pairs

Any old and young tracks become a candidate track pair with the following conditions: (1) the old track has already been terminated and is valid, that is, the update number with measurements is larger than or equal to the threshold; (2) the young track has not been terminated, and the update number with measurements is less than the threshold and larger than or equal to half of it; (3) the young track start frame minus 1 is larger than the old track end frame considering the two-frame differencing initialization; and (4) the gap interval between the old track end frame and the young track start is less than or equal to the threshold.

Figure 3 illustrates two old and three young tracks. The maximum number of consecutive frames without measurements before track termination is set to two frames and the threshold is set to four frames. Tracks O₁ and O₂ can be old tracks from Frames 7 and 8, respectively. Tracks Y₁, Y₂, and Y₃ can be young tracks for Frames 8–9, 7–8, and 11, respectively. Therefore, the candidate track pair is only {O₁, Y₂} at Frame 7. At Frame 8, the candidate track pairs are {O₁, Y₁}, {O₁, Y₂}, and {O₂, Y₁}; {O₂, Y₂} cannot be a candidate because they are too close to each other. At Frame 9, the candidate track pairs are {O₁, Y₁} and {O₂, Y₁}. At Frame 10, no track pair is found. At Frame 11, only the track pair {O₂, Y₃} is a candidate track pair; {O₁, Y₃} cannot be a candidate because the gap interval between two tracks is longer than the threshold.

Figure 3. Illustration of old and young candidate track pairs.

2.5.2. Backward Filtering

Once a candidate track pair is selected from the current frame, the backward filtering in the young track starts from the current frame to the end frame of the old track. The backward filtering is divided into two types: backward updates and backward predictions. Backward updates are performed on frames where the measurements are already associated, while backward predictions are performed elsewhere. In Figure 3, there are two backward updates at Frames 10 and 9 and four backward predictions from Frames 8 to 5 when a candidate track pair {O₂, Y₃} is considered and Frame 11 is the current frame.

The backward IMM filtering is identical to the forward filtering, except that the sampling time is negative and already associated measurements during the forward filtering are reused. The multi-mode interaction is the first step of the backward filtering as

$${\widehat{\mathit{x}}}_{0j}^s\left(k|k\right)={\sum }_{i=1}^M{\widehat{\mathit{x}}}_i^s\left(k|k\right){\mu }_{i|j}^s(k|k), s=1,\dots ,N_y(k)$$

(25)

$$P_{0j}^s\left(k|k\right)=\sum _{i=1}^M\left\{\begin{array}{c}{\mu }_{i|j}^s\left(k|k\right)\left\{\begin{array}{c}{P}_i^s\left(k|k\right)+\\ \left[{\widehat{\mathit{x}}}_i^s\left(k|k\right)-{\widehat{\mathit{x}}}_{0j}^s\left(k|k\right)\right]{\left[{\widehat{\mathit{x}}}_i^s\left(k|k\right)-{\widehat{\mathit{x}}}_{0j}^s\left(k|k\right)\right]}^T\end{array}\right\}\end{array}\right\},$$

(26)

$${\mu }_{i|j}^s\left(k|k\right)={\displaystyle \frac{p_{ij}{\mu }_i^s(k)}{\sum _{i=1}^M{p}_{ij}{\mu }_i^s(k)}}, i,j=1,\dots ,M,$$

(27)

where $N_y\left(k\right)$ is the number of young tracks in candidate track pairs at frame k. It should be noted that the superscript s implies a young track. Mode-matched Kalman filtering is performed individually in the backward direction as

$${\widehat{\mathit{x}}}_j^s\left(k-1|k-1\right)={\widehat{\mathit{x}}}_j^s\left(k-1|k\right)+W_j^s\left(k-1\right){\mathit{\nu }}_{{\widehat{m}}_j^s\left(k-1\right)j}^s\left(k-1\right),$$

(28)

$$P_j^s(k-1|k-1)=P_j^s(k-1|k)-W_j^s(k-1)S_j^s(k-1)W_j^s(k-1{)}^T,$$

(29)

where ${\widehat{\mathit{x}}}_j^s(k-1|k)$ and $P_j^s(k-1|k)$, respectively, are obtained as

$${\widehat{\mathit{x}}}_j^s\left(k-1|k\right)=F\left(-\Delta \right){\widehat{\mathit{x}}}_{0j}^s\left(k|k\right),$$

(30)

$$P_j^s (k-1|k)=F(-\tau )P_{0j}^s(k|k){F\left(-\tau \right)}^T+q\left(-\tau \right)Q_{j}q(-\tau {)}^T,$$

(31)

and $W_j^s(k-1)$ and $S_j^s(k-1)$ are, respectively, obtained as

$$W_j^s(k-1)=P_j^s\left(k-1|k\right)H^T{S}_j^s(k-1{)}^{-1},$$

(32)

$$S_j^s\left(k-1\right)=H{P}_j^s\left(k-1|k\right)H^T+R.$$

(33)

The measurement residue in Equation (28) is

$${\mathit{\nu }}_{{\widehat{m}}_j^s\left(k-1\right)j}^s\left(k-1\right)={\mathit{z}}_{{\widehat{m}}_j^s\left(k-1\right)}\left(k-1\right)-H{\widehat{\mathit{x}}}_j^s\left(k-1|k\right),$$

(34)

where ${\mathit{z}}_{{\widehat{m}}_j^s\left(k-1\right)}\left(k-1\right)$ is the associated measurement with the j-th mode filter of track s at frame $k-1$ during the forward filtering. If no measurement was associated with track s, the state and covariance are, respectively, as

$${\widehat{\mathit{x}}}_j^s\left(k-1|k-1\right)={\widehat{\mathit{x}}}_j^s\left(k-1|k\right),$$

(35)

$$P_j^s\left(k-1|k-1\right)=P_j^s\left(k-1|k\right).$$

(36)

Finally, the state and covariance are updated as

$${\widehat{\mathit{x}}}^s\left(k-1|k-1\right)={\sum }_{j=1}^M{\widehat{\mathit{x}}}_j^s\left(k-1|k-1\right){\mu }_j^s\left(k-1\right),$$

(37)

$$\begin{gathered} P^s\left(k-1|k-1\right)={\sum }_{j=1}^M{\mu }_j^s\left(k-1\right)\left\{P_j^s\left(k-1|k-1\right)+\left[{\widehat{\mathit{x}}}_j^s\left(k-1|k-1\right)- \\ {\widehat{\mathit{x}}}^s\left(k-1|k-1\right)\right]{\left[{\widehat{\mathit{x}}}_j^s\left(k-1|k-1\right)-{\widehat{\mathit{x}}}^s\left(k-1|k-1\right)\right]}^T\right\}. \end{gathered}$$

(38)

The mode probability ${\mu }_j^s\left(k-1\right)$ is obtained in the same manner as in Equations (22)–(24).

2.5.3. Association Testing and Assignment Rule

Association testing is performed only on the candidate track pairs. The testing includes statistical distance testing and Euclidean distance testing between the forward and backward estimates of the state vector at frame $k_e^t$, which is the end frame of the old track t, as

$${\left({\widehat{\mathit{x}}}^t\left(k_e^t|k_e^t\right)-{\widehat{\mathit{x}}}^s\left(k_e^t|k_e^t\right)\right)}^T{\left[P^t\left(k_e^t|k_e^t\right)+P^s\left(k_e^t|k_e^t\right)\right]}^{-1}\left({\widehat{\mathit{x}}}^t\left(k_e^t|k_e^t\right)-{\widehat{\mathit{x}}}^s\left(k_e^t|k_e^t\right)\right)\le {\gamma }_s,$$

(39)

$$‖{\widehat{\mathit{x}}}^t\left(k_e^t|k_e^t\right)-{\widehat{\mathit{x}}}^s\left(k_e^t|k_e^t\right)‖\le {\gamma }_u,\hspace{1em}\hspace{1em}t=1,\dots ,N_o\left(k\right), s=1,\dots ,N_y\left(k\right),$$

(40)

where $N_o\left(k\right)$ is the number old tracks in the candidate track pairs at frame k. It should be noted that in this subsection, the superscript t denotes an older track. Equation (39) is the chi-square hypothesis testing since the forward and backward estimates are assumed to have independent Gaussian errors [41]. If Equations (39) and (40) are satisfied, the cost function for the assignment is identical to the chi-square statistic as follows, and if not satisfied, it becomes infinite:

$$c_{ts}\left(k_e^t\right)={\left({\widehat{\mathit{x}}}^t\left(k_e^t|k_e^t\right)-{\widehat{\mathit{x}}}^s\left(k_e^t|k_e^t\right)\right)}^T{\left[P^t\left(k_e^t|k_e^t\right)+P^s\left(k_e^t|k_e^t\right)\right]}^{-1}\left({\widehat{\mathit{x}}}^t\left(k_e^t|k_e^t\right)-{\widehat{\mathit{x}}}^s\left(k_e^t|k_e^t\right)\right).$$

(41)

The cost function is also infinite if a track pair {O_t, Y_s} is not a selected candidate pair. The assignment rule determines how to match old and young tracks in a 1:1 way. The assignment minimizes the summation of the cost functions as follows:

$${\widehat{a}}_{ts}\left(k_e^t\right)=\underset{a=\mathrm{0,1}}{\min }{\sum }_{i=1}^{N_o\left(k\right)}{\sum }_{s=1}^{N_y\left(k\right)}{a_{ts}\left(k_e^t\right)c}_{ts}\left(k_e^t\right),$$

(42)

such that

$${\sum }_{t=1}^{N_o\left(k\right)}{a}_{ts}\left(k_e^t\right)=1, s=1,\dots ,N_y\left(k\right),$$

(43)

$${\sum }_{s=1}^{N_y\left(k\right)}{a}_{ts}\left(k_e^t\right)=1, t=1,\dots ,N_o\left(k\right),$$

(44)

where $a_{ts}\left(k\right)$ is either 1 or 0, indicating whether it is assigned or not. During TSA, the young tracks that are assigned to the old tracks are terminated, and the corresponding old tracks that have already been terminated become on-going tracks and contain traces of the young tracks, ${\widehat{\mathit{x}}}^s\left(k_e^t|k_e^t\right), {\widehat{\mathit{x}}}^s\left(k_e^t+1|k_e^t+1\right), \dots ,{\widehat{\mathit{x}}}^s\left(k|k\right).$ Figure 4 illustrates the assignment between old and young tracks. No connection implies that the cost function is infinite. If the cost function is finite, old and young tracks are assigned 1:1. Three assigned track pairs, {O₁, Y₂}, {O₂, ${\mathrm{Y}}_{N_y\left(k\right)}$}, and {${\mathrm{O}}_{N_o\left(k\right)}$, Y₁}, are indicated by bold lines in Figure 4.

Figure 4. Illustration of assignment between old and young tracks.

2.6. Track Termination with Validation Testing

Three criteria for track termination are as follows: (1) a track is terminated when the number of consecutive frames without measurements exceeds the maximum number. (2) A potentially terminated track is terminated if it fails to be fused during track association and fusion [40]. (3) The young track assigned to the old track is terminated during TSA. The validity of a terminated track is determined by a minimum number of updates through measurements. Tracks shorter than this number are removed as invalid tracks.

3. Results

3.1. Video Description and Thermal Object Detection

A DJI Inspire 2 drone was equipped with an infrared thermal imaging camera, a FILR Vue Pro R640 (focal length = 19 mm, field of view (FOV) = 32° × 26°, pixel pitch = 17 μm, image resolution = 640 × 512 pixels, frame rate = 30 framer per second (fps)). The thermal imaging camera was operated at a 7.5–13.5 μm spectral bandwidth. Two videos were captured with the thermal camera on a winter night for 120 s. In both videos, three hikers were simulated in distress in the mountains. The drone flew slowly or quickly in various directions and altitudes, and the drone’s viewpoint was arbitrary. Both videos are identical to the videos in Ref. [35].

Figure 5a,b show the 1st, 401st, 801st, 1201st, 1601st, 2001st, 2401st, 2801st, 3201st, and 3601st frames of Videos 1 and 2, respectively. The target ID is shown near the target, on the right or at the bottom. The corresponding visible-light images appear completely black.

Figure 5. Thermal frames of (a) Video 1 and (b) Video 2.

The YOLOv5x object detection results, bounding box, and confidence level are shown in Figure 6 for the same frames in Figure 5. A total of 197 images with 548 training instances were trained on the YOLOv5x pre-trained detection model; the detailed procedure for training is referred to Ref. [35]. Since this study aims to elevate the performance of the multiple-target tracker on the basis of the observation, the same YOLOv5x detection results as in the previous study [35] were used for target tracking.

Figure 6. YOLOv5x detection results of (a) Video 1 and (b) Video 2.

Figure 7 shows the centroids of the YOLOv5x bounding boxes generated over 1801 frames (2 min). These are position measurements input to the target tracker. In Video 1, all the targets moved across the entire image area, while in Video 2, the target paths occupy less than a quarter of the image area; the targets in Video 2 are more congested. The recalls for Videos 1 and 2 are 0.975 and 0.895, respectively. Since there are no false alarms, the precision for both videos is 1. Video 1 has one additional detection for 18 frames but it is not included in the recall and precision calculations because it is a duplicate detection of the same object.

Figure 7. Centroids of YOLOv5x detections for 1801 frames of (a) Video 1 and (b) Video 2.

3.2. Thermal Target Tracking

3.2.1. System Configuration

Table 1 shows the parameter values. The frame interval (sampling time) is set to 0.067 (=1/15) s since every two frames were processed. The pixel-to-coordinate scaling ratio for Video 1 is 0.04 m/pixel, and for Video 2 it is 0.05 m/pixel. The Kalman filter was applied to Video 1, and a two-mode IMM filter was applied to Video 2. The one-mode IMM filter is identical with the Kalman filter. For Video 1, both ${\sigma }_{jx}$ and ${\sigma }_{jy}$ are set to 2.5 m/s², and they are 0.01 m/s² and 1 m/s² for Video 2. Both $r_x$ and $r_y$ are set to 0.5 m for Video 1 and 0.15 m for Video 2. Other parameters were set identically for both videos, except for the maximum established target speed, which is set to 12 m/s and 10 m/s for Videos 1 and 2, respectively. The maximum initial target speed was set to 3 m/s. The gate threshold, ${\gamma }_m$ in Equation (17), was set to 4. The gate threshold, ${\gamma }_t$ in Ref. [34], for the track association and fusion is set to 10, while the angular threshold is set to 90°. Since 90° is the maximum possible angle, the directional gating was not actually applied. The minimum update numbers with the measurements of the old tracks and young tracks are set to 30 and 15 frames, respectively. The maximum update number with the measurements of the young tracks is set to 29 frames. The maximum gap interval between the old and young tracks is set to 30 frames. The gate threshold for TSA, ${\gamma }_s$ in Equation (39), is set to 10, but the maximum Euclidean distance, ${\gamma }_u$ in Equation (40), is set to 4 only for Video 2. The maximum number of consecutive frames without measurements for track termination is set to 19 frames. Tracks shorter than the minimum update number with measurements, 30, were removed as invalid. The gate thresholds, ${\gamma }_m, {\gamma }_t, {\gamma }_s$, were obtained from the chi-square distribution. The degrees of freedom are equal to the dimension of the measurement vector or state vector. They are the maximum statistical distance squared. The smaller threshold indicates a stricter association, which minimizes false associations. The parameters were determined when better results were produced with all the associations applied.

Table 1. Modeling and tracking parameters.

Configurations (Unit)			Video 1	Video 2
System Modeling	Frame Interval (s)		0.067
	Pixel-to-coordinate scaling ratio (m/pixel)		0.04	0.05
	Process noise std. ${\sigma }_{jx}$ = ${\sigma }_{jy} (m/s^2)$	j = 1	2.5	0.01
		j = 2	-	1
	Meas. noise std. $r_{x }$= $r_y(m)$		0.5	0.15
Track Initialization	Max. initial target speed, $V_{max} (m/s)$		3
Measurement Association	Gate threshold, ${\gamma }_m$		4 (86.5% region)
	Max. established target speed, $S_{max} (m/s)$		12	10
Track Association and Fusion	Gate threshold, ${\gamma }_t$		10 (95.6% region)
	Angular threshold, ${\theta }_t$ (degree)		90 (INF)
Track Segment Association	Min. update num. with meas. for an old track (frame)		30
	Min./Max. update num. with meas. for a young track (frame)		15/29
	Max. gap interval between tracks (frame)		30
	Gate threshold, ${\gamma }_s$		10 (95.6% region)
	Max. Euclidean distance, ${\gamma }_u$		INF	4
Track Termination	Max. consecutive frame num. without meas. (frame)		19
	Min. update num. with meas. for a valid track (frame)		30

3.2.2. Evaluation of Tracking Performance

Table 2 and Table 3 show the tracking results of Videos 1 and 2, respectively. The first columns of the tables, Case 1, show the results when all the associations were applied, that is, all A–E blocks in Figure 2. The second columns, Case 2, show the results when TSA is not applied, that is, the result when A–B–C–E blocks are considered. The third columns, Case 3, show the results when only the measurement–track association is applied, that is, A–B–E blocks. The average TTL of Videos 1 and 2 is 0.998 and 0.931, respectively, in Case 1. Since the number of valid tracks in Videos 1 and 2 are three and six, respectively, the average MTL is 0.998 and 0.584 for Videos 1 and 2, respectively. The average TP of Videos 1 and 2 is 1 and 0.982, respectively. The average MTL is significantly improved by TSA, from 0.441 to 0.998 for Video 1 (99.8%) and from 0.167 to 0.584 for Video 2 (250%).

Table 2. Video 1 tracking results.

	Case 1	Case 2	Case 3
Num. of Track Asso. and Fusion	15	15	0
Num. of TSA	4	0	0
Num. of Valid Tracks	3	7	14
Avg. TTL	0.998	0.992	0.996
Avg. MTL	0.998	0.441	0.241
Avg. TP	1	1	1

Table 3. Video 2 tracking results.

	Case 1	Case 2	Case 3
Num. of Track Asso. and Fusion	11	11	0
Num. of TSA	15	0	0
Num. of Valid Tracks	6	18	25
Avg. TTL	0.931	0.905	0.768
Avg. MTL	0.584	0.167	0.099
Avg. TP	0.982	0.998	0.999

Figure 8 and Figure 9 show the tracks of Video 1 in random colors. In Figure 8, the tracks are displayed in the first frame as a background, and in Figure 9, they are shown on a white background. Figure 10 and Figure 11 show the tracks of Video 2 in the same way.

Figure 8. Video 1 tracks on the 1st frame background: (a) Case 1, (b) Case 2, (c) Case 3.

Figure 9. Video 1 tracks on the white background: (a) Case 1, (b) Case 2, (c) Case 3.

Figure 10. Video 2 tracks on the 1st frame background: (a) Case 1, (b) Case 2, (c) Case 3.

Figure 11. Video 2 tracks on the white background: (a) Case 1, (b) Case 2, (c) Case 3.

Six Supplementary files are movies in MP4 format, and they are available online. Supplementary Material Videos S1–S3 show the results of Cases 1–3 of Video 1, and Supplementary Material Videos S4–S6 show the results of Cases 1–3 of Video 2. The YOLOv5x detection results (bounding boxes and their centroids) are displayed in red squares. The valid tracks are displayed in blue circles. The track ID is also displayed, which is a number in the order in which the tracks were initialized.

3.2.3. TSA Analysis

The TSA occurred 4 and 15 times in Videos 1 and 2, respectively. No false TSA was found among them. The results of the four TSAs that occurred in Video 1 are shown in Table 4: the track IDs where the TSAs occurred, the number of backward updates and backward predictions, the statistical distance squared in Ref. [39], and the Euclidean distance in Ref. [40]. The number of TSAs occurred in Track ID 1 is two, and the numbers in Track ID 2 and 3 are both one. The backward predictions include all the backward estimates without measurements even after the young track started. Figure 12a shows the first track of Video 1 with two TSAs. Figure 12b,c show the magnified areas of the TSAs.

Table 4. TSA results of Video 1.

Track ID	Backward Updates	Backward Predictions	Statistical Distance Squared	Euclidean Distance
1	18	3	1.92	2.12
	15	20	0.79	0.46
2	15	6	1.99	2.44
3	18	3	1.37	2.10

Figure 12. First track of Video 1: (a) 2 TSAs, (b) 1st TSA, (c) 2nd TSA.

Table 5 shows the results of 15 TSAs in Video 2. The number of TSAs that occurred in Track IDs 1 to 4 are nine, three, one, and two, respectively. No TSAs occurred on Track IDs 5 and 6. Figure 13a shows the first track of Video 2 with nine TSAs. Figure 13b–f show the magnified areas of the TSAs. In Figure 12 and Figure 13, the red x-marks represent the measurements, blue dots represent the forward position estimates, and green squares and green triangles represent the backward updates and backward predictions, respectively. The red dots and red triangles represent the forward and backward estimates at the end frame of the old track, with the former being replaced by the latter in the track.

Table 5. TSA results of Video 2.

Track ID	Backward Updates	Backward Predictions	Statistical Distance Squared	Euclidean Distance
1	20	1	0.034	0.534
	18	3	0.053	0.550
	15	7	0.163	3.262
	20	1	0.011	0.442
	20	1	0.010	0.226
	20	1	0.007	0.375
	20	1	0.014	0.531
	19	2	0.009	0.528
	15	6	0.007	0.164
2	20	1	0.074	0.503
	20	1	0.034	0.233
	16	5	0.217	2.309
3	20	1	0.045	0.421
4	15	19	0.110	0.826
	15	23	0.017	1.102

Figure 13. First track of Video 2: (a) 9 TSAs, (b) 1st, 2nd TSAs, (c) 3rd TSA, (d) 4th TSA, (e) 5th–8th TSAs, (f) 9th TSA, (x: measurement, o: forward estimation, □: backward update, △: backward prediction).

As shown in Table 4 and Table 5, both the statistical distance squared and the Euclidean distance increased when the track was broken due to the drone’s sudden movement. Only the chi-square hypothesis testing was applied to Video 1, and both the chi-square hypothesis and Euclidean distance testing were applied to Video 2, where the target’s movement is concentrated in a relatively small area.

4. Discussion

Multi-rotor drone-based thermal target detection and tracking is effective for SAR missions. In the videos, the warm objects were captured without ambient light. They are often obscured by other hikers or natural objects in the mountains. The drone was remotely controlled manually and hovered or moved freely.

The contributions of this paper are focused on target tracking and demonstrate that (1) forward filtering is further improved with the IMM approach in block B, and (2) the whole process of the TSA is developed and clarified, which is block C in Figure 2.

YOLO detection is known to stand out for its speed, simplicity, and accuracy. The YOLO model is a pre-trained model that has already been trained on a large dataset and aims to detect multiple classes in visible-light images [42]. A relatively small number of 197 images with 548 training instances were trained mainly for the following reasons: (1) the object detection is limited to one class. The ability to distinguish between multiple classes is not required, but there is likely to be a large variance between the class features and the background features. Better recognition results are expected when the between-class variance is large and the within-class variance is small [44]. (2) The distinguishing features of the class are a few attributes, such as brightness and object contours, because warm objects appear brighter than their surroundings in thermal images. It is well known that the smaller the dimension of the feature, the less training data are required [44]. By training more images of different shapes, it is expected to be able to better detect different poses, such as people sitting or people in groups. Another consideration for training instances is that the rectangular bounding box inevitably includes objects that are not of interest. It seems important to minimize such regions while still including the object of interest.

After track association and fusion, the valid track number in Video 1 was reduced from 14 to 7, and in Video 2, from 25 to 18. Four track breakages were found in Video 1 without TSA, three of which were caused by sudden movements of the drone and one of which was caused by missed detection. In Video 2 without TSA, 15 track breakages occurred, which were caused by measurement–track association failures, missed detections, or sudden movements of the drone.

With TSA, all the track breakages were remedied in Video 1, and the valid track number was reduced from seven to three. In Video 2, the TSA reduced the number of track breakages from 15 to 3 and the valid track number from 18 to 6. The number of tracks is the same as the target number in Video 1; however, three more valid tracks were left in Video 2. These three unresolved breakages in Video 2 were caused by long-term missing detection; a target that is missing for a certain time is recognized as another target, resulting in a track breakage. TSA cannot remedy the breakage if the gap between the two tracks is too long.

Another benefit of TSA is that it can be useful in a multiple-target environment when the strict measurement–track association is applied to avoid wrong measurement–track associations. Strict association leads to more broken tracks, but TSA can resolve these breakages.

The conditions for candidate track pairs require that the old track is valid and the young track is not long enough to be valid. The accuracy of backward filtering may be reduced if the life length of the young tracks is too long or too short. The gap interval cannot be too long because if the backward prediction lasts longer, its error becomes larger.

Figure 1 shows a separate detection-tracking approach. In this scheme, each part can be developed separately; for example, a higher or lighter version of YOLO can be adopted for object detection. The target tracker can utilize other types of measurements, such as range, range rate, and azimuth, obtained from radar [41]. Another advantage of this scheme is that detection and tracking can be implemented together or separately. Only object detection can be implemented onboard, and the position measurements are transmitted to the server on the ground, and target tracking is performed on the server. All the processing can be performed onboard the drone or on the ground, with each case allowing the drone to operate in standalone mode or without requiring onboard implementation.

5. Conclusions

In this paper, multi-rotor drone-based thermal target tracking with TSA was studied for the purpose of SAR missions. Thermal imaging is shown to be beneficial in detecting warm objects in remote areas. Three associations were established for target tracking with the IMM approach, showing robust performance. TSA can effectively reduce track breakages, even when the platform moves suddenly. This target-tracking scheme after object detection has the advantage of being developed and implemented independently, and the target tracker can be applied to various observations.

One advantage of drones is their swarm cooperation [45], which allows them to search vast areas in a timely manner. Target tracking using swarm drones and applying various multi-model filters [46] remain objectives for studies in the future.