3D-SiamMask: Vision-Based Multi-Rotor Aerial-Vehicle Tracking for a Moving Object

Abstract

This paper aims to develop a multi-rotor-based visual tracker for a specified moving object. Visual object-tracking algorithms for multi-rotors are challenging due to multiple issues such as occlusion, quick camera motion, and out-of-view scenarios. Hence, algorithmic changes are required for dealing with images or video sequences obtained by multi-rotors. Therefore, we propose two approaches: a generic object tracker and a class-specific tracker. Both tracking settings require the object bounding box to be selected in the first frame. As part of the later steps, the object tracker uses the updated template set and the calibrated RGBD sensor data as inputs to track the target object using a Siamese network and a machine-learning model for depth estimation. The class-specific tracker is quite similar to the generic object tracker but has an additional auxiliary object classifier. The experimental study and validation were carried out in a robot simulation environment. The simulation environment was designed to serve multiple case scenarios using Gazebo. According to the experiment results, the class-specific object tracker performed better than the generic object tracker in terms of stability and accuracy. Experiments show that the proposed generic tracker achieves promising results on three challenging datasets. Our tracker runs at approximately 36 fps on GPU.

1. Introduction

Multi-rotor aerial vehicles equipped with cameras are capable of covering large areas dynamically, which is why they are valuable for many applications, such as tracking moving objects. When objects in the scene are isolated and distinct from the background, tracking is geneally an achievable task. However, despite decades of research, visual tracking remains a challenging problem in real-world applications due to factors such as partial occlusion, quick and abrupt object motion, lighting changes, and substantial variations in the view point and pose of the target object.

Single-object tracking (SOT) is a fundamental computer vision problem that has several applications, including autonomous vehicles [1,2,3] and surveillance systems [4]. The objective of SOT is to track a defined target within a video sequence using its initial state (position and appearance). SOT approaches [5,6,7,8,9] that utilize the Siamese paradigm are commonly used in 2D and 3D SOT as the Siamese paradigm provides a compromise between performance and speed. Using an appearance-matching strategy, the Siamese model tracks the target in the candidate region using features from the target template and the search area retrieved by a shared backbone.

Since there is no pre-trained object detector involved, single-object trackers are frequently referred to as “generic object trackers” or “model-free trackers” [10,11,12]. From a learning standpoint, model-free visual object tracking is a difficult problem given that there is only one instance of the target in the first frame, and the tracker has to learn the target appearance in the following frames. In our proposed system, we tried to avoid this issue by replacing the template frame with a template set which remains updated based on proposed simple heuristic rules.

The fundamental distinction between detection and tracking is the application of dynamics. During detection, the object is detected independently in each frame. In tracking, we predict the new location of the object in the next frame using estimated dynamics, and then we detect that object. Based on the measurements, we update the estimated dynamics and iterate.

Even while the results of the appearance matching for 3D SOT on the KITTI dataset are decent [13], Zheng et al. [7] noted that KITTI has the following characteristics:

(i)

The subject moves very slightly between two successive frames, preventing a significant change in appearance;

(ii)

There are few or no distractions in the target’s environment.

The aforementioned qualities are not applicable in natural situations. As objects move quickly or the hardware can only handle a limited frame sampling rate, self-occlusion may cause dramatic changes in successive LiDAR views. Additionally, negative samples increase dramatically in scenes where heavy traffic is present. Even humans may find it difficult to identify a target based on its appearance in these situations.

As shown in Figure 1, ourfocus is to track single objects by utilizing only a single RGBD camera rather than the use of other sensors, such as LiDARs, for multiple reasons, such as the use of cameras is cheaper than LiDARs; cameras are lighter in weight than LiDARs, which is very important in the case of multi-rotors; and both sensors cannot see through obstacles, such as heavy rain, snow, and fog.

Based on the above observations, we propose to tackle 3D SOT from a different perspective using the simplicity of a Siamese paradigm [5] with the power of robust monocular depth estimation [14]. The main contributions of the proposed system are as follows:

• Utilize a monocular-based hybrid-depth estimation technique to overcome the limitations of sensor depth maps.
• Employ the template set to cover the target object in a variety of poses over time.
• Introduce the auxiliary object classifier to improve the overall performance of the visual tracker.

2. Related Work

In recent years, visual object tracking has been a prominent research field [15,16]. Due to its necessity and numerous applications, such as autonomous vehicles, robotics, and surveillance, more tracking algorithms are introduced each year [15,16]. Here, we examine the current advancements in the field of visual tracking. To do that, we compare various approaches in terms of the main technique (estimation-based tracking, feature-based tracking, and learning-based tracking); in our comparison, we will consider the advantages, disadvantages, and other details such as the number of tracked objects (single or multiple objects), and whether the tracker works in 2D or 3D space, etc. By the end of this section, we will have outlined a comparison between state-of-the-art deep-learning single-object Trackers.

There are many challenges in object tracking [17,18], such as background clutter, scale variation, occlusion, and fast motion, etc. Figure 2 shows some examples from the OTB dataset [18], with each example depicting one of the main challenges. In recent years, many approaches have been proposed to tackle each of the problems; here, we will discuss the state-of-the-art algorithms based on the used tracking techniques.

Estimation-based tracking: To use the Kalman filter [19] in object tracking [20], a dynamic model of the target movement should be designed. A Kalman filter could be used to calculate the position in the case of linear systems with Gaussian errors. For nonlinear dynamic models, other suitable methods are used, such as the extended Kalman filter [21].

Tracking can benefit from the main properties of the Kalman filter [19], which include:

• Predicting the future location of the object.
• Forecast correction based on current measurements.
• Noise reduction caused by incorrect diagnosis.

The vast majority of tracking problems are non-linear. As a result, particle filters have been investigated as a possible solution to such problems. The particle filter is a non-Gaussian-noise measurement model that employs a statistical calculation method known as recursive Monte Carlo. The main concept of the particle filter is to represent the distribution of a set of particles. Each particle is assigned a probability weight, which represents the probability of sampling that particle using the probability density function. One disadvantage of this method is that the particles with the highest probability are selected multiple times, which is overcome by resampling [22].

The JPDA multi-object tracker [23,24,25] is a tracker that can process numerous target detections from multiple sensors. To assign detections to each track, the tracker employs joint probabilistic data association. With a soft assignment, the tracker allows numerous detections to contribute to each track. Tracks are initialized, confirmed, corrected, predicted, and deleted by the tracker. The tracker receives detection reports from the object detector and sensor fusion as inputs. For each track, the state vector and the state estimate-error covariance matrix are estimated by the tracker. Each detection has at least one track associated to it. The tracker creates a new track if the detection cannot be assigned to an existing track.

Feature-based tracking: Using extracted attributes such as texture, color, and optical flow, this category of visual tracker identifies the most similar objects over the upcoming frames. Here, we discuss some of the most important algorithms associated with feature-based trackers. Lucas–Kanade template tracking [26] is one of the first and most popular trackers, originally designed to be able to monitor the optical flow of the image pixels and to determine how they move over time. The problem with this approach was the inaccurate assumption of a constant flow (pure translation) for all pixels in a bigger window for long periods of time. Zhao et al. [27] used local binary patterns (LBP) to describe moving objects and also used a Kalman filter for target tracking. Meanwhile, Zhao et al. [27] inherited the advantages of both LBP and KF, such as the computational simplicity (LBP and KF), the good performance (LBP and KF), the high discriminative power (LBP), and the invariance to the changes in grascale (LBP). On the other hand, this model also inherited the disadvantages of both of them, such as being not invariant to rotations (LBP), and the state variables being normally distributed (KF). Lastly, in terms of time and space, the complexity of computation increases exponentially with the number of neighbors (LBP).

One of the proposed solutions is to compare features (SIFT and HoG) [28,29] instead of pixels, which are hopefully invariant to changes in scale and rotation. Although these algorithms have been used for a long-time, that does not contradict the fact that they have many issues such as highly dimensional feature descriptors (SIFT and HoG), high computation requirements (SIFT and HoG), and low matching accuracy at large angles of rotation and view, which makes them not as reliable as the current deep-learning models.

Learning-based tracking: Discriminative, generative, and reinforcement learning are the three main paradigms used in learning-based tracking. We will review some of the main deep visual object-tracking (VOT) approaches. With the advancement of deep learning, deep features have been used in object trackers. SiamFC [8] was one of the models that used fully convolutional Siamese networks to solve the problem of object tracking. Although SiamFC has many advantages, it has two main disadvantages [30]. First, it only generates the final response scores using the features from the final layer. These high-level features are resistant to noise, but they lack specific target information, making these features insufficient for discrimination when the distractor falls into the same category as the target. Second, the SiamFC training method ensures that each patch in the search region contributes equally to the final response score map. Therefore, regardless of where a distractor appears in the search region, it may produce a high response score and lead the tracking to fail. Later, in SiamRPN [9], the solution was developed using the region proposal network (RPN), which was originally introduced in Faster R-CNN [31] to solve the object detection problem. Therefore, instead of applying a sliding approach on the features, a bunch of proposals are made and then the model both classifies and regresses these proposals using RPN, which has two branches. The first branch is to classify each proposal, whether or not it looks like the template object, and the second branch regresses an offset for the proposed box.DaSiamRPN [6] focused on learning distractor-aware Siamese networks for precise and long-term tracking. To that end, the features of conventional Siamese trackers were initially examined. Zhu et al. [6] recognized that imbalanced training data reduces the separability of the acquired features. To manage this distribution and direct the model’s attention to the semantic distractions, a powerful sampling approach is used during the off-line training phase. Zhu et al. [6] designed a distractor-aware module to perform incremental learning during inference so that the generic embedding can be successfully transferred to the current video domain. Zhu et al. [6] also introduced a straightforward yet efficient local-to-global search region approach for long-term tracking. ATOM (Accurate Tracking by Overlap Maximization) [32] proposed a novel tracking architecture with explicit components for target estimation and classification. The estimation component is trained offline on large-scale datasets to predict the IoU overlap between the target and a bounding-box estimate. The classification component consists of a two-layer fully convolutional network head and is trained online using a dedicated optimization approach. In SiamMask [5], the authors point out that additional information can be encoded in ROW (response of a candidate window) produced by a fully convolutional Siamese network, to generate a pixel-wise binary map. Transformer meets tracker [33]: In contrast to how the transformer is often used in natural language processing applications, Wang et al. [33] redesigned the encoder and decoder of the transformer into two parallel branches within the Siamese-like tracking pipelines. Through attention-based feature reinforcement, the transformer encoder promotes the target templates, which benefits the construction of better tracking models. The object search procedure is made easier by the transformer decoder, which propagates tracking cues from earlier templates to the current frame.

Since our proposed approach is meant to track moving objects in 3D space, we will discuss object localization in 3D space. Indeed, the knowledge of the structure of the 3D environment and the motion of dynamic objects is essential for autonomous navigation [34]. This importance comes from the fact that the 3D structure implicitly depicts the agent’s relative position, and it is also used to help with high-level scene understanding tasks such as detection and segmentation [35]. Recent breakthroughs in deep neural networks (DNNs) have sparked a surge in interest in monocular depth prediction [14,35] and stereo image depth prediction [36], as well as optical flow estimation [37].

Now we will discuss some of the state-of-the-art 3D visual object trackers that operate in three-dimensional space. Eye in the sky [38] is a novel framework for drone-based tracking and 3D object localization systems. It combines CNN-based object detection, multi-object tracking, ground plane estimation, and, finally, 3D localization of the ground targets. In our proposed framework, we use MiDaS [14], besides the depth map generated by the depth sensor, to generate the hybrid-depth map, which is explained in the Section 3. Unlike the relative depth map generated by the MiDaS algorithm [14], the proposed hybrid-depth map estimates the depth measured in meters.

TrackletNet tracker (TNT) [39] is a multi-object tracking method based on a tracklet graph model, incorporating tracklet vertex creation with epipolar geometry and connectivity edge measurement using a multi-scale algorithm, Tracklet-Net.

This description implies several important characteristics of the task at hand:

• The watermark must include all the required information while remaining placeable and noticeable, even on tiny photos. The created watermark must be compact while still having the ability to hold sufficient information.
• The watermark must be invisible to the human eye to prevent easy tampering (and, preferably, to basic image parsing tools). If the malefactor is unaware of the existence of the watermark, they may not even attempt to remove or disable it.

Beyond 3D Siamese tracking [7]: The authors of this paper also presented a motion-centric paradigm to handle 3D SOT from a new perspective. They suggested the matching-free two-stage tracker $M^2$-Track in line with this concept. At the first stage, $M^2$-Track uses motion transformation to localize the target across a series of frames. The target box is then refined at the second stage using motion-assisted shape completion.

While many 3D VOT studies focused on using LiDAR data as the essential data input to their systems [7,38,39,40,41], in our work, we tried to solve the visual object-tracking task using the RGBD data generated by the lightweight and the affordable sensor [42]. To do that, we relied on the SiamMask architecture [5] which originally works in 2D space. We introduced the hybrid-depth maps (explained in detail in the Section 3) to SiamMask to be able to track the objects in 3D environments. In addition, we replaced the template frame used in SiamMask [5] with the template set to give the model the ability to track the moving object from different points of view and without the need to build a 3D model for the moving object. Lastly, we used stereo image triangulation [43] to deproject the position into 3D space, which is followed by a 3D Kalman Filter [19] which helps to remove the measurement noise.

In

Table 1. Summary of the state-of-the-art DL trackers with different attributes.

Approach	Space	Backbone	Modules	Localization Method	Training Scheme	Input	FPS	Update	Redetection
$M^2$-Track/2022 [7]	3D	PointNet	-	binary classification, BBR	Cross-entropy + Huber-loss + Adam	Lidar	57	no-update	N
3D-SiamRPN/2021 [40]	3D	PointNet-++	-	binary classification, BBR	Focal-loss + Smooth L1-loss	Lidar	20.8	no-update	N
SiamGAT/2021 [44]	2D	Google-LeNet	Inception-V3	binary classification, BBR	BCE-loss+IoU loss+SGD	RGB	70	no-update	N
TransT/2021 [45]	2D	ResNet-50	Block-4	binary classification, BBR	giou loss+ L1-loss + cross-entropy	RGB	50	no-update	N
P2B/20 [41]	3D	PointNet-++	-	binary classification, BBR	BCE-loss + Huber-loss + Adam	Lidar	40	non-linear	N
SiamR-CNN/2020 [46]	2D	ResNet-101-FPN	Block-2,3,4,5	binary classification, BBR	BCE-loss + Huber loss + Momentum	RGB	4.7	no-update	Y
PrDiMP/2020 [47]	2D	ResNet-18/50	Block-4	probabilistic density regression, IoU-prediction	KL-divergence + loss steepest descent method	RGB	30	non-linear	N
SiamMask/2019 [5]	2D	ResNet-50	Stage-1,2,3,4	binary classification, BBR, logistic loss	BCE-loss + SGD	RGB	55	no-update	N
ATOM/2019 [32]	2D	ResNet-18	Block-3,4	binary classification, BBR	MSE + Adam(R), L2-loss + Conjugate Gradient(C)	RGB	30	non-linear	N
SiamRPN++/2019 [48]	2D	ResNet-50	Block-3,4,5	binary classification, BBR	BCE-loss + L1-loss + SGD	RGB	35	no-update	N
SiamRPN/2018 [9]	2D	modified AlexNet	conv5	binary classification, BBR	Smooth L1-Loss + BCE-loss + SGD	RGB	200	no-update	N
DaSiam-RPN/2018 [6]	2D	modified AlexNet	conv5	binary classification, BBR	Smooth L1-Loss + BCE-loss + SGD	RGB	160	linear	Y
SiamFC/2017 [8]	2D	AlexNet	conv5	binary classification, scales searching	logistic loss+SGD	RGB	86	no-update	N

, we outlined a comparison between several state-of-the-art deep-learning trackers, along with their tracking framework, published year, dimensional space (2D/3D), backbone network (the feature-extraction network), modules, classification/regression methods (BBR: bounding-box regression. BCE loss: binary cross-entropy loss), training schemes, update schemes (no-update, linear-update, non-linear update), input data format (RGB, LiDAR), tracking speed, re-detection (yes/no, i.e., Y/N).

3. Methodology

The main goal of this work is to track the moving object from the perspective of the multi-rotor aerial vehicle, Figure 1, where the multi-rotor navigates through the environment while trying to follow the moving object using a real-time visual tracking approach. As shown in Figure 3, the multi-rotor keeps following the moving object until it reaches the location of the moving object and the moving object stops.

There are two sub-tasks in the tracking task: identifying the tracked object and estimating its state. The objective of the tracking mission is to automatically predict the state of the moving object in consecutive frames given its initial state. The proposed framework combines 2D SOT with monocular depth estimation to track moving objects in 3D space. Depth maps are used primarily to maintain awareness of the relevant depth information about target objects.

The proposed system basically has a stream of RGB frames and the depth-map inputs (Figure 4). The target object is tracked by the Siamese network which produces a mask, a bounding box, an object class (optional), and an RPN score [31] for the object. In order to estimate the corresponding point in 3D space, a relative depth map [14] is aligned with the depth map [42] to generate the hyprid-depth map. As a result of our experiments, the proposed hyprid-depth map was able to estimate the distance from objects at a distance of more than 20 m, which is four times the range of the currently used depth sensor, which can only estimate the distance from objects at a distance of 5 m [42].

Loss function. To train the classifier $f_{\beta }$, we used binary cross entropy as the loss function (1) and Adam as the optimizer.

$${\mathcal{L}}_{cls}=-y·log\widehat{y}+(1-y)·log(1-\widehat{y})$$

(1)

where $\widehat{y}$ is the scalar value in the model output, and y is the corresponding target value (0 or 1).

In our experiments, we augmented the architectures of SiamMask [5] and MiDaS [14], where each of them were trained separately:

$${\mathcal{L}}_{4B}={\lambda }_1·{\mathcal{L}}_{\mathrm{mask}}+{\lambda }_2·{\mathcal{L}}_{\mathrm{score}}+{\lambda }_3·{\mathcal{L}}_{\mathrm{box}}+{\lambda }_4·{\mathcal{L}}_{cls}$$

(2)

We refer the reader to [5,8,9] for ${\mathcal{L}}_{\mathrm{mask}}$, ${\mathcal{L}}_{\mathrm{score}}$, and ${\mathcal{L}}_{\mathrm{box}}$. We have not performed hyperparameter optimization for Equation (2) but we merely set ${\lambda }_1=32$, ${\lambda }_2={\lambda }_3=1$, like in [5], and ${\lambda }_4=1$.

For MiDaS, we refer to [14]; the used loss function is called the scale- and shift-invariant loss where the loss for a single sample is represented as

$${\mathcal{L}}_{ssi}\left(\widehat{\mathbf{d}},{\widehat{\mathbf{d}}}^{*}\right)=\frac{1}{2M}\sum _{i=1}^M\rho \left({\widehat{\mathbf{d}}}_i-{\widehat{\mathbf{d}}}_i^{*}\right),$$

(3)

where $\widehat{\mathbf{d}}$ and ${\widehat{\mathbf{d}}}^{*}$ are scaled and shifted versions of the predictions and ground truth, and $\rho$ defines the specific type of loss function.

Before estimating the position of the target object, we need to estimate the depth of the target object by using the hybrid-depth map $\mathbf{H}$. At each time step, $\mathbf{H}$ is calculated as follows

$$\mathbf{H}=\eta ·r·(\mathbf{NIRD}\ast \widehat{\mathbf{M}}\phantom{\mathbf{M}})+\mathbf{D}$$

(4)

where $\mathbf{H}$ is the hybrid-depth map which conveys the depth estimates in meters, $\eta$ is a scaling factor, $\mathbf{D}$ is the depth map generated by the sensor, and depth ratio r can be evaluated using following expression

$$r=\frac{\mathbf{D}}{\mathbf{NIRD}+ϵ}$$

(5)

where $ϵ$ is a small fraction to avoid division by zero, and $\mathbf{NIRD}$ is the normalized inverted relative depth map, which can be evaluated as

$$\mathbf{NIRD}=\frac{{\mathbf{R}}^{†}-{\mathbf{R}}_{\mu }^{†}}{{\mathbf{R}}_{\sigma }^{†}}$$

(6)

To evaluate it, we normalize the inverted relative-depth map ${\mathbf{R}}^{†}$ by subtracting the mean ${\mathbf{R}}_{\mu }^{†}$, then divide by the standard deviation ${\mathbf{R}}_{\sigma }^{†}$. Therefore,      

$${\mathbf{R}}^{†}=1-\mathbf{R}$$

(7)

Here, $\widehat{\mathbf{M}}\phantom{M}$ is the output of applying the XOR operation (denoted by ⊕) on the binary mask $\mathbf{M}$ with 1

$$\widehat{\mathbf{M}}\phantom{M}=\mathbf{M}\oplus 1$$

(8)

$\mathbf{M}$ is the mask for depth points that have values bigger than a small fraction $\gamma$, i.e.,

$$\mathbf{M}=\mathbf{D}>\gamma$$

(9)

Despite the fact that many tracking algorithms have been proposed for various tasks and object tracking has been studied for many years, it is still a challenging problem. There is no single tracking algorithm that can be used to accomplish all tasks and in different scenarios. In the case of SiamMask [9], the tracker showed that sometimes it can be distracted from the target object by other distractors or by similar objects in the scene. Working on solving such critical issues could help make multi-rotor-based tracking applications much safer and more reliable. For that purpose, we proposed an auxiliary object classifier and a mechanism for initiating and updating the template set. This mechanism is designed to cover the target object from different points of view from different distances.

At each time step, the proposed framework has two main inputs (the undistorted RGB and depth frames) and estimates the position of the moving object. At the initial step ($j=0$), the InitTracker initializes the tracker after adding the cropped template of the target object to the template set. The template set has a maximum size (l) and new candidates will be replace old templates over time. All the templates in the template set could be replaced except for the original template inserted at ($j=0$).

The proposed framework components are realised within Algorithm 1.

Algorithm 1 Object visual tracker

Input: Undistorted RGB & depth frames

Input: Undistorted RGB & depth frames

1: $S_0\leftarrow$ InitTracker(${\mathbf{RGB}}_0,{\mathbf{D}}_0$) 2: repeat 3:     ${\mathbf{RD}}_{i+1}\leftarrow$ EstimateRelativeDepthMap(${\mathbf{RGB}}_{i+1}$) 4:     ${\mathbf{HD}}_{i+1}\leftarrow$ GenerateHybridDepthMap(${\mathbf{D}}_{i+1},{\mathbf{RD}}_{i+1}$) 5:     ${\mathbf{S}}_{i+1}\leftarrow$ EstimateObjectState(${\mathbf{S}}_i$) 6:     if StateIsValid($S_{i+1}$) 7:         $<x_{i+1},y_{i+1},z_{i+1}>\leftarrow$ ApplyProjectionAndKF(${\mathbf{HD}}_{i+1},{\mathbf{S}}_{i+1}$) 8:     else 9:         ${\mathbf{S}}_{i+1}\leftarrow {\mathbf{S}}_i$ 10:         $<x_{i+1},y_{i+1},z_{i+1}>\leftarrow <x_i,y_i,z_i>$ 11:     end if 12: until end of sequence

Let us consider all the steps of Algorithm 1 in detail:

• EstimateRelativeDepthMap has the RGB image as an input to estimate the relative depth map using the robust monocular depth estimation network [14]; the predicted relative-depth map with the depth map produced by the sensor [42] will be the inputs for the next step.
• GerenateHybridDepthMap by applying Equation (4). To estimate the hybrid-depth map that has the depth information of the moving object in meters. The process of generating the hybrid-depth map is divided into 3 main steps, as follows

  (i)

  Preprocess the depth map by replacing all non-numeric values with zeros. The result is a 2D array $\mathbf{D}$ with values measured in meters.

  (ii)

  Generate from the relative depth map $\mathbf{R}$ a 2D array with values $\in [0,1]$.

  (iii)

  Apply Equation (4).

• EstimatingObjectState takes the state vector of the moving object $S_i$ as the input and returns the updated state as the output $S_{i+1}$. $S_i$ has the details of the target object such as the hybrid-depth maps and the RGB frames of the current and previous time steps $[{\mathbf{HD}}_i,{\mathbf{HD}}_{i+1},{\mathbf{RGB}}_i,{\mathbf{RGB}}_{i+1}]$. The position will be defined in 2D space as the center of the object $(x,y)$. The depth is $d=\mathbf{HD}[x,y]$.
  To update the template set there are conditions the candidate template needs to meet. These conditions are as follows
  • Class-specific tracker: the object should be classified by the auxiliary classifier as the desired class.
  • Both trackers: if the detected object satisfies the inequalities (3) below, then the candidate template will be inserted to the template set at $j=1$ after the original template frame and before the other templates in the template set:

    $$\begin{array}{c}\hfill |A_{mask}^{i+1}-A_{mask}^i|<{ϵ}_1\\ \hfill |A_{box}^{i+1}-A_{box}^i|<{ϵ}_2\end{array}$$

    where $A_{mask}$ and $A_{box}$ are the area for the generated mask and the bounding box, respectively. ${ϵ}_1$ and ${ϵ}_2$ are small fractions.
  Therefore, if the candidate template violates any of the corresponding conditions, the new candidate template will not be inserted to the template set. Before we explain how the template set is used, we need to know that the original template frame has index $j=0$ at every time step. The usage of the template set is as follows

  (i)

  Use the original template frame.

  (ii)

  The conditions (3) above must be satisfied and the objectness probability (RPN score) should be greater than a threshold $\ge \alpha$. Other than that, the template frame is skipped to the next most recent frame in the template set.

  (iii)

  Repeat (ii) if needed for n times (in our experiments $n=2$).

  (iv)

  Return the estimated object state.

• ApplyProjectionAndKF: A 3D Kalman filter [19] will remove measurement noise after a stereo image triangulation [43] deprojects the position into 3D space.

As shown in Figure 4, the hybrid-depth map is generated from both the MiDaS relative-depth map [14] and the Intel RealSense depth map [42]. The alignment of these two maps was explained in detail in Algorithm 1. Figure 5 shows the generated hyprid-depth map using the proposed approach.

Similarly to SiamMask [5], ResNet50 [49] is used until the final convolutional layer of the 4th stage as the backbone $f_{\theta }$ for both Siamese Network [5] branches. The main goal of $h_{\varphi }$ is to predict $w\times h$ binary masks (one for each RoW). Basically, $h_{\varphi }$ is a simple two-layer network with learnable parameters $\varphi$.

In the proposed system, a ResNet18 [49] was used to classify the object detected by the SiamRPN branches [9], to stabilize tracking and make the model less prone to distractors. We have two different settings to run in the proposed system:

• Generic moving-object tracker: in this case, the object will be classified using the Siamese tracker with the help of the frames in the template set.
• Class-specific moving-object tracker: in this case, the tracker will have an auxiliary object classifier which makes sure to avoid distractors and the possible foreground/background confusions by the RPN network.

The experiment was performed by training ResNet18 with a binary classification layer. Two datasets were utilized to train the binary classifier. The moving object class is represented by the Stanford Cars dataset (16,185 photos of 196 car classes) [50]. In contrast, the Describable Textures Dataset (DTD) [51] was applied to the non-moving object class. The DTD dataset contains 5640 photos sorted into 47 categories influenced by human sight.

4. Results

The trackers were tested in two simulation environments, Figure 6, to assess the performance in different cases. The GitHub repository includes a video with the experiment results. For visual tracker evaluation, the VOT (visual object tracking) benchmarks were used [52,53,54]; all of the benchmark datasets are annotated with rotated bounding boxes. The comparison over the three benchmarks was performed using the official VOT toolkit and the expected average overlap (EAO) [55] (EAO is a metric which takes into consideration the accuracy and the robustness of the visual tracker).

The tracker was tested in the simulation environment utilizing the Iris drone with the Intel Realsense Depth Camera [42]. The tracker was implemented using Python 3.6 and ROS1. We used a computer with a 2.6 GHz CPU, an NVIDIA GeForce RTX 2060 6 GB GPU, and 16 GB of RAM for the simulated environment. The simulated experiments were performed in a Gazebo software. The local planner [56] was used to generate the optimal control to maneuver the quad-rotor considering close-in obstacles and system dynamics. Code and results are available at: https://github.com/mhd-medfa/Single-Object-Tracker.

4.1. Experiment 1: Generic and Class-Specific Trackers

Firstly, this experiment was carried out in the basic simulation environment, Figure 6. We should mention here that the proposed framework has two tracking approaches:

• Generic tracker. Figure 7 shows how the tracker, using the Siamese tracker with the help of the template set and the hybrid depth map, was able to track the target object. The experiment showed that the tracker passed the moving object.
• Class-specific tracker. Figure 8 shows that by involving the auxiliary object classifier, the object tracker did not pass the moving object. The responsibility of this classifier is to stabilize the tracking process.

A primary drawback of the generic tracker is that the multi-rotor may pass the target object due to the fact that the Siamese tracker measures the similarity between the template frame and input frame, which makes the tracker detect the object even if only small part of the object is showing in the scene [5].This issue is handled in the class-specific mode, in which the object classifier classifies the RPN-proposed item at each time step, so that if the target object is closer than d[meters] or if the tracker could not recognize the target object (i.e., the auxiliary classifier classified the detected object into a different class of objects), then the multi-rotor will keep hovering at the same position until the conditions change (i.e., the moving object becomes farther than d[meters]). All that helps keep the target object in the scene without passing the tracked object, as shown in Figure 8.

4.2. Experiment 2: Different Case Scenarios

In this experiment, we try to record the behavior of the tracking system in the following scenarios:

• The basic moving-object tracking Figure 9.
• Tracking one out of multiple identical objects, Figure 10 and Figure 11.
• Tracking a far moving object, Figure 12.

4.2.1. The Basic Moving-Object Tracking

In this case of Figure 9, the multi-rotor followed the moving object. The moving car has no other semantically similar objects in the scene. In the experiment, the car was not very fast, so the multi-rotor kept pace with the moving car. At the beginning, the visual tracker was running without running the planner [56]. When the moving object became far from the multi-rotor (about 10 m away), the multi-rotor tracked the moving object. The multi-rotor stopped when the distance from the moving object became less than the predefined threshold $\alpha =5$ [m]. When the object was far from the multi-rotor, the estimated depth of the moving object was not accurate, but the multi-rotor still managed to track it. As the multi-rotor approached the moving object, depth estimation became more accurate. For more details, check the video attached in the GitHub repository.

4.2.2. Tracking One of Multiple Identical Objects

As shown in Figure 10, the experiment had three identical objects while the goal was tracking one of these objects. Figure 11 shows how the proposed managed to track the car (moving object) with no conflict with the other similar objects.

4.2.3. Tracking a Far Moving Object

The experiment shows how the tracking system will behave in the case of tracking a far moving object (Figure 12). The result shows that the visual object tracker managed to track the far moving objects even when the object was far from the multi-rotor. The experiment is shown in the results video in the GitHub repository.

4.3. Depth-Estimation Analysis

For depth estimation, the intended experiment had the target object moving over almost a straight line so we could observe the depth-estimation accuracy and that explains the relatively small errors on the y and z axes in the medium-range experiment, Table 2, as an example. The experiments are classified into three categories based on the distance between the object and the camera frame and considering a ToF (time-of-flight) camera [57] was used:

• Short (typical) range (5 m or less)
• Medium range (between 5 and 10 m)
• Long range (more than 10 m)

Table 2. Medium-range performance.

	x [m]	y [m]	z [m]	Position [m]
RMSE	1.1156	0.1618	0.2628	1.1576
MSE	1.0314	0.1566	0.2625	1.34
MAE	1.245	0.0262	0.0691	1.4506

Table 2. Medium-range performance.

	x [m]	y [m]	z [m]	Position [m]
RMSE	1.1156	0.1618	0.2628	1.1576
MSE	1.0314	0.1566	0.2625	1.34
MAE	1.245	0.0262	0.0691	1.4506

The experiments were conducted in an open-source [58] customized simulation environment, Figure 6.

4.3.1. Short (Typical) Range

In the typical range, Figure 13, the hybrid-depth estimation matches the sensor depth estimation, and the reason is that in short ranges, our proposed framework relies on the data measured by the sensor. While the claimed max. range of the sensor is up to 6 m [42], in our experiments the sensor depth measurements was available only up to the range of 5 m.

4.3.2. Medium Range

Medium range shows the interesting part of the hybrid-depth approach. The estimation of depth is in the range of 5 to 10 m.

As shown in Figure 14, the experimental results show that the maximum abstract depth error is around $20\%$. The root-mean-square deviation (error) RMSD or RMSE can be computed as

$$RMSE=\sqrt{\frac{{\sum }_{i=1}^N{\left(x_i-{\widehat{x}}_i\right)}^2}{N}}$$

(10)

where N: is the number of depth samples, $x_i$: is the ground truth at point i, and ${\widehat{x}}_i$ is the estimated depth at point i. The mean-square-error MSE is evaluated as follows

$$MSE=\frac{{\sum }_{i=1}^N{\left(x_i-{\widehat{x}}_i\right)}^2}{N}$$

(11)

where N: is the number of depth samples, $x_i$: is the ground truth at point i, and ${\widehat{x}}_i$ is the estimated depth at point i. In addition, the mean absolute error MAE was computed using following expression

$$\mathrm{MAE}=\frac{{\sum }_{i=1}^n\left|x_i-{\widehat{x}}_i\right|}{N}$$

(12)

where N: is the number of depth samples, $x_i$ is the ground truth at point i, and ${\widehat{x}}_i$ is the estimated depth at point i.

To identify how far the predictions fall from the ground truth, RMSE Equation (10), MSE Equation (11) and MAE Equation (12) were measured on random samples collected in the medium range (Table 2). The goal of the experiment is to observe the accuracy of depth estimation by moving the target object along a straight line. This explains the relatively small errors on the y and z axes.

4.3.3. Long Range

Depth estimation performed poorly in long ranges, and the estimated depth values oscillate between 9 and 12 m with a linearly growing error over distance. However, current performance will be enough in the case of a quadrotor with a similar or higher range of speed than the moving object (Figure 15 and Figure 16).

4.4. Visual Object-Tracker Evaluation

In this subsection, we explore the general performance regarding speed and the performance on the established visual-tracking benchmarks VOT-2016 [52],VOT-2018 [53], and VOT-2019 [54]. On the used VOT benchmarks, the performance is measured in terms of accuracy, robustness, and expected average overlap (EAO), as shown in

Table 3. Results for VOT-2016.

	VOT-2016
	EAO ↑	Accuracy ↑	Robustness ↓
SiamMask [5]	0.433	0.639	0.214
The proposed generic tracker	0.433	0.639	0.210

and

Table 4. Results for VOT-2019.

	VOT-2019
	EAO ↑	Accuracy ↑	Robustness ↓
The proposed generic tracker	0.281	0.610	0.527

, where EAO is used to rank trackers. Additionally, as shown in

Table 5. Comparison with the state-of-the-art under the EAO, accuracy, and robustness metrics on VOT-2018. Up arrows indicate that bigger values are better. Down arrows indicate that smaller values are better.

	EAO ↑	Accuracy ↑	Robustness ↓	Speed (fps) ↑
The proposed generic tracker	0.381	0.61	0.286	36
SiamMask [5]	0.380	0.609	0.276	55
DaSiamRPN [6]	0.326	0.569	0.337	160
SiamRPN [9]	0.244	0.490	0.460	200
SiamRPN++ [48]	0.414	0.600	0.234	35
PrDiMP [47]	0.385	0.607	0.217	40
CSRDCF [59]	0.263	0.466	0.318	48.9
SiamR-CNN [46]	0.408	0.609	0.220	15
THOR [60]	0.416	0.5818	0.234	112
TrDiMP [33]	0.462	0.600	0.141	26

, we considered the speed in the comparison on the VOT-2018 dataset.

To run our experiments, we had access to a server with a single Nvidia Quadro P4000 GPU with Ubuntu 18.04. The comparison with other nine state-of-the-art single-object trackers (Table 5), showed that the proposed tracker performed better in terms of accuracy on the VOT-2018 dataset. While in Table 3, the generic tracker had a better robustness score than the SiamMask model [5].

5. Conclusions

A fundamental computer vision problem known as single-object tracking (SOT) has numerous applications, including surveillance systems [4], autonomous vehicles [1,2] and aerial photography [61]. Its objective is to track a particular target within a video series using only its initial state (appearance and location).

In this paper, we proposed a framework which allows multi-rotor aerial vehicles to track moving objects in real time. We suggested having a set of templates, so that we could cover the object from various perspectives throughout time. We developed a method to overcome the limitations of popular commercial depth sensors by combining their data with relative depth maps produced by a machine-learning model for depth estimation [14] to create a hybrid-depth map. Lastly, an object classifier was utilized to overcome the inherited behaviour of the Siamese visual tracker and to limit the effect of distractors and false-positive detected objects.

Apart from the numerous advantages for the proposed framework, there are some limitations to take into consideration, such as the fact that the proposed hybrid-depth estimation approach showed some weaknesses in long-range distances. However, in the case of the moving-object tracking problem, this limitation could not be considered to be a critical issue because the distance estimation will become more and more accurate as the multi-rotor aerial vehicle reaches the target object.

In future work, the main focus will be to build multi-class non-contrastive incremental learning for the following reasons:

• Non-contrastive learning will be used for the opportunity of having a better representation [62] which could tackle problems such as occlusion, illumination variation, etc.
• Self-supervised incremental learning will be used to create a more effective tracking framework which could be trained on the new classes without suffering from the forgetting problem [63].