Adaptive Spatial-Temporal Regularization for Correlation Filters Based Visual Object Tracking

Abstract

Recently, Discriminative Correlation Filters (DCF) have shown excellent performance in visual object tracking. The correlation for a computing response map can be conducted efficiently in Fourier domain by Discrete Fourier Transform (DFT) of inputs, where the DFT of an image has symmetry on the Fourier domain. To enhance the robustness and discriminative ability of the filters, many efforts have been devoted to optimizing the learning process. Regularization methods, such as spatial regularization or temporal regularization, used in existing DCF trackers aim to enhance the capacity of the filters. Most existing methods still fail to deal with severe appearance variations—in particular, the large scale and aspect ratio changes. In this paper, we propose a novel framework that employs adaptive spatial regularization and temporal regularization to learn reliable filters in both spatial and temporal domains for tracking. To alleviate the influence of the background and distractors to the non-rigid target objects, two sub-models are combined, and multiple features are utilized for learning of robust correlation filters. In addition, most DCF trackers that applied 1-dimensional scale space search method suffered from appearance changes, such as non-rigid deformation. We proposed a 2-dimensional scale space search method to find appropriate scales to adapt to large scale and aspect ratio changes. We perform comprehensive experiments on four benchmarks: OTB-100, VOT-2016, VOT-2018, and LaSOT. The experimental results illustrate the effectiveness of our tracker, which achieved a competitive tracking performance. On OTB-100, our tracker achieved a gain of 0.8% in success, compared to the best existing DCF trackers. On VOT2018, our tracker outperformed the top DCF trackers with a gain of 1.1% in Expected Average Overlap (EAO). On LaSOT, we obtained a gain of 5.2% in success, compared to the best DCF trackers.

1. Introduction

An object tracking algorithm aims to track the object’s position in a 2D or 3D input, such as wireless signal, radar (i.e., a radar echo), or camera (i.g., video frame). For example, the Bluetooth 5.1 Direction Finding standard provides the probability of high-precision and real-time tracking of targets based on Angle of Departure (AoD) and the Angle of Arrival (AoA) [1]. Visual object tracking, as the main topic of this paper, is an important area in computer vision, which estimates the trajectory of the target object with visual information from a video sequence. visual object tracking can be applied into many applications, such as video surveillance, motion analyses, human computer interaction, automatic robot navigation, and traffic monitoring.

In recent years, the Discriminative Correlation Filter (DCF) for tracking has attracted a lot of attention due to its efficiency and effectiveness, from the trackers based on hand-crafted features [2,3,4] to the trackers that exploit learning with deep features [5,6,7]. The core idea in DCF trackers is how to learn robust and discriminative filters to adapt to appearance changes online by minimizing a least-squares loss for all circular shifts of a training sample.

Compared with the end-to-end deep trackers that require large-scale training set for offline training, the DCF-based trackers are model-free and do not need offline training. Although the deep trackers obtain superior performance on challenging sequences with attributes such as deformation, rotation, and scale variation. One limitation of deep Siamese trackers, such as SiamRPN++ [8], is that the model can easily lose the target if it is occluded by the similar distractors. While state-of-the-art DCF trackers exhibit more stable performance on non-rigid objects and are even more robust in the case of heavy occlusion caused by similar distractors.

The recent advancements in this area include applying different kinds of regularization techniques [4,6,9,10,11], developing an efficient model update strategy [12], reducing boundary effects [10], employing background information [9,10], applying kernel tricks [13,14], and using the efficient scale estimating scheme [3].

There are a variety of regularizations, such as spatial regularization [4], spatial-temporal regularization [6], context regularization [9,10], and adaptive spatial regularization [11], which can steadily improve the tracking performance on public tracking benchmarks. As spatial-temporal regularized correlation filters (STRCF) [6] and adaptive spatially-regularized correlation filters (ASRCF) [11] merely consider limited auxiliary information, both of them still suffer from severe appearance changes on challenging video sequences.

ASRCF does not consider the temporal consistent, while STRCF does not exploit the changes of spatial constraints over time explicitly. In this paper, we first employ both the temporal information and spatial continuity during the filters learning procedure. Secondly, we study the problem of multiple features (e.g., Histogram of Oriented Gradient (HOG) [15], color features [16], and deep features [17]) combination for correlation filters learning. We found that different feature combinations lead to different impacts on the tracking performance.

The feature representation of the different layers in a deep Convolution Neural Network (CNN) are different, and one limitation of most existing DCF trackers [5,6,7] is the utilization of the specific layer of the CNN model. In some cases, the appearance of the target experiences significant changes caused by deformation or rotation. For example, the DeepSTRCF [6] and ASRCF [11] both fail to track the target on MotorRolling sequence in OTB-100 [18], in which the target rotates in the image plane and has experienced severe non-rigid deformation.

Another limitation of the published DCF trackers is that most of them, such as STRCF [6] and ASRCF [11] utilize the 1-dimensional scale space search method to find the optimal scale. By applying the scale factor, the height and width of the tracking result are adjusted simultaneously in the current frame. However, such a scaling scheme is not sufficient for severe appearance changes, such as deformation, viewpoint change, and aspect ratio change.

In this paper, we explore the scale estimation in 2-dimensional scale space to address the above limitation, in which it can adjust the height and width of target with two different scale factors. By applying the new scale space search scheme, we demonstrate that it leads to better adaptive ability to aspect ratio changes and severe deformation.

The contributions of this paper can be summarized as follows:

• We develop a novel tracker that seamlessly applies temporal and adaptive spatial regularization to learn the correlation filters, which can significantly improve the accuracy and robustness of the tracker for videos with challenging attributes.

• An efficient optimization procedure is presented, and the Alternating Direction Method of Multipliers (ADMM) algorithm is applied for solving each sub-problem.

• We propose a 2-dimensional scale space search method for finding the optimal 2-dimensional aspect ratios to adapt to target deformation, instead of the 1-dimensional scale factor. We validate our approach through sufficient experiments, and the tracking results demonstrate that our new scale search method can achieve significant improvements. On LaSOT, our approach outperforms its counterpart DeepSTRCF in success score by 5.6% on the deformation attribute and 5.4% on the Scale Variation attribute.

• We evaluate our method on OTB-100 [18], VOT-2016 [19], VOT-2018 [20], and LaSOT [21] tracking datasets and report the state-of-the-art tracking performance. On OTB-100, our approach outperforms its counterparts ASRCF and DeepSTRCF by 0.8% and 2.3% in success. Compared to DeepSTRCF [6], our tracker improves by 5.2% in success.

The remainder of the paper is structured as follows. In Section 2, we review the most closely related work on visual object tracking, the traditional DCF tracking procedure is described in Section 3, our approach is presented in Section 4, experimental results are presented in Section 5, and our conclusions are drawn in Section 6.

2. Related Work

Most of the traditional tracking algorithms can be categorized as either generative or discriminative. The generative trackers search the region that is the most similar to the reference target in a video sequence, such as ${\ell }_1$ tracker [22], Multi-Task Tracking (MTT) [23], and Multi-task Correlation Particle Filter (MCPF) [24]. The discriminative trackers aim to distinguish the target from the background by a detection model, such as the Multiple Instance Learning (MIL) classifier used in [25] and the kernelized structured output support vector machine (SVM) applied in Struck [26].

The tracking-by-detection [27] tracker exploited the circular structure for fast learning of the classifier and detecting with the dense sampling strategy. Henriques et al. [13] proposed kernelized correlation filters (KCF) based on linear and kernel ridge regression for fast visual object tracking. In order to utilize multi-channel RGB images or features (e.g., HOG), Kiani Galoogahi et al. proposed the Multi-channel correlation filter (MCCF) strategy for car localization, with the efficient use of memory and computations. Tang et al. [14] demonstrated that the single kernel can be decomposed into multiple kernels that can be applied to fuse multiple channel features for tracking.

Regularization methods are widely used for solving the least-squares problem, such as the spatial regularizer in [4], total generalized variation (TGV) regularizer in [28], and temporal regularizer in [6]. To solve the boundary effect problem caused by the circular assumption, Danelljan et al. [4] proposed spatially regularized discriminative correlation filters (SRDCF), which adds spatial regularization to the optimization problem via a weighted function.

The spatial weights are based on the prior information about the spatial extent of the filter. The framework in [10] allows the learning of background-aware correlation filters (BACF) from both the background and target region. In additional, Mueller et al. [9] applied context image patches around the target to learn discriminative context-aware correlation filters (CACF) for tracking.

Similar to SRDCF [4], Li et al. [6] proposed spatial-temporal regularized correlation filters (STRCF) that exploit both the spatial and temporal regularizations during model learning. The closed solution can be obtained by the ADMM algorithm, which performs the learning procedure only on a single image instead of using multiple samples in SRDCF [4]. In [29], Lukežič et al. proposed a discriminative correlation filter with Channel and Spatial Reliability (CSR-DCF) model, in which the channel and spatial reliability are combined to constrain the filter learning. The unsupervised segmentation mask is utilized as the spatial reliability.

In [30], Ma et al. applied multiple convolutional features with identical resolution for learning independent Correlation Filters (CFs) and then hierarchically estimated the final response. In [7], Danelljan et al. proposed the Continuous Convolution Operator Tracker (C-COT) to learn the CFs in a continuous domain and integrate multi-resolution shallow and deep feature maps. In [12], Danelljan proposed Efficient Convolution Operators (ECO) to reduce the computation cost while preserve the tracking performance. However, the multiple samples used for learning CFs still led to a high computation cost.

Zhang et al. proposed a ensemble tracker based on Multi-Expert Entropy Minimization (MEEM) restoration scheme to address the model drift problem. Li et al. [31] extended the MEEM to multi-expert in a collaborative way, instead of the independent entropy computation in MEEM. Similarly, Li et al. [32] utilized the unified discrete graph to model the relationship of multi-expert, while the computation load of dynamic programming increased significantly with a larger number of experts.

In [33], Wang et al. proposed a strategy to adaptively select the suitable expert from a pool of experts which was generated with different features combinations. Bhat et al. [34] combined the shallow feature model and deep feature model through adaptive weights to enhance the capacity of the model, in which the model can be obtained by solving the Quadratic Programming problem. While the limitation of it is that they only consider the two different features and the fusion happens in the model prediction stage. The weighted confidence margin is computed with the confidence scores and their locations, which does not consider the scale variation of target object.

Dai et al. [11] proposed adaptive spatially-regularized correlation filters (ASRCF) to further improve the accuracy of CFs by adding the adaptive spatial regularization, while they directly concatenate the multiple features (i.e., HOG and deep features) together for CF learning.

Although the advanced CF trackers, like STRCF [6] and ASRCF [11], have achieved state-of-the-art performance in most sequences on the OTB-100 [18] dataset, the sequences, such as $\mathit{Jump}$, $\mathit{MotorRolling}$, and $\mathit{Diving}$ in OTB-100 [18], still pose challenges. Most existing DCF trackers estimate the scale of target by 1-dimensional scale space search, which limits the tracker’s ability to adapt to large scale ratio aspect changes due to challenges, such as deformation, rotation, and viewpoint changes.

Recently, with the development of large scale datasets, the improvement of computing power, and reliable and efficiency computing resources allocation [35], applications based on deep learning have entered a period of rapid development. Trackers based on the deep Siamese network have also drawn great attention and shown significant improvements regarding tracking performance.

In [36], Bertineto et al. proposed the Siamese Fully-Convolutional (SiamFC) tracker based on an end-to-end fully-convolutional Siamese network and achieved high tracking speed beyond real time. In [37], the Correlation Filter Network (CFNet) was proposed for the joint learning of deep features and correlation filters in an end-to-end manner.

In the Siamese region proposal network (SiamRPN) [38], Li et al. took the advantages of the Siamese network and region proposal network (RPN) to improve the accuracy of localization and regression of the target bounding box. Furthermore, SiamRPN++ [8] employed a lightweight Depthwise Cross Correlation (DW-XCorr) layer instead of the Channel Cross Correlation (UP-XCorr) layer in SiamRPN, which achieved state-of-the-art tracking performance.

In contrast to most of the Siamese trackers, Danelljan et al. [39] proposed the Accurate Tracking by Overlap Maximization (ATOM) tracker, which combined the offline trained target estimation module and online learned target classification module for tracking. In [40], a model predictor was designed to estimate the target model, which was trained with a set of multiple samples of the sequence instead of the image pairs in Siamese approaches.

In [41], Zhang et al. designed new residual modules to build deeper and wider backbone networks, in which the cropping-inside residual (CIR) unit was utilized to remove the effect of padding operation, and the downsampling cropping-inside residual (CIR-D) unit was utilized to reduce the spatial size of the feature maps. One limitation in recent deep trackers is that they need to train the target model with large scale datasets in the offline phase.

Different from the DCF trackers that update the model with a continuous strategy in each frame [10,11] or a sparser scheme in every N frames [12], the deep trackers, such as SiamRPN++ [8] do not update the model during tracking, as they are sensitive to sudden changes caused by deformation and viewpoint change. Deep trackers (e.g., SiamRPN [38] and ATOM [39]) also easily lose the target when there are many similar distractors around the target.

3. Revisiting Traditional Correlation Filter Tracking

In this section, we first introduce the standard discriminative correlation filters for visual object tracking. Then, we briefly describe the two baseline trackers, STRCF [6] and SRDCF [11]. These two methods utilize different regularizations for filter learning.

3.1. Revisting Standard Correlation Filter

Given a sequence of images and its initial state, including the position and size of target, the task of standard DCF trackers is to learn CFs using a set of training samples ${\left\{{\mathrm{x}}_i,y_i\right\}}_{i=1}^N$, where ${\mathrm{x}}_i$ denote the multiple channel feature representation of i-th image, $y_i$ is the corresponding desired output. The main objective function can be defined as:

$$\ell =\sum _{i=1}^N\parallel \sum _{d=1}^D{f}^d\star {\mathrm{x}}_i^d-y_i{\parallel }_2^2+\frac{\lambda }{2}{\parallel f\parallel }_2^2$$

(1)

Here, $\lambda$ represents the weight of regularization term. f is the correlation filter. According to Parseval’s theorem, the filter f can be efficiently formulated in the frequency domain as

$$\widehat{f}=\frac{{\sum }_{i=1}^N{\widehat{y}}_i\odot \mathrm{conj}({\widehat{\mathrm{x}}}_i)}{{\sum }_{i=1}^N{\widehat{\mathrm{x}}}_i\odot \mathrm{conj}({\widehat{\mathrm{x}}}_i)+\lambda I}$$

(2)

where ⊙ denotes the element-wise multiplication of two vectors. The $\widehat{}$ denotes the Discrete Fourier Transform (DFT) $\mathcal{F}$, which possesses the property of symmetry, i.e., ${\widehat{\mathrm{x}}}_i=\mathcal{F}\left\{{\mathrm{x}}_i\right\}$ and ${\widehat{y}}_i=\mathcal{F}\left\{y_i\right\}$ are the DFT of the sample and its desired output. $\mathrm{conj}$ denotes the complex conjugation operation. Given a test image patch $z$, the corresponding response can be obtained by:

$$G={\mathcal{F}}^{-1}\left\{\sum _{d=1}^D{\widehat{z}}^d\odot \mathrm{conj}({\widehat{f}}^d)\right\}$$

(3)

where ${\mathcal{F}}^{-1}$ denotes the inverse DFT operation.

3.2. Revisiting STRCF

In STRCF [6], a temporal regularization term $\parallel f-f_{t-1}{\parallel }_2^2$ is incorporated into the objective as follows:

$$\begin{array}{cc}\hfill L=arg\underset{f}{min}& \frac{1}{2}\parallel \sum _{d=1}^D{\mathrm{x}}_t^d\star f^d{-y\parallel }_2^2+\frac{\lambda }{2}\sum _{d=1}^D\parallel \mathrm{w}\odot f^d{\parallel }_2^2+\frac{\mu }{2}{\parallel f-f_{t-1}\parallel }_2^2\hfill \end{array}$$

(4)

where $\mathrm{w}$ denotes the spatial regularization weight as SRDCF [4]. $\lambda$ and $\mu$ are the penalty factors of the two regularization terms. The temporal regularization aims to exploit relationship between adjacent samples by passively updating the correlation filters, making the filter learned in the current frame close to the one in the previous frame.

3.3. Revisiting ASRCF

In ASRCF [11], Dai et al. introduced a regularization based on spatial weight $\mathrm{w}$ and its reference ${\mathrm{w}}^r$. The objective can be formulated as:

$$\begin{array}{cc}\hfill L=arg\underset{f}{min}& \frac{1}{2}\parallel \sum _{d=1}^D{\mathrm{x}}_t^d\star ({\mathrm{P}}^{\top }{f}^d){-y\parallel }_2^2+\frac{\lambda }{2}\sum _{d=1}^D\parallel \mathrm{w}\odot f^d{\parallel }_2^2+\frac{\mu }{2}{\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}\parallel }_2^2\hfill \end{array}$$

(5)

where $\lambda$ and $\mu$ are the penalty factors of the two spatial regularization terms. $\mathrm{P}$ is the cropping matrix to encourage using large number of negative samples from surrounding background for training. The third term is used to adjust the spatial weight during model learning.

4. Our Approach

In this section, we propose a novel tracking framework that simultaneously considers the spatial similarity of the target between current frame and reference frame, and the temporal consistency of filters in adjacent frames. For accurate scale estimation, we proposed a 2-dimensional scale space search method to adapt to appearance changes caused by challenging scenarios, such as deformation, target rotation, and viewpoint change. The Figure 1 shows the diagram of our tracking approach. The two sub-models are refer to two different learning rates.

First, the feature extractor is utilized to extract the features of an image patch. After obtaining the features of a target object, we obtain the appearance model-1 and appearance model-2. Secondly, we learn the two sub-models (filters) with different learning rates, which correspond to $h_t^1$ and $h_t^2$. During the tracking process, the two sub-models are used to produce response maps. Finally, the response maps of the two sub-models are weighted aggregated together to produce the final tracking result.

4.1. Adaptive Spatial-Temporal Regularization

Li et al. [6] introduced a temporal regularization term $\parallel f-f_{t-1}{\parallel }_2^2$ to make the filter learned from the current frame similar to the previous filter. In [11], Dai et al. proposed a spatially-regularized term $\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}{\parallel }_2^2$ to constrain the adaptive spatial weight $\mathrm{w}$ in the current frame be similar to a reference weight ${\mathrm{w}}^{\mathrm{r}}$. We found that both the temporal regularization and adaptive spatial regularization can be combined seamlessly to enhance the discriminative ability of filters. In this paper, we introduce an adaptive spatial-temporal regularization into the discriminative correlation filters framework.

In frame t, we prepare the training pair $\left\{{\mathrm{x}}_t,y\right\}$, where ${\mathrm{x}}_t$ is the sampled image patch, $\mathring{y}$ denotes its desired response generated with Gaussian function. By applying the adaptive spatial-temporal regularization, the corresponding optimization problem can be defined as follows:

$$\begin{array}{cc}\hfill L=arg\underset{f}{min}& \frac{1}{2}\parallel \sum _{d=1}^D{\mathrm{x}}_t^d\star ({\mathrm{P}}^{\top }{f}^d){-y\parallel }_2^2+\frac{\lambda }{2}\sum _{d=1}^D{\parallel \mathrm{w}\odot f^d\parallel }_2^2\hfill \\ \hfill +& \frac{{\mu }_1}{2}\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}{\parallel }_2^2+\frac{{\mu }_2}{2}{\parallel f-f_{t-1}\parallel }_2^2\hfill \end{array}$$

(6)

where $f_{t-1}$ denotes the learned CFs of the $t-1$th frame. f is the CFs of the current frame to be learned, $\parallel \mathrm{w}\odot f^d{\parallel }^2$ and $\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}{\parallel }_2^2$ are the two spatial regularizers. $\mathrm{P}$ is the cropping matrix as in BACF [10]. $\parallel f-f_{t-1}{\parallel }_2^2$ denotes the temporal regularizer. ${\lambda }_1$, ${\lambda }_2$, and $\mu$ are the three different penalty factors, respectively. The ⊤ operator computes the transpose of the matrix. For a complex vector or matrix, it computes the conjugate transpose of them.

By applying the Parseval theorem, the above objective function can be expressed in the frequency for the computation efficiency as:

$$\begin{array}{cc}\hfill & L=arg\underset{\widehat{f}}{min}\frac{1}{2}\parallel \sum _{d=1}^D\mathrm{conj}({\widehat{\mathrm{x}}}^d)\odot {\widehat{\mathrm{g}}}^d-\widehat{y}{\parallel }_2^2+\frac{\lambda }{2}\sum _{d=1}^D{\parallel \mathrm{w}\odot f^d\parallel }_2^2\hfill \\ \hfill & +\frac{{\mu }_1}{2}\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}{\parallel }_2^2+\frac{{\mu }_2}{2}{\parallel \widehat{\mathrm{g}}-{\widehat{\mathrm{g}}}_{t-1}\parallel }_2^2,\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.,{\widehat{\mathrm{g}}}^d=\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }{f}^d,d=1,...,D\hfill \end{array}$$

(7)

Equation (7) can be solved with an Augmentation Lagrangian Method (ALM) as:

$$\begin{array}{cc}\hfill L=arg\underset{\widehat{f}}{min}& \frac{1}{2}\parallel \sum _{d=1}^D\mathrm{conj}({\widehat{\mathrm{x}}}^d)\odot {\widehat{\mathrm{g}}}^d-\widehat{y}{\parallel }_2^2+\frac{\lambda }{2}\sum _{d=1}^D{\parallel \mathrm{w}\odot f^d\parallel }_2^2\hfill \\ \hfill +& \sum _{d=1}^D{({\widehat{\zeta }}^d)}^T({\widehat{\mathrm{g}}}^d-\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }{f}^d)+\frac{{\mu }_1}{2}{\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}\parallel }_2^2\hfill \\ \hfill +& \frac{{\mu }_2}{2}\parallel \widehat{\mathrm{g}}-{\widehat{\mathrm{g}}}_{t-1}{\parallel }_2^2+\frac{\gamma }{2}\sum _{d=1}^D{\parallel {\widehat{\mathrm{g}}}^d-\sqrt{\mathrm{T}}{\mathrm{FP}}^{\mathrm{T}}{f}^d\parallel }_2^2\hfill \end{array}$$

(8)

where the $\widehat{\zeta }$ and $\gamma$ denote the Fourier Transform of the Lagrangian vector and penalty factor, respectively. Inspired form the STRCF [6], we introduce an auxiliary variable $\mathrm{h}=\frac{1}{\gamma }\zeta$, Equation (8) can be reformulated as,

$$\begin{array}{cc}\hfill L=& arg\underset{\widehat{f}}{min}\frac{1}{2}\parallel \sum _{d=1}^D\mathrm{conj}({\widehat{\mathrm{x}}}^d)\odot {\widehat{\mathrm{g}}}^d-\widehat{y}{\parallel }_2^2+\frac{\lambda }{2}\sum _{d=1}^D{\parallel \mathrm{w}\odot f^d\parallel }_2^2\hfill \\ \hfill & +\frac{\gamma }{2}\sum _{d=1}^D{\parallel {\widehat{\mathrm{g}}}^d-\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }{f}^d+{\widehat{\mathrm{h}}}^d\parallel }_2^2\hfill \\ \hfill & +\frac{{\mu }_1}{2}\parallel \mathrm{w}-{\mathrm{w}}^{\mathrm{r}}{\parallel }_2^2+\frac{{\mu }_2}{2}{\parallel \widehat{\mathrm{g}}-{\widehat{\mathrm{g}}}_{t-1}\parallel }_2^2\hfill \end{array}$$

(9)

Then, the ADMM Algorithm [42] is adopted alternatively to solve the objective function.

Subproblem f:

$$\begin{array}{cc}\hfill f^d=& arg\underset{f^d}{min}\frac{\lambda }{2}\parallel \mathrm{W}{f}^d{\parallel }_2^2+\frac{\gamma }{2}{\parallel \widehat{\mathrm{g}}-\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }{f}^d+{\widehat{\mathrm{h}}}^d\parallel }_2^2\hfill \end{array}$$

(10)

where $\mathrm{W}$ denotes the diagonal matrix: $\mathrm{W}=\mathrm{diag}(\mathrm{w})$. Taking the derivative of Equation (10) with respect to $f^d$ and setting it to be zero, we can obtain the solution for $f^d$,

$$\begin{array}{cc}\hfill f^d=& {(\lambda {\mathrm{W}}^{\top }\mathrm{W}+\gamma \mathrm{TP})}^{-1}Q,\hfill \\ \hfill Q=& \gamma \mathrm{TP}(\mathrm{g}+\mathrm{h})\hfill \end{array}$$

(11)

where $\mathrm{g}$ and $\mathrm{h}$ can be obtained by applying the inverse DFT ${\mathcal{F}}^{-1}$ on $\widehat{\mathrm{g}}$ and $\widehat{\mathrm{h}}$, i.e., $\mathrm{g}={\mathcal{F}}^{-1}\left\{\widehat{\mathrm{g}}\right\},\mathrm{h}={\mathcal{F}}^{-1}\{\widehat{\mathrm{h}}\}$.

Subproblem g:

$$\begin{array}{cc}\hfill {\widehat{\mathrm{g}}}^{*}=& arg\underset{\widehat{f}}{min}\frac{1}{2}\sum _{d=1}^D{\parallel \mathrm{conj}({\widehat{\mathrm{x}}}^d)\odot {\widehat{\mathrm{g}}}^d-\widehat{y}\parallel }_2^2\hfill \\ \hfill +& \frac{\gamma }{2}\sum _{d=1}^D{\parallel {\widehat{\mathrm{g}}}^d-\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }{f}^d+{\widehat{\mathrm{h}}}^d\parallel }_2^2\hfill \\ \hfill +& \frac{{\mu }_2}{2}{\parallel {\widehat{\mathrm{g}}}^d-{\widehat{\mathrm{g}}}_{t-1}^d\parallel }_2^2\hfill \end{array}$$

(12)

As the jth element of $\widehat{y}$ is dependent only on the jth element across the D channels of the filters ${\widehat{\mathrm{g}}}^d$ and samples ${\widehat{\mathrm{x}}}^d,d=1,...,D$. We denote ${\mathcal{V}}_j(\widehat{\mathrm{a}})={[\mathrm{conj}({\widehat{\mathrm{a}}}^1(j)),...,\mathrm{conj}({\widehat{\mathrm{a}}}^D(j))]}^{\top }$ the vector of jth element of $\widehat{a}$ along all D channels. Thus, solving Equation (12) equals to solve $\mathrm{T}$ subproblems as:

$$\begin{array}{cc}\hfill \widehat{\mathrm{g}}{(t)}^{*}=& arg\underset{\widehat{g}}{min}\frac{1}{2}{\parallel {\mathcal{V}}_j{({\widehat{\mathrm{x}}}_t)}^{\top }{\mathcal{V}}_j(\widehat{\mathrm{g}})-{\widehat{y}}_j\parallel }_2^2\hfill \\ \hfill +& \frac{\gamma }{2}{\parallel {\mathcal{V}}_j(\widehat{\mathrm{g}})+{\mathcal{V}}_j(\widehat{\mathrm{h}})-{\mathcal{V}}_j(\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }f)\parallel }_2^2\hfill \\ \hfill +& \frac{{\mu }_2}{2}{\parallel {\mathcal{V}}_j(\widehat{\mathrm{g}})-{\mathcal{V}}_j({\widehat{\mathrm{g}}}_{t-1})\parallel }_2^2\hfill \end{array}$$

(13)

Then, we can obtain the solution of $\widehat{\mathrm{g}}{(t)}^{*}$ as:

$$\begin{array}{c}\hfill \widehat{\mathrm{g}}{(t)}^{*}={({\mathcal{V}}_j({\widehat{\mathrm{x}}}_t){\mathcal{V}}_j{({\widehat{\mathrm{x}}}_t)}^{\top }+(\gamma +{\mu }_2)\mathrm{T}{\mathrm{I}}_k)}^{-1}M\end{array}$$

(14)

where $M={\mathcal{V}}_j({\widehat{\mathrm{x}}}_t){\widehat{y}}_j+\gamma \mathrm{T}{\mathcal{V}}_j(\sqrt{\mathrm{T}}{\mathrm{FP}}^{\top }f)-\gamma \mathrm{T}{\mathcal{V}}_j(\widehat{\mathrm{h}})+{\mu }_2\mathrm{T}{\mathcal{V}}_j({\widehat{\mathrm{g}}}_{t-1})$. Then, Equation (14) can be solved by using the Sherman Morrison formula [43], ${(A+u{v}^{\top })}^{-1}=A^{-1}-\frac{A^{-1}u{v}^{\top }{A}^{-1}}{1+v^{\top }{A}^{-1}u}$, where $u{v}^{\top }$ is a rank-1 matrix. $u\in {\mathsf{R}}^{T\times 1}$ and $v\in {\mathsf{R}}^{T\times 1}$ are two column vectors, respectively. Then, Equation (14) can be reformulated as:

$$\begin{array}{c}\hfill \widehat{\mathrm{g}}{(t)}^{*}=\frac{1}{(\gamma +{\mu }_2)\mathrm{T}}\left(I_k-\frac{{\mathcal{V}}_j(\widehat{\mathrm{x}}){\mathcal{V}}_j{(\widehat{\mathrm{x}})}^{\top }}{(\gamma +{\mu }_2)\mathrm{T}+{\mathcal{V}}_j{(\widehat{\mathrm{x}})}^{\top }{\mathcal{V}}_j(\widehat{\mathrm{x}})}\right)M\end{array}$$

(15)

Subproblem w: The solution of spatial regularization weight $\mathrm{w}$ can be obtained by solving the following object function,

$$\begin{array}{cc}\hfill {\mathrm{w}}^{*}=& arg\underset{\mathrm{w}}{min}\frac{\lambda }{2}\sum _{d=1}^D\parallel \mathrm{diag}(\mathrm{w})f^d{\parallel }_2^2+\frac{{\mu }_1}{2}{\parallel \mathrm{w}-{\mathrm{w}}^r\parallel }_2^2\hfill \\ \hfill =& arg\underset{\mathrm{w}}{min}\frac{\lambda }{2}\sum _{d=1}^D\parallel \mathrm{diag}(f^d){\mathrm{w}\parallel }_2^2+\frac{{\mu }_1}{2}{\parallel \mathrm{w}-{\mathrm{w}}^r\parallel }_2^2\hfill \\ \hfill =& {(\lambda \sum _{d=1}^D(\mathrm{diag}{(f^d)}^{\top }\mathrm{diag}(f^d))+{\mu }_{1}I)}^{-1}{\mu }_1{\mathrm{w}}^r\hfill \end{array}$$

(16)

Updating the Lagrangian Multiplier: We update the Lagrangian vector as:

$$\begin{array}{c}\hfill {\widehat{\mathrm{h}}}^{i+1}={\widehat{\mathrm{h}}}^i+\gamma ({\widehat{\mathrm{g}}}^{i+1}-{\widehat{\mathrm{f}}}^{i+1})\end{array}$$

(17)

where the penalty factor $\gamma$ is updated as:

$$\begin{array}{c}\hfill \gamma =min({\gamma }_{max},\beta \gamma )\end{array}$$

(18)

Model Update: Instead of the online update strategy which updates the tracking model at each frame in most of the traditional DCF trackers [2,10,13], we employ the criterion called Peak to Sidelobe Ratio (PSR) [2] to measure the tracking uncertainty and decide when to update the tracking result in upcoming frames. Generally, the larger PSR value indicates the better location accuracy.

4.2. 2-Dimensional Scale Space Search Method

In visual object tracking, trackers often lose the target due to non-rigid deformation, especially for the most of DCFs trackers, which utilize the 1-dimensional fast scale estimation approach [3]. In this method, S number of scales with a scale factor a are used to extract an image patch $J_n$ of size $a^{n}P\times a^{n}P$ centered around the target, where $n\in \left\{-\lfloor \frac{S-1}{2},...,\lfloor \frac{S-1}{2}\rfloor \right\}$. During online tracking, the tracking algorithms learn their appearance models and perform tracking with initialization in the first frame.

The aspect ratio is fixed in this scale estimation approach. For example, if the best scale factor is $a^{*}$, the width and height become $\left[a^{*}\times W_{t-1},a^{*}{H}_{t-1}\right]$, where $W_{t-1}$ and $H_{t-1}$ are the width and height of the target in the previous frame. The limitation of this scheme is that the width $W_{t-1}$ and $H_{t-1}$ are scaled with a same factor $a^{*}$. Since the new estimated state cannot cover the appearance variance of target due to severe non-rigid deformation, the variance of width and height are not the same with each other.

As shown in Figure 2, the size of the ground truth bounding boxes of target objects in raw frames are $W_1=133$, $H_1=146$, $W_2=90$, $H_2=132$, $W_3=95$, and $H_3=93$, respectively. Then, the ratios are: $a_w^1=W_2/W_1=0.676$, $a_h^1=H_2/H_1=0.904$, $a_w^2=W_3/W_2=1.056$, and $a_h^2=H_3/H_2=0.705$. The different ratios of height and width illustrate that applying an identical scale factor to scale estimation is not enough to capture the shape variation.

To adapt to shape changes, we proposed a novel scale space search method, which allows the scale factors to change along two axis directions. As shown in Figure 3, the figure above is the scale variation in conventional trackers that use the 1-dimensional scale space search method, and the figure below is scale variation in our new method. Let the S denotes the size of scale filter, for each $m,n\in \left\{-\lfloor \frac{S-1}{2},...,\lfloor \frac{S-1}{2}\rfloor \right\}$, we extract an image patch $J_{m,n}$ of size $a^{m}P\times a^{n}R$ centered around the previous target location for tracking, where $a^m$ and $a^n$ denote scale factors along two dimensions as shown in the Figure 3.

Compared to the 1-dimensional scale space search method, our new scale search method explores different aspect ratios along two dimensions, aiming to capture severe non-rigid deformation of the target. Afterwards, the final translations and scale factors are computed by Newton’s method as in SRDCF [4].

4.3. The Impact of Learning Rate

In traditional CF-based visual tracking methods, the learning rate is a common parameter that is used for updating the appearance model. In our framework, if we set the learning rate to 1, then the appearance model will degenerate to STRCF [6], which directly feeds the current appearance feature to the learning model. Through the experiments, we found that the learning rate also has a significant impact on the tracking accuracy of the model. For example, the $Car24$ sequence in OTB-100 [18] consists of more than 3k frames, and the appearance of the $Car$ changes slightly when it moves from right side to left side of the road or from far to near as in the examples we present in Figure 4. The larger learning rate implies that the appearance model updates rely more the recent features and less the previous features.

Different from the learning rate in the domain of machine learning and deep learning, which can be adjusted by the model online, we do not have the fully labeled training data to find the optimal hyper parameter, such as the learning rate. In our framework, we utilize the learning rate to update the appearance model, and the main difference is that there are two kinds of learning rates employed in our model, corresponding to two different sub-models as shown in Figure 1.

5. Experiments

5.1. Experimental Implementations

Our approach ASTRCF is implemented with Matlab 2019b using MatConvNet on a PC with an i5-7500 CPU at 3.4 GHz, 64GB RAM, and a NVIDIA Geforce 1080ti GPU. Algorithm 1 provides a brief outline of our tracker. In order to fuse the deep and hand-crafted feature in a proper way for filter learning, we set the standard deviation for the desired correlation output to $1/16$ of the target size as the same with [3]. The initial penalty factor $\gamma$ is set as 1. We set the maximal penalty factor ${\gamma }_{max}$ as 10,000. We set the step $\beta$ as 10 to update $\gamma$ by ${\gamma }^{i+1}=min({\gamma }_{max},\beta {\gamma }^i)$. The spatial regularization factor ${\mu }_1$ is set to $0.001$.

Algorithm 1 The proposed tracking algorithm: iteration at time step t

Inputs:  Images ${\left\{I_t\right\}}_{t=1}^T$, object position in previous frame $p_1$, scale estimation on previous frame $s_{t-1}$, filters $h_{t-1}^1$ and $h_{t-1}^2$. Outputs:  updated correlation filters $h_t^1$ and $h_t^2$, estimated target position $p_t$, estimated scale $s_t$.  1: repeat  2:  extract the feature map $x_t$ based on previous target position $p_{t-1}$.  3:  estimate the current pos $p_t$ and scale $s_t$ using the CFs $h_{t-1}^1$, $h_{t-1}^2$ by the 2-dimensional scale space search method.  4:  extract samples $z_t$ based on current target position $p_t$.  5:  update models of $h_t^1$ and $h_t^2$ based on samples $z_t$ and previous models of $h_{t-1}^1$ and $h_{t-1}^2$.  6: until end of the sequence.

The temporal regularization factor ${\mu }_2$ is set to 15 and 8 for the two sub-models, respectively. The learning rates are set to $0.0185$ and $0.003$ for two sub-models, respectively. The number of scales S is set to 3 and the scale step a is set to 1.05. Then, we extract 9 image patches during tracking in our algorithm. For target representation, we combined hand-crafted features (i.g., HOG and Color-Names) with 27-th (relu-$4\_4$) and 36-th (relu-$5\_4$) layers in the Vgg19 network [44] for model learning.

5.2. Evaluation Methodology

We use precision and success plots as [18] to measure the performance of the trackers. The precision plot shows the percentage of frames for which the distance between estimated location and ground truth is within the specific threshold. The success plot indicates how many times the target is successfully tracked among all frames of a sequence within the IoU (Intersection-over-Union) over the estimated bounding box and the ground truth.

The VOT2016 and VOT2018 datasets evaluate the trackers with three metrics: (1) Accuracy measures the intersection over union between the predicted bounding box and ground truth. (2) Robustness measures the mean number of failures per sequence. (3) The Expected Average Overlap (EAO) integrates the accuracy and robustness and is computed as the average of the expected average overlap over a selected range of sequence lengths.

5.3. Ablation Study

An ablation study was conducted on the OTB-100 [18] dataset to validate the effectiveness of our proposed framework. For a fair comparison, we used deep features include layer-79 from ResNet-101 [45], layer-14 from VggM-2048 [17], layer-27 and layer-36 from Vgg-19 [44], and hand-crafted feature (i.g., HOG) for different trackers. First, we investigate the robustness of deeper features for dealing with the drastic deformation; secondly, we follow the same features setting to fairly compare our framework with the two baseline trackers (e.g., STRCF [6] and ASRCF [11]), conducting the challenging sequences in

Table 1. Average Overlap on challenging sequences of OTB-100 with different model settings.

Tracker	Ironman	MotorRolling	Matrix	Skater2	Singer2	Soccer	KitSurf	Skiing	Skater	Trans	Models
STRCF	0.117	0.094	0.010	0.639	0.039	0.189	0.716	0.088	0.601	0.570	HC+VggM-14
STRCF	0.096	0.154	0.494	0.635	0.764	0.201	0.561	0.099	0.635	0.578	HC+ResNet101-79+VggM-14
ASRCF	0.436	0.117	0.448	0.614	0.765	0.502	0.733	0.513	0.494	0.546	HC+Vgg16-23+VggM-4
ASRCF	0.125	0.119	0.441	0.610	0.788	0.449	0.712	0.515	0.537	0.563	HC+ResNet101-79+VggM-14
Ours	0.515	0.117	0.597	0.588	0.040	0.572	0.743	0.512	0.574	0.551	HC+Vgg16-23+VggM-4
Ours	0.531	0.525	0.581	0.611	0.747	0.477	0.688	0.494	0.600	0.552	HC+ResNet101-79+VggM-14
Ours	0.406	0.583	0.573	0.645	0.739	0.551	0.777	0.574	0.677	0.631	2D search method

.

We also employ two kinds of deep features on STRCF [6] and ASRCF [11]. First, we found that more robust features (e.g., feature maps of layer-79 in ResNet101) improved the performance of STRCF [6], while not significant for ASRCF [11]. For example, the AUC success of $Ironman$ sequence with deep features of ResNet101-79 layer decreases to 0.125 from 0.436 for ASRCF [11]. Secondly, through the $\mathit{Singer}\mathit{2}$ sequence, we found that the low level features and high semantic features contributed differently to the tracking performance.

In such a scenario, the deep semantic features degraded the tracker’s discriminative ability and make the tracker suffer from the background clutter. In order to improve the robustness of our tracker, we constructed a collaborative tracking framework, which includes a shallow model and a deep model, operating as a parallel way during the tracking process. For the $MotorRolling$ sequence, the experimental results show that both the STRCF [6] and ASRCF [11] lose the target with relatively low-level features (e.g., HOG and Vgg16-23). While our tracker with the deeper features from ResNet101 could successfully handle this problem and can achieve promising performance as the Table 1 shown.

In addition, if we increase the learning rate, then our model can obtain large improvements on sequences of the challenging attributes, for example, fast moving, illumination variation, and in-plane rotation. The results of our final model (HC+Vgg19-27+Vgg19-36) with 2-dimensional scale space search method are presented at the last line in Table 1, which outperform the other models by a large margin on these challenging sequences.

5.4. State-of-the-Art Comparison

We evaluated our framework with the selected state-of-the-art trackers, including KCF [13], DeepSRDCF [4], CCOT [7], MDNet [46], ECO [12], ECO_HC [12], DeepSTRCF [6], MCPF [24], ASRCF [11], ACFN [47], CSR-DCF [29], Staple [48], SAMF [49], MEEM [50], LSART [51], SiamRPN [38], SiamRPN++ [8], ATOM [39], and DiMP50 [40] on OTB [18], VOT-2016 [19], VOT-2018 [20], and LaSOT [21] datasets. To further illustrate the effectiveness of our approach, we also present qulitative results on these tracking datasets in Section 5.4.2.

5.4.1. Quantitative Evaluation

OTB-100 Dataset. OTB-100 [18] is an online tracking benchmark that contains 98 fully annotated sequences with 100 different target objects. Each of the $\mathit{Skating}\mathit{2}$ sequence and $\mathit{Jogging}$ sequence contains two annotated targets. The OTB-50 dataset [18] contains 49 sequences with 50 targets selected from the OTB-100 dataset. The detail tracking results are presented in Figure 5 with precision and robustness plots. Our proposed approach outperforms most of the state-of-the-art trackers, achieving an AUC score of $70.0\%$ and a precision score of $93.6\%$.

Compared to DeepSTRCF, our approach achieves a gain of $2.3\%$ AUC score and $5.3\%$ precision score, respectively. Our approach outperforms the ASRCF by $0.8\%$ in AUC and $1.4\%$ in precision, respectively. Compared to other recently proposed deep neural network based trackers, such as ATOM [39], DiMP [40], and SiamRPN++ [8] which obtain AUC scores of 66.0%, 67.8%, and 69.7%, respectively, our method also achieves a competitive tracking performance, with a relative gain of 4%, 2.2%, and 0.3% over them.

We present the quantitative results on different attributes of OTB-100 dataset in terms of precision and success, corresponding to Figure 6 and Figure 7, respectively. The precision and success plots in Figure 6 and Figure 7 illustrate that our approach outperforms other trackers on most of the attributes, such as the precision scores on deformation, fast motion, illumination variation, out-of-plane rotation, out-of-view, and low resolution attributes, and the AUC scores on background clutter, illumination variation, out-of-plane rotation, low resolution, and out of view attributes.

From the tracking results from Figure 5, Figure 6 and Figure 7, we can conclude that our approach achieves state-of-the-art performance compared with others on OTB-100 dataset. The success plots shows that our approach estimates scale more accurately on these dataset. For example, our tracker outperforms ASRCF [11] and DeepSTRCF [6] by a gain of 2.6% and 8.4% in success score on the attribute of in-plane rotation, respectively.

VOT Datasets. VOT-2016 [19] consists of 60 short sequences selected from a large pool of 356 sequences. The sequence selection method is extended to be fully automated compared to VOT2014, and the selection process focus on those sequences that are more likely challenging for tracking. VOT-2018 dataset [20] maintains the same object classes with VOT2016 but the least challenging sequences from latter were replaced by new sequences. The evaluation criteria in VOT-2018 is same as that in VOT-2016. We compare our approach with 20 state-of-the-art trackers.

Table 2. Performance comparison among the state-of-the-art trackers on VOT 2016, and VOT 2018. The results are presented in terms of EAO, Av and Rv. The top three results are marked in red, blue, and green fonts, respectively.

Trackers	VOT2016			VOT2018
	Av	Rv	EAO	Av	Rv	EAO
Ours	0.528	0.177	0.402	0.507	0.225	0.339
ASRCF [11]	0.568	0.187	0.390	0.492	0.234	0.328
DeepSTRCF [6]	0.55	0.257	0.313	0.531	0.272	0.299
ATOM [39]	0.617	0.189	0.424	0.589	0.201	0.401
SA-SIAM [52]	0.543	0.337	0.291	0.5	0.459	0.236
CSR-DCF [29]	0.524	0.239	0.338	0.491	0.356	0.256
CFCF [53]	0.560	0.169	0.384	0.511	0.286	0.283
CCOT [7]	0.541	0.239	0.331	0.494	0.318	0.267
SiamRPN [38]	0.56	0.314	0.340	0.490	0.464	0.244
SiamRPN++[8]	0.637	0.178	0.478	0.6	0.234	0.414
ECO [12]	0.54	0.201	0.374	0.484	0.276	0.281
ECO_HC [12]	0.53	0.304	0.322	0.494	0.435	0.238
DSST[54]	0.535	0.707	0.181	0.395	1.452	0.079
LSART [51]	0.495	0.215	0.323	0.495	0.276	0.323
MEEM [50]	0.490	0.515	0.194	0.463	0.534	0.193
Staple[48]	0.547	0.379	0.295	0.530	0.688	0.169
SRDCF [4]	0.536	0.421	0.247	0.490	0.974	0.119
DeepSRDCF [5]	0.507	0.326	0.276	0.492	0.707	0.154
SiamFC [36]	0.532	0.461	0.88	0.503	0.585	0.188
SAMF [49]	0.507	0.590	0.186	0.484	1.302	0.093
KCF [13]	0.491	0.571	0.192	0.447	0.773	0.135

shows the comparison results of different trackers on VOT-2016 and VOT-2018 datasets. Our approach outperforms most of the trackers in terms of Av, Rv, and EAO metrics. Our approach obtains an EAO score of 33.9%.

In addition, our tracker outperforms ASRCF with a relative gain of 1.5% in Av and 1.1% in EAO on VOT-2018 dataset. Our approach improves the DeepSTRCF by 4% in EAO on VOT-2018 and by 8.9% in EAO on VOT-2016. Our approach also outperforms the deep Siamese trackers, such as SiamRPN [38] and SA-SIAM [52] in Rv and EAO. SiamRPN++ [8] obtains a high EAO score of 41.4%, with the Av score of 60%. The deep neural network based trackers, such as SiamRPN++ and ATOM, employ the bounding box regression branch to estimate the scale variation of the target, which promotes the IoU between estimated target state and its ground truth, comparing with the search strategy applied in the DCF trackers.

LaSOT Dataset. LaSOT [21] dataset contains 1400 sequences with 3.52 million frames and an average of 2512 frames per sequence, among which, the test set contains 280 sequences. There are 70 different object categories in the whole dataset and each category consists of 20 sequences. This dataset provides high-quality dense bounding box annotations for each sequence. Moreover, this dataset categorizes the sequences into 14 attributes, including aspect ratio change (ARC), background clutter (BC), camera motion (CM), deformation (DEF), fast motion (FM), full occlusion (FOC), illumination variation (IV), low resolution (LR), motion blur (MB), out-of-view (OV), partial occlusion (POC), rotation (ROT), scale variation (SV), and viewpoint change (VC).

As shown in Figure 8, our approach achieves the highest performance in terms both the robustness and accuracy compared to previous methods. Our approach obtains a gain of 5.7% and 9.3% in AUC over ASRCF [11] and DeepSTRCF [6], respectively. In addition, our approach obtains a gain of 6.8% and 10.1 % in precision compared with its counterparts ASRCF [11] and DeepSTRCF [6], respectively. Figure 9 and Figure 10 show the success plots and precision plots on LaSOT with different attributes.

Our approach obtained improvements for all attributes in the precision score and success score compared to ASRCF [11] and DeepSTRCF [6]. Our approach outperforms its counterparts ASRCF and DeepSTRCF by a large gain of 5.7% and 5.8% in success on aspect ratio change attribute, respectively. In addition, Our approach outperforms its counterparts ASRCF and DeepSTRCF by a gain of 7% and 5.6% in success on the deformation attribute, respectively. This illustrates the ability of our tracker to deal with challenging sequences.

Our approach still has a gap compared with the end-to-end trained deep tracker SiamRPN++ [8] and DiMP [40], which benefit from the large-scale offline training and bounding box regression network.

5.4.2. Qualitative Results

In this section, we present the qualitative tracking results of different trackers on the OTB-100, VOT-2018, and LaSOT datasets, corresponding to Figure 11, Figure 12 and Figure 13, respectively. The qualitative results illustrate that our approach is capable of tracking scale changes and appearance deformation compared with other DCF trackers. In Figure 11, we observed that the target in the $\mathit{MotorRolling}$ sequence flipped in plane, and the scale also changed dramatically over time. The STRCF [31] and ASRCF [11] both lost the target, while our approach can successfully track the target among most of the frames and obtain relative high AUC score. Furthermore, our tracker achieves promising performance and is on par with the ATOM tracker. For the example of $\mathit{Trans}$ sequence, the target transformed from a robot to a car with the severe appearance changes, as the frames $\#16$, $\#61$, and $\#117$ shown. The results of our tracker are closer to the ground truth than other trackers.

As shown in Figure 12, our tracker outperforms the DCF trackers and deep Siamese tracker SiamRPN++ [8] in frame $\#119$ and $\#145$ in sequence $\mathit{fish}\mathit{1}$ when the posture of fish changes significantly. In $\mathit{bag}$ sequence, our approach can adjust the width and height properly to adapt to the deformation of $\mathit{bag}$. In Figure 13, our approach can still capture the shape of the dog in frame $\#209$ and frame $\#230$, while most of the other DCF trackers have lost the target or merely tracked the part of dog. For the $cat$-3 sequence, only our approach and SiamRPN++ [8] can successfully track the target object, which also illustrates the effectiveness of our method.

The OTB-100 dataset contains more sequences that with rigid target objects, such as car, box, and football. While VOT-2018 and LaSOT contain more sequences with non-rigid target objects. Our approach yields favorable performance on OTB-100 dataset with more stable sequences. However, the end-to-end trained deep trackers exhibits better performance on sequences with more deformable targets. The deep trackers employ deep convolutional network as the backbone to extract features, while our tracker utilizes both the HOG and deep features for correlation filters learning. SiamRPN++ [8] tends to lose the target which was occluded by similar distractor, as the $\mathit{Walking}\mathit{2}$ sequence shown in Figure 11.

6. Conclusions

In this paper, we proposed a new visual object tracking algorithm based on discriminative correlation filters. Our approach learns the discriminative correlation filters (DCFs) via not only adaptive spatial regularization but also temporal regularization for estimating the target state. A 2-dimensional scale space search method was proposed to adapt to appearance variations of the target object and to improve the accuracy of the tracking results.

From the ablation study on OTB-100, we found that the features from deeper neural networks can improve the robustness of the tracker. We also analyzed the impact of different learning rates on learning process of the target model. Comprehensive experiments and analysis were performed on the OTB-100, VOT2016, VOT2018, and LaSOT datasets. The qualitative and quantitative results illustrated the effectiveness of our tracking framework.

Our tracking framework outperformed most of the existing state-of-the-art DCF trackers on challenging sequences and achieved competitive performance compared with end-to-end deep trackers. The results on challenging sequences of three datasets further demonstrate that our novel scale space search method can improve the accuracy of scale estimation.

In future research, we could apply trainable end-to-end deep convolutional neural networks to represent the target during learning of DCFs in the future. An interesting direction for future work is to incorporate bounding box regression technologies in many deep learning based detectors, such as [55], to further improve the performance.