Visual Object Tracking Based on Cross-Modality Gaussian-Bernoulli Deep Boltzmann Machines with RGB-D Sensors

Abstract

Visual object tracking technology is one of the key issues in computer vision. In this paper, we propose a visual object tracking algorithm based on cross-modality featuredeep learning using Gaussian-Bernoulli deep Boltzmann machines (DBM) with RGB-D sensors. First, a cross-modality featurelearning network based on aGaussian-Bernoulli DBM is constructed, which can extract cross-modality features of the samples in RGB-D video data. Second, the cross-modality features of the samples are input into the logistic regression classifier, andthe observation likelihood model is established according to the confidence score of the classifier. Finally, the object tracking results over RGB-D data are obtained using aBayesian maximum a posteriori (MAP) probability estimation algorithm. The experimental results show that the proposed method has strong robustness to abnormal changes (e.g., occlusion, rotation, illumination change, etc.). The algorithm can steadily track multiple targets and has higher accuracy.

1. Introduction

Visual object tracking is one of the key research topics in the field of computer vision. In recent years, it has had a wide range of applications, such as robot navigation, intelligent video surveillance, andvideo measurement [1,2,3,4]. Despite many research efforts, visual object tracking is still regarded as a challenging problem due to changes in object appearance, occlusions, complex motion, illumination variation and background clutter [5].

A typical visual object tracking algorithm often includes three major components: a state transition model, an observation likelihood model and a search strategy. A state transition model is used to model the temporal consistency of the states of a moving object, whereas an observation likelihood model describes the object and observations based on visual representations. Undoubtedly, feature representation is the most important factor in visual object tracking. Most of existing RGB-D trackers [6,7,8] tend to use hand-crafted features to represent target objects, such as Harr-like features [9], histogram of oriented gradients (HOG) [10], and local binary patterns (LBP) [11]. Hand-crafted features aim to describe some pre-defined image patterns, but they cannot capture thecomplex and specific characteristics of target objects. Hand-crafted features may lead to the loss of unrecoverable information which is suitable for tracking in different scenarios.

With the rapiddevelopment of computation power and the emergence of large-scale visual data, deep learning has received much attention and had a promising performance in computer vision tasks, e.g., object tracking [12], object detection [13], and image classification [14]. Wang et al. proposed a so-called deep learning tracker (DLT) for robust visual tracking [15]. DLT trackers learn generic features from auxiliary natural images offline. ADLT tracker cannot obtain deep features with temporal invariance, which is important for visual object tracking. In [16], the authors proposed a video tracking algorithm using learned hierarchical features in which the hierarchical features are learned via a two-layer convolutional neural network. Ding et al. [17] proposed a new tracking–learning–data architecture to transfer a generic object tracker to a blur invariant object tracker without deblurring image sequences. One of the research focuses of this paper is how to use deep learning effectively to extract the features of the target objects in RGB-D data.

To the best of our knowledge, the existing visual tracking methods using deep learning follow a similar procedure, which tracks objects in 2D sequences. Object tracking is performed over 2D video sequences in most early research works like TLD tracker [18], MIL tracker [19] and VTD tracker [20]. With the great popularity of affordable depth sensors, such as Kinect, Asus Xtion, and PrimeSense, an explosive growth of RGB-D data that can be used nowadays has been seen. Reliable depth images can provide valuable information to improve tracking performance. In [21], the author establishesa unified benchmark dataset of 100 RGB-D videos, which provide a foundation for further research in both RGB and RGB-D tracking. One of theresearch focuses of this paper is how to fuse RGB information and depth information effectively to improve the performance of visual object tracking in RGB-D data.

To overcome the problems in the existing methods, we propose a visual object tracking algorithm based on cross-modality feature learning using Gaussian-Bernoulli deep Boltzmann machines (DBM) over RGB-D data. A cross-modality deep learning framework is usedto learn a robust tracker forRGB-D data. The cross-modality features of the samples are input into the logistic regression classifier, andthe observation likelihood model is established according to the confidence score of the classifier. We obtain the object tracking results over RGB-D data using aBayesian maximum a posteriori probability estimation algorithm. Experimental results show that such a cross-modality learning can improve the tracking performance.

The main contributions of this paper can be summarized as follows:

• We present a cross-modality Gaussian-Bernoulli deep Boltzmann machine (DBM) to learn the cross-modality features of target objects in RGB-D data. The proposed cross-modality Gaussian-Bernoulli DBM is constructed with two single-modality Gaussian-Bernoulli DBMs by adding an additional layer of binary hidden units on top of them, which can fuse RGB information and depth information effectively.
• A unified RGB-D tracking framework based on Bayesian MAP is proposed, in which the robust appearance description with cross-modality features deep learning, temporal continuity is fully considered in the state transition model.
• Extensive experiments are conducted to compare our tracker with several state-of-the-art methods on the recent benchmark dataset [21]. From experimental results, we can see that the proposed tracker performs favorably against the compared state-of-the-art trackers.

The remainder of the paper is organized as follows. First, feature learning over RGB-D data with cross-modality deep Boltzmann machines is described in the next section. Then we introduce our tracking framework in Section 3. The implementation of our proposed method is presented in Section 4. Experimental results and analysis are demonstrated in Section 5, and finally we draw conclusions in Section 6.

2. Related Work

2.1. Boltzmann Machine

TheBoltzmann machine (BM) was proposed by Hinton and Sejnowski [22]. A Boltzmann machine is a feedback neural network consisting of fully connected coupled random neurons. The connections between neurons are symmetric, and there is no self-feedback. The outputs of neurons only have two states (active and inactive) which are expressed by 0 and 1, respectively. A set of visible units $v\in {\{0,1\}}^D$ and a set of hidden units $h\in {\{0,1\}}^F$ are included in BM (as shown in Figure 1). The visible units and hidden units are composed ofthe visible nodes and hidden nodes, and D and F represent the number of visible nodes and hidden layer nodes, respectively.

Figure 1. A general Boltzmann machine.

We formulate the energy function over the state $\{v,h\}$ as:

$$E(v,h;\mathrm{\Psi })=-v^{\prime }Wh-\frac{1}{2}{h}^{\prime }Rh-\frac{1}{2}{v}^{\prime }Lv-v^{\prime }B-h^{\prime }A$$

(1)

where $\mathrm{\Psi }=\{W,L,R,B,A\}$ are the model parameters: $W,L,R$ represent the symmetric interaction terms of visible nodes to hidden nodes, visible nodes to visible nodes, and hidden nodes to hidden nodes. The diagonal elements of $L$ and $R$ are set to 0. $B$ and $A$ are the threshold values of the visible layer and the hidden layer.

The model defines a probability distribution over a visible vector $v$ as:

$$P(v;\mathrm{\Psi })=\frac{P^{*}(v;\mathrm{\Psi })}{Z(\mathrm{\Psi })}=\frac{1}{Z(\mathrm{\Psi })}{\displaystyle \sum _{h}exp(-E(v,h;\mathrm{\Psi }))}$$

(2)

where $Z(\mathrm{\Psi })={\displaystyle \sum _v{\displaystyle \sum _{h}exp(-E(v,h;\mathrm{\Psi }))}}$ is called the partition function, and $P^{*}$ is an unnormalized probability.

The following formulations give the conditional distributions over hidden and visible units:

$$P(h_j=1|v,h_{-j})=\sigma ({\displaystyle \sum _{i=1}^D{W}_{ij}{v}_i+{\displaystyle \sum _{m=1\backslash j}^P{J}_{jm}{h}_j}})$$

(3)

$$P(v_i=1|h,v_{-i})=\sigma ({\displaystyle \sum _{j=1}^P{W}_{ij}{h}_j+{\displaystyle \sum _{k=1\backslash i}^D{J}_{ik}{v}_i}})$$

(4)

where $\sigma (x)=1/(1+exp(-x))$ is the logistic function.

2.2. Restricted Boltzmann Machine

Setting both $L=0$ and $R=0$ in Equation (1), we will recover the model of a restricted Boltzmann machine (RBM), as shown in Figure 2.

Figure 2. A restricted Boltzmann machine.

A restricted Boltzmann machine(RBM) is a generative stochastic artificial neural networkthat can learn a probability distribution over its set of inputs. It is an undirected graphical model with each visible unit only connected to each hidden unit. The energy function over the visible and hidden units.

$$E(v,h;\mathrm{\Psi })=-v^{\prime }Wh-v^{\prime }B-h^{\prime }A$$

(5)

where $E:{\{0,1\}}^{D+F}\to \mathrm{ℝ}$, $\mathrm{\Psi }=\{W,A,B\}$ are the model parameters. Equation (6) defines the joint probability distribution over the visible units $v\in {\{0,1\}}^D$ and hidden units $h\in {\{0,1\}}^F$.

$$P(v,h;\mathrm{\Psi })=\frac{1}{Z(\mathrm{\Psi })}exp(-E(v,h;\mathrm{\Psi }))$$

(6)

where the normalizing factor $Z(\mathrm{\Psi })$ denotes the partition function.

2.3. Gaussian-Bernoulli Restricted Boltzmann Machines

When inputs are real-valued images, we formulate the energy function of the Gaussian-Bernoulli RBM over the state $\{v,h\}$ as follows [23]:

$$E(v,h;\mathrm{\Psi })=-{\displaystyle \sum _{i=1}^D\frac{{(v_i-b_i)}^2}{2{\sigma }_i^2}}-{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{j=1}^F{W}_{ij}{h}_j\frac{v_i}{{\sigma }_i}}-}{\displaystyle \sum _{j=1}^F{a}_j{h}_j}$$

(7)

where $\mathrm{\Psi }=\{a,b,W,\sigma \}$ are the model parameters, $b_i$ and $a_j$ are biases corresponding to visible and hidden variables, respectively, $W_{ij}$ is the matrix of weights connecting visible and hidden nodes, and ${\sigma }_i$ is the standard deviation associated with a Gaussian visible variable $v_i$.

2.4. Gaussian-Bernoulli Deep Boltzmann Machine

A deep Boltzmann machine (DBM) [24] contains a set of visible units $v\in {\{0,1\}}^D$, and a sequence of layers of hidden units $h^1\in {\{0,1\}}^{L_1}$, $h^2\in {\{0,1\}}^{L_2}$, …, $h^N\in {\{0,1\}}^{L_N}$. Connections only exist between hidden units in adjacent layers. We illustrate a two-layer Gaussian-Bernoulli deep Boltzmann machine, consisting of learning a stack of modified Gaussian-Bernoulli RBMs (see Figure 3).

Figure 3. A Gaussian-Bernoulli Deep Boltzmann Machine.

The energy function of the joint configuration $\{v,h^{(1)},h^{(2)}\}$ is formulated as:

$$E(v,h^{(1)},h^{(2)};\mathrm{\Psi })=-v^{\prime }{W}^{(1)}{h}^{(1)}-h^{{(1)}^{\prime }}{W}^{(2)}{h}^{(2)}$$

(8)

where $\mathrm{\Psi }=\{W^{(1)},W^{(2)}\}$ are the model parameters, and $h=\{h^{(1)},h^{(2)}\}$ denote the set of hidden units. The probability distribution over a visible vector $v$ can be modelled as:

$$P(v;\mathrm{\Psi })=\frac{1}{Z(\mathrm{\Psi })}{\displaystyle \sum _{h^{(1)},h^{(2)}}exp(-E(v,h^{(1)},h^{(2)};\mathrm{\Psi })})$$

(9)

3. Proposed Tracking framework

3.1. Feature Learning UsingCross-Modality Deep Boltzmann Machines over RGB-D Data

ABoltzmann machine (BM) is an effective tool in representing probability distribution over its inputs. Deep Boltzmann Machines (DBMs) have been successfully used in many application domains, e.g., topic modelling, classification, dimensionality reduction, feature learning, etc. According to the task, DBMs can be trained in either unsupervised or supervised ways. In this paper, we propose the cross-modality DBMs for feature learning in visual tracking over RGB-D data. In this section, we first describe how to establish cross-modality DBMs, review BMs, RBMs and Gaussian-Bernoulli restricted Boltzmann machines, then go over them in detail.

Multimodal deep learning was proposed forvideo and audio [25,26]. In RGB-D data, we can also learn deep features over multiple modalities (RGB modality and depth modality). The proposed cross-modality DBM is constructed with two single-modality Gaussian-Bernoulli DBMs by adding an additional layer of binary hidden units on top of them (see Figure 4). Firstly, we model a RGB-specific Gaussian-Bernoulli DBM with two hidden layers as Figure 4a, where $v^{RGB}\in {\mathrm{ℝ}}^D$ denotes a real-valued image input. Let $h^{\left(1RGB\right)}\in {\{0,1\}}^{F_1^{RGB}}$ and $h^{\left(2RGB\right)}\in {\{0,1\}}^{F_2^{RGB}}$ be the two layers of hidden units in the RGB-specific DBM. Then, the energy function of Gaussian-Bernoulli DBM over $\{v^{RGB},h^{RGB}\}$ is defined as:

$$\begin{array}{r}E(v^{RGB},h^{(1RGB)},h^{(2RGB)};{\mathrm{\Psi }}^{RGB})={\displaystyle \sum _{i=1}^D\frac{{(v_i^{(RGB)}-b_i^{(RGB)})}^2}{2{\sigma }_i^{{(RGB)}^2}}}-{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{j=1}^{F_1^{RGB}}\frac{v_i^{(RGB)}}{{\sigma }_i^{(RGB)}}{W}_{ij}^{(1RGB)}}}{h}_j^{(1RGB)}\\ -{\displaystyle \sum _{j=1}^{F_1^{RGB}}{\displaystyle \sum _{l=1}^{F_2^{RGB}}{W}_{jl}^{(2RGB)}{h}_j^{(1RGB)}{h}_l^{(2RGB)}}}-{\displaystyle \sum _{j=1}^{F_1^{RGB}}{a}_j^{(1RGB)}{h}_j^{(1RGB)}}-{\displaystyle \sum _{l=1}^{F_2^{RGB}}{a}_l^{(2RGB)}{h}_l^{(2RGB)}}\end{array}$$

(10)

where ${\sigma }_i^{(RGB)}$ is thedeviation of the corresponding Gaussian model, and ${\mathrm{\Psi }}^{RGB}$ is the parameter vector of RGB-specific Gaussian-Bernoulli DBM. Therefore, the joint distribution of the energy-based probabilistic model is defined through an energy function as:

$$P(v^{RGB},h^{RGB};{\mathrm{\Psi }}^{RGB})=\frac{1}{Z({\mathrm{\Psi }}^{RGB})}{\displaystyle \sum _{h^{RGB}}exp(-E(v^{RGB},h^{RGB};{\mathrm{\Psi }}^{RGB}))}$$

(11)

where $Z({\mathrm{\Psi }}^{RGB})$ is the partition function.

Figure 4. The illustration of the proposed cross-modal Gaussian-Bernoulli DBM. (a) RGB-specific two-layer Gaussian-Bernoulli DBM; (b) Depth-specific two-layer Gaussian-Bernoulli DBM; (c) a Cross-modal Gaussian-Bernoulli DBM.

Similarly, the corresponding probability assigned to $v^{Depth}$ by Depth-specific DBM has the same form with Equation (11). Let $v^{Depth}\in {\mathrm{ℝ}}^K$ denotes a real-valued depth image input. Let $h^{(1Depth)}\in {\{0,1\}}^{F_1^{Depth}}$ and $h^{(2Depth)}\in {\{0,1\}}^{F_2^{Depth}}$ be the two layers of hidden units in the Depth-specific DBM, as show in Figure 4b. The energy of the Gaussian-Bernoulli DBM and the joint distribution of the energy-based probabilistic model over $\{v^{Depth},h^{Depth}\}$ are defined as:

$$\begin{array}{r}E(v^{Depth},h^{(1Depth)},h^{(2Depth)};{\mathrm{\Psi }}^{Depth})={\displaystyle \sum _{i=1}^D\frac{{(v_i^{(Depth)}-b_i^{(Depth)})}^2}{2{\sigma }_i^{{(Depth)}^2}}}-{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{j=1}^{F_1^{Depth}}\frac{v_i^{(Depth)}}{{\sigma }_i^{(Depth)}}{W}_{ij}^{(1Depth)}}}{h}_j^{(1Depth)}\\ -{\displaystyle \sum _{j=1}^{F_1^{Depth}}{\displaystyle \sum _{l=1}^{F_2^{Depth}}{W}_{jl}^{(2Depth)}{h}_j^{(1Depth)}{h}_l^{(2Depth)}}}-{\displaystyle \sum _{j=1}^{F_1^{Depth}}{a}_j^{(1Depth)}{h}_j^{(1Depth)}}-{\displaystyle \sum _{l=1}^{F_2^{Depth}}{a}_l^{(2Depth)}{h}_l^{(2Depth)}}\end{array}$$

(12)

$$P(v^{Depth},h^{(1Depth)},h^{(2Depth)};{\mathrm{\Psi }}^{Depth})=\frac{1}{Z({\mathrm{\Psi }}^{Depth})}{\displaystyle \sum _{h^{(1Depth)(2Depth)}}exp(-E(v^{Depth},h^{(1Depth)},h^{(2Depth)};{\mathrm{\Psi }}^{Depth}))}$$

(13)

where ${\sigma }_i^{(Depth)}$ is deviation of the corresponding Gaussian model, and ${\mathrm{\Psi }}^{Depth}$ is the parameter vector of Depth-specific Gaussian-Bernoulli DBM.

Let $v^{RGB}\in {\mathrm{ℝ}}^D$ and $v^{Depth}\in {\mathrm{ℝ}}^K$ denote a real-valued RGB input and a real-valued depth input respectively. Consider modeling an image-depth DBM with three hidden layers, let $\{v^{RGB},v^{Depth}\}$ be real-valued Gaussian variables, and $\{h^{(1RGB)},h^{(2RGB)},h^{(1Depth)},h^{(2Depth)},h^{\left(3\right)}\}$ be binary stochastic hidden units. Let $h^{(1RGB)}\in {\{0,1\}}^{F_1^{RGB}}$ and $h^{(2RGB)}\in {\{0,1\}}^{F_2^{RGB}}$ be the two layers of hidden units in the RGB-specific two layer DBM. Similarly, let $h^{(1Depth)}\in {\{0,1\}}^{F_1^{Depth}}$ and $h^{(2Depth)}\in {\{0,1\}}^{F_2^{Depth}}$ be the two layers of hidden units in the depth-specific two layer DBM. The energy of the proposed cross-modality Gaussian-Bernoulli DBM over $\{v,h\}$ can be defined as:

$$\begin{array}{l}E(v,h;{\mathrm{\Psi }}^{cross-modality})={\displaystyle \sum _{i=1}^D\frac{{(v_i^{(RGB)}-b_i^{(RGB)})}^2}{2{\sigma }_i^{{(RGB)}^2}}}-{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{j=1}^{F_1^{RGB}}\frac{v_i^{(RGB)}}{{\sigma }_i^{(RGB)}}{W}_{ij}^{(1RGB)}}}{h}_j^{(1RGB)}\\ -{\displaystyle \sum _{j=1}^{F_1^{RGB}}{\displaystyle \sum _{l=1}^{F_2^{RGB}}{h}_j^{(1RGB)}{W}_{jl}^{(2RGB)}{h}_l^{(2RGB)}}}-{\displaystyle \sum _{l=1}^{F_2^{RGB}}{\displaystyle \sum _{p=1}^{F_3^{RGB}}{h}_j^{(2RGB)}{W}_{lp}^{(3RGB)}{h}_p^{(3RGB)}}}-{\displaystyle \sum _{j=1}^{F_1^{RGB}}{a}_j^{(1RGB)}{h}_j^{(1RGB)}}\\ -{\displaystyle \sum _{l=1}^{F_2^{RGB}}{a}_l^{(2RGB)}{h}_l^{(2RGB)}}+{\displaystyle \sum _{i=1}^K\frac{{(v_i^{(Depth)}-b_i^{(Depth)})}^2}{2{\sigma }_i^{{(\mathrm{Depth})}^2}}}-{\displaystyle \sum _{i=1}^K{\displaystyle \sum _{j=1}^{F_1^{Depth}}\frac{v_i^{(Depth)}}{s_i^{(Depth)}}{W}_{ij}^{(1Depth)}}}{h}_j^{(1Depth)}\\ -{\displaystyle \sum _{j=1}^{F_1^{Depth}}{\displaystyle \sum _{l=1}^{F_2^{Depth}}{h}_j^{(1Depth)}{W}_{jl}^{(2Depth)}{h}_l^{(2Depth)}}}-{\displaystyle \sum _{l=1}^{F_2^{Depth}}{\displaystyle \sum _{p=1}^{F_3^{Depth}}{h}_j^{(2Depth)}{W}_{lp}^{(3Depth)}{h}_p^{(3Depth)}}}-{\displaystyle \sum _{j=1}^{F_1^{Depth}}{a}_j^{(1Depth)}{h}_j^{(1Depth)}}\\ -{\displaystyle \sum _{l=1}^{F_2^{Depth}}{a}_l^{(2Depth)}{h}_l^{(2Depth)}}-{\displaystyle \sum _{p=1}^{F_3}{a}_p^{(3)}{h}_p^{(3)}}\end{array}$$

(14)

Therefore, the joint probability distribution over the cross-modal input $\{v^{RGB},v^{Depth}\}$ can be written as:

$$\begin{array}{l}P(v^{RGB},v^{Depth};{\mathrm{\Psi }}^{cross-modality})={\displaystyle \sum _{h^{(2RGB)},h^{(2Depth)},h^{(3)}}P(h^{(2RGB)},h^{(2Depth)},h^{(3)})}({\displaystyle \sum _{h^{(1RGB)}}P(v^{RGB},h^{(1RGB)},h^{(2RGB)})}\\ ({\displaystyle \sum _{h^{(1Depth)}}P(v^{Depth},h^{(1Depth)},h^{(2Depth)})}\\ =\frac{1}{Z({\mathrm{\Psi }}^{cross-modality})}{\displaystyle \sum _{h}exp(-{\displaystyle \sum _i\frac{{(v_i^{RGB})}^2}{2s_i^2}}}+{\displaystyle \sum _{ij}\frac{v_i^{(RGB)}}{s_i}{W}_{ij}^{(1RGB)}}{h}_j^{(1RGB)}+{\displaystyle \sum _{jl}{W}_{jl}^{(2RGB)}{h}_j^{(1RGB)}{h}_l^{(2RGB)}}\\ -{\displaystyle \sum _i\frac{{(v_i^{Depth})}^2}{2s_i^2}}+{\displaystyle \sum _{ij}\frac{v_i^{(Depth)}}{s_i}{W}_{ij}^{(1Depth)}}{h}_j^{(1Depth)}+{\displaystyle \sum _{jl}{W}_{jl}^{(2Depth)}{h}_j^{(1Depth)}{h}_l^{(2Depth)}}\\ +{\displaystyle \sum _{lp}{W}^{(3RGB)}{h}_l^{(2RGB)}{h}_p^{(3)}}+{\displaystyle \sum _{lp}{W}^{(3Depth)}{h}_l^{(2Depth))}{h}_p^{(3)}})\end{array}$$

(15)

where ${\mathrm{\Psi }}^{cross-modality}$ is the parameter vector of cross-modality Gaussian-Bernoulli DBM. The task of learning the cross-modality Gaussian-Bernoulli DBM is the maximum likelihood learning for Equation (6) with respect to the model parameters.

3.2. Bayesian Framework

In this paper, the object tracking is formulated as a hidden state variable Bayesian maximum a posteriori (MAP) estimation problem in the Hidden Markov model. Given a set of observed variables $Z_t=\{{\mathrm{Z}}_1,{\mathrm{Z}}_2,\dots ,{\mathrm{Z}}_t\}$, we can estimate the hidden state variable $X_t=\left\{X_t^1,X_t^2,\dots \dots X_t^N\right\}$ by using Bayesian MAP theory [27].

The posteriori probability distribution according to the Bayesian theory can be modelled as the following derivation:

$$p(X_t|Z_t)\propto p(Z_t|X_t){\displaystyle \int p(X_t|X_{t-1})}p(X_{t-1}|Z_{t-1})d{X}_{t-1}$$

(16)

where $p(Z{}_t|X_t)$ stands for an observation likelihood model and $p(X_t|X_{t-1})$ is called a state transition model for two consecutive frames. We can obtain the optimal state ${\widehat{X}}_t$ among all the candidates through maximum posterior probability estimation:

$${\widehat{X}}_t=\underset{X_t}{\arg \max }p(X_t|Z_t)$$

(17)

3.2.1. State Transition Model

The state variable is defined as $X_t=\{x_t,y_t,{\theta }_t,s_t,{\alpha }_t,{\varphi }_t\},$ which includes the six parameters of the motion affine transformation, where $x_t$ and $y_t$ denote the x-direction and y-direction translation of the object in the frame t respectively, ${\theta }_t$ represents the rotation angle, $s_t$ stands for the scale change, ${\alpha }_t$ denotes the aspect ratio, and ${\varphi }_t$ represents skew direction at time t.

We assume that the candidate states are generated according to Gaussian distribution:

$$p(X_t|X_{t-1}) = N(X_t;X_{t-1},\Sigma )$$

(18)

where $\Sigma$ is a diagonal covariance matrix whose diagonal elements are ${\sigma }_x^2,{\sigma }_y^2,{\sigma }_{\theta }^2,{\sigma }_S^2,{\sigma }_{\alpha }^2,{\sigma }_{\varphi }^2.$

3.2.2. Observation Likelihood Model

In this paper, the observation model that we use is discriminative. A binary linear classifier is adopted to classify tracking observations into object class and background class during tracking. Observations are represented using features learned from the DBM introduced previously. We can obtain a training dataset with approximate labels after extracting features of positive and negative samples. Deep representations are likely to be linearly separable, and linear classifiers are less prone to overfitting. We adopt the logistic regression classifier owing to its capability of providing predictions in probability estimation.

Let $h_i^3\in {\mathrm{ℝ}}^{r\times 1}$ denote the deep feature for the i-th training sample, and $y_i\in \{-1,+1\}$ represent the label for the i-th training sample. $Z^{+}=[h_{1^{+}}^3,h_{2^{+}}^3,\dots ,h_{D^{+}}^3]\in {\mathrm{ℝ}}^{r\times D^{+}}$ stands for the positive training set with their respective labels as $Y^{+}=[y_{1^{+}},y_{2^{+}},\dots ,y_{D^{+}}]\in {\{-1,+1\}}^{D^{+}\times 1}$. Similarly, $Z^{-}=[h_{1^{-}}^3,h_{2^{-}}^3,\dots ,h_{D^{-}}^3]\in {\mathrm{ℝ}}^{r\times D^{-}}$ represents the negative training set with their respective labels as $Y^{-}=[y_{1^{-}},y_{2^{-}},\mathrm{...},y_{D^{-}}]\in {\{-1,+1\}}^{D^{-}\times 1}$. Training the logistic regression classifier by optimizing:

$$\underset{\pm w}{\min } C^{+}{\displaystyle \sum _{i^{+}=1}^{D^{+}}\log (1+e^{y_{i^{+}}\pm w^T\pm h_{i^{+}}^{(3)}})}+C^{-}{\displaystyle \sum _{i^{-}=1}^{D^{-}}\log (1+e^{y_{i^{-}}\pm w^T\pm h_{i^{-}}^{(3)}})}$$

(19)

where $C^{+}\in \mathrm{ℝ}$ is the parameter to weight the logistic cost of the positive-class and $C^{-}\in \mathrm{ℝ}$ is the parameter to weight the logistic cost of the negative-class logistic. Weight regularization $w$ is added to the cost function in Equation (19) to reduce overfitting. In the prediction stage, the confidence score of the trained logistic regression classifier can be computed as follows:

$$p(Z_t|X_t)=\frac{1}{1+e^{-(\pm w^T\pm z_t)}}$$

(20)

4. The Implementation of Our Proposed Method

Our method has two major components, which are shown in Figure 5 and Figure 6. In the first place, as demonstrated in Figure 5, unlabeled patches in RGB and depth modality are used to train the cross-modality Gaussian-Bernoulli DBM offline.

Figure 5. Offline learning of the proposed cross-modality Gaussian-Bernoulli DBM.

Figure 6. The process of object tracking online based on Bayesian MAP.

Then, the trained cross-modality Gaussian-Bernoulli DBM is transferred to an observational model for visual tracking online based on Bayesian MAP, as shown in Figure 6.

5. Experimental Results and Analysis

The experiments of our proposed tracking algorithm is implemented on MATLAB R2014a, Intel(R) Core(TM) i7-4712MQ, CPU@3.40 GHz and TITAN GPU, 8.00 GB RAM, Windows 8.1 operating system, in Beijing, China.

5.1. Qualitative Evaluation

In order to show the robustness of the visual object tracking algorithm discussed in this paper, we compare our tracker with several state-of-the-art methods on arecent benchmark dataset [21] in different environments with heavy or long-time partial occlusion, rotation, scale change, and fast motion. Given the limited space, in this section we only list four of them to show the experimental results and the forms of data statistics.

We compare our method with several state-of-the-art trackers, including TLD Tracker[18], MIL Tracker [19],VTD Tracker [20], and RGB-D Tracker [28], CT Tracker [29], Struck Tracker [30], Deep Tracker [15], and Multi-cues Tracker [31],andwe ranthe experiments based on the code provided by the authors.

Figure 7 demonstrates that our method performs well in terms of rotation, scale and position when the object undergoes severe occlusion. The MIL tracker and VTD tracker are sensitive to occlusion.

Figure 7. The tracking results on the test video 1 obtained by different methods.

Figure 8 shows the tracking results in the sequence with long-time partial occlusion, pose change and background clutter. We can see that the RGBD, MIL and VTD methods do not perform well and they are less effective in this case.

Figure 8. The tracking results on the test video 2 obtained by different methods.

Figure 9 illustrates the tracking results on the test video with severe occlusion, appearance change and fast motion. From the results, we can notice that the TLD, MIL and VTD methods are sensitive to target appearance change or occlusion.

Figure 9. The tracking results on the test video 3 obtained by different methods.

Figure 10 shows the tracking results in the sequence with all occlusion, pose change and background clutter. We can see that the Stuck, MIL and VTD methods do not perform well and they are less effective in this case.

Figure 10. The tracking results on the test video 4 obtained by different methods.

Figure 11 illustrates the “bad” tracking results of our method,meaning frames where tracking failures are observed. When the objects are all occluded, the tracking results of our method experience a drift phenomenon.

Figure 11. The“bad” tracking results of our method. (a) Frame 47 in test video 1; (b) Frame 93 in test video 1; (c) Frame 66 in test video 3; (d) Frame 37 in test video 3.

As shown in experimental results, the proposed tracking method performs favorably against the state-of-the-art tracking methods in handling challenging video sequences, but there are some limitations for our method. The robustness of the proposed tracking method is not strong enough to solve allocclusion and abrupt movement.

5.2. Quantitative Evaluation

We use two measurements to quantitatively evaluate tracking performances. The first one is called average center location error [32] which measures distances of centers between tracking results and ground truths in pixels. The second one is called success rate (SR) which is calculated according to $\frac{area(R_T\cap R_G)}{area(R_T\cup R_G)}$ and indicates theextent of region overlapping between tracking results $R_T$ and $R_G$.

Figure 12, Figure 13, Figure 14 and Figure 15 report the average center location errors of different tracking methods over three test videos. The comparison results show that the proposed method has a smaller average center location error than the state-of-the-art methods indifferent situations.

Figure 12. Quantitative evaluation in terms of average center location error (in pixel) for the first experiment.

Figure 13. Quantitative evaluation in terms of average center location error (in pixel) for the second experiment.

Figure 14. Quantitative evaluation in terms of average center location error (in pixel) for the third experiment.

Figure 15. Quantitative evaluation in terms of average center location error (in pixel) for the fourth experiment.

Table 1 reports the success rates, where larger scores mean more accurate results.

Table 1. The evaluation results of SR under different categorizations.

Method	Object Type		Movement		Occlusion
	Human	Animal	Fast	Slow	Yes	No
Our Tracker	80.1%	72.9%	77.5%	82.3%	81.2%	82.6%
TLD Tracker	29.0%	35.1%	29.7%	51.6%	33.8%	38.7%
VTD Tracker	30.9%	48.8%	37.2%	57.3%	28.3%	63.1%
MIL Tracker	32.2%	37.2%	31.5%	45.5%	25.6%	49.0%
RGB-D Tracker	47.1%	47.0%	51.8%	56.7%	46.9%	61.9%
Struck Tracker	35.4%	47.0%	39.0%	58.0%	30.4%	63.5%
CT Tracker	31.1%	46.7%	31.5%	48.6%	34.8%	46.8%
Deep Tracker	72.1%	64.8%	70.1%	76.3%	71.4%	72.6%
Multi-cues Tracker	33.2%	49.5%	52.3%	55.6%	44.7%	57.5%

Table 2 lists the average speed of each method on the recent benchmark dataset [21]. The average speed of our method is 0.14 fps, implemented in Matlab without optimization for speed. The fine-tuning of our method is time-consuming.

Table 2. The average speed of each method on the recent benchmark dataset [21].

Method	The Average Speed (fps)
Our Tracker	0.14
TLD Tracker	28.5
VTD Tracker	6.7
MIL Tracker	38.9
RGB-D Tracker	2.6
Struck Tracker	20.8
CT Tracker	64.7
Deep Tracker	0.23
Multi-cues Tracker	40.7

6. Conclusions

By analyzing the problems of the existing technologies, this paper proposes a visual object tracking algorithm based on cross-modality features learning using Gaussian-Bernoulli deep Boltzmann machines (DBM) over RGB-D data. We extract cross-modality features of the samples in RGB-D video data based on across-modality Gaussian-Bernoulli DBM and obtain the object tracking results over RGB-D data using aBayesian maximum a posteriori probability estimation algorithm. The experimental results show that the proposed method greatly improves the robustness and accuracy of thealgorithm. In the future, we will extend the proposed method to solve other vision problems (e.g., object detection, face recognition, etc.).