Adaptive Framework for Multi-Feature Hybrid Object Tracking

Abstract

Object tracking is a computer vision task deemed necessary for high-level intelligent decision-making algorithms. Researchers have merged different object tracking techniques and discovered a new class of hybrid algorithms that is based on embedding a meanshift (MS) optimization procedure into the particle filter (PF) (MSPF) to replace its inaccurate and expensive particle validation processes. The algorithm employs a combination of predetermined features, implicitly assuming that the background will not change. However, the assumption of fully specifying the background of the object may not often hold, especially in an uncontrolled environment. The first innovation of this research paper is the development of a dynamically adaptive multi-feature framework for MSPF (AMF-MSPF) in which features are ranked by a ranking module and the top features are selected on-the-fly. As a consequence, it improves local discrimination of the object from its immediate surroundings. It is also highly desirable to reduce the already complex framework of the MSPF to save resources to implement a feature ranking module. Thus, the second innovation of this research paper introduces a novel technique for the MS optimization method, which reduces its traditional complexity by an order of magnitude. The proposed AMF-MSPF framework is tested on different video datasets that exhibit challenging constraints. Experimental results have shown robustness, tracking accuracy and computational efficiency against these constraints. Comparison with existing methods has shown significant improvements in term of root mean square error (RMSE), false alarm rate (FAR), and F-SCORE.

1. Introduction

The increase in the computational power of existing systems has led to huge investments in automated data analysis. Object tracking is one such class of algorithms that automatically locates the region of interest, possibly obscured by challenging constraints [1]. These constraints are what defines the requirements that should be considered while developing real-time robust object tracking algorithms. Earlier methods successfully tracked objects from stationary cameras using background subtraction techniques [2]. These methods, when combined with data association techniques, can track multiple objects [3]. However, in these methods, the scene structure is known in advance. Fukunage and Hostetlar introduced a method based on meanshift (MS) estimation using gradient descent that follows a reference template/model [4]. Later Comaniciu, Ramesh, and Meer used MS to solve the problem of tracking [5] and ever since it has shown continuous presence in the computer vision community [6,7,8,9].

MS is a non-parametric method that determines the location of the object by measuring distance between the histograms of the target and candidate templates. This is achieved by maximizing a similarity measure, the Bhattacharyya coefficient, until it converges. The MS method is robust against partial occlusion and can adapt to size/scale variation under expensive mixture of Gaussian (MoG) [8,10,11]. Nevertheless, since histogram disassociates the target from the neighbourhood pixel information, it fails under the constraints such as fast object motion, clutter background, and full occlusion. This constrains the object from large displacements in consecutive frames, which as a consequence mandates some part of the object to lie in the basin of search. When the object is lost, the convergence of MS either becomes very slow or it eventually loses the object. MS also becomes slow when the size of the object increases.

On the other hand, the statistical category treats the tracking problem as a recursive computation in state space. Kalman filters (KF) and particle filters (PF) are the most popular methods in this category. KF give optimal estimation under the assumption of a linear state transition and Gaussian measurement and process noise [12]. PF, a popular statistical approach, waives off the restrictive hypothesis of KF and estimates the posterior density by combining random samples associated with weights [13,14,15,16,17,18,19,20,21]. Nummiaro et al. [16], Israd et al. [18] and Pe’rez et al. [19], are some of the most prominent research that has used color histograms with the PF method for object tracking. In [16], a method was developed that uses color with the PF method and the results are compared with the MS method and MS in combination with KF. The color-based PF method showed good performance against occlusion and non-linear object motion. The method of [18] used color and shape to represent the target. However, the method was successful in multi-object tracking, but the tracker distracts from the target when multiple people cross it. Most of these methods use global histogram for object representation and in the process the spatial information is lost. Pe’rez et al. introduced a technique that models multi-part of the target to compensate for some of the spatial information [19]. The method is robust against clutter background, size/scale variation and occlusion [19]. All these research work has proved robustness and accuracy of the PF method over MS and KF, however, the accuracy is dependent on a large number of particles. This dependency is overcome through embedding the MS optimization into the PF framework and that give rise to a new class of hybrid object tracking algorithms which will be discussed next.

2. Materials and Methods

This section discusses hybrid object tracking algorithm based on embedding MS into the PF methodology (MSPF). In this technique, the MS optimization procedure replaces the expensive and inaccurate particle validation process of the PF method. The MS optimization method reduces the particle count by finding the local modes for each particle. That essentially increases the accuracy of the particles’ state, which consequently reduces the need for a large number of particles.

There is a volume of research that approves the accuracy and robustness of MSPF based hybrid tracking methods [22,23,24,25,26,27,28,29]. Shan Caifeng combined PF and MS to track a hand for an application of intelligent wheelchair. Color and motion cues were used in a complementary fashion to offset the error of feature that is not discriminating the object [23]. The results of which are robust against clutter background and light illumination variation. In the same hybrid category, Anbang Yao introduced a particle filter based kernel object tracking (PFKBOT) algorithm. It is based on an incremental Bhattacharyya dissimilarity into the MS optimization method [24]. The idea was to continuously distinguish the background particles from the ones lying in the target region. And, consequently, the result shows robustness to background clutter and occlusion, however, at the cost of large computational power. Likewise, color is complemented with the motion information to solve the problem of background clutter and mild illumination change [26]. A Two Stage Hybrid Tracker (TSHT) was developed that uses color histograms in combination with orientation edge histograms to include the shape and inner edges of the object [27]. The results, however, fail under occlusion but nevertheless are robust towards changes in size/scale and fast motion. TSHT uses 5 MS iterations per particle to compute probable locations, therefore it does not meet the real-time requirements. Summary of the literature review is given in

Table 1. Summary of state-of-the-art hybrid object tracking algorithms.

Ref.	Search Mechanism	Features Models	Robustness Towards Various Constraints
[5]	Deterministic (Mean shift)	Color + Texture	Partial occlusion, clutter, and size/scale
[6]		Color	Partial occlusion, size and scale
[7]		Edge + Texture	Small objects
[8]		Color	Orientation scale and position
[9]		Color	Fast object motion, partial occlusion
[10]		Color	Full occlusion, size and scale
[15]	Statistical (Particle Filters)	Color	Non-rigid deformations, partial occlusions and cluttered background
[16]		Color	Clutter background, Occlusion, size/scale variation and light illumination
[17]		Color and edge orientation	Clutter background and short time period occlusion
[18]		Color and shape	Large object motion, partial occlusion
[19]		Color	Cluttered background, occlusion and size/scale variation
[28]		Motion model	Illumination variation and partial occlusion
[20]	MSPF based hybrid systems	Color + motion Cue	Fast motion, light clutter, illumination change
[21]		Color	Multiple hypothesis, clutter background
[22]		HSV Color	Clutter background, size and scale
[23]		Color + Motion model	Occlusion, clutter and fast motion
[24]		Color + motion model	Fast object motion and clutter background
[25]		Color	Background clutter, full occlusion
[26]		HSV color components	Size/Scale, fast object motion, occlusion
[27]		Color + Edge Orientation histogram	Size/Scale, fast motion
[29]		Color + Motion model	Clutter background, light illumination variation and full occlusion
[30]		Color + local integral orientation	Scale and pose variation

.

These approaches rely on a combination of predetermined fixed features neglecting the adaptive nature of the background that might be caused by the moving object. This creates an impasse when the object moves toward a background that camouflage the object with either a similar texture or any other abrupt change. Consequently, the maneuvering aspects of the object constraints the algorithm. This proposed research work takes into account the above-mentioned shortcomings and presents the following two novelties:

• The proposed AMF-MSPF framework implements a feature ranking module on top of the MSPF methodology. Thereby an adaptive multi-feature framework is implemented that selects the required features on-demand as and when required. Consequently, this enables the object tracking algorithm to discriminate the object locally. The feature ranking module re-initializes the MS procedure with new features and is triggered based on whether re-sampling occurs or not. When resampling occurs, a new set of ${\mathrm{N}}_{₣}$ features are selected using the ranking module that are used for updating the target model.
• As the PF algorithm itself is a very compute intensive method, embedding an MS into its particle validation process increases its computational load. Thereby it is pertinent that the complexity of the MS method is reduced to enable the proposed framework to run in real-time. We propose a novel MS optimization method based on an observation that MS only requires a fraction of sample to accurately track. This has led to huge reduction in computational load without inducing any significant error.

The rest of the paper is organized as follows: The mathematical formulation of the proposed AMF-MSPF framework is explained in Section 3. The experimental results are presented in Section 4. The concluding remarks along with future directions are summarized in Section 5.

3. Proposed Framework

The AMF-MSPF hybrid framework is developed in this section. In this framework, multi-features are adaptively selected from a large pool that is used by the MSPF method. Initially, a set of particles and their associated weights are generated using dynamic state equations (i.e., Equations (16) and (17)). A template of the object is initialized from these particles and processed through the ranking module to select the top ${\mathrm{N}}_{₣}$ features. The MS optimization uses these features to validate the particles by herding them to more precise locations or state. The new state of the particles reinforces PF with more accurate measurements. A mathematical formulation, as shown in the Figure 1, is developed in this subsection.

3.1. Feature Ranking

Unlike the domains of image understanding and medical applications where offline feature ranking and selection are successfully adopted, the broad spectrum of applications in the computer vision domain is directly tied to real-time processing. We have used a simple ranking criterion adopted by Collins [9] to accommodate multiple features that are considered by the tracker on the fly. The ranking module ranks a pool of features based on the variance ratio between the background and foreground. The top ranked features are input into the AMF-MSPF framework, as a result it takes care of the changing variations such as background clutter and abrupt illumination changes. In this subsection, we describe the feature ranking module. Our feature space $₣$ is formed by a linear combination of the RGB color components having coefficients ${\mathrm{c}}_1,{\mathrm{c}}_2,{\mathrm{and} \mathrm{c}}_3$:

$$₣=\{{\mathrm{c}}_1\mathrm{R}+{\mathrm{c}}_2\mathrm{G}+{\mathrm{c}}_3\mathrm{B}\}$$

(1)

where, $\mathrm{c} ϵ \left[-2-1 0 1 2\right]$ and a combination of which produces $5^3$ possible candidates in our feature space. Filtering out the redundant and useless cases, leave us with around 50 features. Let $\mathrm{fg}\left(\mathrm{i}\right)$ and $\mathrm{bg}\left(\mathrm{i}\right)$ be the normalized discrete densities of background and foreground respectively. These densities are discretized to 16 bits for efficiency. The likelihood ratio of these densities is given by Equation (2).

$$\ell \left(\mathrm{i}\right)=\log \frac{\mathrm{fg}\left(\mathrm{j}\right)}{\mathrm{bg}\left(\mathrm{j}\right)}$$

(2)

Multiple likelihood images are generated using these likelihood ratios that are to be used by the MS procedure. We have used the traditional definition of variance, $\mathrm{var}\left(\mathrm{x}\right)=\mathrm{E}{\left(\mathrm{x}\right)}^2-\mathrm{E}{\left(\mathrm{x}\right)}^2,$ to calculate the variance of $\ell \left(\mathrm{i}\right)$ w.r.t the object and background densities to maximize the inter-class variance between them. Equations (3) and (4) are used to calculate the variance:

$$\mathrm{v}\left(\ell ;\mathrm{fg}\right)=\mathrm{E}\left[{\ell }^2\left(\mathrm{j}\right)\right]-{\left(\mathrm{E}\left[\ell \left(\mathrm{j}\right)\right]\right)}^2={\displaystyle \sum }_{\mathrm{j}}\mathrm{fg}\left(\mathrm{j}\right){\ell }^2\left(\mathrm{j}\right)-{[{\displaystyle \sum }_{\mathrm{j}}\mathrm{fg}\left(\mathrm{j}\right)\ell \left(\mathrm{j}\right)]}^2$$

(3)

$$\mathrm{v}\left(\ell ;\mathrm{bg}\right)=\mathrm{E}\left[{\ell }^2\left(\mathrm{j}\right)\right]-{\left(\mathrm{E}\left[\ell \left(\mathrm{j}\right)\right]\right)}^2={\displaystyle \sum }_{\mathrm{j}}\mathrm{bg}\left(\mathrm{j}\right){\ell }^2\left(\mathrm{j}\right)-{[{\displaystyle \sum }_{\mathrm{j}}\mathrm{bg}\left(\mathrm{j}\right)\ell \left(\mathrm{j}\right)]}^2$$

(4)

In the final ranking step, Equation (5) finds the Variance Ratio ($VR$). Essentially, the inter-class variance of the foreground and the background is maximized for the feature space $₣$.

$$VR\left(\mathrm{L};\mathrm{fg},\mathrm{bg}\right)=\frac{\mathrm{v}\left(\ell ;\left(\mathrm{fg}+\mathrm{bg}\right)/2\right)}{\left[\mathrm{v}\left(\ell ;\mathrm{fg}\right)+\mathrm{var}\left(\ell ;\mathrm{bg}\right)\right]}$$

(5)

This variance ratio is sorted for the top $N_{₣}$ features which indicates the highest discrimination score. The likelihood, corresponding to features with the highest score, are used to form new likelihood images. These images are used to initialize the MS optimization procedure that produces new $N_{₣}$ locations for each particle. Symbolically Equations (1)–(5) is represented as $:N_{₣}=\mathrm{feature}Ʀ\mathrm{nk}(₣)$; the ranking module that produces $N_{₣}$ top features. The MS optimization procedure is explained in the next subsection.

3.2. MS Optimization Procedure

The MS procedure is a non-parametric gradient descent method used to move the particles’ state, $s_{\mathrm{k}+1}^{\mathrm{i}}$ to their new state ${\hat{\mathrm{s}}}_{\mathrm{k}+1}^{\mathrm{i}}$ that is more precise and accurate. MS is initialized in each $N_{₣}$ features and applied to all the particles that essentially gives us as many new locations for each particle. These $N_{₣}$ locations are merged, over each particle, using median over them. Evaluating the MS procedure for every particle using multi-features is apparently an intensive computation. However, in the proposed MS optimization technique, the complexity is reduced by an order of magnitude. The weights are estimated for each pixel of the object that is first segmented using normalized cut algorithm based on J. Shi and J. Malik [31]. We pick only a fraction of samples from the reference and candidate regions according to these estimates.

Essentially, the proposed MS operates only on a fraction of samples. This improvement reduces its complexity by an order of magnitude. This subsection describes our proposed innovation introduced in the MS optimization method. The heart of any MS procedure is the maximization of the Bhattacharryya coefficient that is given by Equation (6):

$${\mathsf{\rho }}_{1:N_{₣}}\equiv \mathsf{\rho }[p\left(\mathrm{y}\right),q]={\displaystyle \sum }_{\mathrm{u}=1}^{\mathrm{m}}\sqrt{p\left(\mathrm{y}\right).q_u}$$

(6)

The Bhattacharryya coefficient, ${\mathsf{\rho }}_{1:N_{₣}},$ is maximized over all $N_{₣}$ features. $q_u \mathrm{and} p_u$ are m-bin discrete color histograms of the target and candidates respectively. These bins are simply a series of small intervals that divides the whole range into smaller ones to reduce computational load. These densities are given from Equation (7) through Equation (8):

$$q_u=\mathrm{C}{\displaystyle \sum }_{\mathrm{i}=1}^{N_h}\mathrm{k}\left(\Vert {\mathrm{x}}_{\mathrm{i}}^{\ast }\Vert {}^2\right)\mathsf{\delta }\left[\mathrm{b}\left({\mathrm{x}}_{\mathrm{i}}^{\ast }\right)-\mathrm{u}\right]$$

(7)

$$p_u\left(\mathrm{y}\right)=\mathrm{C}{\displaystyle \sum }_{\mathrm{i}=1}^{N_h}\mathrm{k}\left({\Vert \frac{\mathrm{y}-{\mathrm{x}}_{\mathrm{i}}}{\mathrm{h}}\Vert }^2\right)\mathsf{\delta }\left[\mathrm{b}\left({\mathrm{x}}_{\mathrm{i}}\right)-\mathrm{u}\right]$$

(8)

where, $\{{{\mathrm{x}}_{\mathrm{i}}^{\ast }\}}_{\mathrm{i}=1\dots N_h}$ and $\{{{\mathrm{x}}_{\mathrm{i}}\}}_{\mathrm{i}=1\dots N_h}$ are the pixels of the target and candidates respectively. 𝛿 is the delta function equal to 1 only at the particular bin u and 0 otherwise. $N_h$ represents a fraction of random samples inside the target and candidate. In order to represent $q_u, p_u$ as densities, we multiply by a coefficient $\mathrm{C}$ to restrict the summations ${{\displaystyle \sum }}_{\mathrm{u}=1}^{\mathrm{m}}{p}_u$ and ${{\displaystyle \sum }}_{\mathrm{u}=1}^{\mathrm{m}}{q}_u$ to 1. $\mathrm{k}$ is a monotonic decreasing convex kernel. Traditionally Bhattacharryya coefficient is maximized using densities that are spatially weighted by an Epanechnikov kernel. However, these multivariate kernels are not good at dealing with non-rigid objects. We instead, employed a monotonic decreasing function to select a fraction of samples picked from the object and its immediate surroundings for spatial weighting. By using only a fraction of samples, the computational load reduces considerably. Equation (9) computes the similarity measure based on the distance between the target and candidates using a fraction of samples from them.

$$\mathrm{d}\left(\mathrm{y}\right)=\sqrt{1-\mathsf{\rho }[p\left(\mathrm{y}\right),q]}$$

(9)

The MS procedure maximizes Equation (6), which consequently minimizes Equation (9). The MS optimization process is an iterative process and is initialized in the previous frame with the target position ${\mathrm{y}}_0$. The new location ${\mathrm{y}}_1$ is evaluated based on the convergence of Equation (6). By expanding the Taylor series around coefficient $p_u\left({\mathrm{y}}_0\right)$, the linear approximation of Equation (6) is obtained after some manipulation [6,7,8]:

$${\mathsf{\rho }}_{1:N_{₣}}[p\left(\mathrm{y}\right),q_u]\approx \frac{1}{2}{\displaystyle \sum }_{\mathrm{u}=1}^{\mathrm{m}}\sqrt{p_u\left({\mathrm{y}}_0\right).q_u}+\frac{1}{2}{\displaystyle \sum }_{\mathrm{u}=1}^{\mathrm{m}}{p}_u\left(\mathrm{y}\right)\sqrt{\frac{q_u}{p_u\left({\mathrm{y}}_0\right)}}$$

(10)

y substituting Equation (8) in Equation (10), we obtain

$${\mathsf{\rho }}_{1:N_{₣}}[p\left(\mathrm{y}\right),q_u]\approx \frac{1}{2}{\displaystyle \sum }_{\mathrm{u}=1}^{\mathrm{m}}\sqrt{p_u\left({\mathrm{y}}_0\right).q_u}+\frac{\mathrm{C}}{2}{\displaystyle \sum }_{\mathrm{i}=1}^{N_h}{\mathsf{\omega }}_{\mathrm{i}}\mathrm{k}\left({\Vert \frac{\mathrm{y}-{\mathrm{x}}_{\mathrm{i}}}{\mathrm{h}}\Vert }^2\right)$$

(11)

where, ${\mathsf{\omega }}_{\mathrm{i}}={\displaystyle \sum }_{\mathrm{u}=1}^{\mathrm{m}}\sqrt{\frac{q_u}{p_u\left({\mathrm{y}}_0\right)}}\mathsf{\delta }\left[\mathrm{b}\left({\mathrm{x}}_{\mathrm{i}}\right)-\mathrm{u}\right]$.

The first term, of Equation (11), is actually the previous location of the target which is not dependent on the new coordinate y. Therefore, we only maximize the second term of the equation to get the new target location ${\mathrm{y}}_1$.

$${\mathrm{y}}_1=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{N_h}{\mathrm{x}}_{\mathrm{i}}{\mathsf{\omega }}_{\mathrm{i}}\mathrm{g}\left({\Vert \frac{{\mathrm{y}}_0-{\mathrm{x}}_{\mathrm{i}}}{\mathrm{h}}\Vert }^2\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{N_h}{\mathsf{\omega }}_{\mathrm{i}}\mathrm{g}\left({\Vert \frac{{\mathrm{y}}_0-{\mathrm{x}}_{\mathrm{i}}}{\mathrm{h}}\Vert }^2\right)}$$

(12)

where g(x) is a constant for the kernel profile that reduces Equation (12) to a simple weighted distance average as in Equation (13).

$${\mathrm{y}}_1=\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{N_h}{\mathrm{x}}_{\mathrm{i}}{\mathsf{\omega }}_{\mathrm{i}}}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{N_h}{\mathsf{\omega }}_{\mathrm{i}}}$$

(13)

Update $p\left({\mathrm{y}}_1\right)$ and evaluate ${\mathsf{\rho }}_{1:N_{₣}}[p\left(\mathrm{y}\right),q]$ until $\Vert {\mathrm{y}}_1-{\mathrm{y}}_0\Vert <Є$ for every $N_{₣}$ feature and $Є$ is usually 1 pixel. The particles along the gradient direction of the MS vector move to the local extreme of a probability density function and that determines the best location i.e., where Bhattacharyya coefficient is the highest. The MS procedure is applied to every particle over all the top $N_{₣}$ features given as in Equation (14).

$$[y_{1:N_{₣}}^{1:N_{\mathrm{s}}}]={\mathrm{MnSft}}^{1:N_{₣}}\left(s_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\right)$$

(14)

${\mathrm{y}}_{1:N_{₣}}^{1:N_{\mathrm{s}}}$ are the new $N_{₣}$ estimate locations, for each $N_{\mathrm{s}}$ particles, that are combined to form new estimates using Equation (15).

$$\left[{\hat{\mathrm{s}}}_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\right]=\mathsf{\mu }\mathrm{edian}\left(y_{1:N_{₣}}^{1:N_{\mathrm{s}}}\right)$$

(15)

${\hat{\mathrm{s}}}_{\mathrm{k}+1}$ are the new estimate of the particles that are now closer to their local maxima than $s_{\mathrm{k}+1}$. Figure 1 depicts the working of our framework. The top $N_{₣}$ features along with the initialized particles serves as the input to the MS modules. This gives us new probable locations of the object for all the selected features. Or in other words, the particles after processing through MS, gives rise to $N_{₣}$ locations for each particle. The median over all the locations gives us the particles that are more accurate i.e., ${\hat{\mathrm{s}}}_{\mathrm{k}+1}$. These new hypotheses are used to estimate the posterior density through Equation (20) in the next subsection.

3.3. MS Embedded Particle Filter

The MSPF hybrid method is based on the Bayesian framework which basically deals with the evolution of object state. The belief network is reinforced by the measurement process through a set of dynamic equations as follows:

$$s_{\mathrm{k}+1}={\mathrm{f}}_{\mathrm{k}}\left(s_{\mathrm{k}}+{ᵞ}_{\mathrm{k}}\right)$$

(16)

$$Z_{\mathrm{k}+1}={\mathrm{h}}_{\mathrm{k}}\left(s_{\mathrm{k}+1}+{\mathrm{ɳ}}_{\mathrm{k}}\right)$$

(17)

where, $s_k$ and $s_{k+1}$ are the object states at time k and k + 1. ${\mathrm{f}}_{\mathrm{k}}(.){ \mathrm{and} \mathrm{h}}_{\mathrm{k}}(.)$, are the dynamic equations for drawing particles and taking new measurements $z_{\mathrm{k}+1}. {ᵞ}_{\mathrm{k}} \mathrm{and} {\mathrm{ɳ}}_{\mathrm{k}}$ represent the process and measurement noise respectively. The object state distribution is estimated based on all the previous measurements that theoretically are:

$$\mathrm{p}(s_{\mathrm{k}+1}|z_{1:\mathrm{k}})=\int \mathrm{p}\left(s_{\mathrm{k}+1}|s_{\mathrm{k}}\right)\mathrm{p}\left(s_{\mathrm{k}}|z_{1:\mathrm{k}}\right){\mathrm{ds}}_{\mathrm{k}}$$

(18)

This prior is used in the prediction step when the measurement $z_{\mathrm{k}+1}$ is available. Bayes’ rule recursively updates this prior in the prediction step as in Equation (19):

$$\mathrm{p}(s_{\mathrm{k}+1}|z_{1:\mathrm{k}+1})=\frac{\mathrm{p}\left(z_{\mathrm{k}+1}|s_{\mathrm{k}+1}\right)\mathrm{p}\left(s_{\mathrm{k}+1}|z_{1:\mathrm{k}}\right)}{\int \mathrm{p}\left(s_{\mathrm{k}+1}|s_{\mathrm{k}}\right)\mathrm{p}\left(s_{\mathrm{k}}|z_{1:\mathrm{k}}\right){\mathrm{ds}}_{\mathrm{k}}}$$

(19)

where, $\mathrm{p}\left(z_{\mathrm{k}+1}|s_{\mathrm{k}+1}\right)$ is the likelihood distribution and $\int \mathrm{p}\left(s_{\mathrm{k}+1}|s_{\mathrm{k}}\right)\mathrm{p}\left(s_{\mathrm{k}}|z_{1:\mathrm{k}}\right){\mathrm{ds}}_{\mathrm{k}}$ is the normalization factor. The posterior density function is approximated through summations or weighted summation over all particles, as in Equation (20):

$$\mathrm{p}(s_{\mathrm{k}+1}|z_{1:\mathrm{k}+1})~{\displaystyle \sum }_{\mathrm{i}=1}^{N_{\mathrm{s}}}{w}_{\mathrm{k}+1}^{\mathrm{i}}\left(s_{\mathrm{k}+1}^{\mathrm{i}}-{\hat{\mathrm{s}}}_{\mathrm{k}+1}^{\mathrm{i}}\right)$$

(20)

where, ${\hat{\mathrm{s}}}_{\mathrm{k}+1}^{\mathrm{i}}$ are the new particles obtained by processing $s_{\mathrm{k}+1}^{\mathrm{i}}$ through the MS procedure using Equations (14) and (15). So now, effectively the PF particle validation process is replaced with the MS procedure by inserting ${\hat{\mathrm{s}}}_{\mathrm{k}+1}$ into the posterior distribution function in Equation (20). The vector $w_{\mathrm{k}+1}^{\mathrm{i}}$ are the weights associated with each particle and are calculated through Equation (21).

$$w_{\mathrm{k}+1}^{\mathrm{i}}=w_{\mathrm{k}}^{\mathrm{i}}\frac{\mathrm{p}\left(z_{\mathrm{k}+1}|s_{\mathrm{k}+1}^{\mathrm{i}}\right)\mathrm{p}\left(s_{\mathrm{k}+1}^{\mathrm{i}}|s_{\mathrm{k}}^{\mathrm{i}}\right)}{\mathrm{q}(s_{\mathrm{k}+1}^{\mathrm{i}}|s_{\mathrm{k}}^{\mathrm{i}},z_{\mathrm{k}})}$$

(21)

Usually the likelihood distribution is used to calculate the weights of the particles:

$$w_{\mathrm{k}+1}^{\mathrm{i}}={\mathrm{w}}_{\mathrm{k}}^{\mathrm{i}}\left(\mathrm{p}\left(z_{\mathrm{k}+1}|s_{\mathrm{k}+1}^{\mathrm{i}}\right)\right)$$

(22)

In order to get normalized weights, $w_{\mathrm{k}+1}^{\mathrm{i}}$ is divided by sum of all weights:

$$w_{\mathrm{k}+1}^{\mathrm{i}}=\frac{w_{\mathrm{k}+1}^{\mathrm{i}}}{{{\displaystyle \sum }}_{\mathrm{i}=0}^{N_{\mathrm{s}}}\mathrm{sum}\left(w_{\mathrm{k}+1}^{\mathrm{i}}\right)}$$

(23)

Expectation is used to approximate the posterior densities of Equation (20) given as:

$$\mathbb{E}(s_{\mathrm{k}+1}|z_{1:\mathrm{k}+1})~\frac{1}{N_{\mathrm{s}}}{\displaystyle \sum }_{\mathrm{i}=1}^{N_{\mathrm{s}}}\left(w_{\mathrm{k}+1}^{\mathrm{i}}{s}_{\mathrm{k}+1}^{\mathrm{i}}\right)$$

(24)

We should be very careful about the degeneration phenomenon, also called sample impoverishment, in which a few particles assume the leading role in approximating the posterior. This leads to a situation where many particles will cluster together as a result of which the sample set will have repeated particles. This results in a large percentage of particles getting insignificant tinny weights. Consequently, the tracker loses the target due to particles drifting towards one side. We calculate ${\mathcal{N}}_{\mathrm{e}𝒻𝒻}$, a reasonable measure of degeneracy, using Equation (25)

$${\mathcal{N}}_{\mathrm{e}𝒻𝒻}=\frac{1}{{{\displaystyle \sum }}_{\mathrm{i}=0}^{N_{\mathrm{s}}}{w}_{\mathrm{k}+1}^{\mathrm{i}}}$$

(25)

So, $IF {\mathcal{N}}_{\mathrm{e}𝒻𝒻}\le N_T$ we redistribute the clusters of particles, so that new particles are generated out of this cluster and re-initialize their weights to $\frac{1}{N_{\mathrm{s}}}$. $N_T$ is a threshold that triggers re-sampling if it is greater than ${\mathcal{N}}_{\mathrm{e}𝒻𝒻}$. The idea behind resampling is to get rid of particles with tiny weights and generate more particles from the ones with greater weights. As resampling can only execute after the weight calculation and normalization is done, it becomes a bottleneck in parallelizing the PF computations. Re-sampling although reduces the effects of degeneration, however, it introduces loss of diversity among the particles, also called impoverishment due to the fact that the resultant particles are clustered to close vicinity [21,32]. This is because redundant particles are chosen from the same points and hence the tracker has a chance to lose the object. To mitigate from this problem, the proposed framework executes the feature ranking module to update the model of the object alongside re-sampling. The next subsection presents the pseudo code of the proposed AMF-MSPF framework.

3.4. Pseudo Code of AMF-MSPF

At time $\mathrm{k}$, execute the following steps:

1. 

Particle initialization step:

{$\mathrm{i}=1:N_{\mathrm{s}};\mathrm{k}=1;\mathrm{Flag}=\mathrm{True}$}

${\left\{\left(s_{\mathrm{k}}^{\mathrm{i}}, w_{\mathrm{k}}^{\mathrm{i}}\right)\right\}}_{\mathrm{i}=1}^{N_{\mathrm{s}}}\mathrm{to} \mathrm{give}$[{$s_1^{1:N_{\mathrm{s}}},w_1^{1:N_{\mathrm{s}}}={\left(\frac{1}{N_{\mathrm{s}}}\right)}_1^{1:N_{\mathrm{s}}}$}]

2. 

Feature Ranking step:

$\left[N_{₣}\right]=\mathrm{feature}Ʀ\mathrm{nk}(₣)$

3. 

Propagation step:

{$\mathrm{k}=2,3,4,\dots ..; \mathrm{p}\left(s_{\mathrm{k}+1}|s_{\mathrm{k}}\right)$ = $\left(s_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}, w_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\right)\}$

4. 

MS Optimization Step:

$[y_{1:N_{₣}}^{1:N_{\mathrm{s}}}]={\mathrm{MnSft}}^{1:N_{₣}}\left(s_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\right)$

$\left[{\hat{\mathrm{s}}}_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\right]=\mathsf{\mu }\mathrm{edian}\left(y_{1:N_{₣}}^{1:N_{\mathrm{s}}}\right)$

5. 

Weight Calculation and normalization step:

$w_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}={ \mathrm{w}}_{\mathrm{k}}^{1:N_{\mathrm{s}}}\ast \mathrm{p}\left(z_{\mathrm{k}+1}|s_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\right)$

$w_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}= \frac{w_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}}{{{\displaystyle \sum }}_{\mathrm{i}=0}^{N_{\mathrm{s}}}{\mathrm{w}}_{\mathrm{k}+1}^{\mathrm{i}}}$

${\mathcal{N}}_{\mathrm{e}𝒻𝒻}=\frac{1}{{{\displaystyle \sum }}_{\mathrm{i}=0}^{N_{\mathrm{s}}}{\mathrm{w}}_{\mathrm{k}+1}^{\mathrm{i}}{\ast \mathrm{w}}_{\mathrm{k}+1}^{\mathrm{it}}}$

6. 

Estimation Step: (Posterior Estimation)

$\mathrm{p}(s_{\mathrm{k}+1}{|\mathrm{z}}_{1:\mathrm{k}+1})~{\displaystyle {\displaystyle \sum }_{\mathrm{i}=1}^{N_{\mathrm{s}}}}{w}_{\mathrm{k}+1}^{\mathrm{i}}\left(s_{\mathrm{k}+1}^{\mathrm{i}}-{\hat{\mathrm{s}}}_{\mathrm{k}+1}^{\mathrm{i}}\right)$

7. 

Re-sampling Step: (Particle redistribution and weight re-initialization)

$IF {\mathcal{N}}_{\mathrm{e}𝒻𝒻}\le N_T$

 $\{s_{\mathrm{k}+1}^{1:N_{\mathrm{s}}},w_{\mathrm{k}+1}^{1:N_{\mathrm{s}}}\}=\{s_{\mathrm{k}+1}^{1:N_{\mathrm{s}}},{(\frac{1}{N_{\mathrm{s}}})}_1^{1:N_{\mathrm{s}}}\}$

GOTO STEP 2 and REPEAT

 ELSE

GOTO STEP 3 and REPEAT

4. Experimental Results

This section evaluates and compares the proposed AMF-MSPF framework with the TSHT of [27] and PFKBOT of [24]. The choice of selecting these reference approaches, is because they employ multi-features and an adaptive model updating techniques similar, in some way, to our multi-feature framework. The experiments have been conducted on frames of different sizes and processing is done using an Intel Core i5 2.60 GHZ with 4 GB RAM. To illustrate the efficiency of the proposed AMF-MSPF, we applied it to video sequences having full occlusions, abrupt intensity changes, and clutter background introduced by the moving object. These challenging sequences evidently makes the experiments very difficult. The accuracy and robustness of the proposed AMF-MSPF are tested on the WalkByShop1cor, CAVIAR, and PETS video data sets.

Table 2. Description of video sequence.

Video Sequence	Description	Characteristic	Frame Size	No. of Frames
WalkByShop1cor	Couple walking along a corridor browsing	Regular mild and Severe occlusion	384 × 288	2359
Browse4	Person moves in an area with abrupt intensity change	abrupt illumination change with non-linear motion	384 × 288	1138
PETS 2007	People walking in crowd with different obstacles	Severe occlusion by abrupt intensity variations	576 × 720	3000

highlights some of the important characteristics of these sequences.

The reason for selecting these data sets is because they have rich characteristics such as full occlusion, clutter, and abrupt illumination change. The computer vision research community widely uses these data sets to evaluate their algorithms. The prime goal, of the simulations, is to track manually initialized regions of interest during long video sequences under clutter background and occlusions by intensity. The ground truth locations of the objects are recorded manually for each sequence. Red, blue, green and yellow depicts the results of the ground truth, proposed AMF-MSPF, PFKBOT, and TSHT respectively. For a quantitative analysis F-SCORE, false alarm rate (FAR) and root mean square error (RMSE) are evaluated for all the sequences under test, as shown in

Table 3. RMSE, Accuracy and F-SCORE for all video sequences.

Video Sequence	RMSE			FAR			F_SCORE
	PFKBOT	TSHT	Proposed	PFKBOT	TSHT	Proposed	PFKBOT	TSHT	Proposed
Browse4	55.80	41.30	9.18	0.49	0.47	0.17	0.67	0.71	0.91
WalkBy-Shop1cor	30.70	7.76	5.76	0.35	0.24	0.13	0.79	0.86	0.93
PETS-2007	24.70	15.30	11.12	0.35	0.27	0.15	0.71	0.79	0.91

.

Precision ($Ƥ$) and recall ($Ʀ$) are given by Equations (26) and (27):

$$Ƥ=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(26)

$$Ʀ=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(27)

where, TP, FP, FN and TN are the number of true positives, false positives, false negatives, and true negatives respectively. F-SCORE becomes trivial if we know $Ƥ$ and $Ʀ$ that is given by Equation (28):

$$\mathrm{F}\_\mathrm{SCORE}=2\ast \frac{Ƥ\ast Ʀ}{Ƥ+Ʀ}$$

(28)

Similarly, FAR is calculated as the ratio between the number of negative event wrongly categorized as positive and the total number of actual negative events using Equation (29).

$$\mathrm{FAR}=\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}}$$

(29)

4.1. Visual Tracking Result

We start with the WalkByShop1cor dataset which is a long video sequence exhibiting several constraints. The dataset exhibits multiple full occlusions in addition to similar object appearance in frames 350–410 and 870–930. The results of AMF-MSPF are close to the ground truth and outperforms TSHT and PFKBOT methods as shown in Figure 2. PFKBOT fails on encountering instances having a background similar to the object. The error accumulates until the PFKBOT loses the target because FP strays the particles to false mode. However, in THST, the object recovers from the occasional distraction by the background clutter. This is due to the fact that TSHT method takes into account both the spatiotemporal aspects of the object that leads the particles towards more likely modes. Under mild but regular intensity change, TSHT is quite comparable to the proposed AMF-MSPF in the WalkByShop1cor dataset. Figure 2 highlights the visual tracking results.

Since the results of AMF-MSPF and the reference methods have come in proximity, RMSE metric is evaluated for a closer look. In Table 3, the RMSE, of AMF-MSPF, THST and PFKBOT for WalkByShop1cor dataset are 5.76, 7.76 and 30.70 respectively. Consequently, the FAR of the proposed method is the lowest, due to very few false positives i.e., 0.13 as compared to 0.24 and 0.35 for THST and PFKBOT respectively. Moreover, the F-Score of the proposed method is 0.93 as compared to 0.86, 0.79 for THST and PFKBOT.

In the next experiments, we consider two more sequences from the Browse4.mpg dataset. In these sequences, the object moves in severe intensity occluded areas. The abrupt intensity change occurs in multiple instances from frames 230–410, frames 880–920 and frames 1005–1050. The experimental results show that under abrupt intensity change, the AMF-MSPF is robust and tracks the object, as highlighted in Figure 3 and Figure 4. In Table 3, the RMSE of AMF-MSPF, TSHT, and PFKBOT is 9.18, 41.30 and 55.80 respectively. Low RMSE indicates FAR and consequently a higher F-SCORE that is 0.91 for the proposed.

In the final experiment, AMF-MSPF is tested on the PETS 2007 dataset, which exhibits continuous background change due to occlusion by intensity as well as other objects. The results of AMF-MSPF are convincing as compared to the TSHT and PFKBOT methods as shown in Figure 5. Contrasting TSHT and PFKBOT, the proposed AMF-MSPF framework selects a new set of $N_{₣}$ features that are used to initialize MS procedure for every particle. These features are selected by the ranking module that is triggered whenever the re-sampling step occurs due to the degeneration problem. In contrast to other methods, our method is more accurate because it re-initializes the object based on local features. For a quantitative analysis and highlights of some of the prominent characteristics of our method Table 3 and

Table 4. Results of various trackers under challenging constraint.

Constraint/Methods	[24]	[27]	Proposed Method
Full Occlusion	✓	✓	✓
Clutter Background	🗴	✓	✓
Intensity Change	🗴	🗴	✓

are given in this regard.

4.2. Computational Complexity

After evaluating the proposed AMF-MSPF on several challenging video datasets, we obtained improved results in term of robustness and accuracy. However, the MSPF hybrid methodology implicitly has an additional cost of incorporating the MS optimization into the already complex PF algorithm. One innovation of the proposed research is the simplification of the MS optimization. The simplification comes through an observation, that only a fraction of samples was required by the MS optimization procedure. The computational efficiency of the proposed MS was evaluated on a sequence extracted from the Browse4 dataset. As can be seen in Figure 6, the error introduced, due to dropping more than 75% of the samples, is negligible. And our simplified MS is able to track the object successfully in real-time, even under a Matlab implementation. This has led to a huge computational reduction in the MS method without significantly compromising its accuracy. This as a consequence saves resources for feature ranking module. Thereby, the proposed AMF-MSPF framework implements the feature ranking on top of the MSPF methodology without aggravating its computational complexity. The complexity of proposed framework is considerably reduced as compared to TSHT and PFKBOT because in those implementations, MS takes into account all the samples of object window. The AMF-MSPF framework can process 10–15 frames per second, which is perceived by humans, while none of the reference methods are able to run in real-time.

Let us derive the overall computational cost of the proposed AMF-MSPF framework. Let $I_{ter}$ be the total number of iteration for each MS procedure (in our case $I_{ter}=5$), $T_{p\_gen\_wt}$ be the execution time required for sample generation and their associated weights, $T_{re-sampling}$, $T_{ms}$, and $T_{rnk}$ be the execution times for the resampling, MS procedure and ranking module respectively. Then the execution time of the AMF-MSPF framework is given in Equation (29):

$$T_{Amf-mspf}=\left[T_{p_{\_gen\_wt}}+T_{ms}+T_{re-sampling}+T_{rnk}\right]$$

(30)

Although MS is initialized in multiple features over all particles, however down sampling the number of pixels reduces its execution time by an order of magnitude. Thus with the reduced pixels, $N_h$, the complexity of our proposed MS is $O(m\ast {\mathrm{N}}_{₣}\ast N_h$) which has no significant impact on Equation (29). This saves enormous computational power that is utilized by the ranking module. Since the MS optimization is applied multiple times to every particle until convergence, it moves the particles to new locations where it no longer conforms to the posterior distribution. This is what is known as the degeneracy problem in which a small number of particles dominate the weight race and therefore the estimates are heavily influenced by them. Consequently, the re-sampling stage is introduced to mitigate from the degeneracy problem by replicating particles with larger weights and removing the particles with negligible ones. This, however, constraints the system that re-sampling step be carried out after the particle weighting and normalization steps and that makes it a bottleneck in parallelizing the PF computations. The complexity of the re-sampling module is also equal to the particle generation and weighting step i.e., $O({\mathrm{N}}_{\mathrm{s}}$).

While the ranking module is quadratic in number of samples from the background and foreground i.e., $O\left(₣\ast fg\left(j\right)\ast bg\left(j\right)\right)$, nevertheless, it is only triggered when the re-sampling step is required. Since the re-sampling step is not carried in every step, most of the time the tracking is performed using only the novel optimized MSPF. Therefore, asymptotically speaking, this makes the overall complexity of our proposed AMF-MSPF framework equal to $O\left(n\right)$ with high probability.

5. Summary

In summary, this research work brings improvement to the field of robust object tracking in two novel ways. Firstly, it develops an adaptive multi-feature framework on top of the MSPF methodology. A ranking module ranks the given feature space based on maximizing the inter-class variance between the background and foreground. High variance score signifies the chances for better target discrimination and thereby the top features are selected for updating the target model. The likelihoods corresponding to these features are continuously used to form new likelihood images that dynamically initialize the MS to enable tracking in the local context. Secondly, the computational cost of the MS optimization method of the proposed AMF-MSPF framework is reduced. Consequently, that enables tracking in real-time unlike other methods in the hybrid tracking category that can only process a few frames.

The accuracy and robustness of the proposed AMF-MSPF is tested on the WalkByShop1cor, CAVIAR and PETS video data sets. These sequences are known for their challenging constraints that makes the experiments very difficult. The AMF-MSPF framework was found to provide improved tracking performance compared with other conventional methods based on hybrid object tracking methodology. The experimental results demonstrate successful visual tracking even under extreme intensity variations and full occlusion. The average F-SCORE (over all the video sets) of the AMF-MSPF is 0.92 as compared to 0.79 and 0.72 for TSHT and PFKBOT respectively.

The proposed AMF-MSPF implements a ranking module on top of the MSPF methodology to give it an additional layer of capability. Consequently, this has increased the performance of the MSPF that now takes into account any rapid movement of the object that changes its background. However, the improvement is tied to an assumption that the target reference model deviates significantly from its instance of initialization which is updated at the sampling step. This assumption needs to be carefully studied and there is a need to research on a more robust dynamic technique that switches the required features for the target update. In future research, we also plan to employ statistical estimation or deterministic background weighting mechanisms to boast the target in a cluttered background. As the clutter background eclipses some part of the target, and that can hamper the performance of the feature evaluation process. We have employed a linear combination of the RGB color components. However, the availability of a potentially large spatio-temporal feature space can also be experimented with to further improve tracking using the AMF-MSPF framework.