Target Tracking with Adaptive Morphological Correlation and Neural Predictive Modeling

Abstract

A tracking method based on adaptive morphological correlation and neural predictive models is presented. The morphological correlation filters are optimized according to the aggregated binary dissimilarity-to-matching ratio criterion and are adapted online to appearance variations of the target across frames. Morphological correlation filtering enables reliable detection and accurate localization of the target in the scene. Furthermore, trained neural models predict the target’s expected location in subsequent frames and estimate its bounding box from the correlation response. Effective stages for drift correction and tracker reinitialization are also proposed. Performance evaluation results for the proposed tracking method on four image datasets are presented and discussed using objective measures of detection rate (DR), location accuracy in terms of normalized location error (NLE), and region-of-support estimation in terms of intersection over union (IoU). The results indicate a maximum average performance of 90.1% in DR, 0.754 in IoU, and 0.004 in NLE on a single dataset, and 83.9%, 0.694, and 0.015, respectively, across all four datasets. In addition, the results obtained with the proposed tracking method are compared with those of five widely used correlation filter-based trackers. The results show that the suggested morphological-correlation filtering, combined with trained neural models, generalizes well across diverse tracking conditions.

1. Introduction

Visual object tracking is a central problem in computer vision owing to its broad utility across high-impact applications, including video surveillance, autonomous navigation, human–computer interaction, biomedical imaging, and robotics.

Object tracking consists of estimating the time-varying state (e.g., position and region of support) of a target across an image sequence as it moves through the field of view. This problem is challenging because captured image sequences are often degraded by sensor noise, nonuniform illumination, and blur degradation, among others. Additionally, the target can exhibit appearance changes (pose, rotation, scale), and captured scene images can contain background clutter, partial or prolonged occlusions, and out-of-view events. Furthermore, for applications with real-time constraints, high-rate tracking is important. Because of these factors, tracker designs must achieve both robustness and computational efficiency [1].

Over the years, numerous methods have been proposed to address real-world tracking problems [2,3,4]. Several methods employ offline training, in which a model is pre-trained on labeled data that include the target (true class) and unwanted patterns from the background and distractor objects (false class) before tracking begins [5,6]. Other methods utilize an online training approach that is initialized from minimal target information in the initial frame and adapts the model during tracking operation [7,8]. Online adaptation offers greater flexibility under appearance changes of the target and scene over time. Online training is often preferred in single-object tracking, in which the target exhibits unseen appearance variations during tracking. Also, there are hybrid tracking methods that combine robust offline-learned representations with online learning to balance accuracy and adaptability [9,10].

A successful online training approach to target tracking is tracking-by-detection. This approach learns a discriminative classifier online to distinguish the target from the background. For each frame, image fragments at multiple scales are extracted around the predicted target location. Next, the candidate bounding box with the highest score is selected as the target’s estimate, and the model is updated with new true-class templates near the predicted target location and false-class templates taken from the background. This approach adapts well to appearance variations and is robust to clutter. However, it is sensitive to sensor noise and weakly regularized online updates, which can lead to overfitting and drift.

A widely used tracker within this approach is Multiple Instance Learning (MIL) [11]. MIL considers the training data collected during tracking as bags of candidate image fragments. Positive bags contain at least one true target instance, while negative bags contain only background fragments. Updating the classifier with weak set-level supervision mitigates label uncertainty and enables adaptation to gradual appearance drift.

The Structured Output Tracking with Kernels (Struck) [12] is a widely used tracker that learns a structured support vector machine that directly scores candidate bounding boxes, relating the target’s location space to the online training samples. Struck has demonstrated strong performance on several standard datasets [4].

Another successful tracker is Tracking–Learning–Detection (TLD) [13], which integrates a short-term tracker with a long-term detector and an online learning component. Using a boosted classifier with consistency constraints and a sampling strategy, TLD yields robustness to occlusions and appearance variations. Despite strong performance, the tracking-by-detection approach is computationally intensive because it requires evaluating large sets of candidate templates and updating the classifier every frame, increasing latency.

An alternative widely used approach for target tracking consists of correlation filtering. In this approach, a linear filter is trained to produce a sharp correlation peak at the target location. Next, target detection and filter adaptation are performed efficiently in the Fourier domain by exploiting the convolution theorem [14,15]. A successful approach for correlation filter design consists of employing ridge regularization. In this formulation, the filter is constructed by solving a least-squares problem with ${\ell }_2$ regularization, aimed at controlling overfitting and improving numerical stability. Spatial and temporal regularizers constrain the target’s region of support and promote smooth evolution across frames, improving robustness to clutter and appearance changes.

The Kernelized Correlation Filters (KCFs) [16] are designed by solving a kernelized ridge regression problem on dense circular shifts of the target template using prespecified Gaussian-shaped desired correlation responses. This approach yields good performance under small translations and moderate appearance changes of the target. Additionally, it is suitable for real-time operation. However, the circular boundary assumption induces boundary artifacts and wrap-around effects that introduce background content into the true-class samples, potentially increasing drift errors. In addition, the absence of scale adaptation and of long-term re-detection stages makes this method prone to target loss.

Spatially Regularized Discriminative Correlation Filters (SRDCFs) [17,18] are constructed by solving a kernelized ridge regression problem with spatially varying regularization that penalizes coefficients increasingly with distance from the target origin. This spatial weighting confines the region of support for the target, mitigates boundary artifacts, and enables training with a larger search area while reducing background influence. Compared with KCF, SRDCF improves localization accuracy and robustness in cluttered scenes. A main drawback of SRDCF is the increased computational cost due to spatial regularization and its sensitivity to the spatial weight design. A successful variant of SRDCF incorporates adaptive decontamination of the training set to suppress corrupted samples caused by occlusions, background clutter, or misalignment [19]. This variant is referred to as SRDCFd. During online learning, contaminated updates are identified and assigned reduced weights, which improves peak localization and detection performance in cluttered scenes. A potential drawback is the increased computational cost and limited adaptation when few uncontaminated samples are available.

Background-Aware Correlation Filters (BACFs) [20,21], are learned by solving a ridge regression problem on circulant samples extracted from an enlarged search region. A cropping operator enforces zero coefficients outside the target template, thereby limiting the region of support to the object region. This background-aware formulation provides the learning process from false patterns, alleviating boundary artifacts and improving discrimination in cluttered scenes. Compared with KCF, SRDCF, and SRDCFd, BACF provides improved robustness to scene degradations and false objects. An important limitation of BACF is its higher optimization cost and increased sensitivity to the crop ratio and to the regularization parameters.

Spatial–Temporal Regularized Correlation Filters (STRCFs) [22], are designed using ridge regression with both spatial and temporal regularization. The spatial term penalizes coefficients with increasing distance from the target center, confining the region of support of the filter to the target’s region and mitigating boundary artifacts while reducing background influence. The temporal term penalizes deviations of the filter from the previous frame, stabilizing updates and reducing drift under appearance changes. Compared with KCF, SRDCF, SRDCFd, and BACF, STRCF improves robustness to deformations of the target and partial occlusions. However, this method has two main limitations. First, the spatial and temporal terms increase computational cost. Second, the temporal-regularization term introduces temporal inertia, delaying adaptation under abrupt changes.

In contrast, this work presents an alternative approach for designing correlation filters for target tracking using a morphological-based formulation. The proposed filters are optimized with respect to the Aggregated Binary Dissimilarity-to-Matching Ratio (ABDR) criterion. Because morphological correlation is applied to images after binary threshold decomposition, the resulting filters are more robust to illumination variations and sensor noise. In addition, we introduce a correlation-plane postprocessing method based on synthetic-basis projection that suppresses clutter-induced correlation responses, improving the reliability of target detection. Furthermore, we incorporate trained neural models to predict the target’s center location and estimate the bounding box of the detected target. The proposed tracking method also includes efficient stages for drift correction and tracking reinitialization, improving robustness. As a result, the proposed tracking approach enables accurate estimation of the target state (location and bounding box) in image sequences under challenging conditions such as abrupt motion, partial occlusions, nonrigid deformations, and nonuniform illumination. The main contributions of this research are summarized as follows:

• Morphological-correlation filter design: An adaptive correlation filtering method based on morphological operations is proposed for robust target tracking, providing an alternative to the conventional ridge-regularization formulation and improving robustness to scene perturbations.
• Correlation-plane postprocessing: A postprocessing stage based on synthetic-basis projection is introduced to refine the correlation response and improve target detection in cluttered scenes.
• Hybrid tracking framework: A tracking approach is developed that integrates an online-trained morphological correlation for target detection with neural models trained offline for target location prediction and bounding box estimation, achieving stable tracking trajectories across frames.
• Drift-correction and reinitialization mechanisms: Efficient methods for drift correction and tracking reinitialization are incorporated to maintain tracking stability under occlusions, illumination variations, and temporary target loss.

This paper is organized as follows. Section 2 describes the proposed method for target tracking based on morphological correlation and neural predictive models. First, we present the design of morphological correlation filters. Next, we detail the suggested postprocessing stage for correlation plane denoising. We then describe the operation steps of the proposed tracking method. Section 3 presents the results of the proposed tracking method using four widely used image datasets. Tracking performance is quantified using objective measures of detection efficiency, target location accuracy, and bounding box estimation. We also compare the performance of the proposed tracking method with that of five widely used correlation filter-based trackers. Finally, Section 4 presents the conclusions.

2. Proposed Target Tracking Method

Here, we describe the proposed approach for single-object tracking based on morphological correlation filtering and neural predictive models. Section 2.1 details reliable target detection using morphological correlation filtering. Section 2.2 introduces postprocessing of the correlation plane using synthetic basis projection. Section 2.3 presents the complete tracking algorithm.

2.1. Target Recognition Based on Morphological Correlation

Let $t(x,y)$ be a reference view of a target object to be recognized, located at the arbitrary coordinates $(x_t,y_t)$ in a captured image of a scene, $f(x,y)$, given as

$$f(x,y)=t(x-x_t,y-y_t)+m(x-x_t,y-y_t)b(x,y)+n(x,y),$$

(1)

where $n(x,y)$ is additive sensor noise, $b(x,y)$ is the image of the background, and $m(x,y)$ is the inverse support function of the target, given by

$$m(x,y)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}(x,y)\in t(x,y)\hfill \\ 1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(2)

In conventional correlation pattern recognition [23], the coordinates of the target in the scene are estimated using the linear correlation, as follows:

$$({\widehat{x}}_t,{\widehat{y}}_t)=arg\underset{(x,y)}{max}\left\{{\int \int }_{{\mathbb{R}}^2}f({\tau }_x,{\tau }_y)h(x+{\tau }_x,y+{\tau }_y)d{\tau }_{x}d{\tau }_y\right\},$$

(3)

where $h(x,y)$ is the impulse response of a correlation filter. In recent decades, numerous filter designs have been proposed to optimize different performance criteria [24]. Interested readers are invited to consult the following references: [25,26,27,28,29]. A key advantage of using Equation (3) for target recognition is its efficient frequency-domain computation with the Fast Fourier Transform (FFT) [14,30]. However, a main limitation of this approach for target tracking is that, to construct and update the filter for each captured frame, explicit knowledge of the target’s region of support $m(x,y)$, as well as a statistical characterization of $b(x,y)$ and $n(x,y)$, is required. Consequently, it is necessary to incorporate reliable algorithms to estimate these components dynamically, thereby potentially reducing both recognition performance and computational efficiency.

The proposed approach based on morphological correlation achieves high recognition performance without requiring explicit knowledge of the target’s region of support in each frame, nor estimation of the statistical characteristics of the background and sensor noise.

Consider a digital image $I(x,y)$ with the size of $N_x\times N_y$ elements. This image can be represented through a binary threshold decomposition as follows [31,32]:

$$I(x,y)=(v_{min}-1)+\sum _{v=v_{min}}^{v_{max}}{I}_v(x,y),$$

(4)

where $(v_{min},v_{max})$ denote the minimum and maximum intensity values of $I(x,y)$, with $v_{min}\ge 1$ and $v_{max}\ge v_{min}$, and

$$I_v(x,y)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}I(x,y)\ge v,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(5)

is the binary image corresponding to $I(x,y)$ at the threshold level v. In Equations (4) and (5),

$$\tilde{I}(x,y)=\sum _{v=v_{min}}^{v_{max}}{I}_v(x,y)$$

(6)

is a preprocessed version of $I(x,y)$, obtained by applying a binary threshold decomposition (BTD). Note that if $v_{min}=1$, then $\tilde{I}(x,y)=I(x,y)$.

In this scenario, to reliably detect and locate the target $t(x,y)$ in the input image $f(x,y)$, the Aggregated Binary Dissimilarity-to-Matching Ratio (ABDR) is defined as follows:

$$\mathrm{ABDR}({\tau }_x,{\tau }_y)={\displaystyle \frac{{\displaystyle \sum _{v=v_{min}}^{v_{max}}\sum _{x,y}\left|t_v(x,y)-f_v(x-{\tau }_x,y-{\tau }_y)\right|}}{{\displaystyle \sum _{v=v_{min}}^{v_{max}}\sum _{x,y}\left|t_v(x,y)+f_v(x-{\tau }_x,y-{\tau }_y)-1\right|}}},$$

(7)

where $t_v(x,y)$ and $f_v(x,y)$ are binary images of $t(x,y)$ and $f(x,y)$, respectively, as given in Equation (5). The numerator in Equation (7) is the aggregated binary dissimilarity between $t(x,y)$ and $f(x,y)$ at the coordinates $({\tau }_x,{\tau }_y)$, whereas the denominator is the aggregated binary matching score. The ABDR is zero when the two compared images are identical and tends to infinity when there are no matching elements between them.

Considering the identity $min\left\{I_{1,v}(x,y)+I_{2,v}(x,y),1\right\}=max\left\{I_{1,v}(x,y),I_{2,v}(x,y)\right\}$ [33], Equation (7) can be rewritten as follows:

$$\mathrm{ABDR}({\tau }_x,{\tau }_y)={\displaystyle \frac{{\mu }_t+{\mu }_f-{\displaystyle \frac{2}{V{N}_v}}{\sum }_{v=v_{min}}^{v_{max}}{\sum }_{x,y}min\left\{t_v(x,y),f_v(x-{\tau }_x,y-{\tau }_y)\right\}}{1+{\mu }_t+{\mu }_f-{\displaystyle \frac{2}{V{N}_v}}{\sum }_{v=v_{min}}^{v_{max}}{\sum }_{x,y}max\left\{t_v(x,y),f_v(x-{\tau }_x,y-{\tau }_y)\right\}}},$$

(8)

where ${\mu }_t$ and ${\mu }_f$ are the local mean intensity values of $t(x,y)$ and $f(x,y)$, respectively, $V=(v_{max}-v_{min})$ is the number of threshold values in the decomposition, and $N_v=N_x\times N_y$ is the total number of image elements.

The minimum value of Equation (8) is reached when maximizing the correlation

$$C({\tau }_x,{\tau }_y)={\displaystyle \frac{{\displaystyle \sum _{v=v_{min}}^{v_{max}}\sum _{x,y}min\left\{t_v(x,y),f_v(x-{\tau }_x,y-{\tau }_y)\right\}}}{{\displaystyle {\displaystyle \frac{1}{V{N}_v}}+\sum _{v=v_{min}}^{v_{max}}\sum _{x,y}max\left\{t_v(x,y),f_v(x-{\tau }_x,y-{\tau }_y)\right\}}}},$$

(9)

where the term $(1/V{N}_v)$ is added to the denominator to avoid division by zero.

The computational efficiency of Equation (9) can be improved by interchanging the order of the summation and applying the generalized morphological identity [31]:

$$\sum _{v=v_{min}}^{v_{max}}★\left\{I_{1,v}(x,y),I_{2,v}(x,y)\right\}=★\left\{{\tilde{I}}_1(x,y),{\tilde{I}}_2(x,y)\right\},$$

where $★\in \{min,max\}$. As a result, the morphological correlation in Equation (9) can be expressed as follows:

$$C({\tau }_x,{\tau }_y)={\displaystyle \frac{{\displaystyle \sum _{x,y}min\left\{\tilde{t}(x,y),\tilde{f}(x-{\tau }_x,y-{\tau }_y)\right\}}}{{\displaystyle {\displaystyle \frac{1}{V{N}_v}}+\sum _{x,y}max\left\{\tilde{t}(x,y),\tilde{f}(x-{\tau }_x,y-{\tau }_y)\right\}}}}.$$

(10)

The coordinates of the target in the scene are estimated from Equation (10) as follows:

$$({\tilde{\tau }}_x,{\tilde{\tau }}_y)=\underset{{\tau }_x,{\tau }_y}{argmax}\left\{C({\tau }_x,{\tau }_y)\right\}.$$

(11)

In Equation (10), $\tilde{t}(x,y)$ and $\tilde{f}(x,y)$ denote the preprocessed versions of the reference image $t(x,y)$ and input image $f(x,y)$ obtained through binary threshold-decomposition.

The robustness of target recognition using morphological correlation can be improved by selecting the values $\{v_{min},v_{max}\}$ that characterize the object located at the origin of the image $I_i(x,y)$, where $I_i(x,y)$ corresponds to either $t(x,y)$ or $f(x,y)$. These limits can be defined as follows:

$$\begin{array}{cc}\hfill v_{min}& =I_i(0,0)-{ϵ}_v{\sigma }_{I_i},\hfill \\ \hfill v_{max}& =I_i(0,0)+{ϵ}_v{\sigma }_{I_i},\hfill \end{array}$$

(12)

where ${\sigma }_{I_i}$ denotes the standard deviation of $I_i(x,y)$ relative to the value at the origin $I_i(0,0)$, and ${ϵ}_v$ is a dispersion coefficient. Additionally, by specifying the number of quantization levels Q and defining the quantization step

$${\Delta }_v={\displaystyle \frac{2{ϵ}_v{\sigma }_v}{Q}},$$

(13)

the resulting binary threshold decomposed images are given by

$$\begin{array}{cc}\hfill \tilde{t}(x,y)& =\sum _{v=1}^Q{t}_{(v_{min}+v{\Delta }_v)}(x,y),\hfill \\ \hfill \tilde{f}(x-{\tau }_x,y-{\tau }_y)& =\sum _{v=1}^Q{f}_{(v_{min}+v{\Delta }_v)}(x-{\tau }_x,y-{\tau }_y).\hfill \end{array}$$

(14)

The preprocessed images given in Equation (14) are employed in Equation (10) to reliably detect the target in a captured scene image.

2.2. Postprocessing of the Correlation Plane Through Synthetic Basis Projection

The robustness of target detection using morphological correlation filtering can be improved via projection-based postprocessing. This approach involves the construction of two sets of synthetic bases a priori: one composed of noise-free correlation planes that model the expected responses produced by the target, and another set composed of realizations of spatially correlated noise to model the responses of the background or false objects. The observed correlation plane is reconstructed as a combination of the synthetic basis functions and a residual term. The postprocessed (denoised) plane is obtained by isolating the component associated with the synthetic correlation planes and removing the contribution from the noise realizations. This simple procedure minimizes false-detection errors and improves the reliability of target detection.

The set of $N_p$ noise-free synthetic correlation planes $\{C_k(x,y):k=1,\dots ,N_p\}$ is generated using an inverse-polynomial model of the form

$$C_k(x,y)={\left(1+{\displaystyle \frac{{(x-x_k)}^2+{(y-y_k)}^2}{b_k^2}}\right)}^{-\alpha },$$

(15)

where $(x_k,y_k)$ specify the coordinates of the correlation peak, $b_k$ is a scale parameter controlling the spatial spread, and $\alpha$ is a fixed shape parameter that defines the decay rate. Each synthetic plane produces a single peak located near the origin, with variations in peak width introduced through different values of $b_k$. Furthermore, the set of $N_p$ synthetic noise patterns $\{{\eta }_k(x,y):k=1,\dots ,N_p\}$ is generated by applying a smoothing filter to two-dimensional white noise realizations, inducing spatial correlation.

The amplitude values of each synthetic correlation plane $C_k(x,y)$ are lexicographically reordered into column vectors ${\mathbf{c}}_k$. Then, a matrix of vectorized correlation planes is constructed as follows:

$${\mathbf{Y}}_C=[{\mathbf{c}}_1,{\mathbf{c}}_2,\dots ,{\mathbf{c}}_{N_p}].$$

(16)

To ensure that ${\mathbf{Y}}_C$ is full rank, its columns can be orthogonalized using Singular Value Decomposition (SVD). Similarly, each noise pattern ${\eta }_k(x,y)$ is reordered into a column vector ${\mathit{\eta }}_k$, which is used to construct the following matrix:

$${\mathbf{Y}}_{\eta }=[{\mathit{\eta }}_1,{\mathit{\eta }}_2,\dots ,{\mathit{\eta }}_{N_p}].$$

(17)

In this manner, the regressor matrix is constructed as follows:

$$\mathbf{A}=\left[{\mathbf{Y}}_C{\mathbf{Y}}_{\eta }\right].$$

(18)

From Equation (18), the observed correlation plane can be reconstructed as follows:

$$\mathbf{c}=\mathbf{A}\mathit{\beta }+\mathbf{r},$$

(19)

where $\mathit{\beta }$ is a vector of weighting coefficients, and $\mathbf{r}$ is the residual term. The weighting coefficients can be obtained by solving the regularized least squares problem:

$$\mathit{\beta }=arg\underset{\mathit{\beta }}{min}{∥\mathbf{A}\mathit{\beta }-\mathbf{c}∥}^2+\lambda {∥\mathit{\beta }∥}^2,$$

(20)

where $\lambda >0$ is a regularization parameter. Note that the objective function ${∥\mathbf{A}\mathit{\beta }-\mathbf{c}∥}^2$ is convex and differentiable. Thus, taking its gradient with respect to $\mathit{\beta }$ and setting it to zero yields the following:

$$({\mathbf{A}}^{\top }\mathbf{A}+\lambda \mathbf{I})\mathit{\beta }={\mathbf{A}}^{\top }\mathbf{c}.$$

(21)

The linear system in Equation (21) can be solved efficiently using Cholesky decomposition, since the matrix ${\mathbf{A}}^{\top }\mathbf{A}+\lambda \mathbf{I}$ is symmetric and positive definite.

Note that the correlation plane reconstructed using only the synthetic planes is ${\mathbf{c}}_r=\mathbf{A}\mathit{\beta }$. Thus, the residual term can be computed as $\mathbf{r}=\mathbf{c}-{\mathbf{c}}_r$, and the denoised correlation plane is obtained as follows:

$$\widehat{\mathbf{c}}={\mathbf{Y}}_C{\mathit{\beta }}_C+\mathbf{r},$$

(22)

where ${\mathit{\beta }}_C$ is the vector containing the first $N_p$ elements of $\mathit{\beta }$. Finally, the resultant vector $\widehat{\mathbf{c}}$ is reshaped into a two-dimensional array $\widehat{C}(x,y)$.

Next, a subpixel-accurate Fourier shift can be applied to $\widehat{C}(x,y)$ for correcting the displacement of the correlation peak introduced during postprocessing. The shifted plane is computed as follows:

$${\widehat{C}}_{\mathrm{sh}}(x,y)={\mathcal{F}}^{-1}\left\{\mathcal{F}\left\{\widehat{C}(x,y)\right\}·e^{-j2\pi \left({\displaystyle \frac{\Delta x·u}{N_x}}+{\displaystyle \frac{\Delta y·v}{N_y}}\right)}\right\},$$

(23)

where $(\Delta x,\Delta y)$ represent the coordinate displacement between the peak location $({\widehat{\tau }}_x,{\widehat{\tau }}_y)$ of the observed correlation plane and that of the postprocessed plane, and “$\mathcal{F}\{·\}$, ${\mathcal{F}}^{-1}\{·\}$” denote the two-dimensional Fourier and inverse Fourier transforms, respectively.

2.3. Target Tracking Based on Morphological Correlation Filtering and Neural Predictive Models

The proposed tracking algorithm consists of a sequence of processing stages, as shown in Figure 1. The algorithm begins by capturing an image frame of the scene, from which the target object is selected either manually or automatically to create a target template. The central coordinates of this template are considered to be the current coordinates $(x_t,y_t)$ of the target. This template is then preprocessed using BTD, as defined in Equation (14).

Afterwards, a new image frame is captured, and a region of interest (ROI) is extracted with origin on the coordinates $(x_t,y_t)$. The morphological correlation given in Equation (10) is then computed and postprocessed using Equations (22) and (23) between the preprocessed target template and the ROI to perform target detection. The detection stage determines if the target is successfully recognized within the ROI.

If detection is successful, the target bounding box is predicted using the designed CNN-based model described in Appendix A.2. The coordinates of the target in the current frame are then estimated from the ROI, as given in Equation (11). The target template $t_i(x,y)$ is also updated using the estimated coordinates of the current target detection. To improve robustness, a drift estimation and correction step is incorporated. This stage estimates a small translational offset, which is used to align the target’s template before updating it. The proposed drift correction method is detailed in Appendix B. The target’s template for detecting the target in the next frame is dynamically adapted as follows:

$$t_i(x,y)=(1-\beta )\ast t_i(x,y)+\beta \ast t_{i-1}(x,y),$$

(24)

where $t_{i-1}(x,y)$ is the past target template and $\beta$ is a forgetting parameter (usually $\beta =0.125$). The coordinates of the target for the next frame are then predicted using the designed NN-based model described in Appendix A.1, which uses the past five estimated target locations.

Details of the trained neural models are provided in Appendix A.1 and Appendix A.2. The target position prediction model was trained using a dataset consisting of five-element coordinate vectors derived from annotated image sequences. The bounding box estimation model was trained using a dataset composed of image fragments of 321 × 321 pixels containing targets of varying sizes and background complexity. The data were normalized and randomly partitioned into training and validation subsets with a 70–30 ratio to promote generalization capability. To reduce the risk of overfitting, particularly in cases involving small targets or cluttered scenes, the models incorporated batch normalization and dropout layers, and a mild form of data augmentation was applied during training.

If the target is not detected, the algorithm performs a reinitialization stage, which is detailed in Appendix C, to re-detect the target across the entire scene image. This procedure employs a sliding window approach that evaluates each window location within the scene based on a similarity metric combining Histogram of Oriented Gradients (HOG) and color histogram features. The region of the scene that has the highest similarity score with respect to the archived target’s features is selected as the new bounding box, allowing the tracker to resume operation. This reinitialization step improves robustness against occlusions, appearance changes, and unexpected tracking failures.

The proposed algorithm shown in Figure 1 enables reliable target detection, accurate localization, precise bounding box estimation, and online adaptation throughout the image sequence, yielding efficient and robust tracking.

3. Results

This section presents the results of the proposed tracking method based on morphological correlation and neural predictive modeling. Tracking performance is evaluated using objective measures on video sequences from four widely used datasets. Furthermore, the performance of the proposed method is compared with that of five widely used correlation-based tracking methods.

Section 3.1 describes the experiments performed, including the input image sequences used for tracking evaluation, the performance measures considered to assess the effectiveness of the proposed and existing tracking methods, and the implementation details. Section 3.3 presents the performance results of the proposed and existing methods. Section 3.4 analyzes and discusses the results.

3.1. Description of Experiments

To evaluate the performance of the proposed tracking method, we considered input test sequences from four different widely used and distinct datasets: Large-scale Single Object Tracking (LaSOT) [34], Generic Object Tracking Benchmark (GOT-10k) [35], A Benchmark and Simulator for UAV Tracking (UAV123) [36], and Object Tracking Benchmark (OTB50) [4]. These datasets cover a wide range of target tracking challenges, including deformations and variations in scale, nonuniform illumination, background clutter, occlusions, and abrupt camera motion. Consequently, these datasets enable a comprehensive evaluation of tracker robustness under diverse visual conditions. We employed ten video sequences from each dataset, yielding a total of 38,484 frames distributed as follows: 1195 frames from GOT-10k, 22,853 frames from LaSOT, 4527 frames from OTB50, and 9991 frames from UAV123. Figure 2 shows representative test frames extracted from the selected sequences within each of the four evaluated datasets. Each frame in Figure 2 shows the target estimated by the proposed method, referred to as Morphological Correlation Filtering (MCF), highlighted with a red bounding box. For performance comparison, we evaluated five correlation-based trackers: Kernelized Correlation Filters (KCFs) [16] (magenta), Spatial–Temporal Regularized Correlation Filters (STRCFs) [22] (blue), Background-Aware Correlation Filters (BACFs) [21] (cyan), SRDCF [17] (green), and SRDCF with adaptive template decontamination (SRDCFd) [19] (yellow). Note that the estimate produced by the MCF method closely matches the ground truth, which is highlighted with a white bounding box.

Performance Evaluation Metrics

Tracking performance is evaluated using three criteria: accuracy of the target’s center location, accuracy of the target’s bounding box overlap, and detection rate. Typically, the accuracy and robustness of the target’s center-location estimation and bounding box overlap are characterized by two commonly used metrics: precision and success.

The precision metric assesses the accuracy of the predicted target’s center location by computing the Euclidean distance between the centers of the predicted bounding box and the corresponding ground truth. Let $(x_t^i,y_t^i)$ and $(x_g^i,y_g^i)$ denote the centers of the predicted and ground truth bounding boxes at frame i, respectively. The normalized location error (NLE) is given by

$${\mathrm{NLE}}_i={\displaystyle \frac{1}{D}}\sqrt{{(x_t^i-x_g^i)}^2+{(y_t^i-y_g^i)}^2},$$

(25)

where $D=\sqrt{W^2+H^2}$ is the frame diagonal length, with W and H denoting the frame width and height, respectively. The precision score is given by the proportion of frames where ${\mathrm{NLE}}_i$ is below a specified threshold $\tau$, as follows:

$$\mathrm{Precision}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}\left({\mathrm{NLE}}_i<\tau \right),$$

(26)

where N is the total number of frames and

$$\mathbb{I}\left(A\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}A\mathrm{is}\mathrm{true},\hfill \\ 0\hfill & \mathrm{if}A\mathrm{is}\mathrm{false}.\hfill \end{array}\right.$$

(27)

The success metric evaluates the overlap between the predicted bounding box $B_t^i$ and the ground-truth bounding box $B_g^i$ at each frame. Consider the Intersection over Union (IoU) metric, which quantifies the spatial overlap between the estimated box $B_t^i$ and the ground-truth box $B_g^i$ and is computed as follows:

$${\mathrm{IoU}}_i={\displaystyle \frac{|B_t^i\cap B_g^i|}{|B_t^i\cup B_g^i|}},$$

(28)

where $|·|$ denotes the area of a region. The success metric consists of the fraction of frames in which ${\mathrm{IoU}}_i$ exceeds a threshold $\tau \in [0,1]$, defined as follows:

$$\mathrm{Success}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}\left({\mathrm{IoU}}_i>\tau \right).$$

(29)

Another important characteristic of a tracking algorithm is the detection rate (DR), defined as the percentage of frames for which the tracker detects the target and successfully estimates its state. In this work, the target is considered detected if ${\mathrm{IoU}}_i\ge \tau$, with $\tau =0.5$. Thus, the DR can be computed as follows:

$$\mathrm{DR}={\displaystyle \frac{100}{N}}\sum _{i=1}^N\mathbb{I}\left\{{\mathrm{IoU}}_i\ge 0.5\right\}.$$

(30)

3.2. Implementation Details

The proposed method was implemented in Python 3.13.2 using NumPy 2.2.6, Numba 0.61.2, OpenCV 4.11.0, SciPy 1.15.2, scikit-image 0.25.2, and PyTorch 2.7.0. The existing methods, KCF [16], STRCF [22], BACF [20], SRDCF [17], and SRDCFd [19], were run in MATLAB R2024b using the authors’ released code. The experiments were carried out on a MacBook Pro computer equipped with an Apple M3 Max processor and 64 GB of RAM, running macOS 15.5.

3.3. Performance Evaluation Results

The tracking performance of the proposed method and the considered correlation-based methods is evaluated by processing 40 videos from the GOT-10k, LaSOT, OTB50, and UAV123 datasets, as shown in Figure 2. For all trackers, the initial target location and bounding box are initialized from the ground-truth annotation of the datasets. For each subsequent frame in the sequence, the tracker estimates the target state (center coordinates and bounding box). Additionally, the NLE and IoU metrics are computed between the tracker’s estimate and the ground-truth annotation.

Table 1. Tracking performance of the proposed Morphological Correlation Filter (MCF) and existing methods on 40 sequences (38,484 frames) from the GOT-10k, LaSOT, OTB50, and UAV123 datasets. The results are presented as mean ± standard deviation for the normalized location error (NLE), intersection over union (IoU), and as a percentage for detection rate (DR).

Dataset	Tracker	NLE	IoU	DR (%)
GOT-10k	STRCF	$0.018\pm 0.016$	$0.649\pm 0.228$	70.1
	BACF	$0.015\pm 0.015$	$0.672\pm 0.216$	73.4
	SRDCF	$0.047\pm 0.091$	$0.579\pm 0.299$	59.7
	SRDCFd	$0.051\pm 0.099$	$0.577\pm 0.298$	59.6
	KCF	$0.036\pm 0.065$	$0.463\pm 0.321$	47.0
	MCF	$0.016\pm 0.026$	$0.747\pm 0.174$	90.2
LaSOT	STRCF	$0.033\pm 0.047$	$0.484\pm 0.313$	51.4
	BACF	$0.083\pm 0.150$	$0.482\pm 0.303$	53.5
	SRDCF	$0.122\pm 0.230$	$0.398\pm 0.296$	44.7
	SRDCFd	$0.099\pm 0.115$	$0.408\pm 0.369$	49.2
	KCF	$0.149\pm 0.182$	$0.163\pm 0.233$	12.1
	MCF	$0.032\pm 0.054$	$0.575\pm 0.176$	65.6
OTB50	STRCF	$0.012\pm 0.014$	$0.719\pm 0.152$	91.7
	BACF	$0.016\pm 0.029$	$0.696\pm 0.179$	90.8
	SRDCF	$0.016\pm 0.031$	$0.694\pm 0.187$	87.3
	SRDCFd	$0.012\pm 0.012$	$0.706\pm 0.164$	89.1
	KCF	$0.034\pm 0.085$	$0.592\pm 0.256$	70.8
	MCF	$0.011\pm 0.013$	$0.703\pm 0.211$	89.9
UAV123	STRCF	$0.008\pm 0.009$	$0.634\pm 0.206$	74.4
	BACF	$0.007\pm 0.008$	$0.662\pm 0.168$	81.7
	SRDCF	$0.008\pm 0.011$	$0.636\pm 0.193$	74.5
	SRDCFd	$0.009\pm 0.011$	$0.645\pm 0.187$	78.5
	KCF	$0.045\pm 0.078$	$0.408\pm 0.293$	39.7
	MCF	$0.004\pm 0.005$	$0.754\pm 0.200$	90.1
All datasets	STRCF	$0.017\pm 0.021$	$0.632\pm 0.224$	71.9
	BACF	$0.030\pm 0.050$	$0.628\pm 0.216$	74.8
	SRDCF	$0.048\pm 0.090$	$0.576\pm 0.243$	66.5
	SRDCFd	$0.042\pm 0.592$	$0.584\pm 0.254$	69.1
	KCF	$0.066\pm 0.102$	$0.406\pm 0.275$	42.4
	MCF	$0.015\pm 0.024$	$0.694\pm 0.190$	83.9

presents, for each tracker, the mean ± standard deviation of NLE and IoU, computed over all sequences shown in Figure 2 across the four datasets, as well as the percentage of DR. We observe that the proposed MCF tracker achieves the best overall performance and the best results on most datasets. On the GOT-10k dataset, the MCF and BACF trackers achieve the lowest NLE among all evaluated trackers. However, MCF outperforms BACF in terms of IoU and DR. The STRCF tracker performs slightly below BACF, whereas the SRDCF, SRDCFd, and KCF trackers achieve lower performance on this dataset.

On the LaSOT dataset, the MCF tracker produces the best results among all tested trackers in terms of NLE, IoU, and DR. The STRCF tracker achieves performance comparable to that of MCF in terms of NLE. However, the MCF tracker outperforms STRCF in terms of DR and IoU. The BACF tracker exhibits performance comparable to that of STRCF, achieving a higher DR. The SRDCF and SRDCFd trackers exhibit moderate performance, whereas KCF achieves the lowest performance among the evaluated trackers. Note that the LaSOT dataset presents more difficult tracking challenges, including very long sequences that can induce drift errors, out-of-view events, and long occlusions that make consistent target detection difficult, as well as large appearance changes of the target and strong perturbations, such as non-uniform illumination and background clutter.

On the OTB50 dataset, the MCF tracker yields the lowest NLE, indicating the best center-location accuracy. The STRCF tracker achieves slightly higher IoU and DR than the MCF tracker. The BACF tracker performs comparably to the MCF tracker. The SRDCF and SRDCFd trackers produce competitive results, and the KCF tracker exhibits the weakest performance among the evaluated methods.

On the UAV123 dataset, the proposed MCF tracker achieves the best results among all evaluated methods in terms of NLE, IoU, and DR. The BACF tracker produces competitive results in terms of IoU and DR. The SRDCF and SRDCFd trackers also exhibit good performance, whereas the KCF tracker produces the worst results. Across all datasets combined, the proposed MCF tracker achieves the best overall performance in terms of NLE, IoU, and DR values, indicating that the morphological correlation filtering combined with the trained neural predictive models generalizes well across diverse tracking conditions, particularly on larger and more challenging datasets (GOT-10k, LaSOT, UAV123).

Furthermore, to evaluate robustness, we computed precision and success plots for the proposed MCF tracker and the considered trackers across all datasets. These plots are shown in Figure 3 and Figure 4. Precision plots depict, as a function of the NLE threshold, the fraction of frames for which the center-location error falls below the threshold, whereas success plots depict, as a function of the IoU threshold, the fraction of frames for which the overlap exceeds the threshold. These results plots characterize a tracker’s localization accuracy and robustness across operating points. The area under the curve (AUC) serves as a single scalar metric, obtained by integrating the curve over the threshold range.

On the GOT-10k dataset, the MCF tracker with precision AUC of 0.846 and success AUC of 0.748 yields the best results. The BACF and STRCF trackers are competitive, yielding precision AUCs of 0.840 and 0.817 and success AUCs of 0.675 and 0.649, respectively. The KCF, SRDCF, and SRDCFd trackers yield the worst results. On the LaSOT dataset, the MCF and STRCF trackers achieve the highest center-location accuracy, each with a precision AUC of 0.724. In terms of IoU-based success, MCF achieves the best performance, with a success AUC of 0.575. The BACF tracker yields competitive performance, with a precision AUC of 0.670 and a success AUC of 0.492. The SRDCF and SRDCFd trackers achieve moderate results, with precision AUCs of 0.570 and 0.511, respectively, and success AUCs of 0.411 and 0.426, respectively. The KCF tracker exhibits the weakest performance, with a precision AUC of 0.479 and a success AUC of 0.183.

On the OTB50 dataset, STRCF, BACF, SRDCF, SRDCFd, and MCF exhibit similar performance overall. For center-location accuracy, MCF and STRCF obtain the highest precision AUCs of 0.889 and 0.880, respectively. For IoU-based success, STRCF, MCF, and SRDCFd tracker achieve the highest AUCs of 0.719, 0.707, and 0.704, respectively.

On the UAV123 dataset, MCF yields the best performance in both precision and success, with AUCs of 0.943 and 0.754, respectively. For center-location accuracy, BACF, STRCF, SRDCF, and SRDCFd trackers also perform well, with precision AUCs of 0.922, 0.913, 0.909, and 0.902, respectively. For IoU-based success, SRDCFd, BACF, SRDCF, and STRCF trackers achieve AUCs of 0.664, 0.661, 0.635, and 0.633, respectively. In contrast, KCF yields the lowest performance scores on both metrics, with a precision AUC of 0.735 and a success AUC of 0.417.

When considering all datasets, the MCF tracker yields the strongest overall performance in both precision and success, with AUCs of 0.846 and 0.748, respectively. Note that the center-location accuracy of BACF and STRCF, with precision AUCs of 0.840 and 0.817, are comparable to that of MCF. However, their IoU-based success AUCs of 0.675 and 0.649, respectively, are significantly lower than that of MCF. The SRDCF and SRDCFd trackers yield closely matched results, with precision AUCs of 0.697 and 0.694 and success AUCs of 0.584 and 0.583, respectively. KCF exhibits the lowest overall performance among the evaluated methods.

Finally, for image sequences with a resolution of 800 × 600 pixels, the method achieved an average processing rate of 15.53 frames per second, with a standard deviation of 4.2, on the reference hardware described in Section 3.2. The observed variability in processing speed was primarily due to target scale variations across consecutive frames. Morphological-correlation filtering is inherently well suited to parallel computation, which can considerably enhance processing speed for real-time applications when implemented on dedicated hardware such as a graphics processing unit (GPU).

3.4. Discussion

The results of the extensive experiments across four datasets demonstrate that the proposed MCF tracker achieves superior performance in terms of accuracy of center location estimation (precision/NLE), bounding box overlap (success/IoU), and detection rate (DR). On the GOT-10k dataset, the MCF tracker achieves the lowest NLE and higher IoU and DR scores compared with with those of the evaluated existing trackers. Because GOT-10k primarily assesses category-level generalization across diverse object types and scenes, these results confirm the strong generalization capability of the of MCF tracker to unseen target categories.

On the LaSOT dataset, which evaluates long-term tracking, including robustness to drift and reinitialization after occlusions, the MCF tracker achieves the best performance across all three evaluation criteria.

On the OTB50 dataset, the performance of all tested trackers is quite similar. Note that the limited size and diversity of this dataset reduce appearance and motion variability, thereby narrowing performance variation among trackers. Under these conditions, MCF achieves the lowest NLE, whereas STRCF yields slightly higher IoU and DR.

On the UAV123 dataset, which is characterized by small targets and rapid camera motion, the MCF tracker exhibits the strongest performance across all three criteria, achieving the lowest NLE and the highest IoU and DR. The BACF and STRCF trackers are competitive in NLE but underperform in IoU and DR. SRDCF and SRDCFd yield moderate results, whereas KCF exhibits the weakest performance across all measures.

Across all four datasets, the proposed MCF tracker achieves higher accuracy and robustness, particularly on the larger and more challenging datasets. These results are attributable to the proposed morphological correlation, which improves target localization and suppresses background clutter, and to the learned predictive model, which reduces drift and simplifies reinitialization after occlusions and out-of-view events. The integration of morphological correlation and learned predictive modeling provides a robust and effective approach to visual tracking.

Two primary situations were identified in which the tracking process can fail. The first occurs when the target appearance undergoes substantial variation between consecutive frames relative to the reference template, resulting in a detection error. The second arises when the target motion is highly irregular and the prediction neural model estimates a position corresponding to the background rather than the true target. In both cases, the algorithm activates the reinitialization procedure. In future work, these issues will be addressed to improve tracking performance.

4. Conclusions

A reliable single-object tracking method based on adaptive morphological correlation filtering and neural predictive models was presented. Because morphological correlation was applied to binary threshold decompositions of the input and reference images, improved robustness to illumination variations and sensor noise was obtained. Consequently, reliable target detection and accurate localization in the scene were achieved.

The incorporation of trained neural models enabled accurate estimation of the detected target’s bounding box and prediction of its location in subsequent frames, thereby enabling tracking under target scale variations and abrupt trajectory changes. In addition, the proposed mechanisms for drift correction and tracking reinitialization improved the robustness of target tracking under challenging conditions, including long-term tracking and target occlusions.

The performance of the proposed tracking method was evaluated extensively on four existing image sequence datasets. Performance was quantified using objective measures of target center-location accuracy, bounding box accuracy, and detection rate. For comparison, five widely used correlation filter-based trackers were also evaluated. The proposed tracking method achieved the best overall performance across the considered tracking criteria, thereby validating that the suggested morphological-correlation filtering combined with trained neural models generalized well across diverse tracking conditions, particularly on large and challenging datasets.

The results confirmed that the proposed tracking method is reliable and robust for challenging tracking conditions. Future work will focus on integrating massive parallel processing for high-rate applications and conducting extensive testing in real-world scenarios.