Prompt Self-Correction for SAM2 Zero-Shot Video Object Segmentation

Abstract

Foundation models, exemplified by the Segment Anything Model (SAM), have revolutionized object segmentation with their impressive zero-shot capabilities. The recent SAM2 extended these abilities to the video domain, utilizing an object pointer and memory attention to maintain temporal segment consistency. However, a critical limitation of SAM2 is its vulnerability to error accumulation, where an initial incorrect mask can propagate through subsequent frames, leading to tracking failure. To address this, we propose a novel method that actively monitors the temporal segment consistency of masks by evaluating the distance of object pointers across frames. When a potential error is detected via a sharp increase in distance, our method triggers a particle filter based re-inference module. This framework models object’s motion to predict a corrected bounding box, effectively guiding the model to recover the valid mask and preventing error propagation. Extensive zero-shot evaluations on DAVIS, LVOS v2, YouTube-VOS and qualitative results show that the proposed, parameter-free procedure consistently improves temporal coherence, raising mean IoU by 0.1 on DAVIS, by 0.13 on the LVOS v2 train split and 0.05 on the LVOS v2 validation split, and by 0.02 on YouTube-VOS, thereby offering a simple and effective route to more robust video object segmentation with SAM2.

1. Introduction

Segment Anything Model (SAM) [1], well known as a general-purpose foundation model [2] for object recognition and segmentation tasks, is a visual inference-based architecture that can generate segmentation masks of objects on the fly in response to various types of user prompts. As its name suggests, class-agnostic SAM is essentially comparable to the parametric method that pre-trains the image classes in that it can integrate and process various types of inputs (e.g., points, boxes, masks) [3,4] in a zero-shot [5] manner and can generate accurate segmentation masks even for unseen objects.

The follow-up, SAM2 [6], is proposed as a segmentation model with high generality and accuracy for various visual prompts on high-dimensional dynamic video frames. It borrows the structure of SAM1 as it is, which provides fast zero-shot segmentation performance through fixed image embedding and multi-mask decoding structure, and aims to generate consistent masks for various input prompts by pre-training on a large-scale dataset for video object segmentation. Notably, the memory attention module for propagating the previous predicted mask information to obtain the final result accurately is designed to eliminate the need for cumbersome prompting work.

Despite the impressive performance in video object segmentation (VOS) tasks, SAM2 has a major limitation in that if the incorrect mask is predicted due to the variability of objects or occlusion in the immediate frame, this error can accumulate in the subsequent frames, which leads to the problem of being continuously distorted. Recently, various methods have attempted to post-correct erroneous prompts or masks, or to refine predictions with an auxiliary network, as in [7,8,9]. However, these approaches typically require additional training or computation and lack a principled, a priori criterion for deciding when correction should be invoked, leaving consistency across frames uncertain. Moreover, they generally overlook SAM2’s object pointer, a native signal that can indicate frame-to-frame inconsistency and thus provide a lightweight basis for targeted intervention.

We address the problem in two stages: recognition of an incorrect mask and post-correction. During video segmentation we first recognize whether the current frame’s mask has been perturbed. To this end, we leverage SAM2’s object pointer, a learnable token that cross-attends to image and prompt embeddings and thus summarizes object-specific context without reconstructing the full mask. Our framework computes the exponential moving average of the inter-frame object pointer distance and uses this statistic to assess mask consistency. When the EMA indicates inconsistency, the frame is flagged for post-correction. This pointer-driven criterion provides a lightweight and adaptive signal for recognizing when intervention is warranted.

For post-correction, we integrate a particle filter into SAM2 and use it to construct the re-inference prompt. The filter tracks the object via its bounding box and is updated only when the object score from SAM2 indicates that the target is present, which avoids spurious state updates. When re-inference is triggered, the particle filter predicts a corrected bounding box that serves as the prompt to re-query SAM2, yielding a recovered mask. This design couples a principled, training-free inconsistency recognizer with an efficient motion prior, enabling targeted re-inference that improves temporal stability while keeping computation modest.

The key contributions of the idea can be summarized as follows:

• We propose a mask consistency evaluation method that adaptively determines validity by thresholding the exponential moving average of the distance between the object pointers in consecutive frames (Section 3.1 and Section 3.2).
• We propose a SAM2-based framework that integrates a particle filter-based object tracker into the inference loop, updating it whenever the object score indicates presence. When the mask consistency evaluation triggers re inference, the framework uses the particle filter’s motion prediction to construct the prompt and re-query SAM2, thereby recovering valid masks (Section 3.3).
• Extensive experiments on DAVIS, the LVOS v2 train and validation splits and the YouTube-VOS validation set show that our framework yields small but consistent improvements over baseline SAM2 without using any learnable parameters (Section 4.3).

2. Related Works

This chapter provides a summary of the process for generating object pointers used to evaluate the consistency of masks in SAM2, as well as an overview of previous works on the subsequent approaches of the VOS task and conventional filter-based applications.

2.1. Deep Learning-Based Video Object Segmentation

Early VOS research mainly focused on learning pixel-level similarity based on CNNs to segment objects. S. Caelles et al.’s [10] pioneering study proposed a method to segment objects in an entire video using only the mask information of the first frame. Based on a pre-trained CNN, it learns appearance information and suggests the concept of one-shot VOS that segments objects in the entire video sequence using only the mask information of the first frame of the video, which influenced many subsequent studies. Later, with the advent of the Vision Transformer (ViT) [11], the VOS field gradually shifted to a Transformer-based architecture [12,13] that more effectively models spatio-temporal features. Z. Liu et al. [14] extend the successful Swin Transformer in the image domain to the video domain, proposing a method to efficiently model temporal context and spatial information simultaneously through shifted windows in the spatiotemporal dimension. The following study is known as a representative case that shows how effective the Transformer architecture is in video understanding tasks. These CNN and Transformer-based models achieved high segmentation accuracy for specific video datasets but showed limitations in that they overfitted to the training data [15], resulting in poor generalization performance for new domains or unseen objects.

Recently, Segment Anything Model, a foundation model pre-trained on a large dataset, has emerged, presenting a new paradigm in the VOS field. SAM has demonstrated the ability to precisely segment various objects without separate fine-tuning based on excellent zero-shot generalization performance. Accordingly, follow-up studies are actively being conducted to apply SAM to VOS by attaching additional lightweight modules or combining mask propagation and tracking mechanisms in the inference process. Cheng, Ho Kei et al. [16] presented the idea of decoupling the segmentation and tracking functions to apply the powerful segmentation capability of SAM to video, and proposed an efficient pipeline in which a separate tracker manages and propagates the masks generated by SAM.

2.2. Filter-Based Object Tracking in Deep Learning

Meanwhile, in the field of object tracking [17], traditional probability-based filtering methodologies such as the Kalman filter [18] and Particle filter [19] have been used for a long time. These filter-based algorithms assume the motion model of the object and predict and compensate for changes in the state over time, like position or velocity, thereby providing robust tracking performance even in temporary occlusion or appearance changes.

In the era of deep learning [20], these filter-based tracking techniques have been further developed by combining them with the powerful feature extraction capabilities of deep learning models. Many studies have combined CNN [21] or Transformer-based visual feature extractors and filter-based mechanisms [22] to complementarily utilize appearance information and motion information, thereby greatly improving the accuracy and stability of tracking. A. Bewley et al. [23] used the Kalman Filter to predict linear movements from the results of deep learning-based object detectors, and implemented fast and efficient multi-object tracking with simple matching based on the intersection over Union (IoU). By proposing a simple yet efficient multi-object real-time tracking framework that combines object detection results with a Kalman filter, they demonstrated how the Kalman filter can be used to predict object states and for data association in the era of deep learning. N. Wojke et al. [24] improved the previous work by adding CNN-based appearance features in addition to predicting motion with a Kalman filter, thereby greatly improving tracking performance in complex situations where objects are occluded or interacting. L. Bertinetto et al. [25] proposed a method to learn a similarity map between a target object and a candidate region by utilizing a Siamese Network. This opens up the possibility of combining deep learning-based trackers with filters by incorporating Kalman filters or other postprocessing techniques to improve the stability of tracking results.

While previous studies [23,24,25] have focused on enhancing the segmentation performance of the foundation model in VOS, this study focuses more on maintaining the continuity and stability aspects of tracking the target object over the whole video frame. In this paper, we propose a new approach to supplement unstable inference results and enhance temporal segment consistency while maintaining the strong generalization [26] performance of the foundation models.

3. Methods

In this section, we present a mask consistency evaluation module driven by object pointers, together with a particle filter-based framework for object tracking and re-inference. Figure 1 presents the overall architecture of the proposed framework. Our framework decides whether to update the particle filter based on SAM2’s object score and mask consistency. When this score indicates confident presence and the mask is consistent as shown by the green path in Figure 1 the mask decoder outputs the current mask and we update both the particle filter and the EMA statistics. During inference, if the object is present but the mask is inconsistent, as shown by the red path in Figure 1, we suspend updates to the particle filter and the EMA and run re inference. During re inference we form the prompt from the location predicted by the particle filter. After re-inference we choose between the first pass mask and the re-inferred mask by selecting the one that agrees more closely with the mask from the previous frame.

3.1. Object Pointer in SAM2

In SAM 2, the object pointer is defined as the output embedding of the single mask token after the transformer and serves as a compact representation of the target object’s identity. The decoder forms a single token stream by combining a learned table of mask tokens ${\mathbf{E}}_{\mathrm{mask}}\in {\mathbb{R}}^{M\times D}$ with the IoU token the optional object score token and the prompt tokens, and processes this stream together with image features using a two way attention transformer. After the attention updates the mask token slice ${\mathbf{H}}_{\mathrm{mask}}\in {\mathbb{R}}^{B\times M\times D}$ is produced and the default pointer is taken as ${\mathbf{H}}_{\mathrm{mask}}\in {\mathbb{R}}^{B\times 1\times D}$ with the option to use the multi mask tokens when desired. During video inference the pointer is maintained alongside memory embeddings accumulated from previous frames and performs memory attention at the current step to inject information about the object mask from past frames into the current image representation. The final mask is not chosen by the pointer itself, rather each mask token ${\mathbf{h}}_m$ is mapped by a small MLP to hypernetwork [27] coefficients ${\mathbf{w}}_m$ which act on the upscaled feature map $\mathbf{F}$ to produce logits $\mathbf{L}$, and the final selection is decided by the predicted quality and a stability-based fallback rule. Under this mechanism, the object pointer can be regarded as indicating the mask to be produced.

Figure 2 illustrates the relationship between the object pointer and the generated mask on a representative DAVIS clip. The upper row visualizes SAM2 masks for frames 57–64, and the lower panel plots the L2 distance between successive pointers. When the mask undergoes a pronounced deformation between frames 57 and 58, the pointer-distance curve jumps from 2 to 16 and then quickly returns to a low plateau once the segmentation stabilizes. Similar transient spikes co-occur with moments of degraded temporal consistency, whereas smooth, low-amplitude variations dominate when the same object is segmented reliably.

These dynamics yield a simple but powerful proxy. Because the object pointer is derived from the selected mask token, its temporal similarity reflects the stability of the underlying mask without requiring pixel space comparisons. When the target persists with similar shape and appearance, consecutive pointers remain close in embedding space, exhibiting high cosine similarity and small L2 change. When the target disappears, is occluded, or the hypothesis switches, pointer similarity drops sharply. Therefore, we measure temporal segment consistency directly in pointer space, using pointer-to-pointer distance as an adaptive surrogate for mask agreement across frames.

To quantify temporal stability, we use an exponential moving average of the inter-frame pointer distance. When this distance remains low, memory writes and tracker updates proceed as usual. When it increases rapidly above a threshold while the object score indicates presence, the particle filter guided re-inference is triggered and the mask is replaced only if the re-inferred hypothesis aligns more strongly with recent history. This pointer-derived criterion links recognition and correction without pixel-level overlap calculations, reducing sensitivity to transient geometric distortions and improving long-range identity preservation.

3.2. Mask Consistency Evaluation

Empirically, the Euclidean ($L_2$) distance between the object pointers in consecutive frames correlates strongly with the semantic stability of the corresponding masks. Large deviations of the mask yield large pointer distances, whereas minor shape changes produce only small pointer distances. This observation motivates the consistency criterion developed in this section. Leveraging this fact allows us to evaluate mask consistency without explicitly decoding the mask, thereby saving substantial computation. The instantaneous displacement is defined as follows:

Let the network-predicted pointer for frame t be ${\mathbf{p}}_t\in {\mathbb{R}}^D$.

$$d_t={\parallel {\mathbf{p}}_t-{\mathbf{p}}_{t-1}\parallel }_2=\sqrt{\sum _{i=1}^D{\left(p_{t,i}-p_{t-1,i}\right)}^2}.$$

(1)

During the first N frames, we simply use the mean of the collected values as the estimate. Their mean provides an initial scale estimate

$$d_{\mathrm{EMA}}^{\left(N\right)}={\displaystyle \frac{1}{N}}\sum _{k=1}^N{d}_k.$$

(2)

For $t>N$ the scale estimate is updated online via

$$d_{\mathrm{EMA}}^{\left(t\right)}=(1-\alpha )d_{\mathrm{EMA}}^{(t-1)}+\alpha d_t,\alpha \in (0,1),$$

(3)

where the smoothing coefficient $\alpha$ controls the reactivity–robustness trade-off. During the initial frames until the warm-up phase completes we fix the threshold at ${\theta }_0$ = 17. After the warm-up, we derive a per frame threshold ${\theta }_t$ from the current EMA value.

$${\theta }_t=\left\{\begin{array}{cc}{\theta }_0,\hfill & t\le N,\hfill \\ d_{\mathrm{EMA}}^{\left(t\right)},\hfill & t>N,\hfill \end{array}\right.$$

(4)

Re-inference is invoked when the object is present yet the mask undergoes an abrupt change. This condition uses the EMA threshold ${\theta }_t$ and the object score derived from SAM2’s IoU token. The threshold ${\theta }_t$ is computed from the EMA of the object pointer distances over previous frames, and an abrupt change is detected when the current distance exceeds this threshold, that is $d_t>{\theta }_t$. Because the object pointer distance is valid only when the object is present in adjacent frames, we require the object score to exceed the cutoff in both frames, $s_{t-1}>\kappa$ and $s_t>\kappa$. The re-inference trigger is summarized in Equation (5). The specific values of the EMA threshold $\gamma$ and the object score threshold $\kappa$ were determined empirically through an ablation study in Section 4.4.

$${\mathrm{ReInfer}}_t=\left(s_{t-1}>\kappa \right)\wedge \left(s_t>\kappa \right)\wedge \left(d_t>\gamma {\theta }_t\right),$$

(5)

After re-inference, the mask selection in Equation (15) chooses between the re-inferred mask ${\mathcal{M}}_t^{\mathrm{re}}$ and the baseline mask ${\mathcal{M}}_t^{\mathrm{base}}$. If ${\mathcal{M}}_t^{\mathrm{re}}$ is accepted the newly measured distance $d_t$ is used to update $d_{\mathrm{EMA}}^{\left(t\right)}$ and ${\theta }_t$ as above

3.3. Object Tracking and Re-Inference Based on Particle Filter

3.3.1. Object Tracking Using the Particle Filter

Given the binary mask of the current frame ${\widehat{\mathcal{M}}}_t=\{(y,x)\mid sigmoid\left({\varphi }_t(y,x)\right)>0.5\}$, the bounding rectangle is defined as follows:

$$\begin{array}{cc}{x}_0=\underset{(y,x)\in {\widehat{\mathcal{M}}}_t}{min}x,\hfill & \hfill x_1=\underset{(y,x)\in {\widehat{\mathcal{M}}}_t}{max}x,\end{array}$$

(6)

$$\begin{array}{cc}{y}_0=\underset{(y,x)\in {\widehat{\mathcal{M}}}_t}{min}y,\hfill & y_1=\underset{(y,x)\in {\widehat{\mathcal{M}}}_t}{max}y.\hfill \end{array}$$

(7)

We write the resulting box as ${\mathbf{o}}_t={[c_x,c_y,w_t^{\mathrm{box}},h_t^{\mathrm{box}}]}^{\top }\in {\mathbb{R}}^4$ with

$$c_x={\displaystyle \frac{x_0+x_1}{2}},c_y={\displaystyle \frac{y_0+y_1}{2}},w_t^{\mathrm{box}}=x_1-x_0,h_t^{\mathrm{box}}=y_1-y_0.$$

(8)

Using the box as state vector ${\mathbf{z}}_t={[c_x,c_y,w_t^{\mathrm{box}},h_t^{\mathrm{box}}]}^{\top }$, we maintain P particles ${\left\{{\mathbf{z}}_t^{\left(p\right)}\right\}}_{p=1}^P$ and their weights ${\left\{{\pi }_t^{\left(p\right)}\right\}}_{p=1}^P$, where ${\pi }_t^{\left(p\right)}\ge 0$ and ${\sum }_p{\pi }_t^{\left(p\right)}=1$.

• Initialization ($t=1$):

  $${\mathbf{z}}_1^{\left(p\right)}\sim \mathcal{N}\left({\mathbf{o}}_1,diag{(0.1w_1^{\mathrm{box}},0.1h_1^{\mathrm{box}},0.1w_1^{\mathrm{box}},0.1h_1^{\mathrm{box}})}^2\right),{\pi }_1^{\left(p\right)}={\displaystyle \frac{1}{P}}.$$

  (9)
• Prediction:

  $${\mathbf{z}}_{t|t-1}^{\left(p\right)}={\mathbf{z}}_{t-1}^{\left(p\right)}+{\mathit{ϵ}}_t^{\left(p\right)},{\mathit{ϵ}}_t^{\left(p\right)}\sim \mathcal{N}\left(\mathbf{0},{\sigma }_t^2\mathbf{I}\right),{\sigma }_t=0.05\left[\begin{array}{c}{w}_{t-1}^{\mathrm{box}}\\ h_{t-1}^{\mathrm{box}}\\ w_{t-1}^{\mathrm{box}}\\ h_{t-1}^{\mathrm{box}}\end{array}\right].$$

  (10)
• Update:

  $${\tilde{\pi }}_t^{\left(p\right)}=IoU\left({\mathbf{z}}_{t|t-1}^{\left(p\right)},{\mathbf{o}}_t\right)+\epsilon ,{\pi }_t^{\left(p\right)}={\displaystyle \frac{{\tilde{\pi }}_t^{\left(p\right)}}{{\sum }_q{\tilde{\pi }}_t^{\left(q\right)}}}.$$

  (11)
• Particle-filter confidence:

  $${\rho }_t=1-{\displaystyle \frac{H\left({\mathit{\pi }}_t\right)}{{log}_{2}P}},H\left({\mathit{\pi }}_t\right)=-\sum _p{\pi }_t^{\left(p\right)}{log}_2{\pi }_t^{\left(p\right)}.$$

  (12)
• Resampling: draw indices $I\sim Multinomial(P,{\mathit{\pi }}_t)$ and set

  $${\mathbf{z}}_t^{\left(p\right)}={\mathbf{z}}_{t|t-1}^{\left(I_p\right)},{\pi }_t^{\left(p\right)}={\displaystyle \frac{1}{P}}.$$

  (13)
• State estimate:

$${\widehat{\mathbf{z}}}_t=\sum _{p=1}^P{\pi }_t^{\left(p\right)}{\mathbf{z}}_t^{\left(p\right)}.$$

(14)

Here, $w_t^{\mathrm{box}}$ and $h_t^{\mathrm{box}}$ denote the width and height of the bounding-box component of ${\mathbf{z}}_t$; the symbol ${\pi }_t^{\left(p\right)}$ is reserved exclusively for particle weights and does not conflict with the box width $w_t^{\mathrm{box}}$.

The particle filter infers object presence from the SAM2 object score. When the object is judged to be present, framework update the particles based on the mask mIoU. Through these updates the particles capture the motion of the target, and this motion estimate is used to form the prompt when re inference is triggered.

3.3.2. Re-Inference with Particle Filter Prediction-Based Prompt

If ${\mathrm{ReInfer}}_t$ is true, our framework replace the previous prompt with observation box extracted from the current mask, the particle filter box that predicts the object motion, and a set of negative points sampled outside both boxes to suppress background regions. Negative points are drawn from a uniform distribution over all spatial locations that lie outside both the current bounding box and the particle filter bounding box. These elements constitute the prompt used for re inference at time. {box_coords, box_labels, point_coords,point_labels} and forwarded to the SAM-2 head for a single re-inference pass. The two boxes give complementary spatial hypotheses, one reflecting current appearance, the other captured motion, while the negative points sharpen object boundaries, allowing drift to be corrected without any additional training. Consequently, given the re-inferred mask ${\mathcal{M}}_t^{\mathrm{re}}$ and the base tracker output ${\mathcal{M}}_t^{\mathrm{base}}$, the final mask is chosen as the one that maximizes its agreement with the previous frame.

$${\mathcal{M}}_t=\underset{\mathcal{A}\in \{{\mathcal{M}}_t^{\mathrm{re}},{\mathcal{M}}_t^{\mathrm{base}}\}}{arg\; max}mIoU\left({\mathcal{M}}_{t-1},\mathcal{A}\right).$$

(15)

Our framework selects the final output mask as the candidate with the highest mIoU against the previously accepted mask, reflecting the observation that object shape and position change little over short time intervals [28,29]. This rule encourages gradual evolution of the segmentation rather than abrupt switches, which improves temporal coherence and reduces drift. The selected mask ${\mathcal{M}}_t$ is then used to update the object pointer and to resample the particle-filter state for the next frame.

Algorithm 1 details the proposed mask consistency evaluation and particle filter based object tracking and re-inference mechanism. Our correction scheme assesses mask consistency solely through the lightweight object pointer produced by SAM2, entirely avoiding full-mask operations. Also, an object tracking and subsequent re-inference step is implemented without learnable parameters or additional training, thereby preserving the foundation model’s inherent generalization capability while markedly improving temporal stability in object tracking.

Algorithm 1 Prompt Self-Correction and Re-inference

Require: Base mask ${\mathcal{M}}_t^{\mathrm{base}}$, previous mask ${\mathcal{M}}_{t-1}$, current pointer ${\mathbf{p}}_t$, previous pointer   ${\mathbf{p}}_{t-1}$, object scores $s_{t-1},s_t$, particle set ${\{{\mathbf{z}}_{t-1}^{\left(p\right)},{\pi }_{t-1}^{\left(p\right)}\}}_{p=1}^P$, EMA $d_{\mathrm{EMA}}^{(t-1)}$, threshold ${\theta }_{t-1}$,  frame index t Ensure: Final mask ${\mathcal{M}}_t$, updated particle set, updated $d_{\mathrm{EMA}}^{\left(t\right)}$, ${\theta }_t$  Hyper-parameters: warm-up N, smoothing $\alpha$, EMA threshold $\gamma$, warm-up threshold ${\theta }_0$,  object score cutoff $\kappa$, negatives K  1: $d_t\leftarrow {\parallel {\mathbf{p}}_t-{\mathbf{p}}_{t-1}\parallel }_2$  2: if$t\le N$then  3:     $d_{\mathrm{EMA}}^{\left(t\right)}\leftarrow {\displaystyle \frac{1}{t}}{\sum }_{k=1}^t{d}_k$, ${\theta }_t\leftarrow {\theta }_0$  4: else  5:     $d_{\mathrm{EMA}}^{\left(t\right)}\leftarrow (1-\alpha )d_{\mathrm{EMA}}^{(t-1)}+\alpha d_t$  6:     ${\theta }_t\leftarrow \gamma d_{\mathrm{EMA}}^{\left(t\right)}$  7: ${\mathrm{ReInfer}}_t\leftarrow (s_{t-1}>\kappa )\wedge (s_t>\kappa )\wedge (d_t>{\theta }_t)$  8: if$s_t>0$then  9:     Predict particles with noise ${\sigma }_t$ (Equation (10))  10:     Update weights via IoU (Equation (11))  11:     Resample and compute state ${\widehat{\mathbf{z}}}_t$ (Equation (13))  12: else  13:     Retain previous particle set, set ${\widehat{\mathbf{z}}}_t\leftarrow {\mathbf{o}}_t$  14: if$$then  15:     Build box prompt $\{{\mathbf{o}}_t,{\widehat{\mathbf{z}}}_t\}$ + K negatives  16:     Run SAM2 once ⟶${\mathcal{M}}_t^{\mathrm{re}}$  17:     ${\displaystyle {\mathcal{M}}_t\leftarrow arg\underset{\mathcal{A}\in \{{\mathcal{M}}_t^{\mathrm{re}},{\mathcal{M}}_t^{\mathrm{base}}\}}{max}mIoU({\mathcal{M}}_{t-1},\mathcal{A})}$  18:     if ${\mathcal{M}}_t={\mathcal{M}}_t^{\mathrm{base}}$ then  19:         $d_{\mathrm{EMA}}^{\left(t\right)}\leftarrow d_{\mathrm{EMA}}^{(t-1)}$, ${\theta }_t\leftarrow {\theta }_{t-1}$  20: else  21:     ${\mathcal{M}}_t\leftarrow {\mathcal{M}}_t^{\mathrm{base}}$   return${\mathcal{M}}_t$, particle set, $d_{\mathrm{EMA}}^{\left(t\right)}$, ${\theta }_t$

4. Results

4.1. Datasets

DAVIS (Densely Annotated VIdeo Segmentation) [30] is one of the earliest and most influential benchmarks for VOS. The 2017 multi-object release expands the original corpus to 150 full-HD sequences, providing $\approx 10459$ densely annotated frames and a total of 376 object instances together with per-frame pixel-accurate masks. LVOS v2 (Large-scale Long-term Video Object Segmentation) [31], published in 2024, is designed to stress test temporal generalization. It contains high-resolution (720p) clips sampled at 6 fps, summing to 296,401 annotated frames and 407,945 pixel-accurate masks. YouTube-VOS [32] remains the largest publicly available VOS dataset, comprising 3859 high-resolution clips with more than 232,000 instance masks.

4.2. Settings

All experiments were conducted on the validation splits of three public VOS benchmarks (LVOS v2, DAVIS, and YouTube-VOS) and LVOS v2 train split using a zero-shot protocol in which the ground-truth mask of the first frame in each video was converted into a prompt and no further fine-tuning was applied. A single dot located at the mask centroid (dot prompt) and a bounding box tightly enclosing the mask prompt types wew exmained in the experiments. LVOS v2 validation was evaluated with the dot prompt, whereas DAVIS and YouTube-VOS were evaluated with the bounding-box prompt. Performance was quantified using mean Intersection-over-Union (mIoU). To update the particle filter motion prior and the exponential moving-average (EMA) distance threshold only when the target was reliably present, the SAM2 object score had to exceed $\kappa >0$ before computing the pointer distance, updating the EMA, or resampling particles. During the first five frames a fixed distance threshold of ${\theta }_0=15$ was employed, after which the particle filter tracked 40 particles. Hyperparameters for the adaptive EMA threshold and for the number and type of re-inference prompts were selected in a separate ablation study and then kept fixed for the main evaluation. For determinism seed value was set to 0. All experiments used the SAM2-Tiny backbone and were run on a single NVIDIA A100 GPU.

4.3. Experiment Result Analysis

Extensive evaluations on three public VOS benchmarks demonstrate the effectiveness of our proposed method. When the framework is applied on the SAM2 backbone, mIoU rises by $+0.10$ percentage points on the DAVIS as a standard benchmark dataset for VOS. Detailed experimental results are presented in

Table 1. Video object segmentation performance (mIoU %) and frame per second (fps). LVOS v2 is split into train and validation.

	DAVIS			LVOS v2					YouTube-VOS
Model	mIoU	fps		Train mIoU	Train fps	Val mIoU	Val fps		mIoU	fps
SAM2	90.25	27.78		92.08	18.33	68.47	18.12		82.66	15.36
Ours	90.35	26.84		92.21	16.90	68.52	17.00		82.68	15.13

. For the LVOS v2 train and validation set (averaged over all videos),in Table 1, which assesses long-term temporal consistency the result increases by $+0.13$ and $+0.05$, respectively. Also for YouTube-VOS validation by $+0.02$.

Although the performance improvement in mIoU is marginal, consistent performance enhancements were observed across the train split (420 videos) and validation set (140 videos) of the LVOS dataset, which includes long video sequences of over 2000 frames, as well as on the YouTube VOS and DAVIS datasets. The proposed framework improves performance without additional training, thereby preserving the generalization ability of the foundation model.

Complementing these accuracy gains, the runtime analysis in Table 1 shows that, while our model’s enhanced accuracy entails a modest decrease in frames per second (fps), the processing speed remains at a highly interactive level suitable for practical applications. Specifically, our framework operates at 26.84 fps on DAVIS and maintains a robust 16.90–17 fps even on the complex, long-sequence LVOS dataset. This demonstrates a favorable trade-off, where a significant gain in segmentation quality is achieved while preserving a sufficiently high frame rate for interactive tasks, affirming the usability of our approach.

In addition to mIoU, the temporal consistency of the proposed framework was evaluated on the DAVIS dataset by computing the mIoU between masks of adjacent frames. Figure 3 presents the increase in temporal consistency for DAVIS videos when the proposed framework is applied to the SAM2 baseline. The results indicate that the proposed framework generally improves temporal consistency.

Figure 4 presents qualitative comparisons on LVOS and DAVIS. Green boxes denote cases in which re-inference succeeds, whereas red boxes indicate failures. Success is most pronounced when adjacent frames provide reliable context, i.e., when their masks are close to the ground truth.

In most examples, applying re-inference yields higher quality masks than the SAM2 baseline. The accepted mask is then used to update the EMA of the inter frame pointer distance, which affects the subsequent thresholding and update steps. With this corrected EMA, later frames are processed more reliably, and we observe a gradual improvement in the predicted masks over the sequence.

The second example video shows a case where the man is occluded by a balloon. In this sequence, re-inference on the second frame improves the predicted mask and brings it closer to the ground truth. This suggests that the trigger detects transient inconsistency caused by occlusion and recovers the object without additional training. The correction stabilizes the track and reduces drift in the following frames.

In contrast, by challenging sequences such as the fourth example (zebra), where the baseline has already drifted substantially from the ground truth, the selection rule may favor an inferior mask. This behavior arises because the final mask is chosen by maximizing agreement with the previous frame (mIoU), which can reinforce temporally consistent yet suboptimal hypotheses when the preceding mask is erroneous.

4.4. Ablation Study

The ablation study on the DAVIS dataset, evaluates design factors of the proposed framework, including the EMA threshold ratio that governs the re-inference decision, the object tracking strategy, Object score cutoff, the number of particles in the particle filter and the prompt type for re-inference.

Table 2. Ablation on DAVIS mIoU (%). Effect of EMA threshold ratio ${\gamma }_{\mathrm{EMA}}$.

${\mathit{\gamma }}_{\mathbf{EMA}}$	mIoU (%)
Without EMA	90.25
1.5	90.35
2.0	90.31
2.5	90.31

shows that the EMA threshold ratio, ${\gamma }_{\mathrm{EMA}}$, influences performance on the DAVIS validation set. Without EMA the model attains $90.25\%$ mIoU. Setting ${\gamma }_{\mathrm{EMA}}=1.5$ raises the score to $90.35\%$, indicating that a more responsive trigger corrects drift promptly. Higher values $2.0$ and $2.5$ reduce the gain to $90.31\%$ because re-inference is invoked less often and some errors persist. A ratio of $1.5$ therefore offers the best performance. Additional experiments on the hyperparameter ${\theta }_0$ yielded performance changes below $0.001$ mIoU, therefore detailed results are omitted.

Table 3. Ablation on DAVIS mIoU (%). Effect of tracking method.

Tracking Method	mIoU (%)
Without tracking	90.25
Kalman filter	90.27
Particle filter	90.35

confirms that incorporating an explicit tracking module into the re-prompting stage consistently improves segmentation quality. Re-prompting without any temporal tracker yields 90.25 mIoU, whereas adding a Kalman filter raises the score to 90.27, and substituting the Kalman filter with a particle filter further lifts performance to 90.35. These results indicate, first, that introducing a motion-aware prompt is beneficial in itself, and second, that the sample-based particle filter, maintaining multiple weighted box hypotheses and updating them with the non-linear IoU likelihood, produces a more accurate and robust motion estimate than the single-Gaussian Kalman filter, thereby delivering the highest overall accuracy.

Table 4. Ablation on DAVIS. Effect of number of particles on mIoU and average frames per second (fps).

Particles	mIoU	Avg fps
20	90.33	27.43
40	90.35	26.84
60	90.32	26.39
128	90.29	26.35
256	90.30	23.82

shows a clear speed accuracy trade off as the particle count increases. Moving from 20 to 40 particles yields a small gain in mIoU from 90.33 to 90.35 with a modest drop in throughput from 27.43 to 26.84 fps. Beyond 40 particles the accuracy does not improve and in fact declines slightly to 90.30 at 256 particles while the average fps continues to fall to 23.82. This pattern suggests that the particle filter is already well specified with a modest budget and that additional samples bring little benefit while increasing computational cost. A practical choice is 40 particles when accuracy is the priority and 20 particles for speed sensitive settings since it preserves most of the accuracy at the highest throughput.

In

Table 5. Ablation on DAVIS. Effect of object-score threshold $\kappa$.

Object Score $\mathit{\kappa }$	mIoU
0.0	0.9035
0.1	0.9034
0.2	0.9030
0.3	0.9030
0.4	0.9030
0.5	0.9030
0.6	0.9030

, the object score cutoff $\kappa$ specifies the confidence that the object is present at the time of re inference. A larger object score indicates that SAM2 is more certain that the object exists in the current frame. The ablation shows that initiating re-inference at the earliest positive presence detection, namely by setting $\kappa =0$ which corresponds to the onset of the mask decoder’s object existence decision, yields the highest overall mIoU.

The ablation in

Table 6. Ablation on DAVIS. Effect of re-inference prompt type.

Re-Inference Prompt Type	mIoU (%)
Without re-inference	90.25
1 point (random)	90.23
3 points (random)	90.33
5 points (random)	90.33
10 points (random)	90.32
Bounding box (without tracking)	90.28
Bounding box (with tracking)	90.29
Dual box + 1 negative point	90.32
Dual box + 3 negative points	90.35
Dual box + 5 negative points	90.34
Dual box + 10 negative points	90.31

indicates that prompt design, rather than tracking alone, drives most of the gains. Relative to the no re-inference baseline at 90.25 mIoU, point-only prompts are fragile: a single random point slightly degrades performance (90.23), while adding a few points recovers modest improvements that saturate around three to five points (90.33) and do not further improve with ten points (90.32). A single bounding box yields only a minor benefit (90.28), and adding tracking to the box provides a negligible increment (90.29), suggesting that box localization without boundary sharpening is insufficient. The best results arise from the dual-box prompt augmented with a small number of negative points, peaking at three negatives (90.35), with diminishing or slightly adverse effects as negatives increase to five (90.34) or ten (90.31). These trends support the interpretation that complementary spatial hypotheses from the dual boxes capture motion and extent, while a modest number of negative points sharpens boundaries. Excessive negatives constrain the mask and can clip true object regions. Overall, the improvements are small but consistent, and the dual-box plus three negatives setting offers the most favorable accuracy–robustness trade-off.

5. Conclusions

This work targeted a central failure mode of SAM2 in video object segmentation. When a per frame mask is corrupted by appearance change or occlusion the error can bias subsequent predictions and erode temporal consistency. We addressed this with a selective strategy composed of recognition and correction. Recognition operates in pointer space by monitoring the exponential moving average of the inter frame object pointer distance and it is gated by the SAM2 object existence score so that action is taken only when the target is present. When inconsistency is recognized we invoke correction through a particle filter that predicts a corrected bounding box and turns it into a prompt for SAM2. The approach is training free, lightweight, and integrates cleanly into the inference loop while avoiding pixel level overlap computations.

Experiments across DAVIS, the LVOS v2 train and validation splits, and the YouTube VOS validation set show small but consistent gains over baseline SAM2 while preserving interactive throughput and the zero shot character of the foundation model. Qualitative results indicate that the trigger captures transient failures such as brief occlusions or abrupt appearance shifts and that the particle filter guided prompt helps the model recover a valid mask and maintain identity over time. In practice the rule that selects the candidate with the highest mIoU against the previously accepted mask encourages smooth evolution of the segmentation rather than abrupt switches which improves coherence and reduces drift.

The study suggests several directions for improvement. The current selection can prefer a locally consistent but suboptimal hypothesis when the recent reference has already drifted, so future work can consider multi frame consensus or uncertainty aware agreement. Prompt construction can adapt more flexibly to scene context beyond boxes and a small number of negative points and re detection cues can help in long sequences with heavy occlusions or identity switches. More broadly the same recognition and correction principle can be applied to other promptable segmentation systems, offering a general recipe for test time stabilization without additional learning.