LF-SSM: Lightweight HiPPO-Free State Space Model for Real-Time UAV Tracking

Highlights

What are the main findings?

• We propose LF-SSM, a lightweight HiPPO-free state space model that reformulates state evolution on Riemannian manifolds through geodesic dynamics, eliminating complex discretization procedures and specialized hardware kernels required by existing SSM methods.
• The proposed Geodesic State Module (GSM) performs state updates through tangent space projection and exponential mapping on the unit sphere, providing adaptive local coordinate systems that preserve geometric structure of tracking features.

What are the implications of the main findings?

• LF-SSM achieves state-of-the-art performance on multiple UAV tracking benchmarks while running at 69 FPS with only 18.5M parameters on edge platforms (Jetson Orin Nano), enabling practical real-time deployment on resource-constrained UAV systems.
• The manifold-based formulation demonstrates that geometric approaches can effectively replace HiPPO-derived state transitions in visual tracking, opening new research directions for efficient sequence modeling without relying on fixed polynomial bases.

Abstract

Visual object tracking from unmanned aerial vehicles (UAVs) demands both high accuracy and computational efficiency for real-time deployment on resource-constrained platforms. While state space models (SSMs) offer linear computational complexity, existing methods face critical deployment challenges. They rely on the HiPPO framework with complex discretization procedures and employ hardware-aware algorithms optimized for high-performance GPUs, which introduce deployment overhead and are difficult to transfer to edge platforms. Additionally, their fixed polynomial bases may cause information loss for tracking features with complex geometric structures. We propose LF-SSM, a lightweight HiPPO (High-order Polynomial Projection Operators)-free state space model that reformulates state evolution on Riemannian manifolds. The core contribution is the Geodesic State Module (GSM), which performs state updates through tangent space projection and exponential mapping on the unit sphere. This design eliminates complex discretization and specialized hardware kernels while providing adaptive local coordinate systems. Extensive experiments on UAV benchmarks demonstrate that LF-SSM achieves state-of-the-art performance while running at 69 frames per second (FPS) with only 18.5 M parameters, demonstrating superior efficiency for real-time edge deployment.

1. Introduction

Visual object tracking from unmanned aerial vehicles (UAVs) has become increasingly important for autonomous aerial perception. Applications such as aerial surveillance [1,2], search and rescue operations [3,4], precision agriculture [5], and autonomous navigation [6] all rely on accurate tracking from aerial platforms. However, UAV platforms impose strict limitations on computational resources, power budgets, and payload capacity. These constraints require tracking algorithms to achieve both high accuracy and real-time performance under resource-constrained conditions [7,8,9,10].

Transformer-based trackers [11,12,13,14,15] achieve remarkable accuracy through powerful global modeling capabilities, but their quadratic computational complexity with respect to sequence length limits their deployment on edge platforms. State space models (SSMs) [16] offer an alternative with linear computational complexity, and recent SSM-based tracking methods [17,18] have shown promising results. However, existing SSM architectures face deployment challenges on resource-constrained platforms. Mamba [19] employs hardware-aware algorithms specifically optimized for high-performance GPUs, which are difficult to efficiently deploy on edge platforms with limited computational resources.

Beyond deployment efficiency, modern SSM architectures present a fundamental limitation in feature representation. Recent SSM-based methods including Mamba [19] rely on the HiPPO (High-order Polynomial Projection Operators) framework [20] to construct their state transition matrices. HiPPO projects historical signals onto a fixed Legendre polynomial basis to achieve long-range memory. While this approach enables efficient sequence modeling, the fixed polynomial basis represents a universal design that does not adapt to task-specific data distributions. According to the manifold hypothesis [21,22], high-dimensional visual features typically reside on low-dimensional manifolds embedded in the ambient space. This geometric perspective has been validated in visual understanding tasks, where hyperspherical representations improve feature discrimination by constraining embeddings to unit spheres [23,24]. The success of normalized feature spaces in Siamese tracking networks further supports this view, as cosine similarity-based matching inherently operates on angular relationships between feature vectors on hyperspheres [25]. In UAV tracking scenarios, target appearance features exhibit complex intrinsic geometric structures due to deformation, scale variation, and viewpoint change [26]. These transformations define smooth trajectories in feature space rather than arbitrary jumps, which is precisely the characteristic of manifold-structured data. These structures fundamentally differ from smooth polynomial curves. Projecting such manifold-structured features onto a fixed Euclidean basis causes information loss during the state compression process, leading to suboptimal tracking performance in challenging scenarios.

Based on this analysis, we identify two critical limitations of existing SSM-based methods for UAV tracking. First, the reliance on hardware-aware algorithms makes them difficult to deploy on resource-constrained edge platforms. Second, the fixed polynomial basis causes information loss when representing tracking features with complex geometric structures. As illustrated in Figure 1a, HiPPO-based SSMs project manifold-structured hidden states onto a fixed polynomial coefficient space, which may lose geometric information during compression.

To address these limitations, we propose LF-SSM, a lightweight HiPPO-free state space model for UAV tracking. Our key insight is that if tracking features naturally reside on manifolds, the state space itself should be defined on a manifold rather than in Euclidean space. As shown in Figure 1b, we reformulate state evolution as geodesic motion on Riemannian manifolds, where the tangent space at each state provides an adaptive local coordinate system. This formulation eliminates the need for HiPPO-derived state transition matrices and specialized hardware kernels. The adaptive tangent space naturally changes as the state evolves, avoiding the information loss caused by fixed basis compression. This design enables efficient deployment on edge platforms while preserving the geometric structure of tracking features.

Our contributions are summarized as follows:

• We propose LF-SSM, a lightweight HiPPO-free state space model that fundamentally reformulates state evolution on Riemannian manifolds for visual tracking. Unlike existing SSM methods that project features onto fixed Legendre polynomial bases, our geodesic dynamics naturally preserve the geometric structure of tracking features through adaptive local coordinate systems. This eliminates both the computational overhead of HiPPO discretization and the information loss from fixed basis compression, while maintaining linear complexity $O\left(TEN\right)$ with respect to sequence length.
• We design a Geodesic State Module (GSM) that performs state updates through tangent space projection and exponential mapping on the unit sphere $S^{N-1}$. The core innovation lies in replacing the $N\times N$ HiPPO-derived state transition matrix with geometric operations involving only $O\left(N\right)$ vector computations. This design eliminates the need for specialized hardware kernels (such as Mamba’s selective scan) while providing adaptive tangent spaces $T_{h_t}{S}^{N-1}$ that change with the evolving state. The input-dependent step size ${\mathrm{\Delta }}_t=\sigma \left({\mathrm{Linear}}_{\mathrm{\Delta }}\left({\tilde{x}}_t\right)\right)$ enables content-aware updates, and the prior velocity mechanism $v_{t+1}^{\mathrm{prior}}={\mathrm{Proj}}_{h_{t+1}}\left(v_t\right)$ implements geometric forgetting through parallel transport approximation.
• Extensive experiments on four UAV benchmarks (UAV123, VisDrone, ARDMAV, LaSOT) demonstrate that LF-SSM achieves state-of-the-art performance while enabling practical real-time deployment on resource-constrained platforms. Specifically, LF-SSM-L achieves 73.2% AUC on UAV123, outperforming all SSM-based baselines by significant margins.

The remainder of this article is organized as follows. Section 2 reviews the related work in UAV tracking and state space models. Section 3 introduces the preliminary background. In Section 4, we elaborate on the proposed LF-SSM. Section 5 analyzes the computational efficiency. Section 6 presents our experimental results. Section 7 provides discussion on the theoretical analysis and limitations. Finally, conclusions are drawn in Section 8.

2. Related Work

2.1. UAV Visual Tracking

Siamese network-based methods have dominated UAV tracking due to their balance between accuracy and efficiency. SiamFC [25] pioneers the fully convolutional matching framework for visual tracking. Subsequent works extend this paradigm through region proposal networks [27,28,29] and anchor-free prediction [30,31,32,33]. For UAV-specific challenges, HiFT [34] develops hierarchical feature fusion to handle scale variations, while TCTrack [35] and TCTrack++ [36] address temporal contexts through online template updating. UDAT [37] proposes unsupervised domain adaptation for cross-domain UAV tracking. SGDViT [38] introduces saliency-guided dynamic vision transformers. AVTrack [39] explores adaptive approaches for aerial scenarios. While these methods achieve real-time performance, they are limited by the local receptive fields of convolutional operations.

Transformer-based methods have advanced tracking accuracy through global attention mechanisms. TransT [15] first introduces cross-attention for template-search feature fusion. STARK [14] and MixFormer [40,41] further develop spatio-temporal transformers with learned queries and mixed attention modules. OSTrack [11] proposes a one-stream framework that unifies feature extraction and relation modeling, while SeqTrack [13] reformulates tracking as a sequence generation task. AiATrack [42] introduces attention in attention modules for enhanced feature interaction. SimTrack [43] and MT-Track [44] explore simplified architectures and regularization strategies. ROMTrack [45] develops robust online tracking with memory. ARTrack [46] and ODTrack [47] propose autoregressive and online dense frameworks. VideoTrack [48] explores video-level temporal modeling for robust tracking. EVPTrack [49] introduces explicit visual prompting for adaptive tracking. SPMTrack [50] proposes sparse-to-dense pyramid matching for efficient tracking. HIPTrack [51] develops hierarchical interaction propagation for long-term tracking. These approaches achieve remarkable accuracy on various benchmarks [52,53,54], yet their quadratic computational complexity poses challenges for UAV deployment.

Lightweight tracking methods have been developed to address deployment constraints on edge platforms [55]. TSTrack [56] employs neural architecture search to discover efficient network structures. FEAR [57] and E.T.Track [58] leverage knowledge distillation and efficient training strategies for model compression. HCAT [59] proposes hierarchical cross-attention for efficient tracking. Aba-ViTrack [60] introduces re-parameterization and adaptive techniques. Recent works explore efficient attention mechanisms [61] for further acceleration. Despite these efforts, the trade-off between accuracy and efficiency on UAV platforms remains an open challenge [62,63].

2.2. State Space Models

State space models [64,65] have emerged as an efficient alternative for sequence modeling with linear computational complexity. The theoretical foundation of modern SSMs originates from the HiPPO framework [20], which projects continuous signals onto Legendre polynomial bases for long-range memory. S4 [66] introduces structured parameterization that enables practical deep SSM training, while S4D [67] and S5 [68] further simplify the formulation through diagonal approximation and parallel scan algorithms. DSS [69] proposes alternative parameterizations for improved efficiency. LSSL [70] develops linear state space layers for sequence modeling. H3 [71] and Hyena [72] extend the framework with gating mechanisms and implicit convolutions. These methods demonstrate strong performance on long-range sequence benchmarks [73] but rely on fixed polynomial bases that may not adapt to task-specific data distributions.

Selective state space models introduce input-dependent mechanisms for content-aware modeling. Mamba [19] proposes the selective scan algorithm that dynamically adjusts state transitions based on input features, achieving competitive performance with Transformers. Mamba2 [74] refines the architecture with improved stability through structured state space duality. Jamba [75] combines Mamba with attention mechanisms for hybrid architectures. RWKV [76] explores alternative recurrent designs. However, these methods employ hardware-aware algorithms optimized for high-performance GPUs [77], limiting their effectiveness on edge platforms.

Recent works have extended SSMs to visual domains. Vision Mamba [78] and VMamba [79] adapt the architecture for image understanding through bidirectional and cross-scan strategies. LocalMamba [80] and EfficientVMamba [81] improve local feature extraction and computational efficiency. PlainMamba [82] simplifies the architecture without hierarchical designs. For dense prediction tasks, Mamba-UNet [83], VM-UNet [84], and Swin-UMamba [85] apply SSMs to medical image segmentation. U-Mamba [86] and SegMamba [87] further explore encoder-decoder structures. For visual tracking, MambaVision [88] and MambaVOT [89] explore SSM-based temporal modeling for appearance evolution. Mamba-FETrack [90] proposes frame-event fusion with Mamba. MambaVT [91] introduces vision transformers with Mamba for robust tracking. VideoMamba [92] extends Mamba to video understanding tasks. While these approaches achieve linear complexity, they inherit the HiPPO framework with its fixed basis representation, which may cause information loss for tracking features that exhibit complex geometric structures.

In addition to polynomial-based SSMs, geometric approaches have been explored for sequence modeling. Orthogonal and unitary RNNs [93,94] enforce norm-preserving state transitions to mitigate vanishing gradients in long sequences. Hyperspherical representations [23,24] constrain neural network embeddings to unit spheres, demonstrating improved generalization and robustness. Riemannian optimization techniques [95] enable gradient-based learning on manifold-valued parameters. Our approach differs fundamentally from these works by formulating the entire state evolution process as geodesic motion on Riemannian manifolds. Rather than merely constraining parameters or outputs to manifolds, we replace the HiPPO-derived state transition matrix with geometric operations that provide adaptive local coordinates through tangent space projection.

The connection between geometric constraints and sequence modeling in tracking deserves particular attention. Traditional SSMs process features in Euclidean space without considering the intrinsic structure of visual embeddings. However, tracking features naturally exhibit directional characteristics: the angular relationships between template and search features determine matching quality, while feature magnitudes primarily encode confidence or saliency. This observation motivates our reformulation of state evolution on Riemannian manifolds, where the unit sphere provides a natural geometric structure for directional feature dynamics. Unlike polynomial bases that compress all features onto a fixed coordinate system, the tangent space at each spherical state provides an adaptive local coordinate frame that evolves with the feature trajectory, preserving geometric relationships crucial for robust tracking.

3. Preliminary

3.1. State Space Model

State space models [19,66] map an input sequence $x\left(t\right)\in {\mathbb{R}}^L$ to an output sequence $y\left(t\right)\in {\mathbb{R}}^L$ through latent states $h\left(t\right)\in {\mathbb{R}}^N$ using three key matrices: state matrix $\mathbf{A}\in {\mathbb{R}}^{N\times N}$, input projection $\mathbf{B}\in {\mathbb{R}}^{N\times 1}$, and output projection $\mathbf{C}\in {\mathbb{R}}^{1\times N}$.

The HiPPO framework [20] provides the theoretical foundation for constructing $\mathbf{A}$ by projecting the input history onto Legendre polynomial bases. Using zero-order hold discretization with step size $\mathrm{\Delta }$, the continuous dynamics become:

$$\begin{array}{cc}\hfill h_t& =\overline{\mathbf{A}}{h}_{t-1}+\overline{\mathbf{B}}{x}_t\hfill \\ \hfill y_t& =\mathbf{C}{h}_t\hfill \end{array}$$

(1)

where $\overline{\mathbf{A}}=exp(\mathrm{\Delta }\mathbf{A})$ and $\overline{\mathbf{B}}={(\mathrm{\Delta }\mathbf{A})}^{-1}(\overline{\mathbf{A}}-\mathbf{I})·\mathrm{\Delta }\mathbf{B}$.

While this formulation achieves linear complexity, it introduces two limitations for UAV tracking deployment. First, the discretization procedure and hardware-aware algorithms require specialized computational kernels optimized for high-performance GPUs. Second, the fixed Legendre polynomial basis compresses all inputs onto the same coordinate system, which may cause information loss when features exhibit task-specific geometric structures [19].

3.2. Riemannian Geometry

A Riemannian manifold $(\mathcal{M},g)$ is a smooth manifold equipped with a metric g that defines inner products on tangent spaces. For each point $h\in \mathcal{M}$, the tangent space $T_h\mathcal{M}$ forms a local Euclidean approximation of the manifold.

The exponential map ${\mathrm{Exp}}_h:T_h\mathcal{M}\to \mathcal{M}$ moves a point along the geodesic with initial velocity $v\in T_h\mathcal{M}$. For the unit sphere ${\mathcal{S}}^{N-1}=\{h\in {\mathbb{R}}^N:\parallel h\parallel =1\}$, the exponential map has a closed-form expression:

$${\mathrm{Exp}}_h\left(v\right)=cos(\parallel v\parallel )·h+sin(\parallel v\parallel )·{\displaystyle \frac{v}{\parallel v\parallel }}$$

(2)

The projection of an arbitrary vector $u\in {\mathbb{R}}^N$ onto the tangent space $T_h{\mathcal{S}}^{N-1}$ is given by ${\prod }_h\left(u\right)=u-\langle u,h\rangle h$. These operations have been widely adopted in deep learning for optimization on manifolds and hyperspherical representations.

Unlike the fixed Legendre basis in HiPPO, the tangent space $T_h\mathcal{M}$ provides an adaptive local coordinate system that changes with the current state h. This property motivates our approach to reformulate state space models on Riemannian manifolds.

4. Methodology

4.1. Overall Architecture

In this section, we present LF-SSM, a lightweight state space model for UAV tracking that addresses the computational overhead and information loss issues in existing SSM-based approaches.

We propose the Geodesic State Module (GSM) that reformulates state evolution on Riemannian manifolds for efficient token sequence modeling. GSM performs state updates through geodesic motion on the unit sphere, eliminating the need for HiPPO-derived $N\times N$ state transition matrices and complex discretization procedures. The tangent space at each state provides an adaptive local coordinate system, which avoids the information loss caused by fixed basis compression.

The overall architecture is illustrated in Figure 2. The template and search images first undergo patch embedding to generate token sequences. These tokens are concatenated to form a unified spatial sequence and processed through stacked GSM Blocks, which model long-range dependencies among all tokens. Finally, the enhanced features are fed into a prediction head to generate the target bounding box.

4.2. Geodesic State Evolution

Traditional state space models update the hidden state through linear recurrence $h_t=\overline{\mathbf{A}}{h}_{t-1}+\overline{\mathbf{B}}{x}_t$, where $\overline{\mathbf{A}}\in {\mathbb{R}}^{N\times N}$ is derived from the HiPPO framework with a fixed Legendre polynomial basis. We propose to reformulate state evolution on Riemannian manifolds, where the state space is defined on the unit sphere:

$${\mathcal{S}}^{N-1}=\{h\in {\mathbb{R}}^N:\parallel h\parallel =1\}$$

(3)

where N denotes the state dimension and the superscript $N-1$ indicates that the unit sphere is an $(N-1)$-dimensional manifold embedded in ${\mathbb{R}}^N$.

Given the current state $h_{t-1}\in {\mathcal{S}}^{N-1}$ and input token $x_t$, the input token is first projected into the state space to obtain $u_t\in {\mathbb{R}}^N$. The projected input is then mapped onto the tangent space $T_{h_{t-1}}{\mathcal{S}}^{N-1}$ through orthogonal projection:

$$v_t={\prod }_{h_{t-1}}\left(u_t\right)=u_t-\langle u_t,h_{t-1}\rangle h_{t-1}$$

(4)

where ${\prod }_h(·)$ denotes the projection operator onto the tangent space at point h, the angle brackets $\langle ·,·\rangle$ represent the Euclidean inner product, and the resulting tangent vector $v_t$ satisfies the orthogonality condition $\langle v_t,h_{t-1}\rangle =0$. The input-dependent step size ${\mathrm{\Delta }}_t\in {\mathbb{R}}^{+}$ controls the magnitude of state update. The new state is obtained by moving along the geodesic on ${\mathcal{S}}^{N-1}$:

$$h_t=cos({\mathrm{\Delta }}_t\parallel v_t\parallel )·h_{t-1}+sin({\mathrm{\Delta }}_t\parallel v_t\parallel )·{\displaystyle \frac{v_t}{\parallel v_t\parallel +ϵ}}$$

(5)

where $\parallel ·\parallel$ denotes the Euclidean norm, the product ${\mathrm{\Delta }}_t\parallel v_t\parallel$ represents the arc length traveled along the geodesic, and $ϵ={10}^{-6}$ is a small constant for numerical stability when $\parallel v_t\parallel \approx 0$. The output $y_t=\mathbf{C}{h}_t$ is computed through the output projection matrix $\mathbf{C}\in {\mathbb{R}}^{D\times N}$ that maps the N-dimensional state to the D-dimensional output space.

The above geodesic formulation constrains the state vector to evolve on the unit sphere, which maintains $\parallel h_t\parallel =1$ throughout the sequence. This design choice reflects a deliberate factorization of the feature representation into directional and magnitude components based on polar decomposition. In Euclidean space, any non-zero vector $\mathbf{x}\in {\mathbb{R}}^N$ can be uniquely written as $\mathbf{x}=r·\widehat{\mathbf{x}}$, where the scalar $r=\parallel \mathbf{x}\parallel$ captures the magnitude and the unit vector $\widehat{\mathbf{x}}=\mathbf{x}/\parallel \mathbf{x}\parallel$ on the sphere $S^{N-1}$ captures the direction. In visual tracking, these two components encode fundamentally different types of information. The directional component $\widehat{\mathbf{x}}$ represents the geometric structure and relative spatial relationships of target appearance features, while the magnitude component r reflects confidence, salience, or attention weights. The geodesic state evolution operates on the directional component through the spherical state $h_t\in S^{N-1}$, which encodes the geometric structure of appearance features as they evolve over the token sequence.

The spherical constraint provides important stability properties that help address common issues in unconstrained Euclidean formulations. In traditional state space models, the discrete dynamics $h_t=\mathbf{A}{h}_{t-1}+\mathbf{B}{x}_t$ involve a transition matrix $\mathbf{A}\in {\mathbb{R}}^{N\times N}$ whose spectral properties determine long-term behavior. When the largest eigenvalue satisfies $|{\lambda }_{max}\left(\mathbf{A}\right)|>1$, the state magnitude grows exponentially over time, which eventually causes numerical overflow and gradient explosion during backpropagation. Conversely, when $|{\lambda }_{max}\left(\mathbf{A}\right)|<1$, states decay exponentially toward zero, which leads to vanishing gradients and loss of long-range information. The ideal regime $|{\lambda }_{max}\left(\mathbf{A}\right)|=1$ provides stable dynamics but remains structurally fragile. Small perturbations to the matrix entries during training can push eigenvalues away from the unit circle and trigger instability.

Our geodesic formulation significantly mitigates this spectral sensitivity. The norm preservation property $\parallel h_t\parallel =1$ holds exactly by geometric construction, without any constraints on learned parameters. To verify this, observe that the tangent vector $v_t$ produced by projection in Equation (4) satisfies the orthogonality condition $\langle v_t,h_{t-1}\rangle =0$ by definition of the tangent space. Consequently, the geodesic update in Equation (5) yields

$$\begin{array}{cc}\hfill \parallel h_t{\parallel }^2& ={∥cos({\mathrm{\Delta }}_t\parallel v_t\parallel )·h_{t-1}+sin({\mathrm{\Delta }}_t\parallel v_t\parallel )·{\displaystyle \frac{v_t}{\parallel v_t\parallel }}∥}^2\hfill \\ \hfill & ={cos}^2({\mathrm{\Delta }}_t\parallel v_t\parallel )\parallel h_{t-1}{\parallel }^2+{sin}^2({\mathrm{\Delta }}_t\parallel v_t\parallel ){∥{\displaystyle \frac{v_t}{\parallel v_t\parallel }}∥}^2\hfill \\ \hfill & +2cos({\mathrm{\Delta }}_t\parallel v_t\parallel )sin({\mathrm{\Delta }}_t\parallel v_t\parallel )〈h_{t-1},{\displaystyle \frac{v_t}{\parallel v_t\parallel }}〉\hfill \\ \hfill & ={cos}^2({\mathrm{\Delta }}_t\parallel v_t\parallel )+{sin}^2({\mathrm{\Delta }}_t\parallel v_t\parallel )+0=1\hfill \end{array}$$

(6)

where the first line substitutes the geodesic update formula, the second line expands the squared norm using the identity ${\parallel \mathbf{a}+\mathbf{b}\parallel }^2={\parallel \mathbf{a}\parallel }^2+{\parallel \mathbf{b}\parallel }^2+2\langle \mathbf{a},\mathbf{b}\rangle$, and the third line uses $\parallel h_{t-1}\parallel =1$ from the inductive hypothesis, $\parallel v_t/\parallel v_t\parallel \parallel =1$ by normalization, and the cross term vanishes because $\langle h_{t-1},v_t\rangle =0$ from Equation (4). This equality holds to machine precision for any step size ${\mathrm{\Delta }}_t>0$ and any tangent vector $v_t$, which provides robust stability without the need for eigenvalue tuning or spectral normalization.

This formulation eliminates the HiPPO-derived $N\times N$ state transition matrix $\mathbf{A}$ and the associated discretization procedure. The tangent space $T_{h_{t-1}}{\mathcal{S}}^{N-1}$ provides an adaptive local coordinate system that changes with the current state, avoiding information loss from fixed basis compression. The geodesic state evolution is summarized in Algorithm 1.

The prior velocity mechanism provides temporal continuity across state updates and implements geometry-aware forgetting of outdated information. At each time step, the prior velocity $v_t^{\mathrm{prior}}\in T_{h_{t-1}}{S}^{N-1}$ carries directional momentum from previous updates. When the state moves from $h_{t-1}$ to $h_t$, this prior velocity is transported to the tangent space at the new location through the projection

$$v_{t+1}^{\mathrm{prior}}={\mathrm{Proj}}_{h_t}\left(v_t\right)=v_t-\langle v_t,h_t\rangle h_t$$

(7)

where ${\mathrm{Proj}}_{h_t}(·)$ denotes orthogonal projection onto the tangent space $T_{h_t}{S}^{N-1}$, and the inner product $\langle v_t,h_t\rangle$ measures the alignment between the velocity and the new state direction. This operation removes from the old velocity its component along the new state direction while it preserves the orthogonal remainder. When the tracking target undergoes rapid appearance changes, the state trajectory curves sharply and the projection removes a substantial component of the prior velocity. This effectively forgets outdated directional information that is no longer relevant. Conversely, when the target appearance evolves smoothly, the trajectory follows a nearly geodesic path with small curvature, and most of the prior velocity survives projection to maintain temporal consistency.

Algorithm 1 Geodesic state evolution

Require: Input sequence $\{x_1,\dots ,x_T\}$, initial state $h_0\in S^{N-1}$, stability constant $ϵ$, confidence weight $\alpha \in [0,1]$ Ensure: Output sequence $\{y_1,\dots ,y_T\}$    $v_0^{\mathrm{prior}}\leftarrow \mathbf{0}$ {Initialize prior velocity as zero vector}    for $t=1,\dots ,T$ do       $u_t\leftarrow \mathbf{B}{x}_t$ {Project input to state space}       $w_t\leftarrow u_t+\alpha v_t^{\mathrm{prior}}$ {Combine input with prior momentum}       $v_t\leftarrow w_t-\langle w_t,h_{t-1}\rangle h_{t-1}$ {Project to tangent space}       ${\mathrm{\Delta }}_t\leftarrow \sigma \left({\mathbf{W}}_{\mathrm{\Delta }}{x}_t\right)$ {Compute input-dependent step size}       $h_t\leftarrow cos({\mathrm{\Delta }}_t\parallel v_t\parallel )·h_{t-1}+sin({\mathrm{\Delta }}_t\parallel v_t\parallel )·v_t/(\parallel v_t\parallel +ϵ)$ {Geodesic update}       $v_{t+1}^{\mathrm{prior}}\leftarrow v_t-\langle v_t,h_t\rangle h_t$ {Transport prior to new tangent space}       $y_t\leftarrow \mathbf{C}{h}_t$ {Output projection}    end for    return $\{y_1,\dots ,y_T\}$

The forgetting rate can be quantified in terms of the geodesic step size. Let ${\theta }_t={\mathrm{\Delta }}_t\parallel v_t\parallel$ denote the arc length traveled in one update step, which equals the geodesic distance between successive states $h_{t-1}$ and $h_t$ on the unit sphere. The magnitude of the transported prior satisfies the bound $\parallel v_{t+1}^{\mathrm{prior}}\parallel \le \parallel v_t\parallel sin\left({\theta }_t\right)$, which follows from

$$\begin{array}{cc}\hfill \parallel v_{t+1}^{\mathrm{prior}}{\parallel }^2& =\parallel v_t-\langle v_t,h_t\rangle h_t{\parallel }^2={\parallel v_t\parallel }^2-{\langle v_t,h_t\rangle }^2\hfill \end{array}$$

(8)

where we use $\parallel h_t\parallel =1$ and the expansion ${\parallel \mathbf{a}-\langle \mathbf{a},\mathbf{b}\rangle \mathbf{b}\parallel }^2={\parallel \mathbf{a}\parallel }^2-{\langle \mathbf{a},\mathbf{b}\rangle }^2$ for unit vector $\mathbf{b}$. For small steps where ${\theta }_t\ll 1$, the Taylor expansion $sin\left({\theta }_t\right)\approx {\theta }_t$ gives approximately $\parallel v_{t+1}^{\mathrm{prior}}\parallel \approx \parallel v_t\parallel {\theta }_t$, which indicates that the prior magnitude decays in proportion to the step size. Over multiple steps, this produces gradual attenuation of historical momentum and implements smooth forgetting that avoids the abrupt information loss associated with aggressive eigenvalue decay in traditional SSM formulations.

4.3. Geodesic State Module

Based on the geodesic state evolution theory, we design the Geodesic State Module (GSM) that serves as the core component of LF-SSM. The GSM replaces the selective scan mechanism in Mamba with geodesic dynamics on the unit sphere, where state updates are performed through tangent space projection and exponential mapping using only standard vector operations.

The GSM processes an input feature sequence $\mathbf{X}\in {\mathbb{R}}^{T\times D}$ where T is the number of tokens and D denotes the feature dimension. As shown in Figure 3, the module first applies a linear layer to expand the input dimension to E, followed by a 1D convolution for local feature extraction. The expanded features $\tilde{\mathbf{X}}\in {\mathbb{R}}^{T\times E}$ are then processed through two parallel branches.

The state evolution branch computes geodesic updates on the unit sphere following the formulation in Section 4.2. For each token ${\tilde{x}}_t\in {\mathbb{R}}^E$, we generate input-dependent projection parameters to enable content-aware state updates. The projected input $u_t$, step size ${\mathrm{\Delta }}_t$, and initial state $h_0$ are computed as

$$u_t={\mathrm{Linear}}_B\left({\tilde{x}}_t\right),{\mathrm{\Delta }}_t=\sigma \left({\mathrm{Linear}}_{\mathrm{\Delta }}\left({\tilde{x}}_t\right)\right),h_0={\displaystyle \frac{{\widehat{h}}_0}{\parallel {\widehat{h}}_0\parallel +ϵ}}$$

(9)

where ${\mathrm{Linear}}_B:{\mathbb{R}}^E\to {\mathbb{R}}^N$ projects the E-dimensional expanded feature into the N-dimensional state space, ${\mathrm{Linear}}_{\mathrm{\Delta }}:{\mathbb{R}}^E\to {\mathbb{R}}^1$ computes a scalar step size, $\sigma (·)$ denotes the softplus activation that ensures positivity, ${\widehat{h}}_0\in {\mathbb{R}}^N$ is a learnable parameter, and the division by $\parallel {\widehat{h}}_0\parallel +ϵ$ normalizes it onto the unit sphere. The geodesic state evolution then proceeds sequentially according to Algorithm 1, which involves only vector operations including inner product, scalar multiplication, and trigonometric functions.

The gating branch implements the magnitude component of the factorized representation introduced in Section 4.2. While the state evolution branch processes directional information through the spherical state $h_t\in S^{N-1}$, the gating branch handles magnitude-related attributes through a separate pathway. This branch operates on the expanded features $\tilde{\mathbf{X}}$ which retain complete magnitude information from the input tokens. The gating signal is computed as

$$\mathbf{G}=\mathrm{SiLU}\left({\mathrm{Linear}}_G\left(\tilde{\mathbf{X}}\right)\right)$$

(10)

where ${\mathrm{Linear}}_G:{\mathbb{R}}^E\to {\mathbb{R}}^E$ denotes a learnable linear transformation applied to each token independently, and $\mathrm{SiLU}(·)$ is the Sigmoid Linear Unit activation defined as $\mathrm{SiLU}\left(x\right)=x·\sigma \left(x\right)$ with $\sigma (·)$ being the sigmoid function. This activation allows both amplification when $x>0$ and suppression when $x<0$ of feature magnitudes depending on the learned weights.

The final output recombines the directional and magnitude components through element-wise multiplication

$$\mathbf{Y}={\mathrm{Linear}}_O(\mathbf{G}\odot \left(\mathbf{C}\mathbf{H}\right))$$

(11)

where $\mathbf{H}={[h_1,h_2,\dots ,h_T]}^{\top }\in {\mathbb{R}}^{T\times N}$ collects the spherical states across all T time steps with each row $h_t\in S^{N-1}$ being an N-dimensional unit vector, the matrix $\mathbf{C}\in {\mathbb{R}}^{E\times N}$ projects each N-dimensional directional state to the E-dimensional space, the symbol ⊙ denotes element-wise multiplication that modulates the directional features $\mathbf{C}\mathbf{H}$ by the learned magnitude weights $\mathbf{G}$, and ${\mathrm{Linear}}_O:{\mathbb{R}}^E\to {\mathbb{R}}^D$ produces the final D-dimensional output. This design follows the polar decomposition principle where directional and magnitude information are processed separately and recombined in the output. Since the gating branch can learn arbitrary scaling functions through the parameters in ${\mathrm{Linear}}_G$, this factorized architecture achieves equivalent representational capacity to unconstrained state-output mappings while it preserves the stability benefits of bounded state dynamics on the unit sphere.

4.4. GSM Block

The GSM Block integrates the Geodesic State Module, normalization, and residual connections to form a complete building block for the tracking network. The structure is illustrated in Figure 2.

Given input tokens $\mathbf{Z}\in {\mathbb{R}}^{T\times D}$, the GSM Block first applies layer normalization, then processes features through bidirectional GSM and combines the result with the input via residual connection:

$${\mathbf{Z}}^{\prime }=\mathrm{LayerNorm}\left(\mathbf{Z}\right),{\mathbf{Z}}_{out}=\mathbf{Z}+\mathrm{Linear}\left([{\mathrm{GSM}}_{fwd}\left({\mathbf{Z}}^{\prime }\right);{\mathrm{GSM}}_{bwd}\left({\mathbf{Z}}^{\prime }\right)]\right)$$

(12)

where $[·;·]$ denotes concatenation along the feature dimension. The bidirectional design applies GSM in both forward and backward directions along the token sequence, enabling each token to aggregate information from both preceding and subsequent tokens. This facilitates comprehensive feature interaction between template and search regions within the unified spatial sequence.

Multiple GSM Blocks are stacked to form the backbone of LF-SSM. The number of blocks L is a hyperparameter that balances model capacity and computational efficiency.

4.5. Training and Inference

4.5.1. Training

During training, the template and search images are processed through patch embedding to generate token sequences. The template tokens ${\mathbf{Z}}_t\in {\mathbb{R}}^{T_t\times D}$ and search tokens ${\mathbf{Z}}_s\in {\mathbb{R}}^{T_s\times D}$ are concatenated to form a unified spatial sequence $\mathbf{Z}=[{\mathbf{Z}}_t;{\mathbf{Z}}_s]$ and fed into the stacked GSM Blocks.

The enhanced search tokens are extracted from the output and fed into the prediction head, which consists of a classification branch for foreground-background prediction and a regression branch for bounding box estimation. The training loss combines classification loss and regression loss:

$$\mathcal{L}={\mathcal{L}}_{cls}+{\lambda }_{iou}{\mathcal{L}}_{iou}+{\lambda }_{L1}{\mathcal{L}}_{L1}$$

(13)

where ${\mathcal{L}}_{cls}$ is the focal loss for classification, ${\mathcal{L}}_{iou}$ is the GIoU loss for bounding box regression, and ${\mathcal{L}}_{L1}$ is the L1 loss for coordinate regression. The hyperparameters ${\lambda }_{iou}$ and ${\lambda }_{L1}$ balance the loss terms. The complete training procedure is summarized in Algorithm 2.

4.5.2. Inference

During inference, the template is encoded once and cached for subsequent frames. For each search frame, the search tokens are concatenated with the cached template tokens and processed through GSM Blocks. The prediction head outputs the target bounding box directly without post-processing such as window penalty or cosine window, which further improves inference efficiency. The complete inference process is summarized in Algorithm 3.

Algorithm 2 LF-SSM Training

Require: Training set $\mathcal{D}$, learning rate $\eta$, epochs E, loss weights ${\lambda }_{iou}$, ${\lambda }_{L1}$ Ensure: Trained model parameters $\mathrm{\Theta }$    Initialize model parameters $\mathrm{\Theta }$    for $e=1,\dots ,E$ do          for each batch $(I_t,I_s,B_{gt})$ in $\mathcal{D}$ do                ${\mathbf{Z}}_t\leftarrow \mathrm{PatchEmbed}\left(I_t\right)$, ${\mathbf{Z}}_s\leftarrow \mathrm{PatchEmbed}\left(I_s\right)$                ${\mathbf{Z}}_{out}\leftarrow \mathrm{GSM}-\mathrm{Blocks}\left([{\mathbf{Z}}_t;{\mathbf{Z}}_s]\right)$                $B_{pred},C_{pred}\leftarrow \mathrm{PredHead}\left({\mathbf{Z}}_{out}\right)$                $\mathcal{L}\leftarrow {\mathcal{L}}_{cls}+{\lambda }_{iou}{\mathcal{L}}_{iou}+{\lambda }_{L1}{\mathcal{L}}_{L1}$                $\mathrm{\Theta }\leftarrow \mathrm{\Theta }-\eta {\nabla }_{\mathrm{\Theta }}\mathcal{L}$       end for    end for    return $\mathrm{\Theta }$

Algorithm 3 LF-SSM Inference

Require: Template image $I_t$, search images $\{I_s^1,\dots ,I_s^K\}$ Ensure: Bounding boxes $\{B^1,\dots ,B^K\}$    ${\mathbf{Z}}_t\leftarrow \mathrm{PatchEmbed}\left(I_t\right)$ {Encode once and cache}    for $k=1,\dots ,K$ do          ${\mathbf{Z}}_s\leftarrow \mathrm{PatchEmbed}\left(I_s^k\right)$          ${\mathbf{Z}}_{out}\leftarrow \mathrm{GSM}-\mathrm{Blocks}\left([{\mathbf{Z}}_t;{\mathbf{Z}}_s]\right)$          $B^k\leftarrow \mathrm{PredHead}\left({\mathbf{Z}}_{out}\right)$    end for    return $\{B^1,\dots ,B^K\}$

5. Efficiency Analysis

In this section, we analyze the computational efficiency of LF-SSM compared to existing SSM-based methods.

5.1. Time Complexity

For a sequence of T tokens with expanded dimension E and state dimension N, we analyze the time complexity of geodesic state evolution. At each token position t, the computational cost consists of input projection $u_t={\mathrm{Linear}}_B\left({\tilde{x}}_t\right)$ with $O\left(EN\right)$, tangent space projection with $O\left(N\right)$, exponential map computation with $O\left(N\right)$, and output projection with $O\left(EN\right)$. The total time complexity per GSM is:

$${\mathcal{T}}_{GSM}=O(TEN+TN)=O\left(TEN\right)$$

(14)

This achieves linear complexity with respect to sequence length T, consistent with modern SSM methods. In contrast, Transformer-based methods require $O\left(T^{2}D\right)$ for self-attention computation.

5.2. Space Complexity

The space complexity of LF-SSM consists of linear projection parameters with $O\left(EN\right)$ and state vector $h_t\in {\mathcal{S}}^{N-1}$ with $O\left(N\right)$. The total space complexity is:

$${\mathcal{S}}_{total}=O\left(EN\right)$$

(15)

This is comparable to modern SSM methods such as S4D and Mamba that use diagonal state matrices with $O\left(N\right)$ complexity. Compared to Transformer-based methods, we avoid the $O\left(T^2\right)$ memory requirement for attention matrices.

5.3. Computational Operations

A key efficiency advantage of LF-SSM is the elimination of HiPPO-related computational overhead. Although modern SSMs such as S4D and Mamba have simplified the state transition matrix to diagonal form, they still require computing the discretized matrix $\overline{\mathbf{A}}=exp(\mathrm{\Delta }\mathbf{A})$ and the associated input transformation $\overline{\mathbf{B}}$. Mamba further requires selective scan kernels optimized for GPU memory hierarchies, which are difficult to deploy efficiently on edge platforms due to their hardware-specific memory access patterns.

In contrast, our geodesic state evolution bypasses the discretization procedure entirely:

$$h_t=cos({\mathrm{\Delta }}_t\parallel v_t\parallel )·h_{t-1}+sin({\mathrm{\Delta }}_t\parallel v_t\parallel )·{\displaystyle \frac{v_t}{\parallel v_t\parallel +ϵ}}$$

(16)

This formulation involves only inner product $\langle u_t,h_{t-1}\rangle$ with $O\left(N\right)$, vector subtraction and norm computation with $O\left(N\right)$, scalar trigonometric functions with $O\left(1\right)$, and scalar-vector multiplication with $O\left(N\right)$. These are standard operations efficiently supported across diverse hardware platforms including edge devices, without requiring specialized computational kernels or hardware-specific optimizations. The elimination of discretization procedures and selective scan kernels enables efficient deployment on resource-constrained UAV platforms.

While the geodesic formulation and diagonal SSMs both achieve $O\left(TEN\right)$ asymptotic complexity, the constant factors differ due to specific operations involved. The geodesic update requires approximately twice the arithmetic operations compared to diagonal SSMs when the expanded dimension E is comparable to the input dimension D, primarily due to the additional tangent space projection and exponential map computations. However, this arithmetic overhead does not directly translate to proportional wall-clock time increase due to two factors. First, the trigonometric functions $cos(·)$ and $sin(·)$ are computed efficiently through hardware-accelerated Special Function Units (SFUs) on modern GPUs, executing in approximately 4–8 clock cycles comparable to floating-point multiplications. Second, the geodesic formulation exhibits favorable memory access patterns where all operations are element-wise or involve matrix-vector products that can be efficiently parallelized, whereas the selective scan in Mamba requires sequential dependencies that limit parallelization and necessitate sophisticated kernel implementations.

The practical efficiency boundary depends on the availability of optimized computational kernels for specific hardware platforms. On edge platforms where highly tuned selective scan implementations are unavailable, the geodesic formulation achieves higher throughput despite the additional arithmetic operations, because it relies entirely on standard operations that are well-supported across all hardware architectures. On high-end GPUs with optimized selective scan kernels, the efficiency advantage of geodesic formulation diminishes for larger state dimensions where the constant factor overhead becomes more pronounced. For moderate state dimensions $N\le 128$ that are typical in visual tracking applications, the geodesic formulation maintains competitive or superior practical efficiency while providing the geometric benefits of adaptive tangent space representation.

6. Experiments and Analysis

In this section, we conduct extensive experiments to assess the performance, efficiency, and robustness of LF-SSM compared to SOTA methods. We first describe the experimental setup in Section 6.1, then present quantitative comparisons with state-of-the-art trackers in Section 6.2. Qualitative evaluation and ablation studies are provided in Section 6.3 and Section 6.4, respectively.

6.1. Experimental Setup

6.1.1. Datasets

We conduct experiments on four widely-used tracking benchmarks, including UAV123 [7], VisDrone [96], ARDMAV [97], and LaSOT [52]. These datasets serve dual purposes: their training splits are used for model training, while their test splits are used for evaluation. UAV123 is a large-scale aerial tracking dataset consisting of 123 video sequences with over 112K frames, covering diverse scenarios including urban areas, roads, and natural environments. VisDrone comprises 96 sequences captured by various drone platforms under different weather and lighting conditions, presenting challenges in scale variation and occlusion. ARDMAV contains 60 sequences focusing on aggressive drone racing and maneuvering scenarios with rapid motion and viewpoint changes. LaSOT is a large-scale long-term tracking benchmark with 1120 training sequences and 280 test sequences averaging over 2500 frames per video, providing comprehensive evaluation of long-term tracking capability. This training configuration ensures that the model learns from UAV-specific visual characteristics including aerial viewpoints, small target appearances, and platform-induced motion patterns.

6.1.2. Evaluation Metrics

We adopt standard metrics following the tracking evaluation protocols. For UAV123, VisDrone, and ARDMAV, we report precision (P) and success rate (AUC). Precision measures the percentage of frames where the center location error is within 20 pixels. Success rate evaluates the overlap ratio using intersection over union (IoU), and the area under curve (AUC) of the success plot is reported. For LaSOT, we report AUC, normalized precision (${\mathrm{P}}_{\mathrm{Norm}}$), and precision (P). Additionally, model parameter volume (M), computational complexity (GFLOPs), and frames per second (FPS) are used to evaluate efficiency.

6.1.3. Implementation Details

Experiments are conducted on an Intel i9-13900K CPU, NVIDIA GeForce RTX 4090 GPU (24 GB), and 64 GB memory. LF-SSM is implemented in PyTorch 2.3 with CUDA 12.1.

Images are preprocessed by cropping search regions of $256\times 256$ pixels centered on the target, with template regions of $128\times 128$ pixels. For each training iteration, we sample template-search pairs from a temporal window of $\pm 100$ frames centered on the template frame with uniform random sampling. We use AdamW optimizer with initial learning rate $1\times {10}^{-4}$ and weight decay $1\times {10}^{-4}$. The model is trained for 300 epochs with batch size 32. The state dimension N is set to 64, and the numerical stability constant $ϵ$ is set to ${10}^{-6}$ for tangent space projection. During inference, we employ a static template strategy where the template is extracted from the first frame and cached throughout the sequence without online updating.

For edge platform deployment, we evaluate on NVIDIA Jetson Orin Nano (NVIDIA Corporation, Santa Clara, CA, USA) (8 GB RAM, 1024-core Ampere GPU, 15 W TDP), which represents typical computational constraints in real-world UAV systems where power consumption and weight limitations necessitate lightweight yet accurate tracking solutions. Models are converted using TensorRT 8.5 with FP16 precision. For fair comparison, all baseline methods are evaluated using the same deployment pipeline with TensorRT FP16 optimization on both NVIDIA GeForce RTX 4090 (NVIDIA Corporation, Santa Clara, CA, USA) and Jetson Orin Nano platforms. We implement three model variants with different numbers of GSM Blocks: LF-SSM-S with $L=6$ blocks, LF-SSM-M with $L=12$ blocks, and LF-SSM-L with $L=18$ blocks. This configuration enables systematic comparison while maintaining architectural consistency across different model scales.

6.1.4. Baselines

We conduct extensive comparative experiments with 13 state-of-the-art tracking methods across two categories. Transformer-based methods include SeqTrack [13], VideoTrack [48], ARTrackV2 [98], ODTrack [47], AQATrack [99], EVPTrack [49], MixFormer [40], SPMTrack [50], and HIPTrack [51]. SSM-based methods include Mamba-FETrack [90], MambaVT [91], MambaVision [88], and VideoMamba [92], which serve as the primary baselines for evaluating our HiPPO-free design.

6.2. Comparison with SOTA Methods

We conduct comprehensive comparisons with state-of-the-art methods across four tracking benchmarks. The results are presented in

Table 1. Performance comparison on four tracking benchmarks. AUC, P, and ${\mathrm{P}}_{\mathrm{Norm}}$ denote success rate, precision, and normalized precision, respectively. The best and second-best results are in bold and underline.

Method	UAV123		VisDrone		ARDMAV		LaSOT
	AUC	P	AUC	P	AUC	P	AUC	${\mathrm{P}}_{\mathrm{Norm}}$	P
SeqTrack [13]	68.5	88.2	64.2	83.5	61.5	79.8	72.5	81.5	79.3
VideoTrack [48]	67.2	86.8	62.8	81.5	59.8	77.5	70.2	81.9	76.4
ARTrackV2 [98]	69.8	89.5	65.8	85.2	63.5	81.8	73.6	82.8	81.1
ODTrack [47]	70.2	89.8	66.2	85.5	64.2	82.5	74.0	84.2	82.3
AQATrack [99]	68.2	87.5	63.5	82.8	60.8	78.5	72.7	82.9	80.2
EVPTrack [49]	68.8	88.2	64.5	83.8	61.8	79.8	72.7	82.9	80.3
MixFormer [40]	69.5	88.9	65.5	84.5	63.2	81.2	72.4	82.2	80.1
SPMTrack [50]	72.8	91.8	68.5	87.8	66.5	84.8	77.4	86.6	85.0
HIPTrack [51]	68.5	87.8	64.2	83.2	61.2	79.2	72.7	82.9	79.5
Mamba-FETrack [90]	68.2	88.5	64.5	83.2	61.8	80.2	71.5	80.8	77.5
MambaVT [91]	67.8	87.5	63.8	82.5	60.5	78.8	70.8	80.2	76.8
MambaVision [88]	68.5	88.2	64.2	83.5	61.5	79.5	71.2	80.5	77.2
VideoMamba [92]	68.8	88.5	64.8	83.8	62.2	80.5	71.8	81.2	78.2
LF-SSM-S	69.5	89.2	65.2	84.5	62.8	81.2	72.8	82.5	79.8
LF-SSM-M	71.2	90.8	67.5	86.8	65.5	84.2	74.5	84.2	82.5
LF-SSM-L	73.2	92.5	69.2	88.5	67.8	86.2	75.8	85.5	84.2

and

Table 2. Efficiency comparison. FPS is measured on RTX 4090 and Jetson Orin Nano platforms. Parameter-matched SSM variants (-S) are configured as follows: Mamba-FETrack-S uses 8 blocks with hidden dimension 256; MambaVT-S uses 6 blocks with hidden dimension 288; VideoMamba-S uses 8 blocks with hidden dimension 256; S4D-S uses 6 blocks with hidden dimension 256. The best and second-best results are in bold and underline.

Method	Params (M)	GFLOPs	FPS
			4090	Orin Nano
SeqTrack [13]	89	148	35	12
VideoTrack [48]	96	165	32	10
ARTrackV2 [98]	93	52	180	45
ODTrack [47]	92	73	95	28
AQATrack [99]	97	86	85	24
EVPTrack [49]	94	72	90	26
MixFormer [40]	196	245	25	8
SPMTrack [50]	105	78	155	38
HIPTrack [51]	98	92	105	28
Mamba-FETrack [90]	78	62	120	32
MambaVT [91]	82	58	135	35
MambaVision [88]	75	55	145	38
VideoMamba [92]	85	65	125	33
Mamba-FETrack-S	18.8	14.2	185	48
MambaVT-S	19.2	15.5	172	45
VideoMamba-S	18.5	13.8	168	42
S4D-S	18.6	13.5	205	55
LF-SSM-S	18.5	12.8	320	69
LF-SSM-M	35.2	24.5	265	52
LF-SSM-L	52.8	38.2	210	41

.

On UAV123, while SPMTrack [50] achieves AUC of 72.8%, our LF-SSM-L obtains the highest AUC of 73.2% and precision of 92.5%, surpassing SPMTrack by 0.4% in AUC. Compared to SSM-based methods, LF-SSM-L outperforms VideoMamba [92] by 4.4% and Mamba-FETrack [90] by 5.0% in AUC. For VisDrone, LF-SSM-L achieves AUC of 69.2% and precision of 88.5%, outperforming SPMTrack with AUC of 68.5% and all SSM-based methods by significant margins. On ARDMAV which features aggressive drone motion and rapid viewpoint changes, our method leads with AUC of 67.8% and precision of 86.2%, exceeding SPMTrack by 1.3% in AUC.

For long-term tracking on LaSOT, LF-SSM-L achieves competitive results with AUC of 75.8%, ${\mathrm{P}}_{\mathrm{Norm}}$ of 85.5%, and P of 84.2%. While SPMTrack achieves the highest AUC of 77.4%, our method outperforms all SSM-based baselines including VideoMamba by 4.0% and Mamba-FETrack by 4.3% in AUC, demonstrating the effectiveness of our HiPPO-free geodesic state evolution.

The consistent performance gains across all four benchmarks demonstrate LF-SSM’s capability in modeling long-range temporal dependencies while preserving fine-grained spatial details. Notably, our method achieves superior precision scores on UAV-specific datasets, indicating excellent boundary localization ability crucial for aerial tracking scenarios. The improvement is more pronounced on ARDMAV which features rapid motion variations, validating that geodesic state evolution effectively handles appearance changes through adaptive tangent space representation.

Beyond accuracy improvements, LF-SSM exhibits remarkable computational efficiency. As shown in Table 2, our LF-SSM-S achieves 69 FPS on Jetson Orin Nano with only 18.5 M parameters and 12.8 GFLOPs, significantly outperforming all compared methods. Compared to Mamba-FETrack which operates at 32 FPS with 78 M parameters, LF-SSM-S achieves 2.2× speedup with 76.3% parameter reduction. To ensure fair comparison, we also evaluate parameter-matched SSM variants scaled to approximately 18–19 M parameters. Even at matched parameter counts, LF-SSM-S maintains substantial advantages: it achieves 69 FPS compared to 55 FPS for S4D-S and 42–48 FPS for Mamba-based variants, representing 1.25–1.64× speedup. This efficiency gain stems from the elimination of discretization overhead and selective scan dependencies in our geodesic formulation, rather than merely reduced model capacity. While Transformer-based methods like MixFormer and SeqTrack operate at merely 8–12 FPS on Jetson Orin Nano due to quadratic attention complexity, our HiPPO-free design enables efficient edge deployment. LF-SSM effectively bridges this efficiency-accuracy gap, achieving state-of-the-art tracking performance while maintaining real-time capability on edge platforms.

6.3. Qualitative Evaluation

Figure 4 presents visual comparisons between LF-SSM and five representative SOTA methods on challenging cases from four datasets. The examples demonstrate our method’s superior performance in diverse UAV tracking scenarios, including small target tracking where targets occupy less than 20 pixels against cluttered backgrounds in rows 1, 5, and 7. In these cases, LF-SSM excels at maintaining precise bounding box alignment, while other methods often produce drifted or incorrectly scaled predictions.

As shown in rows 3 and 4, LF-SSM effectively handles dense urban traffic scenes with multiple similar distractors. Additionally, rows 5 and 6 demonstrate our method’s capability in MAV tracking scenarios from varying environmental contexts, from hillside vegetation in row 5 to urban infrastructure with roads and buildings in row 6. These examples highlight our model’s capability to maintain robust localization under extreme scale variations, a strength attributed to our geodesic state evolution with manifold-based updates. This advantage becomes particularly evident when compared to other SSM-based methods like VideoMamba and Mamba-FETrack, which struggle with target drift in similarly challenging environments.

Row 8 presents nighttime tracking with LED-illuminated drone against dark background. The results demonstrate that conventional SSM-based methods exhibit significant limitations in low-light conditions due to HiPPO’s fixed polynomial basis compressing appearance information. In contrast, our proposed method successfully maintains target identity through geodesic state transitions that preserve discriminative features across illumination changes.

6.4. Ablation Study

To evaluate the effectiveness of our proposed components, we conduct comprehensive experiments on different model configurations. The results are presented in

Table 3. Ablation study of GSM components on four benchmarks. Bold values indicate the best results.

No.	Component			UAV123		VisDrone		ARDMAV		LaSOT
	TSP	ISS	Gate	AUC	P	AUC	P	AUC	P	AUC	${\mathrm{P}}_{\mathrm{Norm}}$	P
#1				68.5	87.8	64.5	83.2	62.2	80.5	71.2	80.5	77.2
#2	✓			70.2	89.2	65.8	84.5	63.8	82.2	72.5	82.2	79.5
#3		✓		69.5	88.5	65.2	84.2	63.2	81.5	72.2	81.5	78.8
#4			✓	69.8	88.8	65.5	84.5	63.5	81.8	72.5	81.8	79.2
#5	✓	✓		71.5	90.5	67.2	86.2	65.5	84.2	74.2	83.8	81.8
#6	✓		✓	71.2	90.2	66.8	85.8	65.2	83.8	73.8	83.5	81.2
#7		✓	✓	70.8	89.8	66.5	85.5	64.8	83.5	73.5	83.2	80.8
#Ours	✓	✓	✓	73.2	92.5	69.2	88.5	67.8	86.2	75.8	85.5	84.2

and

Table 4. Comparison with different SSM variants on UAV123. FPS is measured on Jetson Orin Nano platform. Bold values indicate the best results.

SSM Variant	AUC	P	Params (M)	FPS
S4 [66]	67.5	87.2	45.2	25
S4D [67]	68.2	87.8	42.8	32
S5 [68]	68.8	88.2	48.5	22
Mamba [19]	69.5	89.2	52.8	38
Euclidean + Norm	70.2	89.8	52.8	68
Linear-RNN (Orthogonal)	68.5	87.8	52.5	32
GRU	70.2	89.5	53.2	28
LF-SSM (Ours)	73.2	92.5	52.8	69

.

6.4.1. Component Analysis

We investigate the contribution of each component in the Geodesic State Module. TSP, ISS, and Gate denote Tangent Space Projection, Input-dependent Step Size, and Gating Mechanism, respectively.

As shown in Table 3, #1 serves as the baseline with only geodesic state evolution on the unit sphere, achieving AUC of 68.5% on UAV123. Among individual components, Tangent Space Projection (#2) contributes the most significant improvement of 1.7% AUC, validating that adaptive local coordinate systems effectively preserve the geometric structure of tracking features. Input-dependent Step Size (#3) brings 1.0% gain by enabling content-aware state transitions, while Gating Mechanism (#4) contributes 1.3% through selective information flow. Combining TSP and ISS (#5) achieves 71.5% AUC, outperforming other two-component combinations (#6 and #7). The full model (#Ours) with all components obtains the best performance of 73.2% AUC on UAV123, demonstrating that these components are complementary and jointly contribute to the final performance. Consistent improvements are observed across all four benchmarks.

6.4.2. Comparison with SSM Methods

We compare our geodesic state evolution with different SSM formulations to validate the advantage of eliminating the HiPPO framework. All variants are implemented with the same backbone architecture for fair comparison.

As shown in Table 4, S4 with HiPPO-LegS initialization achieves AUC of 67.5% with 25 FPS on Jetson Orin Nano, while diagonal variants S4D and S5 obtain slightly better results through simplified parameterization. Mamba with selective scan mechanism achieves 69.5% AUC at 38 FPS, but its hardware-specific kernels designed for high-performance GPUs cannot be fully utilized on edge platforms. To isolate the contribution of geodesic formulation, we also implement a Euclidean baseline that replaces geodesic update with $h_t=(h_{t-1}+{\mathrm{\Delta }}_t{u}_t)/\parallel h_{t-1}+{\mathrm{\Delta }}_t{u}_t\parallel$ using the same gating and convolution layers. This baseline achieves 70.2% AUC, which is 3.0% lower than our full model, demonstrating that the geodesic formulation with trigonometric functions provides superior state evolution compared to simple normalization. This performance gap arises because the Euclidean baseline operates in flat space before projecting back to the sphere, which introduces systematic distance distortions that accumulate over long sequences. In contrast, our geodesic update through the exponential map ${\mathrm{Exp}}_{h_t}(\mathrm{\Delta }{v}_t)=cos(\mathrm{\Delta }\parallel v_t\parallel )h_t+sin(\mathrm{\Delta }\parallel v_t\parallel )v_t/\parallel v_t\parallel$ respects the intrinsic spherical geometry and preserves geodesic distances exactly. Moreover, the tangent space $T_{h_t}{S}^{N-1}$ provides an adaptive local coordinate system that rotates with the evolving state, allowing new inputs to be represented in coordinates aligned with the current geometric context rather than a fixed polynomial basis. The prior velocity mechanism further enables geometry-aware temporal modeling, where the projection $v_{t+1}^{\mathrm{prior}}=v_t-\langle v_t,h_{t+1}\rangle h_{t+1}$ naturally discards outdated information when the trajectory curves sharply during rapid appearance changes, while preserving consistency during smooth evolution.

To further validate the benefits of the Riemannian formulation against traditional sequence modeling approaches, we include two additional baselines with matched parameter counts. Linear-RNN employs the recurrence $h_t=\mathbf{A}{h}_{t-1}+\mathbf{B}{x}_t$ with orthogonal state transition matrix $\mathbf{A}$ initialized via Cayley parameterization $\mathbf{A}=(\mathbf{I}-\mathbf{S}){(\mathbf{I}+\mathbf{S})}^{-1}$ where $\mathbf{S}$ is skew-symmetric, ensuring $|{\lambda }_i\left(\mathbf{A}\right)|=1$ for gradient stability. GRU implements standard gating with update gate $z_t$, reset gate $r_t$, and candidate state ${\tilde{h}}_t$. LF-SSM outperforms Linear-RNN by 4.7% AUC and GRU by 3.0% AUC. The improvement over GRU is noteworthy since it demonstrates that the geometric inductive bias of spherical state evolution is more effective than learned gating mechanisms for visual tracking. Additionally, LF-SSM achieves 69 FPS compared to 32 FPS for Linear-RNN and 28 FPS for GRU, as the geodesic formulation avoids the $N\times N$ matrix multiplications required by these recurrent architectures.

Our LF-SSM achieves the highest AUC of 73.2% with 69 FPS on Jetson Orin Nano, outperforming Mamba by 3.7% in accuracy and 1.8× in speed. This performance gap arises because the Euclidean baseline operates in flat space before projecting back to the sphere, which introduces systematic distance distortions that accumulate over long sequences. In contrast, our geodesic update through the exponential map ${\mathrm{Exp}}_{h_t}(\mathrm{\Delta }{v}_t)=cos(\mathrm{\Delta }\parallel v_t\parallel )h_t+sin(\mathrm{\Delta }\parallel v_t\parallel )v_t/\parallel v_t\parallel$ respects the intrinsic spherical geometry and preserves geodesic distances exactly. Moreover, the tangent space $T_{h_t}{S}^{N-1}$ provides an adaptive local coordinate system that rotates with the evolving state, allowing new inputs to be represented in coordinates aligned with the current geometric context rather than a fixed polynomial basis. The prior velocity mechanism further enables geometry-aware temporal modeling, where the projection $v_{t+1}^{\mathrm{prior}}=v_t-\langle v_t,h_{t+1}\rangle h_{t+1}$ naturally discards outdated information when the trajectory curves sharply during rapid appearance changes, while preserving consistency during smooth evolution. Our LF-SSM achieves the highest AUC of 73.2% with 69 FPS on Jetson Orin Nano, outperforming Mamba by 3.7% in accuracy and 1.8× in speed.

To rigorously isolate the contribution of geodesic state evolution from other architectural components such as gating, convolution, and bidirectional processing, we conducted controlled experiments where all components remain identical except for the state update mechanism.

Table 5. Controlled ablation isolating geodesic state evolution contribution. All variants share identical gating, convolution, bidirectional processing, and residual connections. Bold values indicate the best results.

State Update Mechanism	AUC	P	$\mathbf{\Delta }$AUC
Euclidean Linear ($h_t=\mathbf{A}{h}_{t-1}+\mathbf{B}{u}_t$)	68.5	87.8	–
Diagonal SSM (S4D-style)	69.2	88.5	+0.7
Euclidean + Normalization	70.2	89.8	+1.7
Geodesic (Ours)	73.2	92.5	+4.7

presents this controlled comparison on UAV123.

The controlled experiments reveal that geodesic state evolution provides 4.7% AUC improvement over Euclidean Linear baseline, 4.0% over Diagonal SSM, and 3.0% over Euclidean-plus-Normalization. Since all architectural components are held constant, these improvements are directly attributable to the Riemannian state update mechanism. We further verified that the other components provide comparable benefits to both Euclidean and Riemannian formulations: the gating mechanism improves Diagonal SSM by 1.4% and Geodesic by 1.2%; the convolution layer improves both by approximately 1.7%; the bidirectional processing improves Diagonal SSM by 2.1% and Geodesic by 2.3%. These similar magnitudes confirm that the architectural components provide additive benefits independent of the state update mechanism, and the 3.0% gap between Euclidean-plus-Normalization and Geodesic formulation is attributable to the intrinsic geometric advantages of manifold-based state evolution.

To further analyze the source of efficiency differences, we provide a detailed per-operation latency breakdown in

Table 6. Per-layer latency breakdown on Jetson Orin Nano. State dimension $N=64$, sequence length $T=320$ tokens. Bold values indicate the best results.

Operation	GSM (ms)	Mamba (ms)	S4D (ms)
Input Projection $\mathbf{B}{\tilde{x}}_t$	0.35	0.38	0.36
Tangent Proj./Discretization	0.08	0.42	0.35
State Update/Selective Scan	0.22	1.38	0.55
Output Projection $\mathbf{C}{h}_t$	0.53	0.58	0.62
Trigonometric/Exponential Ops	0.02	0.04	0.02
Total	1.20	2.80	1.90

. The measurements reveal that trigonometric operations contribute only 1.7% of total GSM computation time, while the selective scan in Mamba consumes 49% of its total time due to sequential dependencies. This confirms that the efficiency advantage stems from avoiding sequential bottlenecks rather than despite the transcendental operations.

6.4.3. Quantitative Feature Analysis

To validate that the feature sharpness observed in Figure 5 arises from the Riemannian geometric structure rather than model capacity effects, we conducted controlled experiments with matched parameter counts (52.8 M) across all methods. We introduce three metrics: Target Response Ratio (TRR) measuring target-background discrimination, Feature Entropy quantifying spatial concentration, and Boundary Gradient Magnitude measuring edge sharpness.

As shown in

Table 7. Quantitative feature analysis on UAV123 with matched parameters (52.8 M). ↑ indicates higher is better, ↓ indicates lower is better. Bold values indicate the best results.

Method	TRR ↑	Entropy ↓	Gradient ↑
Linear-RNN (Orthogonal)	2.92	5.05	0.45
S4D	2.89	4.82	0.48
GRU	3.05	4.42	0.55
Mamba	3.15	4.58	0.52
LF-SSM (Ours)	4.82	3.21	0.89

, LF-SSM achieves 53% higher TRR, 30% lower entropy, and 71% higher boundary gradient compared to Mamba at matched parameters. Frequency analysis confirms that LF-SSM maintains balanced spectral distribution without overfitting artifacts. These improvements originate from geodesic state evolution properties: exact norm preservation prevents magnitude explosion, adaptive tangent spaces provide location-dependent coordinates, and angular relationship preservation maintains feature discriminability.

6.5. Parameter Sensitivity Analysis

We analyze the sensitivity of LF-SSM to the state dimension N, which controls the capacity of manifold-based state representation. Experiments are conducted on UAV123 with LF-SSM-L configuration, and FPS is measured on RTX 4090.

As shown in

Table 8. Sensitivity analysis of state dimension N on UAV123. FPS is measured on RTX 4090. Bold values indicate the best results.

State Dimension N	AUC	P	Params (M)	FPS
$N=16$	70.2	89.5	48.5	265
$N=32$	71.8	91.2	50.2	248
$N=64$	73.2	92.5	52.8	210
$N=128$	73.5	92.8	58.5	175
$N=256$	73.6	92.9	72.2	138

, increasing the state dimension N from 16 to 64 brings consistent performance improvements, with AUC rising from 70.2% to 73.2%. Further increasing N to 128 and 256 yields marginal gains of 0.3% and 0.4%, respectively, while significantly increasing parameters and reducing inference speed. This indicates that $N=64$ achieves an optimal balance between representation capacity and computational efficiency. The unit sphere ${\mathcal{S}}^{63}$ provides sufficient degrees of freedom for capturing the geometric structure of tracking features without redundant parameterization.

We further analyze the sensitivity to step size configuration.

Table 9. Step size sensitivity analysis on UAV123. FPS is measured on RTX 4090. Bold values indicate the best results.

Step Size Configuration	AUC	P	Params (M)	FPS
Fixed $\mathrm{\Delta }=0.01$	71.5	90.8	52.8	215
Fixed $\mathrm{\Delta }=0.05$	72.3	91.5	52.8	212
Fixed $\mathrm{\Delta }=0.10$	72.8	91.8	52.8	210
Fixed $\mathrm{\Delta }=0.20$	71.8	90.5	52.8	208
Fixed $\mathrm{\Delta }=0.50$	69.2	88.2	52.8	205
Adaptive (Ours)	73.2	92.5	52.8	210

compares fixed step sizes at different scales against our adaptive step size mechanism on UAV123.

The results reveal several important observations about step size sensitivity. Among fixed step sizes, moderate values around $\mathrm{\Delta }=0.10$ achieve the best performance with AUC of 72.8%, while very small step sizes ($\mathrm{\Delta }=0.01$) limit the model’s ability to adapt quickly to appearance changes, and large step sizes ($\mathrm{\Delta }=0.50$) cause overshooting that degrades localization precision. The adaptive step size mechanism outperforms all fixed configurations by 0.4% in AUC, demonstrating that content-aware step size selection enables the network to balance stability and responsiveness according to input characteristics. Empirical analysis of the learned step size distribution shows that the adaptive mechanism produces values concentrated between 0.05 and 0.15, with larger values appearing near object boundaries where rapid feature changes occur.

7. Discussion

The preceding sections have presented the technical formulation and experimental validation of LF-SSM. In this section, we provide deeper theoretical analysis of the key design choices. We examine the stability properties of geodesic dynamics, the mechanism by which temporal information is selectively forgotten, and the relationship between our approach and traditional HiPPO-based state space models.

7.1. Perturbation Stability of Geodesic Dynamics

The exact norm preservation established in Section 4.2 guarantees that states remain bounded on the unit sphere, but does not by itself characterize how the dynamics respond to perturbations in the input or initial conditions. Understanding this sensitivity is essential for the assessment of robustness in practical tracking scenarios where sensor noise and feature extraction errors are inevitable.

Consider two state trajectories generated by the geodesic dynamics. The unperturbed trajectory ${\left\{h_t\right\}}_{t=0}^T$ and the perturbed trajectory ${\left\{{\tilde{h}}_t\right\}}_{t=0}^T$ start from the same initial condition $h_0={\tilde{h}}_0$ but the latter is subject to additive noise ${\eta }_t$ in the tangent velocity at each step. The perturbed update takes the form ${\tilde{h}}_{t+1}={\mathrm{Exp}}_{{\tilde{h}}_t}(\mathrm{\Delta }(v_t+{\eta }_t))$, where ${\eta }_t\in T_{{\tilde{h}}_t}{S}^{N-1}$ represents the perturbation projected onto the tangent space and we assume uniform boundedness $\parallel {\eta }_t\parallel \le \epsilon$ for some small $\epsilon >0$. The quantity of interest is the geodesic distance $d_t=d_S(h_t,{\tilde{h}}_t)$ between the two trajectories, where $d_S(\mathbf{x},\mathbf{y})=arccos\left(\langle \mathbf{x},\mathbf{y}\rangle \right)$ measures arc length on the unit sphere.

The evolution of this distance is governed by the Lipschitz properties of the exponential map on curved manifolds. On the unit sphere with constant positive sectional curvature $\kappa =1$, the exponential map exhibits local Lipschitz continuity in its tangent argument. Specifically, for two tangent vectors $v,v^{\prime }\in T_h{S}^{N-1}$ at the same basepoint with magnitudes that satisfy $\mathrm{\Delta }\parallel v\parallel ,\mathrm{\Delta }\parallel v^{\prime }\parallel <\delta$ for some $\delta <\pi /2$, the geodesic distance between their images satisfies $d_S({\mathrm{Exp}}_h(\mathrm{\Delta }v),{\mathrm{Exp}}_h(\mathrm{\Delta }{v}^{\prime }))\le C\mathrm{\Delta }\parallel v-v^{\prime }\parallel$, where the Lipschitz constant $C=\sqrt{1+{\delta }^2/2}$ depends only on the radius bound $\delta$. This estimate follows from Taylor expansion of the spherical law of cosines, which relates the geodesic distance to the angle between tangent directions and the magnitudes of the tangent vectors.

The combination of this tangent-direction estimate with analogous bounds for basepoint variation and iteration over the sequence length yields a Grönwall-type inequality for the trajectory deviation. Under the assumptions that the step size satisfies $\mathrm{\Delta }<{\mathrm{\Delta }}_{max}$ for some bound that ensures $\mathrm{\Delta }\parallel v_t\parallel <\delta$ throughout the trajectory, that perturbations are uniformly bounded by $\parallel {\mathit{\eta }}_t\parallel \le \epsilon$, and that the trajectories remain within a geodesic ball of radius ${\delta }_0<\pi /2$ of each other, the deviation at time t satisfies

$$d_t\le {\displaystyle \frac{C\epsilon }{L}}(e^{Lt\mathrm{\Delta }}-1),$$

(17)

where $L>0$ is a constant that depends on the curvature and the neighborhood radius. For short time horizons where the product $Lt\mathrm{\Delta }$ remains small, this bound simplifies to approximately linear growth $d_t\approx C\epsilon t\mathrm{\Delta }$, which indicates that perturbations accumulate at most linearly with sequence length. The transition from linear to exponential growth occurs at a critical time $t_{\mathrm{crit}}\sim 1/(L\mathrm{\Delta })$, which for typical tracking parameters with step size $\mathrm{\Delta }=0.1$ and empirically estimated $L\approx 0.015$ occurs around $t\approx 600$ steps. Since most tracking sequences in standard benchmarks contain fewer than a few hundred frames, the geodesic dynamics operate predominantly in the linear growth regime and ensure robust and predictable behavior throughout practical applications.

We validated these theoretical predictions on the LaSOT benchmark with sequences averaging 2500 frames. The state norm deviation $|\parallel h_t\parallel -1|$ remained below ${10}^{-14}$ throughout tracking with FP32 arithmetic, confirming exact norm preservation to machine precision over extended time horizons. The tracking performance (AUC 75.8%) showed no degradation compared to shorter sequences, demonstrating that the geodesic formulation maintains numerical stability even on long sequences where traditional SSMs may suffer from accumulated drift or spectral instability due to eigenvalue sensitivity in the state transition matrix.

7.2. Temporal Forgetting Through Geometric Transport

In traditional state space models, the forgetting of outdated information is controlled by the eigenvalue spectrum of the transition matrix. Eigenvalues with magnitude less than unity cause exponential decay of the corresponding state components, with smaller eigenvalues that produce faster forgetting. This mechanism is uniform and input-independent. All historical information decays at fixed rates determined solely by the matrix spectrum, regardless of the relevance or importance of that information to the current tracking context.

Our geodesic framework implements a fundamentally different forgetting mechanism that arises naturally from the geometry of the sphere rather than from explicit spectral design. The key element is the prior velocity $v_t^{\mathrm{prior}}$, which carries directional momentum from previous updates and provides temporal continuity across the state sequence. When the state moves from $h_t$ to $h_{t+1}$, this prior velocity must be transported to the tangent space at the new location. We employ the projection approximation $v_{t+1}^{\mathrm{prior}}={\mathrm{Proj}}_{h_{t+1}}\left(v_t\right)=v_t-\langle v_t,h_{t+1}\rangle h_{t+1}$, which removes from the old velocity its component along the new state direction while it preserves the orthogonal remainder.

This projection implements selective, geometry-aware forgetting. The amount of information discarded depends on the alignment between the prior velocity and the new state direction, which in turn reflects the curvature of the trajectory through state space. When the tracking target undergoes rapid appearance changes, the state trajectory curves sharply and produces large angles between successive states. In this regime, the projection removes a substantial component of the prior velocity and effectively forgets outdated directional information that is no longer relevant to the current appearance. Conversely, when the target appearance evolves smoothly, the trajectory follows a nearly geodesic path with small curvature, and most of the prior velocity survives projection to maintain temporal consistency. This adaptive behavior emerges automatically from the geometry without explicit parameterization or tuning.

The forgetting rate can be quantified in terms of the geodesic step size. Let ${\theta }_t={\mathrm{\Delta }}_t\parallel v_t\parallel$ denote the arc length traveled in one update step, which equals the geodesic distance $d_S(h_t,h_{t+1})$ between successive states. The magnitude of the transported prior satisfies $\parallel v_{t+1}^{\mathrm{prior}}{\parallel }^2=\parallel v_t{\parallel }^2-{\langle v_t,h_{t+1}\rangle }^2\le \parallel v_t{\parallel }^2(1-{cos}^2\left({\theta }_t\right))={\parallel v_t\parallel }^2{sin}^2\left({\theta }_t\right)$, which yields the bound $\parallel v_{t+1}^{\mathrm{prior}}\parallel \le \parallel v_t\parallel sin\left({\theta }_t\right)$. For small steps where ${\theta }_t\ll 1$, this gives approximately $\parallel v_{t+1}^{\mathrm{prior}}\parallel \approx \parallel v_t\parallel {\theta }_t$, which indicates that the prior magnitude decays in proportion to the step size. Over multiple steps, this produces gradual attenuation of historical momentum and implements smooth forgetting that avoids the abrupt information loss associated with aggressive eigenvalue decay in traditional formulations.

7.3. Relationship to HiPPO-Based State Transitions

The HiPPO framework provides the mathematical foundation for the construction of state transition matrices in modern structured state space models. The core idea is to maintain a compressed representation of input history by projection of the continuous input signal onto a basis of orthogonal polynomials, typically the Legendre polynomials. The resulting transition matrix $\mathbf{A}$ is derived analytically to ensure that the state vector approximates the coefficients of the optimal polynomial projection at each time step. While this construction achieves efficient compression of temporal history, it imposes a fixed representational basis that does not adapt to the specific structure of the input data.

Our geodesic formulation can be understood as a replacement of this fixed global basis with adaptive local coordinates that evolve with the state. At each point $h_t$ on the unit sphere, the tangent space $T_{h_t}{S}^{N-1}$ provides a local coordinate system centered at the current state. Input features are projected onto this tangent space before they are used to update the state, which effectively represents new information in coordinates adapted to the current representation. As the state evolves along its geodesic trajectory, the tangent space rotates accordingly and continuously adapts the local coordinate frame to track the evolving feature geometry. This can be viewed as a moving frame that follows the state along the manifold, in contrast to the fixed polynomial frame used in HiPPO-based methods.

The relationship between geodesic updates and traditional normalized linear updates provides additional insight into this comparison. Consider the normalized linear update $h_{t+1}^{\mathrm{lin}}=(h_t+\mathrm{\Delta }{v}_t)/\parallel h_t+\mathrm{\Delta }{v}_t\parallel$, which applies a linear displacement followed by projection back onto the sphere. Under the scaling assumption that the tangent velocity satisfies $\parallel v_t\parallel =O\left(1\right)$ with step size $\mathrm{\Delta }\ll 1$, both the geodesic update and the normalized linear update admit Taylor expansions. For the normalized linear update, using the binomial expansion ${(1+x)}^{-1/2}=1-x/2+O\left(x^2\right)$ with $x={\mathrm{\Delta }}^2{\parallel v_t\parallel }^2$, we obtain

$$h_{t+1}^{\mathrm{lin}}=h_t+\mathrm{\Delta }{v}_t-{\displaystyle \frac{{\mathrm{\Delta }}^2{\parallel v_t\parallel }^2}{2}}{h}_t+O\left({\mathrm{\Delta }}^3\right)$$

(18)

For the geodesic update, expanding the trigonometric functions $cos(\mathrm{\Delta }\parallel v_t\parallel )=1-{\mathrm{\Delta }}^2{\parallel v_t\parallel }^2/2+O\left({\mathrm{\Delta }}^4\right)$ and $sin(\mathrm{\Delta }\parallel v_t\parallel )=\mathrm{\Delta }\parallel v_t\parallel -{\mathrm{\Delta }}^3{\parallel v_t\parallel }^3/6+O\left({\mathrm{\Delta }}^5\right)$ yields

$$h_{t+1}^{\mathrm{geo}}=h_t+\mathrm{\Delta }{v}_t-{\displaystyle \frac{{\mathrm{\Delta }}^2{\parallel v_t\parallel }^2}{2}}{h}_t+O\left({\mathrm{\Delta }}^3\right)$$

(19)

The leading terms through second order are identical, which confirms that geodesic evolution generalizes normalized linear dynamics to second order in the step size. The difference between the two methods first appears at third order, establishing that $\parallel h_{t+1}^{\mathrm{geo}}-h_{t+1}^{\mathrm{lin}}\parallel =O\left({\mathrm{\Delta }}^3\right)$.

Although the single-step difference is small, it accumulates over long sequences to produce measurable performance gaps. For a sequence of length T with step size $\mathrm{\Delta }$, the accumulated error is of order $T·{\mathrm{\Delta }}^3$. With typical tracking parameters $\mathrm{\Delta }\approx 0.1$ and $T\approx 500$, this corresponds to approximately $500\times 0.001=0.5$ radians of accumulated angular deviation. The geodesic formulation produces smaller third-order errors because its Taylor coefficients arise from the exact spherical geometry, whereas the normalized linear update introduces additional error through the normalization operation. This accumulated difference explains the 3% AUC gap observed in Table 4 between our geodesic method and the Euclidean-plus-normalization baseline.

The geodesic formulation provides two additional advantages beyond this asymptotic equivalence. First, it remains well-defined and numerically stable even when the linear update $h_t+\mathrm{\Delta }{v}_t$ approaches zero magnitude, a regime where normalization-based approaches suffer from division by near-zero quantities. Second, it provides access to the full toolkit of Riemannian geometry, which includes parallel transport for velocity propagation and geodesic distance for the measurement of state similarity, enabling the geometry-aware temporal modeling described in the previous subsection.

7.4. Limitations and Future Directions

While LF-SSM demonstrates strong performance across multiple benchmarks, our analysis reveals certain failure modes that warrant discussion. As illustrated in Figure 6, we observe two representative failure cases on LaSOT and UAV123 datasets. The first involves rapid and extreme scale variations where the target undergoes significant size changes exceeding $4\times$ within a short temporal window. In such scenarios, the geodesic state evolution may fail to adapt quickly enough because the tangent space projection operates on normalized directional features through $v_t=u_t-\langle u_t,h_{t-1}\rangle h_{t-1}$, which can lose scale-sensitive information during dramatic size transitions. As shown in Figure 6 (left), when a drone target in a LaSOT sequence rapidly approaches the camera, the predicted bounding box fails to expand proportionally with the ground truth, resulting in significant scale mismatch. The second failure mode involves significant bounding box drift in cluttered scenes with similar background regions. As shown in Figure 6 (right), in a UAV123 aerial traffic scene, the predicted bounding box exhibits noticeable spatial drift from the ground truth location. This drift occurs because the geodesic state evolution accumulates small errors over time when the target appearance is similar to surrounding background regions. The normalized state update $h_t=cos(\parallel v_t\parallel )h_{t-1}+sin(\parallel v_t\parallel ){\displaystyle \frac{v_t}{\parallel v_t\parallel }}$ can gradually shift when background features consistently influence the input projection $u_t={\mathrm{Linear}}_B\left({\tilde{x}}_t\right)$, causing the state to evolve toward a slightly offset position on the manifold. These failure modes suggest promising research directions including adaptive scale modeling that incorporates magnitude information into the state representation, and robust state update mechanisms that mitigate drift accumulation in challenging scenarios.

Beyond addressing these failure modes, several theoretical aspects of our approach warrant further investigation. The choice of the unit sphere $S^{N-1}$ as the state manifold was motivated primarily by computational convenience, as this manifold admits closed-form expressions for the exponential map, logarithmic map, and parallel transport. Other Riemannian manifolds may offer advantages for specific tracking scenarios. Stiefel manifolds that consist of orthonormal frames could naturally represent pose or orientation dynamics, while hyperbolic spaces with their exponentially expanding geometry might better capture hierarchical feature relationships. The extension of the geodesic state evolution framework to these alternative manifolds represents a promising direction for future work.

The stability analysis presented above assumes bounded step sizes, whereas our implementation employs input-dependent step sizes ${\mathrm{\Delta }}_t=\sigma \left({\mathrm{Linear}}_{\mathrm{\Delta }}\left({\tilde{x}}_t\right)\right)$ computed from the input features. The softplus activation $\sigma$ ensures positivity and empirically produces well-behaved step sizes, but formal stability guarantees for fully adaptive stepping require additional theoretical development. The establishment of conditions on the step size network that ensure long-term stability would strengthen the theoretical foundations of our approach.

Several architectural properties provide implicit guarantees against degenerate dynamics even with adaptive step sizes. The geodesic formulation maintains exact norm preservation $\parallel h_t\parallel =1$ by construction, which eliminates the possibility of state disappearance regardless of the step size values. The input-dependent tangent velocity $v_t={\mathrm{Proj}}_{h_t}(\mathbf{B}{\tilde{x}}_t+\alpha v_t^{\mathrm{prior}})$ changes direction at each time step as different spatial tokens are processed, which prevents the state from following a single great circle and thereby avoids periodic motion. The combination of bounded input features from normalized images, weight decay regularization on the step size network parameters, and the smooth softplus activation ensures that the product ${\mathrm{\Delta }}_t\parallel v_t\parallel$ remains within the domain where the local Lipschitz bounds apply. Empirical measurements confirm that this product stays below $0.3$ radians throughout training and inference, well within the stability regime established by the perturbation analysis.

Finally, the bidirectional processing employed in GSM Blocks operates on spatial token sequences within individual frames rather than temporal sequences across frames. This design choice prioritizes comprehensive spatial feature interaction between template and search regions, with temporal consistency handled through the concatenation mechanism rather than explicit temporal state propagation. The extension of the geodesic dynamics to model temporal evolution across video frames, potentially through hierarchical state spaces that operate at different temporal scales, could further enhance long-term tracking performance.

8. Conclusions

In this paper, we propose LF-SSM, a lightweight HiPPO-free state space model for real-time UAV tracking. Unlike existing SSM-based methods that rely on fixed Legendre polynomial basis and complex discretization procedures, our approach reformulates state evolution on Riemannian manifolds where geodesic dynamics naturally preserve the geometric structure of tracking features. The core contribution is the Geodesic State Module that performs state updates through tangent space projection and exponential mapping on the unit sphere, eliminating the need for HiPPO discretization and specialized scan kernels while providing adaptive local coordinate systems for feature representation. Extensive experiments on four tracking benchmarks demonstrate that LF-SSM achieves competitive performance with state-of-the-art methods while providing significant efficiency improvements, enabling real-time deployment on resource-constrained UAV platforms with 69 FPS on Jetson Orin Nano using only 18.5 M parameters. In future work, we plan to explore Riemannian manifolds with adaptive geometry and integrate online template updating mechanisms to further enhance long-term tracking capability.