DST-Mamba: Spatial–Temporal Adapter with Boundary-Aware Mamba for Video Temporal Action Detection ^†

Abstract

Temporal Action Detection (TAD) localizes and classifies action instances within long videos and underlies many downstream video understanding applications. Transformer-based detectors scale poorly to long sequences due to quadratic self-attention, while existing SSM-based variants tend to dilute fine-grained boundary cues during global modeling. To address these limitations, we propose DST-Mamba, a Decoupled Spatial–Temporal Mamba Adapter inserted into a frozen video backbone for parameter-efficient end-to-end TAD. DST-Mamba decouples spatial and temporal modeling into two cooperating branches and explicitly fuses them through cross-branch interaction. Within the temporal branch, we introduce a Temporal Boundary-aware SSM (TB-SSM) with direction-specific forward/backward state-transition matrices, providing a stronger inductive bias for asymmetric action boundaries. Across multiple benchmarks, DST-Mamba consistently outperforms competitive Transformer- and SSM-based baselines while being more computationally efficient.

1. Introduction

Temporal Action Detection (TAD) constitutes a fundamental and challenging research area within the broader field of video understanding. The primary objective of TAD is to precisely categorize and localize action instances within untrimmed video streams by determining their start and end timestamps. This capability is pivotal for a wide range of real-world applications, including intelligent video surveillance [1,2,3,4,5], complex human behavior analysis [6,7,8], and embodied intelligence [9,10,11].

In previous research, diverse model architectures have been developed to tackle the TAD task. Earlier works relied on multi-scale feature pyramids [12] and Graph Convolutional Networks (GCNs) [13] to capture temporal contexts. More recently, Transformer-based architectures [14] have set new benchmarks by leveraging self-attention mechanisms to model global dependencies. However, these existing paradigms face inherent limitations when scaling to long video sequences. CNN-based methods often struggle with limited receptive fields, leading to insufficient modeling of long-term dependencies. Conversely, while Transformers enable global context awareness, their quadratic computational complexity $O\left(N^2\right)$ creates severe memory bottlenecks and redundancy, making them inefficient for processing high-resolution, untrimmed videos. Consequently, balancing global temporal reasoning with computational efficiency remains an unresolved bottleneck in current TAD research. Recently, State-Space Model (SSM) architecture [15,16] has emerged as a competitive paradigm, offering linear computational complexity $O\left(N\right)$ coupled with robust long-sequence modeling capabilities. This presents a significant opportunity to overcome the scalability constraints of Transformers. However, directly applying SSMs to video understanding is not trivial. Purely global modeling in SSMs can sometimes lead to the oversmoothing [17] issue, causing fine-grained boundary details essential for precise action localization to be submerged by global features. Furthermore, retraining large-scale video backbones from scratch is computationally expensive. Therefore, a parameter-efficient architecture [18] is needed to leverage Mamba’s linear efficiency while retaining discriminative boundary cues for accurate boundary detection.

In this paper, we propose a new TAD framework centered on the Decoupled Spatial–Temporal Mamba Adapter (DST-Mamba). The design is motivated by two observations. First, spatial appearance and temporal dynamics play very different roles for action localization in untrimmed video: spatial features mainly carry local appearance context, while temporal cues primarily determine where an action starts and ends, so coupling them inside a single homogeneous module forces one computation path to compromise between two fundamentally different signals. Second, action onsets and offsets are intrinsically asymmetric, with onsets typically being abrupt and offsets fading more gradually, which suggests that a bidirectional temporal model should adopt direction-specific dynamics rather than sharing all parameters across the two scanning directions. Compared with full fine-tuning on a pretrained model, our method integrates a spatial feature processing module and a temporal SSM module into a frozen pretrained video backbone, and lets the two branches exchange information through explicit cross-branch feature interaction rather than running purely in parallel. DST-Mamba improves feature processing efficiency by decoupling spatial and temporal modeling and then fusing them through feature interaction. Specifically, it first applies spatial downsampling to reduce redundant information, and then uses a parameter-efficient Temporal Boundary-aware SSM (TB-SSM) as the core temporal component, in which the forward and backward state-transition matrices are parameterized independently while the input-dependent projections remain shared, so that the two streams can develop different effective memory horizons for onset and offset cues. We adopt the selective scan mechanism of SSMs to capture long-range dependencies with linear complexity and further introduce dedicated spatial and temporal modules to highlight key boundary changes in video human action features. It helps reduce the oversmoothing issue and enables more robust action representation learning under complex backgrounds. The main contributions of this paper are summarized as follows:

• We propose a parameter-efficient end-to-end TAD framework centered on the DST-Mamba Adapter, which integrates the linear-time Mamba architecture into a frozen video backbone and effectively addresses the trade-off between global modeling capability and computational cost.
• Within the DST-Mamba Adapter, we decouple spatial and temporal modeling and perform feature interaction and fusion, enabling the SSM module to better capture long-range dependencies while preserving fine-grained boundary information.
• We propose the TB-SSM structure, where the state-transition matrices for the forward and backward directions are parameterized separately to better model the inherent asymmetry of action boundaries, and complement this design with a theoretical analysis that links direction-specific memory horizons to action onset and offset dynamics.
• We conduct an extensive empirical study on four diverse TAD benchmarks together with an asymptotic complexity analysis and end-to-end efficiency benchmarking. DST-Mamba consistently surpasses strong Transformer- and SSM-based baselines, with the largest gains at strict tIoU thresholds, while occupying a strictly better operating point in the accuracy–memory–throughput trade-off.

This paper is an extended version of our conference paper [19], accepted by the IEEE International Conference on Multimedia and Expo (ICME) 2026.

2. Related Work

2.1. Temporal Action Detection Methods

Temporal Action Detection has progressed through several architectural paradigms. Early methods relied on multi-scale feature pyramids and graph reasoning over snippet features: BMN [20] introduced a boundary-matching mechanism to score dense candidate proposals, G-TAD [13] formulated proposal generation as graph convolution over video snippets, and AFSD [21] adopted an anchor-free regression head to localize boundaries from salient features. With the rise of self-attention, Transformer-based detectors became dominant: ActionFormer [14] built a one-stage detector on top of a multi-scale Transformer pyramid, TadTR [22] introduced end-to-end localization with learnable action queries, and TriDet [12] refined boundary modeling with a relative-boundary head. To reduce the resource cost of Transformer-based pipelines, TemporalMaxer [23] replaces attention with simpler max-pooling, and ETAD [24] adopts sequentialized encoding with gradient checkpointing. More recently, state-space models have emerged as a competitive third paradigm, where ActionMamba [25] and MambaTAD [26] reuse Mamba blocks as the temporal core, while MS-Temba [27] extends this idea to a multi-scale temporal SSM. Compared with these works, our DST-Mamba further specializes the linear-time SSM for action localization through asymmetric forward and backward state transitions and explicit spatial–temporal feature interaction.

2.2. State-Space Models for Video Understanding

State-Space Models (SSMs) were originally proposed as a general-purpose alternative to attention for long-sequence modeling. Mamba [15] introduces input-dependent selective scan with linear time and memory complexity, and the follow-up work [16] formalizes the connection between Transformers and SSMs under a unified framework. Building on this foundation, VisionMamba [28] introduced a bidirectional SSM for image classification, while VideoMamba [29] adapted the architecture to video representation learning. In the TAD setting, ActionMamba [25] adopts a bidirectional parameter-sharing strategy that reduces computational overhead, MambaTAD [26] integrates Mamba into a long-range temporal pyramid, and MS-Temba [27] introduces a multi-scale temporal SSM tailored to dense action detection. Our TB-SSM differs from these designs by parameterizing the forward and backward state-transition matrices ${\mathbf{A}}^f$ and ${\mathbf{A}}^b$ independently while keeping the input-dependent projections shared, so that the two scanning directions can develop different effective memory horizons that match the inherent asymmetry between action onsets and offsets. Concretely, ActionMamba shares a single state-transition matrix across the forward and backward scans and thus cannot model onset–offset asymmetry, whereas MambaTAD and MS-Temba apply conventional symmetric SSMs over pre-extracted features without spatial–temporal decoupling. DST-Mamba instead embeds a boundary-aware SSM with direction-specific forward and backward transitions into a lightweight frozen-backbone adapter and explicitly decouples and fuses its spatial and temporal branches.

2.3. Parameter-Efficient Adapters for Video Models

Parameter-efficient transfer learning (PEFT) has emerged as a practical alternative to full fine-tuning of large pretrained backbones. The original adapter formulation [18] inserts lightweight bottleneck layers into a frozen network and trains only a small number of additional parameters. LoRA [30] reparameterizes weight updates as low-rank matrices to further reduce memory cost, and AdaptFormer [31] extends adapter-style PEFT to vision Transformer backbones. Within end-to-end TAD, AdaTAD [32] introduced Temporal Informative Adapters (TIA) that wrap a frozen video backbone and make end-to-end training with billion-parameter encoders feasible across thousands of frames. The more recent AdaTAD++ [33] extends this idea by decoupling spatial and temporal training and replacing the temporal 1D convolution with a transformer encoder. Our DST-Mamba follows the same frozen-backbone PEFT philosophy but, in contrast to TIA-style adapters, replaces the temporal core with a boundary-aware bidirectional SSM, decouples the spatial and temporal branches with explicit cross-branch fusion, and is specifically designed to capture asymmetric action boundaries while remaining linear-time in the temporal sequence length. Different from SSM-based TAD detectors such as MambaTAD and MS-Temba, which train a full temporal model on pre-extracted features, DST-Mamba confines the boundary-aware SSM to a lightweight adapter inside a frozen backbone, thereby retaining the parameter efficiency of the PEFT setting while enabling end-to-end training.

3. Materials and Methods

This section provides an overview of our proposed TAD framework and presents the internal design details of our core DST-Mamba Adapter. We introduce the overall framework of the video action detection task constructed in this paper, as shown in Figure 1, and summarize the main notation used throughout the paper in Table 1.

Figure 1. Overall architecture of the proposed TAD framework: DST-Mamba Adapters are inserted into the frozen pretrained backbone, and the adapted features pass through the neck and detection head to produce the final action instances.

Table 1. Summary of the main notation used throughout the paper.

Symbol	Description
$T,H,W$	Number of frames, frame height and width
N	Number of input tokens (patches)
D	Backbone embedding dimension
L	Number of pretrained backbone blocks
${\mathbf{X}}_l,{\mathbf{X}}_l^{\prime }\in {\mathbb{R}}^{N\times D}$	Token features before/after the l-th block
${\mathbf{W}}_l(·)$	Pretrained (frozen) main network of block l
${\mathcal{A}}_l(·)$	DST-Mamba Adapter at block l
$\mathbf{Z}\in {\mathbb{R}}^{N\times D_r}$	Down-projected adapter feature (reshaped to $T\times H\times W\times D_r$)
${\mathbf{Z}}_s,{\mathbf{Z}}_t$	Spatial- and temporal-branch outputs
${\mathbf{A}}^f,{\mathbf{A}}^b$	Forward/backward state-transition matrices (TB-SSM)
${\mathbf{B}}_t,{\mathbf{C}}_t,{\Delta }_t$	Input-dependent SSM projections and step size
${\mathbf{h}}_t$	SSM hidden state at step t
$d_{\mathrm{state}}$	SSM state dimension
${\mathbf{Y}}^f,{\mathbf{Y}}^b$	Forward/backward SSM outputs
${\mathbf{F}}_i\in {\mathbb{R}}^{T_i\times D}$	Neck temporal feature at scale i ($T_i$: temporal length)
${\Psi }_p,{\Psi }_g$	Predicted/ground-truth action instance sets
M	Number of predicted action instances
$(t_s,t_e,c,s)$	Start time, end time, class label, confidence score

3.1. Overall Framework

We first define the task object setting. Given an untrimmed video $\mathbf{V}\in {\mathbb{R}}^{T\times H\times W\times D}$, where T, H, W, and D denote the number of frames, height, width, and channel dimension respectively. The TAD task aims to accurately and efficiently predict a set of action instances ${\Psi }_p={\{{\varphi }_i=(t_s^i,t_e^i,c^i,s^i)\}}_{i=1}^M$, where M is the number of predicted action instances, and $t_s^i$, $t_e^i$, $c^i$, and $s^i$ denote the start time, end time, category, and confidence score of the i-th action instance, respectively. The ground-truth action instances are represented as ${\Psi }_g={\{\widehat{{\varphi }_i}=({\widehat{t}}_s^i,{\widehat{t}}_e^i,{\widehat{c}}^i)\}}_{i=1}^N$, where N is the number of ground-truth action instances. The input video is divided into spatio-temporal patches and embedded as $\mathbf{X}=\mathrm{PatchEmbed}\left(\mathbf{V}\right)+{\mathbf{E}}_{pos}$, where ${\mathbf{E}}_{pos}\in {\mathbb{R}}^{T\times D}$ denotes the positional embedding and $\mathbf{X}\in {\mathbb{R}}^{N\times D}$. N is the number of patches as $N=T/t_p\times H/h_p\times W/w_p$, where $t_p,h_p,w_p$ represent the patch size.

We adopt the pretrained backbone with adapter modules. Given an embedded patch sequence $\mathbf{X}\in {\mathbb{R}}^{N\times D}$, the video backbone consists of a stack of L pretrained main blocks. The main block is augmented with an adapter module. For the l-th block, the feature propagation is defined as

$$\begin{array}{cc}\hfill {\mathbf{X}}_l^{\prime }& ={\mathbf{X}}_{l-1}+{\mathbf{W}}_l\left(\mathrm{LN}\left({\mathbf{X}}_{l-1}\right)\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{X}}_l& ={\mathbf{X}}_l^{\prime }+{\mathcal{A}}_l\left({\mathbf{X}}_l^{\prime }\right),\hfill \end{array}$$

(2)

where ${\mathbf{W}}_l(·)$ denotes the pretrained main network. ${\mathcal{A}}_l(·)$ represents the adapter module, which is responsible for efficient temporal feature refinement.

The Neck layer is introduced to transform the backbone features into a temporally organized representation suitable for detection. Temporal downsampling is applied to aggregate temporal information and generate compact temporal feature sequences. At the i-th Neck layer, the temporal features are obtained as:

$${\mathbf{F}}_i=\mathrm{TemporalPool}\left({\mathbf{F}}_{i-1}\right)\in {\mathbb{R}}^{T_i\times D},$$

(3)

where $T_i$ denotes the temporal length after downsampling. This process yields a sequence of temporally aligned feature maps $\left\{{\mathbf{F}}_i\right\}$, enabling the model to capture actions with varying temporal durations.

Finally, the Head layer is applied to predict temporal action instances from the temporal features $\left\{{\mathbf{F}}_i\right\}$. The head consists of two parallel 1D convolutional branches that share the same multi-scale feature pyramid: a classification branch with sigmoid activation that estimates the per-frame action category likelihood, and a regression branch that predicts the start- and end-distance offsets of each candidate action segment. The head jointly performs action classification and temporal boundary regression, and the final prediction set is formulated as

$${\Psi }_p=\mathrm{Head}\left(\left\{{\mathbf{F}}_i\right\}\right)={\left\{{\varphi }_m=(t_s^m,t_e^m,c^m,s^m)\right\}}_{m=1}^M,$$

(4)

where m indexes the predicted action instances and M denotes the total number of predictions. Predictions ${\Psi }_p$ from multiple temporal scales are aggregated and further refined by Non-Maximum Suppression (NMS) [34] in post-processing, which removes redundant and highly overlapping proposals while preserving high-confidence detections.

3.2. Decoupled Spatial–Temporal Mamba Adapter

As illustrated in Figure 2, the DST-Mamba Adapter is inserted into selected pretrained backbone blocks as a lightweight adapter to enhance temporal modeling capability with minimal additional computation.

Figure 2. Internal structure of the DST-Mamba Adapter. The module decouples spatial and temporal modeling into two branches: (1) the spatial branch extracts localized structural cues using depthwise separable convolutions, and (2) the temporal branch employs the TB-SSM to capture asymmetric state dynamics for precise boundary regression. The two branches interact through feature fusion before being projected back to the original channel dimension.

Given the intermediate feature representation ${\mathbf{X}}_l^{\prime }\in {\mathbb{R}}^{N\times D}$ from the l-th pretrained block, the DST-Mamba first applies a channel down-projection to reduce computational cost:

$$\mathbf{Z}={\mathbf{X}}_l^{\prime }{\mathbf{W}}_{down},{\mathbf{W}}_{down}\in {\mathbb{R}}^{D\times D_r},$$

(5)

where $D_r<D$ denotes the rank as the reduced channel dimension.

The projected features are then reshaped into spatio-temporal form $\mathbf{Z}\in {\mathbb{R}}^{T\times H\times W\times D_r}$. The spatial branch focuses on local spatial context modeling by applying depthwise separable convolutions across spatial dimensions:

$${\mathbf{Z}}_s^{\prime }={\mathrm{DWConv}}_{2d}\left(\mathbf{Z}\right),$$

(6)

where ${\mathbf{Z}}_s^{\prime }$ serves as the intermediate spatial feature. We combine it with the temporal features for fusion spatial–temporal modeling. After upsampling, the temporal features ${\mathbf{Z}}_t^{\prime }$ are added to obtain the result, which is further refined through a second 2D convolution to obtain the final spatial feature representation ${\mathbf{Z}}_s$:

$${\mathbf{Z}}_s={\mathrm{DWConv}}_{2d}({\mathbf{Z}}_s^{\prime }+\mathrm{UpSampling}\left({\mathbf{Z}}_t^{\prime }\right)).$$

(7)

For the temporal branch, we first apply a 1D depthwise convolution. The resulting temporal features are then combined with the spatial features, followed by average pooling to reduce spatial dimensions. Finally, the SSM module is employed to capture global temporal dynamics, as formulated below:

$${\mathbf{Z}}_t^{\prime }={\mathrm{DWConv}}_{1d}\left(\mathbf{Z}\right),$$

(8)

$${\mathbf{Z}}_t=\mathrm{TB}-\mathrm{SSM}\left(\mathrm{AvgPool}({\mathbf{Z}}_t^{\prime }+{\mathbf{Z}}_s^{\prime })\right).$$

(9)

The spatial and temporal features are then adaptively fused and projected back to the original channel dimension:

$${\mathcal{A}}_l\left({\mathbf{X}}_l^{\prime }\right)=\left({\mathbf{Z}}_s+{\mathbf{Z}}_t+\mathbf{Z}\right){\mathbf{W}}_{up}.$$

(10)

where ${\mathbf{W}}_{up}\in {\mathbb{R}}^{D_r\times D}$ denotes the channel up-projection. By decoupling spatial refinement and temporal Mamba modeling, the DST-Mamba achieves an effective balance between representation capacity and computational efficiency.

Early blocks of a video backbone primarily encode low-level spatial appearance and short-range patterns, which are largely preserved under our frozen-backbone setting and benefit little from additional temporal modeling. In contrast, deeper blocks aggregate high-level semantic and long-range temporal context, where boundary-aware refinement is most needed for action localization. Restricting adapter insertion to these higher-level blocks therefore concentrates the limited adapter budget where it has the largest effect, while keeping the overall parameter overhead small.

3.3. Temporal Boundary-Aware SSM

The temporal modeling in DST-Mamba is performed by our proposed TB-SSM, illustrated in Figure 3. We build upon the basic Mamba [15], which efficiently captures long-range dependencies through linear state updates. TB-SSM adopts independent state-transition parameters for temporal forward and backward scans, while sharing the input-dependent projection parameters that generate ${\mathbf{B}}_t,{\mathbf{C}}_t$.

Figure 3. Internal structure of the TB-SSM with independent forward/backward state-transition matrices and shared input-dependent projections.

Given the input sequence $\mathbf{X}\in {\mathbb{R}}^{HW\times T\times D_r}$, we first apply layer normalization and an input projection:

$${\mathbf{X}}^f,{\mathbf{X}}^b=\mathrm{Split}\left({\mathbf{W}}_{in}\left(\mathrm{Norm}\left(\mathbf{X}\right)\right)\right),$$

(11)

where ${\mathbf{W}}_{in}\in {\mathbb{R}}^{D_r\times 2D_r}$ denotes the input projection of the TB-SSM, and two streams ${\mathbf{X}}^f\in {\mathbb{R}}^{HW\times T\times D_r},{\mathbf{X}}^b\in {\mathbb{R}}^{HW\times T\times D_r}$ for forward and backward temporal scans split from $\mathbf{X}\in {\mathbb{R}}^{HW\times T\times 2D_r}$.

$${\mathbf{X}}^f=\mathbf{X},{\mathbf{X}}^b=\mathbf{X}.\mathrm{Flip}\left(\right),$$

(12)

For the forward stream, TB-SSM performs selective state updates with an independent transition matrix ${\mathbf{A}}^f$:

$$\begin{array}{cc}\hfill {\mathbf{h}}_t^f& ={\mathbf{A}}^f{\mathbf{h}}_{t-1}^f+{\mathbf{B}}_t{\mathbf{X}}_t^f,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathbf{Y}}_t^f& ={\mathbf{C}}_t{\mathbf{h}}_t^f,\hfill \end{array}$$

(14)

where ${\mathbf{X}}_t^f$ denotes the token at time step t. For the backward stream, we apply the same update rule but with a different transition matrix ${\mathbf{A}}^b$:

$$\begin{array}{cc}\hfill {\mathbf{h}}_t^b& ={\mathbf{A}}^b{\mathbf{h}}_{t-1}^b+{\mathbf{B}}_t{\mathbf{X}}_t^b,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathbf{Y}}_t^b& ={\mathbf{C}}_t{\mathbf{h}}_t^b,\hfill \end{array}$$

(16)

To motivate our boundary-aware design, we revisit the role of the state-transition matrices in the recurrence ${\mathbf{h}}_t=\overline{\mathbf{A}}{\mathbf{h}}_{t-1}+\overline{\mathbf{B}}{\mathbf{x}}_t$, where modes with $|{\lambda }_i\left(\overline{\mathbf{A}}\right)|\approx 1$ retain long-range context while modes with $|{\lambda }_i|\approx 0$ yield rapid forgetting. Parameterizing ${\mathbf{A}}^f$ and ${\mathbf{A}}^b$ independently frees the two scan directions from a shared decay profile and provides an inductive bias for direction-specific memory horizons, namely longer context for onsets and sharper responses for offsets, which the directional selective scan together with the input-dependent $(\Delta ,\mathbf{B},\mathbf{C})$ can exploit.

Finally, we concatenate the bidirectional outputs and apply an output projection to obtain the temporal representation:

$${\mathbf{Z}}_t={\mathbf{W}}_{out}({\mathbf{Y}}^f\parallel {\mathbf{Y}}^b.\mathrm{Flip}\left(\right)),$$

(17)

where ${\mathbf{W}}_{out}\in {\mathbb{R}}^{2D_r\times D_r}$ denotes the output projection. This design enables direction-specific state dynamics for modeling asymmetric action boundaries, while keeping the overall temporal modeling linear in sequence length.

3.4. Loss Function

The proposed framework is trained end-to-end with the same dual-task supervision used by ActionFormer [14] and AdaTAD [32], so that the comparison with these baselines remains fair. The two head branches in Section 3 are optimized jointly: the classification branch is supervised by Focal Loss [35] to mitigate the foreground–background imbalance that is common in untrimmed videos, while the regression branch is supervised by DIoU Loss [36] to directly optimize the temporal alignment between predicted and ground-truth segments.

For a given temporal location m, let $c^m\in {[0,1]}^K$ denote the predicted per-class probabilities over K action categories, ${\widehat{c}}^m$ the ground-truth one-hot label, and $(t_s^m,t_e^m)$ and $({\widehat{t}}_s^m,{\widehat{t}}_e^m)$ the predicted and ground-truth start/end times, respectively. The classification loss is defined as

$${\mathcal{L}}_{\mathrm{cls}}={\displaystyle \frac{1}{N_{\mathrm{pos}}}}\sum _{m\in \Omega }\mathrm{Focal}\left(c^m,{\widehat{c}}^m\right),$$

(18)

where $\Omega$ denotes the set of sampled temporal locations and $N_{\mathrm{pos}}$ the number of positive samples. The boundary regression loss is computed only on positive locations ${\Omega }^{+}$ as

$${\mathcal{L}}_{\mathrm{reg}}={\displaystyle \frac{1}{N_{\mathrm{pos}}}}\sum _{m\in {\Omega }^{+}}\mathrm{DIoU}\left((t_s^m,t_e^m),({\widehat{t}}_s^m,{\widehat{t}}_e^m)\right).$$

(19)

The overall training objective combines the two terms with a balancing weight $\lambda$:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{cls}}+\lambda {\mathcal{L}}_{\mathrm{reg}}.$$

(20)

We follow the default setting $\lambda =1$ used in ActionFormer.

4. Results

4.1. Datasets

To assess our method under diverse video conditions, we conduct experiments on four widely used Temporal Action Detection benchmarks that together cover third-person and first-person viewpoints, sparse and densely concurrent actions, and short to long video durations.

• THUMOS14 [37]. THUMOS14 is a long-standing benchmark for fine-grained temporal action localization in untrimmed sports videos. It contains 413 videos covering 20 action classes, where 200 validation videos are used for training and 213 testing videos for evaluation under the standard TAD protocol. Action instances range from roughly one second to tens of seconds and are separated by long background segments, making the dataset particularly suitable for evaluating boundary localization.
• ActivityNet-1.3 [38]. ActivityNet-1.3 is a large-scale, web-collected benchmark covering 200 everyday human activity categories with 19,994 untrimmed videos split into roughly 10K training, 5K validation, and 5K testing videos; following common TAD practice we train on the training split and report results on the validation split, as the testing labels are withheld. Compared with THUMOS14, its videos are substantially longer (averaging around two minutes) and typically contain only one or two long action instances, which stresses long-range temporal modeling.
• Charades [39]. Charades is a large-scale crowdsourced dataset of indoor daily life activities, comprising 9848 short videos (around 30 s on average) and 157 fine-grained action classes. It provides 66,500 action instance annotations with frequent temporal overlap between concurrent actions performed by the same actor, making it a challenging testbed for handling overlapping segments under cluttered indoor scenes.
• EPIC-Kitchens-100 [40]. EPIC-Kitchens-100 is a large-scale egocentric benchmark of long unscripted cooking videos, with 700 first-person videos totaling roughly 100 h and annotated by 97 verb classes and 300 noun classes. Individual videos can span tens of minutes with densely occurring, rapidly switching actions, which makes precise boundary localization much harder than in third-person datasets. Following recent end-to-end TAD methods, we evaluate on the verb subset, where each ground-truth segment carries a verb label.

4.2. Evaluation Metrics

Following standard practice in Temporal Action Detection, we adopt the mean Average Precision (mAP) computed at multiple temporal Intersection over Union (tIoU) thresholds. Given a predicted segment $\widehat{\varphi }=({\widehat{t}}_s,{\widehat{t}}_e)$ and a ground-truth segment $\varphi =(t_s,t_e)$, the tIoU is defined as

$$\mathrm{tIoU}(\widehat{\varphi },\varphi )={\displaystyle \frac{max\left(0,min({\widehat{t}}_e,t_e)-max({\widehat{t}}_s,t_s)\right)}{max({\widehat{t}}_e,t_e)-min({\widehat{t}}_s,t_s)}}.$$

(21)

At a given threshold $\theta$, a prediction is treated as a true positive if its tIoU with an unmatched ground-truth segment of the same class is at least $\theta$. The per-class Average Precision ${\mathrm{AP}}_c\left(\theta \right)$ is the area under the resulting precision–recall curve, and the per-threshold mAP averages ${\mathrm{AP}}_c\left(\theta \right)$ over all C classes:

$$\mathrm{mAP}\left(\theta \right)={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}{\mathrm{AP}}_c\left(\theta \right).$$

(22)

The average mAP reported in our experiments is the mean of $\mathrm{mAP}\left(\theta \right)$ over the dataset-specific tIoU thresholds.

4.3. Implementation Details

We implemented our method on NVIDIA A100 GPUs, utilizing CUDA 12.8 within the PyTorch 2.8 environment. Throughout the model training process, we adopted the AdamW [41] optimizer and employed a scheduler that integrates linear warm-up and cosine annealing to adjust the learning rate, which was set to 1e-4, with a batch size of 8. For video data, the sampling length of video frames was fixed at 768. Videos exceeding this length were subsampled to the maximum length, while those shorter were padded. Each frame was resized to a resolution of $224\times 224$ pixels. To account for the effect of random initialization, all reported results are averaged over three independent runs with different random seeds.

For completeness, the main hyper-parameters are summarized in Table 2. We adopt the VideoMAEv2 encoder as the backbone and keep it frozen during training. The DST-Mamba Adapters reduce the backbone embedding dimension from $D=1408$ to $D_r=352$, and the TB-SSM adopts a state dimension $d_{\mathrm{state}}=16$. We train for 60 epochs with AdamW and a weight decay of 0.05. One DST-Mamba Adapter is inserted into each block in the latter 50% of the backbone, where high-level semantic features benefit most from temporal modeling. Only the adapter, neck, and detection head are trained, amounting to about 5% of the model parameters.

Table 2. Main hyper-parameters used in our experiments.

Configuration	Value
Input clip	768 frames, $224\times 224$
Backbone embedding dim D	1408
Adapter reduced dim $D_r$	352
TB-SSM state dim $d_{\mathrm{state}}$	16
Optimizer	AdamW
Learning rate	$1\times {10}^{-4}$
Weight decay	0.05
LR schedule	linear warm-up + cosine
Batch size	8
Training epochs	60

4.4. Evaluation and Comparison

We evaluate our method on the THUMOS14 and ActivityNet-1.3 benchmarks, comparing against both end-to-end (E2E) and non-E2E baselines. The quantitative results are presented in Table 3. On THUMOS14, DST-Mamba achieves superior average mAP, maintaining a consistent lead across all evaluated tIoU thresholds. This trend is particularly evident at high tIoU, indicating that our boundary-aware temporal modeling significantly refines boundary regression precision rather than merely improving coarse detection. On ActivityNet-1.3 dataset, DST-Mamba demonstrates strong generalization ability by outperforming competitive baselines. The improvements are consistent across various overlap thresholds, verifying that our method effectively handles long-range temporal reasoning. Furthermore, regarding computational efficiency, our approach maintains a lower memory footprint compared with other E2E baselines, striking a favorable balance between performance and resource consumption. Regarding the comparison setup, the feature-based (non-E2E) baselines are quoted from their original papers and therefore use their respective backbones (I3D [42], CLIP [43], or InternVideo2 [44]); they are listed for completeness. The most controlled comparison is within the end-to-end group, where DST-Mamba, AdaTAD [32], and MambaTAD [26] all share the same VideoMAEv2 [45] backbone, so the gains of DST-Mamba within this group are attributable to the proposed adapter rather than to the underlying video encoder.

Table 3. Comparison with previous methods on THUMOS14 and ActivityNet-1.3 Datasets.

Method	Backbone	E2E	THUMOS14						ActivityNet-1.3
			0.3	0.4	0.5	0.6	0.7	Avg.	0.5	0.75	0.95	Avg.
ETAD [24]	I3D [42]	✓	67.7	64.2	58.2	49.2	38.4	55.6	55.8	40.1	10.3	38.8
AFSD [21]	I3D [42]	✗	73.2	68.5	60.5	49.3	31.2	56.0	54.4	35.7	8.7	35.1
ActionFormer [14]	I3D [42]	✗	82.1	77.8	71.0	59.4	43.9	66.8	55.5	38.1	8.8	37.1
BSFTAL [46]	InternVideo2 [44]	✗	84.2	79.5	73.5	61.8	46.1	69.0	54.8	37.5	9.1	37.1
CLTDR-GMG [47]	InternVideo2 [44]	✗	85.3	80.6	74.8	63.2	48.0	70.4	58.3	40.2	9.4	39.5
TemporalMaxer [23]	InternVideo2 [44]	✗	85.2	81.3	74.2	63.7	49.1	70.7	61.0	40.5	9.8	38.6
TriDet [12]	InternVideo2 [44]	✗	85.8	81.8	75.6	64.6	50.5	71.7	56.4	38.7	9.6	37.5
VideoMamba [29]	InternVideo2 [44]	✗	86.5	82.1	76.3	65.1	50.2	71.8	62.1	43.2	9.5	41.6
ActionMamba [25]	InternVideo2 [44]	✗	87.1	82.7	76.5	65.6	49.9	72.3	62.2	37.1	9.6	41.7
BDRC-Net [48]	InternVideo2 [44]	✗	87.3	83.7	78.1	67.4	53.1	73.9	59.6	38.9	9.8	41.9
MambaTAD [26]	InternVideo2 [44]	✓	87.5	83.8	78.3	67.3	52.9	74.2	63.1	44.2	11.0	43.5
AdaTAD++ [33]	VideoMAEv2 [45]	✓	86.5	81.9	75.6	64.8	49.1	71.6	60.3	43.2	10.1	41.0
AdaTAD [32]	VideoMAEv2 [45]	✓	86.8	82.3	76.1	65.3	49.9	72.1	59.8	42.0	9.7	40.8
MambaTAD [26]	VideoMAEv2 [45]	✓	86.7	82.9	77.0	66.7	52.4	73.1	61.5	43.4	10.5	42.2
DST-Mamba (Ours)	VideoMAEv2 [45]	✓	89.5	85.3	78.9	69.0	54.1	75.3	63.8	44.6	11.2	43.9

Bold values indicate the best performance; ✓ and ✗ indicate whether the method is end-to-end.

We further extend our evaluation to the Charades dataset, as shown in Table 4. DST-Mamba consistently surpasses previous methods across all metrics. The performance gap is most distinct at tighter tIoU thresholds, which supports our motivation that modeling asymmetric boundary dynamics is crucial for preserving discriminative cues in scenarios involving complex activities and heavy background clutter.

Table 4. Comparison with previous methods on Charades Dataset.

Method	Backbone	E2E	0.2	0.5	0.7	Avg.
MS-TST [49]	CLIP [43]	✗	50.1	37.9	19.2	31.9
AAN [50]	CLIP [43]	✗	51.2	38.4	20.7	32.0
MT-Temba [27]	CLIP [43]	✗	51.2	38.4	20.7	32.3
AdaTAD [32]	VideoMAEv2 [45]	✓	53.7	42.9	26.7	37.5
MambaTAD [26]	VideoMAEv2 [45]	✓	54.1	43.2	26.6	37.7
DST-Mamba (Ours)	VideoMAEv2 [45]	✓	57.2	45.5	27.1	38.9

Bold values indicate the best performance; ✓ and ✗ indicate whether the method is end-to-end.

Finally, we evaluate DST-Mamba on the egocentric EPIC-Kitchens-100 verb subset, as reported in Table 5. Compared with strong end-to-end baselines that share the same VideoMAE-L backbone, DST-Mamba consistently improves average mAP and the high-tIoU thresholds, which are particularly sensitive to precise boundary localization. The gain over the strongest prior baseline is moderate yet consistent across thresholds, indicating that the proposed boundary-aware temporal modeling generalizes from third-person sports videos to first-person cooking scenes with dense and rapidly switching actions.

Table 5. Comparison with previous methods on EPIC-Kitchens-100 (verb subset). All end-to-end methods marked with ✓ share the same VideoMAE-L backbone for a fair comparison.

Method	Backbone	E2E	0.1	0.2	0.3	0.4	0.5	Avg.
BMN [20]	TSN	✗	10.8	8.8	8.4	7.1	5.6	8.4
G-TAD [13]	I3D	✗	12.1	11.0	9.4	8.1	6.5	9.4
ActionFormer [14]	I3D	✗	16.6	25.4	22.3	21.3	17.4	20.6
TriDet [12]	I3D	✗	32.7	26.1	24.2	21.8	20.5	25.1
AdaTAD [32]	VideoMAE-L	✓	33.1	32.2	30.4	27.5	25.1	29.3
AdaTAD++ [33]	VideoMAE-L	✓	35.1	34.2	32.9	30.0	27.9	32.0
DST-Mamba (Ours)	VideoMAE-L	✓	35.8	35.0	33.7	30.8	28.5	32.8

Bold values indicate the best performance; ✓ and ✗ indicate whether the method is end-to-end.

4.5. Qualitative Analyses

Figure 4 and Figure 5 show the qualitative visualization results for representative action classes. For each example, the top row presents sampled frames with their timestamps, and the three timelines below show the ground truth, the baseline MambaTAD [26] prediction, and our DST-Mamba prediction, respectively. The colored chips mark the temporal extent of each segment, and the tIoU values above each prediction quantify the alignment with the ground truth. Across all examples, DST-Mamba consistently produces tighter and more accurate boundary localization than the baseline, with noticeably higher tIoU values for the same ground-truth segments. As shown in Figure 6 for two examples, bright regions with high activation align closely with the ground-truth action segments, whereas the regions outside actions remain dark and low-activated. Sharp transitions in activation magnitude appear precisely around action onsets and offsets, indicating that our TB-SSM can better perceive the temporal representations of actions directly from the feature space. To further substantiate the direction-specific behavior discussed in Section 3, Figure 7 plots the per-frame forward and backward hidden-state response magnitudes $\parallel {\mathbf{h}}_t^f\parallel$ and $\parallel {\mathbf{h}}_t^b\parallel$ along the temporal axis, aligned with the ground-truth action segments. Both responses are strongly activated within action intervals and suppressed in background regions, with sharp transitions appearing precisely at action onsets and offsets. Moreover, the forward and backward streams exhibit distinct temporal dynamics rather than identical profiles, providing direct empirical evidence that the two directions specialize for asymmetric boundary modeling.

Figure 4. Qualitative comparison between DST-Mamba and the baseline MambaTAD [26]. Each panel shows sampled frames with timestamps and three timelines annotated with tIoU values.

Figure 5. Additional qualitative comparison between our DST-Mamba and the baseline MambaTAD [26].

Figure 6. TB-SSM internal feature visualization on two examples. Top: LongJump. Bottom: CleanAndJerk. The heatmap shows the averaged $\parallel {\mathbf{h}}^f\parallel /\parallel {\mathbf{h}}^b\parallel$ activation along the temporal axis. Bright regions with high activation align with the ground-truth action intervals, while sharp transitions emerge at action boundaries.

Figure 7. Per-frame forward ($\parallel {\mathbf{h}}_t^f\parallel$, blue) and backward ($\parallel {\mathbf{h}}_t^b\parallel$, red) hidden-state response magnitudes of TB-SSM on an example, aligned with the ground-truth action segments (shaded green). Activations rise inside action intervals and transition sharply at onsets/offsets, while the two directions display distinct temporal dynamics.

We also present two representative failure cases in Figure 8. In the first, the model confuses two visually similar fine-grained actions: the top-1 prediction selects CricketBowling, while the correct CricketShot is recovered only as the top-2 candidate. In the second, the most confident prediction over-extends the action boundary, while a better-localized prediction is ranked only top-2. These cases suggest that fine-grained action disambiguation and confidence calibration remain open challenges for future work.

Figure 8. Two failure cases of our method, showing an incorrect action-class prediction and a mismatched temporal-region prediction. The ✗ marks the incorrect prediction in each case.

4.6. Ablation Studies

We conduct comprehensive ablation studies on THUMOS14 and ActivityNet-1.3 to validate the contribution of each component in DST-Mamba. The results are summarized in Table 6. First, removing the temporal branch results in the most significant performance degradation, confirming that temporal modeling is the primary driver for long-range action reasoning. Second, discarding the spatial refinement branch also leads to consistent drops, suggesting that lightweight spatial processing remains essential for retaining localized appearance cues that aid in temporal boundary discrimination. Third, employing a simple parallel structure without the spatial–temporal fusion mechanism yields suboptimal results, indicating that explicit feature interaction is necessary to effectively couple spatial evidence with temporal dynamics. Finally, replacing TB-SSM with a variant lacking the proposed boundary-aware asymmetric state dynamics leads to inferior performance. This validates that direction-specific temporal state transitions are vital for capturing the intrinsic asymmetry between action onsets and offsets. Overall, the full model delivers the best results, demonstrating that these components contribute complementarily to accurate and robust temporal localization.

Table 6. Ablation study on the effect of each component in the proposed method.

Components				THUMOS14	ActivityNet-1.3
Spatial	Temporal	S-T Fusion	TB-SSM	mAP Avg.	mAP Avg.
✗	✓	✗	✓	74.4	43.0
✓	✗	✗	✗	72.3	41.8
✓	✓	✗	✓	73.9	42.9
✓	✓	✓	✗	73.7	42.6
✓	✓	✓	✓	75.3	43.9

Bold values indicate the best performance; ✓ and ✗ indicate whether the corresponding component is used.

In Table 7, we further investigate various temporal modeling strategies by replacing the temporal block inside our DST-Mamba Adapter with representative design paradigms while keeping the rest of the adapter unchanged. The Transformer row uses a standard self-attention layer, while the VideoMamba and ActionMamba rows correspond to bidirectional SSMs with independent and shared forward and backward state-transition matrices, respectively. Our experiments show that Transformer-based modules offer only marginal gains compared with the baseline without temporal modeling, whereas SSM-based designs consistently yield better performance. Among them, linear-time state-space models like VideoMamba and ActionMamba improve detection accuracy, confirming their suitability for long temporal sequences. However, our TB-SSM achieves the highest performance, demonstrating that introducing boundary-aware asymmetric temporal dynamics provides a stronger inductive bias for temporal localization than conventional bidirectional or shared-parameter SSM designs.

Table 7. Ablation study on different temporal modeling strategies.

Temporal Module Strategy	THUMOS14	ActivityNet-1.3
	mAP Avg.	mAP Avg.
w/o	72.3	41.8
Transformer [51]	72.4	42.0
VideoMamba [29]	73.6	42.4
ActionMamba [25]	73.2	42.7
TB-SSM (Ours)	75.3	43.9

Bold values indicate the best performance.

Beyond the component-level ablations above, we further inspect three internal design choices of DST-Mamba: the spatial pooling size used to compress spatial tokens before TB-SSM, the SSM state dimension $d_{\mathrm{state}}$ inside TB-SSM, and the bidirectional design that controls how the forward and backward streams interact. Since the first two ablations sweep a single scalar hyper-parameter, we visualize them as line plots in Figure 9 for clarity, while the bidirectional design is reported as a separate table.

Figure 9. Ablation on (a) the spatial pooling size applied before TB-SSM ($16\times 16$ runs out of memory and is omitted), and (b) the SSM state dimension $d_{\mathrm{state}}$ inside TB-SSM. Each panel uses twin y-axes to compare THUMOS14 (left, blue) and ActivityNet-1.3 (right, orange) average mAP.

Figure 9a sweeps the spatial pooling size that controls how aggressively the spatial dimensions are compressed before TB-SSM. A pooling size of $1\times 1$ collapses all spatial information into a single token and discards localized appearance cues, leading to noticeably weaker results, while increasing the pooling size beyond $4\times 4$ inflates the effective batch dimension fed to TB-SSM and drives memory cost up sharply ($16\times 16$ already runs out of memory in our setting). The $4\times 4$ configuration retains sufficient spatial granularity for boundary discrimination while keeping the SSM input compact, and is therefore adopted as the default.

Figure 9b ablates the SSM state dimension $d_{\mathrm{state}}$, which directly controls the capacity of the hidden state in TB-SSM. Performance grows when $d_{\mathrm{state}}$ increases from 8 to 16, but then saturates and slightly drops for larger values: further increasing $d_{\mathrm{state}}$ inflates the TB-SSM parameter count proportionally yet does not translate into accuracy gains. This indicates that the gains observed for our full model do not stem from raw capacity but from the structural inductive bias of asymmetric forward and backward state transitions, and we therefore set $d_{\mathrm{state}}=16$ throughout the paper.

Finally, Table 8 isolates the two design factors of bidirectional SSMs: whether the forward and backward streams use shared or independent state-transition matrices ${\mathbf{A}}^f$ and ${\mathbf{A}}^b$, and whether the bidirectional outputs are fused by element-wise addition or by channel concatenation. The first row uses shared $\mathbf{A}$ with concat fusion, the second row uses independent $\mathbf{A}$ with additive fusion, and our TB-SSM combines independent ${\mathbf{A}}^f,{\mathbf{A}}^b$ with concat fusion. Switching from shared to independent $\mathbf{A}$ introduces only a negligible parameter overhead over the shared variant, yet substantially improves accuracy, a much higher accuracy-per-parameter return than simply increasing $d_{\mathrm{state}}$ as discussed above. This further supports our analysis in Section 3: independent state transitions allow the two directions to develop distinct effective memory horizons, and concatenation preserves this directional information rather than averaging it out at the channel level.

Table 8. Ablation on bidirectional SSM design choices inside TB-SSM. $\mathbf{A}$ Params indicates whether the forward and backward state-transition matrices are shared or independent, and Fusion indicates whether the two directional streams are combined by element-wise addition or by channel concatenation. The last row corresponds to our TB-SSM.

A Params		Fusion		THUMOS14	ActivityNet-1.3
Shared	Independent	Add	Concat
✓	✗	✗	✓	73.2	42.7
✗	✓	✓	✗	73.6	42.4
✗	✓	✗	✓	75.3	43.9

Bold values indicate the best performance; ✓ and ✗ indicate whether the corresponding condition is present.

4.7. Complexity and Efficiency

In terms of asymptotic complexity, self-attention scales as $O\left(T^2\right)$ in both time and space, depthwise Conv1d as $O(T·k·C)$, and the selective scan inside Mamba and TB-SSM as $O(T·d_{\mathrm{state}}·C)$ in time and $O(T·d_{\mathrm{state}})$ in space. The quadratic scaling of self-attention is the main reason why Transformer-based detectors become memory-bound on long videos, while TB-SSM simultaneously preserves Mamba’s linear complexity and a global temporal receptive field. The memory–accuracy comparison in Figure 10 further confirms this empirically: among the main compared methods, DST-Mamba sits in the top-left region, achieving the highest average mAP on THUMOS14 while using less peak GPU training memory than the strongest AdaTAD configuration and the SSM-based baselines (ActionMamba and MambaTAD). This demonstrates that our method improves detection performance while simultaneously reducing the computational resource consumption.

Figure 10. Memory–accuracy comparison for the main compared TAD methods.

In addition, Table 9 benchmarks the inference throughput (frames per second, FPS) and peak GPU memory of the main compared methods. DST-Mamba uses the lowest peak memory and achieves the highest performance, further demonstrating the superiority of our method.

Table 9. Inference efficiency comparison with the main compared methods on THUMOS14.

Method	FPS	Peak Mem.	Avg. mAP
AdaTAD [32]	11.3	29.9 G	72.1
ActionMamba [25]	10.8	30.2 G	72.3
MambaTAD [26]	11.2	30.7 G	73.1
DST-Mamba (Ours)	11.6	28.6 G	75.3

Bold values indicate the best performance.

5. Conclusions

We presented DST-Mamba, a parameter-efficient end-to-end TAD framework that inserts a decoupled spatial–temporal adapter into a frozen video backbone. At its core, the TB-SSM parameterizes the forward and backward state-transition matrices independently, yielding direction-specific memory horizons aligned with action onset and offset dynamics, and we further provide a theoretical analysis to ground this design. Extensive experiments on four diverse benchmarks, together with an asymptotic complexity analysis and end-to-end inference efficiency benchmarking, show that DST-Mamba consistently outperforms strong Transformer- and SSM-based baselines across various tIoU thresholds. We hope this work offers useful insights for future research on video temporal action detection.