Hierarchical Prototype Alignment for Video Temporal Grounding

Abstract

Recent advances in vision-language cross-modal learning have substantially improved the performance of video temporal grounding. However, most existing methods directly associate global video features with sentence-level features, overlooking the fact that textual semantics usually correspond to only limited spatio-temporal regions within a video. This limitation often leads to unstable alignment in complex scenarios involving intertwined events and diverse actions. In essence, accurate video temporal grounding requires the joint modeling of fine-grained spatial semantics and heterogeneous temporal event structures. Motivated by this observation, we propose a hierarchical prototype alignment approach that models cross-modal correspondence between video and text through structured intermediate prototype representations. Specifically, the alignment process is decomposed into two complementary stages: object-phrase alignment and event-sentence alignment. In the object-phrase alignment stage, discriminative local visual regions and informative textual words are aggregated to construct object and phrase prototypes, thereby enhancing fine-grained spatial correspondence at the level of entities and localized actions. In the event-sentence alignment stage, object prototypes are further integrated along the temporal dimension to form event prototypes that represent continuous action units, enabling effective alignment with sentence-level semantics and facilitating the modeling of diverse temporal event structures. On this basis, we further directly inject cross-modal alignment information into candidate moment aggregation. This design allows candidate moment representations to emphasize query-relevant temporal regions. Extensive experiments on Charades-STA, ActivityNet Captions, and TACoS demonstrate that the proposed method outperforms existing approaches, validating the effectiveness of hierarchical prototype alignment for improving both cross-modal alignment quality and temporal grounding accuracy.

1. Introduction

Understanding multimodal information [1,2,3] is a fundamental capability that enables computers to perceive and interpret the world. As a core task in multimodal learning [4,5,6], video temporal grounding (VTG) has broad applications in multimodal retrieval, intelligent surveillance, and human–computer interaction, and has therefore attracted significant research attention. VTG [7,8] is defined as the task of localizing a temporal segment within an untrimmed video that corresponds to a given natural language query. Situated at the intersection of computer vision (CV) and natural language processing (NLP), VTG serves as a fundamental component in various video understanding tasks, including video dialogue [9], relationship detection [10,11,12], and question answering [13,14,15].

Videos contain complex spatio-temporal dynamics, involving evolving scenes, entities, and events. In contrast, natural language is inherently abstract and ambiguous, where a single sentence may refer to different combinations of actions and objects. This inherent modality gap [16,17,18] increases the difficulty of associating multimodal data and complicates feature integration in video temporal grounding. In recent years, many studies have focused on improving video-text alignment for temporal localization tasks. Several existing methods [19,20,21,22,23] represent both video and query as single embeddings and perform matching within a global semantic space. While these approaches attempt to establish a holistic semantic correspondence between the video and the query, in practical scenarios, the query usually refers to a specific spatio-temporal segment, which makes global matching susceptible to interference from irrelevant content. To improve matching granularity, later works [24,25,26,27,28] introduce frame-level or segment-level matching strategies that assign different weights to frames or segments for fine-grained alignment, thereby improving discriminative capability. However, videos contain rich visual elements, and a text description may correspond only to a spatio-temporal portion of the video. Consequently, dynamic and fine-grained matching is required beyond global video-level or coarse frame-level matching. More specifically, feature encoding should represent semantically diverse components to support localized and context-sensitive alignment during the matching process. To address this issue, several studies attempt to incorporate intermediate representations to mitigate semantic inconsistencies. For example, Lin et al. [29] decompose videos into semantic events and introduce event prototypes that enable flexible alignment between queries and temporal event structures. As illustrated in Figure 1a, the query specifies not only the action but also the tool and object, and its semantics do not correspond to the entire scene but depend on several localized visual cues. If alignment relies only on global features or coarse event representations, the model may overlook local evidence or become sensitive to background distractions, which may lead to misalignment [30,31].

Based on these observations, we argue that reliable video-text alignment should be grounded in fine-grained local semantics and supported by a structurally stable intermediate representation. Such a representation carries multi-level semantics, preserves necessary spatial correspondence, and progressively integrates temporal context to facilitate hierarchical cross-modal alignment. However, local entity cues and temporal event structures are inherently heterogeneous. Spatial semantics originate from transient visual regions, whereas event-level meanings emerge from temporal aggregation across frames. Modeling them within a single unified representation may blur this distinction, allowing dominant temporal patterns to overshadow localized evidence or causing spatial noise to disrupt temporal coherence. To address this structural mismatch, we organize the alignment process along both spatial and temporal dimensions through a spatio-temporal prototype formulation. We first generate object prototypes in the spatial dimension by aggregating salient local regions to capture phrase-related entities and local action cues. As shown in Figure 1b, object and phrase prototypes enable the model to identify key local regions while suppressing irrelevant background content. On this basis, we further aggregate object prototypes into event prototypes so that the resulting prototypes represent not only within-frame semantics but also the temporal continuity of basic action units. This two-stage prototype representation forms a coherent structure that bridges local and temporal semantics, thereby providing a more stable and explicit foundation for cross-modal alignment. Figure 2 shows that our method progressively constructs cross-modal prototypes across multiple semantic levels, which improves the model’s ability to capture complex semantic structures. Specifically, the spatial prototype module detects semantically salient local regions to form object prototypes, such as a man and a baby. This process reduces visual noise and strengthens fine-grained alignment between visual features and textual phrases. Furthermore, object prototypes are aggregated into event prototypes to represent cross-frame semantics, such as drawing on a pumpkin and carving with a knife, which correspond to complete action sequences or sentence-level semantic units.

Figure 1. Comparison between global alignment and prototype alignment for video temporal grounding. (a) Global alignment assigns similarity scores based on holistic video–sentence representations, which may incorrectly favor visually similar but semantically mismatched candidate moments. The red multiplication sign (×) indicates an incorrect candidate moment, arrows denote similarity matching relationships, and different colors represent different similarity values. (b) Prototype alignment decomposes the sentence into object- and action-related semantic units and aligns them with localized visual regions, enabling fine-grained correspondence between textual phrases and video content and leading to more reliable moment discrimination.

Figure 1. Comparison between global alignment and prototype alignment for video temporal grounding. (a) Global alignment assigns similarity scores based on holistic video–sentence representations, which may incorrectly favor visually similar but semantically mismatched candidate moments. The red multiplication sign (×) indicates an incorrect candidate moment, arrows denote similarity matching relationships, and different colors represent different similarity values. (b) Prototype alignment decomposes the sentence into object- and action-related semantic units and aligns them with localized visual regions, enabling fine-grained correspondence between textual phrases and video content and leading to more reliable moment discrimination.

In addition, existing approaches typically construct candidate moment representations by pooling or convolutional aggregation of frame features across the candidate interval, a design that often struggles to emphasize key semantic regions in the presence of background redundancy or loosely related motions. By contrast, we treat the spatio-temporal prototype matching scores as semantic weights and perform weighted fusion across temporal positions within each candidate interval, thereby aligning the representation construction process with the cross-modal alignment results. The resulting moment representations place greater emphasis on temporal regions relevant to the query, providing more informative inputs for boundary prediction and alleviating the lack of semantic selection in conventional moment representation learning.

The main contributions of this work are summarized as follows:

• We propose a hierarchical prototype alignment method that decomposes cross-modal alignment into two complementary stages, namely object-phrase alignment and event-sentence alignment, thereby improving alignment accuracy from both spatial and temporal perspectives.
• We develop two prototype aggregation networks to model spatial and temporal information in videos, enabling the framework to capture fine-grained local details and diverse semantic patterns.
• Instead of constructing moment candidates through pooling or stacked convolution, we introduce an alignment result to construction strategy that emphasizes temporally localized regions most relevant to the textual semantics.
• Experiments on Charades-STA, ActivityNet Captions, and TACoS demonstrate that the proposed method outperforms existing approaches across multiple evaluation metrics.

Figure 2. Illustration of the hierarchical prototype alignment process. Object prototypes are first constructed by attending to spatial regions within frames, capturing fine-grained entity cues. These object cues are then progressively aggregated along the temporal dimension to form event prototypes, which model action evolution and support sentence-level semantic alignment. Different colors represent different prototype groups, numbers denote prototype indices, arrows indicate the information flow between components, and ellipses represent intermediate prototypes.

Figure 2. Illustration of the hierarchical prototype alignment process. Object prototypes are first constructed by attending to spatial regions within frames, capturing fine-grained entity cues. These object cues are then progressively aggregated along the temporal dimension to form event prototypes, which model action evolution and support sentence-level semantic alignment. Different colors represent different prototype groups, numbers denote prototype indices, arrows indicate the information flow between components, and ellipses represent intermediate prototypes.

2. Related Work

2.1. Video Temporal Grounding

A central challenge in video temporal grounding lies in cross-modal alignment, which requires mapping the video stream to a natural language query in a semantically consistent manner over time. A common line of work formulates localization as global semantic matching by embedding the entire video and the complete sentence into a shared space and using video-sentence similarity to select the target segment. Methods including VSLNet [21], 2D-TAN [19], GDP [32], CPN [23], and DCM [33] generally rely on global video representations or candidate semantic fusion. These approaches typically encode the video as a whole and then score moment candidates in a uniform manner. Such strategies can be effective when the overall semantics are highly consistent. However, natural language queries often describe only a local event or a key action. As a result, global alignment inevitably introduces query-irrelevant visual content and may lead to semantic drift in long videos or complex scenes.

To mitigate the interference introduced by global modeling, some studies incorporate finer temporal modeling or weighting mechanisms within global frameworks. For example, multi-scale designs, cross-modal attention, and candidate enhancement strategies have been adopted to strengthen responses in target regions. MMN [20], PS-VTG [34], ViGA [35], D-TSG [36], and D3G [37] extend this direction by strengthening video-text interaction or introducing structured reasoning mechanisms, which improves localization accuracy to some extent. However, in these approaches cross-modal alignment is often absorbed into fusion or prediction modules rather than being explicitly modeled in terms of the visual structures referred to by the text. Consequently, their ability to represent complex semantic composition remains limited.

With continued progress, recent research increasingly emphasizes fine-grained cross-modal alignment by linking local visual content to discrete semantic units in language. Methods such as LGI [25], MAT [38], and DRN [39] introduce interaction mechanisms between frame-level or region-level cues and textual phrases, enabling the model to capture the visual evidence referred to by the query more precisely. Later works including LCNet [40], CBLN [24], and SeqPAN [27] further strengthen the temporal consistency of local semantic alignment by improving action boundary modeling through hierarchical designs or sequence-aware mechanisms. Recent studies such as MGSL [28], PFU [41], VDI [42], and MESM [26] refine local alignment through frame-level supervision, modality balancing, or semantic enrichment, which improves robustness under challenging conditions. Meanwhile, recent studies have begun exploring large language model (LLM)-based paradigms for multimodal temporal understanding and reasoning, such as PromptCC [43] and Change-Agent [44], which leverage pretrained language models to facilitate semantic interpretation and reasoning over temporal imagery.

Although local alignment approaches substantially improve sensitivity to fine-grained semantics, most existing studies focus on alignment at a single level, such as frame-to-word or region-to-phrase correspondences, and lack a systematic characterization of video semantics spanning from local entities to temporal events. When a query jointly involves objects, actions, and their temporal relations, relying on a single level of local alignment often fails to provide stable and coherent cross-modal support. A key question in video temporal grounding is how to retain local precision while introducing an intermediate representation that captures semantic hierarchy and enables complementary alignment across multiple levels. To address this issue, we propose hierarchical prototype alignment to strengthen cross-modal consistency between video and text.

2.2. Prototype Learning

Prototype learning traces its origins to early developments in machine learning and cognitive psychology, where the central idea is to represent each class with a prototypical exemplar and perform inference based on distance metrics between the input and these prototypes [45,46,47]. In the domains of few-shot and zero-shot learning, prototype-based approaches have attracted significant attention due to their strong generalization capability [48,49]. For example, Gao et al. [50] introduced a hybrid-attention prototype network for relation classification with noisy labels, and Zhao et al. [51] proposed a knowledge-augmented prototype architecture to improve few-shot event detection. These studies demonstrate that it improves the interpretability and consistency of semantic alignment.

More recently, prototype learning has been increasingly adopted in multimodal contexts, showing strong capability in modeling structured semantics and facilitating cross-modal alignment. In the context of event relation extraction, Hu et al. [52] proposed the Prototype-Enhanced Matching (ProtoEM) framework, which models prototype nodes for event types and encodes their semantic dependencies through graph neural networks, thereby improving the recognition of structured event relations. Huang et al. [53] addressed few-shot action recognition by introducing a compound prototype matching method, where videos are represented as a set of compound prototypes, including both global and local prototypes. Their method integrates one-to-one and bipartite matching to enhance temporal alignment at multiple scales. For video-text retrieval, Lin et al. [29] proposed a Text-Adaptive Multiple Visual Prototype Matching framework, in which multiple visual prototypes are learned to represent diverse video semantics. The model performs text-adaptive matching by selecting the most relevant prototype for each query, and a variance loss is introduced to encourage prototype diversity.

Although these studies demonstrate the effectiveness of prototype-based modeling in multimodal tasks, its application to video temporal grounding remains relatively limited, particularly in integrating hierarchical semantic structures with temporal localization. Moreover, in most existing approaches, prototypes are mainly used to improve representation learning or similarity matching, serving as intermediate semantic abstractions that facilitate feature alignment across modalities. In contrast, the proposed framework incorporates hierarchical prototype alignment directly into the temporal grounding process. In this work, we extend prototype modeling to video temporal grounding by proposing a hierarchical prototype alignment method.

3. Methodology

This work aims to strengthen cross-modal alignment between video and text, and the overall framework is illustrated in Figure 3. The framework consists of a multimodal backbone and a two-stage prototype alignment module. The backbone comprises a video encoder and a text encoder that extract visual and textual representations in a shared semantic space. Building on these representations, we further generate object and event prototypes as intermediate representations across semantic levels, allowing alignment to progress from local entities to temporal event units.

3.1. Problem Definition

Given an untrimmed video V and a natural language query sentence S, video temporal grounding (VTG) aims to localize a temporal interval E in V whose visual content is most semantically consistent with S. The query sentence S is represented as a sequence of words $S={\left\{s_i\right\}}_{i=0}^{W-1}$, where $s_i$ denotes the i-th word and W is the total number of words in the sentence. The video V is represented as a sequence of frames $V={\left\{x_j\right\}}_{j=0}^{L-1}$, where $x_j$ denotes the j-th frame and L denotes the total number of frames. The target moment is denoted as $E=[t_s,t_e]$, where $t_s$ and $t_e$ represent the start and end timestamps, respectively. Predicting E can therefore be viewed as a cross-modal alignment problem in which the model identifies video evidence that corresponds to the semantics expressed in S.

Figure 3. Overview of the proposed hierarchical prototype alignment framework. We decompose cross-modal alignment into two complementary components: (1) Object-Phrase Prototype Matching (OPPM), which aligns visual object prototypes and textual phrase prototypes generated by Spatial Prototype Generation (SPG) to emphasize fine-grained spatial semantics; (2) Event-Sentence Prototype Matching (ESPM), which exploits event prototypes progressively generated by Temporal Prototype Generation (TPG) to model temporal action evolution and establish coherent alignment with sentence-level semantics. The resulting matching scores are further integrated by the grounding head for temporal boundary prediction. Overlapping elements represent multiple video frames in the sequence.

Figure 3. Overview of the proposed hierarchical prototype alignment framework. We decompose cross-modal alignment into two complementary components: (1) Object-Phrase Prototype Matching (OPPM), which aligns visual object prototypes and textual phrase prototypes generated by Spatial Prototype Generation (SPG) to emphasize fine-grained spatial semantics; (2) Event-Sentence Prototype Matching (ESPM), which exploits event prototypes progressively generated by Temporal Prototype Generation (TPG) to model temporal action evolution and establish coherent alignment with sentence-level semantics. The resulting matching scores are further integrated by the grounding head for temporal boundary prediction. Overlapping elements represent multiple video frames in the sequence.

3.2. Preparation

In this work, a unified and stable semantic space provides the foundation for prototype formation and cross-modal alignment. Accordingly, we first specify how video and text are represented, and then describe the clip segmentation scheme and the feature encoding process of the backbone. We divide the untrimmed video V into N fixed-length clips ${\left\{v_i\right\}}_{i=0}^{N-1}$, denoted by $V={\left\{v_i\right\}}_{i=0}^{N-1}$. Each clip $v_i$ consists of T consecutive frames, expressed as $v_i={\left\{x_j\right\}}_{j=t_i}^{t_i+T-1}$, where $x_j$ is the j-th frame and $t_i$ is the starting frame index of the i-th clip. Since key actions may occur near the boundary between adjacent clips, we adopt a frame-overlap strategy to avoid truncating critical information, keeping O overlapping frames such that $t_{i+1}=t_i+(T-O)$. With this segmentation, the number of clips N is computed as

$$N=⌊{\displaystyle \frac{L-O}{T-O}}⌋+1.$$

(1)

If the tail segment contains fewer than T frames, we use zero padding to enforce a fixed clip length, which keeps the feature dimensions consistent for later encoding and prototype alignment. The N sampled clips serve as the basic units for constructing moment candidates.

In the feature-encoding stage, we adopt CLIP [54] as the multimodal backbone, which establishes a shared semantic space for vision and text via cross-modal contrastive learning. For each video segment $v_i$, we uniformly sample K frames and feed them into the CLIP visual encoder to obtain the visual feature sequence:

$${\mathit{X}}_k={\left\{{\mathit{x}}_c^k,{\mathit{x}}_{p_1}^k,\dots ,{\mathit{x}}_{p_M}^k\right\}}_{k=1}^K\in {\mathbb{R}}^{K\times (M+1)\times D}.$$

(2)

Here, ${\mathit{x}}_c^k$ denotes the global token of the k-th frame and summarizes its overall semantics. The set ${\left\{{\mathit{x}}_{p_m}^k\right\}}_{m=1}^M$ represents the local features of the frame, consisting of M patch tokens that preserve fine-grained visual cues such as object shape and local motion. After encoding $v_i$, the resulting visual representation captures both global semantics and local structure, serving as the visual foundation for cross-modal alignment. For the query text $S={\left\{s_i\right\}}_{i=0}^{W-1}$, we extract semantic features using the CLIP text encoder. We prepend [SOT] and append [EOT] to mark the sequence boundaries, and represent the encoded feature sequence as:

$$\mathit{Y}=\left\{{\mathit{y}}_S,{\mathit{y}}_{s_1},\dots ,{\mathit{y}}_{s_W},{\mathit{y}}_E\right\}\in {\mathbb{R}}^{(W+2)\times D},$$

(3)

where ${\mathit{y}}_E$ is the global text token feature [EOT], and W and D denote the number of words and the feature dimension, respectively. Beyond the backbone, we decompose the alignment process into two complementary spatio-temporal components. Object-phrase prototype alignment aligns object prototypes from the spatial prototype generator with phrase prototypes, focusing the model on fine-grained spatial evidence. Event-sentence prototype alignment leverages event prototypes progressively generated by the temporal prototype generator to learn dynamic semantic alignment and capture the intrinsic one-to-many relation between video and text. The matching scores from the object-phrase and event-sentence pathways reflect the overall quality of cross-modal alignment.

3.3. Object-Phrase Prototype Alignment

In video temporal grounding, the query text typically describes events within a video segment, and semantic elements such as objects and actions often correspond to local regions in the video. These local regions contain key visual cues for cross-modal alignment, but they are also mixed with substantial background distractions and irrelevant details. Relying only on global features or simple frame-level alignment makes it difficult to establish stable cross-modal correspondences. Therefore, we construct object prototypes and phrase prototypes as spatial prototypes to capture semantic consistency among visual object instances, action-related regions, and textual phrase representations. This section consists of two parts. We first construct video object prototypes and text phrase prototypes, and then evaluate cross-modal alignment based on their semantic similarity.

3.3.1. Spatial Prototype Generation

After the video segment is encoded by the visual encoder in the previous subsection, the resulting visual feature sequence contains a global token and local patch tokens. We aggregate patch tokens into object prototypes that represent fine-grained spatial content, such as object instances, parts, and action-relevant regions. Not every patch should contribute to a prototype, since background patches may carry spatial signals but often introduce noise for cross-modal alignment. Therefore, we adopt sparse aggregation to filter redundant information and generate spatial prototypes. The architecture of the Spatial Prototype Generator (SPG) is shown in Figure 4. In the spatial prototype generator, to alleviate training instability caused by feature scale differences across frames, we first apply layer normalization to the k-th frame feature sequence ${\mathit{X}}_k\in {\mathbb{R}}^{(M+1)\times D}$:

$${\widehat{\mathit{X}}}_k=\mathrm{LayerNorm}\left({\mathit{X}}_k\right).$$

(4)

Then, based on the normalized features ${\widehat{\mathit{X}}}_k$, we use a two-layer fully connected network with ReLU to predict the aggregation weight matrix ${\mathbf{W}}_o^k$ for mapping spatial positions to object prototypes:

$${\mathbf{W}}_o^k={\mathbf{W}}_2\left(\mathrm{ReLU}\left({\mathbf{W}}_1{\widehat{\mathit{X}}}_k+b_1\right)\right)+b_2\in {\mathbb{R}}^{(M+1)\times N_o}.$$

(5)

Here, $N_o$ denotes the preset number of object prototypes. To encourage each prototype to concentrate on a few highly relevant patches and to keep weights comparable across frames, we normalize the scores with Softmax along the spatial dimension, producing the sparse aggregation matrix ${\mathit{A}}_k$:

$${\mathit{A}}_k=\mathrm{Softmax}\left({\mathbf{W}}_o^k\right)\in {\mathbb{R}}^{(M+1)\times N_o}.$$

(6)

In Equation (6), $\mathrm{Softmax}$ normalizes each column j along the spatial index i, ensuring that ${\sum }_{i=1}^{M+1}{\left[{\mathit{A}}_k\right]}_{ij}=1.$ The j-th column can be viewed as a probability distribution over spatial positions (patches) for the j-th object prototype, indicating which patches contribute most to the aggregation. This design encourages each prototype to be supported by a few high-weight patches, thereby suppressing redundant content. Finally, object prototypes are obtained by a weighted aggregation of visual features:

$${\mathit{P}}_o^k={\left({\mathit{A}}_k\right)}^{\top }·{\mathit{X}}_k\in {\mathbb{R}}^{N_o\times D}.$$

(7)

As a result, each object prototype is a D-dimensional vector that represents a spatial semantic unit aggregated from several key patches in the k-th frame. Ideally, each object prototype can adaptively aggregate patches associated with the corresponding object or action.

Figure 4. Framework of the Spatial Prototype Generation (SPG) module. Given patch tokens from different frames, SPG learns frame-adaptive attention weights through a lightweight feed-forward network to aggregate informative patches into object prototypes. The generated prototypes emphasize query-relevant spatial regions while suppressing background noise, providing compact and discriminative representations for subsequent object-phrase prototype matching.

Figure 4. Framework of the Spatial Prototype Generation (SPG) module. Given patch tokens from different frames, SPG learns frame-adaptive attention weights through a lightweight feed-forward network to aggregate informative patches into object prototypes. The generated prototypes emphasize query-relevant spatial regions while suppressing background noise, providing compact and discriminative representations for subsequent object-phrase prototype matching.

For phrase prototypes, since a query contains multiple semantic components, different phrases often correspond to different visual objects or action regions. Therefore, it is necessary to extract referential semantic units from the word sequence rather than aligning visual features with the entire sentence. To this end, we adopt a sparse aggregation strategy analogous to that used on the visual side and design a similar network to aggregate the text feature sequence, producing phrase prototypes ${\mathit{P}}_p\in {\mathbb{R}}^{N_p\times D}$, where $N_p$ denotes the number of phrase prototypes. Phrase prototypes transform a natural language sequence into a set of localized semantic units. Both object and phrase prototypes are dynamically instantiated from the current visual and textual features through sparse aggregation, rather than maintained as static prototype parameters. During training, the prototype representations are recomputed in each forward pass according to the updated aggregation weights, allowing the prototypes to adapt to the semantic content of each input sample.

3.3.2. Object-Phrase Matching

Object-phrase alignment seeks object prototypes in a clip whose semantics most strongly coincide with a phrase prototype across frames. To measure such correspondence, for each frame k we construct an object-phrase prototype similarity matrix using dot-product similarity:

$${\mathbf{M}}^k={\mathit{P}}_p·{\left({\mathit{P}}_o^k\right)}^{\top }\in {\mathbb{R}}^{N_p\times N_o}.$$

(8)

Each element ${\left[{\mathbf{M}}^k\right]}_{ij}$ measures the similarity between phrase prototype i and object prototype j at the k-th frame. For a fixed object index j and frame index k, we retain only the most semantically aligned phrase prototype by taking a maximum over the phrase dimension:

$${\mathbf{m}}_j^k=\underset{1\le i\le N_p}{max}{\left[{\mathbf{M}}^k\right]}_{ij}.$$

(9)

Here, ${\mathbf{m}}_j^k$ captures the best phrase match of object prototype j at frame k; it suppresses weakly related phrases and prevents irrelevant phrases from lowering the matching score. Since objects often persist over time, the same semantic entity may exhibit high similarity to the same phrase across multiple frames. Accordingly, for each object prototype we aggregate its per-frame best match scores along the temporal axis and take the maximum response over the entire clip:

$${\mathbf{m}}_j=\underset{1\le k\le K}{max}{\mathbf{m}}_j^k.$$

(10)

We scan the segment to identify, for each object prototype, its single strongest alignment response to the set of phrase prototypes. Equivalently, a high match to any phrase in any frame is sufficient to treat the object as mentioned by the query, thereby attenuating the impact of occlusion, motion blur, and transitional dynamics in the remaining frames. Finally, we average the maximal responses of all object prototypes to obtain the object-phrase matching score for the video segment:

$$S_{op}={\displaystyle \frac{1}{N_o}}\sum _{j=1}^{N_o}{\mathbf{m}}_j={\displaystyle \frac{1}{N_o}}\sum _{j=1}^{N_o}\underset{1\le k\le K}{max}\underset{1\le i\le N_p}{max}{\left[{\mathit{P}}_p·{\left({\mathit{P}}_o^k\right)}^{\top }\right]}_{ij}.$$

(11)

$S_{op}$ indicates whether each object prototype can find, in at least one frame within the segment, a phrase prototype that matches it semantically.

3.4. Event-Sentence Prototype Alignment

3.4.1. Temporal Prototype Generation

This section investigates how to organize the temporal evolution of object prototypes into a set of event prototypes with structural diversity while preserving local semantic resolution. The core idea is to progressively aggregate spatial object prototypes into frame-level prototypes and then perform temporal modeling on top of them to generate event prototypes with coherent semantics. The architecture of the Temporal Prototype Generator (TPG) is shown in Figure 5. We first design a frame decoder that aggregates the per-frame object prototypes ${\mathit{P}}_o^k\in {\mathbb{R}}^{(K\times N_o)\times D}$ into frame-level prototypes ${\mathit{p}}_f\in {\mathbb{R}}^{K\times D}$. The decoder assigns each frame a learnable query ${\mathit{Q}}_f\in {\mathbb{R}}^{K\times D}$, and uses linear projections to form the corresponding key ${\mathbf{K}}_o$ and value ${\mathbf{V}}_o$ for the object prototypes:

$${\mathbf{K}}_o={\mathit{P}}_o^k{\mathbf{W}}_o^K,{\mathbf{V}}_o={\mathit{P}}_o^k{\mathbf{W}}_o^V.$$

(12)

where ${\mathbf{W}}_o^K,{\mathbf{W}}_o^V\in {\mathbb{R}}^{D\times D}$ are learnable parameter matrices. To ensure that each frame query attends only to the object prototypes within the same frame, we construct a mask matrix ${\mathbf{M}}_f\in {\mathbb{R}}^{K\times (K\times N_o)}$ to restrict the attention scope, with elements defined as follows:

$${\mathbf{M}}_f(i,j)=\left\{\begin{array}{cc}0,\hfill & i·N_o\le j<(i+1)·N_o,\hfill \\ -\infty ,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(13)

With this mask, the frame-level prototypes are obtained through masked attention:

$${\mathit{p}}_f=\mathrm{Softmax}\left({\mathbf{M}}_f+{\mathit{Q}}_f{\mathbf{K}}_o^{\top }\right){\mathbf{V}}_o+{\mathit{Q}}_f.$$

(14)

Here, $\mathrm{Softmax}$ normalizes the weights row-wise, so that the frame query in row i distributes attention only among the $N_o$ object prototypes of the same frame, yielding a frame-level prototype vector that aggregates within-frame objects. Still, object-prototype aggregation alone can miss non-object signals, including scene context and background cues. Inspired by [55], we fuse the frame-level prototype with the global feature ${\mathit{x}}_c^k$ of the same frame by averaging to supplement scene-level semantic cues:

$${\mathit{p}}_f^k={\displaystyle \frac{{\mathit{p}}_f^k+{\mathit{x}}_c^k}{2}}.$$

(15)

To ensure that event prototypes capture continuity across adjacent frames rather than degenerating into isolated frame representations, we impose a 1D temporal convolution with kernel size 3 on the frame-level prototypes to encode local context among neighboring frames. The frame-level prototypes after temporal convolution are expressed as:

$${\tilde{\mathit{p}}}_f={\mathrm{Conv}}_{\mathrm{temp}}\left({\mathit{P}}_f\right)\in {\mathbb{R}}^{K\times D}.$$

(16)

Since the receptive field of the 1D convolution covers only the current frame and several neighboring frames, it introduces local temporal context without compromising frame-level details. We then obtain the frame-level prototypes with local temporal modeling via a residual connection:

$${\widehat{\mathit{p}}}_f={\mathit{p}}_f+{\tilde{\mathit{p}}}_f.$$

(17)

After obtaining frame-level prototypes ${\widehat{\mathit{p}}}_f$ with local temporal context, we develop an event decoder to learn temporal relations among frame-level prototypes and generate diverse event prototypes ${\mathit{P}}_e\in {\mathbb{R}}^{N_e\times D}$ to represent rich action information in videos. We begin by applying linear projections to ${\widehat{\mathit{p}}}_f$ to obtain the keys ${\mathbf{K}}_f$ and values ${\mathbf{V}}_f$:

$${\mathbf{K}}_f={\widehat{\mathit{p}}}_f{\mathbf{W}}_f^K,{\mathbf{V}}_f={\widehat{\mathit{p}}}_f{\mathbf{W}}_f^V,$$

(18)

where ${\mathbf{W}}_f^K,{\mathbf{W}}_f^V\in {\mathbb{R}}^{D\times D}$ are learnable projection matrices. We then compute event prototypes using scaled dot-product attention:

$${\mathit{P}}_e=\mathrm{Softmax}\left({\displaystyle \frac{{\mathit{Q}}_e{\mathbf{K}}_f^{\top }}{\sqrt{D}}}\right){\mathbf{V}}_f+{\mathit{Q}}_e.$$

(19)

Here, ${\mathit{Q}}_e\in {\mathbb{R}}^{N_e\times D}$ denotes the event query vectors, and $N_e$ is the predefined number of event prototypes. During training, each query vector represents a potential event type, and using multiple queries promotes diversity among event prototypes. For the text modality, we directly use the global token feature ${\mathbf{y}}_E$ as the sentence prototype and align it with the event prototypes ${\mathit{P}}_e$. Similarly, the frame-level and event-level prototypes are dynamically generated from the current object prototypes and temporal features rather than stored as independent learnable representations. These prototypes are recomputed in each forward pass through attention-based aggregation parameterized by learnable query vectors, projection matrices, and temporal modeling modules such as temporal convolution and event decoding.

Figure 5. Framework of the Temporal Prototype Generation (TPG) module. Starting from object prototypes extracted at each frame, TPG first employs a frame decoder to construct frame-level prototypes. Temporal convolution is then applied to model inter-frame dependencies and capture action evolution over time. Finally, an event decoder aggregates temporally-aware features into event prototypes, which serve as semantic anchors for event-sentence prototype matching.

Figure 5. Framework of the Temporal Prototype Generation (TPG) module. Starting from object prototypes extracted at each frame, TPG first employs a frame decoder to construct frame-level prototypes. Temporal convolution is then applied to model inter-frame dependencies and capture action evolution over time. Finally, an event decoder aggregates temporally-aware features into event prototypes, which serve as semantic anchors for event-sentence prototype matching.

3.4.2. Event-Sentence Matching

The same video segment can be described by different texts, some emphasizing fine-grained action details while others focus on overall semantics. Restricting alignment to phrase- and object-level representations can sharpen fine-grained correspondence, but it often overlooks sentence-level intent, causing the predicted boundaries to drift from the global semantics of the query. To address this issue, we use the global sentence feature ${\mathit{y}}_E$ produced by the text encoder as the sentence prototype and match it with the event prototype set ${\left\{{\mathit{P}}_e^i\right\}}_{i=1}^{N_e}$ to quantify event-level alignment. Because events differ in their semantic roles and only a few correspond clearly to the actions or states described in the query, a simple average over event prototypes can be dominated by irrelevant events, shifting the result away from the true target interval. To mitigate this effect, we define the event-sentence matching score using the maximum similarity:

$$S_{es}=\underset{1\le i\le N_e}{max}〈{\mathit{y}}_E,{\mathit{P}}_e^i〉.$$

(20)

Here, $\langle ·,·\rangle$ denotes the inner product that measures the alignment between the sentence prototype and the i-th event prototype in the semantic space, and $N_e$ denotes the preset number of event prototypes. In contrast to weighted averaging over all event prototypes, the maximum similarity score directs the model toward the principal event that best explains the query instead of distributing evidence across multiple events. Ultimately, the event-sentence prototype matching score $S_{es}$ is regarded as the event-level alignment between the video segment and the query, guiding the model to attend to the interval that is most semantically consistent with the text along the timeline. Event-sentence matching provides temporal alignment and, together with object-phrase matching in the spatial dimension, forms a two-stage cross-modal alignment process from local semantics to global semantics.

3.5. Fusion of Matching Results

In the preceding alignment stage, the model derives per-clip matching scores along the spatial and temporal dimensions, and these two scores encode cross-modal alignment evidence at different semantic levels. However, the relative importance of spatial details and temporal structures varies depending on the query. For example, when a query emphasizes the actor or salient objects, spatial prototype alignment becomes more influential, whereas queries describing event ordering or action dynamics rely more on temporal prototype alignment. Therefore, a key step for accurate temporal grounding is to couple the two dimensions so that the final clip-level score can adaptively reflect the semantic focus of the query. Motivated by this observation, we design a dynamic fusion mechanism to integrate the two matching scores. Specifically, we first stack the object-phrase matching score $S_{op}$ and the event-sentence matching score $S_{es}$ to form a two-dimensional input vector, and then map it to a high-dimensional representation ${\mathbf{H}}_m$ through a learnable fully connected layer, thereby enhancing the model’s ability to exploit complementarity across semantic levels. This process is formulated as follows:

$${\mathbf{H}}_m=\mathrm{ReLU}\left({\mathbf{W}}_m\left[\begin{array}{c}{S}_{op},S_{es}\end{array}\right]+b_m\right).$$

(21)

The projected representation is then passed through a linear transform followed by Softmax normalization, yielding two adaptive weights $\alpha$ and $\beta$:

$$\left[\begin{array}{c}\alpha ,\beta \end{array}\right]=\mathrm{Softmax}\left({\mathbf{W}}_a{\mathbf{H}}_m+{\mathit{b}}_a\right).$$

(22)

Here, $\alpha$ and $\beta$ quantify the relative contributions of spatial and temporal prototype matching, respectively. This dynamic weighting scheme, built on high-dimensional feature mapping, allows the model to learn the relative salience of different semantic levels during training and adjust the contributions of the two matching modules accordingly, improving robustness across diverse video-query settings. Given the learned weights, the composite score for each clip $v_i$ is defined as:

$$S_{\mathrm{match}}\left(v_i\right)=\alpha S_{op}\left(v_i\right)+\beta S_{es}\left(v_i\right).$$

(23)

This dynamic fusion enables the model to adapt the spatial-versus-temporal semantic contributions to each specific video-query pairing.

3.6. Grounding Head

3.6.1. Moment Candidates Generation

In video temporal grounding, constructing candidate moment features is a prerequisite for boundary prediction, and the quality of these representations directly determines the upper bound of candidate moment scores in the subsequent two-dimensional temporal map. Following Section 3.2, we segment the input video into N consecutive clips ${\left\{v_i\right\}}_{i=0}^{N-1}$, and candidate segments $[a,b]$ are subsequently formed by pairing start and end indices within this clip sequence. Prior work commonly constructs candidate moment features using max pooling [7,8] or stacked convolutions [56]. However, candidate moment features generated by these paradigms are not explicitly aligned with the query semantics, which limits the accuracy of subsequent localization. In general, video clips that better match the query semantics are more likely to fall within the ground-truth moment and should therefore receive larger weights when constructing candidate segment features. Conversely, clips that correspond to background content or are semantically irrelevant should be suppressed. To this end, we use the composite matching score $S_{\mathrm{match}}\left(v_i\right)$ as a cross-modal semantic weight to guide candidate moment feature construction.

We first build a unified visual representation for each clip. Using average pooling, we aggregate the set of K sampled global token features ${\left\{{\mathit{x}}_c^k\right\}}_{k=1}^K$ from clip $v_i$ into a clip-level feature ${\mathbf{f}}_i^v\in {\mathbb{R}}^D$:

$${\mathbf{f}}_i^v={\displaystyle \frac{1}{K}}\sum _{k=1}^K{\mathbf{x}}_c^k,i=0,1,\dots ,N-1.$$

(24)

We then treat ${\left\{{\mathbf{f}}_i^v\right\}}_{i=0}^{N-1}$ as the primitive elements from which we build the feature map of all candidate segments. For a candidate interval $[a,b]$, we apply Softmax normalization to the composite scores of clips within the interval to obtain cross-modal semantic weights ${\mathbf{w}}_i$:

$${\mathbf{w}}_i={\displaystyle \frac{exp\left(S_{\mathrm{match}}\left(v_i\right)\right)}{{\sum }_{j=a}^{b}exp\left(S_{\mathrm{match}}\left(v_j\right)\right)}},i\in [a,b].$$

(25)

The weights encode how much each clip contributes to the query-relevant semantics for a fixed candidate moment. Based on these weights, we form the visual representation of a candidate moment $[a,b]$, denoted as ${\mathbf{f}}_{a,b}^M$, by a weighted aggregation of all clip features inside the interval:

$${\mathbf{f}}_{a,b}^M=\sum _{i=a}^b{\mathbf{w}}_i{\mathbf{f}}_i^v.$$

(26)

Here, a and b are the start and end indices of a candidate moment, satisfying $0\le a\le b\le N-1$. Relative to stacked convolutions, this construction selectively amplifies text-consistent clips and attenuates query-irrelevant noise. In contrast to max pooling, the weighted aggregation better preserves temporal continuity within the candidate moment. Given the candidate moment features, we follow the [19] formulation and arrange them into a two-dimensional temporal feature map ${\mathbf{F}}^M\in {\mathbb{R}}^{N\times N\times D}$, whose first two axes represent segment start and end locations and whose third axis is the feature dimension. Since valid candidates satisfy $a\le b$, we zero-pad entries that fall into the region where $a>b$ in practice.

3.6.2. Answer Prediction

After constructing candidate segment features, we predict the best-matching moment for the query sentence among all candidates. The pipeline proceeds in three stages: cross-modal fusion, contextual modeling, and score prediction. First, we fuse the two-dimensional temporal feature map ${\mathbf{F}}^M$ with the query representation ${\mathbf{f}}^s={\mathbf{y}}_E\in {\mathbb{R}}^D$. Since the features are extracted by CLIP in a shared multimodal embedding space, we perform cross-modal fusion via a Hadamard product followed by Frobenius norm normalization, yielding the fused map $\mathbf{F}$:

$$\mathbf{F}={∥\left({\mathbf{f}}^s·{\mathbf{1}}^{\top }\right)\odot {\mathbf{F}}^M∥}_F.$$

(27)

Here, ${\mathbf{1}}^{\top }$ denotes the transpose of an all ones vector, ⊙ is the Hadamard product, and ${∥·∥}_F$ indicates Frobenius normalization. Next, we build a temporal adjacent network on the fused map $\mathbf{F}$ to produce a two-dimensional temporal map. The network contains $L_c$ convolutional layers, each with kernel size $K_c$, and is defined as:

$${\mathbf{F}}^{(l+1)}=\mathrm{ReLU}\left(\mathrm{Conv}2{\mathrm{D}}^{\left(l\right)}\left({\mathbf{F}}^{\left(l\right)}\right)\right),l=0,1,\dots ,L_c-1,$$

(28)

where the network progressively incorporates richer context from adjacent segments. Finally, the output feature map is fed into a fully connected layer followed by a sigmoid function to produce a two-dimensional score map. We gather the valid entries of the score map into a set $P={\left\{p_i\right\}}_{i=1}^{C_n}$, where $C_n$ denotes the number of candidate moment. Each $p_i$ measures the consistency between a candidate moment and the query sentence, and the maximum score corresponds to the best-matched moment.

3.7. Training Objective

To jointly optimize cross-modal semantic alignment and temporal boundary prediction, we adopt a combination of a cross-modal alignment loss and a binary cross-entropy loss during training. The former enforces semantic consistency between video and text in a shared embedding space, while the latter guides the model to learn accurate temporal boundary prediction. In this process, the prototype representations are dynamically constructed from the current visual and textual features rather than stored as independent trainable parameters. Specifically, the object, phrase, frame-level, and event-level prototypes are recomputed in each forward pass through aggregation operations parameterized by learnable components, including aggregation networks, query vectors, projection matrices, and temporal modeling modules. All learnable components are initialized using standard random initialization, and the prototype representations gradually emerge from the aggregation of input features during early training stages. As optimization proceeds, the prototype structures are updated through end-to-end backpropagation under the supervision of the alignment and boundary prediction losses. To regularize the learning of prototype representations during training, several normalization and structural constraints are incorporated into the prototype construction process. These include layer normalization applied to input features to control feature scale, Softmax-based sparse aggregation to constrain prototype assignment distributions, masked attention to restrict cross-frame interference, and residual temporal modeling to stabilize temporal feature propagation. Together, these operations act as implicit regularization mechanisms that constrain the feature aggregation behavior and prevent unstable prototype assignments during optimization.

3.7.1. Cross-Modal Alignment Loss

In the prototype alignment stage, the model computes a composite matching score $S_{\mathrm{match}}\left(v_i\right)$ for each video clip $v_i$ based on the spatial prototype alignment score $S_{op}$ and the temporal prototype alignment score $S_{es}$. This score reflects the alignment strength between the clip and the query text in the shared semantic space. We optimize $S_{\mathrm{match}}$ using a contrastive learning objective. To construct positive and negative sample sets, we compare the temporal interval of clip $v_i$, denoted as $\mathrm{Time}\left(v_i\right)=[i·\delta ,(i+1)·\delta ]$, with the ground-truth $\mathrm{GT}=[g_s,g_e]$. If $\mathrm{IoU}(\mathrm{Time}\left(v_i\right),\mathrm{GT})\ge \theta$, the clip is treated as a positive sample; otherwise, it is treated as a negative sample. The positive set $\mathcal{P}$ and negative set $\mathcal{N}$ are defined as:

$$\mathcal{P}=\{\mathrm{IoU}(\mathrm{Time}\left(v_i\right),\mathrm{GT})\ge \theta \},\mathcal{N}=\{\mathrm{IoU}(\mathrm{Time}\left(v_i\right),\mathrm{GT})<\theta \}.$$

(29)

Next, we apply a Softmax operation to normalize the composite matching scores over all clips into a confidence distribution $c_i$:

$$c_i={\displaystyle \frac{exp\left(S_{\mathrm{match}}\left(v_i\right)/\tau \right)}{{\sum }_{j=0}^{N-1}exp\left(S_{\mathrm{match}}\left(v_j\right)/\tau \right)}}.$$

(30)

Here, $\tau$ is a temperature parameter that controls the smoothness of the distribution. The confidence $c_i$ reflects the model’s belief that clip $v_i$ is semantically matched to the query text. Based on this distribution, the cross-modal alignment loss is defined as:

$${\mathcal{L}}_{\mathrm{Match}}=-{\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{i\in \mathcal{P}}log{c}_i=-{\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\sum _{i\in \mathcal{P}}log{\displaystyle \frac{exp\left(S_{\mathrm{match}}\left(v_i\right)/\tau \right)}{{\sum }_{j=0}^{N-1}exp\left(S_{\mathrm{match}}\left(v_j\right)/\tau \right)}}.$$

(31)

This objective strengthens the alignment structure by maximizing the normalized probability of positive segments, encouraging them to be closer to the query semantics in the cross-modal space while suppressing negatives by pushing them toward semantically distant regions.

3.7.2. Binary Cross-Entropy Loss

To supervise boundary probabilities on the two-dimensional score map, we follow 2D-TAN [19] and use binary cross-entropy as the primary supervision signal. Concretely, for each candidate moment, we denote its temporal overlap with the ground truth as $o_i$ and linearly map it to a soft label $y_i$ using two predefined thresholds $t_{min}$ and $t_{max}$:

$$y_i=\left\{\begin{array}{cc}0,\hfill & o_i\le t_{min},\hfill \\ {\displaystyle \frac{o_i-t_{min}}{t_{max}-t_{min}}},\hfill & t_{min}<o_i<t_{max},\hfill \\ 1,\hfill & o_i\ge t_{max}.\hfill \end{array}\right.$$

(32)

This label reflects the semantic proximity between a candidate moment and the ground truth. Given the predicted confidence $p_i$ for each candidate moment on the two-dimensional score map, the binary cross-entropy loss is defined as:

$${\mathcal{L}}_{\mathrm{BCE}}=-{\displaystyle \frac{1}{C_n}}\sum _{i=1}^{C_n}{y}_{i}log{p}_i+(1-y_i)log(1-p_i).$$

(33)

By using soft labels, this loss provides coarse-to-fine temporal supervision, enabling the model to learn continuous boundary variation patterns across temporal scales and yielding smoother and more stable localization results. We adopt a weighted joint training strategy that combines the two losses into a unified optimization objective:

$${\mathcal{L}}_{\mathrm{Total}}={\mathcal{L}}_{\mathrm{BCE}}+\lambda {\mathcal{L}}_{\mathrm{Match}}.$$

(34)

Here, $\lambda$ is a balancing coefficient that controls the relative contribution of the semantic alignment term to the overall objective.

4. Experiments

4.1. Datasets

To comprehensively evaluate the effectiveness and generalizability of the proposed hierarchical prototype matching framework, we conduct experiments on three widely used benchmark datasets for video temporal grounding: Charades-STA, ActivityNet Captions, and TACoS. These datasets are commonly adopted in the literature, enabling rigorous and fair comparisons with existing approaches.

Charades-STA [7] is an extended version of the Charades action recognition dataset that is specifically designed for video temporal grounding. It contains 5338 videos and 12,408 query-moment pairs for training, as well as 1334 videos with 3720 query-moment pairs for testing. The original Charades dataset was recorded using RGB cameras in indoor environments, where participants perform scripted activities. A notable characteristic of Charades-STA is its high semantic density, as each video is associated with multiple overlapping or even conflicting queries. This property often leads to ambiguous event boundaries and poses challenges for precise cross-event alignment.

ActivityNet Captions [57] is a large-scale dataset comprising 19,209 videos and over 72,000 annotated video-sentence pairs. Specifically, it is divided into 37,417 training pairs, 17,505 validation pairs, and 17,031 testing pairs. Compared with Charades-STA, video segments in ActivityNet Captions are typically longer and cover a broader range of events with more complex semantic structures; individual sentences often describe multi-event sequences and nuanced temporal relations. These characteristics make the dataset suitable for evaluating the generalization ability and temporal reasoning capability of the proposed event-level prototypes.

TACoS [58], derived from the MPII Cooking Activities dataset, consists of 127 videos and 18,818 video-sentence pairs, partitioned into 10,146 for training, 4589 for validation, and 4083 for testing. Videos in TACoS are on average over four minutes long and exhibit highly similar event semantics (e.g., “cutting tomato” versus “cutting onion”), which makes fine-grained semantic discrimination particularly challenging. This property aligns well with the object-phrase prototype matching strategy in our framework and provides a valuable testbed for evaluating its ability to capture fine-grained semantic distinctions.

4.2. Experimental Settings

4.2.1. Evaluation Metrics

Following existing video temporal grounding works, we evaluate performance using two main metrics.

mIoU: This metric measures the mean Intersection-over-Union (IoU) between the predicted temporal intervals and the corresponding ground-truth intervals. By averaging IoU over all queries in the test set, mIoU reflects the overall localization accuracy of the model.

Recall: We adopt “$\mathrm{Rank}@\mathrm{n},\mathrm{IoU}=\mathrm{m}$”, as the evaluation metrics, following [7]. The “$\mathrm{Rank}@\mathrm{n},\mathrm{IoU}=\mathrm{m}$” represents the percentage of language queries having at least one result whose IoU between $top-n$ predictions with ground-truth is larger than m. In our experiments, we reported the results of $\mathrm{n}=1$ and $\mathrm{m}\in \left\{0.3,0.5,0.7\right\}$.

4.2.2. Implementation Details

Because the datasets differ substantially in event density, action duration, and scene transition pace, we do not adopt a single configuration for clip partitioning, frame overlap, or frame sampling. Instead, these settings are adapted to the characteristics of each dataset to ensure that prototypes are constructed on a stable temporal structure. Following previous work [19], we set the clip length to $T=8$ with no frame overlap ($O=0$) for Charades-STA, where events are relatively short and temporally discrete. For ActivityNet Captions, due to longer events and smoother temporal transitions, we use $T=16$ without frame overlap ($O=0$) to preserve semantic completeness within each clip. For TACoS, where event density is high and semantic continuity is strong, we set $T=16$ with substantial frame overlap ($O=12$, approximately $75\%$ overlap) to enhance the model’s ability to capture events that span across adjacent clips.

Within each clip, we uniformly sample $K=8$ frames. We adopt CLIP (ViT-B/32) [54] as the multimodal backbone and initialize it with pre-trained weights. Following the spatial and temporal modeling requirements described earlier, we set the numbers of object, phrase, and event prototypes to $N_o=16$, $N_p=12$, and $N_e=3$, respectively. All prototypes are 512-dimensional, and these hyperparameters are kept fixed across datasets to maintain architectural consistency. We train the model using the Adam optimizer [59] with a learning rate of $1\times {10}^{-4}$ and a batch size of 32 for 100 epochs, while setting the loss balancing coefficient $\lambda$ to 0.5. During inference, candidate scores are post-processed using non-maximum suppression (NMS) [60] with a threshold of 0.5 to remove redundant intervals and produce the final predictions.

All experiments were conducted using a workstation equipped with an NVIDIA RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA). The implementation was based on Python 3.8 and PyTorch 1.13.0 (Meta Platforms, Inc., Menlo Park, CA, USA). The video frames were extracted using FFmpeg (Version 4.3.1).

4.3. Comparison to State-of-the-Art Methods

4.3.1. Comparative Methods

We select a set of representative VTG methods as baselines and categorize them into two groups according to their cross-modal alignment strategies: global alignment and local alignment. Global alignment methods learn a shared semantic space between whole videos and complete sentences, typically via global representation modeling and cross-modal fusion. In contrast, local alignment methods emphasize fine-grained correspondence by matching local visual regions (or frame-level cues) with words/phrases to better capture complex actions.

• Global alignment methods: VSLNet [21], 2D-TAN [19], GDP [32], CPN [23], DCM [33], CI-MHA [61], MMN [20], PS-VTG [34], ViGA [35], D-TSG [36], D3G [37], and MRTNet [22].
• Local alignment methods: LGI [25], MAT [38], DRN [39], LCNet [40], CBLN [24], SeqPAN [27], MGSL [28], PFU [41], VDI [42], and MESM [26].

4.3.2. Quantitative Comparison

As shown by the comparative results in Table 1, Table 2 and Table 3, our method achieves strong and consistent performance across all three benchmark datasets. On ActivityNet Captions, our approach attains state-of-the-art results on the primary metrics. At $\mathrm{IoU}=0.7$, it improves upon D-TSG [36] by roughly two percentage points and further outperforms MAT [38] by about four percentage points. This trend is consistent with the characteristics of ActivityNet Captions: the captions are relatively detailed and often describe objects and actions with clear phrase boundaries. Such informative queries facilitate the formation of stable semantic links from local evidence to higher-level event semantics, making prototype alignment more effective when the textual description is sufficiently specific. The results on TACoS are also informative. At $\mathrm{IoU}=0.3$, our method outperforms CPN [23], a representative global-alignment approach. In terms of overall performance, it remains slightly below MESM [26], a strong local alignment method. Taken together, the results across the three benchmarks suggest that our method benefits most from informative queries with clear semantic structures. When the semantics are sparse, the performance margin narrows, while the method remains stable.

Table 1 shows that on Charades-STA our method remains competitive, but the advantage is less pronounced than on the other two benchmarks. This outcome aligns with the nature of Charades-STA, where indoor clips are visually homogeneous and temporally smooth, and the queries are brief with limited object vocabulary and weak descriptive modifiers. This sparsity in both the visual stream and the textual descriptions makes it difficult for object and phrase prototypes to extract representative semantic cues from local segments, and it also hinders event prototypes from forming robust representations when clear temporal-event boundaries are absent. Nevertheless, for candidate-segment construction, our method still yields a clear advantage over approaches such as 2D-TAN [19] and MMN [20] that rely on max-pooling to build video representations.

As shown in Table 2 and Table 3, our method outperforms global-alignment approaches such as D-TSG [36], D3G [37], CPN [23], and MRTNet [22] on multiple evaluation metrics. These methods typically rely on holistic representations over long temporal spans for cross-modal matching, which helps capture long-range semantic structures expressed in the query sentence. However, when alignment is dominated by global features, fine-grained differences among candidate segments tend to be attenuated, leaving limited representational capacity for local actions, short-term interactions, or subtle details. Consequently, global methods are more prone to misalignment when queries involve complex semantic structures, dense entities, or rapidly changing actions. In contrast, local-alignment methods such as SeqPAN [27], MAT [38], MGSL [28], and VDI [42] align sentences and videos within restricted temporal and spatial neighborhoods and can therefore capture fine details more sensitively. Yet these approaches commonly adopt a single-granularity matching strategy, for example aligning frames to words or local segments to the full sentence. When a query simultaneously refers to multiple entities and events, or when the video exhibits abrupt action changes, local features can become overly sensitive to noise and outliers, which degrades cross-modal alignment. Our method models alignment at two semantic granularities. Object-phrase prototypes constrain visual attention in the spatial dimension, focusing the model on query-relevant entities, while event-sentence prototypes further strengthen the correspondence between action trajectories and sentence semantics in the temporal dimension. This design yields more stable alignment for long and compositional queries.

Table 1. Comparison of our method with existing methods on the Charades-STA dataset.

Type	Method	Rank@1			mIoU
		$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Global Alignment	VSLNet [21]	70.46	54.19	35.22	50.02
	2D-TAN [19]	57.31	39.70	23.31	39.23
	GDP [32]	54.54	39.47	18.49	–
	CPN [23]	64.41	46.08	25.06	43.90
	DCM [33]	–	47.80	28.00	43.10
	CI-MHA [61]	69.87	54.68	35.27	–
	MMN [20]	–	47.31	27.28	–
	PS-VTG [34]	–	39.22	20.17	–
	ViGA [35]	60.22	36.72	17.20	38.62
	D-TSG [36]	–	65.05	42.77	–
	D3G [37]	–	41.64	19.60	–
	MRTNet [22]	59.23	44.27	25.88	40.59
Local Alignment	DRN [39]	–	42.90	23.68	–
	LCNet [40]	59.60	39.19	18.87	38.94
	CBLN [24]	–	43.67	24.44	–
	SeqPAN [27]	73.84	60.86	41.34	53.92
	MGSL [28]	–	63.98	41.03	–
	PFU [41]	71.57	54.66	28.34	48.65
	VDI [42]	–	46.47	28.63	41.60
	UniVTG [62]	70.81	58.01	35.65	50.10
	MS-DETR [63]	71.34	59.62	36.48	50.59
Ours		69.91	49.32	33.68	45.03

Table 1. Comparison of our method with existing methods on the Charades-STA dataset.

Type	Method	Rank@1			mIoU
		$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Global Alignment	VSLNet [21]	70.46	54.19	35.22	50.02
	2D-TAN [19]	57.31	39.70	23.31	39.23
	GDP [32]	54.54	39.47	18.49	–
	CPN [23]	64.41	46.08	25.06	43.90
	DCM [33]	–	47.80	28.00	43.10
	CI-MHA [61]	69.87	54.68	35.27	–
	MMN [20]	–	47.31	27.28	–
	PS-VTG [34]	–	39.22	20.17	–
	ViGA [35]	60.22	36.72	17.20	38.62
	D-TSG [36]	–	65.05	42.77	–
	D3G [37]	–	41.64	19.60	–
	MRTNet [22]	59.23	44.27	25.88	40.59
Local Alignment	DRN [39]	–	42.90	23.68	–
	LCNet [40]	59.60	39.19	18.87	38.94
	CBLN [24]	–	43.67	24.44	–
	SeqPAN [27]	73.84	60.86	41.34	53.92
	MGSL [28]	–	63.98	41.03	–
	PFU [41]	71.57	54.66	28.34	48.65
	VDI [42]	–	46.47	28.63	41.60
	UniVTG [62]	70.81	58.01	35.65	50.10
	MS-DETR [63]	71.34	59.62	36.48	50.59
Ours		69.91	49.32	33.68	45.03

Table 2. Comparison of our method with existing methods on the ActivityNet Captions dataset.

Type	Method	Rank@1			mIoU
		$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Global Alignment	2D-TAN [19]	59.45	44.51	26.54	43.29
	VSLNet [21]	63.16	43.22	26.16	43.19
	GDP [32]	56.17	39.27	–	39.80
	CI-MHA [61]	61.49	43.97	25.13	–
	CPN [23]	62.81	45.10	28.10	45.70
	DCM [33]	–	44.90	27.70	43.30
	D-TSG [36]	–	54.29	33.64	–
	MMN [20]	65.05	48.59	29.26	–
	PS-VTG [34]	59.71	39.59	21.98	–
	ViGA [35]	59.61	35.79	16.96	40.12
	D3G [37]	58.25	36.68	18.54	–
	MRTNet [22]	60.71	45.59	28.07	44.54
Local Alignment	LGI [25]	58.52	41.51	23.07	41.13
	DRN [39]	58.52	41.51	23.07	43.13
	SeqPAN [27]	61.65	45.50	29.37	45.11
	CBLN [24]	66.34	48.12	27.60	–
	MAT [38]	–	48.02	31.78	–
	LCNet [40]	48.49	26.33	–	34.29
	MGSL [28]	–	51.87	31.42	–
	VDI [42]	–	32.35	16.02	34.32
	PFU [41]	59.63	36.35	16.61	40.15
Ours		67.73	56.81	35.48	46.03

Table 2. Comparison of our method with existing methods on the ActivityNet Captions dataset.

Type	Method	Rank@1			mIoU
		$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Global Alignment	2D-TAN [19]	59.45	44.51	26.54	43.29
	VSLNet [21]	63.16	43.22	26.16	43.19
	GDP [32]	56.17	39.27	–	39.80
	CI-MHA [61]	61.49	43.97	25.13	–
	CPN [23]	62.81	45.10	28.10	45.70
	DCM [33]	–	44.90	27.70	43.30
	D-TSG [36]	–	54.29	33.64	–
	MMN [20]	65.05	48.59	29.26	–
	PS-VTG [34]	59.71	39.59	21.98	–
	ViGA [35]	59.61	35.79	16.96	40.12
	D3G [37]	58.25	36.68	18.54	–
	MRTNet [22]	60.71	45.59	28.07	44.54
Local Alignment	LGI [25]	58.52	41.51	23.07	41.13
	DRN [39]	58.52	41.51	23.07	43.13
	SeqPAN [27]	61.65	45.50	29.37	45.11
	CBLN [24]	66.34	48.12	27.60	–
	MAT [38]	–	48.02	31.78	–
	LCNet [40]	48.49	26.33	–	34.29
	MGSL [28]	–	51.87	31.42	–
	VDI [42]	–	32.35	16.02	34.32
	PFU [41]	59.63	36.35	16.61	40.15
Ours		67.73	56.81	35.48	46.03

Table 3. Comparison of our method with existing methods on the TACoS dataset.

Type	Method	Rank@1			mIoU
		$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Global Alignment	VSLNet [21]	29.61	24.27	20.03	24.11
	2D-TAN [19]	37.29	25.32	13.32	25.19
	GDP [32]	24.14	–	–	16.18
	CPN [23]	47.69	36.33	–	34.49
	D-TSG [36]	46.32	35.91	–	–
	PS-VTG [34]	23.64	10.00	3.35	–
	ViGA [35]	19.62	8.85	3.22	15.47
	MMN [20]	38.57	27.24	–	–
	D3G [37]	27.27	12.67	4.70	–
	MRTNet [22]	37.81	26.01	14.95	26.29
Local Alignment	DRN [39]	–	23.17	–	–
	CBLN [24]	38.98	27.65	–	–
	SeqPAN [27]	31.72	27.19	21.65	25.86
	MAT [38]	48.79	37.57	–	–
	MGSL [28]	42.54	32.27	–	–
	MESM [26]	52.69	39.52	–	36.94
	UniVTG [62]	51.44	34.97	17.35	33.60
	MS-DETR [63]	53.16	39.65	23.42	37.01
Ours		49.65	34.77	21.86	33.89

Table 3. Comparison of our method with existing methods on the TACoS dataset.

Type	Method	Rank@1			mIoU
		$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Global Alignment	VSLNet [21]	29.61	24.27	20.03	24.11
	2D-TAN [19]	37.29	25.32	13.32	25.19
	GDP [32]	24.14	–	–	16.18
	CPN [23]	47.69	36.33	–	34.49
	D-TSG [36]	46.32	35.91	–	–
	PS-VTG [34]	23.64	10.00	3.35	–
	ViGA [35]	19.62	8.85	3.22	15.47
	MMN [20]	38.57	27.24	–	–
	D3G [37]	27.27	12.67	4.70	–
	MRTNet [22]	37.81	26.01	14.95	26.29
Local Alignment	DRN [39]	–	23.17	–	–
	CBLN [24]	38.98	27.65	–	–
	SeqPAN [27]	31.72	27.19	21.65	25.86
	MAT [38]	48.79	37.57	–	–
	MGSL [28]	42.54	32.27	–	–
	MESM [26]	52.69	39.52	–	36.94
	UniVTG [62]	51.44	34.97	17.35	33.60
	MS-DETR [63]	53.16	39.65	23.42	37.01
Ours		49.65	34.77	21.86	33.89

4.4. Ablation Study

4.4.1. Effect of Individual Components

This section presents an ablation study with several variants to quantify the contributions of the key components in our framework, including Object-Phrase Prototype Matching (OPPM) and Event-Sentence Prototype Matching (ESPM). All experiments are conducted on ActivityNet Captions, and the results are reported in

Table 4. Ablation results of the proposed method on the ActivityNet Captions dataset.

ID	OPPM		ESPM		Rank@1			mIoU
	SPG	Matching	TPG	Matching	$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
1	✗	✗	✓	✓	61.39	52.93	32.91	42.05
2	✓	✓	✗	✗	63.39	53.88	33.43	43.35
3	⟲TPG	✓	✓	✓	64.06	54.39	34.13	43.89
4	✓	✓	⟲SPG	✓	63.39	53.91	33.82	43.93
5	✓	P–P	✓	✓	62.73	53.42	32.52	42.97
6	✓	O–W	✓	✓	62.37	52.91	32.21	42.57
7	✓	✓	–F	✓	64.73	54.37	33.74	43.81
8	✓	✓	–M	✓	65.40	54.86	34.04	44.27
9	✓	✓	–R	✓	64.06	54.88	33.13	44.89
10	✓	F–W	✓	✓	64.73	54.37	34.04	44.81
11	✓	✓	✓	$\overline{\mathrm{F}}$–S	63.39	53.39	33.87	43.43
12	✓	mean	✓	✓	66.18	54.73	32.69	44.20
13	✓	✓	✓	✓	67.73	56.81	35.48	46.03

✗ indicates removing the corresponding module;✓ indicates retaining the corresponding module;⟲ indicates replacing the corresponding module.

. We first examine the performance when only OPPM or only ESPM is enabled. The results show that removing either module leads to a substantial performance drop. With only OPPM, the model can suppress irrelevant regions via object-phrase relations. However, it lacks explicit modeling of action evolution, making it difficult for candidate segments to remain semantically consistent with the query over longer time spans. With only ESPM, the model can aggregate event information within video segments, but without object-level local semantic support, boundary predictions become less stable in segments with dense entities or alternating actions. When both prototype modules are present, local entity cues and temporal action structure act jointly, allowing cross-modal alignment to leverage both fine-grained evidence and global semantics.

Next, we analyze how alternative prototype generation schemes affect performance. If event prototypes are derived directly from frame features ($⟲\mathrm{TPG}$), they lose the entity-grounded local semantics distilled by object prototypes and become overly driven by appearance cues. If they are built only from object features ($⟲\mathrm{SPG}$), inter-frame dependencies are insufficiently modeled and temporal action evolution is overlooked. Both variants degrade alignment performance, indicating that event prototypes must integrate object cues with cross-frame aggregation to remain stable. Moreover, replacing spatial prototype matching with token-to-token alignment (P–P) or object-to-word alignment (O–W) also leads to a marked performance drop. P–P fails to aggregate semantics across frames and produces fragmented event cues. O–W ignores sentence-level structure, making action-text correspondence ambiguous.

Third, we assess how the spatial and temporal decoders influence prototype construction. Removing the frame decoder (–F), the attention mask (–M), or the residual connection (–R) leads to performance degradation to varying degrees. Removing –F undermines object-level spatial coherence and increases the model’s sensitivity to local distractions during object-phrase alignment. Removing –M weakens inter-frame coupling and disrupts the temporal integration of event prototypes. Removing -R hampers feature propagation and destabilizes temporal modeling, particularly in longer videos.

Finally, we compare two alternative global alignment schemes, namely global frame-feature alignment (F–W) and mean pooling of frame features ($\overline{\mathrm{F}}$–S). Both choices lead to substantial performance degradation. The former struggles to preserve object-level details, whereas the latter suppresses action dynamics, leaving alignment without a clear local semantic basis. We further evaluate a variant that replaces the max-based matching with mean aggregation (mean). This variant also leads to performance degradation across all evaluation metrics, likely because mean aggregation dilutes the most discriminative object–phrase correspondences by averaging similarities over many irrelevant prototypes and frames.

4.4.2. Analysis of Prototype Configuration

This subsection examines how different prototype configurations affect model performance, with results reported in

Table 5. Analysis of prototype number configurations on the ActivityNet Captions dataset.

$\{{\mathit{N}}_{\mathit{o}},{\mathit{N}}_{\mathit{p}},{\mathit{N}}_{\mathit{e}}\}$	Rank@1			mIoU
	$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
$\{6,14,3\}$	60.31	51.82	32.13	41.95
$\{6,18,3\}$	61.24	52.41	33.34	42.69
$\{12,14,3\}$	63.33	54.52	33.86	43.17
$\{12,18,1\}$	62.52	53.83	32.95	42.52
$\{12,18,3\}$	63.73	54.81	33.48	43.79
$\{12,18,4\}$	62.55	53.33	32.34	42.98
$\{16,12,3\}$	67.73	56.81	35.48	46.03

. The candidate ranges for the numbers of prototypes are determined according to the structural roles of the three prototype levels and the statistical characteristics of VTG datasets. For object prototypes, CLIP represents each frame using 49 spatial tokens (image patches), while only a few salient entities typically appear within a frame; therefore, we evaluate moderate spatial compression levels (6, 12, and 16). For phrase prototypes, queries in Charades-STA, ActivityNet Captions, and TACoS usually contain multiple semantic components (objects, actions, and relations), so we evaluate 12 and 18 phrase prototypes to capture these compositions. For event prototypes, queries generally refer to one or a few actions within a short temporal interval; thus, a small range (1–4) is sufficient to model diverse event structures.

The results indicate that suboptimal prototype settings can substantially degrade performance. With an insufficient number of spatial prototypes, the model fails to represent the diversity of object instances; the matching becomes sparse, and local alignment accuracy deteriorates. By contrast, an excessive number of phrase prototypes induces competition over a limited set of object prototypes, which weakens the semantic aggregation associated with each prototype. The number of event prototypes also matters: too few may under-cover distinct action segments, while too many may over-segment the temporal signal and make each event prototype less stable semantically. Among all configurations, the setting $16,12,3$ performs best. Compared with the preceding groups, this configuration sharpens entity granularity through object prototypes, curbs text-side over-fragmentation by moderately reducing phrase prototypes, and keeps event prototypes at a moderate scale. As a result, alignment among entities, phrases, and event segments becomes clearer.

4.4.3. Analysis of the Fusion Mechanism

To further assess how object-phrase matching and event-sentence matching interact under different query semantics, we compare static and dynamic fusion strategies on the ActivityNet Captions dataset. In static fusion, the two matching scores are linearly combined using preset weights, whereas dynamic fusion learns the weights during training and adaptively adjusts the contribution of each mechanism according to the query content and video structure.

Table 6. Comparison of fusion strategies on the ActivityNet Captions dataset.

Fusion Strategy	$\mathit{\alpha }$	$\mathit{\beta }$	Rank@1			mIoU
			$\mathbf{IoU}=\mathbf{0}.\mathbf{3}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{5}$	$\mathbf{IoU}=\mathbf{0}.\mathbf{7}$
Static fusion	0.0	1.0	61.39	52.93	32.91	42.05
	0.3	0.7	64.10	53.82	33.62	43.72
	0.5	0.5	65.55	54.23	34.85	44.68
	0.7	0.3	64.23	53.95	33.68	43.92
	1.0	0.0	63.39	53.88	33.43	43.35
Dynamic fusion	–	–	67.73	56.81	35.48	46.03

summarizes the results of these fusion variants. When $\alpha =0$, the model relies solely on event-sentence matching and achieves the weakest performance. This observation suggests that, without entity-level constraints, boundary predictions become more prone to drift. As the weight on object-phrase matching increases, the $\mathrm{IoU}=0.7$ metric improves steadily, highlighting the role of entity cues in fine-grained alignment. However, excessive emphasis on spatial alignment is not optimal. When $\alpha$ approaches $1.0$, performance at $\mathrm{IoU}=0.5$ decreases slightly, indicating that relying exclusively on local entities cannot fully capture event semantics. Static fusion attains its best overall performance when $\alpha =0.5$, where spatial and temporal cues contribute in a more balanced manner. Under this setting, the model can leverage both entity-level details and the event sequence, leading to strong alignment accuracy across metrics.

Dynamic fusion improves all evaluation metrics and surpasses static fusion by approximately two percentage points at $\mathrm{IoU}=0.5$. This result suggests that, as query semantics vary, automatically adjusting the relative weights of the two prototype types better captures cross-level semantic dependencies. Consequently, the resulting matching score aligns more closely with the event structure and semantic intent expressed in the query sentence.

4.4.4. Parameter Sensitivity

In the joint objective, the parameter $\lambda$ controls the relative weight of the cross-modal alignment loss ${\mathcal{L}}_{\mathrm{Match}}$ and the binary cross-entropy loss ${\mathcal{L}}_{\mathrm{BCE}}$, thus determining whether the training objective emphasizes semantic alignment or temporal boundary accuracy. To identify a reasonable balance, we evaluate $\lambda \in \{0.0,0.1,0.3,0.5,0.7,1.0\}$ on ActivityNet Captions and report the results in Figure 6. When $\lambda$ is small (e.g., 0 or $0.1$), training relies almost exclusively on ${\mathcal{L}}_{\mathrm{BCE}}$. In this regime, object and event prototypes do not yet impose strong discriminative constraints over candidate segments, so local semantic consistency is unstable and the model tends to select segments primarily by visual similarity. Consequently, performance remains at a relatively low level. As $\lambda$ increases to $0.5$, the influence of ${\mathcal{L}}_{\mathrm{Match}}$ strengthens, enabling object-phrase matching to more reliably suppress segments irrelevant to query-mentioned entities and event-sentence matching to induce temporally coherent action semantics, which improves candidate ranking with respect to the query. When $\lambda$ exceeds this point, ${\mathcal{L}}_{\mathrm{Match}}$ dominates and ${\mathcal{L}}_{\mathrm{BCE}}$ is weakened; the model focuses on semantic separation while boundary precision is under-constrained, and performance at high IoU deteriorates noticeably. Overall, setting $\lambda$ around $0.5$ yields a stable synergy between prototype-level semantic consistency and boundary learning, allowing both object-phrase and event-sentence matching signals to contribute effectively to candidate filtering and ranking, thereby achieving the best temporal grounding performance.

4.4.5. Analysis of Kernel Size

To examine the effect of temporal receptive field design, we conduct a sensitivity analysis on the kernel size of the temporal convolution used in the Temporal Prototype Generation module. Specifically, we evaluate kernel sizes of 3, 5, and 7 on the ActivityNet Captions dataset while keeping all other settings unchanged. The results are illustrated in Figure 7.

Figure 6. Parameter sensitivity analysis of the loss balancing coefficient $\lambda$ on the ActivityNet Captions dataset. The figure reports the performance variations under different $\lambda$ values, reflecting the trade-off between the cross-modal alignment loss and the boundary prediction loss.

Figure 6. Parameter sensitivity analysis of the loss balancing coefficient $\lambda$ on the ActivityNet Captions dataset. The figure reports the performance variations under different $\lambda$ values, reflecting the trade-off between the cross-modal alignment loss and the boundary prediction loss.

As shown in the figure, the performance differences among the three configurations are relatively small across all evaluation metrics. Increasing the kernel size does not yield consistent improvements in temporal grounding accuracy. This behavior can be attributed to the fact that the temporal convolution in our framework operates on clip-level prototypes constructed from uniformly sampled frames, where local temporal patterns have already been captured. Larger kernels therefore provide only limited additional context while potentially introducing excessive temporal smoothing. Based on these observations, we adopt a kernel size of 3 as a balanced choice for modeling temporal evolution while maintaining sensitivity to precise temporal boundaries.

Figure 7. Performance comparison of temporal convolution kernel sizes (3, 5, and 7) on the ActivityNet Captions dataset.

Figure 7. Performance comparison of temporal convolution kernel sizes (3, 5, and 7) on the ActivityNet Captions dataset.

4.5. Efficiency Analysis

To evaluate the computational efficiency of the proposed framework, we report the total parameter count, floating-point operations (FLOPs), and inference latency under the same experimental settings used in our evaluations. The results are summarized in

Table 7. Computational efficiency of the proposed method on three benchmark datasets.

Dataset	FLOPS (B)	Params (M)	Times (ms)
Charades-STA	1.454	162.261	130.674
ActivityNet Captions	2.275	162.261	560.992
TACoS	2.293	162.261	570.736

. All experiments were conducted on a workstation equipped with an NVIDIA RTX 4090 GPU. The model was implemented using Python 3.8 and PyTorch 1.13.0. Video frames were extracted using FFmpeg (version 4.3.1), and the inference time was averaged over the entire test split under identical hardware and batch configurations.

It is worth noting that the CLIP backbone remains frozen during both training and inference, and visual features are extracted offline. Consequently, the trainable components mainly consist of the prototype generation and matching modules, whose computational overhead is relatively small. In practice, the prototype matching network contributes less than 5 GFLOPs, indicating that the additional computational cost introduced by the hierarchical prototype alignment is lightweight compared with typical video temporal grounding architectures.

As shown in Table 7, the parameter size of the model is largely dominated by the backbone, while the learnable modules account for only a small fraction of the overall capacity. Since the model architecture remains unchanged across datasets, the parameter count is consistent, while FLOPs and inference latency vary slightly due to different clip partitioning and temporal sampling strategies.

4.6. Qualitative Analysis

4.6.1. Visualization of Prototypes

In this section, we present qualitative visualizations of both spatial and event prototypes, as summarized in Figure 8. We begin by rendering object prototypes on individual frames, which show that irrelevant regions are largely suppressed and that the retained high-response areas align locally and explicitly with the corresponding textual phrases. Building on this observation, we further compute, according to Equation (19), the cross-frame attention weights for three event queries over eight video frames and visualize them using line plots. The three event prototypes exhibit clear differences along the temporal axis. For example, event prototype 1 concentrates its attention on the first two frames, capturing the action of drawing on the pumpkin; event prototype 2 attains the highest weights over frames 3–5, aligning with carving using a knife; and event prototype 3 peaks around frame 6, corresponding to the stage in which the interior is removed. These observations indicate that the model effectively learns temporal relations and that event prototypes can represent diverse semantic patterns.

4.6.2. Visualization of Score Map

By comparing the two-dimensional score maps produced under different constructions of candidate moment features, we can directly observe how the three methods differ in their ranking patterns, as shown in Figure 9. When candidate moment features are constructed using max pooling, local frame activations are amplified, and a single salient location can yield nearly identical scores for a large set of candidates that span it; this widens the ranking distribution, weakens inter-segment discrimination, and destabilizes boundary prediction under local perturbations. Stacked temporal convolutions partially mitigate this over-amplification effect, but the resulting moment features are still largely governed by appearance similarity; consequently, videos with visually similar actions produce overly smooth score maps whose ranking boundaries are not sharply constrained by semantics. By contrast, the proposed approach aligns candidate moment features with the query through matching results, which sharpens the score distribution by concentrating probability on text-relevant candidates while rapidly suppressing irrelevant ones. With a more decisive ranking distribution, the model becomes less susceptible to interference from neighboring candidates when selecting the best moment, and boundary prediction becomes more stable.

4.6.3. Grounding Results

This section presents qualitative visualizations of grounding results on representative videos from three benchmarks, comparing the full model with a variant without the cross-modal alignment loss, as illustrated in Figure 10. Overall, these visualizations highlight the role of prototype matching in stabilizing boundary estimation under different data conditions.

In the Charades-STA example (Figure 10a), the short indoor clip contains small-scale daily actions, where object and event prototypes capture limited cues; therefore, removing the cross-modal alignment loss causes little change in prediction. In this setting, prototypes mainly help maintain stable candidate ranking.

By contrast, ActivityNet Captions videos typically involve multiple objects and concurrent events. As shown in Figure 10b, the video includes children and adults with objects such as a drum and a guitar, while drumming and instrument playing occur simultaneously. Object prototypes align with phrase-level descriptions and event prototypes with sentence-level semantics, concentrating relevant candidates while suppressing visually similar yet irrelevant intervals. Without the alignment loss, predicted boundaries drift toward nearby background segments, indicating that cross-modal alignment is important for candidate ranking in complex scenes.

TACoS videos contain long cooking procedures with clear temporal structure and ordered action steps. The case in Figure 10c shows two successive actions, peeling and cutting into pieces, which have weak visual contrast but span a long duration. The full model uses event prototypes to maintain semantic linkage over time, concentrating candidate rankings around the main action trajectory. Without the cross-modal alignment loss, the model is more easily distracted by adjacent segments and produces boundary spillover, showing that event prototypes provide useful temporal constraints in this scenario.

4.7. Case Study

To further examine the behavior of the dynamic fusion mechanism under different query semantics, we present two representative examples in Figure 11. The first query mainly emphasizes spatial object-level cues (e.g., objects and spatial relations), while the second query focuses on temporal event evolution. As shown in Figure 11a, the dynamic fusion strategy assigns a higher spatial weight for the spatially detailed query, leading to a prediction closer to the ground-truth segment compared with static fusion. In contrast, for the temporally focused query in Figure 11b, the model increases the temporal matching weight, allowing event-level alignment signals to dominate the final grounding decision. These examples illustrate that the dynamic fusion mechanism can adaptively adjust the relative contributions of spatial and temporal prototype matching according to query semantics, thereby improving temporal grounding accuracy compared with fixed-weight fusion.

5. Conclusions

We propose a hierarchical prototype alignment framework for video temporal grounding that decomposes cross-modal alignment into complementary object-phrase and event-sentence alignments. In the object-phrase alignment stage, we generate spatial prototypes to emphasize discriminative regions and strengthen fine-grained, entity-level correspondence. In the event-sentence alignment stage, we progressively aggregate object cues over time to form event prototypes, enabling coherent alignment with sentence semantics and improving robustness under complex event compositions. Beyond performance gains, this study highlights the value of coupling local entity evidence with temporal event structure for stable grounding, and it offers a principled direction for improving VTG models through structured alignment design rather than purely heuristic architectural tuning.