Video Temporal Grounding with Multi-Model Collaborative Learning

Abstract

Given an untrimmed video and a natural language query, the video temporal grounding task aims to accurately locate the target segment within the video. Functioning as a critical conduit between computer vision and natural language processing, this task holds profound importance in advancing video comprehension. Current research predominantly centers on enhancing the performance of individual models, thereby overlooking the extensive possibilities afforded by multi-model synergy. While knowledge flow methods have been adopted for multi-model and cross-modal collaborative learning, several critical concerns persist, including the unidirectional transfer of knowledge, low-quality pseudo-label generation, and gradient conflicts inherent in cooperative training. To address these issues, this research proposes a Multi-Model Collaborative Learning (MMCL) framework. By incorporating a bidirectional knowledge transfer paradigm, the MMCL framework empowers models to engage in collaborative learning through the interchange of pseudo-labels. Concurrently, the mechanism for generating pseudo-labels is optimized using the CLIP model’s prior knowledge, bolstering both the accuracy and coherence of these labels while efficiently discarding extraneous temporal fragments. The framework also integrates an iterative training algorithm for multi-model collaboration, mitigating gradient conflicts through alternate optimization and achieving a dynamic balance between collaborative and independent learning. Empirical evaluations across multiple benchmark datasets indicate that the MMCL framework markedly elevates the performance of video temporal grounding models, exceeding existing state-of-the-art approaches in terms of mIoU and Rank@1. Concurrently, the framework accommodates both homogeneous and heterogeneous model configurations, demonstrating its broad versatility and adaptability. This investigation furnishes an effective avenue for multi-model collaborative learning in video temporal grounding, bolstering efficient knowledge dissemination and charting novel pathways in the domain of video comprehension.

1. Introduction

Video Temporal Grounding (VTG) seeks to accurately pinpoint the corresponding time segment in an untrimmed video based on natural language queries [1]. As a pivotal task at the intersection of computer vision and natural language processing, VTG lays the groundwork for a variety of downstream video comprehension tasks, such as video dialogue systems [2], video relationship detection [3,4,5], and video question answering systems [6,7,8]. The majority of traditional VTG strategies concentrate on refining a single model, typified by the Span-based Model and the 2D-Map Model shown in Figure 1, both of which display distinct advantages and limitations in particular contexts [9]. The Span-based Model explicitly forecasts the probabilistic distribution of a target interval’s start and end points, showcasing commendable capabilities in delineating precise temporal boundaries. Nevertheless, given its dependence on localized feature extraction, it exhibits a diminished aptitude for holistic semantic context modeling, thereby restricting its effectiveness in handling semantically intricate videos [10,11]. In comparison, the 2D-Map Model devises a two-dimensional temporal mapping, partitioning the entire video span into a 2D matrix in which every element corresponds to a prospective temporal clip. Operating within a two-dimensional framework, this model maximizes the score region, thereby adeptly capturing the contextual interplay among temporal segments. However, the expansive search space for candidate clips renders its computational overhead substantial, ultimately undermining boundary prediction precision [12,13,14].

Despite the impressive performance of these models in specific settings, their independent optimization strategies fail to fully exploit the complementary nature among models, disregarding the potential for collaborative learning and thereby limiting further performance enhancement. In recent years, knowledge flow, which encompasses knowledge distillation and transfer, has increasingly become a critical strategy for achieving multi-model cooperative optimization in VTG endeavors. Through the supervised transfer of extensive knowledge from large-scale models to more compact ones, knowledge distillation empowers lightweight models to attain robust performance under constrained computational budgets [9,15,16]. By leveraging cross-modal alignment and data exchange, knowledge transfer fortifies the synergy between visual and textual representations, facilitating effective multimodal co-modeling [17,18]. Nevertheless, existing work often restricts knowledge flow to a unidirectional paradigm, such as transferring insights from larger models to smaller counterparts or employing simplistic cross-modal alignments. Such approaches generate pseudo-labels of limited quality and are prone to gradient conflicts in multi-model collaboration [19,20], thus failing to fully harness the potential of bidirectional cooperative learning and dynamic knowledge sharing. These shortcomings have emerged as a key bottleneck restricting further advancements in VTG performance.

To solve the aforementioned deficiencies, this study introduces a Multi-Model Collaborative Learning (MMCL) framework, illustrated in Figure 1. Within this framework, a bidirectional collaborative learning paradigm is instituted across multiple models, yielding significant enhancements in the performance of video temporal grounding. Concretely, to surmount the constraints of one-way knowledge transfer, the MMCL architecture adopts a bidirectional knowledge transmission scheme. Through an equitable exchange of pseudo-labels across different models, predictions from Model A furnish pseudo-supervision for Model B, while Model B’s output similarly informs Model A. This bidirectional collaboration mechanism fully leverages the complementary learning capacities among models, compensates for the limitations of one-way knowledge flow, and further capitalizes on each model’s strengths in boundary prediction and context modeling. To bolster the fidelity of pseudo-labels, the MMCL framework introduces a CLIP-guided pseudo-label generator. Leveraging the CLIP [21] model’s cross-modal priors, this module calculates the similarity between the video and the query text to provide precise temporal constraints for the pseudo-label generator, thereby reducing noise while enhancing temporal accuracy and semantic consistency of the pseudo-labels. Moreover, to counteract gradient clashes within multi-model collaborative learning, the MMCL framework embeds an iterative training algorithm. During each optimization cycle, one model’s parameters remain static while the other model’s supervision and transfer objectives are refined. This alternating optimization tactic substantially alleviates gradient contention among models and ensures their respective optimization autonomy. Through adaptive calibration of consistency thresholds and transfer loss coefficients, the framework achieves a nuanced balance between cooperative learning and autonomous optimization, culminating in superior overall performance.

Experimental results show that the proposed MMCL framework exhibits pronounced strengths and robust versatility in video temporal grounding. Its inherent adaptability makes the MMCL framework suitable for both heterogeneous and homogeneous model collaboration contexts, where it consistently yields appreciable performance gains. From a theoretical perspective, this study broadens the applicability of knowledge transfer and collaborative learning, addressing limitations in pseudo-label quality and model collaboration mechanisms, while offering an efficient and flexible solution for video temporal grounding. The key contributions of this work are enumerated as follows:

• We propose a multi-model collaborative learning framework employing a bidirectional knowledge transfer scheme to foster cooperative learning across distinct models, thus transcending the confines of isolated model optimization and amplifying cross-model complementarity.
• We design a CLIP-guided pseudo-label generator that harnesses CLIP’s cross-modal priors to optimize pseudo-label generation, improving their quality and robustness, and bolstering the accuracy of knowledge transfer.
• We present an iterative multi-model training algorithm that alternately freezes and optimizes model parameters to effectively mitigate gradient conflicts, safeguard the balance between cooperative and independent optimization, and further boost training stability and model performance.
• Experimental findings reveal that our method effectively enhance the performance of various baseline methods for video temporal grounding. By incorporating both the VSLNet [22] and 2D-TAN [23] baseline models, performance on multiple datasets surpasses current mainstream approaches.

2. Related Work

2.1. Video Temporal Grounding

Video Temporal Grounding (VTG) involves aligning temporal video segments with natural language queries, and existing methods predominantly fall into two categories: Span-based Models and 2D-MAP Models. Span-based Models, such as VSLNet [22], frame the VTG task as a span-based question-answering problem, where video segments serve as context, the target temporal fragment as the answer span, and the linguistic query as the question. These models excel at accurately detecting temporal boundaries and efficiently localizing target segments through boundary regression techniques. However, their focus on explicit boundary prediction often limits their ability to capture broader semantic contexts and handle long-duration interactions, making them less effective in complex video scenarios [10,11]. In contrast, 2D-MAP Models encode all potential start-end intervals as elements of a two-dimensional matrix, refining high-scoring regions to locate the desired temporal segment [9]. The 2D-TAN model [23], for example, constructs a temporal map to model contextual interactions among video segments, offering robust capabilities in modeling context and segment dependencies [24]. Despite their strengths, these models may struggle with boundary prediction accuracy in densely overlapping temporal segments, particularly under intricate temporal distributions [12,13,14]. While both approaches offer distinct advantages, their independent optimization limits the potential of leveraging their complementary strengths, highlighting the need for a collaborative learning framework like MMCL to integrate these methodologies effectively.

2.2. Knowledge Flow in Video Temporal Grounding

The utilization of knowledge flow in multimodal tasks is gaining momentum, predicated on the concept of disseminating knowledge across different models or modalities to augment collective model capability [25]. Present investigations can be bifurcated into two principal streams: knowledge distillation and knowledge transfer. Via knowledge distillation, the expertise of a large model is conferred upon a more compact model, ensuring the latter maintains commendable performance while remaining computationally efficient [26]. By integrating both local and global modality information and applying distillation techniques, Bao et al. improved the feature representation of video understanding models, thereby achieving greater accuracy in temporal localization tasks [16]. Other works leverage the robust representational capacity of large language models (LLMs) to generate weakly supervised pseudo-labels via distillation, thereby offering a novel optimization pathway for VTG tasks [15]. Furthermore, Liang et al. introduced a distillation approach employing multiple teacher models to collectively steer the student model’s learning, significantly boosting temporal localization accuracy for VTG [9]. Diverging from the principles of knowledge distillation, knowledge transfer focuses on facilitating cross-modal information exchanges that foster cooperative learning among distinct modalities. Weng et al. proposed a vision-prompted knowledge transfer method that integrates visual and textual modality features, significantly enhancing multimodal reasoning in video question answering tasks [17]. Wu et al. devised a dynamic knowledge transfer mechanism that enables cross-modal knowledge sharing between visual and textual predictors, effectively bolstering temporal localization and answer generation in video question answering tasks [18]. Although existing knowledge flow methods moderately improve model performance, most studies are confined to unidirectional knowledge transfer (e.g., from large to small models) or cross-modal knowledge migration, failing to fully explore the potential of bidirectional cooperative learning and thus limiting the depth and breadth of knowledge sharing. Additionally, the phenomenon of gradient contention within multi-model collaborative learning undercuts the advantages of coordinated optimization, thus imposing further limitations on knowledge-sharing efficacy and holistic performance improvements.

2.3. Pseudo-Label in Video Temporal Grounding

Pseudo-labels are extensively utilized in video temporal grounding tasks, fundamentally relying on model-generated predictions to form surrogate supervision signals. Such an approach bolsters performance in weakly or semi-supervised settings, while likewise demonstrating considerable viability in low-resource or unsupervised domains [27]. Xu et al. proposed a pseudo-label generation method based on predictive confidence, where high-confidence time segments are filtered for training via a confidence threshold [28]. Nevertheless, this approach remains vulnerable to threshold selection, risking the exclusion of valuable segments or the inclusion of suboptimal ones [29]. An alternative tactic is to utilize the model’s predicted temporal segments as pseudo-labels outright, for instance, leveraging initial predictions as pseudo supervision in weakly supervised video localization [30]. Yet the fidelity of such pseudo-labels is wholly determined by the initial model’s predictive prowess, impeding effective mitigation of noisy segments [30]. While these methods may excel in particular scenarios, pseudo-label quality generally hinges on the accuracy of the underlying model predictions [31]. Inferior pseudo-labels can erode the efficacy of model training and inject noise into the learning procedure, thereby impeding potential gains in performance [32]. Moreover, existing methods lack explicit constraints on the pseudo-label generation range, making it difficult for the model to filter out irrelevant or low-confidence segments.

3. Methodology

In this study, we introduce an MMCL framework tailored for video temporal grounding. The overall architecture of MMCL is illustrated in Figure 2.

3.1. Problem Formulation

The objective of video temporal grounding is to identify the optimal time segment M, consistent with the query’s semantics, given an untrimmed video V of duration L and a natural language query Q. In this task, the video V is denoted by a frame sequence $V={\left\{f_i\right\}}_{i=1}^T$, where $f_i$ represents the i-th frame of the video and T is the total number of frames; the natural language query $Q={\left\{q_j\right\}}_{j=1}^W$ comprises the j-th word of the query, with W being the total number of words. Presently, mainstream approaches to video temporal grounding typically employ a multimodal fusion paradigm, mapping both the video V and the query Q into a unified shared feature space X. Within the feature space X, the model learns the probability distribution of the time range, predicting the start and end times $\left[t_s,t_e\right]$ of the target segment to ensure high semantic consistency with the query.

3.2. General Model

The base architecture of the MMCL framework encompasses a feature extractor, a cross-modal fusion module, and a visual predictor. To address various model requirements under the MMCL framework, the basic model unifies the structure of video temporal grounding architectures, enabling effective multi-model cooperation. The feature extractor obtains features from the input video V and the natural language query Q, encoding both into high-dimensional representations used by the subsequent cross-modal fusion component to model their semantic interconnections. The video is partitioned into N equally spaced segments, each of duration $\mathrm{\Delta }t={\displaystyle \frac{L}{N}}$. A pretrained visual model (e.g., 3D ConvNet [33] or CNN [34]) extracts the visual features for each segment, denoted as follows:

$$F_V^k={\mathrm{FeatureExtractor}}_V\left(V\right),F_V^k\in {\mathbb{R}}^{N\times d_v}$$

(1)

Here, $d_v$ is the dimensionality of the video features, and k signifies the current model. The query is processed by a pretrained linguistic model (e.g., GloVe [35] or BERT [36]) to obtain features, denoted by the following:

$$F_Q^k={\mathrm{FeatureExtractor}}_Q\left(Q\right),F_Q^k\in {\mathbb{R}}^{M\times d_q}$$

(2)

In this case, $d_q$ represents the dimensional size of the query feature vector. The aforementioned extracted features are subsequently passed to the Cross-Modal Fusion module. The Cross-Modal Fusion module combines the video features $F_V^k$ and the query features $F_Q^k$ to generate the multimodal fused representation $X^k$, formally given by the following:

$$X^k={\mathrm{Fusion}}_k(F_V^k,F_Q^k),X^k\in {\mathbb{R}}^{N\times d_x}$$

(3)

In this formulation, $d_x$ signifies the dimensional scale of the fused feature space. The Cross-Modal Fusion module effectively captures the interactive information between video segments and the query semantics, providing high-quality multimodal representations to the visual prediction module. Under the MMCL framework, each model maintains its native multimodal processing approach while leveraging pseudo-labels to learn from the strengths of the other model. MMCL acts as a flexible collaborative layer that supports diverse architectural choices and fosters model synergy. By using pseudo-labels as soft supervision signals, MMCL facilitates knowledge transfer without altering the inherent multimodal interaction strategies of each model, allowing, for example, the 2D-TAN model to learn fine-grained boundary prediction from VSLNet in fast-changing scenarios, and vice versa. The visual predictor estimates the target time range $P_k$ from the fused features $X_k$, with the output format varying by model architecture. Span-based models yield 1D probability distributions for the start and end time points, expressed as follows:

$$P_s^k,P_e^k={\mathrm{Predictor}}_k\left(X^k\right),P_s^k,P_e^k\in {\mathbb{R}}^N$$

(4)

Meanwhile, 2D-MAP-based models output a two-dimensional temporal probability matrix, represented by the following:

$$P_{i,j}^k={\mathrm{Predictor}}_k\left(X^k\right),P_{i,j}^k\in {\mathbb{R}}^{N\times N}$$

(5)

One dimension i of the matrix corresponds to the start boundary, and the other dimension j to the end boundary, with $i<j$. Each element $P_{i,j}$ in the matrix represents the probability that the video segment begins at index i and ends at index j.

3.3. Pseudo-Label Generator

3.3.1. CLIP-Guided Module

The CLIP-Guided Module leverages the cross-modal alignment capability of the CLIP model between video frames and textual queries, providing a coarse-grained temporal constraint for pseudo-label generation to ensure semantic consistency. The overall process can be broken down into three distinct stages:

(1) Semantic Vector Extraction

The input video V is uniformly sampled into $n_f$ frames across its total length L, represented by ${\left\{f_i\right\}}_{i=1}^{n_f}$. Every frame is paired with a timestamp ${\left\{t_i\right\}}_{i=1}^{n_f}$, where $t_i={\displaystyle \frac{i}{n_f}}·L$. For each frame f, the CLIP model’s visual encoder yields its semantic vector $v_i$, formulated as follows:

$$v_i={\mathrm{CLIP}}_{\mathrm{image}}\left(f_i\right),i=1,2,\dots ,n_f$$

(6)

Concurrently, the query Q is passed through CLIP’s text encoder to obtain its textual representation:

$$u_i={\mathrm{CLIP}}_{\mathrm{text}}\left(Q\right)$$

(7)

(2) Computing Similarities

To gauge the semantic congruence between $v_i$ and $u_i$, we calculate their cosine similarity. The calculation is as follows:

$$S_i=\mathrm{cosine\_similarity}(v_i,u_i)={\displaystyle \frac{v_i·u_i}{\parallel v_i\parallel \parallel u_i\parallel }},i=1,2,\dots ,n_f$$

(8)

In this formulation, $S_i\in [-1,1]$, with higher values signifying greater semantic harmony between Q and frame $f_i$. The sequence of similarity scores for the $n_f$ frames is denoted by $S=\{S_1,S_2,\dots ,S_{n_f}\}$.

(3) Generating the Constrained Interval

Drawing on the similarity sequence S, a threshold $\tau$ is utilized to filter relevant frames by their time span. First, all frames with $S_i\ge \tau$ are identified, and their minimum and maximum indices, $i_{min}$ and $i_{max}$, are determined according to Equation (9).

$$i_{min}=min\{i\mid S_i\ge \tau \},i_{max}=max\{i\mid S_i\ge \tau \}$$

(9)

Subsequently, these frame indices are mapped to actual temporal boundaries: $t_{cs}=t_{i_{min}};t_{ce}=t_{i_{max}}$. Finally, $[t_{cs},t_{ce}]$ denotes the coarse-grained temporal interval guided by the CLIP model, which constrains pseudo-label generation.

3.3.2. Pseudo-Labels Generation

In the process of pseudo-label generation, we utilize the coarse-grained temporal interval $[t_{cs},t_{ce}]$ provided by the CLIP model and apply normalization to constrain the range of pseudo-label creation, thereby ensuring semantic consistency and producing pseudo-labels $[{\widehat{t}}_s,{\widehat{t}}_e]$ that align with semantic priors. The generation of pseudo-labels depends on the prediction format of different models. Below, we separately describe the pseudo-label generation processes for Span-based models and 2D-MAP models.

(1) Pseudo-Labels Generation for Span-based Models

As stated in Section 3.2, a Span-based model (such as VSLNet) produces probability distributions $P_s\left(t\right)$ and $P_e\left(t\right)$ corresponding to the start and end times, respectively. The original probability distributions are normalized over the entire video span $\left[1,N\right]$, which contains a substantial number of query-irrelevant time segments. To enhance semantic consistency, it is necessary to constrain and re-normalize these distributions based on the temporal window $[t_{cs},t_{ce}]$ provided by CLIP. Following restriction of these probability distributions to the selected interval, the normalized forms ${\widehat{P}}_s\left(t\right)$ and ${\widehat{P}}_e\left(t\right)$ are specified as follows:

$${\widehat{P}}_s\left(t\right)=\left\{\begin{array}{cc}{\displaystyle \frac{P_s\left(t\right)}{{\sum }_{t=t_{cs}}^{t_{ce}}{P}_s\left(t\right)}},\hfill & t\in [t_{cs},t_{ce}],\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(10)

$${\widehat{P}}_e\left(t\right)=\left\{\begin{array}{cc}{\displaystyle \frac{P_e\left(t\right)}{{\sum }_{t=t_{cs}}^{t_{ce}}{P}_e\left(t\right)}},\hfill & t\in [t_{cs},t_{ce}],\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(11)

The normalization step guarantees that the distributions remain confined to semantically pertinent intervals, thereby narrowing the pseudo-label search domain and fortifying semantic coherence. Upon finalizing the normalization, the pseudo-label boundaries ${\widehat{t}}_s$ and ${\widehat{t}}_e$ emerge by identifying the indices that maximize the normalized probability distributions, formally expressed as follows:

$${\mathrm{index}}_s=argmax{\widehat{P}}_s\left(t\right),{\mathrm{index}}_e=argmax{\widehat{P}}_e\left(t\right)$$

(12)

Subsequently, the indices are mapped to pseudo-labels ${PL}^k$ by Equation (13).

$$P{L}^k=[{\widehat{t}}_s={\mathrm{index}}_s\times \mathrm{\Delta }t,{\widehat{t}}_e={\mathrm{index}}_e\times \mathrm{\Delta }t]$$

(13)

(2) Pseudo-Labels Generation for 2D-MAP Models

Under the 2D-MAP model, the pseudo-labels derive from a 2D temporal probability matrix $P_{i,j}$, with i and j designating the start and end indices, respectively, and adhering to $i\le j$. For $P_{i,j}$, elements within the CLIP-guided temporal window are retained, while probabilities outside this range are set to zero. The remaining elements are then normalized to ensure the sum is one. The definition of the normalized probability matrix ${\widehat{P}}_{i,j}$ is as follows:

$${\widehat{P}}_{i,j}=\left\{\begin{array}{cc}{\displaystyle \frac{P_{i,j}}{{\sum }_{i=t_{cs}}^{t_{ce}}{\sum }_{j=i}^{t_{ce}}{P}_{i,j}}},\hfill & i,j\in [t_{cs},t_{ce}]\mathrm{and}i\le j,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(14)

Subsequently, the highest-probability entry in ${\widehat{P}}_{i,j}$ pinpoints the start and end indices of the pseudo-labels:

$$({\mathrm{index}}_i,{\mathrm{index}}_j)=argmax{\widehat{P}}_{i,j}$$

(15)

Lastly, Equation (16) translates these indices into the pseudo-labels ${PL}_k$.

$$P{L}^k=[{\widehat{t}}_s={\mathrm{index}}_i\times \mathrm{\Delta }t,{\widehat{t}}_e=({\mathrm{index}}_j+1)\times \mathrm{\Delta }t]$$

(16)

By following the procedure above, we leverage CLIP’s semantic guidance to generate high-quality pseudo-labels, thus offering reliable supervisory signals for training.

3.4. Mutual Knowledge Transfer

At the heart of the MMCL framework is the harnessing of inter-model knowledge exchange and cooperative optimization, capitalizing on each model’s complementary capabilities to boost collective outcomes. Within this framework, pseudo-label generation and loss function design constitute the essential elements for enabling collaborative learning. Pseudo-label generation (detailed in Section 3.3) furnishes high-quality supervisory signals for cross-model knowledge migration, while the loss function employs a suitable optimization strategy to effectively merge single-model learning with inter-model collaboration. Accordingly, we construct the comprehensive loss function as a weighted amalgamation of supervision loss and transfer loss, satisfying both each model’s need for accurate learning and the bidirectional knowledge exchange among multiple models. In the supervised learning stage, the supervision loss steers Models A and B to refine their performance using genuine labels as guidance. Formally, it is specified as follows:

$$L_{\sup }^A=\mathcal{L}(P^A,GT),L_{\sup }^B=\mathcal{L}(P^B,GT)$$

(17)

Here, $P^A$ and $P^B$ denote the predictions of Model A and Model B, respectively; $GT$ stands for ground truth; and $\mathcal{L}$ refers to the particular form of the loss function, which varies depending on the model category. The role of the supervision loss is to align $P^A$ and $P^B$ more tightly with $GT$, providing direct supervised feedback to each model. In the knowledge transfer phase, the transfer loss drives collaborative learning between models by optimizing the pseudo-labels produced by the counterpart model, and leverages those pseudo-labels as indirect supervision signals to achieve cross-model knowledge transfer. Specifically, Model A’s transfer loss $L_{trans}^A$ uses the pseudo-labels ${PL}^B$ produced by Model B, aligning them with A’s predictions $P^A$ to encourage Model A to learn B’s predictive characteristics. Analogously, Model B’s transfer loss $L_{trans}^B$ imposes knowledge constraints on Model B via ${PL}^A$, nudging its optimization objective toward Model A’s predictions. The transfer loss is defined as follows:

$$L_{\mathrm{trans}}^A=\mathcal{L}(P^A,P{L}^B),L_{\mathrm{trans}}^B=\mathcal{L}(P^B,P{L}^A)$$

(18)

Building upon the aforementioned supervision and transfer objectives, the MMCL framework accomplishes multi-model cooperative learning through a weighted synthesis of these losses. The aggregate loss function is given by the following:

$$\mathrm{LOSS}=L_{\sup }^A+L_{\sup }^B+\alpha L_{\mathrm{trans}}^A+\beta L_{\mathrm{trans}}^B$$

(19)

Here, $\alpha$ and $\beta$ serve as hyperparameters governing the balance between transfer losses. With this scheme, every model benefits from solitary supervision-driven optimization toward ground-truth labels and, in parallel, participates in inter-model knowledge exchange facilitated by the transfer losses.

3.5. Iterative Training Algorithm

Within the MMCL framework, the core aim of the loss optimization strategy is to foster coherent predictions among models, yet preserve each model’s autonomy in its individual optimization processes. To accomplish this, we introduce an iterative training algorithm that blends consistency evaluation with dynamic loss tuning, thereby enabling inter-model knowledge exchange and collective enhancement. In this section, we elaborate on the loss function’s optimization strategy from three perspectives: consistency assessment, dynamic loss adjustment, and algorithmic implementation.

3.5.1. Consistency Assessment

Within the realm of multi-model collaborative learning, the temporal intervals forecasted by various models can at times diverge, resulting in inconsistency. To assess and quantify this inconsistency, we introduce the Intersection over Union (IoU) as a measure of consistency. Concretely, IoU is calculated by dividing the intersection of the two predicted time spans by their union, expressed as follows:

$$\mathrm{IoU}(A,B)={\displaystyle \frac{max\left(0,min(t_e^A,t_e^B)-max(t_s^A,t_s^B)\right)}{(t_e^A-t_s^A)+(t_e^B-t_s^B)-max\left(0,min(t_e^A,t_e^B)-max(t_s^A,t_s^B)\right)}}$$

(20)

In this context, $\left[t_s^A,t_e^A\right]$ and $\left[t_s^B,t_e^B\right]$, respectively, signify the predicted time intervals for Model A and Model B. On the basis of the IoU value, we define an IoU threshold $\lambda$ to gauge the degree of concordance between the models’ predictions. When $\mathrm{IoU}>\lambda$, the alignment between the two models’ predictions is relatively strong, prompting a decrease in the transfer loss weight. Conversely, if $\mathrm{IoU}\le \lambda$, the predictions are deemed inconsistent, warranting an increase in the transfer loss weight so as to guide both models’ results toward consistency. By quantifying the degree of consistency, the MMCL framework can dynamically adjust inter-model learning intensity, ensuring the effectiveness of collaborative learning. Consistency evaluation offers a dynamic adjustment signal that directly influences the adaptive loss adjustment procedure (see Section 3.5.2), guiding transfer loss weighting and providing a basis for iterative training.

3.5.2. Dynamic Loss Adjustment

To balance the independent optimization of each model with the consistency of cross-model predictions in collaborative learning, we devise a dynamic loss adjustment strategy founded on IoU. Building on top of the supervision and transfer losses, we incorporate a dynamic weighting factor $\omega$ (refer to Equation (21)) to regulate the influence of transfer loss on the total loss. The definition of the dynamic factor $\omega$ is as follows:

$$\omega =\left\{\begin{array}{cc}1-\mathrm{IoU}(A,B),\hfill & \mathrm{if}\mathrm{IoU}\le \lambda ,\hfill \\ \gamma ·(1-\mathrm{IoU}(A,B\left)\right),\hfill & \mathrm{if}\mathrm{IoU}>\lambda \hfill \end{array}\right.$$

(21)

Here, $\gamma \in (0,1)$ is a parameter that governs the attenuation of the transfer loss under high consistency conditions. The total loss function after dynamic adjustment can be expressed as follows:

$$\mathrm{LOSS}=L_{\sup }^A+L_{\sup }^B+\alpha ·\omega ·L_{\mathrm{trans}}^A+\beta ·\omega ·L_{\mathrm{trans}}^B$$

(22)

In this expression, $\alpha$ and $\beta$ are the intrinsic weights for the transfer loss, and the factor $\omega$ modulates the strength of cross-model transfer in accordance with IoU. Under conditions of strong predictive alignment, the framework grants each model greater latitude for autonomous optimization, whereas lower alignment triggers a heavier emphasis on cross-model knowledge exchange, thereby bolstering model concordance. Leveraging this strategy, the MMCL framework deftly orchestrates the magnitudes of both supervision and transfer losses, securing an optimal trade-off between inter-model knowledge sharing and standalone optimization.

3.5.3. Algorithmic Realization

In the course of knowledge dissemination, inter-model gradients can impede one another, thereby curtailing each model’s capacity to learn independently. This phenomenon, known as gradient contention, occurs when gradients produced by different models or tasks conflict with each other, leading to ineffective optimization. Specifically, when gradients push shared model parameters in opposing directions, they may neutralize each other’s effect, hindering both models’ learning processes. To preclude gradient contention across models and, at the same time, enable effective knowledge transmission and reinforce each model’s autonomy in optimization, we devise an iterative training algorithm. This algorithm partitions loss optimization into several phases and carries them out in a sequentially alternating fashion. Initially, the procedure optimizes Model A’s supervised loss $L_{sup}^A$ while holding Model B’s parameters static, thus allowing Model A to refine its performance under the supervision of true labels; subsequently, it optimizes Model B’s supervised loss $L_{sup}^B$ while freezing Model A’s parameters. Thereafter, the framework employs the transfer losses $L_{trans}^A$ and $L_{trans}^B$ to impose knowledge constraints on Models A and B, respectively, using pseudo-labels to guide inter-model refinement. See Algorithm 1 for details. This staggered iterative update scheme safeguards each model’s independent feature learning, using pseudo-labels to facilitate a more adaptable form of collaboration. In the end, after repeated iterations, the predictions from both models increasingly align, culminating in a synergistic improvement in overall performance. This algorithm deftly circumvents gradient contention while bolstering both models’ learning capabilities and operational robustness.

Algorithm 1: Dynamic Loss Adjustment with IoU-based Weighting

4. Experiments

4.1. Datasets

To evaluate the performance of our proposed MMCL, we conduct experiments on three challenging video temporal grounding datasets:

Charades-STA [1] consists of videos portraying routine activities, sourced from the Charades dataset [37]. It encompasses 6672 videos, 16,128 annotations, and 11,767 temporal moments. The uneven distribution of clip lengths and a high degree of overlap place considerable demands on local feature modeling. Of the annotated moments, 12,408 are allocated for training and 3720 for testing.

ActivityNet Caption [38] was initially constructed for dense video captioning. It comprises around 20,000 YouTube videos with diverse contents, averaging 120 s in length. The dataset features semantic variety and broad domains, placing higher demands on the multimodal representation capability of models. Following the experimental setup in [22,23], val1 is used as the validation set and val2 as the test set, containing 37,421 and 17,505 annotated temporal moments for training and testing, respectively. Currently, this is the largest dataset in this task.

TACoS [39] is a video temporal grounding dataset focusing on kitchen scenarios, derived from the MPII Cooking Composite Activities dataset [39]. It comprises 127 videos with annotated time segments, totaling over 18 h in duration. The dataset features a large number of consecutive actions, with dense temporal distributions and a high overlap ratio, posing greater challenges for boundary-based models. Following [22,23], 10,146, 4589, and 4083 temporal annotations are used for training, validation, and testing, respectively.

4.2. Baselines

We employ VSLNet [22] and 2D-TAN [23] as two baseline networks in the framework, representing the typical Span-based and 2D-MAP models, respectively. Both have publicly available source code, and we train and evaluate the MMCL framework on the three datasets described in Section 4.1.

4.3. Implementation Details

4.3.1. Evaluation Metrics

Following existing video grounding works, we evaluate the performance on two main metrics:

mIoU: This metric measures the mean Intersection over Union (IoU) between the model’s predicted intervals and the reference intervals. By averaging the IoU values over the entire test set, it reflects the overall localization accuracy of the model and effectively evaluates the precision of temporal segment predictions.

Recall: We adopt “$Rank@n,\mathrm{IoU}=m$” as the evaluation metrics, following [1]. The “$Rank@n,\mathrm{IoU}=m$” represents the percentage of language queries having at least one result whose IoU between $top-n$ predictions with ground-truth is larger than m. In our experiments, we reported the results of $n=1$ and $m\in \left\{0.3,0.5,0.7\right\}$.

4.3.2. Experimental Settings

For each baseline method, VSLNet follows the configurations reported in their original paper, adopting the C3D-feature version of VSLNet. As for 2D-TAN, the components of the framework likewise follow the same parameters as presented in its original documentation. The hyperparameters $\alpha$ and $\beta$ (refer to Section 3.4) regulate the weighting for the transfer losses $L_{trans}^A$ and $L_{trans}^B$. Based on validation tuning, we choose $\alpha =0.5$ and $\beta =0.5$ to strike a balance between supervision and transfer objectives in model optimization. The IoU threshold $\lambda$ (see Section 3.5) evaluates predictive alignment between models, set at 0.5 to maintain a reasonable level of discriminatory power. The factor $\gamma$ (see Section 3.5), responsible for dynamically modulating weights, is fixed at 0.8 to attenuate the transfer loss under scenarios of high model consistency. The number of video frames $n_f$ in the CLIP-Guided Module (see Section 3.3.1) is set to 32 to balance semantic representational power and computational efficiency. The similarity threshold $\tau$ (see Section 3.3.1) is set to 0.4, filtering out highly relevant frames to generate a more precise guiding temporal range. The learning rates ${\eta }_A$ and ${\eta }_A$ are both set to 0.0001, and the number of iterative optimization cycles K (see Algorithm 1) is set to 10.

4.4. Training and Inference

This subsection provides an examination of how the MMCL framework performs throughout both training and inference stages. Figure 3 juxtaposes the framework’s outcomes in loss optimization, test accuracy, signal learning, and noise memorization.

Observing the training loss curve reveals that MMCL demonstrates a noticeably swifter reduction in loss compared with the baseline model. In addition, as training epochs progress, the loss steadily converges to a more stable, lower plateau. This expedited convergence owes much to the framework’s bidirectional knowledge transfer design, which allows the two models to exchange learned representations and leverage pseudo-labels as supplementary supervisory signals, thus steering clear of local optima. In the testing phase, the cooperative learning facilitated by the MMCL framework markedly enhances the accuracy of both models. Especially during the final training iterations, VLSNet’s contextual reasoning and 2D-TAN’s temporal boundary localization both become more refined, indicating that this collaborative mechanism can successfully broaden each model’s performance ceiling. The signal learning trajectory reveals that the MMCL framework incrementally fortifies the cooperative models’ capacity to capture relevant signals, subsequently stabilizing and outperforming any single-model baselines. CLIP’s guidance mechanism, leveraging semantic priors to refine pseudo-labels’ precision and coherence, plays a key role in these gains. The juxtaposition of noise memorization curves offers additional evidence of the efficacy of CLIP-guided pseudo-labels. By more effectively suppressing spurious or low-fidelity time segments, the MMCL framework supplies purer supervisory signals for training.

4.5. Comparison with State-of-the-Arts

4.5.1. Comparative Methods

We compare our proposed MMCL with state-of-the-art video temporal grounding methods on three public datasets. These methods can be grouped into two categories:

(1) Span-based Models: CTRL [1], ROLE [40], ACL [41], SM-RL [42], ExCL [43], DRN [44], CBLN [45], PS-VTG [28], D3G [46], PFU [47], TGN [48], ABLR-af [49], ABLR-aw [49], ACRN [50], VSLNet [22], MRTNet(VSLNet) [51];

(2) 2D-MAP Models: SAP [52], QSPN [53], DEBUG [54], MAN [55], GDP [56], CI-MHA [57], CMIN [58], 2D-TAN [23], MMN [59], MRTNet(2D-TAN) [51]

4.5.2. Quantitative Analysis

The outcomes on the three benchmarks, as reported in

Table 1. Performance comparison with the state-of-the-art models on Charades-STA dataset.

Methods	Publication	Rank@1			mIoU
		$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{3}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{5}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{7}$
CTRL [1]	ICCV2017	–	23.63	8.9	–
ROLE [40]	MM2018	25.26	12.12	–	–
ACL [41]	WACV2019	–	30.48	12.20	–
SAP [52]	AAAI2019	–	27.42	13.36	–
SM-RL [42]	CVPR2019	–	24.36	11.17	–
QSPN [53]	AAAI2019	54.70	35.60	15.80	–
DEBUG [54]	EMNLP2019	54.95	37.39	17.92	36.34
ExCL [43]	ACL2019	–	44.10	22.40	–
MAN [55]	CVPR2019	–	46.53	22.72	–
GDP [56]	AAAI2020	54.54	39.47	18.49	–
DRN [44]	CVPR2020	–	42.90	23.68	–
CBLN [45]	CVPR2021	–	43.67	24.44	–
CI-MHA [57]	SIGIR2021	69.87	54.68	35.27	–
MMN [59]	AAAI2022	–	47.31	27.28	–
PS-VTG [28]	ITM2022	–	39.22	20.17	–
D3G [46]	ICCV2023	–	41.64	19.60	–
2D-TAN [23]	AAAI2020	57.31	39.70	23.31	39.23
MRTNet (2D-TAN) [51]	ICASSP2024	59.23	44.27	25.88	40.59
MMCL (2D-TAN)	–	60.58	46.95	28.26	42.17
VSLNet [22]	ACL2020	70.46	54.19	35.22	50.02
MRTNet (VSLNet) [51]	ICASSP2024	70.88	56.19	36.37	50.74
MMCL (VSLNet)	–	72.31	58.86	38.52	51.96

,

Table 2. Performance comparison with the state-of-the art models on ActivityNet Caption dataset.

Methods	Publication	Rank@1			mIoU
		$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{3}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{5}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{7}$
TGN [48]	EMNLP2018	43.81	27.93	–	–
CMIN [58]	SIGIR2019	63.61	43.40	23.88	–
QSPN [53]	AAAI2019	45.30	27.70	13.60	–
ABLR-af [49]	AAAI2019	53.65	34.91	–	35.72
ABLR-aw [49]	AAAI2019	55.67	36.79	–	36.99
DEBUG [54]	EMNLP2019	55.91	39.72	–	39.51
GDP [56]	AAAI2020	56.17	39.27	–	39.80
DRN [44]	CVPR2020	58.52	41.51	23.07	43.13
CI-MHA [57]	SIGIR2021	61.49	43.97	25.13	–
SeqPAN [60]	ACL2021	61.65	45.50	29.37	45.11
CBLN [45]	CVPR2021	66.34	48.12	27.60	–
MMN [59]	AAAI2022	65.05	48.59	29.26	–
PS-VTG [28]	ITM2022	59.71	39.59	21.98	–
PFU [47]	ARXIV2023	59.63	36.35	16.61	40.15
D3G [46]	ICCV2023	58.25	36.68	18.54	–
2D-TAN [23]	AAAI2020	59.45	44.51	26.54	43.29
MRTNet (2D-TAN) [51]	ICASSP2024	60.71	45.59	28.07	44.54
MMCL (2D-TAN)	–	62.15	47.21	29.35	46.53
VSLNet [22]	ACL2020	63.16	43.22	26.16	43.19
MRTNet (VSLNet) [51]	ICASSP2024	64.17	44.09	27.43	44.82
MMCL (VSLNet)	–	65.59	46.82	28.59	45.98

and

Table 3. Performance comparison with the state-of-the-art models on TACoS dataset.

Methods	Publication	Rank@1			mIoU
		$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{3}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{5}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{7}$
CTRL [1]	ICCV2017	18.32	13.30	–	–
ACRN [50]	SIGIR2018	19.52	14.62	–	–
TGN [48]	EMNLP2018	21.77	18.90	–	–
SM-RL [42]	CVPR2019	20.25	15.95	–	–
ACL [41]	WACV2019	22.07	17.78	–	–
ABLR-aw [49]	AAAI2019	18.90	9.30	–	12.50
DEBUG [54]	EMNLP2019	23.45	11.72	–	16.03
CMIN [58]	SIGIR2019	24.64	18.05	–	–
CBLN [45]	CVPR2021	38.98	27.65	–	–
SeqPAN [60]	ACL2021	31.72	27.19	21.65	25.86
MMN [59]	AAAI2022	38.57	27.24	–	–
PS-VTG [28]	ITM2022	23.64	10.00	3.35	–
D3G [46]	ICCV2023	27.27	12.67	4.70	–
2D-TAN [23]	AAAI2020	37.29	25.32	13.32	25.19
MRTNet (2D-TAN) [51]	ICASSP2024	37.81	26.01	14.95	26.29
MMCL (2D-TAN)	–	39.50	27.50	15.12	27.85
VSLNet [22]	ACL2020	29.61	24.27	20.03	24.11
MRTNet (VSLNet) [51]	ICASSP 2024	32.35	25.84	21.31	26.14
MMCL (VSLNet)	–	34.15	26.81	22.57	27.83

, indicate that our MMCL framework significantly enhances the baseline models’ performance.

Summarizing the results in Table 1, Table 2 and Table 3, we see that on the Charades-STA dataset, 2D-TAN underperforms VSLNet mainly because its 2D-MAP mechanism struggles with short segments and high-overlap video scenarios, whereas VSLNet’s boundary-classification approach allows more precise modeling of short-segment boundaries. After incorporating the MMCL framework, 2D-TAN learns the semantic modeling capability of VSLNet’s boundary-classification approach, improving its mIoU to $42.17$ and achieving a relative improvement of $7.5\%$, demonstrating that the MMCL framework effectively compensates for 2D-TAN’s shortcomings in modeling short segments. By contrast, on the TACoS dataset, VSLNet underperforms 2D-TAN because its span-based structure finds it difficult to handle the dense and uneven segment distribution of TACoS, while 2D-TAN’s 2D-MAP mechanism is better suited for complex temporal-relationship modeling. After integrating the MMCL framework, VSLNet acquires long-segment relational modeling and temporal context-capturing abilities from 2D-TAN, raising its mIoU to $27.83$ and yielding a $15.4\%$ relative improvement. These results demonstrate that the MMCL framework, through knowledge transfer and collaborative learning, significantly improves both models’ performance on the datasets where they originally fared poorly.

From the results in Table 2, both models achieve improved performance on the ActivityNet Caption dataset. With its semantic modeling advantage, VSLNet achieves an mIoU of $43.19$ on this dataset. However, by learning richer temporal relationship modeling from 2D-TAN through the MMCL framework, its mIoU rises to $45.98$, a relative improvement of $6.4\%$. Similarly, by sharing knowledge with VLSNet, 2D-TAN advances its mIoU from $43.29$ to $46.53$, achieving a $7.1\%$ relative improvement.

In sum, through cross-model knowledge sharing and joint learning, the MMCL framework allows the two baseline models to mutually address their shortcomings and amplify their respective advantages. On TACoS, VLSNet reaps the benefits of 2D-TAN’s temporal relationship modeling; on Charades-STA, 2D-TAN inherits stronger boundary-modeling capabilities from VLSNet. This reciprocal learning and knowledge transfer characteristic endows the MMCL framework with robust adaptability across different task scenarios. Moreover, relative to other competing methods and MRTNet, the MMCL framework achieves superior results on all metrics in each of the three datasets, offering additional validation of its efficacy.

4.6. Performance of Isomorphic Models

Even under identical model architectures, the random sampling of training data and mini-batch dynamics induce divergent biases in temporal segments, feature representations, or modality alignment. Such discrepancies lay the groundwork for knowledge transfer between structurally identical models. To assess the MMCL framework’s efficacy and potential constraints in same-architecture collaborative learning, we designate 2D-TAN as both Model A and Model B and conduct experiments on the Charades-STA dataset, employing the hyperparameters outlined in Section 4.3.2. According to Figure 4, the models utilizing the MMCL framework demonstrate a degree of performance enhancement relative to the baseline model. This finding suggests that even with a uniform model design, differences in data inputs and the stochastic nature of feature learning give rise to variations in predictive outcomes. The MMCL framework capitalizes on these variations for inter-model knowledge transfer, thereby achieving a synergistic boost in overall performance.

4.7. Ablation Studies

In an effort to streamline the experimental design, the following ablation studies are carried out on the Charades-STA dataset, leveraging VSLNet and 2D-TAN as baseline architectures.

4.7.1. Loss Function

To evaluate the effectiveness of each component in the proposed loss function, we conduct a series of tests with different loss settings, as shown in

Table 4. Performance comparison of MMCL with different losses on Charades-STA dataset.

Experimental Setting		Rank@1			mIoU
		$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{3}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{3}$	$\mathbf{IoU}\mathbf{=}\mathbf{0}\mathbf{.}\mathbf{3}$
MMCL (VSLNet)	Ours	72.31	58.86	38.52	51.96
	w/o $L_{sup}^A$	68.56	52.82	33.89	48.11
	w/o $L_{sup}^B$	70.12	56.71	36.25	49.92
	w/o $L_{trans}^A$	70.85	56.71	36.79	50.25
	w/o $L_{trans}^B$	71.92	57.91	37.65	51.35
MMCL (2D-TAN)	Ours	60.58	46.95	28.26	42.17
	w/o $L_{sup}^A$	59.41	45.63	27.11	41.03
	w/o $L_{sup}^B$	55.81	38.89	22.18	37.84
	w/o $L_{trans}^A$	59.83	46.42	27.94	41.72
	w/o $L_{trans}^B$	58.78	45.56	27.21	40.85

.

Supervision loss ($L_{sup}^A$ and $L_{sup}^B$) underpins the independent optimization of each model; removing this component forces the affected model to depend predominantly on transfer loss and pseudo-labels for training. As an example, omitting $L_{sup}^A$ strips Model A of its capacity for standalone optimization, resulting in a notable performance drop. Concurrently, the pseudo-labels transferred to Model B deteriorate in quality, undermining the efficacy of knowledge transfer. However, since Model B still retains $L_{sup}^B$, its performance decline is relatively limited. Similarly, removing $L_{sup}^B$ leads to a significant reduction in Model B’s performance, especially when capturing strict boundaries (e.g., $IoU=0.7$). Model A is also affected by the degraded quality of pseudo-labels, but to a lesser extent, as it still depends on $L_{sup}^A$. Excluding the transfer losses ($L_{trans}^A$ and $L_{trans}^B$) disrupts the inter-model knowledge-sharing mechanism. Removing $L_{trans}^A$ prevents Model A from assimilating supplementary insights from Model B, leading to a performance dip. Model B, retaining $L_{trans}^B$ and its corresponding supervisory signals, suffers only marginal setbacks. Conversely, removing $L_{sup}^B$ robs Model B of its ability to acquire knowledge from Model A, causing a significant performance drop, whereas Model A’s performance sees only a minor decline since it can still rely on its own supervision signals for independent optimization.

4.7.2. Hyperparameters of $\alpha$ and $\beta$

From Equation (22) in Section 3.5.2, it is evident that the base weights $\alpha$ and $\beta$ for the transfer loss exert a direct influence on the strength of inter-model knowledge transfer. In this study, we hold $\gamma$ constant while varying the fundamental weights $\alpha$ and $\beta$. Referring to Figure 5, once $\alpha$ and $\beta$ approximate 0.5, knowledge transfer finds an equilibrium that enables mutual learning and leads to superior overall performance. When $\alpha$ and $\beta$ remain small, the force of knowledge transfer lessens, causing each model to rely heavily on its own supervision and thereby producing results akin to the baseline model. This situation suggests that inter-model cooperation is curtailed, and it becomes challenging to compensate for the limitations inherent in a solitary model. If $\alpha$ and $\beta$ are severely skewed (e.g., $\alpha$ is much larger than $\beta$ or vice versa), knowledge transfer adopts a one-sided flow, resulting in asymmetrical performance. Conversely, if $\alpha$ and $\beta$ are both sizable, the frequency of knowledge transfer escalates, injecting abundant “noise” that hampers the model’s ability to distinguish useful supervisory cues from extraneous transfer signals, ultimately diminishing cooperative gains. Taken together, properly tuning $\alpha$ and $\beta$ is essential for upholding equilibrium and moderation in inter-model knowledge sharing; doing so maximizes each model’s unique advantages while averting skewed transfer or excessive disruption that might undermine overall performance.

4.7.3. Hyperparameters of $\lambda$ and $\gamma$

The hyperparameters $\lambda$ and $\gamma$, which govern the dynamic adjustment of transfer loss (see Section 3.5.2), play key roles in determining recall, precision, and overall inter-model cooperation. In this study, $\alpha$ and $\beta$ remain fixed at 0.5, while performance metrics are gathered for various $\gamma -\lambda$ pairs (see Figure 6). As the $IoU$ consistency threshold, $\lambda$ measures the degree of similarity in the models’ predictions. A relatively small $\lambda$ defines consistency more leniently, capturing a greater number of predictions and thereby boosting recall, yet at the risk of admitting low-quality knowledge flow that can erode precision. By contrast, a larger $\lambda$ imposes tighter consistency requirements, fostering higher precision but limiting collaborative interactions between models and thereby diminishing recall. Experimental findings suggest that a suitably moderate $\lambda$ strikes an effective compromise between shared knowledge transfer and independent optimization. Meanwhile, the dynamic factor $\gamma$ dictates the degree of transfer-loss attenuation when models exhibit high consistency. A larger $\gamma$ enables models to lean more on their internal supervision signals, thus bolstering precision. Conversely, a smaller $\gamma$ strengthens cross-model knowledge sharing, likely improving recall at the expense of reduced independence in optimization. Under lenient consistency requirements (e.g., $Rank@1,IoU=0.3$), the influence of $\lambda$ and $\gamma$ on recall is minimal, producing relatively stable performance. This suggests that collaborative knowledge exchange reliably enhances the model’s overall recall capability. Under more balanced consistency thresholds (e.g., $Rank@1,IoU=0.5$), the models grow increasingly sensitive to variations in $\lambda$ and $\gamma$, making prudent parameter choices pivotal for balancing recall and precision. Under stringent consistency metrics (e.g., $Rank@1,IoU=0.7$), the models display the highest sensitivity to $\lambda$ and $\gamma$, and moderate $\lambda$ in combination with a mid-to-high $\gamma$ yields peak performance during collaborative learning. In summary, $\lambda$ and $\gamma$ jointly manage the trade-off between recall and precision and, by dynamically calibrating the strength of knowledge transfer, serve as key levers for enhancing the overall efficacy of cooperative learning.

4.7.4. Effectiveness Analysis of the CLIP-Guided Module

In the CLIP-guided module, both the number of sampled video frames $n_f$ and the similarity threshold $\tau$ play a key role in determining the quality of pseudo-labels, thereby shaping the outcomes of inter-model knowledge transfer as well as final performance. In this subsection, we separately discuss the impact of varying the number of video frames and the similarity threshold on the MMCL framework, with the corresponding results presented in Figure 7. First, holding the similarity threshold constant at $\tau =0.4$, we gradually raise the number of sampled frames. As depicted in Figure 7a, the model’s performance (mIoU) progressively increases alongside the number of frames. This indicates that a higher frame count enriches the temporal scope and contextual fidelity of the pseudo-labels, effectively expanding the reservoir of knowledge they convey. Nevertheless, upon reaching 64 frames, performance gains level off or marginally drop because incorporating excessive frames does not translate into significant feature benefits but rather introduces unwarranted noise and squanders computational overhead. Subsequently, keeping $n_f=32$ constant, we adjust the similarity threshold $\tau$ to gauge how performance responds. As illustrated in Figure 7b, the model’s performance follows a nonlinear trajectory in response to variations in the similarity threshold $\tau$. A too-low threshold incorporates excessive low-grade segments in the pseudo-labels, weakening their utility in guiding optimization. Conversely, a too-stringent threshold narrows the scope of pseudo-labels, hindering the capture of the video’s broader context and thereby stifling collaborative improvement. In comparison, a moderate threshold ($\tau =0.4$) attains an equilibrium between filtering out noise and retaining adequate coverage, yielding pseudo-labels of superior quality for facilitating knowledge transfer. In sum, judiciously determining the number of frames and the similarity threshold is vital for producing robust pseudo-labels, thereby proving indispensable for successful inter-model knowledge transfer and performance enhancement.

4.7.5. Qualitative Analysis

Figure 8 shows qualitative results of MMCL(VSLNet) on the Charades-STA dataset and MMCL(2D-TAN) on the TACoS dataset. From the examples, we can observe that in the Charades-STA dataset, which features short action intervals with rapidly shifting boundaries, MMCL(VSLNet) demonstrates heightened temporal sensitivity and refined boundary delineation under CLIP guidance. As for the TACoS dataset, known for its longer action durations, MMCL(2D-TAN) demonstrates greater boundary-capturing precision under CLIP guidance. In the absence of CLIP guidance, the model frequently yields more expansive time spans than the actual annotations, hindering its ability to precisely identify the critical moments aligned with the query. This result reaffirms the crucial function of the CLIP-guided module in refining pseudo-label accuracy.

4.8. Case Study

The case study evaluates the MMCL (Multi-Model Collaborative Learning) framework’s ability to harness the complementary strengths of VSLNet and 2D-TAN models across various video temporal grounding scenarios, including short and long videos, complex temporal relationships, and fast-changing scenes.

In scenarios with short videos (Figure 9, left), where precise boundary detection is critical, MMCL improved 2D-TAN’s performance by allowing it to learn from VSLNet’s accuracy in temporal boundary prediction. Conversely, in long video scenarios (Figure 9, right), which require a broader temporal context, MMCL enhanced VSLNet’s performance by incorporating 2D-TAN’s strength in extended temporal modeling. This demonstrates the framework’s capacity to facilitate effective mutual learning between models with distinct advantages. In Figure 10, the performance on “difficult” samples was examined through complex temporal scenarios and fast-changing scenes. Difficult samples in our study refer to scenarios that require a deep understanding of extended temporal dependencies or rapid visual changes. The task demands not only temporal segmentation but also an understanding of the sequential order of events. VSLNet, typically challenged by such scenarios, showed notable improvement under the MMCL framework, demonstrating enhanced temporal reasoning by leveraging 2D-TAN’s strengths. Conversely, in fast-changing scenarios like the BMX race video, which required precise alignment between fast-paced visual cues and the query, MMCL (2D-TAN) showed a significant performance gain. This result indicates that 2D-TAN, through bidirectional knowledge transfer, learned from VSLNet’s dynamic attention mechanism, which is particularly effective in fast-changing environments.

In specific challenging samples (Figure 11 and Figure 12), the standalone models struggled with accurate predictions, whereas the MMCL-enhanced models succeeded. The MMCL (2D-TAN) model adapted better to fast-changing scenarios by learning from VSLNet’s strengths, while MMCL (VSLNet) demonstrated improved performance in complex temporal scenarios by leveraging 2D-TAN’s proficiency in modeling long-range temporal dependencies. These results confirm that the MMCL framework not only preserves each model’s inherent strengths but also facilitates adaptive knowledge transfer, leading to improved performance on difficult samples and demonstrating the intended collaborative learning effect.

5. Conclusions

In this work, we proposed a Multi-Model Collaborative Learning (MMCL) framework that achieves efficient collaboration among multiple models through bidirectional knowledge transfer, a CLIP-guided pseudo-label generation mechanism, and an iterative training algorithm. Extensive evaluations across three benchmark datasets confirm that MMCL surpasses conventional baselines. Furthermore, the MMCL framework is highly versatile and can be employed in both isomorphic and heterogeneous model configurations, thus providing a novel paradigm for multi-model collaboration. While our current study focused on well-established datasets such as Charades-STA, ActivityNet Captions, and TACoS, we recognize the value of validating the MMCL framework on more recently released datasets, such as TVR and QVHighlight. These datasets introduce unique challenges, including longer video durations, more complex queries, and multi-view video sources. Exploring these datasets in future work will allow us to assess the generalization ability of MMCL under more diverse and evolving conditions. Even so, opportunities remain to enhance the framework’s computational overhead and training efficiency. Future work will concentrate on devising a more compact architecture and real-time optimizations, broadening the applicability of the framework in diverse real-world contexts.