A Survey of Multi-Model Collaboration in Video Understanding

Abstract

The rapid development of multimodal foundation models has shifted video understanding from perception-centered recognition toward more general semantic interpretation, reasoning, and decision-making over dynamic visual content. As video understanding tasks increasingly require fine-grained perception, long-range temporal modeling, multimodal grounding, and adaptive reasoning, collaboration among heterogeneous functional units, including specialized models, modules, agents, memory systems, and external tools, has emerged as an important system-level paradigm. However, existing surveys mainly organize video understanding methods by architectures, learning strategies, or task categories, leaving the collaborative structure of modern systems insufficiently examined. This survey provides a structured narrative review of multi-model collaboration in video understanding, which we formulate as collaborative video understanding. We introduce a unified analytical framework that characterizes collaborative systems through functional units, inter-unit communication mechanisms, and collaborative state representations, and organize existing methods according to their coordination dynamics into static collaboration and dynamic collaboration, with the latter further distinguished into controller-based and agent-based collaboration. We further review representative benchmarks, evaluation metrics, and empirical analysis, showing that current evaluation protocols mainly capture task-level performance but provide limited insight into collaborative organization, memory use, adaptive execution, and system-level collaborative capability. Finally, we discuss key challenges and future directions, including adaptive task decomposition, semantically aligned inter-unit communication, persistent shared memory, uncertainty-aware error containment, evidence-grounded reasoning, and collaboration-centric evaluation. By reinterpreting video understanding from a collaborative systems perspective, this survey aims to provide a structured foundation for developing more adaptive, reliable, and scalable video understanding systems.

1. Introduction

Video understanding has become a fundamental research topic in computer vision due to its broad applications in surveillance [1], autonomous driving [2], industrial monitoring [3], human–computer interaction [4], and video-language reasoning [5]. Different from static image understanding, video understanding requires effective modeling of temporal dynamics, long-range dependencies, and cross-modal semantic interactions within evolving video content [6]. These characteristics pose substantial challenges for spatiotemporal representation learning, temporal reasoning, and multimodal semantic alignment, making video understanding considerably more complex than static image understanding.

Video understanding methods have evolved through several major technological paradigms. Early approaches primarily relied on handcrafted features [7,8] and conventional statistical models [9], which exhibited limited generalization capability in complex real-world environments. With the rapid advancement of deep learning, Convolutional Neural Networks (CNN) were developed to learn hierarchical spatiotemporal representations from video data, significantly enhancing local spatiotemporal feature extraction and motion modeling capability [10,11]. Nevertheless, these methods still suffered from limited capability in modeling long-range temporal dependencies. Transformer-based architectures further advanced the field by introducing global contextual interaction through self-attention mechanisms [12,13], thereby enabling more effective long-range temporal reasoning.

More recently, video-oriented multimodal foundation models have transformed video understanding from perception-oriented recognition toward more general-purpose semantic understanding and reasoning [14,15]. This evolution has expanded the scope of video understanding, requiring modern systems to support efficient perception, temporal abstraction, multimodal grounding, and adaptive decision-making. However, when video-oriented multimodal foundation models are implemented as monolithic architectures, they still struggle to meet these heterogeneous requirements because a single parameterization and fixed inference pathway must jointly balance perception, temporal abstraction, multimodal grounding, and adaptive reasoning. To address these limitations, multi-model collaboration has emerged as a promising paradigm for video understanding [16,17]. In this survey, we define collaborative video understanding as approaches that coordinate two or more independently callable and replaceable computational units with functionally differentiated roles through explicit information exchange, adaptive routing, shared state maintenance, or decision aggregation [18]. These units may be instantiated as separate neural models, foundation models, retrieval or memory modules, external tools, or autonomous agents, provided that they can be invoked and replaced as distinct computational units during inference [19]. Unlike simple feature fusion, single-model multimodal pretraining, or conventional ensemble averaging, these methods focus on how computational units communicate, divide labor, and adapt their roles during video understanding [20,21]. By exploiting the complementary strengths of specialized components, collaborative video understanding methods can better balance perception, reasoning, efficiency, and scalability than monolithic methods [22].

Despite the rapid progress in this area, existing surveys [23,24,25,26,27,28] have primarily reviewed video understanding from model-centric perspectives, focusing on application tasks, architectural designs, learning paradigms, and video-language models, as summarized in

Table 1. Comparison of representative video understanding and multimodal-agent surveys and their analytical perspectives.

Survey	Primary Scope	Analytical Perspective	Collaboration-Centric
[23]	Temporal action segmentation	Core techniques and supervision regimes	×
[24]	Video Transformers	Pipeline design and architectural adaptations	×
[25]	Self-supervised learning for videos	Learning objectives and training paradigms	×
[28]	Vision-language pre-training	VLP design dimensions	×
[26]	Video-LLMs	Framework architectures and LLM functional roles	Partial
[29]	Large multimodal agents	Agent components, taxonomy, collaboration, and evaluation	Partial
[30]	Multimodal Agent AI	Agent integration, learning, categorization, and applications	Partial
Ours	Multi-model collaboration	Collaboration paradigms and mechanisms	✓

Note: “✓”, “Partial”, and “×” indicate primary, partial, and non-central collaboration focus, respectively.

. Recent multimodal-agent surveys [29,30] have also discussed collaboration-related aspects, such as memory, planning, tools, and multi-agent coordination; however, their discussions are situated in broader multimodal-agent settings rather than focusing specifically on video understanding. Although these surveys provide valuable overviews of existing developments, they leave several collaboration-related questions underexplored: how are specialized units selected, what information is exchanged, how is shared state maintained, and how does coordination evolve from fixed pipelines to adaptive agentic methods [20,31]. To bridge this gap, we present a structured survey of multi-model collaboration in video understanding from the perspective of collaborative components and coordination mechanisms.

To identify relevant studies, we searched Web of Science, Scopus, IEEE Xplore, and ACM Digital Library, supplemented by Google Scholar, using combinations of keywords related to video understanding, including video understanding, video reasoning, video question answering, long-video understanding, and Video-LLM, as well as keywords related to collaborative systems, including multi-model collaboration, multi-agent collaboration, modular, memory-augmented, tool-augmented, retrieval-augmented, routing, and planning. We prioritized studies that (1) address video understanding tasks, (2) involve multiple independently callable and replaceable computational units with functionally differentiated roles, and (3) incorporate explicit collaboration mechanisms, such as coordination, routing, memory, tool use, retrieval, or decision aggregation. Studies focusing only on unimodal image understanding or pure video generation were generally excluded unless they directly informed the proposed taxonomy.

The main contributions of this survey are summarized as follows:

• We provide a structured analysis of video understanding from a collaborative systems perspective by characterizing task requirements, collaborative components, and coordination challenges across heterogeneous video understanding scenarios.
• We introduce a unified system-level framework for collaborative video understanding, in which collaborative methods are analyzed through three core components: functional units, inter-unit communication mechanisms, and collaborative state representations.
• We further organize existing collaborative methods from the perspective of coordination mechanisms into two overarching paradigms, namely static collaboration and dynamic collaboration, revealing a progression from predefined execution structures toward adaptive and increasingly autonomous coordination.
• We review representative benchmarks, evaluation metrics, and empirical evidence across major video understanding tasks, and discuss why current evaluation protocols remain insufficient for measuring collaborative organization, memory use, adaptive execution, and system-level collaborative capability.
• We identify key challenges and future directions for collaborative video understanding, including structured task orchestration, semantically aligned inter-unit communication, persistent shared memory, uncertainty-aware error containment, evidence-grounded reasoning, trustworthy collaboration, and collaboration-centric evaluation.

The remainder of this survey is organized as follows. Section 2 formalizes the video understanding problem and summarizes its major task categories. Section 3 presents the collaborative video understanding framework and analyzes representative collaboration paradigms from the perspective of coordination mechanisms. Section 4 discusses benchmarks, evaluation metrics, and empirical evidence. Section 5 analyzes open challenges and future directions, and Section 6 concludes the survey.

2. Video Understanding Problem Definition

2.1. Problem Definition

Video understanding refers to a family of computational tasks that extract temporally grounded representations from video streams to support recognition [32], localization [33], cross-modal grounding [34], description [35], and reasoning [36]. Unlike image understanding, which operates on static visual observations, video understanding must model how entities, actions, and contexts evolve over time, thereby transforming frame-level observations into coherent event-level interpretations [37]. Early studies often treated video understanding as a recognition-oriented problem, focusing mainly on action recognition and event classification [23,24]. With the rise of multimodal foundation models and video-language models, the field has expanded toward open-ended semantic and causal reasoning, where systems are expected to infer what occurs in a video, when it occurs, how it evolves, and why it occurs under specific contextual conditions [5,38,39].

Formally, a video can be represented as $V=\{I,A^{*},L^{*}\}$, where $I={\left\{I_t\right\}}_{t=1}^T$ denotes an ordered sequence of T visual frames, $A^{*}$ denotes an optional audio modality, and $L^{*}$ denotes optional language information such as subtitles, transcripts, metadata, or task instructions. Given an optional task query Q, video understanding can be formulated as learning a task-dependent mapping $Y=\varphi (V,Q)$, where Y denotes the desired output. Depending on the application, Y may represent a recognition label, temporal localization result, textual description, question answer, structured prediction, reasoning trace, tool-use plan, or decision output [39,40,41,42]. Following the taxonomy in Figure 1, this survey groups video understanding tasks by their dominant output form and functional objective into four categories:

• Recognition and Localization Tasks identify entities or events in video content and estimate their spatial or temporal locations [43,44]. The output Y typically takes the form of categorical labels, spatial regions, temporal segments, trajectories, or spatio-temporal event locations. Representative tasks include action recognition, event classification, object tracking, temporal action localization, and spatio-temporal action localization [45]. These tasks primarily rely on video-internal evidence to determine what occurs and where or when it occurs [46].
• Cross-modal Matching and Grounding Tasks establish semantic correspondences between video content and external modalities, such as language or audio [47]. The output Y typically takes the form of retrieval rankings, similarity scores, matched video-text pairs, or spatially/temporally grounded regions or segments associated with an external query. Representative tasks include video-text retrieval, video-language alignment, video grounding, and audio-visual grounding [48,49]. Their defining characteristic is the explicit alignment of video content with information from another modality [50].
• Generative Description Tasks convert video content into natural-language descriptions or condensed semantic summaries [51,52]. The output Y is typically a free-form or structured textual representation, such as a caption, a sequence of event descriptions, or a condensed summary. Representative tasks include video captioning, dense video captioning, video summarization, and event description [48,49,53]. These tasks emphasize semantic abstraction and narrative organization by converting observed entities, actions, and temporal relations into coherent linguistic representations [50].
  Reasoning and Decision-making Tasks infer implicit information beyond directly observable video content by integrating evidence across temporal, semantic, and multimodal contexts [26,54]. The output Y may take the form of answers, inferred temporal or causal relations, explanations, decisions, or action plans. Representative tasks include video question answering, temporal reasoning, causal reasoning, commonsense reasoning, embodied video reasoning, and video-agent decision-making [55,56]. These tasks are closely related to collaborative video understanding because they often require coordinated perceptual grounding, evidence retrieval, memory maintenance, and multi-step reasoning [57].

Figure 1. Taxonomy of video understanding tasks organized by their dominant output forms and functional objectives. Recognition and localization tasks predict semantic labels or spatial and temporal locations; cross-modal matching and grounding tasks align video content with language or audio; generative description tasks produce captions, event descriptions, or summaries; and reasoning and decision-making tasks integrate temporal and multimodal evidence to answer questions, infer implicit relations, or select actions.

Figure 1. Taxonomy of video understanding tasks organized by their dominant output forms and functional objectives. Recognition and localization tasks predict semantic labels or spatial and temporal locations; cross-modal matching and grounding tasks align video content with language or audio; generative description tasks produce captions, event descriptions, or summaries; and reasoning and decision-making tasks integrate temporal and multimodal evidence to answer questions, infer implicit relations, or select actions.

2.2. Fundamental Characteristics of Video Understanding

Video understanding differs from static visual analysis due to the spatiotemporal and multimodal nature of video data [58,59]. From a fundamental perspective, video understanding can be characterized by three essential properties:

• Temporal Dependency. Video semantics often emerge from information distributed across multiple temporal segments rather than individual observations [60]. Effective video understanding therefore requires modeling temporal continuity, motion evolution, and long-range temporal interactions [61].
• Event-Centric Semantics. The semantic meaning of a video is primarily determined by events and interactions that evolve over time rather than isolated visual observations [62]. Consequently, effective video understanding requires identifying participating entities, modeling their relationships, and characterizing event evolution [36]. For example, understanding an interaction such as a person picking up an object and handing it to another person requires coordinated entity perception, relation modeling, and event-level temporal reasoning.
• Multi-level and Multimodal Reasoning. Effective understanding requires integrating information across multiple semantic levels, ranging from visual appearance and motion patterns to event structures, semantic relationships, and contextual knowledge [35]. In addition, video content is often associated with complementary modalities such as language and audio, which provide information unavailable from visual observations alone [63]. Consequently, modern video understanding increasingly relies on multi-level and multimodal reasoning to integrate heterogeneous information sources and construct comprehensive semantic interpretations of dynamic scenes [64].

3. Collaborative Video Understanding Framework

Modern video understanding requires systems to simultaneously handle temporal dependency, event-centric semantics, and multi-level multimodal reasoning across heterogeneous task outputs. These requirements challenge monolithic video understanding methods, as their single parameterization and fixed inference pathways limit the balance among specialization, adaptability, and computational efficiency. Collaborative video understanding offers a system-level alternative by coordinating functionally specialized units through information exchange, shared state maintenance, and adaptive control, thereby reducing the trade-offs inherent in monolithic architectures [64,65]. To characterize existing collaborative approaches, this section presents a unified framework that analyzes collaborative systems from two complementary perspectives: collaborative components and collaborative mechanisms.

3.1. Collaborative Components

Collaborative video understanding methods can be analyzed through a unified system-level abstraction composed of three elements: (1) collaborative functional units, (2) inter-unit communication mechanisms, and (3) collaborative state representations, as illustrated in Figure 2. This abstraction provides a common basis for analyzing collaborative video understanding across different implementations, including Video-LLM-based systems, retrieval-augmented frameworks, long-video understanding methods, and agent-based architectures [66].

Collaborative functional units constitute the fundamental components that perform specialized functions, such as perception, retrieval, memory, reasoning, planning, or other tools. Formally, the collaborative system is modeled as

$$U=\{u_1,u_2,\dots ,u_N\},$$

(1)

where $u_i$ denotes the i-th collaborative functional unit participating in the overall video understanding process. In practice, a collaborative functional unit may be instantiated as a foundation model [67], a retrieval component [64,68], a memory system [69], a reasoning module [56], a planning controller [17,70], an external tool [16,71], or an autonomous agent [18].

Inter-unit communication mechanisms define how collaborative functional units exchange task-relevant collaborative message during collaboration. Such collaborative message may involve visual features, semantic representations, retrieved evidence, memory contents, reasoning traces, planning instructions, or intermediate predictions [72,73]. The collaborative message transmitted from unit $u_i$ to unit $u_j$ at execution step t can be abstracted as

$$m_{i\to j}^t=c_{ij}^t(o_i^t,S_t,Q),$$

(2)

where $o_i^t$ denotes the intermediate output of unit $u_i$, $S_t$ denotes the current collaborative state representation, Q denotes the optional task query, and $c_{ij}^t(·)$ denotes the inter-unit communication function between two units. In this formulation, collaborative messages serve as the operational interface through which functional units condition one another, transfer evidence, and coordinate intermediate decisions during collaborative inference [74].

Collaborative state representations preserve and organize task-relevant context accumulated throughout the collaborative process. The system maintains a collaborative state representation $S_t$, which denotes task-relevant information accessible globally, locally, or through a controller-mediated interface at execution step t [75]. Depending on the implementation, $S_t$ may contain contextual representations, retrieved knowledge, memory records, intermediate results, tool-call traces, routing decisions, uncertainty estimates, or other information required for coordination among collaborative functional units [76]. State evolution can be written as

$$S_{t+1}=g(S_t,{\{m_{i\to j}^t\}}_{i,j},{\{o_i^t\}}_{i=1}^N),$$

(3)

and the final task output is generated by

$$Y=h(S_T,Q),$$

(4)

where $g(·)$ denotes the state update function and $h(·)$ denotes the task-specific output function.

3.2. Coordination Mechanisms

While collaborative functional units, inter-unit communication mechanisms, and collaborative state representations define the structural components of collaborative video understanding, coordination mechanisms determine how these components are organized during execution. From this perspective, existing collaborative systems are first distinguished according to whether their coordination policies remain fixed or are adaptively adjusted during execution, resulting in two primary paradigms: static collaboration and dynamic collaboration, as illustrated in Figure 3 and summarized in

Table 2. Characteristics of representative collaboration paradigms.

Paradigm	Coordination Type	Policy	Execution Behavior	Main Limitation
Static Collaboration	–	Predefined	Fixed execution pipeline	Limited adaptability
Dynamic Collaboration	Controller-based	Adaptive	Runtime routing, scheduling, or resource allocation	Policy-bound coordination
	Agent-based	Agent-driven	Planning and iterative execution	High complexity

. Dynamic collaboration can be further characterized by its dominant coordination mechanism, including controller-based collaboration and agent-based collaboration. This hierarchical organization separates the primary distinction in coordination behavior from the mechanisms used to realize adaptive coordination, while allowing different implementations to be analyzed under a common framework.

To ground the proposed taxonomy in representative literature,

Table 3. Evidence mapping of representative collaborative video understanding systems to the proposed framework. Four representative studies are selected for each coordination category. Selected metrics are reported for compactness; not all metrics from the original studies are listed.

Coordination	Study	Functional Units	Communication/Load	State/Memory	Tasks	Benchmarks	Evaluation
Static	Collaborative Experts [77]	Object, scene, motion, face, audio, ASR, and OCR experts; gating	Pairwise feature gating; one-pass	Fused representation; no persistent memory	Video retrieval	MSR-VTT; MSVD; LSMDC; DiDeMo; ActivityNet-Captions	R@K; MdR; MnR
	Video-RAG [78]	LVLM, Whisper, EasyOCR, APE, CLIP, Contriever, FAISS; scene graph	JSON retrieval requests and evidence; one-pass	OCR, ASR, and scene-graph databases	Long-video QA; retrieval	Video-MME; MLVU; LongVideoBench; VNBench	Accuracy; token cost; ablation
	VideoMulti–Agents [79]	Text, video, and scene-graph agents; Organizer	Independent reports and synthesis; parallel inter-agent execution	Agent memory and Organizer context	VideoQA	NExT-QA; IntentQA; EgoSchema	Accuracy; architecture ablation
	MAGNET [80]	AV-RAG, salient-frame selector, AV agents, and meta-agent	Evidence tuples and meta-agent synthesis; parallel map-reduce	Precomputed AV database; aggregation context	Audio-visual multi-video QA	AVHaystacks-50; AVHaystacks-Full	BLEU/CIDEr; Text Sim; GPT Eval; R@K; MTGS; STEM; human evaluation
Controller-based	MoReVQA [81]	Event parsing, grounding, reasoning, and prediction; PaLM-2, PaLI-3, OWL-ViT, CLIP	Structured API calls and grounded outputs; staged adaptive execution	Frame IDs, event queue, QA/OCR flags, and grounded evidence	VideoQA; grounded VideoQA	NExT-QA; iVQA; EgoSchema; ActivityNet-QA; NExT-GQA	Accuracy; Acc@GQA; mIoP; mIoU
	DrVideo [82]	Video-document, retrieval, augmentation, planning, interaction, and answer modules	Binary sufficiency feedback and typed frame requests; iterative	Augmented document and analysis history	Long-video QA	EgoSchema; MovieChat-1K; Video-MME	Accuracy; Gemini-Pro score; ablation
	VideoAgent [17]	GPT-4 controller; LaViLa/CogAgent; EVA-CLIP	JSON actions, confidence, and captions; multi-round	Chronological caption history and retrieved frames	Long-form VideoQA	EgoSchema; NExT-QA	Accuracy; frame count; frame efficiency
	Video-EM [69]	Query agent, CLIP, TransNetV2, Grounding-DINO, Qwen2.5-VL/Qwen3, downstream Video-LLMs	Query decomposition, narratives, and verification; recursive	Episodic event memory and scene relations	Long-video QA; narrative reasoning	Video-MME; LVBench; HourVideo; EgoSchema	Accuracy; frame count; runtime; FLOPs; ablation
Agent-based	OmAgent [19]	Video2RAG, Conqueror, Divider, Rescuer, Tool Manager, Conclusive Synthesis	Typed results, subtasks, and tool outputs; recursive D&C	Knowledge database, timestamps, and task tree	Complex long-video understanding	OmAgent long-video benchmark; MBPP; FreshQA	Accuracy; event localization; summary-QA accuracy
	TraveLER [18]	Planner, retriever, extractor, evaluator, and summarizer	Plans, timestamps, QA, and feedback; iterative replanning	Timestamp-keyed caption/QA memory	Zero-shot VideoQA	NExT-QA; EgoSchema; STAR; Perception Test; Causal-VidQA	Accuracy; frame efficiency; inference cost; ablation
	VideoChat-M1 [83]	Policy agents, retrieval/perception tools, voting, and lead agent	Shared policy memory and tool updates; multi-round	Policies, tool logs, and shared KV buffer	Long-video QA; spatial/temporal reasoning	LongVideoBench; Video-MME; MLVU; Video-Holmes; VSI-Bench; Charades-STA	Accuracy; M-avg/G-avg; category accuracy; mIoU; latency
	HiCrew [84]	Problem/task planning; text, visual, evidence, and answer agents	Workflow specifications, confidence, and tool outputs; ReAct	HybridTree and task state	Long-form VideoQA	EgoSchema; NExT-QA	Accuracy; role/planning/HybridTree ablations

maps selected collaborative video understanding systems to their functional units, communication patterns, state or memory designs, coordination mechanisms, tasks, benchmarks, and evaluation criteria. The mapping illustrates how the proposed coordination categories can be consistently applied across systems with different architectures and application settings. In particular, systems are classified according to their dominant coordination behavior: static collaboration follows a predefined execution policy, whereas dynamic collaboration adapts execution online through either an explicit controller or autonomous agents.

3.2.1. Static Collaboration

Static collaboration represents the most fundamental collaboration paradigm for video understanding [85]. Its defining characteristic is that the coordination policy is predetermined before inference and remains invariant throughout execution [86]. Under this paradigm, collaborative functional units, their invocation order, inter-unit communication topology, and collaborative state evolution process are predefined during system design rather than dynamically adjusted according to the input video, intermediate results, or execution context [87,88,89]. Although the information processed by each unit varies across inputs, the collaborative structure itself remains fixed [90].

Under the unified analytical framework introduced in Section 3.1, static collaboration can be characterized by preconfigured unit assignment [91], execution ordering [92], and inter-unit interaction [93]. Individual collaborative functional units assume fixed responsibilities, such as visual perception, temporal modeling, evidence retrieval [94], semantic fusion, reasoning [95], or language generation [96], while collaboration is realized through predefined inter-unit communication among these units.

Representative implementations include cascaded processing pipelines [97,98], modular multi-stage frameworks [99], retrieval-augmented systems with predefined retrieval-generation workflows [100], and multi-component architectures employing fixed routing [101] among specialized modules. Despite substantial architectural differences, these systems share the same coordination principle: the collaborative strategy is specified before execution and remains unchanged throughout inference.

The primary limitation of static collaboration arises from its input-invariant coordination policy. Since unit selection and interaction patterns cannot adapt to the execution context, the system is unable to dynamically adjust computational resources [102], information routing [103], or reasoning depth according to video complexity [104], semantic ambiguity [105], or intermediate uncertainty [106]. Such limitations become increasingly evident in long-video understanding, multimodal reasoning, and open-ended inference [104], where different inputs often require substantially different collaborative behaviors [107]. These limitations have motivated the development of dynamic collaboration, in which coordination policies are adaptively adjusted during execution.

3.2.2. Dynamic Collaboration

Dynamic collaboration extends static collaboration by allowing the coordination policy to be adaptively adjusted during execution according to the input video, intermediate results, and execution context [17]. Its defining characteristic is that the collaborative structure is no longer fully predetermined before inference but can be partially reconfigured online as the task unfolds [18]. In this survey, we further distinguish two representative implementation mechanisms: controller-based collaboration and agent-based collaboration. Controller-based collaboration relies on an explicit controller, router, scheduler, or policy to determine the participation, ordering, interaction, or computational allocation of functional units, whereas agent-based collaboration introduces autonomous agents that directly participate in planning and coordination.

Controller-Based Collaboration

Controller-based collaboration represents a major implementation mechanism for dynamic coordination in video understanding. Under the unified analytical framework introduced in Section 3.1, dynamic collaboration is characterized by adaptive unit selection, execution scheduling, and inter-unit interaction [108]. In controller-based systems, these adaptive decisions are typically mediated by an explicit controller, router, scheduler, or policy that continuously evaluates the current execution state. Instead of following a fixed coordination topology, the system continuously evaluates the current collaborative state to determine which functional units should be activated, what evidence should be retrieved, how information should be routed, and whether additional computation is required [82]. Individual functional units still assume specialized responsibilities, such as temporal search, retrieval, memory access, perception, reasoning, or response generation, but their participation and interactions vary across different inputs [109]. Collaborative state representations further support adaptive coordination by maintaining intermediate evidence, retrieval history, uncertainty signals, or memory contents, thereby enabling subsequent coordination decisions to be conditioned on the evolving execution state [110].

Representative implementations include adaptive temporal search frameworks that progressively refine observation scope according to query relevance [111], dynamic retrieval-augmented systems that adjust retrieval pathways or evidence granularity according to question complexity and intermediate findings [112], memory-adaptive reasoning frameworks that iteratively determine which memory type or temporal scale should be accessed next [113], and uncertainty-aware inference pipelines that decide whether to continue evidence collection or terminate reasoning according to prediction reliability [114]. StreamChat targets streaming video understanding and multi-round interaction by combining hierarchical short-term, long-term, and dialogue memories with parallel scheduling for frame selection, memory formation, and response generation [115]. Under our taxonomy, it is characterized as controller-based collaboration, as its overall processing structure is predefined while component execution and memory updates adapt to the incoming video stream. Thinking-QwenVL decouples evidence-aware reasoning control from progressive memory integration through an Active Thinking Decision Maker and a Hierarchical Progressive Semantic Integration module, enabling adaptive response timing, confidence monitoring, and cross-clip state updates in online video understanding [116]. Under our taxonomy, it is characterized as controller-based collaboration, with ATDM serving as the reasoning controller and HPSI serving as the memory and integration unit. The two units coordinate through the evolving cognitive state, in which HPSI progressively integrates cross-clip visual evidence while ATDM conditions subsequent decisions on the updated state together with progress and confidence signals, enabling adaptive execution and self-triggered reflection. Although these approaches differ substantially in implementation, they share the same coordination principle: collaborative decisions are determined online according to the current execution state rather than being completely specified before inference. In contrast to fixed one-shot retrieval or single-pass modular pipelines [78,117], these methods explicitly modify the collaborative process during execution.

Compared with static collaboration, controller-based collaboration provides substantially greater flexibility, efficiency, and adaptability [118,119]. By tailoring coordination policies to individual inputs, the system can selectively activate specialized units, focus computation on relevant temporal regions, reduce unnecessary retrieval or reasoning steps, and improve robustness under heterogeneous video understanding demands [120]. Such adaptive coordination is particularly beneficial for long-video understanding, retrieval-intensive reasoning, and multimodal question answering, where different queries often require markedly different observation strategies, memory access patterns, evidence aggregation, and reasoning depth [121].

Nevertheless, controller-based collaboration typically operates within coordination policies whose decision space, optimization objective, and interaction patterns are specified implicitly or explicitly during training or system design [122]. Although the system adapts its collaborative behavior online, such policy-bound coordination limits autonomous planning, open-ended role negotiation, and self-directed coordination among heterogeneous computational units [123]. Consequently, controller-based methods remain primarily adaptive execution frameworks rather than fully autonomous collaborative systems [124]. These limitations motivate agent-based collaboration, in which autonomous agents actively plan, coordinate, and invoke heterogeneous computational resources according to task objectives [125].

Agent-Based Collaboration

Agent-based collaboration represents a more autonomous form of dynamic coordination, allowing autonomous agents to participate in goal decomposition, coordination planning, tool invocation, and iterative self-refinement during execution [17,125]. Under the unified analytical framework introduced in Section 3.1, this paradigm is characterized by agentized collaborative functional units, task-driven inter-unit communication, and policy-driven updates of collaborative state representations. Functional units are no longer restricted to predefined roles but are upgraded into agents capable of subgoal formation, strategy refinement, and execution control. These agents dynamically decide when and how to invoke other agents, tools, or memory components [126].

Inter-unit communication in this paradigm extends beyond intermediate features or retrieved evidence to include high-level semantic signals such as plans, critiques, confidence estimates, and tool-selection actions [127]. Collaborative state representations are correspondingly enriched with reasoning traces, interaction histories, agent feedback, and execution plans, thereby supporting more expressive coordination across agents [128]. In many systems, agents also participate in execution control by deciding whether to continue, terminate, backtrack, or replan. Formally, both the message function $m_{i\to j}^t$ and state evolution process $S_t$ can be conditioned on agent policies defined over evolving goals and observations.

Representative implementations typically organize video understanding as a task-driven multi-agent process rather than a fixed pipeline. Complex tasks are decomposed into subgoals and assigned to specialized agents responsible for retrieval, grounding, reasoning, memory access, or generation [18]. These agents iteratively exchange evidence, plans, and feedback through explicit interaction protocols, enabling progressive refinement of the collaborative workflow [19]. Some methods further introduce reflection, critique, or self-revision mechanisms to filter unreliable intermediate outputs and improve execution robustness [17]. DoraemonGPT further exemplifies this paradigm by constructing task-related symbolic video memory and using an LLM-driven Monte Carlo Tree Search planner to adaptively schedule specialized sub-task tools, external knowledge tools, and utility tools [70]. Under our taxonomy, it represents agent-based collaboration because its tools are selected and sequenced online according to intermediate observations and planning feedback. Despite diverse implementations, these approaches share a common principle: coordination emerges from agent-mediated interactions over goals, evidence, and decisions rather than being fully predefined.

Compared with controller-based collaboration, agent-based collaboration provides stronger autonomy, flexibility, and generalization capability. By enabling agents to perform subgoal planning, tool selection, strategy revision, and iterative reasoning, this paradigm better supports long-horizon reasoning, multimodal grounding, and decision-oriented video understanding tasks [129]. It also offers a unified framework for heterogeneous tasks such as question answering, retrieval, planning, and embodied reasoning [130]. These properties make it particularly effective for long-video understanding scenarios requiring iterative reasoning and multi-step evidence aggregation. Nevertheless, despite its enhanced autonomy, several challenges still remain in practical deployments. These issues are further analyzed in Section 5.

4. Benchmarks, Metrics, and Empirical Analysis

Benchmarks provide the empirical basis for defining video understanding tasks, specifying annotation protocols, and measuring model performance [26]. This section reviews representative benchmarks, evaluation metrics, and empirical evidence across major task categories, focusing on the capabilities they assess, the protocols they adopt, and the comparability of reported results under heterogeneous settings [131,132].

4.1. Representative Benchmarks

This section analyzes representative datasets for major video understanding tasks from a task-oriented perspective [26]. Representative datasets across four benchmark categories are summarized in

Table 4. Representative datasets for major video understanding tasks.

Task	Datasets	Size	Sources	Remarks
R&L	Kinetics [10]	240K clips	CVPR 2017	Broad-scale benchmark evaluating generic human action recognition.
	Something-Something V2 [133]	220.8K videos	ICCV 2017	Fine-grained benchmark emphasizing temporal dynamics and object interaction understanding.
	THUMOS14 [134]	18.4K videos	CVIU 2016	Benchmark supporting action recognition and temporal action localization
	ActivityNet v1.3 [135]	19,994 videos	CVPR 2015	Large-scale untrimmed-video benchmark for activity recognition and temporal action localization.
	AVA [136]	437 clips	CVPR 2018	Dense spatio-temporal benchmark for localized atomic human action recognition.
CMG	MSR-VTT [137]	10K clips	CVPR 2016	Open-domain video-language benchmark for cross-modal retrieval and alignment.
	DiDeMo [138]	10.5K videos	ICCV 2017	Benchmark for natural-language moment localization in unedited personal videos.
	Charades-STA [139]	10K videos	ICCV 2017	Sentence-grounded temporal localization benchmark built on indoor activity videos.
	TACoS [140]	127 videos	TACL 2013	Early cooking-domain benchmark for language-to-video grounding.
	ActivityNet Captions [141]	20K videos	ICCV 2017	Dense event benchmark for temporal grounding with natural-language queries.
	QVHighlights [142]	10.1K videos	NeurIPS 2021	Query-conditioned benchmark with joint moment retrieval and highlight detection.
	NExT-GQA [143]	1.6K videos	CVPR 2024	Visually grounded VideoQA benchmark extending NExT-QA with temporal grounding annotations.
GD	MSVD [144]	2.1K clips	ACL 2011	Early open-domain benchmark for short-video caption generation.
	MSR-VTT [137]	10K clips	CVPR 2016	Large open-domain benchmark for captioning diverse web videos.
	VATEX [145]	41.3K videos	ICCV 2019	Large-scale multilingual video-language dataset with English and Chinese captions.
	ActivityNet Captions [141]	20K videos	ICCV 2017	Multi-event captioning benchmark with temporally localized descriptions.
	YouCook2 [146]	2K videos	AAAI 2018	Procedural video benchmark with step-aligned instructional narrations.
	TVSum [147]	50 videos	CVPR 2015	Topic-diverse benchmark for supervised shot-level video summarization.
RD	TGIF-QA [148]	71.7K GIFs	CVPR 2017	GIF-based VideoQA benchmark for temporal reasoning, counting, and transition understanding.
	MSR-VTT-QA [149]	10K videos	PR 2020	VideoQA benchmark constructed from MSR-VTT videos with large-scale QA annotations.
	NExT-QA [150]	5.4K videos	CVPR 2021	VideoQA benchmark emphasizing causal and temporal reasoning.
	AGQA [151]	9.6K videos	CVPR 2021	Compositional reasoning benchmark built on structured action graphs.
	STAR [152]	22K clips	NeurIPS 2021	Situated reasoning benchmark covering interaction, sequence, prediction, and feasibility.
	Video-MME [132]	900 videos	CVPR 2025	Long-context multimodal video understanding benchmark with visual, audio, and subtitle information.
	EgoSchema [153]	5K QA pairs	NeurIPS 2023	Long-form egocentric VideoQA benchmark with five-way multiple-choice questions over three-minute clips.
	LongVideoBench [154]	3.8K videos	NeurIPS 2024	Long-context video-language benchmark with interleaved video and subtitle inputs for referring reasoning over videos up to one hour.

Note: R&L, CMG, GD, and RD denote Recognition and Localization, Cross-modal Matching and Grounding, Generative Description, and Reasoning and Decision-making, respectively. For THUMOS14, the reported 18.4K videos refer to the complete challenge collection, while the standard temporal action localization setting uses 413 temporally annotated untrimmed videos, including 200 validation videos for training and 213 test videos for evaluation [155].

. The selection is intended to be representative rather than exhaustive, emphasizing benchmarks that are widely adopted, influential in the development of their respective task categories, or particularly relevant to recent collaborative and long-context video understanding research. Additional datasets exist across these task categories but are not individually listed because of limited adoption, narrower task scope, substantial overlap with representative benchmarks, or specialized evaluation objectives. The following discussion focuses on their objectives, annotation characteristics, evaluated capabilities, and limitations for system-level assessment.

• Recognition and Localization Benchmarks evaluate whether systems can identify actions or events and locate their spatial or temporal occurrences. Representative datasets cover action recognition, temporal action localization, and spatio-temporal localization, with annotations ranging from category labels to temporal segments and spatial regions [10,133]. These benchmarks primarily assess visual perception, motion modeling, and temporal aggregation under predefined label spaces and explicit supervision. For collaborative video understanding, they provide evidence for coordinated perceptual and temporal processing, while offering limited evaluation of long-range memory, multi-step reasoning, and adaptive execution [132].
• Cross-modal Matching and Grounding Benchmarks evaluate whether systems can align video content with language or other modalities. Representative datasets cover video-text retrieval, temporal grounding, query-conditioned moment retrieval and highlight detection, as well as visually grounded video question answering, with annotations including paired instances, ranked results, and temporal spans [137,138]. These benchmarks further differ in whether grounding is evaluated through global retrieval, localized temporal evidence, saliency or highlight identification, or explicit answer-evidence alignment. These benchmarks assess multimodal alignment and retrieval capabilities but exhibit substantial heterogeneity in supervision forms, query formulations, annotation granularity, and modality configurations, which limits direct comparison across datasets [156].
• Generative Description Benchmarks evaluate whether systems can generate high-level textual representations from video content. Representative datasets cover video captioning, dense captioning, event description, and summarization, with annotations typically provided as sentence-level or event-level references [157,158]. These benchmarks assess semantic abstraction, temporal organization, and content selection, but their performance interpretation is highly sensitive to reference quality, annotation diversity, and evaluation protocols, complicating fair comparison across datasets.
• Reasoning and Decision-making Benchmarks evaluate whether systems can infer implicit information by integrating temporal, semantic, and multimodal evidence. Representative datasets cover video question answering, causal reasoning, commonsense inference, and decision-oriented tasks, requiring stronger capabilities in evidence integration, memory utilization, and reasoning consistency [131,150,159]. Recent benchmarks further extend this category toward long-form and long-context video understanding, requiring models to integrate evidence over substantially longer temporal horizons and, in some cases, across multiple input modalities such as visual, audio, and subtitle information [132,153,154]. However, variations in task formulations, input configurations, and supervision strategies introduce substantial challenges for fair comparison across benchmarks. In more advanced long-horizon and agentic settings, these tasks further require coherent state maintenance, tool use, planning, and adaptive execution across extended inference processes [160].

4.2. Evaluation Metrics

Evaluation metrics translate benchmark outputs into measurable evidence of model capability. Existing protocols mainly emphasize final task outcomes [28,41], whereas collaborative video understanding further requires process-level evaluation of consistency, efficiency, memory use, adaptive routing, and interaction cost.

• Task Outcome Metrics. Task outcome metrics remain the predominant evaluation paradigm in existing video understanding benchmarks. Recognition tasks commonly rely on accuracy-based measures, such as Top-1 and Top-5 accuracy, whereas localization tasks use overlap-based measures, including mAP under different temporal IoU thresholds and IoU-based matching criteria [161,162]. Cross-modal retrieval tasks typically employ ranking metrics, such as Recall@K, Mean Rank, and Median Rank, while temporal grounding tasks additionally use IoU-based criteria to assess localization accuracy [163,164]. Generation tasks use reference-based metrics, such as BLEU, METEOR, CIDEr, and SPICE, to measure textual similarity and semantic consistency, whereas dense video captioning additionally employs the task-specific SODA_c metric [161,165]. Reasoning-oriented and composite video understanding tasks typically adopt accuracy-based or task-specific measures, including answer accuracy, grounded answer accuracy, category-level accuracy, event localization accuracy, and summary-based question answering accuracy [151]. Decision-oriented settings may additionally use success rate or task completion rate to assess decision effectiveness. These examples are representative rather than exhaustive, as task-specific benchmarks may adopt additional evaluation measures tailored to their particular objectives. Although these metrics effectively characterize task outcomes, they primarily evaluate final predictions under predefined task settings and provide limited insight into system-level behaviors.
• Beyond Task-level Evaluation. Beyond task-level outcomes, recent evaluations have increasingly considered system-level properties, including output quality, consistency, and efficiency [17,166]. Quality and consistency assessments complement conventional task metrics by examining textual or semantic similarity, cross-modal alignment, and temporal consistency, while human and model-based evaluations provide additional assessment for open-ended outputs [167]. Efficiency-oriented criteria further characterize the computational and interaction costs of collaborative systems, including token cost, frame count, frame efficiency, inference cost, runtime, FLOPs, and latency [168]. These measures are particularly relevant to multi-model and agent-based systems, where adaptive coordination may introduce additional computation, iterative interactions, or redundant evidence processing [169]. However, such system-level evaluations remain less standardized than conventional task metrics, and their adoption varies substantially across benchmarks and studies [170].

4.3. Quantitative Performance Trends and Comparative Analysis

This subsection summarizes representative results across four video understanding regimes. Because these results are obtained on different datasets and under heterogeneous metrics, supervision settings, and inference protocols, they should not be interpreted as a unified ranking of overall video understanding capability. We instead use them to identify task-specific performance trends, compare successive technical paradigms, and determine what current evidence can reveal about collaborative video understanding.

• Recognition and Localization.  Table 5 summarizes representative quantitative evidence for both non-collaborative and collaborative methods across recognition and localization tasks. Recognition and localization exhibit the clearest gains across successive technical paradigms. On Kinetics-400, Top-1 accuracy increases from 74.7 for TSM to 86.1 for MViTv2, 90.0 for VideoMAE V2, and 92.1 for InternVideo2. SSv2 shows a similar but increasingly saturated trajectory, improving from 64.3 for TSM to 73.3 for MViTv2 and 77.4 for InternVideo2. For temporal localization, InternVideo2 reaches 41.2 average mAP on ActivityNet v1.3 and 72.0 on THUMOS14, exceeding InternVideo by 2.2 and 0.4 points, respectively. On AVA v2.2, frame-mAP increases from 27.4 for X3D to 42.6 for VideoMAE V2 among the representative non-collaborative methods. The collaborative methods show a more heterogeneous pattern: Feature Hallucination VideoMAE V2 and Feature Hallucination InternVideo2 reach 87.5 and 91.6 on Kinetics-400, respectively, while their SSv2 results are 77.4 and 77.3. Other collaborative methods report task-specific results, including 68.3 for TrAction + V-JEPA 2-L on SSv2, 49.8 for MLLM4WTAL on THUMOS14, and 45.1 for LART-Hiera on AVA v2.2. These results show substantial gains from advances in model architectures and large-scale pre-training, while the relatively small gap among the strongest recent models on several established recognition benchmarks suggests an emerging saturation trend. Collaborative methods, however, do not exhibit a uniform performance advantage, partly because they target different objectives and are evaluated under heterogeneous backbone, training, and computational settings. Therefore, controlled comparisons with matched single-model baselines and computational budgets are needed to isolate and quantify the actual contribution of collaboration.
• Cross-modal Matching and Grounding. Cross-modal matching and grounding show substantial but less uniform gains across the available dataset–metric combinations. On MSR-VTT retrieval, the reported Recall@1 values range from 20.9 for Collaborative Experts and 29.6 for TeachText to 62.8 for InternVideo2 and 78.6 for MAVIS. On Charades-STA, InternVideo2 obtains 70.03 Recall@1, IoU = 0.5, compared with 57.91 for EMTM and 38.4 for Moment-GPT; on QVHighlights, InternVideo2 reaches 49.24 moment-retrieval mAP, compared with 43.63 for UniVTG. As reflected in Table 6, these results demonstrate substantial progress in cross-modal matching and grounding, but the magnitude of improvement remains highly task-dependent. The mixed results of collaborative methods further suggest that the effectiveness of collaboration depends on how the collaboration mechanism aligns with the specific cross-modal objective and evaluation protocol.

Table 5. Representative quantitative evidence across recognition and localization tasks.

Method Type	Method	Video Recognition		Temporal Localization		Spatio-Temporal Action Localization
		Kinetics-400 Top-1	SSv2 Top-1	ActivityNet v1.3 Avg. mAP	THUMOS14 Avg. mAP	AVA v2.2 Frame-mAP
Non-collaborative	TSM [171]	74.7	64.3	–	–	–
	X3D [172]	79.1	–	–	–	27.4
	UniFormer [173]	83.0	71.4	–	–	–
	Video Swin [174]	84.9	69.6	–	–	–
	MViTv2 [175]	86.1	73.3	–	–	34.4
	MaskFeat [176]	87.0	75.0	–	–	39.8
	VideoPrism [177]	87.2	68.5	37.8	–	–
	VideoMAE [178]	87.4	75.4	–	–	39.5
	VideoMAE V2 [179]	90.0	77.0	–	69.6	42.6
	InternVideo [180]	91.1	77.2	39.0	71.6	41.0
	InternVideo2 [181]	92.1	77.4	41.2	72.0	–
Collaborative	TrAction + DINOv2-B [182]	–	62.3	–	–	–
	TrAction + V-JEPA 2-L [182]	–	68.3	–	–	–
	LART-Hiera [183]	–	–	–	–	45.1
	GAP + CLIP [184]	–	–	31.8	32.9	–
	MLLM4WTAL [185]	–	–	–	49.8	–
	JEDI Student [186]	82.08	–	–	–	–
	Feature Hallucination VideoMAE V2 [187]	87.5	77.4	–	–	–
	Feature Hallucination InternVideo2 [187]	91.6	77.3	–	–	–

Table 5. Representative quantitative evidence across recognition and localization tasks.

Method Type	Method	Video Recognition		Temporal Localization		Spatio-Temporal Action Localization
		Kinetics-400 Top-1	SSv2 Top-1	ActivityNet v1.3 Avg. mAP	THUMOS14 Avg. mAP	AVA v2.2 Frame-mAP
Non-collaborative	TSM [171]	74.7	64.3	–	–	–
	X3D [172]	79.1	–	–	–	27.4
	UniFormer [173]	83.0	71.4	–	–	–
	Video Swin [174]	84.9	69.6	–	–	–
	MViTv2 [175]	86.1	73.3	–	–	34.4
	MaskFeat [176]	87.0	75.0	–	–	39.8
	VideoPrism [177]	87.2	68.5	37.8	–	–
	VideoMAE [178]	87.4	75.4	–	–	39.5
	VideoMAE V2 [179]	90.0	77.0	–	69.6	42.6
	InternVideo [180]	91.1	77.2	39.0	71.6	41.0
	InternVideo2 [181]	92.1	77.4	41.2	72.0	–
Collaborative	TrAction + DINOv2-B [182]	–	62.3	–	–	–
	TrAction + V-JEPA 2-L [182]	–	68.3	–	–	–
	LART-Hiera [183]	–	–	–	–	45.1
	GAP + CLIP [184]	–	–	31.8	32.9	–
	MLLM4WTAL [185]	–	–	–	49.8	–
	JEDI Student [186]	82.08	–	–	–	–
	Feature Hallucination VideoMAE V2 [187]	87.5	77.4	–	–	–
	Feature Hallucination InternVideo2 [187]	91.6	77.3	–	–	–

Table 6. Representative quantitative evidence across cross-modal matching and grounding tasks.

Method Type	Method	Video-Text Retrieval	Temporal Grounding		Moment Retrieval
		MSR-VTT Recall@1	Charades-STA Recall@1, IoU = 0.5	ActivityNet Captions Recall@1, IoU = 0.5	QVHighlights mAP
Non-collaborative models	LGI [188]	–	59.46	41.51	–
	Moment-DETR [142]	–	55.65	–	36.14
	UniVTG [189]	–	60.19	–	43.63
	QD-DETR [190]	–	57.31	–	40.19
	VideoCLIP [191]	30.9	–	–	–
	Frozen in Time [192]	32.5	–	–	–
	CLIP4Clip [193]	44.5	–	–	–
	InternVideo2 [181]	62.8	70.03	–	49.24
Collaborative models	Moment-GPT [194]	–	38.4	31.1	35.0
	EMTM [195]	–	57.91	44.73	–
	Collaborative Experts [77]	20.9	–	–	–
	TeachText [196]	29.6	–	–	–
	Learning from Image Captioning [197]	39.2	–	–	–
	MAVIS [198]	78.6	–	–	–

Table 6. Representative quantitative evidence across cross-modal matching and grounding tasks.

Method Type	Method	Video-Text Retrieval	Temporal Grounding		Moment Retrieval
		MSR-VTT Recall@1	Charades-STA Recall@1, IoU = 0.5	ActivityNet Captions Recall@1, IoU = 0.5	QVHighlights mAP
Non-collaborative models	LGI [188]	–	59.46	41.51	–
	Moment-DETR [142]	–	55.65	–	36.14
	UniVTG [189]	–	60.19	–	43.63
	QD-DETR [190]	–	57.31	–	40.19
	VideoCLIP [191]	30.9	–	–	–
	Frozen in Time [192]	32.5	–	–	–
	CLIP4Clip [193]	44.5	–	–	–
	InternVideo2 [181]	62.8	70.03	–	49.24
Collaborative models	Moment-GPT [194]	–	38.4	31.1	35.0
	EMTM [195]	–	57.91	44.73	–
	Collaborative Experts [77]	20.9	–	–	–
	TeachText [196]	29.6	–	–	–
	Learning from Image Captioning [197]	39.2	–	–	–
	MAVIS [198]	78.6	–	–	–

• Generative Description. Generative description exhibits substantial progress, although the strongest method varies across datasets and metrics. Table 7 summarizes representative results from both non-collaborative and collaborative methods across global and dense video captioning. On MSVD, CIDEr increases from 49.8 for RETTA and 51.7 for Enc–Dec + Local + Global to 146.2 for Vid2Seq, while Vid2Seq reaches 64.6 on MSR-VTT compared with 53.8 for both SwinBERT and Multiple Networks Consensus + OracleNet. Among the listed methods, no single method consistently dominates all ActivityNet Captions metrics: Masked Transformer records the highest METEOR score of 9.56, while Vid2Seq achieves the highest CIDEr and SODA_c scores of 30.10 and 5.80, respectively. The collaborative methods span both global and dense captioning, with APML Ensemble reaching 109.5 CIDEr on MSVD and ECG obtaining 7.06 METEOR on ActivityNet Captions. Overall, the metric- and method-dependent variations indicate that captioning progress cannot be represented by a single performance ranking. The mixed performance of collaborative methods further suggests that collaboration is beneficial only when its design effectively supports the specific generation objective and evaluation criterion.

Table 7. Representative quantitative evidence across generative description tasks categorized by collaboration type.

Method Type	Method	Global Video Captioning			Dense Video Captioning
		MSVD	MSR-VTT	VATEX	ActivityNet Captions
		CIDEr	CIDEr	CIDEr	METEOR	CIDEr	SODAc
Non-collaborative	Dense-Captioning Events [141]	–	–	–	4.82	17.29	–
	Masked Transformer [199]	–	–	–	9.56	–	–
	PDVC [200]	–	–	–	7.50	25.87	5.26
	Enc–Dec + Local + Global [201]	51.7	–	–	–	–	–
	SwinBERT [202]	120.6	53.8	73.0	–	–	–
	Vid2Seq [161]	146.2	64.6	–	8.50	30.10	5.80
Collaborative	Multiple Networks Consensus + OracleNet [203]	–	53.8	–	–	–	–
	Weakly Supervised Dense Event Captioning [204]	–	–	–	6.30	18.77	–
	ECG [205]	–	–	–	7.06	14.25	–
	RETTA [206]	49.8	24.3	23.8	–	–	–
	APML Ensemble [207]	109.5	52.7	–	–	–	–

Table 7. Representative quantitative evidence across generative description tasks categorized by collaboration type.

Method Type	Method	Global Video Captioning			Dense Video Captioning
		MSVD	MSR-VTT	VATEX	ActivityNet Captions
		CIDEr	CIDEr	CIDEr	METEOR	CIDEr	SODAc
Non-collaborative	Dense-Captioning Events [141]	–	–	–	4.82	17.29	–
	Masked Transformer [199]	–	–	–	9.56	–	–
	PDVC [200]	–	–	–	7.50	25.87	5.26
	Enc–Dec + Local + Global [201]	51.7	–	–	–	–	–
	SwinBERT [202]	120.6	53.8	73.0	–	–	–
	Vid2Seq [161]	146.2	64.6	–	8.50	30.10	5.80
Collaborative	Multiple Networks Consensus + OracleNet [203]	–	53.8	–	–	–	–
	Weakly Supervised Dense Event Captioning [204]	–	–	–	6.30	18.77	–
	ECG [205]	–	–	–	7.06	14.25	–
	RETTA [206]	49.8	24.3	23.8	–	–	–
	APML Ensemble [207]	109.5	52.7	–	–	–	–

• Reasoning and Decision-making. Reasoning-oriented tasks show the largest apparent gains but also the greatest sensitivity to evaluation protocols. Table 8 summarizes representative results from non-collaborative and collaborative methods across video question answering, situated reasoning, and long-context multimodal understanding. On STAR, the reported accuracies range from 33.3 for HCRN and 36.8 for ClipBERT to 44.9 for TraveLER, although the latter is obtained under zero-shot evaluation. On Video-MME, VILA-1.5-34B reports 59.0 overall accuracy, while the collaborative methods Video-EM, A4VL, LVAgent, and VideoChat-M1 report 62.0, 77.2, 81.7, and 83.2, respectively; however, the subtitle setting is not explicitly specified for VideoChat-M1. MSR-VTT-QA also requires caution because the available results are reported under different methodological settings, and the table does not support a direct ranking across all methods. More broadly, these results do not isolate whether an improvement arises from visual representation, language-model scale, prompting, memory, or collaboration among functional units. Nevertheless, the stronger results of collaborative methods on Video-MME suggest that explicit coordination may be particularly beneficial for long-context reasoning, where complementary perception, memory, and decision processes must be integrated. This apparent advantage should, however, be interpreted cautiously because the non-collaborative evidence in the current comparison remains limited.
• Cross-benchmark Comparability. Reliable comparison is restricted to results obtained on the same dataset with the same metric and evaluation protocol. Even within an individual column, differences in pre-training data, supervision resources, input modalities, model scale, inference configurations, and collaboration mechanisms can confound attribution. This limitation is particularly evident for reasoning-oriented tasks, where the reported results include both conventional evaluations and zero-shot settings, and the subtitle condition is not uniformly specified. The four tables should therefore be read as evidence of task-specific capability trends rather than as a global leaderboard of video understanding systems or a uniform ranking of collaborative versus non-collaborative approaches.
• Implications for Collaborative Video Understanding. Across the four task regimes, performance gains appear most consistent on standardized recognition benchmarks, whereas grounding, generation, and reasoning exhibit stronger dataset, metric, and protocol dependence. The results also document a broad transition from task-specific architectures to large-scale pre-trained and foundation models, alongside an emerging body of collaborative approaches. However, the quantitative evidence does not establish a corresponding improvement attributable to collaboration across tasks. Existing metrics primarily evaluate final task outputs and do not isolate the contributions of functional-unit selection, inter-unit communication, shared-state maintenance, or adaptive execution. Consequently, Table 5, Table 6, Table 7 and Table 8 provide evidence of task-level capability evolution and the emerging quantitative presence of collaborative methods rather than direct evidence of collaboration effectiveness. Collaboration-centric evaluation should complement outcome metrics with matched-cost single-model baselines, unit-level ablations, communication and routing efficiency, memory fidelity, uncertainty calibration, and robustness to component failures.

Table 8. Representative quantitative evidence across reasoning and decision-making tasks categorized by collaboration type.

Method Type	Method	Paradigm	Video Question Answering	Situated Reasoning	Long-Context Multimodal Understanding
			MSR-VTT-QA Standard Accuracy	STAR Accuracy	Video-MME Overall Accuracy
Non-collaborative	VILA-1.5-34B [208]	Video-capable VLM	–	–	59.0
	HME [209]	Heterogeneous multimodal attention	33.0	–	–
	HCRN [210]	Conditional relation network	35.6	33.3 †	–
	ClipBERT [211]	Sparse-sampling Transformer	37.4	36.8 †	–
	HQGA [212]	Query-conditioned graph hierarchy	38.6	–	–
	MERLOT [213]	Temporal multimodal Transformer	43.1	–	–
	VIOLET [214]	Video-language Transformer	43.9	–	–
Collaborative	Video-EM [69]	Multi-model tool orchestration with episodic memory	–	–	62.0
	A4VL [215]	Multi-agent perception-action alliance	–	–	77.2
	LVAgent [169]	Multi-round MLLM collaboration	–	–	81.7
	VideoChat-M1 [83]	Multi-agent reinforcement learning	–	–	83.2 *
	TraveLER [18]	Modular multi-LMM agents	–	44.9 ‡	–
	RADI-P [216]	Training-stage teacher–student ranking distillation	48.2	–	–

Note: ^† denotes macro-averaged STAR accuracy over the four question types; ^‡ denotes zero-shot STAR evaluation; ^* indicates that the subtitle setting is not explicitly specified for VideoChat-M1. Video-MME values are overall accuracies.

Table 8. Representative quantitative evidence across reasoning and decision-making tasks categorized by collaboration type.

Method Type	Method	Paradigm	Video Question Answering	Situated Reasoning	Long-Context Multimodal Understanding
			MSR-VTT-QA Standard Accuracy	STAR Accuracy	Video-MME Overall Accuracy
Non-collaborative	VILA-1.5-34B [208]	Video-capable VLM	–	–	59.0
	HME [209]	Heterogeneous multimodal attention	33.0	–	–
	HCRN [210]	Conditional relation network	35.6	33.3 †	–
	ClipBERT [211]	Sparse-sampling Transformer	37.4	36.8 †	–
	HQGA [212]	Query-conditioned graph hierarchy	38.6	–	–
	MERLOT [213]	Temporal multimodal Transformer	43.1	–	–
	VIOLET [214]	Video-language Transformer	43.9	–	–
Collaborative	Video-EM [69]	Multi-model tool orchestration with episodic memory	–	–	62.0
	A4VL [215]	Multi-agent perception-action alliance	–	–	77.2
	LVAgent [169]	Multi-round MLLM collaboration	–	–	81.7
	VideoChat-M1 [83]	Multi-agent reinforcement learning	–	–	83.2 *
	TraveLER [18]	Modular multi-LMM agents	–	44.9 ‡	–
	RADI-P [216]	Training-stage teacher–student ranking distillation	48.2	–	–

Note: ^† denotes macro-averaged STAR accuracy over the four question types; ^‡ denotes zero-shot STAR evaluation; ^* indicates that the subtitle setting is not explicitly specified for VideoChat-M1. Video-MME values are overall accuracies.

5. Challenges and Future Directions

Collaborative video understanding has been increasingly explored as a system-level paradigm to address the limitations of monolithic video models [16]. However, achieving effective collaboration requires systematic optimization of three fundamental system-level components: collaborative functional units, inter-unit communication mechanisms, and collaborative state representations. From this perspective, we identify six challenges that are critical to advancing collaborative video understanding.

5.1. Adaptive Task Decomposition and Functional Units Orchestration

Existing collaborative systems are often characterized by manually specified functional roles and relatively fixed execution workflows. For example, VideoMultiAgents assigns visual, textual, and scene-graph reasoning to different agents and aggregates their predictions through an organizer agent [79]. LongVideoAgent employs a master agent to coordinate grounding and visual perception agents for long-video understanding [217]. Although these designs provide effective structural priors, predefined decomposition strategies may struggle to accommodate videos and queries with diverse reasoning requirements. Future collaborative systems should explore adaptive task decomposition and functional unit orchestration through hierarchical planners that progressively decompose and refine complex tasks [19], learned routers that select functional units according to query characteristics, and uncertainty-aware mechanisms that invoke additional perception or verification when required [18]. Beyond adaptive routing, self-reconfigurable collaboration strategies are also essential for dynamically adjusting system topology when functional units exhibit failures, generate unreliable evidence, or encounter out-of-distribution inputs.

5.2. Efficient and Semantically Aligned Inter-Unit Communication

Collaborative video understanding requires the exchange of heterogeneous messages among functional units, including visual tokens, video clips, audio segments, subtitles, temporal intervals, scene graphs, natural-language descriptions, tool outputs, and intermediate predictions. These messages vary substantially in modality, temporal resolution, spatial granularity, and semantic abstraction. Without explicit alignment mechanisms, these messages may be difficult to integrate and may correspond to inconsistent temporal or semantic contexts [16]. A promising direction is to develop unified multimodal communication protocols in which each message encodes semantic content together with its temporal support, spatial support, source identity, and confidence. Adaptive-bandwidth communication is another important direction, where low-resolution summaries are exchanged initially and higher-resolution evidence is selectively requested when uncertainty remains high [114]. Multimodal interfaces that preserve visual evidence while maintaining linguistic abstraction may provide a more reliable basis for collaboration than purely language-based communication [66].

5.3. Persistent and Temporally Consistent Shared Memory

Shared or mediated states allow functional units to preserve task-relevant context across multiple interactions. However, constructing such states for long videos remains challenging because systems must determine what information should be retained, organized, updated, or discarded. Early approaches compress dense video tokens into sparse short- and long-term memories [160], while agent-based systems maintain structured temporal and object-centric memories that can be accessed through specialized tools. These mechanisms improve scalability, but aggressive compression may remove brief yet decisive visual evidence [218]. Conversely, retaining all frames or intermediate conclusions increases computational overhead and undermines the efficiency objective of memory mechanisms. Future collaborative systems should employ hierarchical, multimodal, and multi-timescale memory mechanisms [219]. Such designs may combine fine-grained visual memory for object attributes and local interactions, episodic memory for temporally localized events, and semantic memory for persistent relations and high-level knowledge.

5.4. Uncertainty-Aware Collaboration and Error Containment

Collaboration creates additional pathways through which errors can propagate. Errors produced by one functional unit may propagate to subsequent units, resulting in incorrect evidence retrieval or unsupported reasoning outcomes. Because many functional units are built upon similar foundation models or training data, their errors may also be correlated. Agreement among units therefore does not necessarily imply correctness, and a central organizer may inadvertently amplify a shared failure rather than resolve it [220]. Uncertainty-aware collaboration should incorporate disagreement detection, confidence calibration, evidence-level verification, and selective re-observation [120]. When conflicts arise, the system should identify the disputed evidence and acquire additional observations rather than resolve disagreements solely through linguistic priors [221].

5.5. Evidence-Grounded and Verifiable Collaborative Reasoning

A collaborative system may produce a correct answer without grounding it in the underlying visual evidence, particularly when language priors or subtitles provide plausible shortcuts. Conversely, the retrieved video evidence may be relevant while the intermediate reasoning process remains inconsistent with the underlying observations. Final-answer accuracy alone therefore does not establish whether the collaborative reasoning process is faithful [143]. This issue becomes more severe in long videos, where relevant events are often sparse and temporally dispersed, increasing the risk of reasoning from incomplete summaries or incorrectly retrieved evidence. Future systems should interleave retrieval, inspection, reasoning, and verification rather than treating perception and reasoning as isolated stages. Each intermediate claim should be linked to the supporting frame, clip, audio segment, or subtitle from which it was derived [222].

5.6. Collaboration-Centric Evaluation and Scalable Deployment

Existing video benchmarks have substantially expanded the coverage of video duration, task diversity, and multimodal inputs. Recent benchmarks have introduced long-context reasoning, multimodal understanding, online perception, and hallucination evaluation settings, including LongVideoBench [154], Video-MME [132], OVO-Bench [223], and VidHalluc [224]. However, these benchmarks primarily evaluate externally observable task outputs. They do not directly assess whether functional units provide complementary capabilities, whether exchanged messages contribute necessary information, whether shared states preserve decisive evidence, or whether collaboration yields improvements under comparable computational budgets. Collaboration-centric evaluation should therefore consider task performance, evidence grounding, unit contribution, communication efficiency, uncertainty calibration, robustness, and system cost [225]. Specifically, collaboration gains should be evaluated against strong individual or computationally matched baselines, unit contributions through ablations or marginal effects, communication efficiency through interaction volume, transmitted information, and latency, and adaptive execution through comparisons with fixed strategies under matched computational budgets [226]. Collaborative state and memory should further be assessed for evidence retrieval, temporal consistency, stale information, and conflict handling, complemented by oracle and random routing, message perturbation, and failure-robustness analyses [227]. Nevertheless, existing video benchmarks generally lack standardized protocols for these collaboration-specific dimensions, leaving the development of collaboration-oriented benchmarks that jointly evaluate task performance, collaboration effectiveness, and computational efficiency as an open research gap.

6. Conclusions

Recent advances in video understanding have expanded the field from perception-oriented recognition toward semantic interpretation, multimodal reasoning, and decision-making over dynamic visual content. This survey examined this transition from the perspective of collaborative video understanding, where specialized functional units, inter-unit communication mechanisms, and shared state representations are coordinated as a system-level framework for addressing heterogeneous video understanding tasks. Through this perspective, we reviewed existing methods under a unified analytical framework and organized them into two overarching paradigms: static collaboration and dynamic collaboration, with the latter further divided into controller-based and agent-based collaboration. This taxonomy reveals a broader evolution from predefined execution structures toward adaptive and increasingly autonomous coordination. We also analyzed representative benchmarks, metrics, and empirical evidence, showing that current evaluation protocols mainly capture task-level outcomes and remain insufficient for assessing collaboration quality, memory use, adaptive execution, evidence grounding, and system-level robustness. Advancing collaborative video understanding will require progress not only in the capabilities of individual models but also in the principles governing how heterogeneous functional units are selected, coordinated, updated, verified, and evaluated. Key research directions include adaptive task decomposition, semantically aligned inter-unit communication, persistent and temporally consistent memory, uncertainty-aware error containment, evidence-grounded reasoning, and collaboration-centric evaluation under realistic computational constraints. By consolidating existing progress and identifying these unresolved challenges, this survey aims to provide a structured foundation for developing collaborative video understanding systems that are adaptive, reliable, interpretable, and scalable.