Generative AI for Text-to-Video Generation: Recent Advances and Future Directions

Abstract

Text-to-video (T2V) generation has recently emerged as a transformative technology within the field of generative AI, enabling the creation of realistic, temporally coherent videos based on natural language descriptions. This paradigm provides significant added value in many domains such as creative media, human-computer interaction, immersive learning, and simulation. Despite its growing importance, systematic discussion of T2V is still limited compared with adjacent modalities such as text-to-image and image-to-video. To alleviate the scarcity of discussions in the T2V field, this paper provides a systematic review of works published from 2024 onward, consolidating fragmented contributions across the field. We survey and categorize the selected literature into three principal areas—namely, T2V methods, datasets, and evaluation practices—and further subdivide each area into subcategories that reflect recurring themes and methodological patterns in the literature. Emphasis is then placed on identifying key research opportunities and open challenges that need further investigation.

1. Introduction

Recent advances in generative artificial intelligence (AI) are reshaping the production of multimedia content, including text, images, music, and video. A promising frontier is text-to-video (T2V) generation, which has gained growing popularity in recent years. Video synthesis has become a core driver of this popularity, since it enables the creation of realistic videos with high visual fidelity and coherent temporal transitions. Among the technological frameworks that supported this progress are diffusion models, which have shown remarkable success in generating high-quality video content; this is exemplified by Sora’s groundbreaking video generation [1], which demonstrates the performance of Diffusion Transformer (DiT) architectures. Moreover, the emergence of alternative generative paradigms like autoregressive models (e.g., Video Pixel Networks [2]) and variational autoencoders (VAEs; e.g., Stochastic Video Generation (SVG) [3]) has also advanced this field. Whether trained from scratch or fine-tuned from pre-existing architectures, these models have rejuvenated research in T2V generation.

Alongside synthesis, video understanding plays a supporting role in enabling controllable, interpretable, and evaluable T2V systems. Rather than serving as an independent objective, video understanding techniques contribute to tasks such as verifying semantic alignment between generated videos and input text, maintaining temporal and narrative consistency, and supporting post-generation analysis and refinement. Integrating Large Language Models (LLMs) like GPT [4] or LLaMA [5,6] with visual encoders [7,8] in vision and language modeling has significantly improved performance on video understanding tasks such as caption generation, action recognition, and temporal event localization. Another pillar in this field is objective assessment and benchmarking to evaluate key aspects of the generated videos like visual realism, text–video alignment, and alignment with human preferences, which enables measuring and guiding progress in T2V research. Another active research direction involves developing benchmark datasets, ranging from short-form video collections to more complex and prompt-driven datasets.

In order to enrich the literature on T2V generation, this paper provides a comprehensive analysis of recent advances in video synthesis, together with a review of evaluation frameworks and benchmark datasets and a discussion of video understanding methods insofar as they support generation and assessment. In this paper, we focus on works published from 2024 onward. This temporal focus is intentional and reflects a paradigm shift in T2V research. Prior studies published before 2024 have been extensively reviewed and largely emphasize short-duration video synthesis, GAN-based architectures, or early extensions of image diffusion models with limited temporal reasoning [9,10,11,12]. In contrast, post-2024 research reflects a reorganization of the field, in which video generation, video understanding, and evaluation increasingly co-evolve within unified foundation-model pipelines. Accordingly, we review the current state of the field through the lens of the following questions:

• RQ1: What are the key technological advances that have driven progress in the aforementioned video research fields?
• RQ2: What are the current challenges and best practices in evaluating T2V models, and how do benchmark datasets support the development of robust T2V generators?
• RQ3: What are the primary technical challenges and future research directions in leveraging Generative AI and LLMs for T2V generation?

Throughout this review, performance-related claims are reported as described in the original studies. It is important to note that evaluation practices in T2V research remain highly heterogeneous, spanning automated metrics, custom benchmarks, and often small-scale human studies. Dataset sizes, prompt distributions, and baseline selections vary substantially across works, and independent replication is rare. Consequently, reported improvements should be interpreted within the scope of each study’s evaluation protocol rather than as universally established gains. In light of these considerations, the review emphasizes comparative trends, design principles, and methodological patterns over definitive performance rankings.

The remainder of this paper is structured as follows. Section 2 presents an overview of LLMs and generative AI, including variational autoencoders (VAEs), autoregressive models, and Diffusion Transformers (DiTs), as well as the pre-training and transfer learning paradigms that underpin them. Section 3 describes the research methodology in this review, including the research strategy and inclusion criteria. In Section 4, statistical analyses of the selected literature are presented, covering publication types, industrial sectors, shop typology, and other relevant factors. Section 5 presents a classification and literature review of the selected publications. Section 7 discusses the main findings derived from the reviewed papers and identifies future research needs.

2. Background

It is imperative to understand the background of generative AI and LLMs to contextualize their performance and versatility in T2V generation. This section presents a brief discussion of generative AI and LLM architectures and their main characteristics. Moreover, it briefly covers the fundamental learning paradigms that support these models; namely, pretraining and transfer learning.

2.1. Large Language Models and Generative Architectures

Generative AI has enabled machines to generate coherent, high-quality outputs across many modalities, such as text, images, audio, and video. Within this framework, deep learning-based generative models, including diffusion-based models, autoregressive models, and variational autoencoders (VAEs), have shown high performance in generating realistic, multimodal content across a wide range of applications. In order to generate new, realistic samples that mimic those from the original dataset, generative AI models attempt to learn the underlying data distribution of a given modality.

Variational Autoencoders (VAEs) are generative models that use two networks: an encoder network and a decoder network. While the encoder network learns to approximate the posterior distribution of latent variables, the decoder network learns to reconstruct the original input from the latent representation. This framework enables VAEs to generate samples that are consistent with the underlying data distribution by optimizing a variational lower bound on the data likelihood. In the context of video generation, VAEs have been extended to work with spatiotemporal data by modeling temporal dependencies within the latent space by using temporal transformers or recurrent layers for sequence modeling, or by introducing hierarchical latent structures. For example, the Stochastic Video Generation (SVG) framework [3] uses recurrent latent variables to capture the inherent uncertainty and dynamics in video sequences, thereby enabling the generation of multiple futures conditioned on previous frames.

Diffusion models are another type of generative model that have recently attracted increasing attention in video generation. Unlike VAEs and autoregressive models, diffusion models generate data through a gradual denoising process. These models learn to reverse a fixed noising procedure that incrementally corrupts the data—typically by adding Gaussian noise over multiple steps—until it becomes pure noise. This denoising process is often parameterized by conditional neural networks—commonly U-Nets or transformer architectures—that predict either the noise added at each step or the clean data itself. Training is generally performed by optimizing a denoising score-matching objective. In video generation, diffusion models have been extended to capture both spatial and temporal correlations through various architectural designs. For instance, Video Diffusion Models (VDMs) [13] incorporate 3D convolutional denoising networks, as well as motion decoders and temporal attention modules, enabling the model to operate on video clips as spatiotemporal volumes.

Autoregressive models are another salient class of generative models that create data one element at a time by modeling the conditional probability of each element given the preceding ones. These models are trained to maximize the likelihood of observed sequences, typically by minimizing their negative log-likelihood. In video generation, autoregressive models usually synthesize video frames in a fixed temporal order, frame by frame or pixel by pixel, using previously generated content as a guide. Modern autoregressive video models frequently incorporate temporal transformers and attention layers to effectively model long-range temporal relationships and dependencies among frames. A notable example includes Video Pixel Networks (VPNs) [2], which decompose video generation into conditional distributions over spatiotemporal patches. Architecturally, VPNs extend PixelCNN [14] with convolutional Long Short-Term Memory (LSTM) to model both spatial and temporal dependencies. Specifically, VPNs factorize video distributions into a four-dimensional dependency chain corresponding to spatial width, spatial height, temporal frames, and color channels. Each pixel in a frame is generated based on previously generated pixels within the same frame, as well as pixels from earlier frames, thereby capturing both intra-frame and inter-frame correlations.

2.2. Pre-Training and Transfer Learning Paradigms

In order to generate temporally coherent video sequences, T2V models are commonly trained using supervised learning on large-scale captioned video datasets, such as LAION-5B [15], Panda-70M [16], and WebVid-10M [17]. In these corpora, video–text examples serve as labeled data guiding the model to learn how to map textual descriptions (prompts) to corresponding visual and temporal features. Two main training paradigms are widely used.

• Training from Scratch: The model weights are randomly initialized, and all spatial and temporal patterns are learned entirely from large-scale paired video–text datasets. This method enables full flexibility when designing model architectures (e.g., spatiotemporal transformers or 3D convolutional networks) [18].
• Fine-Tuning: In this paradigm, the model is initialized with pretrained weights and then adapted to the video domain through further training on paired video–text datasets. Fine-tuning involves updating the entire model or selectively updating specific modules (e.g., temporal transformers or attention layers). Techniques such as lightweight adapters, Low-Rank Adaptation (LoRA) layers [19], or partial-freeze schedules [20] (e.g., freezing the text encoder and early spatial layers while updating temporal transformer blocks) are often used to reduce computational cost.

Due to the high computational cost and data requirements of training from scratch and the growing availability of pretrained T2V backbones, fine-tuning has emerged as the most popular method. Furthermore, the spatial and semantic priors encoded in the pretrained models enable faster convergence, more efficient adaptation to temporal dynamics, and high-quality video generation with fewer resources [21].

3. Research Methodology

This study adopts a systematic literature review (SLR) methodology due to its rigor in identifying, evaluating, and synthesizing all relevant research related to a specific research question, domain, or phenomenon. An SLR enables evidence-based synthesis and identification of research gaps and research opportunities within the existing body of work [22]. Following established guidelines, the SLR process is structured into three main phases: planning, conducting, and reporting the review, each involving several methodological steps as illustrated in Figure 1.

3.1. Review Protocol and Research Questions

This systematic literature review was conducted in alignment with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines [23], (Supplementary Materials). The review protocol was prospectively registered on the Open Science Framework (OSF) under the registration identifier 29vna and is publicly accessible at https://osf.io/29vna (accessed on 16 February 2026). The registered protocol outlines the review objectives, research questions, eligibility criteria, search strategy, and planned data extraction procedures.

No substantive deviations from the registered protocol were made during the conduct of the review. Minor refinements were limited to clarifying inclusion criteria wording and improving the categorization structure of reviewed studies, without altering the predefined research questions or search strategy.

As stated in Section 1, the objective of the present research is to scrutinize the current status of research on T2V generation and identify potential research directions. Research questions were formulated in Section 1. Research question 1 (RQ1) aims to identify the main innovations that have accelerated progress in T2V, including advances in T2V models and video–text datasets. Research question 2 (RQ2) examines the evaluation landscape for T2V models, focusing on both persistent challenges and emerging best practices. Finally, research question 3 (RQ3) aims to identify the challenges and future research opportunities in T2V generation.

3.2. Search Strategy

To collect a comprehensive range of publications, data were gathered from two prominent academic databases: Google Scholar and arXiv. All English-language publications registered as journal articles, conference proceedings, or established preprints were included, with no specific restrictions on scientific journals. Searches were performed using combinations of the following keywords: “text-to-video,” “text to video,” “T2V,” “video generation,” “video synthesis,” and “generative video”. Queries were applied mainly to titles and abstracts to ensure relevance, while full-text inspection was subsequently used during the screening stage.

The search was restricted to works published from January 2024 onward, consistent with the stated scope of the review. The initial searches were conducted between 24 October 2025 and 28 October 2025. After removing duplicates, papers were screened based on relevance to text-to-video generation, excluding works without technical contributions, as well as studies that focused on image-to-video generation, video prediction without text conditioning, or non-generative video analysis.

3.3. Study Selection

The initial database searches returned 78 records from Google Scholar and arXiv. After removing duplicate entries, 76 unique records remained for screening. Title and abstract screening was then performed to remove papers that were clearly outside the scope of T2V generation, resulting in the exclusion of 4 records.

The remaining 72 papers underwent full-text review to assess methodological relevance and alignment with the inclusion criteria. During this stage, 3 papers were excluded due to a lack of generative content or insufficient technical detail. The final corpus consisted of 69 publications, which are listed in

Table A1. Selected literature.

	Authors	Publication Title
1	Bahmani et al. [62]	AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers.
2	Bao et al. [38]	Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models.
3	Cai et al. [68]	DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation.
4	Chen et al. [82]	ShareGPT4Video: Improving Video Understanding and Generation with Better Captions.
5	Chen et al. [52]	VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models.
6	Choi et al. [73]	We’ll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedback.
7	Cuttano et al. [79]	SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation.
8	Dalal et al. [37]	One-Minute Video Generation with Test-Time Training.
9	Fei et al. [47]	Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs.
10	Gal et al. [56]	Breathing Life Into Sketches Using Text-to-Video Priors.
11	Girdhar et al. [51]	Factorizing Text-to-Video Generation by Explicit Image Conditioning.
12	Guo et al. [64]	SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models.
13	Guo et al. [96]	Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models.
14	Guo et al. [97]	T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation.
15	Henschel et al. [33]	StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text.
16	Huang et al. [89]	VBench: Comprehensive Benchmark Suite for Video Generative Models.
17	Jeong et al. [60]	VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models.
18	Jiang et al. [67]	VideoBooth: Diffusion-based Video Generation with Image Prompts.
19	Ju et al. [84]	MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions.
20	Kou et al. [88]	Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment.
21	Li et al. [76]	PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation
22	Li et al. [58]	Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization.
23	Liao et al. [93]	Evaluation of Text-to-Video Generation Models: A Dynamics Perspective.
24	Lin et al. [39]	Open-Sora Plan: Open-Source Large Video Generation Model.
25	Liu et al. [41]	VideoDPO: Omni-Preference Alignment for Video Diffusion Generation.
26	Liu et al. [75]	Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model.
27	Lv et al. [48]	GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning.
28	Ma et al. [63]	Follow Your Pose: Pose-Guided Text-to-Video Generation Using Pose-Free Videos.
29	Menapace et al. [53]	Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis.
30	Miao et al. [98]	T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models.
31	Mohamed and Lucke-Wold [29]	Text-to-Video Generative Artificial Intelligence: Sora in Neurosurgery.
32	Nan et al. [85]	OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-Video Generation.
33	Qin et al. [36]	xGen-VideoSyn-1: High-Fidelity Text-to-Video Synthesis with Compressed Representations.
34	Qing et al. [46]	Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation.
35	Qu et al. [90]	Exploring AIGC Video Quality: A Focus on Visual Harmony Video-Text Consistency and Domain Distribution Gap.
36	Rawte et al. [95]	ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Models.
37	Ren et al. [61]	Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models.
38	Sharan et al. [101]	Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification.
39	Si et al. [74]	FreeU: Free Lunch in Diffusion U-Net.
40	Tan et al. [50]	Mimir: Improving Video Diffusion Models for Precise Text Understanding.
41	Tian et al. [35]	VideoTetris: Towards Compositional Text-to-Video Generation.
42	Wang et al. [43]	A Recipe for Scaling up Text-to-Video Generation with Text-free Videos.
43	Wang et al. [54]	LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models.
44	Wang and Yang [83]	VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models.
45	Sun et al. [94]	T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation.
46	Wang et al. [87]	LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation.
47	Wang et al. [91]	T2VBench: Benchmarking Temporal Dynamics for Text-to-Video Generation
48	Wang et al. [44]	WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation.
49	Wei et al. [69]	DreamVideo: Composing Your Dream Videos with Customized Subject and Motion.
50	Weng et al. [31]	ART-V: Auto-Regressive Text-to-Video Generation with Diffusion Models.
51	Wu et al. [86]	Towards A Better Metric for Text-to-Video Generation.
52	Wu et al. [65]	DragAnything: Motion Control for Anything Using Entity Representation.
53	Xie et al. [32]	Progressive Autoregressive Video Diffusion Models.
54	Xing et al. [66]	DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors.
55	Yang et al. [40]	CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer.
56	Yang et al. [78]	IPO: Iterative Preference Optimization for Text-to-Video Generation.
57	Yin et al. [30]	From Slow Bidirectional to Fast Autoregressive Video Diffusion Models.
58	Yin et al. [42]	Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model.
59	Yuan et al. [92]	ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation.
60	Yuan et al. [45]	MagicTime: Time-lapse Video Generation Models as Metamoraphic Simulators.
61	Yuan et al. [77]	Inflation With Diffusion: Efficient Temporal Adaptation for Text-to-Video Super-Resolution.
62	Yuan et al. [70]	Identity-Preserving Text-to-Video Generation by Frequency Decomposition.
63	Zhang et al. [71]	CAMEL: CAusal Motion Enhancement Tailored for Lifting Text-driven Video Editing.
64	Zhang et al. [72]	Style-A-Video: Agile Diffusion for Arbitrary Text-Based Video Style Transfer.
65	Zhang et al. [57]	Tora: Trajectory-oriented Diffusion Transformer for Video Generation.
66	Zhang et al. [55]	Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation.
67	Zhao et al. [59]	MotionDirector: Motion Customization of Text-to-Video Diffusion Models.
68	Zhou et al. [49]	HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models.
69	Zhu et al. [81]	Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation.

and are included in the bibliometric analysis and literature review. Figure 2 presents the corresponding PRISMA flow diagram.

3.4. Data Extraction

Data extraction from the included studies was conducted by two reviewers. The reviewers worked sequentially: the first reviewer extracted study characteristics and evaluation details using a predefined data extraction template, and the second reviewer independently verified the extracted information for accuracy and completeness. Any discrepancies were resolved through discussion until consensus was reached. No automation tools, machine learning systems, or AI-assisted screening software were used during the identification, screening, or data extraction stages; all processes were conducted manually.

4. Descriptive Statistics of Selected Literature

To deepen our understanding of the evolving landscape of T2V research, we categorized and analyzed the 69 selected publications along several dimensions. These statistics provide insight into key trends, common practices, and gaps in the field. It is important to note that the bibliometric analysis presented here is not intended to be a statistically exhaustive or longitudinal survey of the entire T2V literature. Instead, consistent with the deliberate focus of this review on works published from 2024 onward, the analysis provides a descriptive snapshot of emerging research directions during a period of rapid methodological consolidation and transition.

Regarding publication types (Figure 3), conference papers dominate (72%), with preprints/tech reports at 21%, while peer-reviewed journal articles present only 7%. Figure 4 categorizes papers by research theme. More than three-quarters (77%) of the corpus focuses on video synthesis directly. In contrast, only 18% focus on objective assessment and benchmarks, and a mere 5% on video understanding.

Figure 5 shows that an encouraging 91% of the surveyed papers publicly release their code, reflecting the open research culture that has similarly accelerated progress in T2V generation models. However, this openness contrasts sharply with the limited accessibility of large-scale video datasets. The remaining 9% are typically corporate tech reports with either (i) proprietary data pipelines or (ii) safety concerns around video content. Figure 6 summarizes the training strategies adopted in the surveyed models. Almost two-thirds of recent T2V work (68%) relies on fine-tuning existing T2V models. Meanwhile, 21% of works still pursue training from scratch, especially when introducing novel T2V architectures. Specifically, 11% adopt training-free strategies, indicating a growing interest in lightweight, low-compute alternatives appropriate for rapid experimentation and deployment.

In terms of architectural design (Figure 7), we observe the dominance of diffusion models, often enhanced by transformer backbones. This reflects the field’s convergence on scalable, generative architectures capable of producing high-fidelity, temporally consistent outputs. Other architectures include GAN-based models and hybrid transformer–CNN models, yet these are becoming less common.

Finally, Figure 8 shows the distribution of dataset types that have been used in the selected literature. More than half of the studies (54%) leveraged large-scale, web-scraped video–text corpora such as WebVid-10M or Panda-70M. Domain-specific collections (15%) and benchmark/evaluation suites (14%) together account for under one-third, while synthetic/LLM-generated video–text pairs appear in 10%. High-quality, fully curated datasets occupy only 7%.

5. Literature Review

As outlined in Section 1, the scope of T2V research covers three primary themes: video synthesis, video understanding, and objective assessment and benchmarking. This section surveys and categorizes key contributions from the literature, structured around the following research questions:

RQ1: What are the key technological advances that have driven progress in the aforementioned video research fields?

RQ2: What are the current challenges and best practices in evaluating T2V models, and how do benchmark datasets support the development of robust T2V generators?

5.1. Video Synthesis

Driven by recent advances in transformer architectures [24,25] and diffusion models [26,27], visual content generation has shown remarkable capabilities when conditioned on text prompts. These developments have sparked widespread interest and a surge of new research efforts.

A notable milestone is the introduction of Sora [1], OpenAI’s DiT-based video generation model, which demonstrated striking results and marked a leap forward, ushering in a new era of text-driven video synthesis. A key innovation in Sora is its integration of a descriptive captioning model with GPT-based prompt expansion (adapted from DALL-E 3 [28]). This “recaptioning” strategy enriches training inputs with detailed textual annotations, improving alignment between user prompts and generated videos. Similar to other diffusion models, generation in Sora starts with a noisy grid of latent patches, which are iteratively denoised. This process is guided by the transformer model and conditioned on user-provided text to generate coherent and text-aligned videos.

Building on the promise of Sora, Mohamed and Lucke-Wold [29] investigated its applicability within the medical domain, particularly in neurosurgery. Their study emphasizes the promise of such models for medical education, surgical simulation, and patient communication by rapidly generating tailored procedural or anatomical videos from natural language prompts. However, they also identify several key challenges hindering clinical adoption, including physically implausible or anatomically inaccurate motion, generation of non-existent or irrelevant entities, unnatural object morphing, abrupt transitions, and flawed modeling of physical interactions.

Alongside Sora, other innovations have emerged, covering primarily four subcategories; namely, automatic T2V generation, controllable T2V generation, video style transfer and editing, and video quality and inference enhancement (see Figure 9).

5.1.1. Automatic T2V Generation

In this category, related works can be further grouped into four sub-categories: long-form video generation, user preference and human alignment, physics-aware and reasoning-grounded generation, and hybrid and factorized generation.

(a)

Long-form video generation

Generating long videos poses unique challenges that go beyond temporal extension. As video duration increases, the generation task shifts from synthesizing isolated visual events toward modeling coherent narrative structure. Unlike short clips that typically depict a single event, long-form videos require event sequencing, continuity of entities, and coherence across narrative segments. Consequently, many recent approaches to long-form T2V generation incorporate narrative primitives, such as temporal ordering, scripting, and event-level decomposition, even when not explicitly framed as story generation.

In this context, while many existing diffusion models rely on bidirectional attention mechanisms that require access to the entire sequence—including future frames—for generating each frame, Yin et al. [30] introduced CausVid, a video diffusion framework that re-engineers diffusion transformers into autoregressive transformers with causal attention. This enables frame-by-frame generation based only on preceding frames, enabling interactive and progressive synthesis. To accelerate inference, CausVid extends Distribution Matching Distillation (DMD) to video, reducing sampling steps from around 50 to just 4 via a teacher–student framework, where the original bidirectional model supervises the autoregressive student. Training is stabilized using an initialization scheme based on the teacher’s continuous ODE trajectories, while an asymmetric distillation strategy mitigates error accumulation during generation. Additionally, key-value caching reuses attention states across frames, reducing redundant computation and significantly boosting speed. Despite being trained primarily on short clips, CausVid reports improved temporal consistency and real-time inference speed relative to its bidirectional teacher model, making it suitable for low-latency applications.

Building on the autoregressive paradigm, Weng et al. [31] introduced ART-V, an auto-regressive (AR) T2V framework built on pretrained text-to-image diffusion models, extended for long video synthesis at resolutions up to 768 × 768. ART-V generates each frame sequentially, each conditioned on one or more previous frames, using a lightweight T2I-Adapter to incorporate temporal context. In order to mitigate appearance drifting—a common issue in AR video generation where content gradually deviates—ART-V employs a masked diffusion mechanism that learns to copy stable regions from reference frames and selectively generates new content where changes are needed, minimizing error accumulation. An anchored conditioning strategy further enhances global consistency in scene layout and appearance throughout the video by conditioning on a fixed anchor frame (user-defined or generated) alongside previous frames. To simulate real-world inference conditions and boost robustness, ART-V applies noise augmentation to reference frames during training. The framework supports diverse conditioning modes—text prompts, static images, or hybrids—enabling applications such as long-form video synthesis from sequential prompts and animation of still images with textual descriptions.

While CausVid and ART-V focus on causal attention and appearance stabilization, other works further refine autoregressive diffusion by controlling how uncertainty propagates over time. Along this line, Xie et al. [32] proposed Progressive Autoregressive Video Diffusion Models (PA-VDM) that is able to generate long videos—up to 60 s (1440 frames, at 24 FPS) at resolutions ranging from $176\times 320$ to $240\times 424$. PA-VDM introduces progressive noise scheduling, assigning noise levels per frame rather than uniformly across the entire video. Frames nearer to the current generation step are assigned lower noise (indicating higher certainty), while more distant frames receive higher noise, enabling smooth information propagation and temporal continuity. To further enhance coherence, the model adopts autoregressive frame generation with overlapping attention, progressively denoising small frame intervals instead of the entire sequence at once. This overlapping structure allows stronger correlations between adjacent frames and mitigates artifacts often seen in previous autoregressive approaches. Notably, PA-VDM is model-agnostic and can be integrated into existing architectures like DiT or U-Net. During training, pretrained diffusion models are fine-tuned with the new noise schedule, while inference proceeds through progressive denoising to generate temporally coherent videos.

Together, these autoregressive diffusion approaches address the quadratic memory and latency bottlenecks of bidirectional attention by restricting temporal context. From a narrative perspective, these autoregressive approaches can be interpreted as event-sequencing mechanisms. By conditioning generation on prior frames or segments, they enforce a temporal ordering of events and causal dependency between visual states. However, because narrative structure emerges indirectly from architectural constraints rather than event representations, cumulative error and semantic drift remain challenges in very long sequences.

To overcome these limitations while retaining efficiency, a second line of work shifts from frame-level autoregression to chunk-based generation. In this setting, Henschel et al. [33] proposed StreamingT2V that generates videos in chunks of 8–16 frames in the latent space of a pretrained Vector Quantized Variational Autoencoder (VQ-VAE) variant, specifically a VQ-GAN [34], which compresses video frames into latent codes for efficient spatiotemporal diffusion. Training leverages short text–video datasets and pretrained short-video generators for initialization. Temporal coherence is maintained through a Conditional Attention Module (CAM) that maintains short-term consistency by conditioning each chunk on its preceding one, and an Appearance Preservation Module (APM) that retains high-level scene attributes from the initial chunk to prevent semantic drift over time. A randomized blending strategy enables seamless video extension, allowing StreamingT2V to generate videos ranging from 80 to over 1200 frames (2+ min) at $720\times 720$ resolution.

Focusing on compositionality and multi-object dynamics in long videos, Tian et al. [35] proposed VideoTetris, which models complex spatiotemporal semantics by tracking and manipulating multiple objects over time using attention-based control. By explicitly modeling object interactions and configurations, VideoTetris moves closer to event-centric narrative modeling, where scenes evolve through interactions among entities rather than independent frame synthesis.

While chunk-based and compositional strategies alleviate computational constraints and improve controllability, they often rely on auxiliary modules and preprocessing pipelines whose complexity can limit generalization. Moreover, maintaining coherence across narrative segments remains challenging when global context evolves significantly.

In parallel to these architectural strategies, another stream of work improves long-form generation efficiency through latent compression and adaptive inference. For instance, Qin et al. [36] proposed xGen-VideoSyn-1, a hybrid model for long-form video generation (over 14 s at 720p resolution) that combines latent diffusion architecture with a Video VAE (VidVAE) for spatial–temporal compression. The VidVAE extends the traditional VAE to 3D, enabling compression of video data both spatially and temporally, which significantly reduces the length of visual tokens and computational demands for generating long-sequence videos. Token segmentation divides long sequences into overlapping segments that are independently processed and merged to preserve temporal consistency. Training is conducted on 13 million curated video–text pairs, collected through an automated pipeline.

Complementarily, Dalal et al. [37] introduced Test-Time Training (TTT) layers to dynamically extend the temporal capacity of T2V models. TTT layers replace static hidden states with learnable modules that are fine-tuned during inference, which improves temporal consistency without retraining on long videos. The model is initialized from a pretrained CogVideo-X model, originally limited to 3-second clips, and trained on a curated 7-hour Tom and Jerry dataset. By integrating TTT layers and restricting self-attention to 3-second segments, the proposed model generates coherent one-minute videos (~63 s) at 16 FPS and around 256 × 256 resolution. Training proceeds in progressive stages, extending temporal context from 3 to 9, 18, 30, and finally 63 s.

In contrast to autoregressive and chunk-based designs, several recent models emphasize full spatiotemporal attention to maximize coherence and visual fidelity. For instance, Bao et al. [38] proposed Vidu, which combines a video autoencoder with a U-shaped Vision Transformer to capture long-range dependencies across 3D spatiotemporal patches and to generate videos up to 1080p and 16 s in a single pass. Similarly, Lin et al. [39] introduced Open-Sora Plan, an open framework designed for long-form, Sora-like video synthesis of up to 16 s at 720p resolution. It integrates a Wavelet-Flow Variational Autoencoder (VAE) for video compression, a 3D full attention transformer with Skiparse Attention, and a conditional control module. Open-Sora is trained via a progressive multi-stage strategy: it starts with image-pretrained weights and sparse attention to initialize spatiotemporal modeling, followed by joint training on both images and raw video at varying resolutions and durations, and finally fine-tuning on a cleaned Panda-70M subset. Finally, Yang et al. [40] proposed CogVideoX, which employs a 3D VAE, an Expert Transformer, and progressive multi-resolution training to generate high-resolution videos.

In summary, long-form T2V methods trade off among (i) temporal context modeling (autoregressive vs. chunked vs. full attention), (ii) computational and memory efficiency (causal attention, latent compression, distillation), and (iii) explicitness of narrative representations (sequencing vs. object-centric or event-centric models).

Table A2. Summary of T2V models mentioned in the long-form video generation category.

Reference	Model Architecture	Methods	Training Strategy	Training Dataset	Evaluation Results	Baselines	Project Code
Yin et al. [30]	Transformer-based diffusion model	Bidirectional-to-autoregressive transformer adaptation + DMD + Asymmetric distillation + KV caching	Teacher–student distillation with ODE-based initialization; training on short clips	UCF-101 (~13,000 clips)	Inference speed: ~9.4 FPS (single GPU) (SOTA); Authors also report their VBench-Long total score 84.27	CogVideoX-5B; OpenSORA; StreamingT2V; Pyramid Flow	https://github.com/tianweiy/CausVid
Weng et al. [31]	Pretrained image diffusion model (Stable Diffusion 2.1) + T2I-Adapter	Autoregressive generation + Masked diffusion to reduce drifting	Noise augmentation + Anchored conditioning on initial frame	WebVid-10M	UCF-101: FVD 567.20 (text-only), 315.69 (GT-image-conditioned); IS 26.89/50.34. MSR-VTT: FVD 291.08	VideoFusion; MagicVideo; LVDM; CogVideo	https://github.com/WarranWeng/ART.V
Xie et al. [32]	Transformer-based latent diffusion model	Progressive noise scheduling + Autoregressive denoising with overlapping attention	Progressive autoregressive training on latent frames	Large-scale filtered dataset (~1M videos, unnamed)	FVD 358 on long-video benchmarks (SOTA)	StreamingSVD; SVD-XT; FIFO-Diffusion	https://github.com/desaixie/pa_vdm
Henschel et al. [33]	Autoregressive diffusion with CAM + APM + enhancer	Chunk-based autoregressive diffusion + Randomized blending	Pretrained initialization + Streaming refinement	Large-scale public video collection (720 × 720, unnamed)	MAWE 52.3 (SOTA), SCuts 0.04 (∼SOTA), CLIP Score 31.73 (SOTA)	I2VGen-XL; SVD; DynamiCrafterXL; OpenSoraPlan v1.2; SEINE; SparseControl; OpenSora v1.2; FreeNoise	https://github.com/Picsart-AI-Research/StreamingT2V
Tian et al. [35]	Spatio-temporal compositional diffusion + ControlNet-based AR diffusion	Compositional region-based video generation + Consistency regularization	Auto-regressive training on enhanced video–text data	Filtered Panda-70M	-Short prompts: VBLIP-VQA 0.556, VUnidet 0.235, CLIP-SIM 0.931 -Long prompts: VBLIP-VQA 0.484, VUnidet 0.214, CLIP-SIM 0.952 (∼SOTA)	Diffusion models without compositional control; FreeNoise; StreamingT2V	https://github.com/YangLing0818/VideoTetris
Qin et al. [36]	VidVAE + DiT with spatial and temporal attention	Latent diffusion with video compression + Divide-and-merge strategy	Progressive multi-resolution training (240p, 480p, 720p)	13M+ curated video–text pairs (unnamed)	VBench: Consistency 0.714, Temporal 0.947, Aesthetic 0.655, Spatial 0.523	ImageVAE; OpenSoraPlan; ModelScope	Not released
Dalal et al. [37]	Pretrained Diffusion Transformer (e.g., CogVideo-X) + TTT layers	Test-Time Training layers with segmented inference	Progressive fine-tuning from 3 s to 63 s sequences	~7 h Tom and Jerry cartoons + storyboards	Human evaluation: +34 Elo improvement over non-TTT baselines	CogVideo-X without TTT; fixed-context diffusion models	https://github.com/test-time-training/ttt-video-dit
Bao et al. [38]	Diffusion model + U-ViT backbone + Video autoencoder	Transformer-based denoising of 3D patches + Re-captioning	Multi-length video training with auto-captioning	Large-scale text–video datasets (unnamed)	Evaluation primarily qualitative	Not reported	https://www.shengshu.com/en
Lin et al. [39]	STDiT + PixArt-$\alpha$ pretrained T2I backbone + spatial VAE	Wavelet-Flow VAE + Joint image–video training + Skiparse attention	Large-scale video pretraining + image-to-video finetuning	~70M curated video dataset (256 × 256)	PSNR 32.32, LPIPS 0.051, rFVD 186 (SOTA)	CV-VAE; OD-VAE; CogVideoX-2B/5B; Mochi-1	https://github.com/PKU-YuanGroup/Open-Sora-Plan
Yang et al. [40]	Diffusion Transformer with 3D full attention + 3D VAE + Expert Transformer	Expert-conditioned latent diffusion + LLaMA2 captioning	Progressive long-sequence training + separate VAE training	~35M video–text pairs + 2B images (LAION-5B, COYO)	CogVideoX-5B total score 2.74 vs. Kling 2.17	T2V-Turbo; AnimateDiff; VideoCrafter-2.0; OpenSora V1.2; Show-1; Gen-2; Pika; LaVie-2; Kling	https://github.com/THUDM/CogVideo

All project links were accessed on 16 February 2026. Metrics. FVD (Fréchet Video Distance) measures distributional similarity between generated and real videos. IS (Inception Score) evaluates visual quality and diversity. CLIP-SIM denotes CLIP-based text–video similarity scores. VBench reports normalized scores across multiple dimensions such as temporal consistency, spatial coherence, and aesthetics. MAWE and SCuts are motion-aware and temporal cut metrics used in streaming and long-video evaluation. VBLIP-VQA and VUnidet assess video-language understanding and object consistency. PSNR and LPIPS measure reconstruction fidelity and perceptual similarity for video autoencoders. rFVD is a reconstruction-based variant of FVD. Elo scores derive from pairwise human preference studies. All metrics are reported as described by the original authors and are not directly comparable across different evaluation settings.

provides an overview of the studies mentioned above, including their model architectures, methodological contributions, training strategies, evaluation results (with SOTA indicating state-of-the-art performance among the baselines), and reference baselines.

(b)

User preference and human alignment

Addressing the critical challenge of aligning T2V models with diverse human preferences, several works have focused on integrating user feedback and preference signals into video generation. For instance, Liu et al. [41] proposed VideoDPO, a pioneering pipeline that adapts Direct Preference Optimization (DPO)—originally developed for language and image generation—to video diffusion models. Pretrained T2V systems often struggle to balance visual quality and semantic fidelity across varied user expectations. VideoDPO addresses this limitation through preference-driven fine-tuning tailored for video generation. At the core of VideoDPO is OmniScore, a unified metric that jointly captures visual fidelity and semantic alignment by combining intra-frame quality (clarity, aesthetics), inter-frame coherence (temporal consistency, motion stability), and text–video relevance using vision-language models. Training data are generated by sampling multiple video outputs per prompt from a base diffusion model, automatically ranking them via OmniScore, and forming preference pairs weighted by score differences. The resulting re-weighted DPO loss fine-tunes the backbone model, improving alignment with user preferences across both visual and semantic dimensions. While effective, this approach depends on the quality and bias of the underlying scoring model, which may influence the learned preference distribution.

Building on preference-aligned optimization at a larger scale, Yin et al. [42] introduced Step-Video-T2V, a 30-billion-parameter T2V foundation model capable of generating videos up to 204 frames long. It employs a Deep Compression Video Variational Autoencoder (Video-VAE), achieving $16\times 16$ spatial and $8\times$ temporal compression without sacrificing visual fidelity. The encoder utilizes causal Res3D blocks and a convolution-attention MidBlock, while the decoder symmetrically reconstructs frames from the compressed latent representation. A dual-path structure with causal 3D convolutions preserves high-frequency details while effectively compressing structural information. Generation is guided by a 48-layer DiT with full 3D attention—48 heads per layer—trained using Flow Matching loss to improve denoising stability and spatiotemporal consistency. Bilingual generation in English and Chinese is enabled by dual pretrained multilingual text encoders conditioning the diffusion process. Finally, human-aligned fine-tuning via Video-based DPO enhances realism and temporal coherence by reducing artifacts based on preference signals. For evaluation, the authors propose Step-Video-T2V-Eval, a benchmark of 128 video prompts across 11 categories, enabling comparison against state-of-the-art T2V systems.

In parallel to preference-based fine-tuning, Wang et al. [43] proposed TF-T2V, which approaches alignment from a different perspective by decoupling text understanding from motion modeling. To overcome the scarcity and cost of labeled video–text datasets, TF-T2V leverages large-scale unlabeled videos (e.g., from YouTube) to learn motion dynamics, while maintaining semantic alignment using limited labeled data. The framework adopts a dual-branch architecture based on a 3D-UNet diffusion model, including a text-conditioned content branch for spatial generation, and a motion branch trained on unlabeled videos. Joint training with shared weights ensures motion coherence, while separating text decoding from motion modeling enables effective exploitation of text-free video data. At the same time, incorporating high-quality image-text datasets such as LAION-5B strengthens text-conditioned spatial generation. This design illustrates an alternative alignment strategy that prioritizes data efficiency and modularity, though it relies on the assumption that motion learned from unlabeled videos generalizes effectively across diverse semantic contexts.

(c)

Physics-aware and reasoning-grounded generation

Traditional T2V models often struggle to simulate abstract physical laws, leading to implausible or inconsistent dynamics. To address this limitation, recent work has focused on embedding explicit physical knowledge or structured reasoning into the generation process, either through physics-aware modeling or higher-level semantic planning.

Within this direction, Wang et al. [44] proposed WISA, a T2V generation framework that involves interpretable physical knowledge to improve realism and consistency. WISA decomposes physical understanding into three components: textual descriptions of expected physical behavior; qualitative categories covering 17 physical phenomena across dynamics, thermodynamics, and optics; and quantitative physical properties such as density or temperature. Its key innovation is the Mixture-of-Physical-Experts Attention (MoPA), where individual attention heads specialize in distinct physical categories and selectively activated based on a learned physical classifier. Adaptive Layer Normalization (AdaLN) is further used to inject continuous physical values into the generation process, enabling fine-grained control based on real-world measurements. To support training and evaluation, the authors released WISA-32K, a curated dataset of 32,000 videos labeled with physical categories and properties to support physics-aware generation. While effective within its defined scope, this approach relies on predefined physical taxonomies and annotations, which may limit scalability to unconstrained or open-domain scenarios.

Building on physics-grounded synthesis for natural phenomena, Yuan et al. [45] proposed MagicTime, which targets metamorphic time-lapse video generation by learning real-world physical transformations from time-lapse data. MagicTime introduces the MagicAdapter module to decouple spatial and temporal learning, allowing pretrained T2V models to better capture long-term physical variability of metamorphic phenomena, while retaining general video synthesis capabilities. Temporal fidelity is further improved through Dynamic Frames Extraction, which prioritizes frames with significant transitions, and a specialized text encoder designed to distinguish metamorphic-specific language. Training is conducted on the ChronoMagic, a curated dataset of 2265 time-lapse videos with auto-generated captions. Although effective for modeling gradual physical processes, this approach is inherently specialized, and its reliance on domain-specific data limits direct applicability to more general video generation tasks.

Shifting from domain-specific physics to general realism, Qing et al. [46] proposed HiGen, a diffusion-based framework that hierarchically decouples spatial and temporal components of video synthesis to improve realism and motion stability. HiGen separates structure and content modeling through a two-stage training process: spatial reasoning first produces coherent static priors conditioned on text, which subsequently guide temporal reasoning to generate smooth motion. At the content level, the model extracts motion-related and appearance-related cues from training data to better handle spatial and temporal variations independently. This hierarchical design reduces interference between appearance and motion modeling; however, it does not explicitly encode physical laws and instead relies on learned statistical regularities to improve realism, while the enforced separation may limit the model’s ability to capture tightly coupled spatiotemporal interactions, such as object collisions or rapid scene changes.

While physics-based models focus on capturing natural phenomena through learned physical transformations and domain-specific dynamics, some works integrate LLMs to provide high-level reasoning, planning, and semantic structure. For instance, Fei et al. [47] introduced Dysen-VDM, a diffusion model that leverages LLMs for enhanced scene understanding and action planning. The framework constructs dynamic scene graphs by extracting and ordering actions from text, enriching them using in-context LLM reasoning, and encoding them with a recurrent graph Transformer. This representation is injected into a latent video diffusion model, enabling more temporally coherent generation. Training proceeds in multiple stages, combining large-scale pretraining with reinforcement learning to refine scene graph coherence. While effective at imposing high-level structure, this approach introduces additional architectural complexity and depends on the reliability of LLM-generated scene representations.

A related but distinct strategy is explored in GPT4Motion by Lv et al. [48], which combines LLM-based planning, physics simulation, and diffusion-based rendering in a training-free pipeline. Here, an LLM translates text prompt into an executable Blender scripts that define scenes and simulate physics-driven dynamics such as object collisions and fluid motion. The resulting outputs are then used to condition a text-to-image diffusion model to synthesize video frames. This modular design improves physical consistency by offloading dynamics to a physics engine, but it also introduces external dependencies and limits end-to-end optimization.

Beyond guiding physical realism, LLMs have also been employed as semantic planners to improve text–video alignment. For instance, Zhou et al. [49] proposed HiTVideo, which introduces a hierarchical video tokenizer built on a 3D causal VAE. By separating high-level semantic tokens from low-level spatiotemporal detail tokens, the model enables coarse-to-fine video generation using an autoregressive LLM conditioned on a frozen text encoder. This hierarchical representation reduces token length and improves semantic alignment, though it increases system complexity and requires careful coordination between tokenization and generation stages.

Finally, Tan et al. [50] proposed Mimir, an end-to-end training framework that combines conventional text encoders with LLMs for improved semantic reasoning. Mimir uses a dual-branch architecture in which a standard text encoder captures local syntax, while a decoder-only LLM models global semantics. A token fuser module integrates these representations to condition a latent diffusion model. Joint training across all components enables semantically coherent video generation within a compact latent space. However, the approach depends on large-scale curated datasets and extensive training resources, which may hinder its applicability in low-resource settings.

An overview of the papers discussed in categories (b) and (c) is presented in

Table A3. Summary of T2V models mentioned in categories b) and c).

Reference	Model Architecture	Methods	Training Strategy	Training Dataset	Evaluation Results	Baselines	Project Code
Liu et al. [41]	Post-training adaptation of existing diffusion models; no architecture change	Direct Preference Optimization (DPO) + Omni-preference alignment + VideoDPO Loss	Multi-stage: T2V pretraining → DPO fine-tuning with re-weighted preference pairs	VidProM dataset with 10,000 human prompts + millions of videos for preference alignment	OmniScore > 78% (SOTA)	VC2; SFT; VADER	https://github.com/CIntellifusion/VideoDPO
Yin et al. [42]	Video-VAE with causal Res3D blocks + DiT with 3D attention + bilingual text encoders; + Video-DPO module	Latent video diffusion, Flow Matching training, multilingual text encoding, preference-based Direct Preference Optimization (Video-DPO)	Multi-stage training: Video-VAE pretraining → DiT diffusion model training → DPO finetuning	Large-scale, filtered video corpus (unnamed)	VBench (10/30 s): Temporal 94.7/94.9, Frame 64.4/63.4; 9.4 FPS	CogVideoX-5B; OpenSORA; Pyramid Flow; MovieGen; Gen-L-Video; FreeNoise; StreamingT2V; FIFO-Diffusion; Bidirectional Teacher	https://github.com/stepfun-ai/Step-Video-T2V
Wang et al. [43]	Two-branch diffusion model: content branch (text-conditioned), motion branch (image-conditioned) with shared weights	Disentangled content-motion learning, temporal coherence loss	Joint optimization on video–text and text-free video data; semi-supervised approach	WebVid10M video–text pairs + large-scale unlabeled videos from YouTube	Qualitative comparison	Gen-2; Pika	https://github.com/ali-vilab/Vgen
Wang et al. [44]	Enhanced T2V model with physics-aware modules	Decomposition of physical principles into textual, qualitative, and quantitative components; MoPA attention; Physical classifier	Joint training with generative loss and physical classifier supervision	WISA-32K: 32,000 videos on 17 physical laws (dynamics, thermodynamics, optics)	VideoPhy (SA/PC): 0.67/0.38 (SOTA) PhyGenBench (SA/PC): 0.40/0.43 (∼SOTA); Inference ∼220 s (∼SOTA)	VideoCrafter2; HunyuanVideo; CogvideoX-5B; Cosmos; PhyT2V	https://github.com/360CVGroup/WISA
Yuan et al. [45]	DiT backbone with MagicAdapter modules in temporal layers, enhanced text encoder	MagicAdapter for physics encoding; Dynamic Frame Extraction for adaptive sampling + Magic Text-Encoder for prompt understanding + auto-captioning for annotation	Fine-tuning pretrained T2V models on metamorphic time-lapse videos	Curated ChronoMagic dataset (~2265 annotated metamorphic time-lapse videos)	FID 58.12 (∼SOTA), FVD 441.17 (SOTA), CLIP-SIM~0.33 (SOTA)	MakeLongVideo; ModelScopeT2V; VideoCrafter2; Zeroscope; LaVie; T2V-zero; Latte; Animatediff	https://github.com/PKU-YuanGroup/MagicTime
Qing et al. [46]	Diffusion model with unified denoiser for spatial-temporal reasoning + motion and appearance cue extraction modules	Decoupled spatial-temporal diffusion + hierarchical conditioning on motion and appearance variations	Two-step training: spatial prior from text → temporal motion from spatial priors	MSR-VTT (10,000 video–text pairs)	SSIM ~0.97 (SOTA), PSNR 39.37 (∼SOTA), rFVD 3.61 (SOTA)	CogvideoX-1.5; OpenSora-1.2; Cosmos-VAE; HunyuanVideo	https://github.com/ali-vilab/Vgen
Fei et al. [47]	Latent video diffusion model + Dysen module (action extractor, DSG constructor, LLM-based enrichment, graph Transformer encoder)	Dynamic Scene Graphs (DSGs) + LLMs (ChatGPT) + Recurrent Graph Transformer (RGT)	Pre-training on large-scale video–text datasets → Fine-tuning Dysen module and RGT	3M WebVid (pre-training) + UCF-101, MSR-VTT, ActivityNet (fine-tuning)	UCF-101: IS 95.23, FVD 255.42 (SOTA); Zero-shot: IS 35.57, FVD 325.42 (SOTA); Human eval (ActivityNet): 86.6/92.4/87.3 (SOTA)	CogVideo; MagicVideo; MakeVideo; AlignLatent; Latent-VDM; Latent-Shift; VideoFactory; InternVid	https://github.com/scofield7419/Dysen
Lv et al. [48]	Pipeline of GPT-4 (planner) + Blender physics engine (simulator) + ControlNet-augmented Stable Diffusion	GPT-4 generates Blender scripts for simulation + Blender simulates motion + Stable Diffusion generates frames	Training-free framework; uses pretrained GPT-4 and Stable Diffusion; no fine-tuning	None	Motion 0.993, CLIP-SIM 0.260, Flickering 0.990 (SOTA)	AnimateDiff; ModelScope; Text2Video-Zero; DirecT2V	https://github.com/jiaxilv/GPT4Motion
Zhou et al. [49]	3D causal VAE with hierarchical discrete token layers + LLaMA 3B for token generation conditioned on frozen Flan-T5-XL embeddings	Hierarchical autoregressive token generation from coarse semantic to fine visual detail	Two-stage training: Training of the 3D causal VAE → Training of LLM autoregressively with text conditioning	Pexels Videos dataset	Reduced bits-per-pixel (bpp) by ≈70% (SOTA)	MAGVIT-v2; CogVideoX; EMU3	Not released
Tan et al. [50]	Latent diffusion backbone + Dual-text encoding (ViT-style encoder + decoder-only LLM) + Token Fuser	Token fusion to harmonize encoder and LLM embeddings + Semantic stabilization	Joint training on large video–text datasets with supervised latent denoising conditioned on fused semantic embeddings	Large-scale video–text corpora (unnamed)	VBench (8 dims): 97.68–26.22 (SOTA); User study (Instruction/Physics/Visual %): 82.00/83.65/89.65 (SOTA)	ModelscopeT2V; OpenSora; OpenSoraPlan; CogVideoX-2B/5B	https://lucaria-academy.github.io/Mimir

All project links were accessed on 16 February 2026. Metrics. OmniScore is a composite preference-based metric combining visual fidelity, temporal coherence, and text–video alignment. SA and PC denote Success Accuracy and Physical Consistency scores used in physics-aware video benchmarks. VideoPhy and PhyGenBench are physics-grounded evaluation suites measuring compliance with physical laws in generated videos. Motion and Flickering scores quantify motion intensity and temporal stability, respectively (higher is better). Structural Similarity Index Measure (SSIM) evaluates perceptual similarity between generated and reference videos by comparing luminance, contrast, and structural information. Bits-per-pixel (bpp) measures video compression efficiency, with lower values indicating more compact representations. Reported human evaluation scores correspond to task-specific criteria defined by the original authors (e.g., instruction following, physical plausibility, and visual quality). All metrics are reported as described in the original papers and are not directly comparable across different evaluation settings.

.

(d)

Hybrid and factorized generation

Unlike end-to-end generative models, hybrid and factorized generation approaches decompose the video synthesis process into modular stages or separate spatial and temporal modeling, thereby improving synthesis quality and efficiency while reducing computational overhead.

A representative example of this paradigm is Emu Video [51], which factorizes T2V generation into two sequential stages. First, an image is generated from a text prompt using a frozen text-to-image latent diffusion model. This image then serves as a conditioning signal for a video diffusion transformer that animates the scene over time. By decoupling static scene composition from temporal animation, Emu Video simplifies the generation process compared to fully end-to-end or cascaded pipelines. The model employs tailored noise schedules and a multi-phase training strategy to achieve stable, high-resolution ($512\times 512$) outputs while avoiding the complexity of multi-stage diffusion pipelines. However, this factorization assumes that a single static image sufficiently captures global scene structure, which may constrain flexibility in scenarios involving significant scene evolution.

Focusing on disentangled spatial–temporal modeling, Chen et al. [52] presented VideoCrafter2, a method designed to improve the visual fidelity when training on low-quality, large-scale datasets such as WebVid-10M. Its architecture separates spatial and temporal learning by employing factorized 3D U-Net temporal modules alongside pretrained spatial modules responsible for appearance and semantic grounding. Training proceeds in two stages: first, joint training on large video datasets captures motion dynamics and coarse semantics; subsequently, only the spatial modules are fine-tuned on high-quality synthetic images to enhance visual fidelity while preserving learned motion. Although this design improves robustness to low-quality video data, the dependence on staged training and dataset-specific tuning highlights a trade-off between visual refinement and pipeline complexity.

In parallel, Menapace et al. [53] introduced Snap Video, which prioritizes speed through a video-first transformer architecture. Rather than extending 2D image-based U-Nets with temporal layers, Snap Video models spatial and temporal features jointly within a compressed latent space using a scalable transformer. The approach adapts the Efficient Diffusion Model framework to handle the spatial and temporal redundancies in video, achieving a $3.31\times$ speedup in training and $4.5\times$ faster inference compared to U-Net baselines. This unified latent representation improves efficiency, though it reduces the use of spatially localized inductive biases, which may affect fine-grained control in complex scenes.

While the above methods emphasize factorization or disentanglement within a single generation stage, other approaches adopt multi-stage pipelines to balance quality and efficiency. LaVie [54] exemplifies this strategy through a three-stage cascaded latent diffusion architecture. It first generates low-resolution short clips conditioned on text, then increases temporal resolution through frame interpolation, and finally improves spatial resolution via video super-resolution. Each stage is optimized for a specific aspect of video quality, enabling high-definition outputs while maintaining temporal coherence. Joint fine-tuning on image and video data further supports visual diversity. However, the cascading design introduces additional training and inference stages, increasing system complexity and cumulative error sensitivity across stages.

Similarly, Zhang et al. [55] proposed Show-1, a hybrid T2V generation framework that integrates pixel-based and latent-based video diffusion models within a two-stage coarse-to-fine pipeline. Low-resolution videos are first generated using a pixel-based diffusion model with strong semantic grounding inherited from image-pretrained weights. These outputs are then refined and upscaled by a latent diffusion model acting as a super-resolution expert. Sharing a common text encoder across stages ensures semantic consistency, while selectively fine-tuning temporal attention layers enables motion customization. This hybrid design balances semantic alignment and computational efficiency, but, like other staged pipelines, depends on careful coordination between stages to avoid propagating artifacts from early generation steps.

Table A4. Summary of T2V models mentioned in the hybrid and factorized generation section.

Reference	Model Architecture	Methods	Training Strategy	Training Dataset	Evaluation Results	Baselines	Project Code
Girdhar et al. [51]	Two-stage factorized generation: frozen T2I image + latent video diffusion with temporal layers and image conditioning	Factorized conditioning + tuned noise schedules + classifier-free guidance with separate image/text weights	Frozen T2I init + multi-stage training: 256px image-conditioned → 512px zero terminal-SNR → high-motion subset fine-tune (1.6K clips)	34M licensed text–video pairs (unnamed)	FVD 317.10 (SOTA), IS 42.7 (∼SOTA)	MagicVideo; Align Your Latents; Make-A-Video; PYOCO	https://emu-video.metademolab.com
Chen et al. [52]	Stable Diffusion backbone + factorized 3D U-Net temporal modules + separate spatial and temporal modules	Disentangled spatial/temporal training + partial temporal tuning + LoRA fine-tuning + frame rate conditioning	Temporal modules trained on low-quality videos → spatial modules trained on high-quality images → separate fine-tuning	WebVid-10M (low-quality videos) + JDB (Midjourney-synthesized high-quality images)	Visual Quality 63.28 (<Gen-2 67.35), Text–Video Alignment 64.67 (<AnimeDiff 74.79 and VideoCrafter1 66.76), Motion Quality 53.95 (>Show-1 53.74 and AnimeDiff 51.38)	Gen-2; Pika Labs; VideoCrafter1; Show-1; AnimateDiff	https://github.com/AILab-CVC/VideoCrafter
Menapace et al. [53]	Spatiotemporal transformer (FIT) + extended EDM diffusion for video	Joint modeling of spatial and temporal redundancies	Two-stage training: pre-training on lower-res videos → fine-tuning on high-res videos	Large-scale video–text datasets (unnamed)	UCF-101 (zero-shot): FVD ∼200 ($512\times 288$) (SOTA), 260 ($288\times 288$) (SOTA), IS 38.9 (∼SOTA)	CogVideo; MagicVideo; LVDM; Video LDM; VideoFactory; Make-A-Video; PYoCo	https://snap-research.github.io/snapvideo
Wang et al. [54]	Cascaded latent diffusion models + temporal self-attention + rotary positional encoding	Latent diffusion + temporal self-attention for frame coherence + temporal interpolation in latent space	Joint image-video fine-tuning + cascaded training	Curated Vimeo25M dataset with 25 million text–video pairs	UCF101: FVD 526.30 (SOTA), MSR-VTT: CLIP-SIM 0.29 (∼SOTA)	CogVideo; Make-A-Video; VideoFusion; Magic Video; LVDM; Video LDM	https://github.com/Vchitect/LaVie
Zhang et al. [55]	Two-stage hybrid: pixel-based diffusion for low-res generation + latent diffusion for high-res upscaling	Hybrid pixel-latent diffusion pipeline + expert translation module for super-resolution	Multi-stage training (keyframe generation → frame interpolation → SR → expert fine-tuning)	WebVid-10M	IS 36.02 (SOTA), FVD 369.33 (∼SOTA)	CogVideo; Make-A-Video; MagicVideo; Video LDM; VideoFactory	https://github.com/showlab/Show-1

All project links were accessed on 16 February 2026. Metrics. Visual Quality, Text–Video Alignment, and Motion Quality are composite evaluation scores reported by the original authors to assess perceptual fidelity, semantic alignment between text and generated video, and motion realism, respectively. These metrics are typically computed using a combination of automated perceptual measures and learned evaluators defined within each work. Zero-shot evaluation indicates that models are assessed without task-specific fine-tuning on the target benchmark. All metrics are reported as described in the original papers and are not directly comparable across different evaluation settings.

provides an overview of the studies mentioned in this section.

5.1.2. Controllable T2V Generation

Unlike fully automatic T2V generation, controllable video generation focuses on synthesizing videos in which specific aspects of the content—such as object appearance, spatial layout, motion trajectory, temporal dynamics, or style—can be manipulated by the user. This category covers many control modalities, including trajectory and motion trajectory control, camera and pose control, entity- and object-level Control, multi-prompt and temporal editing, and frequency- and identity control. Figure 10 illustrates the works from the selected literature, organized into these subcategories, which are detailed in the following sections.

(a)

Trajectory and motion trajectory control

Trajectory and motion trajectory control in T2V generation refers to methods that allow users to define the spatial and temporal movements of objects, camera viewpoints, and scene elements. Rather than relying solely on text prompts, these approaches incorporate user-defined motion specifications—such as predefined paths, keyframes, or movement constraints—to guide the generation process.

One class of methods provides indirect motion control by extracting motion priors from pretrained models. For example, Gal et al. [56] introduced a method to animate static sketches using pretrained T2V diffusion models guided by text prompts. Instead of training a dedicated model, their approach distills motion priors from large T2V models to produce short, semantically meaningful vector-based animations. Motion is decomposed into local stroke deformations and global affine transformations, ensuring both fine detail and structural coherence. The key mechanism is a score-distillation optimization scheme that adjusts motion parameters based on the pretrained model’s learned video distribution, encouraging motions that are both natural and semantically aligned with the text prompt. This approach is computationally efficient and training-free; however, because motion is inferred implicitly from pretrained priors, the degree of user control remains limited, and extending the method to complex or long-range trajectories may be challenging.

In contrast to indirect control, some works pursue direct trajectory conditioning so users can draw or specify exact paths. For instance, Zhang et al. [57] introduced Tora, the first trajectory-oriented DiT framework, enabling controllable video generation conditioned on text, image, and explicit 2D motion trajectories for precise user guidance. At its core, the Trajectory Extractor (TE), which encodes 2D trajectories—represented as sequences of points—into a compact latent forms by converting them into RGB flow-like maps and processing them with a 3D VAE. These motion representations are injected into a Spatial—Temporal Diffusion Transformer via a Motion-Guidance Fuser to ensure that generated content adheres closely to the specified paths. Training proceeds in two stages, first learning motion from dense optical flow data and then adapting to sparse trajectory annotations. While this design enables control over motion, it also introduces additional architectural components and supervision requirements, which may increase system complexity and limit accessibility for casual users.

Bridging indirect motion priors and direct trajectory specification, Li et al. [58] proposed Video-MSG, a training-free guidance framework that enhances T2V diffusion models’ ability to follow complex text prompts involving spatial layouts and object trajectories. Video-MSG operates by first generating a Video Sketch—a structured spatiotemporal blueprint derived from multimodal planning that combines text, detected objects, segmentation results, and large multimodal language model reasoning. This sketch is then used to initialize the diffusion process through structured noise initialization, injecting spatially and temporally coherent priors without modifying the underlying model. Guided diffusion further refines the output to match the intended layout and dynamics. Although Video-MSG avoids retraining or architectural changes, its reliance on multiple pretrained components and accurate upstream detections introduces dependencies that may affect robustness in complex scenes.

A different line of work focuses on motion transfer: extracting motion from a reference clip and applying it to new subjects while preserving appearance diversity. Zhao et al. [59] proposed MotionDirector, a framework that disentangles motion from appearance and frames motion customization as adapting a pretrained T2V model to reproduce reference motions independently of subject appearance. The framework employs a dual-path LoRA architecture that separates spatial appearance modeling from temporal motion modeling. An appearance-debiased temporal loss further reduces interference between motion and appearance, which allows motion patterns to generalize across subjects. By updating only low-rank adapters and freezing the base model, MotionDirector achieves efficient adaptation without full retraining. Nevertheless, its effectiveness depends on the quality and representativeness of the reference motion, and transferring highly complex or interacting motions may remain challenging.

Similarly, Jeong et al. [60] introduced VMC (Video Motion Customization), a framework for customizing T2V diffusion models to generate videos with user-specified motion patterns. VMC introduces a motion distillation loss based on residual latent-frame differences to capture smooth, low-frequency motion while removing high-frequency artifacts. To decouple motion from appearance, prompts are transformed into appearance-invariant forms, enabling consistent motion reproduction across varied scenes. However, its focus on low-frequency motion may limit expressiveness for highly dynamic or abrupt movements.

Finally, Ren et al. [61] proposed Customize-A-Video, a framework for one-shot motion customization that transfers motion patterns from a single reference video to new subjects or scenes. The method employs a staged training strategy using LoRA-based adaptation of temporal attention layers, combined with appearance absorbers that isolate and neutralize appearance information before motion learning. By freezing the pretrained backbone and adapting only plug-and-play modules, the framework enables efficient motion transfer with minimal data. However, the reliance on staged training and auxiliary modules adds to system complexity and may need careful tuning for stable performance.

(b)

Camera and Pose Control

Camera and pose control in T2V generation refers to methods that enable users to manipulate camera viewpoints and motion (e.g., position, rotation, focal length, and temporal trajectories) as well as human or character poses and their temporal trajectories, thereby enabling the creation of cinematic visual content.

A representative approach targeting explicit camera control is AC3D, introduced by Bahmani et al. [62], which focuses on precise 3D camera control within video diffusion transformer models. Built on transformer-based video diffusion backbones—extending VD3D and VDiT architectures—AC3D processes video latent tokens alongside dedicated camera tokens. Camera poses are encoded using Plücker coordinates and transformed into camera tokens via a fully convolutional encoder, ensuring compatibility with video tokens in both spatial and channel dimensions. These camera tokens are further processed by lightweight transformer blocks before being injected into the diffusion model. A key insight of this work is that camera-induced motion primarily manifests in low-frequency video components. Based on this analysis, the authors redesign pose-conditioning schedules and restrict camera conditioning to early network layers, where pose information is shown to concentrate. Consequently, the number of trainable parameters is reduced by $4\times$ while improving visual fidelity by 10%. To better disentangle camera and scene motion, AC3D introduces a 20K-video dataset recorded with stationary cameras, enhancing the model’s ability to generate natural, pose-conditioned videos. However, the reliance of AC3D on structured camera representations and controlled data may limit its applicability to unconstrained, in-the-wild video scenarios.

Moving from camera motion to character control, Follow Your Pose, proposed by Ma et al. [63], addresses pose-controllable character video generation without requiring paired video–pose–caption datasets. The approach builds on a pretrained text-to-image (T2I) diffusion model backbone extended for video generation. In the first stage, pose-conditioned T2I generation is learned using keypoint–image pairs, injecting pose information through a zero-initialized convolutional encoder while preserving the compositional and editing capabilities of the pretrained model. The second stage fine-tunes this model on large-scale pose-free videos by introducing learnable temporal self-attention and cross-frame attention blocks, enabling temporal consistency without explicit pose labels. This two-stage strategy allows pose control to be learned with limited annotation while leveraging abundant unlabeled video data. However, the absence of pose labels during video fine-tuning means that pose accuracy over long sequences depends on the model’s learned temporal priors, which may degrade under complex or fast motions.

Finally, Guo et al. [64] presented SparseCtrl, a plug-and-play framework that enhances controllability in T2V diffusion models via temporally sparse structural conditioning. Unlike traditional methods that require dense per-frame conditioning inputs (e.g., depth maps or sketches), SparseCtrl uses only a few condition frames scattered across the video. A modality-agnostic condition encoder with lightweight adapter heads propagates this sparse information through time using temporal-aware transformer blocks, while keeping the pretrained T2V backbone frozen. During training, random temporal sparse masks simulate different sparsity levels by varying the number and positions of conditioning frames, improving robustness at inference. This design significantly lowers annotation cost and enables plug-and-play controllability without full-model retraining. At the same time, relying on sparse conditioning introduces a trade-off between control precision and flexibility, as fine-grained or rapidly changing camera and pose motions may be difficult to enforce with limited conditioning signals.

(c)

Entity- and object-level control

Entity- and object-level control in T2V generation refers to methods that allow users to manipulate individual objects or entities within a video at a fine-grained semantic level, controlling appearance, motion, spatial placement, and interactions via signals beyond text prompts such as reference images and user-drawn trajectories.

For instance, Wu et al. [65] proposed DragAnything, a framework for controllable video generation that enables precise, intuitive motion manipulation at the object level by operating on semantically meaningful entity representations rather than pixel-level cues. DragAnything is built on Stable Video Diffusion, which employs a 3D U-Net backbone to encode and decode video latent representations. The framework extracts object-level embeddings by indexing latent diffusion features with segmentation masks, for example obtained from the Segment Anything Model (SAM). Users guide video generation by drawing motion trajectories, which are converted into 2D Gaussian-weighted signals focused on object regions to emphasize coherent movement of entire entities. During denoising, the model conditions generation on the initial frame, the entity embedding, and the trajectory signal, enabling localized motion while preserving the rest of the scene. Training relies on annotated video segmentation datasets and object tracking data generated via Co-Tracker. This design enables fine-grained and interpretable control, but it also depends on accurate segmentation and tracking, which makes performance sensitive to errors in upstream object masks or trajectory extraction, particularly in cluttered or occluded scenes.

While DragAnything emphasizes entity embeddings and trajectory guidance, other works exploit motion priors learned by pretrained T2V models to animate arbitrary visual content. For instance, DynamiCrafter, introduced by Xing et al. [66], animates still images by transferring motion priors from pretrained diffusion-based T2V models. Unlike domain-specific animation methods, DynamiCrafter generalizes across diverse visual content, including objects, animals, CGI, and art, while preserving high-resolution appearance (up to $576\times 1024$). The framework employs a dual-stream image injection design: semantic context stream projects the input image into a semantic embedding space aligned with the pretrained T2V model using a CLIP encoder and a learnable query transformer, while a visual detail stream concatenates the full-resolution image with the initial latent noise to retain fine-grained appearance. A three-stage training procedure progressively adapts the context network and spatial layers while preserving temporal motion priors. This separation enables flexible animation of static images, though the motion remains governed by dynamics learned by the pretrained model, which limits user control over specific object interactions or complex motion trajectories.

Finally, VideoBooth, proposed by Jiang et al. [67], integrates both image and text prompts to produce precise, user-controlled videos. VideoBooth introduces a coarse-to-fine conditioning strategy: high-level visual features extracted from image prompts are mapped into the text embedding space to guide global semantics, while multi-scale image features are injected into cross-frame attention layers to refine spatial detail and temporal coherence. By training only lightweight visual encoders and attention injection modules on WebVid-10M while keeping the T2V backbone frozen, VideoBooth enables feed-forward inference without per-instance fine-tuning. This design provides strong identity preservation and controllability; however, its reliance on image prompts and pretrained backbones restricts its flexibility in scenarios that require substantial changes to object appearance or behavior beyond what can be inferred from a single reference image.

(d)

Multi-prompt and temporal editing

Multi-prompt and temporal editing in T2V generation refers to methods that enable coherent and semantically consistent video synthesis guided by multiple sequential or overlapping text prompts over time. Unlike single-prompt methods that assume a static instruction, these methods handle dynamic scenarios where the content, scene, or actions evolve with changing textual inputs.

A representative training-free approach in this category is DiTCtrl, proposed by Cai et al. [68], which enables the generation of long-form videos from multiple sequential text prompts within a Multi-Modal Diffusion Transformer (MM-DiT) framework. MM-DiT employs full 3D attention to jointly model spatial and temporal dependencies. DiTCtrl reformulates multi-prompt generation as a temporal video editing problem rather than treating each prompt independently. Its core innovation is a mask-guided attention control mechanism that modifies MM-DiT’s 3D full attention to token subsets associated with each prompt, allowing accurate conditioning over different temporal regions of the video. To preserve semantic continuity across segments, DiTCtrl introduces a key-value sharing across attention layers, propagating contextual information such as object identity and motion patterns between segments. To further enhance temporal coherence, a latent blending strategy merges overlapping latent video representations at segment transitions using position-dependent weights to reduce abrupt visual changes. This design enables multi-prompt generation without retraining; however, because control is exercised indirectly through attention manipulation, the approach may be sensitive to prompt segmentation choices and relies on the pretrained model’s capacity to maintain consistency under constrained attention.

In parallel to attention-based temporal editing, DreamVideo, introduced by Wei et al. [69], addresses multi-prompt and temporal variation by decoupling subject identity and motion. DreamVideo is a personalized T2V framework that generates videos of custom subjects (from images) performing custom motions (from videos) by learning appearance and motion separately within a unified diffusion-based architecture. Built on a pretrained video diffusion backbone with temporal attention, the method introduces two lightweight adapters: an identity adapter, integrated primarily into the spatial cross-attention layers to encode subject-specific appearance, and a motion adapter, inserted into temporal layers to capture motion dynamics. Training proceeds in two stages. First, a textual identity embedding is learned from a small number of images using Textual Inversion and refined through the identity adapter while freezing the backbone. Second, the motion adapter is trained on motion reference videos with the identity components fixed. During inference, the independently trained adapters are combined without additional optimization. This modular design enables flexible composition of subjects and motions and supports temporal editing across prompts. However, the frozen-backbone assumption and sequential training place constraints on how strongly identity and motion can interact, which may restrict expressiveness in scenarios with complex appearance—motion coupling.

(e)

Frequency- and identity control

Frequency- and identity-control methods in T2V generation refers to those designed to generate videos that preserve human identity—especially facial fidelity—across video frames. Instead of relying only on spatial conditioning or fine-tuning, recent approaches exploit frequency-domain characteristics of facial features to achieve more stable identity preservation.

A representative example is ConsisID, introduced by Yuan et al. [70], a tuning-free framework designed to maintain facial identity consistency in T2V generation. The framework is built on a pretrained DiT backbone, which replaces conventional U-Net architectures with high-capacity transformer blocks for video denoising. The core idea is to decompose facial identity into low- and high-frequency components corresponding to different identity representations. Low-frequency features, encoding global aspects such as face shape and structure, are extracted using a global facial feature module and injected into early transformer layers to stabilize identity. High-frequency features, encoding fine details like skin texture and subtle appearance cues, are derived from a local extractor and integrated into deeper layers to preserve appearance realism. A hierarchical training strategy adapts a pretrained video diffusion model by incorporating these frequency-specific features, transforming it into an identity-preserving T2V system (IPT2V) that ensures coherence and realism across generated frames.

This frequency-aware design provides a principled mechanism for maintaining both structural and textural aspects of identity across frames, leveraging the representational hierarchy of DiT architectures. However, since identity preservation is enforced through predefined frequency separation and fixed feature injection points, the approach assumes that facial identity can be cleanly decomposed along frequency lines. This assumption may restrict flexibility when identity cues interact with dynamic expressions, lighting changes, or occlusions. Moreover, while tuning-free operation improves practicality, it also limits the system’s ability to adapt identity representations beyond the pretrained feature extractors.

Table A5. Summary of T2V models mentioned in the controllable T2V generation section.

Reference	Model Architecture	Methods	Training Strategy	Training Dataset	Evaluation Results	Baselines	Project Code
Gal et al. [56]	Lightweight network controlling sketch strokes + pretrained T2V motion prior with local deformation and global affine components	Score distillation sampling (SDS) loss + vector Bézier curve sketch representation	Optimization-based, no training or fine-tuning	None	Alignment 0.142; Sketch consistency 0.965 (both SOTA)	ZeroScope; ModelScope; VideoCrafter	https://github.com/yael-vinker/live_sketch
Zhang et al. [57]	DiT backbone (OpenSora) with: Trajectory Extractor (3D VAE) + Spatial–Temporal DiT blocks + Motion-guidance Fuser	Trajectory-conditioned video generation + MGF hierarchical fusion + diffusion with text/visual conditions + alt. spatial-temporal attention	3D Training of 3D VAE on flow maps + Joint training of diffusion transformer and MGF on trajectory-annotated video–text data	630,000 videos from: Panda-70M + Mixkit + Pexels + Internal sources	FVD 494, CLIP-SIM 0.2418, TrajError 11.72 (∼SOTA across all metrics)	VideoComposer; DragNUWA; AnimateAnything; TrailBlazer; MotionCtrl; OpenSora	https://github.com/alibaba/Tora
Li et al. [58]	Pretrained T2V diffusion model + pipeline of multimodal LLM planner	Background planning, foreground layout and trajectory planning + structured noise init. + MLLM/vision models	No training or fine-tuning	None	Consist-attr 0.7109, Spatial 0.6070, Motion 0.4487, Numeracy 0.3647 (∼SOTA across all metrics)	Pika; ModelScope; ZeroScope; AnimateDiff; Latte; Show-1; Open-Sora 1.2; VideoCrafter2; CogVideoX-5B	https://github.com/jialuli-luka/Video-MSG
Zhao et al. [59]	Pretrained 3D U-Net T2V diffusion backbone + dual-path spatial and temporal LoRA modules	Motion customization via decoupled LoRA tuning + appearance-debiased temporal loss + Temporal Attention Purification	Fine-tune spatial LoRAs on single frames + temporal LoRAs on multiple frames + backbone frozen	UCF Sports Action	CLIP metrics: Alignement 27.82 (∼SOTA), Appearance Diversity 28.48 (SOTA), Temporal Consistency 93.00 (SOTA), Pick Score 20.74 (SOTA); Human Eval: ∼SOTA across all metrics	VideoComposer; Control-a-Video; VideoCrafter; Tune-a-Video	https://github.com/showlab/MotionDirector
Jeong et al. [60]	Pretrained cascaded VDM backbone + adapted temporal attention layers	Motion distillation via residual latent frame vectors + appearance-invariant prompt transformation	Parameter-efficient fine-tuning on temporal attention layers only + motion distillation loss + frozen backbone	Few, short videos	CLIP metrics: Text Alignment 0.801, Temporal Consistency 0.959; User study: Text Alignment 4.56, Temporal Consistency 4.57 (SOTA across all metrics)	VideoComposer; Gen-1; Tune-A-Video; Control-A-Video	https://github.com/HyeonHo99/Video-Motion-Customization
Ren et al. [61]	Pretrained T2V diffusion backbone + LoRA modules on temporal attention layers + appearance absorbers	One-shot motion customization from single video + appearance absorption before motion adaptation + LoRA tuning for temporal attention	Parameter-efficient LoRA fine-tuning + two-stage training (appearance absorber training → motion LoRA tuning)	Few, short videos	SORA in text alignment and temporal consistency	ModelScope; Tune-A-Video; Video-P2P; MotionDirector	https://github.com/customize-a-video/customize-a-video
Bahmani et al. [62]	Transformer-based diffusion backbone (VDiT/VD3D) + Plücker coordinate-based camera pose encoding + lightweight DiT-XS blocks	Motion spectral analysis + layer-specific camera knowledge probing + truncated normal noise schedule + feedback connections	Training with camera conditioning only in early transformer layers + standard diffusion denoising loss + truncated normal noise	Curated 20,000 video–text pairs from RealEstate10K	RealEstate10K: 0.358 / 1.18 / 36.55 / 28.76 (SOTA); MSR-VTT: 0.428 / 5.34 / 110.71 / 28.58 (SOTA) (order: TransErr / FID / FVD / CLIP Score)	MotionCtrl; CameraCtrl; VD3D	https://github.com/snap-research/ac3d
Ma et al. [63]	Zero-initialized convolutional pose encoder + Pretrained text-to-image diffusion backbone + temporal and cross-frame self-attention blocks	Learnable temporal attention for motion coherence + preservation of pretrained T2I’s editing ability	Two-stage training: training on image-pose pairs → finetuning on pose-free videos + minimal tuning of pretrained backbone	Curated 20,000 video–text pairs from RealEstate10K	CLIP Score 24.09 (SOTA), Quality 39.96 (SOTA), Pose accuracy 34.92 (SOTA), Frame consistency 93.36 (∼SOTA)	FOMM; Everybody dance now; Tune-A-Video; ControlNet; T2I adapter; Masactrl	https://github.com/mayuelala/FollowYourPose
Guo et al. [64]	Condition encoder (shared backbone + modality heads) + frozen diffusion T2V model (AnimateDiff)	Sparse temporal control with condition propagation + masking-based sparsity simulation + purging noised ControlNet inputs + multimodal control support	Training of encoder only + freezing T2V backbone	WebVid-10M	CLIP metrics: 31.39/87.29/98.00 (∼SOTA); User study: 2.21/2.28/2.82 (∼SOTA) (Order: Alignement/Domain similarity/Motion smoothness)	Text2Video-Zero; Tune-a-Video; Gen-2; Pika Lab	https://github.com/guoyww/AnimateDiff
Wu et al. [65]	Stable Video Diffusion backbone (3D U-Net) + entity representation + conditional denoising autoencoder	Segmentation tool (SAM) + 2D Gaussian creation + user trajectory input + Co-Tracker for trajectories	Supervised training with MSE loss focused on entity regions	VIPSeg	FVD 494 (SOTA), FID 33 (SOTA)	DragNUWA	https://github.com/showlab/DragAnything
Xing et al. [66]	Pretrained T2V diffusion backbone + CLIP image encoder + query Transformer + gated fusion mechanism with image/text conditioning	Dual-stream image injection (text-aligned context + visual detail guidance) + generative frame interpolation + looping videos	Three-stage training: training the image context network → adapting with T2V → joint fine-tuning with VDG	WebVid-10M	UCF-101: 429.23/62.47/0.60; MSR-VTT: 234.66/13.74/0.58 (Order: FVD/KVD/PIC)	VideoComposer; I2VGen-XL	https://github.com/Doubiiu/DynamiCrafter
Jiang et al. [67]	Pretrained T2V diffusion backbone + CLIP image encoder + attention injection module + cross-frame and temporal attention layers	Hierarchical image prompt embedding + attention injection into cross-frame attention layers + conditioning on text and image jointly	Two-stage coarse-to-fine training: training MLP encoder → training attention injection module	WebVid-10M	CLIP-Text 30.09 (∼SOTA), CLIP-Image 74.79 (SOTA), DINO 65.09 (SOTA)	Textual Inversion; DreamBooth; ELITE	https://github.com/Vchitect/VideoBooth
Cai et al. [68]	Multi-Modal Diffusion Transformer (MM-DiT) backbone with 3D full attention	Mask-guided attention sharing + latent blending + prompt token reweighting	No training or fine-tuning	None	CSCV 84.90%, Motion smooth 97.80%, Text-image sim 30.68% (∼SOTA)	Gen-L-Video; FreeNoise; Video-Infinity; FreeNoise+DiT	https://github.com/TencentARC/DiTCtrl
Wei et al. [69]	Pretrained video diffusion backbone U-Net + image retention branch + convolutional image feature extractor	Image-to-video generation + low-level image feature concatenation + double-condition guidance	Two-stage training: training of the identity adapter → fine-tuning of the motion adapter	Pexels 300K	Qualitative Eval	Gen-2; Pika	https://github.com/ali-vilab/Vgen
Yuan et al. [70]	DiT backbone + global facial extractor + local facial extractor	Frequency-aware identity control with LF/HF features + dynamic mask loss (face) + dynamic cross-face loss	Hierarchical frequency-aware training + joint optimization of facial extractors with DiT backbone	Large-scale human face video datasets (unnamed)	CLIP Score 27.93 (SOTA), FID 151.82	ID-Animator	https://github.com/PKU-YuanGroup/ConsisID

All project links were accessed on 16 February 2026. Metrics. Alignment, Sketch Consistency, and Consist-attr quantify how well generated videos preserve user-specified attributes, structures, or constraints. TrajError measures deviation between generated motion and target trajectories (lower is better). Numeracy evaluates correctness in object counting tasks. Pick Score reflects preference-based ranking derived from learned or human-aligned evaluators. Quality, Pose Accuracy, and Frame Consistency assess perceptual fidelity, pose correctness, and temporal stability, respectively, as defined by the original authors. TransErr denotes camera or motion translation error. KVD (Kernel Video Distance) measures distributional similarity between generated and real videos. PIC evaluates perceptual image consistency in animated frames. DINO measures feature similarity using self-supervised vision representations. CSCV (CLIP Similarity Coefficient of Variation) measures semantic consistency across frames under multi-prompt generation. Motion Smooth quantifies temporal continuity of motion. CLIP-Text and CLIP-Image measure alignment between generated videos and text or image prompts. Domain Similarity measures visual distribution consistency across domains. Frequency-aware identity metrics evaluate preservation of low- and high-frequency facial identity features. All metrics are reported as defined by the original authors and are not directly comparable across different evaluation settings.

provides an overview of the T2V models discussed in the controllable T2V generation section.

5.1.3. Video Style Transfer and Editing

Video style transfer and editing aim to apply visual styles or semantic modifications from a reference video to a target video, while preserving temporal coherence and motion dynamics. Unlike image-based style transfer, videos must ensure consistent frame-to-frame continuity, avoiding flicker or visual artifacts.

One representative example is CAMEL, proposed by Zhang et al. [71], which frames text-driven video editing as a problem of separating motion dynamics from appearance content. Built on a pretrained latent video diffusion model, CAMEL introduces motion prompts; namely, learned embeddings that capture motion characteristics from template videos. These prompts are injected into the latent space of diffusion models to guide the editing process, allowing textual modifications to alter appearance while preserving underlying motion patterns. To further improve temporal consistency, CAMEL incorporates a causal motion-enhanced attention mechanism and a causal motion filter to isolate high-frequency motion features from low-frequency appearance details. Drawing from wavelet transform principles, the framework decomposes video sequences into frequency bands, enabling fine-grained control over motion and appearance. This design addresses limitations of earlier approaches that tightly couple motion and appearance, though it also introduces additional modeling assumptions about frequency separability and relies on suitable motion templates to guide editing effectively.

By contrast, Style-A-Video, introduced by Zhang et al. [72], emphasizes flexibility and ease of use through zero-shot text-guided stylization. Rather than learning motion representations from reference videos or requiring style-specific training, Style-A-Video enables arbitrary style transfer directly from textual prompts without fine-tuning. The framework combines a generative pretrained transformer with an image latent diffusion model to produce stylized videos directly from text prompts. To balance stylization and structural fidelity, the model refines guidance conditioning during the denoising process, preserving essential content features while applying the desired artistic transformations. Additionally, a temporal consistency module and optimized sampling strategies further reduce inter-frame flicker and improves visual continuity. While this zero-shot design provides strong flexibility and low deployment cost, the absence of motion modeling means that temporal coherence relies mainly on the pretrained model’s dynamics, which may limit control over complex motion patterns.

5.1.4. Video Quality and Inference Enhancement

Complementing generation-focused approaches, some efforts have emerged to address post-generation refinement by improving the perceptual clarity, fidelity, and temporal coherence of videos produced by T2V models. Rather than redesigning architectures or retraining large models, many of these methods operate at inference time or as external refinement modules.

One such approach is NeuS-E, introduced by Choi et al. [73], which targets semantic and temporal inconsistencies through a neuro-symbolic enhancement pipeline. NeuS-E operates independently of the underlying T2V model and does not require retraining or architectural modification. The method translates text prompts into formal Temporal Logic specifications that encode desired temporal relationships and event sequences. Generated videos are abstracted into automaton representations using a vision–language model, where frames correspond to states labeled with content-derived predicates (e.g., abrupt object appearances or contradictory scene transitions). Probabilistic model checking is then used to verify whether the video satisfies the specified temporal logic, outputting a confidence score that reflects semantic and temporal alignment. Finally, the detected semantic inconsistencies are fed back to guide targeted video edits of problematic frames or segments. However, the framework relies on the accuracy of both the vision—language abstraction and the logic specification, which may limit scalability to highly complex or ambiguous prompts.

Along similar lines of improving generation quality without retraining, FreeU, introduced by Si et al. [74], focuses on enhancing diffusion-based models through a lightweight inference-time adjustment. The authors observe that in U-Net architectures, the backbone is primarily responsible for semantic structure, while skip connections inject high-frequency details that can overwhelm semantic structure. FreeU rebalances the contributions of these components using two scaling factors applied during inference. This lightweight modification improves generation fidelity without altering model parameters, retraining, or increasing computational cost. However, because it does not integrate explicit semantic or temporal reasoning, its improvements are mainly perceptual and may not address deeper inconsistencies in motion or narrative structure.

While NeuS-E and FreeU emphasize quality refinement, other methods directly target inference efficiency. TeaCache, introduced by Liu et al. [75], accelerates video diffusion inference through a training-free, model-agnostic caching method. While video diffusion models produce high-fidelity outputs, their sequential denoising process leads to slow inference. Rather than caching intermediate outputs at fixed timesteps, TeaCache estimates output variation using timestep embeddings that modulate noisy inputs during diffusion. By identifying timesteps where output changes are minimal, the method selectively reuses cached results and avoids redundant computation. TeaCache also incorporates a rescaling strategy that refines variation estimates without additional model calls. This adaptive caching improves efficiency while preserving visual quality; however, its effectiveness depends on accurate variation estimation and may diminish in scenarios with highly dynamic content where outputs change rapidly across timesteps.

Beyond perceptual quality and efficiency, PhyT2V, proposed by Li et al. [76], addresses physical realism through an inference-time self-refinement loop guided by large language models. PhyT2V operates as a plug-and-play module compatible with existing T2V diffusion models and does not require retraining or additional data. The framework parses the original prompt to extract entities and implicit physical rules, generates a candidate video, and captions it to summarize observed behavior. An LLM then compares expected physical behaviors with the generated content, identifies violations, and refines the prompt to include clearer physical constraints. This iterative loop—including rule extraction, mismatch detection, and prompt refinement—repeats until the output achieves satisfactory physical fidelity. While this approach leverages commonsense reasoning, it introduces additional inference overhead and depends on the reliability of automatic captioning and LLM-based reasoning to correctly diagnose physical errors.

In parallel, Inflation With Diffusion, proposed by Yuan et al. [77], focuses on enhancing spatial resolution and temporal coherence by adapting pretrained image super-resolution diffusion models for video generation. The method inflates a pretrained text-to-image super-resolution diffusion model into a video UNet by sharing residual and cross-attention components across frames, preserving spatial priors learned from image data. A lightweight temporal adapter is added to capture inter-frame dynamics, enabling temporal consistency without sacrificing the spatial quality inherited from the original image model. However, its reliance on limited temporal adaptation may constrain performance in videos with complex motion patterns.

Finally, IPO, proposed by Yang et al. [78], improves T2V generation through iterative post-training preference optimization. Unlike one-shot fine-tuning, IPO refines T2V models through multiple optimization rounds using preference feedback from pairwise comparisons and pointwise quality scores. A critic model, trained using multimodal large language models, evaluates generated videos based on criteria such as semantic relevance, motion smoothness, subject consistency, and aesthetic quality. The T2V model is then updated using preference-based objectives such as Direct Preference Optimization or Kahneman–Tversky Optimization. This iterative loop gradually aligns outputs with inferred human preferences without requiring extensive manual annotation. However, the approach introduces additional training complexity and depends on the critic’s ability to provide reliable and unbiased preference judgments.

5.1.5. Comparative Analysis of Architectural Trade-Offs in Video Synthesis Models

While recent T2V models vary widely in architectural design, many of their performance gains arise from recurring trade-offs between generation quality, temporal coherence, and computational efficiency. In particular, the choice between U-Net-based diffusion architectures and DiT-based models plays a key role in shaping output quality and system scalability.

U-Net-based diffusion models remain attractive due to their relatively low memory footprint and strong inductive biases for local spatial structure. When extended with temporal convolutions or lightweight attention modules, they can effectively generate short video clips with stable motion and low computational requirements. However, their reliance on localized receptive fields limits their capability to model long-range temporal dependencies, which results often in temporal drift, object inconsistency, or low coherence in longer videos. This limitation motivates the use of transformer-based diffusion architectures in recent work.

Diffusion Transformer-based models address these challenges by replacing convolutional inductive biases with global self-attention across spatiotemporal tokens. This design enables more expressive modeling of long-range temporal relationships, complex object interactions, and temporal consistency, which is critical for long-form and high-resolution video synthesis. Models such as Sora-style DiTs, CogVideoX, Vidu, and Open-Sora Plan demonstrate that full spatiotemporal attention improves semantic alignment and motion consistency. However, these gains come at the cost of significantly higher computational and memory demands, as attention complexity scales quadratically with the number of tokens.

To mitigate these costs, recent DiT-based approaches adopt efficiency-oriented architectural refinements, including token compression via 3D VAEs, sparse or skip attention mechanisms, causal or autoregressive attention, and key-value caching. Autoregressive diffusion variants, such as CausVid, ART-V, and PA-VDM, further trade bidirectional context for causal generation, enabling progressive synthesis with lower inference latency and memory usage. While autoregressive designs may sacrifice some global optimality compared to fully bidirectional diffusion, they provide better temporal scalability and interactive generation capabilities, making them well suited for long-duration or real-time applications.

Beyond architectural design, post-training preference alignment represents an orthogonal strategy for improving video generation quality. Rather than modifying the backbone architecture, preference-aligned methods refine pretrained T2V models using human or proxy feedback signals to improve perceptual quality, semantic alignment, and user satisfaction. These approaches enhance output quality without increasing inference-time complexity but they introduce additional optimization stages and depend on reliable preference data or reward models.

Overall, these trends indicate that recent progress in video synthesis arise not from a single dominant architecture, but from carefully balancing expressiveness and efficiency. Transformer-based diffusion models favor generation quality and long-range coherence, whereas U-Net-based and autoregressive designs emphasize computational tractability. Preference-aligned post-training further refines generation quality without altering model structure. The increasing convergence of these approaches through hybrid architectures, factorized generation, adaptive attention, and post-training alignment, reflects that scalable, high-quality T2V systems must handle the trade-off between model capacity and computational cost rather than optimizing any single factor in isolation.

Table 1. Summary of architectural trade-offs in text-to-video generation models.

Architecture	Key Characteristics	Strengths (Quality)	Limitations (Compute / Scalability)
U-Net–based diffusion models	Convolutional backbone with temporal convolutions or lightweight attention; strong local inductive bias	Efficient training and inference; stable short-term motion; good spatial fidelity for short clips	Limited long-range temporal modeling; temporal drift in long videos; reduced narrative coherence
Diffusion Transformer (DiT)–based models	Full spatiotemporal self-attention over video tokens; global receptive field	Superior long-range temporal coherence; strong semantic alignment; high visual consistency in long-form videos	High memory usage; quadratic attention cost; expensive inference for long or high-resolution videos
Autoregressive diffusion models	Causal attention or frame-wise generation; diffusion conditioned on past frames only	Improved scalability; reduced memory footprint; supports progressive and interactive generation	Reduced global optimality compared to bidirectional diffusion; error accumulation over long sequences
Latent-space diffusion with 3D VAE	Video compressed into spatiotemporal latent tokens prior to diffusion	Enables longer videos and higher resolutions; improved efficiency without severe quality loss	Compression may discard fine-grained motion details; additional VAE training complexity
Factorized attention models	Sparse, skip, or windowed attention; decoupled spatial and temporal modeling	Better balance between quality and efficiency; scalable to longer durations	Requires careful tuning; may underperform full attention on complex global interactions
Hybrid or cascaded generation pipelines	Multi-stage generation (e.g., low-res to high-res, image-to-video factorization)	High visual fidelity; modular optimization of spatial and temporal quality	Increased pipeline complexity; longer end-to-end generation time
Preference-aligned post-trained models	Fine-tuning via human or proxy preference signals (e.g., DPO)	Improved perceptual quality and semantic alignment without architectural changes	Requires preference data or reliable reward models; limited impact on raw generation capacity

consolidates these observations by contrasting major architectural families in terms of their defining characteristics, quality-related strengths, and limitations related to computation and scalability.

5.2. Video Understanding

Video understanding focuses on interpreting and analyzing video content to support tasks such as object segmentation, activity recognition, tracking, and high-level video reasoning. In the context of T2V generation, these capabilities are essential for guiding generation quality, as they allow models to verify that generated scenes contain the correct objects, maintain spatial accuracy, and preserve object boundaries across frames.

One line of work extends strong vision-first segmentation models to better handle language and temporal cues. Cuttano et al. [79] introduced SAMWISE, a lightweight extension to the Segment Anything 2 (SAM2) framework [80], designed for streaming-capable referring video object segmentation (RVOS). Built on a frozen SAM2 backbone that combines a hierarchical Vision Transformer visual encoder, a prompt encoder, and memory-based mask decoder, SAMWISE augments both visual and textual encoders with a Cross-Modal Temporal adapter. This adapter injects multimodal and temporal information directly into the feature extraction pipeline without modifying the pretrained backbone. A key contribution is a learnable tracking bias adjustment mechanism that addresses SAM2’s tendency to over-focus on previously tracked objects, enabling attention to shift when textual prompts change. This design improves flexibility and prompt alignment in dynamic scenarios. However, because SAMWISE is based on a segmentation-centric backbone, its reasoning capacity is mainly restricted to object-level tracking and may not extend naturally to more abstract or long-range semantic understanding.

In contrast, Zhu et al. [81] explored an alternative direction: leveraging pretrained T2V diffusion models themselves as frozen backbones for video understanding. Their framework, VD-IT, is based on the hypothesis that T2V models inherently encode temporally coherent and semantically aligned representations suitable for language-conditioned video understanding. Using ModelScopeT2V as a feature extractor, VD-IT avoids full-model fine-tuning and instead introduces lightweight adaptation modules. A Text-Guided Image Projection module fuses text embeddings from the T2V model with CLIP-extracted visual tokens to produce enriched frame-level representations, while a Video-Specific Noise Prediction Module refines reverse-diffusion features to improve temporal consistency. These representations are passed to a trainable mask decoder that produces frame-wise segmentation masks with temporal smoothness regularization. This approach demonstrates that generative models can be used as both synthesis and understanding backbones. However, its dependence on diffusion latent spaces introduces additional abstraction layers, which may complicate interpretability and limit performance in scenarios that require accurate boundary delineation under rapid motion.

Moving beyond segmentation, Chen et al. [82] introduced ShareGPT4Video, a large-scale initiative aimed at improving both video understanding and T2V generation through video captioning. The project combines three components: a curated dataset of 40,000 videos annotated with GPT-4V-generated captions, a scalable captioning system trained on this dataset that has produced millions of additional video–text pairs, and a large video-language model trained on these resources. A notable contribution is the differential captioning strategy, which enables annotation across videos of varying length and complexity by capturing both fine-grained visual details and temporal evolution. These resources improve the training signal available for large video-language models and downstream generative systems. However, the effectiveness of such data-centric approaches depends on the accuracy and consistency of automatically generated captions, and potential biases introduced by the captioning model may propagate into both understanding and generation tasks.

5.3. Objective Assessment and Benchmark

Alongside the preceding sections on T2V generation and video understanding, parallel research efforts have focused to create curated datasets and standardized evaluation protocols to benchmark T2V models. This category is divided into two subcategories—datasets and evaluation protocols—which are summarized in Figure 11 and discussed in detail in the following sections.

5.3.1. Dataset

Researchers increasingly recognize that progress in T2V depends not only on architectural advances, but also on the availability of large-scale, high-quality datasets that combine diverse video content with rich textual descriptions.

One effort in this direction is VidProM, introduced by Wang and Yang [83], which focuses on modeling real-world user interaction with T2V systems. VidProM is designed to capture authentic user prompts alongside corresponding generated outputs. It contains 1.67 million unique prompts sourced from real users, paired with 6.69 million videos generated by four state-of-the-art diffusion models: Pika, Text2Video-Zero, VideoCraft2, and ModelScope. In addition to the prompt–video pairs, VidProM includes extensive metadata such as timestamps, semantic embeddings, and toxicity scores across multiple safety categories. This rich annotation supports benchmarking, preference analysis, and research on safety and alignment. However, because the videos are generated rather than captured from the real world, the dataset mainly captures the limitations of the underlying models, which may constrain its usefulness for learning new motion or visual patterns.

In contrast to user-centric prompt datasets, MiraData, introduced by Ju et al. [84], prioritizes long-duration videos and dense, structured captions to address limitations commonly found in existing datasets, such as short clips, weak motion, and sparse textual descriptions. MiraData features videos with an average of 72 s and pairs each clip with six types of captions, ranging from high-level summaries to detailed descriptions of objects, backgrounds, camera motion, and visual style. These captions provide a multi-perspective and hierarchical understanding of each video, with an average length exceeding 200 words. The dataset is curated from diverse, manually selected sources to ensure quality and motion richness. To enable thorough evaluation, the authors introduce MiraBench, a benchmark suite with 150 prompts and 17 evaluation metrics to assess temporal consistency, motion strength, 3D coherence, visual quality, text–video alignment, and distribution similarity in generated videos. While MiraData reports improved descriptive depth and temporal coverage under the evaluated settings, its dependence on curated sources and extensive annotation may increase collection cost compared to more automated pipelines.

Addressing visual fidelity and resolution at scale, OpenVid-1M, introduced by Nan et al. [85], focuses on providing a large corpus of high-quality, high-resolution videos paired with detailed captions. Unlike commonly used datasets such as WebVid-10M and Panda-70M, that suffer from low visual fidelity or high computational demands, OpenVid-1M provides over 1 million diverse, high-resolution videos paired with rich, descriptive captions. The dataset is curated based on an automated framework that emphasizes aesthetics, temporal consistency, and clarity, ensuring that each video depicts a coherent scene. To support research on high-definition video generation, OpenVid-1M also includes a high-definition subset called OpenVidHD-0.4M with 433,000 videos at 1080p resolution. To demonstrate dataset’s utility, the authors introduced a Multi-modal Video Diffusion Transformer that jointly models visual and textual tokens. While OpenVid-1M improves visual quality and scale, its automated curation process implies a trade-off between coverage and manual verification, and its effectiveness depends on the robustness of the filtering criteria.

5.3.2. Evaluation Protocol

Early evaluation efforts generally focus on individual aspects such as temporal consistency or content continuity, using simple metrics like Fréchet Inception Distance (FID), Inception Score (IS), or CLIP similarity. However, these metrics often fail to capture the full complexity of generated videos, particularly dynamic temporal behavior and text–video alignment. Consequently, many works have moved toward human-aligned and task-specific evaluation protocols that provide a more diagnostic framework for T2V research.

A first category of work emphasizes joint assessment of video quality and text alignment. Wu et al. [86] proposed T2VScore, along with the TVGE dataset, to evaluate alignment and visual quality in a unified framework. T2VScore combines multiple specialized sub-metrics in a mixture-of-experts design to approximate human judgment across dimensions such as spatial resolution, temporal coherence, and overall visual appeal. The accompanying TVGE dataset provides human annotations for 2543 generated videos, enabling direct correlation analysis with human scores. While this approach improves over single-metric evaluations, it remains dependent on the choice and weighting of sub-metrics, which may influence sensitivity to specific failure modes.

Scaling this direction, Wang et al. [87] introduced LOVE, an LMM-based evaluation framework built around AIGVE-60K—the largest and most detailed human-annotated corpus for video evaluation to date. AIGVE-60K comprises 58,500 AI-generated videos from 30 state-of-the-art T2V models and 2.6 million human-provided annotations, including mean opinion scores evaluating perceptual video quality and QA pairs measuring semantic-text alignment and task-specific accuracy. Its dual-encoder design and instruction-tuned language model backbone support richer spatiotemporal reasoning than earlier metrics. However, the scale and complexity of this framework introduce significant computational overhead, and its dependence on large language models raises questions about evaluation bias and generalization beyond the training distribution.

Similarly, Kou et al. [88] addressed the need for human-aligned evaluation of T2V generative models by introducing the largest and most rigorously annotated T2V evaluation dataset, along with a quality assessment metric. Their contributions include the T2V Quality Assessment DataBase (T2VQA-DB), comprising 10,000 videos generated by nine state-of-the-art T2V models. They also introduce the T2V Quality Assessment (T2VQA) model, a transformer-based metric designed to predict perceptual quality by jointly analyzing semantic relevance and visual fidelity. This setup improves reproducibility and annotation reliability, but its fixed resolution and frame rate may limit its ability to reflect real-world generation scenarios with higher variability in duration and motion.

Huang et al. [89] complemented these human-centred efforts with VBench, a hierarchical benchmark suite designed to evaluate the quality of modern video generative models across a diverse set of fine-grained, disentangled dimensions. Addressing the limitations of existing metrics—which often misalign with human perception and provide limited diagnostic value—VBench decomposes “video generation quality” into 16 interpretable dimensions, including subject identity consistency, motion smoothness, temporal flickering, and spatial relationships. VBench incorporates human preference annotations to better align with perceptual quality. However, as with other dimension-based benchmarks, its effectiveness depends on the completeness and independence of the selected dimensions.

Other efforts focus on perceptual structure and distributional analysis. Qu et al. [90] proposed an evaluation framework that is structured around three key dimensions: visual harmony, video–text consistency, and domain distribution gap. Visual harmony is evaluated using DOVER, a no-reference quality assessment model with aesthetic and technical branches based on ConvNeXt and Swin Transformer, respectively. Video–text consistency is addressed using a dual-stream architecture, where textual features guide visual alignment through a cross-attention mechanism, enabling the model to evaluate how well the video reflects the intended semantics. To capture domain distribution gaps, the model includes an auxiliary classification task that predicts the generative model source from video features, improving feature discriminability and robustness across styles and quality variations among T2V models. While comprehensive, its multi-component design increases evaluation complexity and may obscure the interpretation of individual scores.

While the above efforts focus on generated video quality and semantic alignment, other benchmarks prioritize temporal reasoning and motion dynamics. For instance, Wang et al. [91] introduced T2VBench to evaluate the temporal reasoning capabilities of T2V generative models. While many models generate photorealistic frames, their ability to model temporal coherence, including realistic object motion, event sequencing, and camera transitions, remains underexplored. T2VBench addresses this gap with multi-level annotated prompts to evaluate temporal coherence, event sequencing, and camera motion. Human evaluations reveal that even visually strong models struggle with consistent temporal reasoning, highlighting gaps not captured by conventional metrics. Similarly, ChronoMagic-Bench, proposed by Yuan et al. [92], focuses on metamorphic time-lapse generation by introducing metrics to quantify meaningful temporal transformation and coherence. While these benchmarks provide targeted insight into temporal reasoning, their domain specificity limits generalization to broader video generation tasks.

Shifting focus to motion dynamics, Liao et al. [93] introduced DEVIL, an evaluation protocol specifically designed to evaluate video dynamics in T2V generation. DEVIL includes a benchmark with text prompts covering varying levels of dynamic content—from static scenes to highly dynamic actions—paired with metrics that analyze motion at different temporal granularities. This enables fine-grained evaluation of how motion unfolds over time. However, its focus on dynamics means it does not fully account for semantic correctness or visual fidelity.

With the emergence of multi-prompt generation, Cai et al. [68] introduced MPVBench to evaluate prompt transitions and temporal editing. MPVBench features 130 long-form prompts with 10 diverse transition types crafted using GPT-4. To overcome limitations of standard metrics like CLIP similarity—which often overlook abrupt semantic shifts at prompt boundaries—MPVBench proposes the CLIP Similarity Coefficient of Variation (CSCV) metric to penalize abrupt semantic changes that indicate poor transition smoothness. Nonetheless, its dependence on CLIP-based similarity inherits the known limitations of vision—language embeddings in capturing fine-grained temporal semantics.

Another line of evaluation targets compositional reasoning and hallucinations. For instance, Sun et al. [94] introduced T2V-CompBench that is designed to evaluate the compositional reasoning capabilities of T2V generative models. The benchmark covers seven core compositional categories, including attribute binding, spatial relations, motion and action binding, object interactions, and generative numeracy. Its hybrid evaluation framework combines detection, tracking, and multimodal large language model (MLLM)-based reasoning, which provides a detailed analysis of compositional failures. Addressing hallucinations, Rawte et al. [95] introduced ViBe, a large-scale benchmark that categorizes hallucination types in generated videos. These benchmarks provide valuable insight into failure modes, but they also reveal that many errors are deeply rooted in model architecture rather than easily addressable through post-hoc evaluation.

Focusing on numerical precision, Guo et al. [96] introduced T2VCountBench, a specialized human evaluation benchmark designed to evaluate the ability of ten state-of-the-art T2V generative models, available up to 2025, to correctly follow numerical instructions involving object counting. Despite advances in producing realistic, high-quality videos, as demonstrated by the authors, the study reveals a critical limitation: current models struggle to generate videos with precise object counts, especially for numbers up to nine. Factors such as video style, temporal dynamics, and multilingual prompts have minimal impact on performance.

Addressing physical plausibility, Guo et al. [97] introduced T2VPhysBench, which evaluates adherence to fundamental physical laws across multiple domains. The results reveal systemic shortcomings: no model consistently respects all tested physical laws, highlighting critical gaps in temporal and physical coherence and underscoring the need for physics-aware video generation techniques.

While prior evaluations focused on video quality, concerns over unsafe, illegal, or unethical generated content have escalated. Miao et al. [98] introduced T2VSafetyBench to evaluate the safety risks of T2V generative models, covering illegal content, violence, hate speech, misinformation, and other harms. The benchmark includes a diverse malicious prompt dataset drawn from real-world unsafe prompts, LLM-generated content, and jailbreak attack prompts designed to bypass filters, testing models under realistic adversarial conditions. By combining adversarial prompts with both human and automated evaluation, the benchmark exposes persistent safety weaknesses across nine state-of-the-art T2V models, which highlights an ongoing trade-off between safety and usability.

6. Discussion

To shape and consolidate knowledge in the emerging field of T2V generation, this study synthesizes fragmented research efforts and proposed T2V models. This section aims to answer the following research questions:

• RQ1: What are the key technological advances that have driven progress in the aforementioned video research fields?
• RQ2: What are the current challenges and best practices in evaluating T2V models, and how do benchmark datasets support the development of robust T2V generators?
• RQ3: What are the primary technical challenges and future research directions in leveraging Generative AI and LLMs for T2V generation?

RQ1: What are the key technological advances that have driven progress in the aforementioned video research fields?

In recent years, several foundational breakthroughs have underpinned the rapid progress in T2V generation. Among these, diffusion models have become a cornerstone since they have demonstrated unprecedented capabilities in generating high-fidelity, temporally coherent videos from text prompts. An important milestone was the release of Sora, OpenAI’s DiT model that can generate a two-minute video with remarkable realism, marking a leap forward in video synthesis. These diffusion transformers (DiTs) exemplify how transformer-based denoising architectures can maintain temporal consistency across frames, outperforming earlier approaches. In parallel with Sora, several large-scale proprietary systems, such as Google Veo [99] and Runway Gen-3 [100], have further illustrated the maturation of DiT-based video generation at scale. Although detailed architectural and training information for these systems is not publicly disclosed, their reported capabilities in long-form generation, motion coherence, and cinematic quality reinforce the broader trends observed in the research literature.

At the same time, alternative generative paradigms like autoregressive Video Pixel Networks and VAE-based video generators have provided important groundwork; however, their impact has been overshadowed by the superior quality and flexibility of diffusion models. Modern T2V systems often employ hybrid architectures that leverage strengths from multiple paradigms; for example, latent video diffusion models combine a VAE compression of video frames with transformer-guided diffusion, exploiting spatial detail and temporal smoothness. This convergence on scalable generative architectures, particularly diffusion models enhanced by transformer backbones, has enabled the generation of videos that are both more realistic and of longer duration than was previously achievable.

Another key driver is the explosion of large-scale video–text data and robust pretraining regimes. Models now leverage massive datasets (e.g., WebVid-10M, LAION-5B, and other web-scale video-caption collections) to learn rich cross-modal representations. By pretraining on millions of diverse video–text pairs and then fine-tuning for T2V tasks, generative models acquire a broad “world knowledge” of visuals and language. This transfer learning paradigm is bolstered by open, high-quality datasets specifically curated for video generation research. For example, the OpenVid-1M dataset introduced 1 million high-resolution video clips with descriptive captions to endorse T2V training, and the VidProM dataset collected 1.67 million real user prompts with corresponding model-generated videos to reflect authentic use cases. The availability of such data has been pivotal as researchers increasingly recognize that progress in T2V “hinges on open datasets combining varied, real-world video content with detailed textual descriptions”. In parallel, some T2V models have leveraged multimodal learning by integrating visual encoders with LLMs to improve the semantic understanding of video content. The pretrained LLMs (e.g., GPT or LLaMA) can be employed to guide video generation or interpret prompts with nuanced context, which improves text–video alignment. For example, vision-language transformers that use both visual features and LLM-driven text embeddings enable narrative coherence and object relevance to be maintained in videos. Overall, the synergy of generative architectures (especially diffusion transformers), training on large-scale video–text corpora, and incorporation of LLM-based semantic reasoning has dramatically accelerated progress in T2V generation, laying a strong foundation for future state-of-the-art models.

RQ2: What are the current challenges and best practices in evaluating T2V models, and how do benchmark datasets support the development of robust T2V generators?

Despite rapid technical progress, evaluating T2V models remains a complex challenge. Traditional evaluation metrics inherited from image generation (e.g., FID for visual quality or CLIP-based similarity for relevance) often fail to capture the full temporal and semantic nuances of generated videos. A persistent issue is that many models can generate photorealistic individual frames but still struggle with consistency across time, yet simple metrics may not penalize subtle temporal glitches or logical breaks in a video. Early evaluation efforts tended to examine multiple aspects of quality, such as per-frame fidelity or text alignment. As a result, best practices in T2V evaluation have shifted toward multi-dimensional and human-centered approaches. For example, the benchmark VBench decomposes video quality into 16 fine-grained dimensions (e.g., motion smoothness, temporal consistency, object permanence, absence of flicker, spatial relationships), providing a more diagnostic assessment of generative performance beyond a single aggregate score. In general, combining automated metrics with human judgment is now considered essential: benchmarks often report human evaluation of text–video alignment and overall realism to validate whether numeric scores (e.g., from CLIP Score or FID) truly reflect perceptual quality. For instance, the T2V-CompBench suite integrates “multimodal LLM-based metrics, detection-based assessments, and motion tracking techniques” and validates them against human judgments to ensure reliability. Aligning metric outcomes with human perception is crucial, as purely algorithmic scores can be misleading in capturing narrative coherence or subtle errors.

Another best practice is the development of specialized evaluation benchmarks targeting known failure modes of T2V models. Researchers have started targeting capabilities like temporal reasoning, compositional understanding, and realism under physical constraints. For example, T2VBench focuses on temporal dynamics, providing 1600+ annotated videos to test if models maintain coherent object motion, event sequencing, and scene transitions over time. This addresses the gap that while many models produce sharp frames, they often fail on long-term coherence and plausible scene progressions. Likewise, T2V-CompBench was introduced as the first benchmark for compositional reasoning in generated videos, covering attributes binding, spatial relations, object interactions, and even basic counting (numeracy) in complex prompts, enabling fine-grained evaluation of whether a model can correctly combine multiple concepts (e.g., “red cube on a blue sphere while a cat jumps”) in one video. Results on this benchmark have revealed that current models often struggle with attributing the right properties to the right entities or maintaining multiple object relationships over time, reflecting weaker compositional capabilities compared to image generation. In response, the benchmark pairs automated checks (object detectors, spatial consistency metrics) with LLM-based semantic evaluations, striving for a holistic measure of compositional faithfulness.

Moreover, emerging benchmarks tackle even more specific challenges: hallucinations in videos are addressed by the ViBe dataset, which defines five types of video hallucinations and evaluates how frequently models introduce objects or actions that were never mentioned or plausible. Early analyses show that even state-of-the-art models can noticeably hallucinate—for instance, by adding extraneous elements or morphing object identities—a shortcoming that ViBe helps to quantify. Counting ability is probed by a dedicated evaluation where humans judge whether models accurately represent numeric quantities in the video (e.g., “five apples”). Guo et al. [96] found that across ten contemporary T2V models, counting beyond very small numbers remains unreliable, prompting the creation of a T2VCountBench with human evaluation to drive improvements. Physical realism is another critical evaluation dimension: a recent benchmark tested various generative models against fundamental physical laws (e.g., gravity, object permanence, reflection) and found “no model consistently respects all tested physical laws”, with average compliance scores under 60%. This underscores that physics awareness in video generation is nascent, motivating new metrics for physical consistency and efforts to embed physical knowledge into models. Finally, content safety has become an evaluation focus amid growing concerns that models may generate harmful or inappropriate videos. Miao et al. [98] introduced T2VSafetyBench, which evaluates T2V models on 14 safety aspects (violence, hate, illicit behavior, etc.) using a suite of adversarial prompts and human/GPT-4 assessments. Initial results show no single model excels in all safety dimensions, and that there are trade-offs between minimizing unsafe outputs and retaining creative freedom. The proliferation of these benchmarks and protocols illustrates the community’s best practice: use a wide spectrum of evaluation datasets to stress-test models across multiple dimensions of quality and reliability. By benchmarking on diverse tasks, from temporal coherence to ethical safety, researchers can obtain a more robust picture of a model’s strengths and weaknesses and drive the development of more “robust T2V generators”, as called for in RQ2.

RQ3: What are the primary technical challenges and future research directions in leveraging Generative AI and LLMs for T2V generation?

Despite the notable progress in the T2V field, significant technical challenges still need further research. One major challenge is scaling to longer-duration videos without compromising temporal coherence or incurring prohibitive computation costs. Generating videos beyond a few seconds poses challenges in maintaining consistent narratives and preventing drift in visual details or motion. Current T2V models typically produce only short clips (often 2–5 s) at modest resolutions. Moving to minute-long or even narrative-length videos will require new strategies to handle long-range temporal dependencies. Promising directions include architectural innovations like causal or autoregressive video generation that generate frames sequentially rather than with full sequence attention (e.g, CausVid). This avoids needing to attend to an entire video at once and, combined with distillation techniques to reduce diffusion sampling steps, such advances illustrate a path forward for longer video generation; in this context, hierarchical or chunked generation (splitting a long video into manageable segments) and efficient sampling will be key research areas. Additionally, even with better architectures, training models for long outputs is resource-intensive—pointing to a need for optimized training paradigms, perhaps using curriculum learning (start with short sequences, gradually increase length) or leveraging video continuation tasks where a model extends a given clip.

A second major challenge is aligning T2V generation with human preferences, intent, and safety requirements. As models become more powerful, simply optimizing for pixel quality is not enough—they must also produce videos that are useful and trustworthy to users. This has led to the incorporation of human feedback loops and preference modeling into the generation process. For instance, “addressing the critical challenge of aligning T2V models with diverse human preferences” has spurred techniques like Direct Preference Optimization for video, exemplified by the VideoDPO framework that combines multiple criteria (visual clarity, temporal stability, text relevance) to rank outputs, improving both the fidelity and text alignment of videos as judged by users. Such approaches, akin to reinforcement learning from human feedback in text domains, are a promising direction to ensure generative videos meet qualitative expectations. Going forward, we anticipate more work on multi-criteria reward models (for example, jointly evaluating realism, relevance, creativity, and safety) and on efficient collection of preference data specific to video. Another aspect of alignment is content safety: as noted, no model currently excels at avoiding all forms of unsafe content. Future research must integrate safety constraints into generation, possibly by modeling forbidden content within the training objective or using real-time detectors to steer generation away from problematic scenes. Likewise, bias and fairness in T2V outputs (e.g., how different demographics are portrayed) will need examination as the technology matures, building on the initial safety benchmarks.

A third research direction involves enhancing the world knowledge, reasoning, and physical realism of generative video models. Today’s models often generate superficially plausible videos that nevertheless break logical rules or physical laws upon close inspection (e.g., objects appearing/disappearing, unnatural motion, physics violations). This limitation stems from models learning correlations in training data but lacking an understanding of underlying principles. Future T2V generators will benefit from being “physics-aware and reasoning-grounded”, meaning they have mechanisms to enforce basic consistency and continuity in the worlds they simulate. One promising direction is to explicitly inject knowledge of physics or common sense into the generative process like the WISA framework. Another approach is neuro-symbolic integration, where symbolic logic or external reasoning modules work alongside neural generators. In general, coupling LLMs with video generation is a tantalizing research avenue: LLMs could help maintain narrative coherence (by generating high-level storyboards or shot lists from a script), ensure semantic alignment (verifying that each scene logically follows the description), or augment evaluation (as GPT-4V is already used to assess video captions and safety). Early efforts like ShareGPT4Video have used GPT-4 with vision to generate detailed video annotations at scale, effectively leveraging an LLM to improve video datasets. Future systems might extend this idea, using multimodal LLMs as intelligent directors or critics in the video generation loop.

Finally, researchers are exploring hybrid and factorized generation strategies to overcome current limitations of end-to-end models. Rather than training a single colossal model to do everything, one direction is to split the problem into sub-tasks or stages that can be optimized independently. For example, to exploit abundantly available image data and unlabeled videos, one can decouple what to render from how it moves: a content branch generates high-quality key frames (leveraging image-text data), while a motion branch, trained on unlabeled video, ensures those frames evolve smoothly over time. Similarly, factorized diffusion models like Show-1 break the generation into an image generation step followed by a video extension step, reusing powerful image diffusion models for efficiency. Such modular designs make it easier to scale certain components (e.g., using a very large text-to-image model for detail, and a specialized temporal model for motion). The trade-off, however, is managing consistency between components; hence, research is needed on optimal interfaces between image and video generation modules (e.g., how to condition a video model on key frames or latent codes without drift). Model scalability is another concern: pushing parameter counts and model capacity is a clear trend (with recent T2V foundation models reaching tens of billions of parameters), but training such models is expensive. Efficient fine-tuning methods (e.g., parameter-efficient adapters or layer freezing) will be vital to make large T2V models accessible for wider use. There is also growing interest in multi-lingual and multimodal video generation. So far, most T2V benchmarks and models assume English text inputs, but recent work like Step-Video-T2V introduced dual text encoders to support bilingual generation in English and Chinese. This points toward a future where T2V systems handle many languages and cultural contexts. Along the same lines, adding other modalities such as audio is largely uncharted territory since current generators typically produce silent videos, a notable limitation for real-world applications. Future research should integrate text-to-speech or sound effect generation synchronized with video frames, enabling fully multimodal storytelling (e.g., a video with dialog or ambient sound matching the scene). Addressing audio generation, alongside visual content, would enable T2V models to generate complete immersive experiences rather than just silent movies.

7. Conclusions

This study presented a systematic literature review of scientific publications exploring text-to-video (T2V) generation, with a specific emphasis on works published from 2024 onward that address methods, datasets, and evaluation practices for generating temporally coherent and semantically consistent video from natural language prompts. In total, 69 studies were gathered to consolidate and structure the fragmented knowledge in this emerging field. The main findings indicate that transformer architectures combined with diffusion models has lead to a remarkable progress in generating visual content from text prompts. Moreover, large language models (LLMs) are increasingly utilized for better alignment between text prompts and videos. Another observation pertains to the diversity in evaluation practices: some studies rely only on automated metrics, while others combine human judgments, introducing variability that complicates direct comparisons across methods.

Subsequently, this review contributed to the theory by clarifying the current research landscape on T2V and proposing directions for future investigation. Specifically, the main contributions of this study are as follows: (1) mapping the main approaches in T2V generation addressed by recent research, (2) outlining the existing evaluation landscape of T2V models, and (3) providing a structured roadmap for researchers and practitioners to support further research on T2V generation.

Neverthless, it is important to acknowledge certain limitations of this research. (1) The review focuses specifically on T2V generation and does not cover related modalities such as text-to-image and image-to-video; (2) only English-language journal papers and conference proceedings were considered; and (3) although the consulted databases are regularly updated, only papers retrieved at the consultation time were reviewed. Nevertheless, by following a rigorous methodology and addressing the main research questions, this review provides a consolidated foundation for future work in T2V generation.