Symmetry-Aware Advances in Multimodal Large Language Models: Architectures, Training, and Evaluation

Abstract

With the exponential growth of multimodal data, the limitations of traditional unimodal models in cross-modal understanding and complex scenario reasoning have become increasingly evident. Built upon the foundation of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) retain strong reasoning abilities and demonstrate unique capabilities in multimodal understanding. This survey provides a comprehensive overview of the current research landscape of MLLMs. It systematically analyzes mainstream model architectures, training, fine-tuning strategies, and task classifications, while offering a structured account of evaluation methodologies. Beyond synthesis, the paper highlights emerging trends that aim for balanced integration across modalities, tasks, and components, and critically examines key challenges together with potential solutions. The survey specifically emphasizes recent reasoning-oriented MLLMs, with a focus on DeepSeek-R1, analyzing their design paradigms and contributions from the perspective of symmetric reasoning capabilities. Overall, this work offers a comprehensive overview of cutting-edge advancements and lays a foundation for the future development of MLLMs, especially those guided by symmetry principles.

1. Introduction

In the era of artificial intelligence, MLLMs have become a central driver toward artificial general intelligence. The rapid growth of multimodal data exposes the inherent limitations of unimodal models in cross-modal understanding and reasoning. By extending LLMs, MLLMs offer innovative solutions to these challenges, underscoring the need for theoretical perspectives that reveal structural patterns and guide future model development. The concept of symmetry—broadly defined as structural or functional balance across modalities, components, or learning processes—provides such a framework. In the context of multimodal learning, symmetry manifests in several key aspects, including cross-modal alignment, encoder–decoder consistency, and unified architectures that process heterogeneous inputs in a modality-invariant manner. Representative models such as CLIP [1] and Flamingo [2] have achieved breakthrough progress in tasks like image–text retrieval and visual question answering by constructing joint cross-modal semantic spaces through pretraining.

Research in multimodal learning has mainly followed two technical pathways: the first, represented by discriminative models such as CLIP and ALIGN [3], leverages contrastive learning to build cross-modal joint embedding spaces; the second adopts generative architectures like OFA [4], unifying multimodal tasks within a sequence-to-sequence framework. Both paradigms reflect an implicit pursuit of structural or functional symmetry. The technical evolution of MLLMs exhibits three key trends: first, model architectures have advanced from early dual-tower contrastive learning approaches (e.g., CLIP) to hybrid expert systems based on Transformer [5] architectures (e.g., VLMO [6]); second, novel training paradigms such as multimodal instruction tuning and reinforcement learning have significantly enhanced the models’ capacity to comprehend and execute new instructions; third, application scenarios have expanded from basic vision-language tasks to specialized domains.

Recently, MLLMs have made substantial breakthroughs in complex reasoning and multi-task processing. Many of these breakthroughs are closely tied to improved cross-modal alignment and deeper bidirectional reasoning, both of which resonate with symmetry-aware design principles. The DeepSeek-R1 model [7] represents a cutting-edge advancement in this field. Based on the DeepSeek-V3 [8] architecture, DeepSeek-R1 introduces systematic optimization under a staged reinforcement learning framework, significantly enhancing performance in complex reasoning tasks and outperforming peer models in comprehensive evaluations. Notably, the Grok-3 series of models has further pushed the boundaries of reasoning-focused MLLMs. Similarly, the year 2025 has witnessed remarkable advances in MLLMs, exemplified by VideoLLaMA 3 [9] for image and video understanding, VITA-1.5 [10] for real-time vision–speech interaction, each reflecting cutting-edge progress in modality integration and task-specific capabilities.

Nevertheless, several critical challenges continue to hinder the development of multimodal intelligence. Despite remarkable technical progress, current research faces multiple core contradictions. First, the heterogeneity of modalities makes it difficult to balance representation alignment efficiency and the preservation of fine-grained semantics—studies such as ViLT [11] have demonstrated that simple feature-level fusion is inadequate for supporting complex reasoning tasks. Second, the ever-increasing scale of models creates tension with limited computational resources, prompting the emergence of lightweight solutions such as MobileLLaMA [12]. Moreover, issues like answer quality bottlenecks caused by noisy data, multimodal hallucination, and representational bias call for systematic governance mechanisms. Addressing these challenges may benefit from explicitly considering symmetry and asymmetry across modalities, data distributions, and task formulations.

Given the rapid development and promising prospects of this field, this survey aims to provide a systematic review of the current state of research on MLLMs, offering both theoretical guidance and practical references for researchers in related domains. This survey is primarily based on the literature retrieved from Google Scholar, IEEE Xplore, ACM Digital Library, and arXiv, with emphasis on representative studies from the past five years published in leading international venues and directly related to MLLM architectures, training methods, and evaluation. Works lacking empirical evidence, showing weak relevance to multimodality, or being outdated were excluded to ensure systematic coverage and reproducibility. Additional information is provided in Appendix A.

The structure of this survey is as follows: Section 2 provides a comprehensive review of key elements in MLLMs, including mainstream model architectures and a comparative analysis of representative models, with LLaVA [13] used as a case study. Section 3 discusses complete training and fine-tuning strategies, focusing on methods employed in classical models such as CLIP and reasoning-oriented MLLMs like DeepSeek-R1. Section 4 examines the categorization of tasks and systematic evaluation methods for MLLMs. Section 5 analyzes major challenges in current research and application, identifying bottlenecks and proposing corresponding solutions. Section 6 explores future research directions from various perspectives. Finally, Section 7 summarizes the development of MLLMs and presents future outlooks. Throughout this review, we emphasize how the concept of symmetry provides a unifying perspective for understanding and advancing MLLM research.

2. Mainstream Model Architectures

Typical MLLMs can be abstracted into three core components: a pretrained modality encoder, a pretrained LLM, and a modality connector that bridges the two (refer to Figure 1). These components play critical and interrelated roles. The modality encoder is responsible for converting raw multimodal data, such as visual or auditory signals, into feature representations that can be processed by downstream models. The LLM serves as the central intelligence engine, leveraging its powerful language understanding and generation capabilities to perform complex semantic reasoning. Acting as the bridge, the modality connector ensures effective alignment between encoded features and the LLM’s input space, thereby enabling the model to comprehend and integrate multimodal information. The synergy among these three components propels the development and application of multimodal technologies. Notably, many mainstream MLLM architectures intrinsically incorporate symmetry-oriented design principles. These include encoder–decoder structural balance, which maintains bidirectional information flow and architectural parity; modality-parallel processing pathways, which enable simultaneous and structurally analogous treatment of different input modalities; and the use of shared latent representation spaces, which enforce alignment and transformation invariance across modalities. Such symmetry-aware features facilitate more unified, interpretable, and scalable multimodal systems by reducing modality-specific biases and promoting consistent cross-modal reasoning.

2.1. Modality Encoders

As the perceptual front-end of MLLMs, modality encoders convert raw multimodal inputs (e.g., images, audio, video) into low-dimensional semantic features. The design of modality encoders must strike a balance between computational efficiency and cross-modal alignment to ensure high-quality understanding by downstream LLMs. To enhance both performance and efficiency, many existing works adopt pretrained encoders that have achieved semantic alignment across modalities in other tasks. Vision encoders are a major research focus. For example, the CLIP series of models employs a ViT-based architecture trained with image–text contrastive learning on large-scale datasets to achieve cross-modal alignment. In contrast, Osprey [14] adopts ConvNext-L [15], a convolutional-based encoder, to exploit higher resolutions and multi-level feature representations. The choice of encoder often depends on factors such as input resolution, parameter size, and pretraining data. Empirical studies have shown that input resolution tends to have a greater impact on model performance than the composition of training data. Current methods for improving input resolution fall into two main categories:

• Direct Scaling: feeding higher-resolution images directly into the encoder.
• Image Partitioning: Splitting high-resolution images into smaller patches, which are then processed individually by a lower-resolution encoder. For example, Monkey [16] divides large images into multiple patches and combines them with a downsampled global image, allowing the patches to capture local features while the low-resolution image retains global context.

For other modalities, representative encoders have also been developed. For instance, Pengi [17] uses CLAP [18] as an audio encoder. ImageBind [19] serves as a unified encoder for multiple modalities, including images, text, audio, depth, thermal imagery, and inertial measurement unit data.

2.2. Pretrained LLMs

Compared to training from scratch, using pretrained LLMs is more efficient and practical. These models are trained on massive web-scale corpora and have embedded extensive world knowledge, exhibiting strong generalization and reasoning abilities. Most LLMs follow the autoregressive decoder architecture, derived from the GPT-3 [20] framework. Among the early widely adopted models is the FlanT5 [21] series, which has been applied in works such as BLIP-2 [22]. Currently, the LLaMA [23] series stands as a representative open-source LLM, widely used in academia.

Most LLMs are built on the Transformer architecture, though recent advancements include the Mamba [24] architecture, which introduces a time-dependent state-space model. This model has linear scaling in both inference cost and memory usage with respect to sequence length. Furthermore, thanks to efficient GPU implementations, Mamba offers faster training while maintaining high performance. Similar to Transformers, Mamba is pretrained via next-token prediction for language modeling.

2.3. Modality Connectors

Since LLMs are inherently designed to process only textual data, a bridging mechanism is required to connect them with other modalities. Training a MLLM from scratch is costly; thus, a more practical approach is to introduce trainable connectors that integrate pretrained vision encoders with LLMs. Another feasible strategy is to employ expert models to convert visual inputs into textual descriptions, which are then fed into LLMs.

Various learnable connectors have been employed in MLLMs, ranging from simple linear layers and multi-layer perceptrons (MLPs) to advanced Transformer-based solutions such as Q-Former, or conditional cross-attention layers integrated directly into the LLM.

2.3.1. Learnable Connectors

A direct approach to projecting visual features into the text embedding space is via linear MLP projections. LLaMA-Adapter [25] uses a single linear layer for multimodal alignment, while LLaVA-1.5 [26] adopts a two-layer MLP, demonstrating enhanced multimodal capabilities. Despite its simplicity, linear projection remains effective in both early and advanced MLLM frameworks [27,28,29], making it a widely used and efficient technique for aligning visual features with text embeddings.

Q-Former, first introduced in BLIP-2, is a Transformer-based architecture that has since been adopted in several methods [30,31]. It features a dual-layer self-attention module, enabling alignment between visual and textual representations. Q-Former interacts with visual features through learnable queries and cross-attention mechanisms. Models like mPLUG-Owl2 [32] simplify the Q-Former structure and propose a visual abstraction component that compresses visual information into learnable tokens, enhancing the semantic richness of visual representations. Similarly, Qwen2-VL [33] compresses visual features using learnable queries and integrates 2D positional encodings.

The cross-attention layer approach, first proposed in Flamingo, incorporates dense cross-attention blocks within pretrained LLM layers. While this adds a significant number of trainable parameters and increases computational cost, it improves training stability and performance. To reduce overhead, this strategy is often combined with Perceiver-based [34] modules that compress visual tokens before feeding them into the LLM. This method has been adopted in multiple recent models [35,36].

2.3.2. Expert Models

Expert models serve as intermediaries that convert multimodal inputs directly into natural language descriptions, allowing LLMs to understand multimodal content without requiring additional retraining. This strategy greatly enhances model flexibility and applicability across modalities. For example, image captioning models can transform visual elements such as objects, scenes, and actions into descriptive text. Similarly, models like VideoChat—Text [37] extract motion features from videos using pretrained vision models and combine them with speech recognition outputs, converting both visual and auditory content into detailed textual descriptions. This enables LLMs to comprehend video content effectively.

However, despite their convenience, expert models are less flexible than learnable connectors. The conversion process inevitably results in information loss. For instance, in the case of video-to-text conversion, the complex spatiotemporal relationships and dynamic variations present in video content may be distorted or omitted, leading to incomplete semantic representations. Therefore, practical applications must carefully balance the convenience of expert models against the need for semantic fidelity.

2.4. Representative MLLMs

The “representative MLLMs” discussed in this survey include MLLMs with broad impact in both academia and industry. They are considered representative because they achieved state-of-the-art results on major benchmarks and introduced methodological innovations that have substantially influenced subsequent research.

In this section, we take LLaVA as a representative example to illustrate the typical architecture of a MLLM:

• Encoding: LLaVA employs the pretrained Vicuna model as the language backbone ${\mathrm{f}}_{\mathsf{\varphi }}\left(.\right)$ and uses the ViT-L/14 encoder from CLIP as the vision encoder.
• Modality Alignment: For the image ${\mathrm{X}}_{\mathrm{v}}$, a visual encoder $\mathrm{g}\left({\mathrm{X}}_{\mathrm{v}}\right)$ is first employed to encode the input image into the visual feature representation ${\mathrm{Z}}_{\mathrm{v}}$, which is subsequently transformed via a linear projection layer $\mathrm{W}$ into ${\mathrm{H}}_{\mathrm{v}}$. For the language instruction ${\mathrm{X}}_{\mathrm{q}}$, it is converted into a textual feature representation ${\mathrm{H}}_{\mathrm{q}}$ through a tokenizer, which can be written as the following equation:

  $$\begin{gathered} {\mathrm{Z}}_{\mathrm{v}}=\mathrm{g}\left({\mathrm{X}}_{\mathrm{v}}\right), \\ {\mathrm{H}}_{\mathrm{v}}=\mathrm{W}·{\mathrm{Z}}_{\mathrm{v}} \end{gathered}$$

  (1)
• Connection: The visual feature ${\mathrm{H}}_{\mathrm{v}}$ and textual representation ${\mathrm{H}}_{\mathrm{q}}$ are concatenated and fed into the language model ${\mathrm{f}}_{\mathsf{\varphi }}\left(.\right)$ to produce the final output ${\mathrm{X}}_{\mathrm{a}}$. This process can be formally expressed as

  $$\begin{gathered} {\mathrm{X}}_{\mathrm{a}}=\mathrm{f}\left(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(\left[{\mathrm{H}}_{\mathrm{v}},{\mathrm{H}}_{\mathrm{q}}\right]\right)\right) \\ =\mathrm{f}\left(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(\left[\mathrm{W}\cdot {\mathrm{Z}}_{\mathrm{v}}{\mathrm{H}}_{\mathrm{q}}\right]\right)\right) \\ =\mathrm{f}\left(\mathrm{concat}\left(\left[\mathrm{W}\cdot \mathrm{g}\left({\mathrm{X}}_{\mathrm{v}}\right),{\mathrm{H}}_{\mathrm{q}}\right]\right)\right) \end{gathered}$$

  (2)

Moreover, the field of MLLMs has been developing rapidly, with numerous innovative architectures emerging (refer to Figure 2).

2.5. Classification of MLLM Architectures

Based on whether the visual and language modalities possess equivalent representational capacity in terms of dedicated parameters and computational resources, and whether these modalities interact within deep neural networks, four prototypical architectures of MLLMs can be identified (refer to Figure 3).

Visual–semantic embedding models such as SCAN [38] fall into the first category in Figure 3. These models employ separate encoders for images and texts, with the image encoder typically being more complex. The similarity between the two modalities is then computed using simple dot products or shallow attention layers.

CLIP represents the second category, as it uses independent but equally complex Transformer-based encoders for each modality. However, the interaction between image and text features remains shallow (e.g., via dot product). Although CLIP demonstrates remarkable zero-shot performance in image–text retrieval, its effectiveness on other downstream tasks remains limited. For instance, on NLVR2 [39], fine-tuning with dot-product multimodal representations yields only 50.99 ± 0.38 accuracy, close to chance level, indicating insufficient capacity for complex reasoning tasks.

In contrast, some vision-language models employ deep Transformer layers to model cross-modal interactions between image and text features. Certain modulation-based vision-language models [40] belong to the third category in Figure 3. ViLT [11] is the first model representative of the fourth category, where the image embedding layer is as shallow and computationally lightweight as textual token embeddings. Conventional VLP models (e.g., UNITER [41], Pixel-BERT [42]) rely on convolutional backbones, leading to high latency (∼900 ms for UNITER-Base, ∼60 ms for Pixel-BERT-R50). In contrast, ViLT removes CNNs by employing lightweight linear embeddings and a Transformer, reducing inference time to ∼15 ms while achieving competitive performance (76.1 on NLVR2, 83.5 R@1 on Flickr30K). This demonstrates that MLLMs can improve efficiency without sacrificing effectiveness by minimizing modality-specific encoders and emphasizing cross-modal interaction.

After presenting the taxonomy of MLLMs, we further compare representative models in terms of their architectural designs and task specializations.

Table 1. Comparison of MLLMs in architecture and capabilities.

Model	Visual Encoder	Adapter	LLM Backbone	I→O	Main Tasks
BLIP-2 [22]	EVA ViT-g	Q-Former	Flan-T5/OPT	I + T→T	VQA, Captioning, Retrieval
Flamingo [2]	NFNet-F6	LLM MH-Attn	Chinchilla-1.4B/7B/70B	I + V + T→T	VQA, Captioning
LLaVA [35]	CLIP ViT-L	Linear	Vicuna-7B/13B	I + T→T	VQA, Captioning
mPLUG-Owl [43]	CLIP ViT-L	Q-Former	LLaMA-7B	I + T→T	Visual Dialogue, VQA
MiniGPT-4 [44]	EVA ViT-g	Linear	Vicuna-13B	I + T→T	VQA, Captioning
InstructBLIP [45]	EVA ViT-g	Q-Former	Flan-T5/Vicuna	I + V + T→T	VQA, Captioning
Otter [46]	CLIP ViT-L	LLM MH-Attn	LLaMA-7B	I + T→T	VQA, Captioning
Shikra [47]	CLIP ViT-L	Linear	Vicuna-7B/13B	I + T→T + I	VQA, Captioning, Referring
CogVLM [27]	EVA ViT-E	MLP	Vicuna-v1.5-7B	I + T→T	VQA, Captioning, REC
VILA [29]	CLIP ViT-L	Linear	LLaMA-2-7B/13B	I + T→T	VQA, Captioning
LLaVA-1.5 [26]	CLIP ViT-L	MLP	Vicuna-v1.5-7B/13B	I + T→T	VQA, Captioning
MiniGPT-v2 [48]	EVA ViT-g	Linear	LLaMA-2-Chat-7B	I + T→T	VQA, Captioning, Referring
PaLI-X [49]	ViT-22B	LLM MH-Attn	UL2-32B	I + T→T	Multilingual, VQA
ChatSpot [50]	CLIP ViT-L	Linear	Vicuna-7B/LLaMA	I + T→T	VQA, Captioning, Referring
MiniGPT-5 [51]	EVA ViT-g	Q-Former	Vicuna-7B	I + T→I + T	Image Generation
LaVIT [52]	EVA ViT-g	LLM MH-Attn	LLaMA-7B	I + T→I + T	Captioning, Image Generation
GILL [53]	CLIP ViT-L	Linear	OPT-6.7B	I + T→I + T	Retrieval, Image Generation

summarizes the key components of different MLLMs.

3. Training and Fine-Tuning Strategies

3.1. Pretraining

Pretraining serves as the initial stage in the training of MLLMs. Its primary objective is to align different modalities and enable the model to acquire multimodal world knowledge. A common pretraining strategy involves freezing the visual encoder and the LLM, while training only the learnable interface modules [54]. This approach aims to align modalities while preserving the knowledge encoded in the pretrained components. Some methods [55], however, unfreeze more components to introduce additional trainable parameters, thereby enhancing modality alignment. Many pretraining strategies implicitly reflect symmetry-related principles, such as encouraging modality-invariant representations, aligning visual and textual semantics, or balancing the contribution of each modality through joint learning objectives. Specifically, pretraining a MLLM can be carried out in either a single-stage or multi-stage manner.

3.1.1. Single-Stage Pretraining

LLaMA-Adapter introduces additional trainable parameters to encapsulate visual knowledge while managing instruction tuning for textual inputs. The model is jointly trained on image–text pairs and instructions across different parameter subsets. Koh [56] proposes a model that adjusts the overall loss by incorporating two contrastive loss functions for image–text retrieval; during training, only three linear layers are updated. Flamingo and its open-source variants [36] train cross-attention layers and Perceiver-based components to connect visual features to the frozen backbone of the LLM.

3.1.2. Multi-Stage Pretraining

In the first stage of multi-stage training, the objective is typically to align image features with the text embedding space. The output after this stage is often fragmented. The second stage then focuses on enhancing the model’s multimodal dialogue capabilities. LLaVA is among the first models to adopt a visual instruction tuning paradigm. In its first stage, only the multimodal adapter is trainable, while the second stage fine-tunes both the adapter and the language model parameters. MiniGPT-4 [44] trains the linear alignment layers in both stages. After initial training, it uses the model itself to curate and refine high-quality training data for the second stage. Unlike previous approaches, mPLUG-Owl2 updates the visual backbone during the initial stage to capture both low-level and high-level visual features. In the second stage, it jointly trains on both unimodal (text-only) and multimodal data to enhance alignment. VLMo adopts a phased pretraining strategy: it first trains visual experts and self-attention modules using a BEiT-style [57] masked image modeling method on a large image dataset to learn visual representations; then freezes these modules and trains text experts on a large-scale text corpus using masked language modeling to capture deep semantic features. Finally, it conducts joint training on image–text pairs to achieve cross-modal learning and improve generalization and representation capacity. VideoLLaMA 3 [9] also serves as a representative multi-stage pretrained model, progressively building multimodal understanding through staged visual adaptation, modality alignment, and task fusion, highlighting the effectiveness of multi-stage pretraining for complex modality integration. The choice of training strategy is closely tied to data quality. For short and noisy subtitle data, lower image resolution can be used to accelerate training; for long, high-quality data, higher resolution is recommended to mitigate hallucinations. ShareGPT4V [58] shows that unfreezing the visual encoder significantly improves alignment when high-quality subtitle data is used in pretraining.

3.2. Fine-Tuning

3.2.1. Instruction Tuning

Instruction tuning aims to enhance the model’s ability to understand and follow user instructions, enabling better generalization to unseen tasks and improving zero-shot performance. This approach has achieved notable success in natural language processing, exemplified by models such as FLAN [21]. A multimodal instruction sample typically consists of three components: an instruction, an input, and an output. The instruction is a natural language description of the task; the input can be an image–text pair [59] or a single modality [60]; and the output is the model-generated response. Instruction templates are highly flexible and can be manually designed to suit different tasks. They can also be extended to multi-turn dialogue settings. Formally, a multimodal instruction sample can be represented as a triplet (I, M, R), where I, M, and R denote the instruction, modality input, and ground-truth response, respectively. The model generates predicted responses based on the instruction and input. SVLA [61] applies end-to-end joint instruction tuning using multimodal instruction data to align speech, vision, and language modules, exemplifying the effectiveness of instruction tuning in multimodal integration.

Instruction data can be collected via data adaptation, self-instruction, and dataset mixing. Task-specific datasets are a key source of high-quality instruction data. For example, the VQA dataset consists of (image, question, answer) triplets, which can be naturally transformed into instruction samples. Notably, answers in existing datasets tend to be short, which may limit the model’s generative capabilities. Possible solutions include: explicitly specifying output length in the instruction and using GPT to extend the answer. To support real-world scenarios such as multi-turn dialogue, some studies employ self-instruction methods [62], where LLMs generate instruction-following data based on a small set of human-labeled examples. LLaVA extends this method to the multimodal domain by converting images into text descriptions and bounding boxes, prompting GPT-4 to generate instruction data, resulting in the LLaVA-Instruct-150k dataset. With the advent of GPT-4V, some studies now leverage it to generate higher-quality training data.

Studies show that the quality of instruction-tuning data significantly impacts model performance. Lynx [63] finds that models pretrained on smaller but higher-quality datasets outperform those trained on larger yet noisier datasets. Experimental results also suggest that diversified prompts improve model performance. Among task types, visual reasoning tasks offer more significant performance gains than captioning or Q&A. Moreover, increasing instruction complexity appears to be more effective than simply expanding task diversity or adding fine-grained spatial annotations.

3.2.2. Parameter-Efficient Fine-Tuning (PEFT)

When a pretrained LLM needs to be adapted to a specific domain or application, PEFT provides an effective alternative to full model tuning. Instead of updating the entire model, PEFT methods introduce a small number of additional trainable parameters. LaVi [64] introduces an internal visual feature modulation mechanism within the language model, enabling efficient multimodal integration and exemplifying the use of parameter-efficient fine-tuning in vision–language tasks.

Among these approaches, prompt tuning [65,66] learns a small set of vectors that are prepended to the input text as soft prompts. In contrast, Low-Rank Adaptation ( LoRA) [67] constrains the number of new parameters by learning low-rank matrices to approximate weight updates. This technique is orthogonal to quantization-based methods such as QLoRA [68], which further reduces the memory consumption of LLMs by representing weights with lower precision, offering greater memory savings compared to standard half-precision models.

3.2.3. Alignment Tuning

Alignment tuning is commonly applied when models need to align with human preferences in specific contexts. Currently, two main techniques dominate this area: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO).

RLHF leverages reinforcement learning algorithms to fine-tune LLMs based on human feedback. This approach incorporates human-annotated preferences as supervision signals to guide model behavior during training. The RLHF pipeline typically involves three key stages: supervised fine-tuning, reward modeling, and reinforcement learning.

Researchers have also extended RLHF to multimodal settings. For example, LLaVA-RLHF [69] collects preference pairs based on human feedback to fine-tune the model, resulting in reduced hallucinations and more aligned outputs. On the other hand, DPO simplifies the pipeline by directly optimizing a binary preference loss, thereby avoiding explicit reward modeling required in traditional methods.

3.2.4. Representative Model Case Studies

This section presents the complete training and fine-tuning pipelines of CLIP and the latest inference model, DeepSeek-R1, providing readers with a comprehensive understanding of training practices.

CLIP is a foundational MLLM that has inspired numerous variants. Its text encoder typically adopts a Transformer architecture, using self-attention mechanisms to capture dependencies between words and generate text embeddings. For visual encoding, CLIP employs the Vision Transformer (ViT) [70], which encodes images via self-attention and projects the resulting representation into the same embedding space as the text vector through a fully connected layer.

CLIP was pretrained on a massive dataset containing 400 million image–text pairs collected from the internet. This diverse dataset spans a wide range of domains and topics, allowing the model to learn rich multimodal representations. To enhance generalization, CLIP applies various image augmentations during training, such as random cropping, rotation, and scaling, enabling the model to handle image distortions and noise more robustly.

The core idea behind CLIP’s learning objective is contrastive learning—maximizing the similarity between matched pairs while minimizing the similarity between mismatched ones. Specifically, CLIP adopts the InfoNCE loss, formulated as follows:

$$\mathrm{L}=-{\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}\log {\displaystyle \frac{\exp \left(\cos \left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{I}},{\mathrm{z}}_{\mathrm{i}}^{\mathrm{T}}\right)/\mathsf{\tau }\right)}{{\sum }_{\mathrm{j}=1}^{\mathrm{N}}\exp \left(\cos \left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{I}},{\mathrm{z}}_{\mathrm{j}}^{\mathrm{T}}\right)/\mathsf{\tau }\right)}}$$

(3)

Here, $\mathrm{N}$ denotes the batch size, $z_i^I,z_i^T$ represent the embedding vectors of the image and its corresponding text, and $\cos \left(z_i^I,z_i^T\right)$ indicates the cosine similarity between two vectors. The parameter $\mathsf{\tau }$ is a temperature scalar used to scale the cosine similarity values. Contrastive pretraining methods promote bidirectional semantic consistency across modalities, which aligns with functional notions of symmetry in cross-modal representation learning. The CLIP model has demonstrated broad applicability across various domains, including image retrieval, text generation, image classification, and multimodal analysis tasks.

DeepSeek-R1 is a recently released inference-oriented MLLM, optimized from DeepSeek-V3 to enhance reasoning capabilities. This section details its innovative training and fine-tuning strategies. The DeepSeek team employed a multi-stage reinforcement learning pipeline, starting with the DeepSeek-R1-Zero phase, where reward modeling was introduced. The final DeepSeek-R1 stage involved a systematic multi-phase training procedure. The goal is to improve reasoning, problem-solving, and context-aware response generation.

The training procedure overview is followed by a detailed analysis of each stage. In the DeepSeek-R1-Zero phase, DeepSeek-V3 acts as the agent, generating answers and reasoning processes. Correct responses or logically sound reasoning chains receive positive rewards, guiding the model to update its policy and promote higher-reward generations.

Traditional reinforcement learning frameworks often incorporate a critic model to assess the quality of the agent’s output and provide feedback. Typically, the critic is of comparable scale and complexity to the policy model. However, DeepSeek takes a different approach with its Group-based Reinforcement Policy Optimization (GRPO) algorithm. Rather than employing a critic model of equal size, GRPO estimates the baseline directly from group-level scores. For each question $\mathrm{q}$, the GRPO algorithm samples a set of outputs $\left\{{\mathrm{o}}_1,{\mathrm{o}}_2,\dots ,{\mathrm{o}}_{\mathrm{G}}\right\}$ from the previous policy and then updates the current policy ${\mathsf{\pi }}_{\mathsf{\theta }}$ by maximizing the following objective:

$$\begin{array}{ll}{\mathcal{J}}_{\mathrm{G}\mathrm{R}\mathrm{P}\mathrm{O}}\left(\mathsf{\theta }\right)& =\mathrm{E}\left[\mathrm{q}\sim \mathrm{P}\left(\mathrm{Q}\right),{\left\{{\mathrm{o}}_{\mathrm{i}}\right\}}_{\mathrm{i}=1}^{\mathrm{G}}\sim {\mathsf{\pi }}_{{\mathsf{\theta }}_{\mathrm{o}\mathrm{l}\mathrm{d}}}(\mathrm{O}\mid \mathrm{q})\right]\\ & {\displaystyle \frac{1}{\mathrm{G}}}\sum _{\mathrm{i}=1}^{\mathrm{G}}\left(\min \left({\displaystyle \frac{{\mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}{{\mathsf{\pi }}_{{\mathsf{\theta }}_{\mathrm{o}\mathrm{l}\mathrm{d}}}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}}{\mathrm{A}}_{\mathrm{i}},\mathrm{clip}\left({\displaystyle \frac{{\mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}{{\mathsf{\pi }}_{{\mathsf{\theta }}_{\mathrm{old}}}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}},1-\mathsf{\epsilon },1+\mathsf{\epsilon }\right){\mathrm{A}}_{\mathrm{i}}\right)-\mathsf{\beta }{\mathrm{D}}_{\mathrm{K}\mathrm{L}}\left({\mathsf{\pi }}_{\mathsf{\theta }}\parallel {\mathsf{\pi }}_{\mathrm{r}\mathrm{e}\mathrm{f}}\right)\right),\\ & \end{array}$$

(4)

$${\mathrm{D}}_{\mathrm{K}\mathrm{L}}\left({\mathsf{\pi }}_{\mathsf{\theta }}\parallel {\mathsf{\pi }}_{\mathrm{r}\mathrm{e}\mathrm{f}}\right)={\displaystyle \frac{{\mathsf{\pi }}_{\mathrm{r}\mathrm{e}\mathrm{f}}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}{{\mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}}-\log {\displaystyle \frac{{\mathsf{\pi }}_{\mathrm{r}\mathrm{e}\mathrm{f}}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}{{\mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{o}}_{\mathrm{i}}\mid \mathrm{q}\right)}}-1,$$

(5)

In this formulation, ε and β are predefined hyperparameters, while ${\mathrm{A}}_{\mathrm{i}}$ represents the computed advantage. A group of rewards $\left\{{\mathrm{r}}_1,{\mathrm{r}}_2,\dots ,{\mathrm{r}}_{\mathrm{G}}\right\}$ is assigned to the corresponding outputs within each sample group:

$$A_i={\displaystyle \frac{r_i-\mathrm{mean}\left(\left\{r_1,r_2,\dots ,r_G\right\}\right)}{\mathrm{std}\left(\left\{r_1,r_2,\dots ,r_G\right\}\right)}}$$

(6)

The reward serves as the source of the training signal and determines the direction of optimization. To train DeepSeek-R1-Zero, the team employed a rule-based reward system, which primarily includes two types of rewards: accuracy reward and format reward.

DeepSeek-R1-Zero excels in reasoning but struggles with readability and language consistency. To enhance its performance, the DeepSeek team employed cold-start data collection and Supervised Fine-Tuning (SFT). They provided the DeepSeek-V3 Base model with chain-of-thought exemplars, enabling structured reasoning and reducing inconsistencies. Direct prompting was used to ensure the model explicitly detailed its reasoning process.

The team also refined outputs from DeepSeek-R1-Zero, creating a high-quality dataset for fine-tuning the DeepSeek-V3 Base model. Reinforcement learning, with an extended reward mechanism including language consistency, was reintroduced. Approximately 600,000 optimal reasoning samples were retained after rigorous filtering. An additional 200,000 non-reasoning samples were included to bolster general capabilities. The final stage involved reinforcement learning across all scenarios to align the model with human preferences, enhancing usefulness and harmlessness.

After multiple rounds of training, the model outperformed both open-source and proprietary counterparts on several popular benchmark datasets, and was officially named DeepSeek-R1 (refer to Figure 4).

Following the description of the training pipeline of DeepSeek-R1, we next provide a broader comparative perspective by examining the performance of representative MLLMs across key evaluation dimensions. As illustrated in Figure 5, the radar chart highlights the relative strengths and weaknesses of these models in tasks such as mathematics, knowledge-intensive reasoning, code generation, and general-purpose inference.

4. Task Taxonomy and Evaluation Methods for MLLMs

4.1. Task Taxonomy

In the field of MLLMs, tasks can be broadly categorized into two fundamental types based on their intrinsic logic and core objectives: understanding tasks and generation tasks. These two categories are closely interconnected and mutually reinforcing, forming the foundational framework for the technological development and practical applications of MLLMs. We further summarize the representative benchmarks and commonly used evaluation metrics associated with each category.

Table 2. Summary of task taxonomy, benchmarks, and evaluation metrics for MLLMs.

Task Taxonomy	Benchmark	Evaluation Metrics
Understanding Tasks	VQA [59] OK-VQA [71] VCR [72] ChartQA [73] VSR [13] MMBench [74] MMStar [75] MMMU [76] MathVista [77] MM-Vet [78]	Accuracy Top-k Accuracy [79] Precision F1 Score mAP NDCG
Generation Tasks	PaintSkills [80] HRS-Bench [81] GenEval [82] DPG-Bench [83] EditBench [84] ConceptMix [85] Emu-Edit [86] EditVal [87] Reason-Edit [88] AnyEdit [89]	CIDEr SPICE FID [90] IS [91] Relevance Faithfulness GPTScore

provides an overview of task types, benchmarks, and evaluation measures, which serves as a reference for systematic comparison. From a symmetry-oriented perspective, task taxonomies can benefit from maintaining a balanced coverage across modalities, interaction types, and cognitive levels, which ensures that MLLMs are evaluated under structurally diverse yet functionally comparable scenarios.

4.1.1. Understanding Tasks

Understanding tasks aim to extract deep semantic information from diverse and complex multimodal inputs, accurately analyze the intrinsic associations among modalities, and achieve deep cross-modal alignment. This process involves the systematic analysis and comprehensive understanding of multimodal data such as images, videos, texts, and audio, laying the cognitive foundation for subsequent interaction and content generation. Specifically, understanding tasks encompasses the following closely coupled and interdependent sub-directions:

• Deep Visual Content Analysis: This focuses on a comprehensive understanding of images and videos. In visual question answering (VQA), the model is required to respond accurately to natural language questions based on the given image or video content. For example, BLIP [92] significantly improves performance on VQA tasks through an innovative joint training strategy of vision-language models, enabling the model to better capture the semantic association between vision and language. GLaMM [93] introduces a region encoder to support fine-grained visual grounding in question answering, interpreting, and responding to specific regions within an image. Image captioning tasks aim to generate precise and descriptive text for input images. Region understanding and visual grounding tasks focus on locating specific regions in images and extracting their semantic content. VistaLLM [94] and LLaFS adopt unique point-sequence encoding approaches to process polygonal regions, providing key technical support for precise region localization.
• Cross-modal Association and Retrieval: This focuses on establishing tight correlations between different modalities within a shared embedding space and enabling efficient bidirectional retrieval. In cross-modal retrieval, CLIP leverages contrastive learning to align image–text features, laying a solid foundation for image-to-text and text-to-image retrieval. ImageBind further expands these capabilities by incorporating images, videos, and audio, significantly enriching the application scenarios and possibilities of cross-modal retrieval.

4.1.2. Generation Tasks

The core objective of generation tasks is to creatively generate output in one modality based on input from another modality. These tasks demand high-quality, relevant, and controllable content generation. Generation tasks include:

• Image-related Generation and Editing: Image generation tasks aim to create high-quality images based on text descriptions. GILL [53] maps frozen MLLMs to diffusion models to achieve multimodal image generation, offering new paradigms for such tasks. DALL·E 2 [95], based on advanced diffusion models, enables high-resolution image generation, significantly advancing visual quality. Image editing tasks modify images locally or globally according to instructions or text. InstructPix2Pix [96] combines LLMs to generate editing instructions, offering more intelligent and flexible editing capabilities. MLLM-Guided Editing [97] further utilizes multimodal dialogue understanding to refine editing requirements, allowing personalized and precise modifications.
• Cross-modal and Domain-specific Generation: Multimodal generation tasks aim to generate content across modalities, such as text-to-video or image-to-text. NExT-GPT [98], a major breakthrough in this domain, supports arbitrary modality-to-modality generation, expanding the application scope and innovation potential. VITA-1.5 [10] enables interactive generation across speech and vision modalities, supporting speech output from both speech and image inputs, and exemplifies an end-to-end approach to multimodal generation tasks. Domain-specific generation focuses on areas like healthcare and design, leveraging domain knowledge and multimodal data. For instance, Medical Image Synthesis [99] combines medical knowledge bases to generate high-quality diagnostic images, serving as a valuable tool in medical diagnosis. Robot Action Generation [100] translates visual inputs into robotic action commands, enabling intelligent decision-making and control in complex environments, driving real-world deployment of robotic technologies.

4.2. Evaluation Methods

Evaluation plays a critical role in the research and development of MLLMs. It is not only essential for measuring model performance, but also drives optimization and technical advancement. Symmetry principles may also inform the design of evaluation protocols, such as using bidirectional tests or balanced benchmark construction, to reduce modality or task biases and promote more reliable comparisons. Compared to traditional LLMs, MLLMs exhibit more complex characteristics, necessitating a comprehensive evaluation of their multifunctionality, emergent capabilities, and adaptability to intricate interaction scenarios.

Currently, evaluation methods are mainly divided into closed-ended and open-ended approaches. Each is suited to different settings and comes with its own strengths and limitations.

4.2.1. Closed-Ended Evaluation

Closed-ended evaluation focuses on tasks where answers are predefined within a limited set of options. This method relies heavily on structured datasets and standardized metrics, such as accuracy and CIDEr [101] scores. Its main advantage lies in its quantifiability and reproducibility, offering objective benchmarks for model performance.

In a closed-ended evaluation, benchmark datasets play a pivotal role. In tasks such as VQA, image captioning, and scientific reasoning, models are typically evaluated under zero-shot or fine-tuned settings. Comprehensive benchmarks have also emerged to provide a broader assessment. MME [102] covers 14 perceptual and cognitive tasks, ensuring fairness and effectiveness. MMBench [74] evaluates multidimensional capabilities by mapping open responses to predefined options, enriching the evaluation perspectives. LENS [103] introduces a hierarchical visual reasoning benchmark consisting of approximately 3400 real-world images and over 60,000 human-crafted QA pairs across perception, understanding, and reasoning levels. It serves as a closed-set evaluation tool to systematically assess model performance across varying depths of reasoning. Additionally, specific capability tests, such as POPE [104], use adversarial samples to detect hallucinations and assess model reliability in content generation.

4.2.2. Open-Ended Evaluation

Open-ended evaluation requires models to generate free-form responses, which poses higher demands on semantic understanding, reasoning, and creativity. As a result, it is more complex.

Human evaluation remains a key approach in open-ended settings. Experts design question sets covering various dimensions, including visual understanding and complex reasoning, and subjectively assess the quality of model responses. For instance, GPT4Tools [105] assesses the logical coherence of model outputs from multiple perspectives, offering qualitative insights.

GPT-based scoring enables automatic evaluation of response relevance and accuracy using LLMs. With advancements in vision-language capabilities, GPT-4V has emerged, capable of analyzing visual content directly, thereby improving evaluation reliability. Woodpecker [106] scores models based on image–text alignment, offering evaluations more reflective of real-world multimodal tasks. HumaniBench [107] is a human-centered benchmark for MLLMs, covering dimensions such as fairness, ethics, and linguistic inclusivity, and is well-suited for open-set evaluation of social alignment.

However, open-ended evaluation also presents several challenges. On one hand, it relies on subjective criteria or proxy models, potentially introducing bias and affecting result accuracy. On the other hand, human annotation is resource-intensive, while automated evaluation methods still require further validation to ensure robustness and reliability.

Together, the task taxonomy and evaluation methods provide a structured lens through which the capabilities of MLLMs can be assessed. By linking different categories of tasks with appropriate evaluation protocols, they establish a consistent basis for analyzing progress and identifying areas that require further improvement.

5. Key Challenges in MLLM Research

Despite the remarkable progress in research and practical applications of MLLMs, several key challenges remain unresolved. These challenges significantly affect the further enhancement of model performance, broader application deployment, and the sustainable development of the underlying technology. Many of the challenges faced by MLLMs—ranging from hallucination to data noise and ethical risks—can be understood through the lens of representational and functional asymmetries across modalities, model components, and learning signals. The following sections examine these key challenges in detail.

5.1. Multimodal Hallucination

Multimodal hallucination is a prominent issue that severely impacts the reliability and accuracy of model outputs in real-world applications. According to current studies, multimodal hallucination can be broadly categorized into three types:

• Existence hallucination: the most fundamental type, where the model incorrectly claims the presence of objects that do not actually exist in the image, leading to fundamental misinterpretations.
• Attribute hallucination: This refers to incorrect descriptions of object attributes, such as misidentifying the color of a dog. Since attribute descriptions are based on object existence, this type is often closely related to existence hallucination.
• Relation hallucination: a more complex type where the model misrepresents relationships between objects—such as relative positions or interactions—misleading the understanding of spatial and contextual relationships within a scene.

Multimodal hallucination often arises from asymmetric learning signals across modalities, where dominant modalities override weaker ones. Mitigating such an imbalance through symmetry-aware learning objectives may reduce hallucinated outputs and improve semantic fidelity. To mitigate hallucinations, strategies have been proposed across three stages:

• Pre-generation mitigation: This involves fine-tuning with curated data. For instance, LLaVA-RLHF [108] leverages human preference data and reinforcement learning to reduce hallucinations and generate more expected responses. LRV-Instruction [109] introduces a fine-grained visual instruction tuning dataset with negative instructions to guide models toward image-faithful outputs.
• In-process correction: Architectural design or feature representations are modified to reduce hallucinations. For example, VCD [110] attributes hallucinations to dataset bias and strong language priors, proposing a decode-then-contrast design. HACL [111] explores visual and language embedding spaces using contrastive learning to pull aligned cross-modal pairs closer and push apart hallucinated representations.
• Post-generation correction: Methods like Woodpecker provide hallucination correction after the output is generated. Woodpecker is a training-free, general hallucination correction framework that integrates expert models to supplement image context and employs a stepwise correction pipeline with intermediate result inspection.

5.2. Challenges of Noisy Data

Automatically collected image–text pairs from the internet often contain various forms of noise, significantly hindering model training and generalization. Without human curation, textual descriptions may not semantically align with corresponding images, resulting in mislabeled or inaccurate annotations that provide misleading signals during training.

Noise and inconsistencies in multimodal data can amplify asymmetries in representation learning, leading to biased or unstable predictions. Encouraging structural or distributional symmetry during training may help improve model resilience to noise. Researchers have developed several techniques. For instance, ALBEF [112] improves robustness to noisy web data using momentum distillation, where a “momentum model” generates pseudo-targets as additional supervision, thereby reducing reliance on noisy labels. BLIP incorporates a captioning and filtering module that generates synthetic captions and filters them based on image–text matching quality, extracting high-quality pairs from large-scale web data. MVCD [107] introduces a zero-training inference-time contrastive decoding strategy that compares output distributions between original and perturbed visual inputs, enhancing robustness of multimodal reasoning under noisy visual data—demonstrating a practical approach to tackling input noise.

5.3. Computational and Storage Costs

The development of MLLMs demands extensive computational resources and storage, presenting a major bottleneck for widespread deployment. Traditional vision-language pretraining approaches rely on convolutional neural networks (CNNs) like ResNet and Faster R-CNN [113], which incur high computational costs for region extraction and other steps.

Imbalanced architectural complexity or processing pathways across modalities can lead to inefficient computation and scaling bottlenecks. Designing symmetric or unified processing blocks may reduce redundancy and facilitate more efficient parameter sharing. To reduce these costs, several innovations have emerged. Vision Transformer divides images into patches and uses transformers for processing, avoiding convolutions and improving efficiency. ViLT further simplifies the architecture by removing CNNs and region features, instead using simple linear projections for patch embedding and fusion with text, achieving strong downstream task performance with reduced cost. ALBEF separates the decoder into two parts: the first processes unimodal text without cross-attention, reducing complexity; the second performs cross-modal attention with the image encoder.

5.4. Safety and Ethics

As MLLMs are increasingly applied in critical domains, they face growing risks of adversarial attacks. Malicious inputs, such as adversarial samples, can disrupt model performance and lead to incorrect or harmful outputs, undermining decision-making reliability.

Moreover, these models may absorb various societal biases from training data—such as those related to gender, race, or culture. Studies have shown that web-scraped data can lead to models generating inappropriate or biased content. Although efforts have been made in text-to-image generation to mitigate such biases [114,115], further exploration is necessary for MLLMs.

These biases can manifest in model outputs, resulting in unfair treatment of certain groups and violating ethical principles. For example, facial recognition performance may vary across racial groups, and text generation may contain gender stereotypes. Ethical risks often stem from asymmetric treatment of modalities, tasks, or user groups during training and deployment. Incorporating symmetry-aware fairness constraints—such as balanced representation, response parity, or equal risk exposure—may contribute to safer and more equitable MLLM systems.

These challenges underscore not only the technical limitations of current MLLMs but also the broader concerns of reliability, efficiency, and ethical responsibility. Recognizing and systematically addressing these issues will lay the foundation for sustainable progress in the field.

6. Future Research Directions for MLLMs

Building on the significant progress achieved thus far, future research in MLLMs holds immense potential to overcome existing technical bottlenecks, expand application boundaries, and advance towards more efficient, intelligent, and practical systems. Addressing the limitations of current MLLMs will increasingly require principled frameworks that go beyond empirical scaling. Among these, symmetry—conceived as a structural and functional balance across modalities, temporal sequences, and representational layers—offers a compelling lens through which to guide future research. Symmetry-informed approaches have the potential to enhance robustness, coherence, and generalization in multimodal reasoning by enforcing consistency across modalities, aligning semantic structures, and balancing information flow. In the following, we outline three directions where symmetry-aware principles may play an important role in shaping the next generation of MLLMs.

6.1. Enhancing Long-Range Cross-Modal Reasoning

Long-range cross-modal reasoning poses unique challenges in maintaining coherent semantic dependencies over extended sequences. Symmetry principles—particularly those promoting temporal and structural alignment across modalities—may support the development of models capable of sustaining consistent reasoning across diverse inputs and time spans. Traditional vision-language models have laid the foundation for dynamic reasoning through explorations in modality fusion mechanisms. For instance, VLMO proposes a hybrid modality expert architecture that combines modality-specific experts with shared attention, supporting task-driven encoding mode switching. BEiT V3 achieves modality-adaptive encoding under a unified architecture, effectively addressing issues such as manual task format conversion and inefficient parameter sharing across modalities.

DeepSeek-R1 extended its hybrid expert routing mechanism to the multimodal domain. Its reinforcement learning-based dynamic expert selection strategy improves cross-modal resource allocation and demonstrates the effectiveness of dynamically adjusting modality weights in mathematical reasoning tasks. Grok-3 reinforces the stability of modality interaction through ultra-large-scale training.

Furthermore, multimodal in-context learning expands upon traditional in-context learning by integrating multimodal inputs such as images and texts. This enables inference with minimal annotation and promotes reasoning transfer using only a few examples, with performance heavily influenced by example quantity and order. Multimodal chain-of-thought reasoning mimics human step-by-step reasoning by generating reasoning steps and final answers for a given task. It enables complex multimodal reasoning by combining image interpretation with textual understanding, supporting decomposition and solution of intricate tasks, and significantly enhancing the model’s reasoning capabilities in complex multimodal scenarios.

6.2. Finer-Grained Cross-Modal Interaction

To enhance user–agent interaction, research increasingly focuses on supporting finer-grained multimodal inputs and outputs. Achieving fine-grained multimodal understanding requires tightly coupled interactions between modalities at multiple levels of abstraction. Symmetry-guided mechanisms, such as bidirectional attention balancing and shared representational mapping, offer promising means to enhance the precision and reciprocity of such interactions.

On the input side, Ferret [116] introduces a hybrid representation framework that supports diverse forms such as points, bounding boxes, and sketches to improve fine-grained object grounding. Osprey integrates a pretrained segmentation model to enable precise part-level localization with a single click, overcoming the limitations of pixel-level interaction.

On the output side, LISA [117] achieves mask-level visual understanding and reasoning, enabling pixel-wise semantic segmentation tasks for more precise and accurate model outputs.

Recent VLA models such as Helix [118], GR00T N1 [119], and Gemini Robotics [120] demonstrate fine-grained cross-modal interaction through unified architectures. By integrating perception, instruction, and control via cross-modal attention and aligned representations, they exemplify precise coordination across semantic and action layers. Finer-grained interaction across modalities is expected to be a key future direction. By exploring improved alignment strategies, interaction mechanisms, and fusion techniques, the performance and applicability of MLLMs can be further enhanced.

6.3. New Application Paradigms

As MLLMs are increasingly deployed in open-ended and dynamic scenarios, ensuring stable adaptation across novel tasks and modalities becomes essential. Here, symmetry-aware adaptation strategies can help maintain functional balance, reduce modality dominance, and support consistent performance across shifting application contexts. In graphical user interface interactions, CogAgent [121] supports GUI element parsing for automating app operations. Domain-specific adaptation is another prominent trend. In document understanding, leveraging structural data without OCR improves table recognition accuracy. In medical imaging, LLaVA-Med incorporates medical knowledge graphs and achieves 89% clinical consistency on radiological VQA tasks, demonstrating the potential of MLLMs in domain-specific applications.

6.4. Expansion Toward Omni-MLLMs

Omni-MLLMs represent a major extension of MLLMs in terms of modality breadth. Unlike earlier models that mainly focused on text–image tasks, Omni-MLLMs integrate a wider range of modalities within a unified framework, including speech [122], video, 3D environments, and IMU sensor signals [123]. These models are also exhibiting fine-grained cross-modal interaction abilities, such as ImageBind-LLM [124], which supports joint reasoning across images, audio, and 3D data, and CoDi-2 [125], which extends the capability to “Any-to-Any” generation by producing both audio and image outputs from interleaved multimodal contexts. At the same time, the application scope of Omni-MLLMs has been significantly broadened, ranging from real-time multimodal dialogue [126] to world simulation [127] and multi-sensor autonomous driving [128]. In addition to open-source models, several closed-source systems have also pushed forward omni-modal capabilities in both research and industry. Nevertheless, despite the rapid emergence of Omni-MLLMs, a systematic evaluation framework, standardized data construction methods, and a deeper understanding of cross-modal generalization remain underexplored, underscoring important directions for future research. Overall, future research on MLLMs is converging toward more general, fine-grained, and versatile systems that can operate across diverse modalities and tasks. Such directions highlight the field’s ambition to move closer to robust and truly universal multimodal intelligence.

7. Conclusions

This survey has reviewed the landscape of MLLMs, covering their architectures, training strategies, evaluation methodologies, and task classifications. We highlighted major challenges—such as multimodal hallucination, noisy data, computational costs, and safety concerns—and connected them to open research opportunities. Future directions point to enhancing long-range reasoning, enabling finer-grained cross-modal interaction, exploring new application paradigms, and expanding toward Omni-MLLMs. By aligning current limitations with these emerging trends, this work not only consolidates the present state of MLLM research but also outlines concrete paths toward more reliable, efficient, and generalizable multimodal intelligence.