A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision–Language Tasks

Abstract

This survey provides a comprehensive guide to Multimodal Large Language Models (MLLMs) with a focus on vision–language tasks, including image captioning, visual question answering, cross-modal retrieval, visual grounding, multi-image reasoning, long-video understanding, and embodied AI. We examine architectures, training pipelines, and practical applications, covering visual encoders, language model backbones, connector modules, contrastive pre-training, instruction tuning, and preference alignment. We also foreground first-principles constraints—information bottlenecks, data-processing limits, and statistical co-occurrence bias—that shape architecture, robustness, and evaluation. This survey centers on vision–language systems and does not cover audio-only models or code-generation tools without visual inputs. Through task-level analysis and system-level case studies, we examine prominent MLLM implementations while addressing key challenges in scalability, memory, energy use, inference cost, robustness, and cross-modal learning. We present a unified taxonomy of the MLLM design space, a comparative overview of representative models and evaluation benchmarks, and a discussion of open problems. Concluding with ethical considerations and responsible AI development, this survey offers theoretical frameworks and practical insights for researchers, practitioners, and students working at the intersection of natural language processing and computer vision.

1. Introduction to Multimodal Large Language Models (MLLMs)

• Positioning relative to existing surveys. Several concurrent surveys cover overlapping ground. Yin et al. [1] provide an extensive taxonomy of MLLM architectures and training paradigms but do not include quantitative cross-model benchmark comparisons. Zhang et al. [2] focus on architectural evolution and representative models, yet omit design-space gap analysis and systematic failure mode documentation. Caffagni et al. [3] survey the broader MLLM landscape with emphasis on generative capabilities, but do not offer a self-contained tutorial path for newcomers. Our work complements these surveys by combining a pedagogical progression with original analytical contributions, as detailed below.

Figure 1 summarizes the organization and dependencies of the survey.

• Contributions of this survey. Building on this positioning, our work makes the following distinct contributions:

• End-to-end technical guide. We provide a self-contained progression from NLP foundations through MLLM architectures, training strategies, applications, and open challenges, designed to serve both as a reference for active researchers and as an accessible entry point for newcomers to the field.
• Unified empirical comparison. We compile a systematic, side-by-side benchmark comparison of fifteen major open-source and proprietary vision–language MLLMs—including recent 2024 models such as GPT-4o, Qwen2-VL, InternVL2, DeepSeek-VL, and LLaVA-OneVision—across five standardized benchmarks (VQAv2, GQA, TextVQA, MMBench, MMMU), supporting side-by-side architectural context and performance analysis (Section 2.3).
• Design-space gap analysis. We introduce a structured mapping of the Visual Encoder × LM Backbone × Connector design space (Section 6.7), identifying under-explored architectural combinations and training–strategy pairings that represent concrete opportunities for future research.
• Candid failure mode analysis. Unlike surveys that focus on aggregate benchmark scores, we document recurring failure modes of state-of-the-art multimodal assistants—including spatial reasoning errors, object hallucination, and multi-turn context loss—providing a more realistic assessment of current capabilities and limitations.

1.1. Definition and Importance of MLLMs

Multimodal Large Language Models (MLLMs) represent a significant evolution in artificial intelligence (AI), enabling the integration and understanding of inputs such as text, images, and video, with audio considered where it appears in multimodal systems. Unlike unimodal models restricted to a single input type, MLLMs process multiple modalities simultaneously, providing a more comprehensive understanding that reflects real-world interactions.

The key features and importance of MLLMs include:

Cross-Modal Learning: In the vision–language setting, MLLMs learn correspondences between text and visual inputs; broader multimodal systems may also incorporate audio or other sensory streams. This capability allows them to create connections between different modalities, enabling tasks that require comprehension and generation of content across diverse data types. For example:

• Text-to-Image Generation: MLLMs can generate detailed images from textual descriptions, supporting creative workflows in areas such as graphic design and advertising. For example, a user might describe a “futuristic cityscape at sunset” and have an AI generate a corresponding image.
• Visual Question Answering: These models can analyze images and provide accurate answers to natural language questions, enhancing educational tools and accessibility technologies. For instance, an MLLM could answer questions about the contents of a photograph, such as “What breed of dog is in this image?”
• Multimodal Content Creation: MLLMs facilitate the creation of content that integrates text, visuals, and audio, such as illustrated stories or multimedia presentations. This could involve generating a coherent story with matching illustrations based on a brief prompt.

Unified Representation: MLLMs achieve integrated representations of multimodal data through joint embedding spaces, learned projection modules, or shared token representations, enabling coordinated processing across different modalities. This architectural approach offers several key capabilities:

• Seamless translation between modalities (e.g., describing a photograph or generating an image from text).
• Cross-modal retrieval, where the model can find relevant images based on text queries or match sounds with visual content.
• More natural and intuitive interactions between humans and AI systems.

Intuitively, unified representations act like a shared cataloging system in which related textual and visual items can be retrieved or compared within the same semantic space.

Enhanced Contextual Understanding: By integrating multiple modalities, MLLMs generate more accurate and context-aware responses. This capability is particularly valuable in fields such as:

• Healthcare: Assisting clinicians by analyzing medical images alongside patient records and physician notes. For example, an MLLM could assist a clinician by jointly presenting evidence from a patient’s X-ray, medical history, and symptoms for human review.
• Security: Interpreting surveillance footage in conjunction with audio data to support situational awareness. This could involve analyzing video feeds and audio recordings to flag events for human review.
• E-commerce: Enhancing product searches by understanding both textual queries and visual product attributes. An MLLM could help a customer find a “blue floral summer dress” by understanding both the text description and visual characteristics of available products.

Generalization Across Modalities: MLLMs demonstrate flexibility in handling various tasks across different modalities, including:

• Image captioning and visual question answering.
• Cross-modal retrieval and content generation.
• Audio–visual integration for tasks like video subtitling or lip-syncing.
• Multimodal translation, such as converting a video into a textual summary.
• Enhanced human–computer interaction through simultaneous interpretation of gestures, facial expressions, speech, and text.

Advancements in Robotics and Embodied AI: In robotics, MLLMs contribute to systems that can perceive and interact with their environment more effectively. By processing visual, auditory, and sensory data, robots powered by MLLMs can perform complex tasks such as object manipulation, navigation, and human–robot interaction. For instance, a household robot could understand and execute a verbal command like “Please bring me the red mug from the kitchen counter,” by combining language understanding with visual recognition and spatial navigation.

Real-World Application Potential: This breadth makes MLLMs relevant to applications where decisions depend on both visual and textual context. For instance:

• In autonomous vehicles, these models can integrate visual data from cameras with textual information from maps and traffic reports, supporting navigation and safety features. An MLLM could help a driving system interpret a road sign as one input to a broader perception and control stack.
• In scientific research, MLLMs can analyze molecular structures, research papers, and experimental data simultaneously to help researchers identify candidate compounds or literature patterns for further validation.

Bridging the Gap Between AI and Human Cognition: MLLMs’ ability to process multiple modalities is inspired by aspects of human multimodal perception more closely than unimodal models. This alignment can lead to AI systems that are more intuitive to use and better at handling complex, context-dependent situations. For example, an MLLM-powered virtual assistant could adapt its response based on a user’s tone of voice, facial expression, and choice of words, while still requiring careful evaluation before use in sensitive settings.

Table 1. MLLM capability categories: overview and application examples.

Capability	Description	Representative Applications
Cross-modal Understanding	Processing and reasoning over inputs from multiple modalities simultaneously	Visual question answering (VQA), audio–visual speech recognition
Text-to-Image Generation	Generating images conditioned on natural language descriptions	DALL-E 3, Stable Diffusion, Midjourney
Image Captioning	Producing natural language descriptions of image content	NoCaps, MS COCO captioning
Cross-modal Retrieval	Finding relevant content across modalities given a query	CLIP-based image search, video retrieval
Visual Grounding	Localizing and identifying objects referenced by language	Referring expression comprehension, Grounding DINO
Multimodal Generation	Producing coherent content spanning text and vision jointly	Illustrated story generation, slide deck synthesis
Embodied AI	Controlling agents in physical or simulated environments through multimodal perception	Robotic manipulation, vision–language navigation
Document Understanding	Parsing and reasoning over text-rich images such as charts and PDFs	DocVQA, ChartQA, InfographicVQA

Color coding: light blue rows denote core perception capabilities, light teal rows denote advanced capabilities, light gray rows denote alternating entries, and yellow cells mark representative examples.

summarizes the key capability categories of MLLMs, their descriptions, and representative application examples.

1.2. The Convergence of NLP and Computer Vision

The emergence of MLLMs stems from decades of parallel progress in NLP and computer vision, which converged through three key phases: (1) CNN–RNN pipelines for image captioning [4], (2) task-specific vision–language models for VQA [5], and (3) transformer-based architectures such as ViLBERT [6] and CLIP [7] that learn aligned vision–language representations from web-scale data. Section 2 traces the NLP-side evolution in detail; here we highlight three theoretical pillars that underpin this convergence:

• Representation learning enables joint embeddings that capture cross-modal semantic relationships, allowing models to map language and visual concepts into a shared space.
• Transfer learning allows large-scale pre-trained models to be adapted to downstream multimodal tasks with minimal task-specific data.
• Cross-modal attention extends the self-attention mechanism of transformers to selectively attend across modalities, forming the basis of modern MLLM fusion modules.

• These foundations, combined with the architectural innovations discussed in Section 2.3—visual encoders, cross-modal connectors, and large language model backbones—have enabled the current generation of MLLMs. The challenges associated with this convergence, including bias, interpretability, efficiency, and cross-modal consistency, are examined in Section 6 and Section 7.

2. Foundations and Architecture of Vision–Language MLLMs

This section traces the path from text-only NLP to today’s vision–language MLLMs. As introduced in Section 1, MLLMs extend LLMs with modality-specific encoders and cross-modal fusion modules, enabling joint reasoning over language and vision. Their design space is organized along three architectural axes: the visual encoder (e.g., ViT, CLIP), the language model backbone (e.g., GPT, LLaMA), and the connector module (e.g., Q-Former, linear projection). Before detailing these components, we foreground several first-principles constraints—information bottlenecks, data-processing limits, and statistical co-occurrence bias—because they govern the later discussion of connector design, benchmark interpretation, hallucination, and efficiency.

Figure 2 presents the taxonomy of vision–language MLLMs covered in this survey, organized along five design axes.

Table 2. Representative vision–language MLLMs: architectural and training overview.

Model	Yr	Visual Encoder	LM Backbone	Connector	Training Objective	Post-Training
Flamingo	2022	NFNet/CLIP ViT	Chinchilla (70B)	Gated cross-attn.	LM + ITC	None
BLIP-2	2023	CLIP ViT-L/14	OPT/FlanT5	Q-Former	ITC + ITM + ITG	None
MiniGPT-4	2023	EVA-CLIP ViT	Vicuna-13B	Linear proj.	Captioning	Instr. tuning
LLaVA	2023	CLIP ViT-L/14	LLaMA-13B	Linear proj.	VQA + caption	Instr. tuning
LLaVA-1.5	2023	CLIP ViT-L/14	Vicuna-13B	MLP proj.	VQA + caption	Instr. tuning
InstructBLIP	2023	CLIP/EVA-CLIP	Vicuna/FlanT5	Q-Former	Instr.-following	Instr. tuning
mPLUG-Owl	2023	CLIP ViT-L/14	LLaMA-7B	Visual abstractor	LM + ITC	Instr. tuning
IDEFICS	2023	OpenCLIP ViT-H	LLaMA	Gated cross-attn.	LM + ITC	Instr. tuning
InternVL	2024	InternViT-6B	InternLM-20B	MLP	Contrastive + gen.	Instr. tuning
Qwen-VL	2023	OpenCLIP ViT-G	Qwen-7B	Cross-attn.	VQA + det. + cap.	Instr. + RLHF
DeepSeek-VL	2024	SigLIP + SAM-B	DeepSeek-LLM-7B	MLP proj.	VQA + cap. + OCR	Instr. tuning + DPO
LLaVA-OneVision	2024	SigLIP-SO400M	Qwen2-7B	MLP proj.	VQA + cap.	Instr. tuning
Qwen2-VL	2024	ViT-675M (native)	Qwen2-7B	MLP-based merger	Multimodal	Instr. + DPO
InternVL2	2024	InternViT-6B	InternLM2-Chat	MLP	Contrastive + gen.	Instr. + RLHF
Cambrian-1 [8]	2024	Multi-encoder (CLIP + SigLIP + DINOv2 + ConvNeXt)	Vicuna/LLaMA-3	Spatial Vision Aggregator	VQA + cap.	Instr. tuning
CogVLM [9]	2023	EVA2-CLIP-E	Vicuna/LLaMA	Visual expert module	Pre-train + SFT	Instr. tuning
MiniCPM-V 2.6 [10]	2024	SigLIP-400M	MiniCPM/Qwen2-7B	Perceiver Resampler	Multi-task	Instr. + DPO
Yi-VL [11]	2024	CLIP ViT-H/14	Yi-6B/Yi-34B	MLP proj.	Vision–language SFT	Instr. tuning
GPT-4V	2023	Proprietary	GPT-4	Proprietary	Undisclosed	RLHF
GPT-4o	2024	Proprietary (native)	GPT-4o	Proprietary (native)	Undisclosed	RLHF †
Gemini	2023	Proprietary	Gemini	Proprietary	Multimodal	RLHF/instr.
Claude 3	2024	Proprietary	Claude 3	Proprietary	Undisclosed	Constitutional AI †

Open-source models (teal rows): architectures and weights publicly available. Proprietary models (pink rows): closed systems with undisclosed implementation details. † Inferred from published model cards and product documentation; not formally disclosed by the developer. Abbreviations: ITC = image–text contrastive; ITM = image–text matching; ITG = image–text generation; RLHF = reinforcement learning from human feedback; LM = language modeling.

provides a comparative overview of representative vision–language MLLMs discussed in this survey, summarizing their key design choices across visual encoder, language backbone, connector type, training objective, and post-training strategy.

2.1. From NLP to LLMs: A Brief Overview

• This subsection provides contextual background for readers less familiar with the NLP progression that led to modern LLMs. Readers already versed in this history may proceed directly to Section 2.2.

MLLMs are the most recent stage of a decades-long progression in natural language processing: from symbolic, rule-based systems [12], to statistical methods such as n-gram models, HMMs, and SVMs [13], to distributed word representations [14,15], and to recurrent sequence models with soft attention [16,17]. The transformer [18] replaced recurrence with self-attention, enabling full parallelization and more effective modeling of long-range context, and became the substrate for BERT-style masked pre-training and the GPT family of autoregressive language models. Scaling these architectures together with instruction tuning and reinforcement learning from human feedback produced today’s LLMs (GPT-3/4, LLaMA, PaLM), whose few-shot generalization and pre-trained representations are precisely what MLLMs extend to jointly reason over textual and visual inputs for tasks such as image captioning, visual question answering, and multimodal dialogue.

2.2. First-Principles Constraints on Multimodal Learning

MLLM design is often described as a menu of engineering choices—which visual encoder, which language backbone, which connector, and which instruction tuning recipe. These choices are important, but they operate under several first-principles constraints that should be made explicit before specific architectures are compared.

Information bottlenecks. Let $X_v$ denote the visual input, $X_t$ the text input, and Y a downstream target such as the correct answer to a VQA question. Any learned visual representation $Z_v=f_v\left(X_v\right)$ and textual representation $Z_t=f_t\left(X_t\right)$ can preserve at most the task-relevant information contained in the original inputs: by the data-processing inequality, $I(Z_v,Z_t;Y)\le I(X_v,X_t;Y)$. The information bottleneck framework [19] further clarifies that compression is never free: a connector that maps hundreds or thousands of visual tokens into a small set of summary tokens may reduce latency and memory, but it can discard spatial, textual, or temporal evidence needed for grounding, OCR, counting, and multi-image comparison.

Statistical co-occurrence bias. MLLMs are trained on large image–text corpora whose labels are not neutral observations of the world; they contain web-scale priors such as “beach” co-occurring with “umbrella” or “kitchen” with “refrigerator”. A maximum-likelihood model learns $p_{\mathrm{train}}(y\mid x_v)$, not an ideal causal distribution $p_{\mathrm{data}}(y\mid x_v)$. Some hallucinations therefore arise from training-distribution priors rather than from a single defective module. This point later explains why object hallucination should be separated into reducible failures caused by insufficient visual evidence and irreducible-under-the-training-distribution failures caused by biased co-occurrence patterns.

Comparability limits. Benchmark tables mix systems trained under different data scales, data quality regimes, post-training procedures, and disclosure levels. A reported score can compare observed behavior under a benchmark protocol, but it cannot by itself identify whether the cause is architecture, data curation, instruction tuning, reinforcement learning, proprietary post-training, or test contamination. Consequently, the empirical comparisons below should be read as performance summaries rather than controlled causal experiments.

2.3. Architecture of MLLMs

The architecture of MLLMs combines language modeling with modules that encode and align visual information. This integration allows MLLMs to perform tasks that require reasoning across both textual and visual domains.

At its core, a vision–language MLLM usually follows an encoder–connector–language-model pipeline. A visual encoder extracts image or video features, a connector module projects those features into a representation compatible with the language model, and the language model backbone generates task-appropriate text, labels, or actions conditioned on both modalities.

Different systems instantiate this pipeline in different ways. Some use cross-attention layers or Q-Former-style query tokens to select visual evidence, whereas others use simpler linear or MLP projections to preserve patch-level features. The choice of connector affects alignment quality, spatial fidelity, memory use, and inference cost, which is why later sections compare connector designs and benchmark behavior in detail.

Figure 3 summarizes the encoder–connector–language-model pipeline described above.

Transformer Backbone:

Figure 4 shows the transformer encoder–decoder backbone used by many MLLM language backbones.

The transformer architecture, introduced by [18], serves as a central backbone for MLLMs because self-attention can model relationships among text tokens, image patches, and other sequence elements without relying on fixed local neighborhoods. This makes transformers well suited to multimodal tasks that require long-range dependencies, parallel processing, and nuanced relationships between visual and textual representations.

The transformer’s architecture typically consists of multiple layers of self-attention and feed-forward neural networks. In MLLMs, this structure is often adapted to include:

• Encoder layers: These process the input data, whether it is text tokens or visual features, and create contextualized representations.
• Decoder layers: These generate an output based on the encoded representations, often incorporating cross-attention mechanisms to attend to relevant parts of the input.
• Multi-head attention: This allows the model to attend to different aspects of the input simultaneously, enhancing its ability to capture diverse relationships in multimodal data.
• Position encodings: These are crucial for maintaining spatial or sequential information, especially important when dealing with image patches or text sequences.

Scalability is another factor that contributes to the transformer’s popularity among MLLMs. Models such as GPT-3 and CLIP show that larger transformer-based systems can improve performance and generalization, but these gains depend strongly on data quality, training objectives, architecture, and available compute. This scalability allows MLLMs to leverage large multimodal datasets, while also increasing the importance of efficient training and inference.

Furthermore, the transformer’s architecture facilitates efficient parallel processing, making it well-suited for handling the computational demands of large-scale multimodal tasks. This efficiency is crucial when processing high-dimensional inputs like images alongside text.

Multimodal Embedding:

Figure 5 illustrates a general multimodal embedding pipeline for image, text, and audio inputs.

For this survey, the relevant case is vision–language embedding, where text and visual information are represented in a shared or aligned space so their relationships can be leveraged effectively.

Multimodal embedding is a crucial component in the architecture of MLLMs, enabling these models to represent and reason about diverse types of information within a unified computational framework. In the vision–language instantiation emphasized here, visual and textual inputs are embedded and aligned for tasks such as image–text retrieval, captioning, and visual question answering. The resulting embeddings capture semantic relationships not only within each modality but also across modalities, facilitating more sophisticated cross-modal reasoning and analysis.

The process of creating multimodal embeddings typically involves several key steps:

• Modality-Specific Encoding: Initially, each input modality is processed through specialized encoders. For textual data, this typically involves tokenization followed by learned token embeddings or contextual representations from a language model. Visual data is typically encoded using Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs) to extract salient features.
• Dimensionality Alignment: The embeddings from different modalities are often of different dimensionalities. A crucial step is to project these embeddings into a common dimensional space, usually through learnable linear transformations or more complex neural network layers.
• Joint Representation Learning: The aligned embeddings are then further processed to create a truly joint representation. This often involves attention mechanisms or fusion layers that allow the model to learn complex interactions between the modalities.
• Contrastive Learning: Many state-of-the-art MLLMs employ contrastive learning techniques during training. This approach encourages the model to produce similar embeddings for semantically related text–image pairs while pushing apart unrelated pairs in the embedding space.
• Fine-tuning for Downstream Tasks: The joint embeddings are then fine-tuned on specific downstream tasks, allowing the model to adapt its representations for particular applications while retaining the general cross-modal understanding gained during pre-training.

Stages for creating multimodal embeddings

Figure 6 breaks this process into the main stages used to construct multimodal embeddings.

The effectiveness of multimodal embeddings in MLLMs is reflected in their performance on various tasks requiring cross-modal understanding. For instance, in image–text retrieval tasks, these embeddings enable the model to find semantically relevant images given a textual query, and vice versa. In visual question answering, the joint embedding space allows the model to reason about textual questions in the context of visual information.

Moreover, recent research has shown that multimodal embeddings can capture nuanced relationships between concepts across modalities. For example, they can represent abstract concepts that are difficult to visualize directly but are often associated with certain visual patterns or contexts. This capability enables MLLMs to perform higher-level reasoning tasks, such as visual commonsense inference or generating creative descriptions of images.

However, creating effective multimodal embeddings also presents several challenges. These include dealing with the semantic gap between modalities, handling the different statistical properties of visual and textual data, and ensuring that the embeddings generalize well across diverse tasks and domains. Ongoing research in this area focuses on developing more sophisticated embedding techniques, exploring ways to incorporate additional modalities, and improving the interpretability of these high-dimensional representations.

Cross-Attention Layers: To enable interaction between text and images, MLLMs often employ cross-attention mechanisms. These layers allow the model to focus on relevant parts of an image when processing a piece of text (or vice versa), enhancing the ability to understand the relationships between the two.

Cross-attention layers are a fundamental component of MLLMs, facilitating the intricate interplay between textual and visual modalities. These layers enable the model to dynamically focus on relevant aspects of one modality while processing information from the other, thereby enhancing the model’s capacity to understand and reason about multimodal inputs.

The mechanism of cross-attention is derived from the self-attention concept introduced in the transformer architecture. However, unlike self-attention, which operates within a single modality, cross-attention allows the model to attend to information across different modalities. In the context of MLLMs, this typically involves attention between textual and visual representations.

The cross-attention process can be formalized as follows:

Let Q represent the query vectors from one modality (e.g., text), and K and V represent the key and value vectors from another modality (e.g., image). The cross-attention operation can be expressed as follows:

$$Attention(Q,K,V)=\mathrm{softmax}\left(Q{K}^T/\sqrt{d_k}\right)V.$$

(1)

where $d_k$ is the dimensionality of the key vectors, and the softmax operation is applied row-wise.

This formulation allows the model to compute attention weights that determine the relevance of each element in one modality to each element in the other modality. For instance, when processing a textual query about an image, the cross-attention layer enables the model to focus on specific regions of the image that are most pertinent to the query.

The benefits of cross-attention layers in MLLMs are multifold:

• Fine-grained multimodal alignment: Cross-attention facilitates precise alignment between elements of different modalities, allowing the model to capture nuanced relationships between specific words and image regions.
• Contextual understanding: By attending to relevant parts of one modality while processing the other, the model can develop a more contextual and holistic understanding of the multimodal input.
• Flexibility in handling varying input sizes: Cross-attention can naturally handle inputs of different lengths or sizes, making it suitable for processing variable-length text and images of different resolutions.
• Improved interpretability: The attention weights produced by cross-attention layers can be visualized, providing insights into which parts of an image the model focuses on when processing specific textual inputs, and vice versa.

Benefits of cross-attention layers

Figure 7 summarizes the practical benefits of cross-attention layers for multimodal alignment and reasoning.

Recent advancements in cross-attention mechanisms for MLLMs include the development of more efficient attention computations to handle large-scale inputs, the incorporation of multi-head cross-attention for capturing diverse cross-modal relationships, and the exploration of hierarchical cross-attention structures to model interactions at different levels of abstraction.

Vision Encoders and Language Model Backbones: MLLMs typically include visual encoders (often based on CNNs or Vision Transformers), connector modules, and language model backbones. These components transform visual inputs into representations that can be consumed by a language model for tasks such as captioning, question answering, retrieval, or action selection.

Vision encoders and language model backbones are fundamental components of MLLMs, serving as the interface between raw visual data, textual prompts, and the model’s internal representations. The visual encoder extracts modality-specific features, while the connector and language model backbone align these features with token-level language representations.

Vision encoders are responsible for extracting salient features from visual inputs. Traditionally, Convolutional Neural Networks (CNNs) have been the predominant architecture for this task. CNNs excel at capturing hierarchical visual features, from low-level edges and textures to high-level semantic concepts. Notable CNN architectures employed in MLLMs include ResNet, Inception, and EfficientNet. These networks typically consist of multiple convolutional layers, pooling operations, and non-linear activations, culminating in a dense representation of the input image.

More recently, Vision Transformers (ViTs) have emerged as a powerful alternative to CNNs for visual encoding. Introduced by [20], ViTs adapt the transformer architecture, originally designed for sequence modeling, to image processing. In a ViT, an image is divided into fixed-size patches, which are linearly embedded and treated as a sequence of tokens. This approach leverages the self-attention mechanism to capture global dependencies in the image, potentially offering advantages over the local receptive fields of CNNs.

The output of the vision encoder is typically a set of feature vectors or a single aggregated vector representing the image. This representation is then projected into a shared embedding space or directly into the language model’s token space through a connector.

The language model backbone is responsible for generating coherent and contextually appropriate textual outputs based on the model’s internal representations. In many current MLLMs, this backbone is a decoder-only transformer that operates autoregressively, generating text one token at a time while conditioning on visual tokens inserted through a connector or cross-attention pathway.

A key feature of modern MLLMs is their ability to handle prompts or queries. By conditioning generation on both the prompt and the visual representation, these systems can produce responses that are relevant to the visual input and tailored to specific instructions or questions.

The interplay between vision encoders, connectors, and language model backbones is crucial for MLLM performance. The quality of visual features extracted by the encoder directly affects the language model’s ability to generate accurate and relevant textual outputs, while the connector determines how much fine-grained visual evidence is preserved.

Recent advancements in this domain include more efficient connector architectures, sparse attention mechanisms for longer sequences, dynamic-resolution processing, and alternative visual encoding strategies. These innovations aim to capture fine-grained visual details while maintaining computational efficiency.

Table 3. Key architectural components of MLLMs: functions and trade-offs.

Component	Function	Common Implementations	Key Trade-Offs
Visual Encoder	Extracts spatial and semantic features from images	CLIP ViT-L/14, EVA-CLIP, InternViT-6B	Higher resolution → richer features but higher cost
Language Backbone	Processes text; generates output tokens autoregressively	LLaMA, Vicuna, GPT-4, Qwen, FlanT5	Larger LM → better reasoning but higher memory
Connector/Fusion Module	Bridges visual and language representation spaces	Q-Former (BLIP-2), MLP proj. (LLaVA), gated cross-attn. (Flamingo)	Q-Former reduces tokens but may lose spatial detail
Cross-modal Attention	Focuses the language model on relevant image regions	Standard cross-attention; sparse cross-attention	Full attention is expressive but costly for long images
Tokenizer (vision)	Converts image patches into discrete or continuous tokens	Patch embedding (ViT); discrete VQ-VAE codes	Discrete tokens: compact; continuous: higher fidelity
Output Head/Decoder	Generates text or structured outputs conditioned on fused representations	Autoregressive LM head; classification head	Autoregressive provides flexibility; classification is faster

compares the main architectural components of MLLMs, their functions, common implementations, and key trade-offs.

Critical comparison of connector designs. While Table 3 summarizes the three dominant connector families, it is important to analyze how their design choices propagate into downstream performance, training stability, and scalability. Standard cross-attention inserts attention layers whose queries come from the language stream and keys/values from the visual encoder. Although conceptually straightforward, injecting unregulated visual signals into a pre-trained language model can destabilize its learned representations. Gated cross-attention, introduced in Flamingo [21], mitigates this by adding learnable gating parameters (initialized near zero) that control how much visual information flows into each language model layer, yielding more stable training at the cost of additional parameters and design complexity.

The Q-Former module in BLIP-2 [22] takes a fundamentally different approach: a set of $N=32$ learnable query tokens attend to the visual encoder output through cross-attention, compressing a ViT patch-token sequence down to 32 summary tokens. For example, common ViT settings produce $\sim 256$ patch tokens at 224 px or $\sim 576$ patch tokens at 384 px; using the latter as P, this reduces the sequence length seen by the language model, lowering the self-attention cost within the LM from $\mathcal{O}(L · {(T+P)}^2)$ to $\mathcal{O}(L · {(T+N)}^2)$ with $N=32$. The trade-off is that aggressive compression can discard fine-grained spatial information needed for tasks such as visual grounding or dense captioning.

Conversely, the simple MLP projection used in LLaVA [23] preserves every visual token by projecting each patch embedding into the language model’s input space. This retains maximal spatial fidelity but shifts the computational burden to the language model, whose self-attention cost grows quadratically with the combined text-plus-vision sequence length—making it significantly more expensive for long-context or high-resolution inputs.

Critically, none of these connector designs scale gracefully to long video or very-high-resolution images, where the number of visual tokens grows linearly (temporal frames) or quadratically (spatial resolution) with input size. This limitation motivates ongoing work on token-merging strategies, hierarchical visual abstraction, and dynamic resolution adapters—an open challenge we revisit in Section 6.

Empirical Comparison of Vision–Language MLLMs

To complement the architectural taxonomy presented above, we provide a systematic, side-by-side comparison of representative vision–language MLLMs on widely adopted benchmarks.

Table 4. Unified benchmark comparison of representative vision–language MLLMs. Scores denote accuracy (%). An asterisk (*) indicates third-party evaluation results rather than numbers reported in the original paper. A dash (–) indicates that no published result is available.

Model	Params (B)	Visual Encoder	LM Backbone	Connector	VQAv2	GQA	TextVQA	MMBench	MMMU
BLIP-2 [22]	12.1	ViT-G/14	FlanT5-XXL	Q-Former	65.0	41.0	40.6	–	–
InstructBLIP [29]	12.1	ViT-G/14	FlanT5-XXL	Q-Former	–	49.5	50.7	36.0	–
MiniGPT-4 [30]	8.0	ViT-G/14	Vicuna-7B	Linear	–	30.8	–	23.0	–
LLaVA-1.5 (7B) [31]	7.2	CLIP ViT-L/14	Vicuna-7B	MLP	78.5	62.0	58.2	64.3	35.4
LLaVA-1.5 (13B) [31]	13.4	CLIP ViT-L/14	Vicuna-13B	MLP	80.0	63.3	61.3	67.7	36.4
Qwen-VL-Chat [32]	9.6	ViT-G	Qwen-7B	Cross-attn	78.2	57.5	61.5	60.6	–
InternVL-Chat-V1.5 [33]	26.0	InternViT-6B	InternLM2-20B	MLP	80.6	64.8	68.0	82.2	46.8
DeepSeek-VL [34]	7.0	SigLIP + SAM-B	DeepSeek-7B	MLP	–	–	–	73.2	36.6
LLaVA-OneVision (7B) [35]	8.0	SigLIP-SO400M	Qwen2-7B	MLP	–	–	–	80.8	48.8
Qwen2-VL (7B) [36]	8.3	ViT-675M	Qwen2-7B	MLP-based merger	–	–	84.3	83.0	54.1
InternVL2 (8B) [37]	8.1	InternViT-300M	InternLM2.5-7B	MLP	–	–	77.4	81.7	51.8
GPT-4V [38]	–	Undisclosed	Undisclosed	Undisclosed	77.2 *	62.0 *	78.0 *	75.1 *	56.8 *
GPT-4o [39]	–	Native	Native	Native	–	–	–	83.4 *	69.1 *
Gemini Pro [40]	–	Native	Native MoE	Native	71.2 *	–	74.6 *	73.6 *	47.9 *
Claude 3 Opus [41]	–	Undisclosed	Undisclosed	Undisclosed	–	–	–	76.7 *	59.4 *

Note: Results marked with * are sourced from third-party evaluation studies (e.g., MMBench and MMMU leaderboards) rather than the models’ original publications, as proprietary systems do not always report results on all benchmarks. For proprietary models, architecture, training data scale and quality, post-training procedure, and inference stack are undisclosed; therefore, the table supports behavioral comparison under reported benchmark protocols, not controlled causal attribution to architecture alone.

consolidates published results from original papers and, where noted, third-party evaluations across five established benchmarks: VQAv2 [24], GQA [25], TextVQA [26], MMBench [27], and MMMU [28]. These benchmarks span general visual question answering, compositional reasoning, scene-text understanding, holistic multimodal ability, and expert-level knowledge reasoning, respectively. By unifying results across architectural families, this comparison supports a side-by-side view of how reported design choices—visual encoder capacity, language model scale, and connector type—coincide with downstream performance.

Table 5. Benchmark comparability and disclosure caveats for representative model classes.

Disclosure Class	Examples	Known Quantities	Comparison Caveat
Open weights or detailed papers	LLaVA-1.5, Qwen2-VL, InternVL2, LLaVA-OneVision	Approximate parameters, encoder, backbone, connector, many benchmarks	More suitable for architecture-level comparison, although data mixtures and post-training recipes may still be incomplete.
Partially disclosed	Gemini, GPT-4V technical reports, Claude model cards	Product capability, selected benchmarks, safety notes	Useful for behavioral comparison, but architecture and training data remain too opaque for controlled causal claims.
Closed proprietary	GPT-4o, Claude 3, hosted multimodal assistants	Public product behavior and third-party benchmark scores	Training data scale, data quality, multimodal mixture, post-training cost, and serving stack are undisclosed and likely not comparable to open-source systems.

summarizes the disclosure limitations that should guide interpretation of these benchmark scores.

Several noteworthy patterns emerge from this comparison, provided that the disclosure caveats in Table 5 are kept in view. First, 2024-generation open-source models report scores that approach or exceed some proprietary systems on perception-oriented public benchmarks, but these comparisons combine architecture, data scale, data quality, post-training, evaluation protocol, and disclosure level. They should therefore be interpreted as behavioral benchmark observations rather than evidence that any single architectural choice has matched a proprietary training stack. Qwen2-VL (7B) reaches 83.0 on MMBench while GPT-4o reports 83.4; InternVL2 (8B) and LLaVA-OneVision (7B) achieve 81.7 and 80.8, respectively. These numerical comparisons require a statistical-power caveat that the MLLM literature habitually omits. The public MMBench-Dev EN split contains 1164 unique questions; at a reported accuracy of $\approx 0.83$, the standard error of each model’s score is approximately $1.1$ percentage points, and the standard error of the difference between two independent model scores is approximately $1.6$ pp. The observed $0.4$ pp gap between Qwen2-VL and GPT-4o is therefore not statistically distinguishable from zero under a simple binomial approximation. The $2.2$ pp gap between Qwen2-VL and LLaVA-OneVision and the $1.3$ pp gap between Qwen2-VL and InternVL2 should likewise be read as suggestive benchmark differences rather than definitive rankings. We believe this kind of rudimentary uncertainty analysis should be standard practice in MLLM benchmarking, and we highlight it here not as a criticism of any specific model but as a pattern the community should correct.

With that caveat in mind, the aggregate pattern is still clear: all three 2024-generation open-source models—Qwen2-VL, InternVL2, and LLaVA-OneVision—report MMBench scores above older proprietary baselines such as GPT-4V ($75.1$) and Claude 3 Opus ($76.7$) by several percentage points. These larger gaps are less likely to be explained by sampling noise alone, but they remain benchmark-level observations rather than controlled architectural comparisons. This supports the narrower conclusion that public perception-oriented benchmarks no longer cleanly separate open and closed systems; they do not prove a comparable data scale, post-training investment, or deployment efficiency.

Second, simpler MLP projections (LLaVA-1.5, InternVL, Qwen2-VL, LLaVA-OneVision) report higher scores than the Q-Former-based systems (BLIP-2, InstructBLIP) in this table. This initially counter-intuitive pattern merits deeper analysis. We identify three contributing factors: (i) Information bottleneck: Q-Former compresses $\sim 256-576$ visual tokens into a fixed set of 32 query tokens, discarding fine-grained spatial and textual details that are critical for tasks requiring localization (visual grounding, TextVQA) or dense scene understanding (GQA). In contrast, MLP projections preserve every patch token, maintaining spatial fidelity. (ii) Encoder–backbone co-evolution: when both the visual encoder and language backbone are strong (e.g., InternViT-6B paired with InternLM2-20B, or SigLIP-SO400M with Qwen2-7B), the alignment problem becomes simpler—the representations are already semantically rich, and a lightweight linear transformation suffices to bridge the embedding spaces. Q-Former’s additional cross-attention layers were designed for weaker encoder–backbone pairings and introduce unnecessary parameters that can overfit or under-train in data-limited instruction tuning regimes. (iii) Resolution scaling: MLP connectors naturally accommodate higher input resolutions and dynamic resolution schemes (e.g., AnyRes in LLaVA-OneVision, Naive Dynamic Resolution in Qwen2-VL) because they process each patch independently, while Q-Former’s fixed query count acts as a resolution-invariant bottleneck that cannot exploit additional visual tokens from higher-resolution inputs.

However, this does not imply that MLP projection is universally optimal. For efficiency-critical deployment scenarios—such as edge devices or real-time video processing—Q-Former’s token compression reduces the sequence length fed to the LM by an order of magnitude, proportionally cutting attention cost. The right connector choice thus depends on the deployment context: MLP projection for accuracy-critical benchmarks and fine-grained tasks, Q-Former or similar compression modules for latency- and memory-constrained settings. Cross-attention connectors offer a middle ground, selectively attending to visual features without compressing them, but at the cost of additional trainable parameters within the LM layers. Table 3 and the benchmark results above should therefore be read in conjunction: connector “quality” is not absolute but task- and resource-dependent.

Third, a notable shift in evaluation practice is visible: several newer models in this comparison omit VQAv2 and GQA from their primary reported results, consistent with a broader move toward more challenging evaluation suites. Finally, the MMMU benchmark—which tests expert-level, multi-discipline reasoning—reveals the most dramatic progress: GPT-4o reaches 69.1 and Qwen2-VL 54.1, a large improvement over their predecessors (GPT-4V: 56.8, Qwen-VL: not reported), yet a gap between open-source and proprietary models persists on reasoning-intensive tasks.

Figure 8 visualizes the relationship between approximate parameter count and MMBench accuracy for the open-source models in the comparison.

2.4. Training Methodologies and Data Requirements

Training MLLMs is a multi-stage process encompassing large-scale pre-training, task-specific fine-tuning, and increasingly, instruction tuning and preference alignment. During pre-training, models learn cross-modal representations from massive image–text corpora (typically hundreds of millions to billions of pairs) through objectives such as contrastive learning, masked language/image modeling, and image–text matching. The choice of pre-training dataset—including scale, diversity, and alignment quality—has a direct impact on downstream performance: misaligned pairs introduce noise, while domain-limited data restricts generalization.

Fine-tuning adapts the pre-trained model to specific downstream tasks (e.g., VQA, image captioning, visual grounding) using curated, task-specific datasets. Recent work has demonstrated that few-shot and zero-shot capabilities can reduce the need for extensive task-specific data, while multi-task fine-tuning on heterogeneous objectives can yield mutually beneficial transfer across tasks.

A comprehensive treatment of pre-training strategies, fine-tuning techniques, few-shot and zero-shot learning, and instruction tuning is provided in the dedicated Training section of this survey (Section 3). The remainder of this subsection focuses on two cross-cutting concerns—data quality and computational cost—that apply across all training stages.

The quality and scale of training data are paramount. State-of-the-art models are trained on datasets ranging from million-scale corpora such as Conceptual Captions to billion-scale corpora such as LAION-5B. Ensuring accurate image–text alignment, managing semantic coherence across diverse domains, and maintaining ethical standards (privacy, bias, copyright) remain significant challenges in dataset curation. Training frontier-scale models is computationally intensive, often requiring distributed processing across many GPUs or TPUs, with implications for both environmental sustainability and equitable access to MLLM research.

2.5. Cross-Modal Understanding and Visual Reasoning

Cross-modal understanding—the ability to jointly reason over visual and textual information—is a defining capability of MLLMs and distinguishes them from unimodal language or vision models. This capability underpins a range of downstream tasks, including visual question answering (VQA), image captioning, visual grounding, cross-modal retrieval, and multimodal dialogue.

Rather than treating each task in isolation, recent MLLMs approach cross-modal understanding through a shared set of learned mechanisms: the visual encoder extracts spatial and semantic features from images, the connector module projects these features into the language model’s embedding space, and the language model backbone generates task-appropriate outputs conditioned on both modalities. The quality of this pipeline—particularly the fidelity of cross-modal alignment—determines performance across all downstream applications. Detailed discussions of individual tasks, including image captioning and VQA methodologies, benchmark datasets, and evaluation protocols, are provided in the Applications section of this survey.

Beyond task-specific performance, several dimensions of visual reasoning remain active research problems for current MLLMs. Spatial reasoning—understanding relative positions, sizes, and arrangements of objects—continues to produce errors even in state-of-the-art models, as documented in recent evaluations of GPT-4V and Gemini. Compositional reasoning, which requires understanding novel combinations of known concepts (e.g., “a blue banana on a red table”), exposes brittleness in models trained primarily on naturalistic image–text pairs. Temporal reasoning over video sequences demands tracking object states and causal relationships across frames, a capability that current frame-sampling approaches handle only coarsely. Finally, commonsense and counterfactual reasoning—inferring what is likely, plausible, or physically possible in a scene—requires world knowledge that goes beyond pattern matching on training data. Progress on these reasoning dimensions, rather than aggregate benchmark scores alone, will be critical for advancing MLLMs toward robust multimodal intelligence.

2.6. Multi-Image and Long-Video Reasoning

The vision–language tasks surveyed above are often evaluated on a single image paired with a short prompt. This is no longer sufficient for current MLLMs. Many real use cases require reasoning over multiple visual inputs or temporally extended video: comparing two images (“Which object is larger, the one in image A or the one in image B?”), detecting changes across before/after frames, following a person through a five-minute video, or answering temporal order questions such as “What did the person do first and then what?” These tasks stress a different set of architectural choices because the model must preserve visual identity, spatial scale, temporal order, and cross-frame coreference rather than merely describe one static scene.

Three technical routes dominate current systems. Frame-sampling and image-LLM reuse convert a video into a small set of frames and feed them through an image-trained visual encoder; this is simple and compatible with LLaVA-style architectures, but it loses events between sampled frames and scales poorly when dense temporal evidence is required. Temporal encoders and video–text contrastive/captioning models explicitly adapt image–text foundations to video. VideoCoCa [42], for example, reuses a pre-trained image–text contrastive captioner and adapts it to flattened frame embeddings for video classification, retrieval, captioning, and VQA; Video-LLaMA [43] adds video and audio Q-Formers to align temporal visual and auditory information with an LLM; and Video-LLaVA [44] unifies image and video representations before projection so that mixed image–video instruction tuning can improve both modalities. Memory or repository-based long-video systems compress long streams into structured intermediate state. LangRepo [45] maintains an interpretable textual repository updated over multi-scale video chunks, allowing the LLM to read and write compressed event summaries rather than attend over every frame.

Evaluation is also shifting. LongVideoBench [46] evaluates long-context interleaved video–language understanding with videos up to an hour long and questions that require retrieving and reasoning over referred temporal contexts. Such benchmarks expose the gap between short-clip perception and long-horizon understanding: performance depends not only on a strong visual encoder but also on frame budget, temporal indexing, memory compression, and the ability to avoid confusing early and late events. These requirements connect directly to embodied AI, where a robot or agent must compare current observations with previous states, maintain task history, and decide which visual evidence is relevant for action.

3. Training and Fine-Tuning Multimodal Large Language Models (MLLMs)

Figure 9 illustrates a four-stage training and evaluation workflow common to modern vision–language MLLMs: pre-training on large-scale image–text corpora [47], instruction fine-tuning [48,49], preference alignment, and deployment evaluation.

3.1. Pre-Training Strategies

Pre-training exposes the model to large-scale multimodal datasets of paired text and images, building general-purpose language and visual representations that can later be adapted to specific tasks.

3.1.1. Contrastive Learning (CLIP, ALIGN)

Contrastive learning is a key method in training Multimodal Large Language Models (MLLMs) [50]. Key methods include the following.

Basic Concept: Contrastive learning involves training models to differentiate between similar and dissimilar pairs of data. In the case of MLLMs, this often means aligning text and image pairs while distinguishing them from mismatched pairs. This approach helps in creating shared embedding spaces for different modalities, which is crucial for tasks like cross-modal retrieval [7].

Hallucination-Augmented Contrastive Learning: Later robustness-oriented variants, such as hallucination-augmented contrastive learning, introduce synthetic hard examples or generated negatives to strengthen robustness and generalization, especially in zero-shot scenarios where the model must handle tasks it has not been explicitly trained on [51].

Img-Diff: Contrastive Data Synthesis: This technique involves creating a novel dataset that enhances the quality of contrastive learning by synthesizing new data points. The Img-Diff dataset, for instance, focuses on improving the quality of multimodal data, which is essential for the effective training of high-performance MLLMs [52].

Integration with Other Techniques: Contrastive learning is often combined with other methods like masked language modeling and visual question answering to enhance the model’s understanding of multimodal data. This integration helps in building robust models that can handle a wide range of tasks across different modalities [7].

Two landmark models illustrate this paradigm: CLIP (OpenAI) learns to associate images with text descriptions via contrastive objectives over 400M web-collected pairs, enabling strong zero-shot transfer; ALIGN (Google) demonstrates that the same approach scales to noisier, billion-scale datasets while retaining competitive performance.

3.1.2. Masked Multimodal Modeling

Masked multimodal modeling extends traditional masked language modeling beyond missing words to include masked image regions or visual tokens. The model must recover missing textual and visual evidence from the surrounding multimodal context, forcing it to learn joint representations of text and images [53].

Downstream embodied settings can benefit from representations learned in this way: a robot following an instruction must align words, visual regions, and action-relevant objects before estimating an action such as an end-effector pose [54].

The training of MLLMs often involves a combination of image–text contrastive learning, image–text matching, and masked language modeling. These tasks collectively help the model to align visual and textual modalities, improving its performance in tasks such as image captioning, visual question answering, and more complex multimodal interactions [50].

Ongoing research in this area focuses on enhancing the efficiency and accuracy of MLLMs by refining the MLM techniques used, exploring new architectures, and integrating more diverse datasets. These efforts aim to create models that are not only more capable of understanding multimodal inputs but also more efficient in terms of computational resources [54].

3.1.3. Visual Question Answering (VQA) Pre-Training

In VQA pre-training, models are exposed to paired questions and images and must learn to infer cross-modal relationships, typically via cross-attention mechanisms [55].

In medical VQA, recent pre-training work combines unimodal and multimodal contrastive losses with retrieval-augmented methods to improve grounding in domain-specific visual and textual evidence [56].

However, challenges remain due to the limited availability of diverse multimodal datasets, especially in niche areas like medical VQA. This scarcity necessitates innovative approaches to data augmentation and transfer learning to ensure robust model performance across various applications [55].

3.1.4. Vision-And-Language Pre-Training (VLP)

Vision-and-Language Pre-training (VLP) combines objectives such as image–text matching, masked language or region modeling, word-region alignment, contrastive learning, and caption generation within a multimodal context, producing models with broad cross-modal reasoning capabilities [57].

Models like UNITER, ViLBERT, and OSCAR are prime examples of this multitask approach, but they instantiate the architecture differently: ViLBERT uses a two-stream co-attention design, whereas UNITER and OSCAR use single-stream transformer variants that jointly encode visual and textual tokens. These designs allow models to process and align visual and textual data, improving their performance across various multimodal tasks [57].

Beyond centralized pre-training, VLP objectives are also being adapted to privacy-sensitive or cross-institutional settings. Recent VLP work has addressed model and data heterogeneity in federated learning environments, particularly in specialized domains such as biomedical applications, where data cannot always be pooled into a single training corpus [58].

Table 6. Comparison of pre-training strategies for vision–language MLLMs.

Strategy	Method	Representative Models	Key Strength
Contrastive Learning	Align image–text pairs	CLIP, ALIGN, FLAVA	Zero-shot transfer
Masked LM/MLM	Predict masked tokens	VisualBERT, VL-BERT, UNITER	Joint cross-modal representation
VQA Pre-training	Answer image questions	ViLBERT, LXMERT	Visual reasoning
VLP (multi-task)	Image–text matching + MLM	OSCAR, ALBEF, BLIP	Comprehensive multimodal understanding
Generative (LM)	Predict next token	Flamingo, BLIP-2, LLaVA	Open-ended generation

summarizes the main pre-training strategies used for vision–language MLLMs.

3.2. Fine-Tuning for Specific Tasks

After pre-training, MLLMs are typically fine-tuned on task-specific data to adapt their general knowledge to the requirements of a target application. Fine-tuning adjusts the model’s parameters to improve performance on particular domains—such as image captioning, visual question answering, or cross-modal retrieval—while preserving the representations acquired during pre-training. Techniques such as instruction tuning and parameter-efficient methods (e.g., multiway adapters, LoRA) have been developed to reduce the computational cost of fine-tuning and improve adaptability across diverse tasks [59,60,61,62].

Table 7. Comparison of fine-tuning techniques for MLLMs.

Technique	Description	Param. Efficiency	Use Case
Full Fine-tuning	Update all parameters	Low	Large task-specific datasets
LoRA	Low-rank adaptation of weight matrices	High	General downstream tasks
QLoRA	Quantized LoRA (4-bit)	Very High	Memory-constrained settings
Adapter Tuning	Insert small adapter modules	High	Multi-task adaptation
Prefix Tuning	Prepend trainable prefix tokens	High	Prompt-sensitive tasks
Instruction Tuning	Fine-tune on NL instruction datasets	Medium	Interactive/assistant systems
Multitask Fine-tuning	Train on multiple tasks jointly	Medium	Related task groups

compares the main fine-tuning techniques used for MLLMs.

3.2.1. Task-Specific Datasets

Task-specific datasets provide the domain knowledge needed to adapt pre-trained MLLMs to particular applications such as image captioning, VQA, and cross-modal retrieval.

For instance, in image captioning, datasets like the Microsoft Common Objects in Context (MS COCO) are extensively used. MS COCO comprises over 330,000 images, each annotated with multiple captions, offering a rich resource for training models to generate descriptive textual representations of visual content. The diversity and volume of this dataset help models learn to associate visual features with corresponding textual descriptions, enhancing their ability to generate accurate and contextually appropriate captions.

In the realm of visual question answering, the VQA 2.0 dataset is a prominent resource. It contains over 1.1 million questions based on over 200,000 images, with each question designed to test the model’s ability to comprehend and reason about visual information in conjunction with textual queries. The dataset includes questions that require an understanding of object recognition, counting, and spatial reasoning, among other skills, thereby challenging models to develop a nuanced understanding of the interplay between visual and textual data.

Beyond these, specialized datasets cater to niche applications. For example, the TextVQA dataset focuses on questions that require reading and understanding text present within images, pushing models to integrate optical character recognition with visual reasoning. Similarly, the GQA dataset emphasizes compositional question answering, assessing a model’s ability to handle complex queries that involve multiple reasoning steps.

Dataset selection directly affects fine-tuning outcomes: annotation quality, sample diversity, and volume must align with the target domain to ensure strong generalization.

3.2.2. Learning Rate Scheduling and Optimization

Fine-tuning typically uses smaller learning rates than pre-training to preserve acquired representations while adapting to the target task.

In practice, stable fine-tuning depends less on a particular scheduler name than on matching the update scale to the frozen and trainable components. Common recipes use small learning rates, warmup followed by decay, or adaptive reductions when validation loss plateaus; AdamW remains common because decoupled weight decay helps regularize large models. More adaptive schemes, including gradient-based learning-rate tuning, can reduce manual tuning effort, but in MLLMs these choices should be evaluated through downstream stability, preservation of vision–language alignment, and compute cost [59,63].

3.2.3. Multitask Fine-Tuning

In certain scenarios, models are fine-tuned on multiple tasks simultaneously, a technique known as multitask learning; analogous multitask fine-tuning strategies have also been used to improve specialist LLMs such as code models [64]. This approach enhances the model’s ability to generalize across various related tasks [65,66]. For example, a model might be fine-tuned concurrently on image captioning and visual question answering datasets, allowing it to perform well across both contexts [65]. Multitask learning leverages shared features among tasks, which improves overall performance and efficiency [66].

However, multitask fine-tuning also presents challenges, such as task interference and increased computational demands. To address these, researchers have developed parameter-efficient fine-tuning (PEFT) techniques. Methods like Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA) adjust only a subset of model parameters, reducing computational costs while maintaining strong performance [65]. In multimodal settings, the same principle is useful when a model must share capacity across captioning, VQA, grounding, retrieval, and instruction-following objectives without overfitting to a single task family.

These studies show the potential of multitask fine-tuning in developing efficient, versatile models that can perform well in diverse tasks with fewer resource requirements.

3.2.4. Cross-Modal Tasks

Fine-tuning is essential for tasks that require the model to reason across modalities, such as cross-modal retrieval or referring expression comprehension, where the model must identify specific objects in an image based on a text description. The goal during this phase is to effectively align the visual and textual representations [67]. Recent studies have demonstrated that fine-tuning large pre-trained models for cross-modal tasks yields impressive results but can be computationally expensive [68]. To address this, parameter-efficient transfer learning techniques have been developed, which update only a subset of parameters to adapt pre-trained models to downstream tasks efficiently [69].

For example, the M2IST framework (Multimodal Interactive Side-Tuning) introduces side-tuning with a mixture of multimodal interactive side adapters, enabling better vision–language alignment and efficient fine-tuning for tasks like referring expression comprehension [67]. Additionally, methods such as LOUPE focus on learning fine-grained semantic alignment between visual regions and textual phrases, enhancing the model’s capacity for cross-modal tasks [68]. These approaches underscore the importance of fine-tuning in achieving effective cross-modal alignment, critical for integrating visual and textual information in complex tasks.

3.3. Few-Shot and Zero-Shot Learning in Multimodal Large Language Models

Few-shot and zero-shot learning have emerged as powerful capabilities of Multimodal Large Language Models (MLLMs), enabling them to generalize to new tasks with minimal or no task-specific data. This is particularly valuable when labeled datasets are scarce or expensive to curate. For instance, the Frozen model demonstrates that a pre-trained language model can be extended to multimodal tasks without updating its weights, effectively transferring few-shot learning abilities to settings involving both vision and language [70]. Similarly, Kosmos-1 is a Multimodal Large Language Model trained from scratch on web-scale multimodal corpora, including interleaved text and images, image–caption pairs, and text data, showcasing the model’s capacity to perform various tasks without gradient updates or fine-tuning [71]. These advancements highlight the potential of MLLMs in scenarios where traditional supervised learning is impractical, paving the way for more adaptable and efficient AI systems.

3.3.1. Few-Shot Learning

In few-shot learning, a model adapts to a new task from a small number of examples. For Multimodal Large Language Models (MLLMs), this adaptation may occur through in-context examples, light parameter updates, or limited fine-tuning after extensive pre-training. This capability is particularly advantageous in scenarios where labeled data is scarce or expensive to obtain.

The process of few-shot learning in MLLMs typically involves presenting the model with a few examples of the new task as demonstrations or adaptation data. These examples serve as a guide, enabling the model to infer the task’s structure and requirements. For instance, in image captioning, the model might be shown a few images along with their corresponding captions, using this limited information to generate captions for new, unseen images.

A notable example of this approach is the “Frozen” model, which integrates a pre-trained language model with a Vision Encoder. In this setup, the language model’s weights remain unchanged (“frozen”), while the Vision Encoder is trained to produce embeddings that the language model can interpret. This design allows the model to perform tasks such as visual question answering and image captioning with minimal task-specific data, effectively transferring the few-shot learning capabilities of language models to multimodal settings [70].

Few-shot learning is especially useful for niche tasks where only a limited amount of data is available. In such cases, traditional supervised learning methods, which require large datasets, are impractical. Few-shot learning enables models to generalize from a small number of examples, making it feasible to develop AI systems for specialized applications without extensive data collection and annotation.

However, few-shot learning also presents challenges. The model must generalize effectively from limited examples, which can be difficult if the new task differs significantly from those encountered during pre-training. Additionally, the quality of examples provided during prompting or adaptation is crucial; poor-quality examples can lead to suboptimal performance.

To address these challenges, researchers are exploring various strategies. One approach involves enhancing the pre-training phase by incorporating a diverse range of tasks and data modalities, equipping the model with a broader knowledge base to draw upon during few-shot learning. Another strategy focuses on developing more sophisticated prompting and adaptation techniques that can better leverage the limited examples available.

3.3.2. Zero-Shot Learning

Zero-shot learning refers to a model’s ability to perform tasks without having seen any examples of that task during training. In the context of Multimodal Large Language Models (MLLMs), this capability enables the model to generalize across tasks and domains by leveraging its understanding of the relationships between different modalities, such as text and images.

A prominent example of this is the CLIP (Contrastive Language–Image Pre-training) model, which is trained on a vast dataset of text–image pairs. By learning to associate textual descriptions with corresponding images, CLIP develops a rich multimodal representation that allows it to perform zero-shot image classification. This means that CLIP can assign labels to images it has never encountered before, simply by interpreting the textual descriptions associated with those labels. For instance, when presented with an image of a previously unseen object, CLIP can often classify it by matching the image to the most relevant textual label from its training data [7].

The success of zero-shot learning in MLLMs like CLIP is attributed to their ability to capture high-level semantic information across modalities. By aligning visual and textual representations in a shared embedding space, these models can transfer knowledge from one domain to another, facilitating tasks such as image retrieval, captioning, and classification without task-specific training data.

Recent research has further explored the potential of zero-shot learning in MLLMs. For example, the study “Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages” demonstrates that multilingual language models can serve as pivots for zero-shot multimodal learning across different languages. By leveraging a strong multilingual large language model, multimodal models pre-trained on English-only image–text data can generalize to other languages in a zero-shot manner, even surpassing models trained on image–text data in native languages [72].

Another study, “MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning,” investigates the enhancement of zero-shot performance on various unseen multimodal tasks through instruction tuning. The researchers fine-tuned a multimodal model on a diverse set of tasks and instructions, demonstrating strong zero-shot performance and reduced sensitivity to variations in instructions for each task [48].

These advancements highlight the growing importance of zero-shot learning in the development of versatile and efficient AI systems. By enabling models to perform tasks without task-specific training data, zero-shot learning reduces the reliance on large labeled datasets, making it particularly valuable in scenarios where data collection is challenging or impractical.

3.3.3. Transfer Learning

Few-shot and zero-shot learning are made possible by transfer learning, where knowledge gained from pre-training on one set of tasks is transferred to new, unseen tasks. This is particularly effective in MLLMs because they are trained on large, diverse multimodal datasets that cover a wide range of text and visual domains, allowing for strong generalization across tasks [72].

Figure 10 contrasts the zero-shot and few-shot paradigms described above.

3.4. Instruction Tuning for MLLMs

Instruction tuning is a newer technique that enhances the ability of MLLMs to follow human instructions across modalities. It involves fine-tuning the model using explicit instructions in natural language, enabling the model to perform a broader range of tasks with greater flexibility and accuracy [73].

3.4.1. Natural Language Instructions

Instruction tuning uses datasets where tasks are framed as natural-language commands rather than as fixed task labels. For instance, instead of only pairing an image with a caption target, the model may receive a prompt such as “Describe the image in detail.” This format teaches the model to interpret user intent while grounding its response in the visual input [73].

3.4.2. Multimodal Instruction Tuning

For vision–language MLLMs, instruction tuning pairs natural-language commands with images, documents, screenshots, or video frames. A task may include an image and an instruction such as “What is the person in the image doing?”, requiring the model to jointly parse the prompt and the visual evidence [73].

3.4.3. Improving Generalization

By presenting diverse tasks in a shared instruction format, instruction tuning can improve transfer to new prompts and task variants without extensive retraining. It also complements few-shot and zero-shot learning by making task intent explicit in natural language.

3.4.4. Applications of Instruction Tuning

Instruction-tuned MLLMs are particularly useful in interactive systems where users provide visual inputs together with natural-language requests. Examples include support agents that inspect screenshots, receipts, forms, product photos, or interface states; creative tools that revise images or storyboards from prompts; and educational systems that answer questions about diagrams or visual materials.

In these settings, accuracy depends on the coverage of the instruction-tuning data, the quality of visual grounding, and the availability of external retrieval or tool support [74].

3.4.5. Illustrative Overview of Instruction Tuning

Figure 11 provides a visual overview of this instruction-tuning workflow.

4. Applications of MLLMs in Vision–Language Tasks

This section surveys the major application areas of MLLMs in vision–language tasks, covering image captioning, visual question answering (VQA), visual storytelling, content creation, cross-modal retrieval, and accessibility.

Table 8. Key vision–language benchmark datasets.

Dataset	Task	Size	Key Features	Ref.
MS COCO	Captioning, Detection	330,000 images	Multi-caption annotations (5 per image)	[75]
Flickr30K	Captioning, Retrieval	31,000 images	5 captions per image, entity grounding	[76]
VQAv2	Visual QA	1.1 million questions	Balanced yes/no answer distribution	[24]
TextVQA	Text-in-image QA	28,000 images	Requires OCR + visual reasoning	[77]
GQA	Compositional QA	22 million questions	Multi-step spatial reasoning	[25]
OK-VQA	Knowledge-based QA	14,000 questions	Requires external world knowledge	[78]
NoCaps	Novel object captioning	15,000 images	Evaluates generalization to new objects	[79]
NLVR2	Visual reasoning	107,000 pairs	Sentence–image pair reasoning	[80]

provides an overview of the key benchmark datasets used to evaluate these applications, and

Table 9. Evaluation benchmarks for MLLMs: tasks, known weaknesses, and reporting recommendations.

Benchmark	Task Type	Evaluation Dimension	Known Weaknesses/Traps	Recommended Reporting
MMBench	General multimodal	Perception, reasoning, knowledge	Prompt-sensitive; leakage risk	Report prompt template; model + split version
SEED-Bench	General multimodal	12 evaluation dimensions	Skewed toward spatial tasks	Report all 12 dimensions separately
MME	Perception + cognition	Y/N + open-ended	Binary bias; ceiling effects	Report perception and cognition scores separately
MMMU	Multi-discipline	College-level knowledge	Requires domain expertise	Report per-discipline breakdown
HallusionBench	Hallucination	Visual grounding accuracy	Model-specific failure modes	Pair with POPE for cross-validation
POPE	Object hallucination	Precision/recall on objects	Adversarial vs. random split gap	Report all three splits (random, popular, adversarial)
VQAv2	Open-ended VQA	Accuracy on answer strings	Answer distribution bias	Use test-dev; avoid leakage
GQA	Compositional reasoning	Accuracy, consistency, validity	Data contamination risk	Report accuracy + consistency jointly

summarizes prominent evaluation benchmarks along with known limitations relevant to fair comparison.

How benchmark weaknesses affect performance interpretation. The known limitations catalogued in Table 9 are not merely methodological footnotes; they materially affect how reported scores should be interpreted and compared. VQAv2’s short-answer format, for example, rewards shallow pattern matching over genuine visual understanding: because the answer distribution is highly skewed (“yes”, “no”, and a handful of common nouns account for a large fraction of ground-truth answers), models can achieve deceptively high accuracy by learning answer distribution biases rather than truly grounding their responses in the image [24]. MMBench and SEED-Bench adopt a multiple-choice format, which inherently inflates scores relative to open-ended evaluation. A random baseline already achieves 25% on four-option questions, and models can exploit option-elimination heuristics that would not transfer to free-form generation settings, making cross-format comparisons misleading. POPE specifically targets hallucination, but its scope is limited to object existence (“Is there a dog in the image?”). It does not probe attribute hallucinations (“Is the car red?”) or relational hallucinations (“Is the cat on the table?”), so a strong POPE score should not be taken as evidence that a model is hallucination-free in the broader sense. MMMU tests expert-level, multi-discipline knowledge, yet its per-domain sample sizes are small, leading to high variance: a handful of lucky or unlucky guesses can shift a sub-domain score by several percentage points, making fine-grained comparisons between models unreliable without confidence intervals. Taken together, these weaknesses underscore a critical methodological point: no single benchmark captures true multimodal understanding. Researchers should report results across multiple complementary benchmarks—spanning open-ended and closed-form formats, perception and reasoning tasks, and both general and adversarial evaluation splits—to provide a more faithful picture of model capabilities and limitations.

Systemic risks in current evaluation paradigms. Beyond individual benchmark weaknesses, four systemic issues threaten the reliability of the entire MLLM evaluation ecosystem: (i) Data contamination. As benchmark datasets become standard training targets, the risk of train–test leakage grows. Web-scraped pre-training corpora may inadvertently contain benchmark questions and answers; contamination has been documented for text-only LLMs and likely affects multimodal models as well. Without rigorous decontamination protocols and held-out test splits that are never released publicly, reported improvements may partly reflect memorization rather than genuine capability gains. (ii) “Expert-level multiple choice” is not “expert-level reasoning”. A high MMMU score is evidence only that a model performs well on an expert-level multiple-choice question bank, not that it possesses expert-level reasoning in the open-ended sense. Three specific format artefacts must be acknowledged. First, option-order bias: Pezeshkpour and Hruschka [81] and Zheng et al. [82] show that large language models are strongly sensitive to the order in which multiple-choice options are presented—relabelling the correct answer from position A to position D can shift reported accuracy by 5–15 pp on a wide range of benchmarks, so a single-run MMMU score over-estimates robust accuracy. Second, elimination-strategy exploits: models can rule out implausible distractors using language priors alone and then guess among the remaining options, a test-taking strategy that does not reflect the reasoning a benchmark was intended to probe. Third, format generalization gap: real expert decision-making is open-ended, ambiguous, and requires quantifying uncertainty; MCQ scores measure selection from a fixed menu, which under-estimates the difficulty of real tasks and over-estimates the model’s transfer to them.

For these reasons, an MMMU score should be treated as a lower bound on how much the model would be wrong in an open-ended expert-decision setting, not as an upper bound. Reporting MCQ accuracy together with circular-permutation accuracy (averaging over all possible option orderings) and with a free-form generation variant would substantially reduce the interpretive gap; we recommend this as standard practice for future MLLM reasoning benchmarks. The same caveats apply to VQA-style multiple-choice evaluation more generally: strong MCQ scores do not predict reliable performance on real-world visual assistance tasks, where queries are unconstrained and failure consequences are more significant.

(iii) Absence of statistical rigour. When performance differences between models are 1–2 percentage points, as is common on MMBench, VQAv2, and GQA, these differences may not be statistically significant—yet papers routinely claim improvements without confidence intervals, significance tests, or multiple-run variance reporting. A concrete instance: Qwen2-VL (83.0) versus GPT-4o (83.4) on the public MMBench-Dev EN split ($N=1164$ unique questions) gives a per-model standard error of roughly $1.1$ pp near $83\%$ accuracy and a standard error of the difference of roughly $1.6$ pp under a simple binomial approximation, so the observed $0.4$ pp gap should not be interpreted as a reliable ranking.

Larger gaps between recent open-source models and older proprietary baselines are less likely to be explained by sampling noise alone, but they remain benchmark-level observations rather than definitive evidence of overall model superiority. This 1–2 pp noise floor is a property of the benchmarks themselves, and the current community practice of reporting un-caveated numerical rankings hides genuine noise as apparent progress. Readers are encouraged to apply the same back-of-the-envelope binomial standard error argument to any close comparison they encounter in the MLLM literature. (iv) Lack of stratified failure reporting. Current reporting aggregates accuracy into a single number, hiding structural failure patterns that are far more informative for researchers designing next-generation systems. Section 6.3.2 argues that object hallucinations should be disaggregated into “popular-object” (irreducible under the training distribution) and “adversarial-split” (reducible by architectural and data interventions) components; we further recommend that benchmark scores be reported alongside matched per-capability sub-scores (spatial reasoning, counting, fine-grained attribute, relational) so that improvements can be attributed to specific architectural changes.

4.1. Image Captioning and VQA

Image captioning and VQA, introduced earlier, remain two of the most mature MLLM application areas. Here we focus on recent advances and real-world deployments rather than restating foundational concepts.

Figure 12 summarizes the application-domain taxonomy used in this section.

4.1.1. Image Captioning: Overview and Advances

Image captioning refers to automatically generating textual descriptions for images by combining visual and linguistic processing. Early methods were limited to hand-crafted rules, while modern deep learning approaches, including MLLM-based systems, have substantially improved caption quality on standard datasets. By training on large image–text datasets such as MS COCO and Flickr, these models can generate richer and more context-aware captions than earlier rule-based or narrow supervised methods, although their outputs still require evaluation for grounding errors and hallucinations [83,84,85].

Key advances include the following techniques:

• OSCAR (Object-Semantics Aligned Pre-training): Enhances captioning by aligning object tags with textual descriptions during pre-training. This leads to better object recognition and semantically rich captions [84].
• VIVO (Visual Vocabulary Pre-training): Introduces a vocabulary of visual concepts, helping models describe objects that are underrepresented or absent in captioning training data [86].
• Dense Captioning: Generates region-specific captions for different parts of an image, supporting detailed image understanding and retrieval [85].
• Generative Adversarial Networks (GANs): Have been explored for refining caption fluency and coherence through adversarial training, although most recent captioning advances rely on transformer-based vision–language pre-training [87].
• Meta-Learning Approaches: Aim to help models adapt to new captioning or VQA tasks with limited data, especially when task-specific training examples are scarce [88].

4.1.2. Visual Question Answering (VQA): Overview and Advances

VQA is an interdisciplinary field requiring a system to answer questions posed about images by combining visual and textual reasoning [89]. Unlike image captioning, which describes an image holistically, VQA focuses on specific queries about an image’s content.

MLLMs have improved VQA performance by utilizing the same underlying multimodal architectures:

• MCAN (Multimodal Co-Attention Network): Uses co-attention mechanisms to fuse image and text features, resulting in improved understanding of image–question relationships [90].
• Knowledge-Enhanced VQA Models: Incorporate external knowledge graphs for commonsense reasoning, improving performance on complex VQA tasks [91].

4.1.3. Applications of Image Captioning and VQA

The advancements in MLLMs have expanded their applications across diverse domains, offering solutions to various real-world problems by bridging the gap between visual data and language.

• Assistive Technologies Image captioning and VQA can support assistive technologies for blind and low-vision users by turning visual inputs into spoken descriptions, OCR results, or question-driven responses. For instance, applications such as Microsoft Seeing AI use AI-powered visual recognition and narration to describe objects, people, scenes, and text in a user’s environment. VQA-style interaction can further allow users to ask targeted questions such as “What is the name on the sign?” or “Is there anyone near me?”, but these systems should be understood as assistive aids whose outputs require error-tolerant interaction design rather than as perfectly reliable perception systems [92].
• Autonomous Systems and Vehicles: In autonomous-driving research, captioning and VQA-style modules can provide interpretable summaries of road conditions, obstacles, pedestrians, and traffic signs as auxiliary inputs for monitoring, debugging, or human–vehicle interaction. They should not be framed as stand-alone safety controllers: deployed driving stacks still require specialized perception, prediction, planning, redundancy, and verification before visual-language outputs can influence safety-critical decisions.
• Medical Imaging and Healthcare: In medical imaging, research prototypes have demonstrated that MLLM-based captioning systems can draft preliminary descriptive reports from X-rays, CT scans, and MRIs [56]. Such systems are being investigated as a way to support—rather than replace—radiologists, with the goal of accelerating report drafting and surfacing candidate findings for expert review. We emphasize, however, that the published evidence comes from retrospective evaluations on benchmark datasets; current systems have not been prospectively validated in clinical workflows, their error characteristics under distribution shift are poorly understood, and claims of “high accuracy” or workload reduction must therefore be interpreted as early research findings rather than properties of deployed medical devices. Research-stage tools in this space (e.g., CaptionHealth) operate in this same pre-deployment regime. VQA models are similarly being explored for question-driven interrogation of medical images (“What abnormalities are present?”), again with the aim of supporting clinician decision-making rather than functioning as autonomous diagnostic agents.
• Content Moderation and Search Engines: On social media and search platforms, multimodal classifiers and captioning models can help tag images, detect text in images, and prioritize potentially objectionable content for review. VQA-style querying may further support human moderators by asking targeted questions such as “Does this image contain violence?” or “Is there inappropriate text in the background?”, but moderation decisions still depend on policy context, human review, and error monitoring.

Figure 13 summarizes representative domains in which captioning and VQA are applied.

4.2. Visual Storytelling and Scene Understanding

Visual storytelling and scene understanding extend captioning to coherent, multi-image narratives and structured scene representations, with applications in autonomous driving, interactive media, and 3D modeling [93,94].

4.2.1. Technologies for Visual Storytelling and Scene Understanding

Visual storytelling, defined as generating coherent narratives from sequences of images or videos, has evolved from simpler object recognition methods to models that combine visual semantics, temporal context, and language generation. Early models often used scene graphs to capture relationships between objects but struggled with narrative coherence. The introduction of MLLMs has improved the ability to connect visual relationships with narrative language, although long-horizon consistency remains difficult.

For instance, the Multidimensional Semantic Augmented Network provides a method for merging scene, object, and action semantics, supporting richer narrative construction [93]. These models go beyond static object descriptions by modeling how elements interact over time to form stories [93]. In another example, Kosmos-1 leverages cross-modal knowledge transfer between vision and language, producing descriptions that can incorporate broader contextual cues [71]. By drawing on both visual and textual data, these models support more fluent storytelling than earlier systems, while still requiring checks for temporal drift and unsupported inferences.

Scene understanding, another key task, involves comprehending the spatial and relational structure of objects within a scene. Advanced models like Scene-LLM integrate 3D visual data with textual descriptions, allowing for high-level reasoning about spatial relationships and object interactions. Hybrid 3D feature representations, which combine global scene-level information with local object-centric details, can support reasoning about dynamic environments [95]. This is particularly relevant to research on robotics and autonomous driving, where scene understanding must ultimately be paired with real-time perception, calibration, and safety validation before deployment.

4.2.2. Applications

MLLMs are being explored in several domains where their ability to generate, understand, and manipulate multimodal data can provide useful support.

In the entertainment industry, MLLMs can assist with drafting narratives for films, games, and other media. By analyzing sequences of images or video, these models can propose storylines that respond to characters’ actions or environmental changes. For instance, games can use AI-assisted narrative components that adapt to player decisions, creating more interactive experiences while still requiring authorial control and consistency checks [96]. Similarly, in content generation for streaming platforms, MLLMs can help draft personalized narrative variants based on viewer preferences [71].

In the realm of autonomous driving, scene understanding is vital for perception and planning research. MLLMs like OmniDrive aim to help autonomous systems interpret 3D environments by analyzing traffic situations, detecting potential hazards, and reasoning about the actions of other vehicles [97]. These models are trained on large datasets that include diverse driving scenarios, but their generalization to complex urban settings remains an empirical question. By modeling spatial and temporal relationships between objects, such systems may support safer decision-making modules, provided they are integrated with specialized driving stacks and rigorous validation [97].

For augmented reality (AR) and interactive storytelling, MLLMs offer the potential to create narratives or instructions that respond to user interactions with their environment. In AR applications, these models can analyze the physical space around the user and generate context-aware guidance. For example, in an AR-based learning environment, an MLLM might draft contextual stories or educational content based on the objects and scenes detected in a classroom or outdoor setting [98]. This ability to blend physical and digital cues can support education, entertainment, and marketing applications, but the generated content still requires grounding and safety checks.

In robotics and embodied AI, MLLMs like Scene-LLM can support robots’ abilities to navigate and interact with their environments. Robots equipped with scene understanding models may perform bounded tasks in complex, unstructured environments, such as warehouses or hospitals. These systems use 3D scene representations to understand their surroundings, make decisions about where to move, and interact with objects based on their spatial relationships [95]. In healthcare-related settings, such capabilities should be viewed as research-stage support for navigation or logistics rather than as validated patient-care autonomy.

In content generation, MLLMs have started to enable platforms that provide automatic captioning and summarization of images and videos. Applications such as digital marketing, social media content creation, and journalism can benefit from draft narratives generated from visual content [99]. Such automation may reduce drafting time and support personalization, but editorial review remains necessary for factuality, tone, and audience fit.

Future directions for MLLMs in these applications include improving real-time processing, enhancing model interpretability, and reducing the computational cost of deploying these large models in dynamic, resource-constrained environments. Addressing these challenges will be crucial for expanding MLLMs’ use in more diverse, real-world scenarios [100].

4.3. MLLM Applications in Content Creation and Editing

MLLMs enable cross-modal content generation and editing across text, image, video, and audio [101].

4.3.1. Technologies Behind MLLM in Content Creation

Current multimodal content systems combine several technological families, including transformer language models, diffusion and latent-diffusion image/video generators, multimodal encoders, and vision–language instruction tuning. GANs remain historically important for image synthesis, but most recent text-to-image and text-to-video systems discussed here rely primarily on diffusion- or transformer-based generation. These technologies enable systems to understand and generate multimodal content for entertainment, marketing, journalism, and education.

The integration of GPT-style text generation with DALL-E and other image generation systems allows creators to draft multimedia pieces from natural-language prompts [101]. Models such as Video-LLaMA combine visual and textual inputs for video understanding and dialogue, while separate video-generation systems support editing and synthesis workflows. Self-supervised learning and multimodal training datasets help these models learn relationships between text, images, and videos, making them useful tools for content creation when paired with human review [71].

4.3.2. Applications in Content Creation and Editing

The following are the key applications of MLLMs in the field of content creation and editing:

• Multimodal Content Generation: MLLMs and adjacent generative systems allow artists and creators to draft images, text, and video content from simple inputs. These models can interpret text prompts to create visual art, design characters, or prototype environments for video games and films. This application supports multimedia storytelling, where different types of content can be integrated into a common creative workflow [101,102]. MLLMs are also used in marketing and social media to help draft personalized content from user preferences and trends [96].
• Real-Time Video and Image Editing: MLLMs and multimodal editing systems are advancing video and image editing by enabling natural-language or sketch-guided modifications. Systems like ExpressEdit allow users to sketch over a video frame or provide verbal commands to alter specific elements in a scene [103]. Related tools can support color correction, object tracking, and scene enhancement, although latency and edit fidelity vary by model, hardware, and video length [104].
• Multimodal Script and Article Drafting: Text-only drafting becomes multimodal when paired with visual inputs such as storyboards, reference images, video outlines, product photographs, or design briefs. In these workflows, MLLMs and adjacent tools can produce structured drafts that human editors fact-check, revise, and align with the visual source material [105,106].
• Collaborative Content Creation: Collaborative content development tools powered by MLLMs allow teams to simultaneously work on different aspects of a project, from visual elements to text descriptions. Cloud-based platforms enable real-time collaboration, ensuring that all contributors are working on the most up-to-date version of the project [107]. In interactive design, tools like U-CREATE help creators develop augmented reality experiences and location-based services, streamlining the authoring process [108].
• Mobile Multimedia Editing: Mobile applications integrating MLLMs have made content creation accessible to a broader audience. These apps allow users to edit and generate multimedia content on their devices using intuitive commands and gestures. By automating some technical aspects of editing, MLLM-powered mobile apps can help social media creators and small businesses produce usable drafts without advanced editing skills [109].
• Content Repurposing and Multilingual Adaptation: MLLMs can assist in repurposing content for different platforms and adapting it to various languages. Whether reformatting a blog post for social media or translating a promotional video into multiple languages, these systems can draft adapted content that human reviewers then check for meaning, tone, and cultural nuance [105]. For global marketing campaigns, MLLMs can support localization workflows, but they do not remove the need for regional review [110].
• Creative Personalization: MLLMs can support content personalization based on user behavior and preferences. For example, recommendation or marketing systems may use multimodal models to analyze viewing habits, likes, and social trends [110]. This application is useful in e-commerce, where product descriptions and promotional materials can be adapted to user segments, subject to privacy and fairness constraints [102].
• Dynamic Multimedia Creation: MLLMs can dynamically generate multimedia presentations by combining textual, visual, and audio elements. This capability is particularly useful in industries like education and training, where interactive and adaptive content is essential for engaging users. In educational technology, MLLMs assist in creating lesson plans, video tutorials, and interactive learning modules that adapt to the learner’s pace and preferences [106].

MLLMs and adjacent generative systems have become important tools for content creation and editing by automating parts of drafting, transformation, and revision. From multimodal content generation to collaborative editing, these models can integrate text, image, and video into shared workflows, while final quality still depends on human direction, factual review, and platform-specific constraints. Continued progress is likely to improve personalized content, interactive editing, and multilingual adaptation in industries ranging from entertainment to marketing [101,104].

4.4. MLLM Applications in Cross-Modal Retrieval and Search

Cross-modal retrieval—finding information in one modality based on a query from another—has been transformed by MLLM architectures that learn shared vision–language embedding spaces (see Section 2).

4.4.1. Technological Foundations of Cross-Modal Retrieval

Cross-modal retrieval relies on a combination of multimodal transformers, dual-encoder architectures, and self-supervised or contrastive learning to manage relationships between different data types. For instance, CLIP-style Vision–Language Models map images and text into a shared embedding space, enabling efficient image–text retrieval [93,111]. Generative systems such as DALL-E address a related but different problem: producing new images from text prompts rather than retrieving existing items. Other architectures, such as cross-attention mechanisms, can then refine the relationships between elements of different modalities.

Multimodal assistants such as GPT-4V extend retrieval-centered workflows by interpreting retrieved images or documents and generating textual explanations, summaries, or follow-up queries. In such systems, retrieval and generation are distinct stages: the retrieval module supplies candidate evidence, while the multimodal model reasons over that evidence and produces the final response [112].

4.4.2. Applications of MLLMs in Cross-Modal Retrieval and Search

• Image–Text Retrieval: One of the most widespread applications of MLLMs in cross-modal retrieval is the search for images based on textual queries or vice versa. This has applications in areas such as e-commerce, where users can search for products using images or text descriptions. Models like CLIP map images and text into the same latent space, making it possible to retrieve images that semantically match the input text [93]. Advanced models also allow retrieval of more abstract visual concepts, such as emotion or style, based on text queries.
• Video–Audio–Text Retrieval: Cross-modal retrieval extends to the video and audio domains as well, with applications in multimedia search engines and content recommendation systems. For example, users can retrieve relevant video clips by providing a text description or even an audio snippet. Systems like Video-LLaMA and Speech2Text search can match video content with textual or spoken queries, allowing for accurate retrieval in large multimedia databases [112,113]. This has significant implications for media platforms, allowing users to discover video content based on both audio and text inputs.
• Generation from Cross-Modal Prompts: Generative models complement retrieval by enabling systems to create new content from user queries. For example, DALL-E and similar models allow users to generate images from detailed textual descriptions, while other systems generate music or video content from text-based prompts. This generative paradigm should be distinguished from retrieval: instead of finding existing content, users synthesize a new artifact that still requires evaluation for fidelity, originality, and safety [1,114].
• Multi-Lingual and Cross-Lingual Retrieval: MLLMs can support cross-lingual retrieval when their text, speech, and visual components are trained or aligned on multilingual data. Such systems may retrieve multimedia content across languages directly through multilingual embeddings or indirectly through translation and reranking. This is useful in global applications, where users may search in one language and receive content from another [93,112]. For instance, a user could search using an English query and retrieve relevant French or Spanish media files, provided the system has adequate multilingual coverage.
• Cross-Modal Music Retrieval: Cross-modal retrieval extends beyond text, images, and video into music as well. Researchers have developed systems that enable users to find musical pieces based on descriptions of melodies, moods, or even visual stimuli such as album covers or sheet music [114]. These systems can enhance music recommendation platforms, allowing users to find music across various modalities, leading to more immersive listening experiences.
• Lecture Video Retrieval: Cross-modal retrieval systems are becoming increasingly important in educational technology. For example, specialized multimodal lecture models have been applied to index and retrieve lecture videos based on both spoken content and text on slides [113]. This allows students to find relevant portions of lecture videos by searching with either keywords or topics, improving content retrieval in educational platforms.
• Content-Based Image Retrieval in Medical Domains: In the medical domain, cross-modal retrieval is being explored as a way to link textual descriptions (e.g., symptoms or diagnoses) with relevant medical images such as X-rays or MRIs. MLLMs trained on multimodal medical datasets have shown the potential to support—rather than replace—clinicians’ diagnostic workflows by retrieving and analyzing medical data across text and images. It is important to stress that existing systems such as RAMM [56] and the multimodal medical VQA framework of [55] are research prototypes evaluated on curated benchmark datasets; they have not yet been clinically validated and are therefore best understood as early-stage capabilities rather than deployed systems.
• Interactive Search in AR/VR: Augmented reality (AR) and virtual reality (VR) applications are using cross-modal retrieval to enable more interactive and immersive experiences. Users can search for virtual objects, spaces, or experiences by describing them with words or gestures, and MLLMs process these inputs to retrieve or generate corresponding virtual environments or objects. This is particularly useful in gaming, training simulations, and virtual tourism [93].

Despite the progress in cross-modal retrieval, several challenges remain. One major challenge is handling domain-specific data such as medical or legal information, where cross-modal models must be validated on specialized terminology, images, and workflows rather than assumed to generalize from web data. Another challenge is the scalability of cross-modal retrieval in latency-sensitive systems, where response time depends on embeddings, index structure, reranking, network overhead, and generation. Lastly, improving the accuracy and contextual relevance of cross-modal retrieval results, especially in complex scenarios involving abstract concepts or multilingual inputs, remains an ongoing area of research [101,112].

Future directions include the enhancement of personalized retrieval systems that adapt to user preferences and the exploration of cross-modal reasoning capabilities, where models not only retrieve content but also infer and reason across modalities [1]. Additionally, integrating real-time data processing for AR and VR applications could open new possibilities in immersive and interactive cross-modal experiences.

MLLMs have advanced cross-modal retrieval and search by offering more flexible ways to embed, retrieve, rerank, and explain multimodal data. From image–text search to generation from multimodal prompts, these systems are changing how users interact with multimedia content. Their impact in fields such as healthcare, education, and entertainment will depend on domain-specific validation, retrieval quality, and safeguards around privacy, bias, and factuality.

4.5. MLLMs in Enhancing Accessibility for People with Disabilities

MLLMs’ cross-modal capabilities can support accessibility by bridging vision, language, and audio for users with disabilities, provided that systems are designed around reliability, latency, privacy, and user control [71].

4.5.1. Technological Foundations

MLLMs, such as VIAssist and Kosmos-1, utilize cross-modal understanding to process inputs from different modalities, making them relevant to accessibility research. Models like VIAssist are trained to recognize objects, generate descriptive text, and provide contextual answers to questions based on visual inputs [100]. These models combine techniques such as transformers, visual grounding, and natural language processing to create an interaction layer between users and digital environments.

Moreover, cross-modal retrieval frameworks like those used in Kosmos-1 can support text generation from images or video streams, facilitating tasks such as captioning, OCR, and object recognition [112]. Whether these capabilities are usable in real time depends on model size, hardware, network conditions, and the tolerance of the accessibility task for delay or error.

4.5.2. Applications of MLLMs in Accessibility

• Text-to-Speech and Speech-to-Text Systems: Multimodal systems can improve speech recognition and generation by combining audio with visual or textual context. Models trained on large speech and text datasets can transcribe spoken content into text and synthesize speech from text, but accuracy varies with accent, noise, domain vocabulary, and speaker overlap. Real-time subtitles can improve access to live presentations, videos, and other media content, while still requiring fallback mechanisms when latency or transcription errors occur [115].
• Visual Assistance for the Blind and Visually Impaired: MLLMs are being leveraged to build systems that describe the surrounding environment to blind and low-vision users. Applications like VIAssist use cameras to capture images and then provide narration or detailed descriptions of objects, people, or text present in the environment [100]. These models can identify relevant aspects of scenes and summarize key details, but they may omit hazards or hallucinate objects, so navigation-critical uses require conservative interface design and user safeguards. This technology also extends to recognizing text from images, making documents and signs more accessible [71].
• Object Detection and Recognition: MLLMs enable object recognition systems that may benefit blind, low-vision, and deaf or hard-of-hearing users. For instance, wearable devices equipped with cameras can use multimodal recognition models to identify objects and provide audio descriptions when latency and power constraints permit [93]. For deaf or hard-of-hearing users, visual object recognition can support lip-reading assistance or visual cues, but such systems remain sensitive to occlusion, lighting, and user-specific variation [96].
• Assistive Text Summarization and Captioning: For people with combined vision and hearing loss, captioning and summarization of digital content can be valuable when paired with accessible output formats. MLLMs can generate captions or summaries for events, meetings, and video content, making it easier for users to stay informed when outputs are accurate and timely [116]. They can also summarize books, articles, and documents before conversion into audio, braille, or tactile formats, but the summaries must preserve essential information and avoid unsupported simplification [112].
• Real-Time Sign Language Translation: Another important application for deaf or hard-of-hearing users is sign language recognition and translation. MLLMs trained on multimodal datasets that include sign-language video can support translation from signs into text or spoken language, facilitating communication between sign-language users and those unfamiliar with a given sign language [71]. These models must recognize hand gestures, facial expressions, body movement, signer identity, and regional variation, so current systems should be described as promising research and assistive tools rather than as universally reliable translators [71].
• Personalized Accessibility Tools: With the advancement of MLLMs, personalized accessibility tools are becoming more feasible. These tools can learn individual user preferences and adapt their output accordingly, whether by adjusting speech patterns, text formatting, or the detail level of visual descriptions [71]. Personalized accessibility can improve usability, but it also raises privacy and consent questions because adaptation often depends on sensitive user data.

While MLLMs offer numerous advantages for accessibility, challenges remain. One of the main issues is ensuring that the models can generalize effectively across diverse environments and users. Models trained on specific datasets may struggle in real-world applications where the visual or auditory environment differs significantly from the training data. Additionally, the computational cost of running MLLMs in real-time for assistive technologies, particularly on mobile devices, remains high, limiting their widespread use [71].

Future research will likely focus on improving the accuracy and calibration of real-time systems, expanding datasets to cover a broader range of users and environments, and optimizing the models for lower-power devices. Moreover, advancements in multimodal reasoning, where models infer meaning and context across diverse input types, may further enhance their effectiveness in accessibility technologies when paired with careful evaluation [115].

Multimodal Large Language Models represent a promising approach to enhancing accessibility for people with disabilities. From sign-language support to visual assistance for blind and low-vision users, these technologies can reduce some communication and information-access barriers. Their practical value will depend on reliability under diverse conditions, transparent failure modes, privacy-preserving personalization, and interfaces that keep users in control [71,93,100].

5. Case Studies: MLLMs and the Surrounding Multimodal Ecosystem

5.1. Purpose and Scope of the Case Studies

The systems examined in this section differ substantially in how closely they align with our survey’s central focus on architecturally Multimodal Large Language Models. To avoid conflating fundamentally different classes of systems—and to respond to the concern that a flat enumeration of commercial tools can read as a product catalogue rather than as technical analysis—we explicitly organize the discussion into three tiers:

• Tier A—Architecturally Multimodal MLLMs: systems whose architecture jointly reasons over visual and textual (or, in some cases, audio or action) tokens through a shared language model backbone. This tier is the primary object of our survey and includes general-purpose assistants such as GPT-4V, GPT-4o, Claude 3, and Gemini, together with the vision–language–action models used in embodied AI (RT-2, SayCan/PaLM-SayCan, ManipLLM, PaLM-E, VoxPoser, and LLM-VLMaps).
• Tier B—Adjacent Generative Systems: text-to-image and text-to-video models (Midjourney, DALL-E 3, Stable Diffusion, Imagen, Flux.1, Runway, Pika, Kling, ModelScope, and related platforms). These are mostly diffusion-based generators rather than multimodal-understanding MLLMs, but they share encoders, tokenizers, and evaluation protocols with Tier A, and are therefore included as ecosystem context.
• Tier C—Supporting Infrastructure and Retrieval Systems: vector databases, RAG frameworks, and visual or cross-modal search engines (Pinecone, Chroma, FAISS, Vespa, Weaviate, Qdrant, LangChain, the OpenAI Retrieval Plugin reference implementation [117], Speculative RAG, Google Lens, Bing Visual Search, You.com, Perplexity, MARVEL, InteR, and SPECTER). These are not MLLMs in their own right but are the components on which many multimodal applications depend, so we document them for completeness while making their role explicit.

These systems are discussed as surveyed infrastructure rather than software used to generate new experimental results; no specific software versions were installed, executed, or evaluated in this review.

For each subsection that follows, a short Scope note makes clear which tier is being discussed so that readers can weigh the technical claims accordingly. We intentionally exclude general-purpose text-only coding assistants such as GitHub Copilot and Amazon CodeWhisperer, which fall outside the vision–language scope stated in the abstract.

Table 10. Bridge from the task-level analysis in Section 4 to the system-level case studies in this section.

Section 4 Principle	Case Study System	How the System Extends the Principle	Deployment Limitation
Cross-modal retrieval	Google Lens, Bing Visual Search, MARVEL	Adds OCR, object/entity linking, ANN retrieval, reranking, and web or knowledge-graph grounding on top of CLIP-like shared embedding principles.	Product internals are partly closed; retrieval quality depends on corpus freshness and indexing.
Image/video generation	DALL-E 3, Midjourney, Stable Diffusion, Runway, Kling	Converts task-level generation into deployed prompt rewriting, diffusion, safety filtering, and iterative editing workflows.	Often closed-source or commercial; benchmark protocols are inconsistent.
Evidence-grounded generation	LangChain, FAISS, Pinecone, Speculative RAG	Combines cross-modal retrieval with external evidence selection and LLM generation to reduce knowledge staleness and hallucination.	End-to-end quality depends on embedding, retrieval recall, reranking, and generation, not on any single component.
Multimodal dialogue	GPT-4V/4o, Claude 3, Gemini	Integrates visual encoders, connector modules, instruction tuning, and safety post-training into interactive assistants.	Proprietary systems disclose limited architectural, data, and serving details.
Embodied/agent tasks	RT-2, PaLM-E, SayCan, VoxPoser	Extends vision–language reasoning into action selection, affordance grounding, spatial planning, and robot control.	Physical deployment requires safety constraints, calibration, and real-time sensory feedback.

bridges the task-level analysis to the system-level case studies that follow.

Table 11. Classification of prominent multimodal systems by core technical paradigm and primary application domain, as covered in the case studies of this section.

Model/System	Core Paradigm	Primary Domain
Midjourney, DALL-E 3	Text-to-image generation	Creative generation
Stable Diffusion	Open-weight latent diffusion	Research & art
GPT-4V/4o, Claude 3, Gemini	Multimodal assistant models	Multimodal assistants
Google Lens, MARVEL	Dense visual retrieval	Search & e-commerce
BLIP-2, LLaVA	VLP + instruction tuning	VQA & captioning

classifies the systems examined in this section by their primary technical paradigm and application domain.

These case studies examine how MLLMs and adjacent multimodal systems are deployed across creative, scientific, and commercial domains. For each application area we analyze the underlying technical approach, document observed failure modes, and distill practical lessons. Key technical developments that inform these case studies include hierarchical text-conditional image generation with CLIP latents [118], latent diffusion models [119], and photorealistic text-to-image synthesis with deep language understanding [120].

A note on the evidentiary basis of these case studies. Several of the systems we discuss, particularly closed commercial products such as Midjourney, Kling AI, or managed vector-database services, do not have peer-reviewed publications or open technical specifications. For these systems, we cite product documentation and combine it, where possible, with peer-reviewed technical work on the underlying algorithmic family (e.g., diffusion models, approximate nearest-neighbour indexing) or with independent external evaluations such as Borji’s comparative study of text-to-image generators [121]. Where no such independent evidence exists, we restrict our claims to architectural description and intended use, and we do not report benchmark numbers that are not independently verifiable.

5.2. Case Studies

5.2.1. Image Generation

• Scope: Tier B—adjacent generative systems. The systems below are text-to-image diffusion models rather than multimodal-understanding MLLMs, but they share encoders, tokenizers, and evaluation protocols with the Tier A systems discussed later.

Text-to-image generation has advanced rapidly through diffusion-based and autoregressive approaches. We first examine three widely used systems, then briefly cover Imagen and Flux.1 before summarizing characteristic failure modes.

The first three systems represent distinct design philosophies: Midjourney [122] is a closed, proprietary diffusion system optimized for aesthetic quality and stylistic coherence, widely adopted by artists despite offering no architectural transparency; independent comparisons with other text-to-image systems are limited to black-box user studies such as [121]. DALL-E 3 [123] improves prompt following through a text-rewriting and recaptioning pipeline: user prompts can be expanded into more detailed descriptions before image generation. This yields strong prompt fidelity in product use, but the full generation architecture and training details are not publicly disclosed, and the tight integration with ChatGPT can reduce user control over exact wording. Stable Diffusion [124] operates in a compressed latent space via a variational autoencoder, substantially lowering computational cost and enabling open-weight distribution on consumer hardware; later variants such as Stable Diffusion XL improve image quality and resolution, but reported benchmark numbers are not directly comparable unless evaluation prompts, samplers, resolutions, and filtering protocols are matched.

These three systems highlight a fundamental design trade-off: open-weight reproducibility and community extensibility (Stable Diffusion) versus closed, commercially tuned aesthetic optimization (Midjourney) versus LLM-integrated prompt understanding (DALL-E 3). Because closed systems rarely disclose training data, inference settings, or benchmark protocols, human-preference and prompt-alignment comparisons should be interpreted cautiously. Publicly comparable FID-style evaluations across major systems remain limited because prompts, samplers, resolutions, and filtering protocols differ, illustrating a broader challenge in the field: the absence of a unified, open evaluation protocol for generative image quality.

Imagen and Flux.1

Google’s Imagen [120] employs a cascaded diffusion architecture: a base model generates a $64\times 64$ image conditioned on T5-XXL text embeddings, followed by two super-resolution stages that upscale to $1024\times 1024$. This cascaded design achieves strong photorealistic detail in the original report but remains closed-source. Flux.1 [125] is described as a rectified-flow transformer family for text-to-image generation, with some variants distributed as open weights and others offered through hosted or commercial channels. Both systems limit full reproducibility because training data, serving configuration, and evaluation protocols are only partially disclosed.

Despite their capabilities, diffusion-based image generation models exhibit a number of well-documented failure modes. Text rendering within generated images remains unreliable: models frequently produce misspelled or visually distorted characters, limiting their utility for infographics and document generation [126]. Fine-grained anatomical details, particularly human hands and fingers, are often rendered with incorrect counts or fused joints [119]. Prompt adherence degrades when instructions involve negation, precise spatial arrangements, or multiple interacting objects, causing models to omit or conflate specified elements. Training data memorization poses both privacy and copyright risks, as diffusion models have been shown to reproduce near-identical copies of training images under certain prompts [127]. Finally, all current systems require prompt engineering expertise to achieve reliable results, which limits accessibility for non-specialist users.

Table 12. Comparison of representative image-generation, visual-search, and cross-modal retrieval systems.

Model	Type	Core Technique	Key Strength	Notable Limitation	Access
Midjourney	Text-to-img	Diffusion (proprietary)	Artistic quality; creative style	Closed-source; limited control	Discord
DALL-E 3	Text-to-img	Closed generation pipeline; prompt rewriting	High prompt fidelity; ChatGPT-integrated	Usage cost; content restrictions	OpenAI API
Stable Diffusion	Text-to-img	Latent diffusion	Open weights; large community ecosystem	Needs tuning; safety filtering	Open weights
Imagen	Text-to-img	Cascaded diffusion	Photorealistic detail	Closed source; limited reproduction	Google
Flux.1	Text-to-img	Rectified-flow transformer	High-fidelity image generation	Partial disclosure; mixed access	API/open weights
Google Lens	Visual search	CNN + Transformer	Real-world object/text recognition	Retrieval only; no generation	Mobile/Web
MARVEL	Cross-modal retrieval	Dense retrieval	Dense multimodal retrieval	Retrieval only; no generation	Research

compares the representative image-generation, visual-search, and retrieval systems discussed in this subsection.

5.2.2. Search and Information Retrieval

• Scope: Tier C—retrieval and search infrastructure. The systems surveyed below combine classical information retrieval with vision and language models; most are not themselves end-to-end MLLMs but rather components of multimodal search pipelines.

Section 4.4 introduced cross-modal retrieval as a task-level principle: images and text are mapped into a shared semantic space so that one modality can retrieve the other. Deployed visual search systems extend this principle into a multi-stage pipeline rather than simply applying CLIP-style similarity search. A typical pipeline contains (i) visual encoding and region proposal, (ii) OCR and text-in-image extraction, (iii) object and entity linking, (iv) dense embedding search over products, landmarks, or web images, (v) ANN indexing and filtering, (vi) cross-modal reranking, and (vii) grounding against web pages, knowledge graphs, or shopping catalogues. The case study question is therefore not whether Google Lens or Bing Visual Search uses cross-modal retrieval, but how these systems operationalize and extend the retrieval principles introduced in Section 4.4.

Google Lens and Bing Visual Search. Google Lens [128] and Bing Visual Search [129] are best understood as deployed visual search stacks. Their user-facing function—identifying objects, text, products, landmarks, or similar images from a camera input—depends on the same shared-representation idea as CLIP, but adds OCR, object detection, region-level search, entity disambiguation, and web-scale reranking. This is the technical link to Section 4.4: CLIP-style embeddings supply candidate semantic matches, while product databases, maps, OCR indexes, and web-ranking signals turn those candidates into actionable results. The limitation is that the core retrieval index and ranking logic are closed, so these systems demonstrate deployment integration more than reproducible MLLM architecture.

You.com and Perplexity. You.com [130] and Perplexity [131] illustrate a different extension: retrieval is coupled to conversational answer synthesis. Their primary contribution is not a new visual encoder, but the orchestration of web retrieval, source selection, summarization, and natural language response generation. In multimodal settings, this pattern becomes a bridge from Section 4.4’s cross-modal retrieval to RAG: the system retrieves external evidence and then asks a language model to synthesize the result, making grounding, citation quality, freshness, and source ranking central deployment variables.

MARVEL

MARVEL (Multimodal Dense Retrieval Model for Vision–Language Understanding) [132] illustrates dense retrieval for vision–language tasks. It retrieves relevant documents, images, or other data points from large datasets using visual and textual inputs. Its core contribution is to align these modalities in a dense retrieval space, enabling queries that combine image content with natural-language context.

In e-commerce, for example, users may search by uploading a product image and receiving visually and semantically related results. The same retrieval pattern can support education, research, and content discovery when the corpus and indexing pipeline provide appropriate multimodal evidence.

InteR

InteR (Interactive Retrieval) [133] is a framework that focuses on enhancing the synergy between traditional search engines and large language models (LLMs). By creating a feedback loop between search results and LLMs, InteR ensures that search engines not only retrieve relevant content but also present it in a format that is easily understood by users. InteR leverages LLMs to refine and summarize search results, presenting more focused and actionable information.

The framework is particularly useful in contexts where search results need to be distilled into a concise, clear format, such as in legal research, academic work, or medical information retrieval. InteR’s combination of structured search capabilities and LLM-based summaries allows users to quickly find and comprehend relevant information without being overwhelmed by excessive details or irrelevant results.

Semantic Scholar’s SPECTER

SPECTER, developed by Semantic Scholar, is a scientific paper embedding model designed to enhance academic search [134]. SPECTER helps researchers find relevant academic papers by creating high-quality document embeddings that capture the semantic content of research articles. Unlike traditional keyword-based search, SPECTER uses these embeddings to connect papers based on their deeper conceptual relationships, even when they do not share specific keywords.

SPECTER further improves the model’s ability to understand complex academic content, making it easier for researchers to discover papers that are contextually related to their work. By offering a more nuanced understanding of scientific literature, SPECTER is particularly useful for researchers navigating vast databases of academic papers, streamlining the research process and improving knowledge discovery.

5.2.3. Retrieval-Augmented Generation (RAG)

• Scope: Tier C—supporting infrastructure. The frameworks and vector databases below are not MLLMs themselves; they are the retrieval components that multimodal applications depend on, and we document them for completeness.

In the present survey, RAG is treated as the system-level extension of the cross-modal retrieval principles introduced in Section 4.4. A multimodal RAG pipeline first embeds a text, image, video frame, or region-level query; retrieves external evidence such as text passages, image–caption pairs, product records, or domain-specific visual exemplars; optionally reranks that evidence with a cross-modal model; and then conditions generation on the retrieved context. The technical question is therefore not whether RAG “adds search” to an LLM, but how retrieval recall, evidence granularity, grounding quality, and generation interact. This framing avoids repeating the task-level retrieval description and focuses instead on the deployment chain: cross-modal retrieval + evidence grounding + generation.

LangChain

LangChain, released in 2022, provides orchestration abstractions for building applications that connect large language models with retrievers, vector stores, document stores, APIs, and external tools, including Retrieval-Augmented Generation pipelines [135]. In this role, it helps compose retrieval, evidence selection, prompting, and generation into repeatable workflows. For multimodal applications, the relevant design question is how these orchestration layers connect visual or cross-modal retrieval components to downstream language-model reasoning.

Such workflows can support chatbots, research assistants, and enterprise systems that need external knowledge sources, but their quality depends on retrieval recall, reranking, prompt construction, and model calibration. LangChain should therefore be viewed as supporting infrastructure rather than as an MLLM or a retrieval model in its own right.

A growing ecosystem of vector databases and retrieval frameworks underpins practical RAG deployments.

Table 13. Comparison of vector databases and retrieval frameworks for RAG pipelines.

System	Index Type	Multimodal	Scalability	License	Latency Note
Pinecone [136]	Managed ANN	Text + image emb.	Billions of vectors	Proprietary	Workload-dependent managed service
Chroma [137]	HNSW	Text + image emb.	Millions	Open-source	Local setup dependent
FAISS [138]	IVF/HNSW/PQ	Embedding-agnostic	Billions (GPU)	Open-source	Hardware/index dependent
Weaviate [139]	HNSW + filters	Native multimodal	Billions	Open-source	Cluster/config dependent
Qdrant [140]	HNSW + payload	Text + image emb.	Billions	Open-source	Cluster/config dependent
Vespa [141]	ANN + BM25	Text + tensor	Billions	Open-source	Cluster/config dependent

compares six widely used systems along dimensions relevant to multimodal RAG pipelines.

What latency reporting means for an end-to-end multimodal system. The latency notes in Table 13 should not be read as directly comparable benchmark numbers because query time depends on hardware, index parameters, corpus size, top-K, filtering, network placement, batching, and reranking. In many multimodal RAG pipelines, total user-perceived latency includes visual embedding, first-stage ANN lookup, optional cross-modal reranking, prompt construction, and autoregressive generation by the MLLM. Retrieval is therefore only one part of the budget, and in hosted assistant settings the generation step or visual encoder can dominate the end-to-end response time. A fair latency comparison should report the full deployment setting rather than a single vector-database number; latency-sensitive choices become especially important when the corpus is very large, when aggressive compression such as IVF or PQ is needed for memory reasons [142], or when the deployment target is an edge device with a tight response-time budget.

Recall–latency trade-off curves for different index structures. The choice between HNSW, IVF, and PQ is better understood as a position on a recall–latency Pareto curve than as a choice between fast and slow libraries. For a fixed corpus of N vectors, HNSW graphs [143] often provide high recall at low query time for million-scale corpora, but their memory footprint grows with graph connectivity and can become difficult to manage at very large scale without sharding. IVF with product quantization (IVF-PQ) trades some recall for a smaller memory footprint and can scale to larger corpora [138,142]. For multimodal RAG pipelines that perform a subsequent cross-modal reranking pass over the top-K retrieved candidates, an index that provides adequate recall at high throughput may be preferable to a slower index with marginally higher first-stage recall, because the reranker can repair some imperfect first-stage retrieval while latency cannot be recovered. HNSW is therefore attractive for smaller, latency-sensitive corpora; IVF-PQ is attractive for larger corpora; and learned quantizers attempt to improve the Pareto frontier at intermediate scales. The latency notes in Table 13 should be read in this broader context rather than as a direct quality ranking.

RAG is particularly consequential for Multimodal Large Language Models, where hallucination and knowledge staleness are amplified by the added complexity of visual inputs. By grounding generation in retrieved evidence—whether textual passages, image–caption pairs, or structured knowledge-graph triples—RAG pipelines can reduce some factual confabulation when the retrieved evidence is relevant and correctly used. In the multimodal setting, retrieval can supply region-level visual descriptions or domain-specific imagery that the base model did not memorize during pre-training, thereby improving domain coverage. However, retrieval alone does not guarantee visual grounding: object hallucination benchmarks such as POPE and CHAIR still require explicit evaluation, and a retrieved context can also introduce errors if it is irrelevant, stale, or misinterpreted [144].

Speculative RAG

Speculative RAG [145] is a framework designed to enhance traditional Retrieval-Augmented Generation systems by introducing a draft-and-verify mechanism into the generation process. In this framework, smaller specialist language models generate candidate answer drafts in parallel from retrieved subsets, and a larger generalist model verifies and synthesizes the final response. This approach aims to improve efficiency and contextual relevance by reducing the amount of retrieved context that the larger model must process directly.

Speculative RAG offers a promising direction for improving RAG systems, particularly in scenarios where retrieval coverage and answer verification both matter. For legal research, scientific analysis, or other high-stakes settings, however, it should be paired with citation checking and human review rather than treated as a guarantee of correctness.

These tools and frameworks represent an active area of Retrieval-Augmented Generation research, where efficient retrieval systems and generative models are combined to improve grounding, relevance, and context awareness. By integrating external data and improving retrieval quality, RAG systems can mitigate some limitations of static language models, but their reliability still depends on corpus quality, retrieval recall, reranking, citation fidelity, and generation behavior.

5.2.4. Multimodal Assistants and Chatbots

• Scope: Tier A—core architecturally multimodal MLLMs. The systems below (GPT-4V, GPT-4o, Claude 3, Gemini) are the primary object of this survey: they jointly reason over visual and textual tokens through a shared language model backbone.

Advancements in Multimodal Large Language Models (MLLMs) have expanded human–AI interaction by enabling conversations with systems that can process and respond to textual and visual inputs. These models are increasingly capable of understanding and generating across multiple modalities, making them useful in contexts such as customer support, education, and complex problem-solving, while still requiring careful evaluation for factuality and grounding. Below are some of the most notable advancements in multimodal assistants and chatbots.

GPT-4V (Visual)

GPT-4V (Visual) is OpenAI’s 2023 GPT-4 variant that incorporates the ability to analyze and respond to images as well as text [38]. This multimodal capability allows users to upload images alongside text prompts to receive visual question-answering, explanation, and document-interpretation assistance. For example, users can ask the model to analyze a photograph of a chart, identify objects in a picture, or provide explanations for visual content such as diagrams or architectural designs.

These capabilities support tasks such as visual QA, document interpretation, chart reading, and diagram explanation. In healthcare-related research, GPT-4V could assist with explanation or triage of visual information, but medical-image interpretation requires clinical validation and expert oversight before use in care. In education, it can support interactive learning experiences in which students ask questions about visual aids, diagrams, or their own sketches. Domain-specific uses require validation because reliability, grounding, and safety vary by task and visual input type.

GPT-4o

GPT-4o (“omni”), released in May 2024, represents a shift toward a natively multimodal assistant. OpenAI describes it as trained end-to-end across text, vision, and audio inputs and outputs, enabling lower-latency multimodal interaction than earlier voice-mode pipelines [39]. Public benchmark reports list GPT-4o at 69.1 on MMMU and 83.4 on MMBench, improving over reported GPT-4V scores (56.8 and 75.1, respectively), but OpenAI has not disclosed architectural details such as the Vision Encoder type, parameter count, or training mixture. The model therefore remains difficult to compare causally with open systems and still requires evaluation for spatial reasoning, hallucination, and high-stakes use.

Claude 3

Claude 3, developed by Anthropic and announced in March 2024, is a multimodal model family with visual understanding capabilities [41]. Similar to GPT-4V, Claude 3 is designed to analyze and respond to images alongside text, providing users with deeper insights and more context-aware interactions. One of Claude 3’s distinguishing design goals is safety-oriented post-training, intended to make responses more helpful and less harmful, although accuracy and fairness still require task-specific evaluation.

In support workflows, visual inputs such as receipts, screenshots, product images, and charts can provide context for troubleshooting or explanation. Claude 3 can analyze these inputs alongside text, allowing users to request summaries, comparisons, or follow-up questions grounded in the submitted visual material. In creative and design contexts, the same visual-understanding capability can support feedback on drafts or analysis of reference images, while final decisions remain task- and domain-dependent.

Anthropic has placed particular emphasis on safety and responsible deployment. For industries like legal services and education, this makes Claude 3 relevant as an assistant, but outputs should still be checked by domain experts because reliability and ethical considerations are context dependent.

Gemini

Gemini is Google’s family of multimodal AI models for assistant and reasoning tasks [40]. The Gemini series is designed to understand and process multiple modalities, including text, images, audio, and video. By integrating these capabilities into a single model family, Gemini supports human–AI interaction in which users can provide inputs in different formats and receive contextually integrated responses.

One of Gemini’s key strengths lies in its reported scalability across model sizes and its support for multiple input modalities. It is designed to handle tasks ranging from basic assistant interactions to more complex problem-solving scenarios that require the model to synthesize information from multiple data types. As an illustrative—rather than validated—example, Gemini has been demonstrated on prototype tasks that combine medical history, diagnostic imagery, and clinical audio to produce multimodal summaries; such usage in production clinical settings remains an aspirational use case rather than a clinically validated deployment, and published evaluations of the model do not include the regulatory approvals, prospective clinical trials, or external audits that would be required before such a system could responsibly be placed in the diagnostic loop.

In educational settings, Gemini’s multimodal capabilities enhance learning by allowing students to interact with visual and auditory content while asking questions or requesting clarifications. Its ability to switch between different types of inputs can support dynamic, interactive learning environments. Moreover, Gemini may support entertainment workflows where users draft content across formats, from textual scripts to visual storyboards.

Google’s Gemini report describes a family of models trained for multimodal reasoning across text, image, audio, and video inputs. The goal is to create an integrated multimodal experience, but product behavior depends on the specific Gemini version, interface, safety policies, and serving configuration.

These advancements in multimodal assistants and chatbots are broadening how humans interact with AI. Models like GPT-4V, Claude 3, and Gemini allow users to input and receive information in multiple formats, thereby increasing the richness of AI-driven conversations. Whether in healthcare research, education, customer service, or creative industries, these multimodal models make AI more interactive and context-aware, while their reliability remains task- and deployment-dependent.

Current multimodal assistants exhibit several recurring failure modes that constrain their reliability in practice.

Spatial reasoning. Yang et al. [146] systematically tested GPT-4V on visual reasoning tasks and documented recurring errors in relative position, counting, and distance or size judgment. These errors can persist even when the relevant objects are clearly visible, suggesting limitations in the spatial encoding of visual tokens rather than failures of visual perception per se.

Architectural roots of the spatial reasoning errors. It is tempting to treat these errors as model-specific quirks, but the pattern connects directly to three architectural choices shared by GPT-4V and most CLIP-ViT-based MLLMs. First, ViT-based encoders [20] represent an image as a sequence of non-overlapping $14\times 14$ or $16\times 16$ patches with learned 1-D positional embeddings; the explicit 2-D geometry is reconstructed only implicitly by attention over these flat position indices, which is known to be weaker for relative-position relations than for absolute or part-of relations. Second, the connector between the visual encoder and the LM typically compresses the visual-token sequence—a Q-Former with 32 queries discards approximately 88% of the spatial signal of a 256-patch image, while even MLP projections are “blind” to 2-D structure once the patches have been serialized. Third, instruction tuning corpora are known to under-represent explicit spatial-relation descriptions relative to object-naming and attribute descriptions; under-represented supervision yields under-represented behaviour. A straightforward falsifiable prediction that follows from this architectural reading is that spatial reasoning error rates should decrease monotonically with (a) encoder input resolution, (b) visual-token budget retained through the connector, and (c) the density of spatial-relation supervision in the training mix; recent results on dynamic-resolution and AnyRes schemes (e.g., Qwen2-VL, LLaVA-OneVision) are consistent with (a) and (b). We emphasise that this connection between failure mode data and architectural design remains underemphasized in many MLLM surveys.

Object hallucination. On the POPE benchmark, GPT-4V achieves precision of $\sim 86\%$ under the adversarial split but only $\sim 79\%$ under the popular-object split, indicating that the model can hallucinate objects that are semantically plausible in a given scene context [144]. Similar qualitative patterns have been reported for other multimodal assistants, but direct percentage comparisons are protocol- and version-dependent. Crucially, these are existence hallucinations; attribute and relational hallucinations (e.g., “the red car” when the car is blue) remain less studied but are likely more prevalent. Following the decomposition in Section 6.3.2, the popular-object failure rate is most informative about training-distribution co-occurrence priors (and is therefore partially irreducible under the current pre-training data), whereas the adversarial-split failure rate is most informative about the connector’s capacity to preserve fine-grained visual evidence (and is therefore addressable by higher-resolution encoders, richer connectors, and grounding-aware training).

Multi-turn context loss. In extended dialogues, multimodal assistants may lose track of visual context established in earlier turns, producing responses that contradict previously acknowledged image content. This is unlikely to be only an attention-length limitation because the nominal context windows of GPT-4V, Claude 3, and Gemini are large enough for many short conversations. A more plausible architectural cause is that visual tokens and textual tokens compete for the same context and KV-cache budget, and the relative weight that the LM assigns to visual tokens may decay as additional textual turns dilute the cache. This predicts (testably) that the drop-off should shift later when visual tokens are re-injected into each turn, and should shift earlier when the model is asked to reason over increasingly long image sequences; these observations remain under-systematized in published evaluations.

Chart and diagram interpretation. Fine-grained chart reading is error-prone, particularly for numerical scales, overlapping labels, and non-standard plot types. Models frequently misread axis values or confuse data series, limiting their utility for data analysis applications. This failure mode has the same architectural signature as the spatial reasoning error analysed above—low effective resolution at small textual features combined with a connector that does not preserve fine-grained layout—and is therefore expected to improve with the same encoder-resolution and dynamic-patching interventions.

These failure patterns underscore the need for targeted evaluation benchmarks that stress-test spatial, counting, and cross-turn visual reasoning, rather than relying solely on aggregate VQA accuracy; equally, they reveal concrete architectural levers (encoder resolution, connector compression ratio, supervision density) that future MLLM designers can tune against, rather than treating failure rates as intrinsic to a model.

Table 14. Comparison of prominent multimodal assistants: GPT-4V, Claude 3, and Gemini.

Feature	GPT-4V (OpenAI)	Claude 3 (Anthropic)	Gemini (Google)
Input modalities	Text + image (JPEG, PNG)	Text + image/PDF pages	Text, image, audio, video
Architecture	Undisclosed †; multimodal extension of GPT-4	Undisclosed †; multimodal assistant model	Mixture-of-Experts + multimodal [40]
Alignment methodology	Moderation filters + RLHF	Constitutional AI + RLHF †	Responsible AI principles
Strongest use case	Visual QA, charts, documents	Long-document + image reasoning	Multimodal reasoning and interaction
Known weakness	Spatial reasoning errors	Closed architecture; visual reasoning limits	Audio/video latency; hallucination
Access	OpenAI API/ChatGPT	Anthropic API/Claude.ai	Google AI Studio/Gemini App
Key benchmarks	Strong reported vision-language performance; version-dependent	Strong reported multimodal and long-context performance	Strong reported multimodal performance; version-dependent

^† Not publicly documented by the developer; descriptions are inferred from published model cards, product behavior, and third-party analyses.

compares the disclosed capabilities and limitations of representative multimodal assistants.

5.2.5. Video Analysis and Generation

• Scope: Tier B—adjacent generative systems. Most of the platforms below are diffusion-based text-to-video generators or AI-assisted editors rather than multimodal-understanding MLLMs; they are discussed because their encoders, training data, and evaluation conventions are closely related to those of the Tier A systems.

Video should be separated into two related but distinct problems. Video generation systems synthesize or edit frames from text or image prompts. Video understanding systems answer questions, retrieve clips, summarize events, or reason over temporal order. The reviewer’s examples—“which object is larger in image A or image B?” and “what did the person do first and then what in a five-minute video?”—belong primarily to the second category and require multi-image or long-video reasoning rather than short-clip generation.

Representative technical approaches. VideoCoCa [42] adapts image–text contrastive captioners to video by treating video frames as flattened visual-token sequences, preserving the contrastive/captioning objectives that made image–language models effective. Video-LLaMA [43] introduces video and audio Q-Formers that align temporal visual and auditory signals with an LLM. Video-LLaVA [44] instead argues for unified visual representations before projection into the language model, enabling shared image–video instruction tuning.

For long videos, LangRepo [45] uses a language repository as external memory: the model processes multi-scale chunks, writes textual state, and later reads that state to answer long-horizon questions. LongVideoBench [46] provides an evaluation setting for such capabilities by testing interleaved video–language understanding over long videos and referred temporal contexts.

Table 15. Representative systems and benchmarks for video and multi-image reasoning.

System	Primary Route	Capability Addressed	Remaining Limitation
VideoCoCa	Video–text contrastive/captioning	Reuses image–text pre-training for video retrieval, captioning, VQA, and classification.	Flattened frame embeddings can miss long-range event structure.
Video-LLaMA	Video/audio Q-Formers	Aligns temporal visual and auditory information with an LLM for video dialogue.	Fixed query compression can bottleneck fine temporal evidence.
Video-LLaVA	Unified image–video representation	Improves transfer between image and video instruction tuning.	Still depends on frame budget and short-context visual evidence.
LangRepo	Language memory repository	Maintains interpretable state across long videos through chunk-level reading and writing.	Depends on summary quality and may lose visually fine-grained details.
LongVideoBench	Benchmark	Tests long-context video–language understanding and temporal retrieval over extended clips.	Benchmark success may still underrepresent open-ended real-world video tasks.

summarizes representative systems and benchmarks for video and multi-image reasoning.

Adjacent video-generation platforms such as Runway [147], ModelScope [148], Pika Labs [149], and Kling AI [150] remain relevant as ecosystem context, but they do not by themselves solve long-video temporal understanding. Their contribution lies mainly in text-to-video synthesis, editing, and creator-facing workflows. In contrast, the core MLLM research challenge is to preserve event order, object identity, cross-frame spatial relations, and evidence provenance over many frames while keeping memory and energy costs tractable.

5.2.6. Robotics and Embodied AI

• Scope: Tier A—architecturally multimodal MLLMs extended with action. The systems below (RT-2, SayCan, PaLM-SayCan, ManipLLM, PaLM-E, VoxPoser, LLM-VLMaps) jointly reason over visual, textual, and action tokens, and are therefore within the core scope of this survey.

The integration of Multimodal Large Language Models (MLLMs) with robotics and embodied AI is creating new research directions for language-conditioned interaction with the physical world. By combining language understanding with robotic perception and control, these systems can process commands, interpret visual or sensory information, and select actions in bounded environments. They should be understood as research systems that expose useful design patterns rather than as general-purpose autonomous robots ready for unconstrained deployment. Below are representative projects and models in this area.

Embodied AI also clarifies why multi-image and long-video reasoning are not optional add-ons. A robot observes a sequence of states: an object before and after manipulation, a person entering then leaving a room, or a tool moving from one surface to another. Vision–language–action models must therefore compare observations across time, preserve object identity, infer temporal order, and map language to feasible actions. The systems below are discussed as vision–language–action architectures rather than as ordinary chatbots with camera input.

RT-2 (Robotic Transformer 2)

RT-2 [151], or Robotic Transformer 2, is Google’s 2023 vision–language–action model. RT-2 is notable because it transfers web-scale vision–language knowledge into a robotic action-token setting. The model enables robots to interpret visual inputs, understand language commands, and map them to executable action tokens within the robot’s trained action space. This integration allows RT-2 to perform research tasks such as recognizing objects in a scene, following instructions, and interacting with the environment accordingly.

What makes RT-2 particularly important is its attempt to generalize from knowledge gained from web data and apply it to robotic control. This can reduce the need to hand-specify every object category or instruction pattern a robot may encounter. Instead, the system combines web-scale vision–language pre-training with robotic training data to improve generalization to some novel tasks. Its implications for manufacturing, logistics, and home robotics remain research-oriented because physical deployment still requires safety constraints, calibration, feedback control, and extensive environment-specific testing.

SayCan

SayCan [152] is a method developed by Google to ground large language models in robotic affordances, allowing robots to select natural-language action plans that are both plausible and physically feasible. SayCan works by linking language models, which can interpret and generate human language, with a robot’s understanding of its abilities (affordances) in a physical environment. This enables a robot to rank possible next steps based on both language-model likelihood and affordance scores from the robot’s skills.

For example, a robot powered by SayCan could process an instruction such as “grab the red cup from the table” and break it down into feasible skill calls, such as identifying the cup, navigating to it, and choosing a grasping action. By grounding language models in robotic affordances, SayCan helps robots assess whether proposed steps are feasible for the available skills, bridging language understanding and practical action selection.

PaLM-SayCan

PaLM-SayCan [152] builds on the SayCan methodology by integrating it with Google’s PaLM (Pathways Language Model) for robotic planning. PaLM-SayCan combines PaLM’s language understanding capabilities with the affordance-grounding principles of SayCan, enabling more structured multi-step action selection by robots. Its contribution is to rank language-generated plans by both semantic plausibility and physical feasibility rather than to guarantee robust execution in all environments.

The use of PaLM enhances the robot’s ability to interpret nuanced language inputs and decompose them into candidate steps. For instance, a user could ask a PaLM-SayCan-powered robot to “prepare a cup of coffee,” and the planner would propose steps such as identifying necessary items, moving to relevant locations, and invoking manipulation skills when available. PaLM-SayCan exemplifies how integrating language models with robotic affordance models can improve task planning, while practical deployment in domestic, warehouse, or healthcare environments requires separate validation of perception, manipulation, and safety.

ManipLLM

ManipLLM [54] is an embodied Multimodal Large Language Model designed specifically for object-centric robotic manipulation. The model is designed to connect language instructions, visual perception, and manipulation representations for tasks that require reasoning about physical objects.

ManipLLM integrates visual and language information for object-centric manipulation planning, allowing robots to reason about objects, task instructions, and candidate actions. For example, a robot powered by ManipLLM could be studied on assembly-style tasks by interpreting both the visual layout of components and the language instructions, then selecting manipulation steps through a downstream controller. Potential applications include manufacturing and logistics settings where object manipulation and assembly tasks are common, but the system should be described as a research prototype rather than as validated for safety-critical medical manipulation.

PaLM-E

PaLM-E [153] is an embodied multimodal language model that extends language modeling with continuous sensor inputs such as images and robot-state features. Rather than directly outputting arbitrary continuous robot actions, PaLM-E produces text-form decisions, plans, or action-related tokens that can be grounded through downstream robotic policies and controllers. It therefore bridges perception, language understanding, and action selection while still relying on low-level control systems for physical execution.

PaLM-E’s ability to combine language instructions with embodied observations opens up new possibilities for human–robot interaction. A robot equipped with PaLM-E-style planning could respond to commands like “clean the kitchen” or “pick up the toys in the living room” by proposing a sequence of steps that involve navigation, object recognition, and manipulation, subject to the capabilities and safety limits of the robot platform. This model represents an important research step toward more flexible robots that can use natural-language inputs, but robust autonomy remains an open systems problem.

VoxPoser

VoxPoser [154] is a system that combines large language models with 3D scene representations to support robotic manipulation in structured environments. By integrating 3D perception with language-model reasoning, VoxPoser helps translate instructions into spatial constraints and affordance maps that a robot can use for planning.

For instance, in a scenario where a robot is asked to “place the vase on the table between the two books,” VoxPoser can help interpret the 3D scene, identify relevant object positions, and generate spatial constraints for the robot controller. The system’s ability to map language to 3D space is useful for applications that require spatial awareness and object placement, such as logistics, assembly research, and home-assistance prototypes.

LLM-VLMap

LLM-VLMap (Large Language Model Visual-Language Mapping) [155] is a framework designed to support language-conditioned robot navigation through visual-language maps. The framework associates visual observations with language labels or descriptions, allowing robots to localize objects and plan navigation steps based on verbal instructions.

LLM-VLMap is particularly relevant to scenarios where robots need to navigate large or complex environments, such as warehouses, hospitals, or offices. The model can support commands such as “find the blue box in the warehouse” by using its visual-language mapping system to localize the object and plan navigation. LLM-VLMap demonstrates how MLLMs can support robotic navigation, while manipulation, obstacle avoidance, and safety remain responsibilities of the broader robotic system.

The integration of MLLMs with robotics and embodied AI has made progress in how robots connect language, perception, and action. From vision–language–action models like RT-2 and systems like SayCan and PaLM-SayCan that improve action selection, to frameworks such as LLM-VLMap that support navigation, these technologies show how natural language can interface with robotic skills. Their industrial impact will depend on safety validation, reliability under distribution shift, hardware constraints, and integration with conventional robotic control.

5.2.7. Case-Study Synthesis

The case studies presented in this section provide a broad view of how Multimodal Large Language Models (MLLMs) and adjacent multimodal systems are influencing content creation and retrieval, multimodal assistants, and robotics by integrating diverse data types and improving human–AI interaction. These developments, driven by advanced neural network architectures and training methods, are making AI systems more capable of understanding and generating outputs across modalities such as text, images, audio, and video.

From creative industries using tools like Midjourney [122] and DALL-E 3 [123] to draft high-quality visuals, to video-generation platforms such as Runway [147] lowering some barriers to video production, the generative-system layer (Tier B) complements the architecturally multimodal MLLMs that are the core focus of this survey. Audio-assisted multimodal platforms can also support accessibility and personalized content.

The integration of MLLMs with infrastructure platforms, such as Hugging Face, LangChain [156], and Ollama, has provided developers with tools to experiment, prototype, and deploy MLLM-based applications. These platforms make it easier to build applications across sectors, but deployment quality still depends on model selection, data governance, evaluation, latency, and monitoring.

Moreover, the rise of Retrieval-Augmented Generation (RAG) systems—seen through the development of tools like Pinecone [136]—represents a significant shift in AI, where models can supplement pre-trained knowledge with retrieved external information to generate more contextually relevant outputs [157]. This blend of retrieval and generation can improve utility in applications such as legal research, personalized recommendations, and technical support, provided that retrieved sources are relevant, current, and accurately cited.

In robotics and embodied AI, recent systems show how MLLMs can help robots interpret visual inputs and natural-language commands for planning, navigation, and manipulation. These advances create new possibilities for automation in manufacturing, logistics, and service settings, while robust real-world deployment remains constrained by safety, calibration, feedback control, and evaluation.

Overall, these examples show that practical multimodal systems depend on interactions between core MLLMs, adjacent generative models, retrieval infrastructure, and embodied-control components. Current systems can link language, vision, audio, retrieval, and action, but robust general multimodal reasoning still requires stronger evaluation, transparency, safety, and deployment discipline.

Taken together, the case studies in this section provide an ecosystem snapshot rather than an exhaustive MLLM benchmark: Tier A systems (GPT-4V, GPT-4o, Claude 3, Gemini, and the embodied models discussed above) are the core of the survey, Tier B systems illustrate how closely related generative techniques feed into and are influenced by multimodal-understanding models, and Tier C systems document the retrieval and serving infrastructure on which practical multimodal deployments depend. This tiered framing is intended to make clear, system by system, what kind of evidence each case study rests on and how far its claims generalize.

6. Challenges and Limitations of Multimodal Large Language Models

Despite the rapid progress documented in the preceding sections, MLLMs face challenges that span architectural design, cross-modal alignment, reliability, and ethics. Figure 14 organizes these into eight categories, each examined below.

6.1. Model Architecture and Scalability

6.1.1. Designing Efficient Multimodal Architectures

MLLM architectures must process and integrate information from multiple modalities while maintaining coherent internal representations. This introduces design trade-offs not present in unimodal models.

Cross-Modal Attention Mechanisms

Cross-modal attention (defined in Section 2.3) poses several design challenges in an MLLM fusion context that go beyond its baseline formulation:

• Computational Complexity: Traditional attention mechanisms scale quadratically with input size, which becomes problematic for multimodal inputs. This scaling challenge is particularly acute when dealing with high-dimensional inputs such as images or video sequences combined with text. Recent work on efficient attention, such as Performer [158], offers promise but requires adaptation for multimodal settings. These adaptations must balance computational efficiency with the ability to capture nuanced cross-modal relationships.
• Modality-specific Biases: Attention weights may be biased towards certain modalities, leading to suboptimal integration. This bias can result in models that overemphasize one modality while neglecting important information from others. Kim et al. [159] proposed ViLT, which uses a single transformer for both vision and language, but balancing attention across modalities remains an open problem. Addressing these biases requires careful architectural design and training strategies that ensure equal representation and importance across all modalities.
• Long-range Dependencies: Capturing long-range dependencies across modalities is crucial but computationally expensive. These dependencies are essential for understanding complex relationships between different parts of multimodal inputs, such as connecting visual elements with their textual descriptions or understanding temporal relationships in video content. Techniques like Longformer [160] could be adapted for multimodal contexts but require careful design to handle cross-modal interactions effectively.

Modality-Specific vs. Unified Encoders

The choice between separate encoders for each modality and a unified encoder for all modalities presents significant trade-offs that must be carefully considered in the design of MLLMs:

• Separate Encoders: Models like CLIP [7] use separate encoders for images and text, allowing for modality-specific pre-training. This approach enables specialized processing of each modality and can leverage existing pre-trained models. However, this approach may struggle with fine-grained cross-modal reasoning due to the potential semantic gap between different encoder spaces. The challenge lies in effectively bridging these separate representations while maintaining the benefits of specialized processing.
• Unified Encoders: Early adjacent image–text generative systems such as DALL-E [161] illustrate one route to placing text and image tokens in a shared transformer, but they should be treated as historical and adjacent examples rather than central vision–language MLLMs. More general unified-interface systems, including Unified-IO [162], are closer to the survey’s multimodal framing because they use modality-specific tokenization with shared sequence modeling. The key challenge is designing architectures that can integrate diverse modality characteristics without losing task-specific efficiency or fine-grained evidence.
• Hybrid Approaches: Recent work by [162] on Unified-IO proposes a hybrid approach, using modality-specific tokenizers followed by a shared transformer. This promises a balance between specialization and integration but introduces additional complexity. These approaches attempt to combine the benefits of both separate and unified encoders while minimizing their respective drawbacks. The challenge lies in managing the increased architectural complexity while maintaining efficient training and inference.

Scaling Laws for Multimodal Models

Understanding how MLLM performance scales with model size and dataset characteristics is crucial for efficient development. This understanding guides resource allocation and architectural decisions in the development of increasingly capable models:

• Modality-specific Scaling: Work by [163] on language model scaling laws needs extension to multimodal settings. Different modalities may exhibit distinct scaling characteristics due to their inherent properties and computational requirements. Preliminary studies suggest that different modalities may have different optimal scaling relationships, necessitating careful consideration of how to allocate model capacity across modalities.
• Cross-modal Scaling: The relationship between model size and cross-modal performance is not well-understood. This relationship is particularly complex due to the interactions between different modalities and the potential for emergent capabilities as models scale. Initial studies on scaling vision–language models provide promising insights, but more comprehensive investigations are needed to fully characterize the scaling dynamics of multimodal systems.
• Dataset Scaling: The impact of dataset size and quality on MLLM performance across different tasks and modalities requires further investigation. The quality and diversity of training data play crucial roles in model performance, but the relationships between dataset characteristics and model capabilities are not yet fully understood. Diverse and high-quality multimodal datasets are essential for effective scaling, emphasizing the importance of careful data curation strategies.

6.1.2. Computational Efficiency, Memory, Energy, and Deployment Cost

The computational demands of MLLMs present significant challenges for both training and inference. As these models grow in complexity and capability, managing their computational requirements becomes increasingly critical for practical applications. The intersection of multiple modalities introduces unique computational challenges that exceed those of traditional unimodal models, requiring innovative solutions across various aspects of model design and deployment.

Efficiency cannot be reduced to latency alone. A fair deployment comparison must jointly report parameter count, numerical precision, weight memory, activation and KV cache memory, visual-token budget, hardware class, throughput, energy use, carbon proxy, and inference cost. A 13B model quantized to 4 bits has a raw weight footprint of roughly $13\mathrm{B}\times 4/8\approx 6.5$ GB before activation memory, KV cache, Vision Encoder memory, and runtime overhead. By contrast, a 70B model in FP16 has a raw weight footprint of roughly $70\mathrm{B}\times 2\approx 140$ GB and normally requires multi-GPU serving. These systems should not be compared solely by a benchmark score or a single end-to-end latency number, especially when one runs locally on a mobile or edge-class device and the other runs on an 8 × H100 server.

Table 16. Resource dimensions required for meaningful MLLM efficiency reporting.

Dimension	What to Report	Why It Matters	Example Interpretation
Model size/precision	Parameters; FP16, BF16, INT8, 4-bit	Determines weight memory and hardware feasibility	13B 4-bit weights are about 6.5 GB before overhead; 70B FP16 weights are about 140 GB.
KV/activation memory	Context length, batch size, layers, hidden size	Often dominates long-video or multi-image inference	More frames increase visual tokens and KV cache even if weight memory is unchanged.
Visual-token budget	Image resolution; frames sampled; connector compression	Controls attention cost and evidence retention	Q-Former-style compression saves memory but may discard fine spatial or temporal evidence.
Hardware class	Mobile NPU, single GPU, multi-GPU server	Latency and energy are hardware-dependent	A mobile 13B run and an 8 × H100 70B run answer different deployment questions.
Throughput/latency	Tokens/s, requests/s, time-to-first-token	Separates interactive usability from batch efficiency	A fast retrieval layer may be irrelevant if autoregressive generation dominates latency.
Energy/carbon proxy	$E\approx P\times t\times u$; carbon $\approx E \times$ regional carbon intensity	Avoids unsupported absolute carbon claims while making environmental cost reportable	Report power, runtime, utilization, and region rather than a context-free carbon number.
Inference cost	Hardware rental/API cost per request or per 1 K tokens	Determines deployability at scale	A model with slightly lower accuracy may be preferable if it reduces cost by an order of magnitude.

summarizes these reporting dimensions and their deployment interpretation.

For this reason, benchmark papers should pair accuracy with a resource card: hardware, precision, memory footprint, throughput, input resolution, number of frames or images, and an energy/cost proxy. Without these fields, claims that one MLLM is more “efficient” than another are underspecified.

Inference Optimization

Low-latency MLLM inference depends not only on language-model size but also on the cost of visual encoding, connector projection, visual-token retention, and multi-image or video-frame processing. Practical systems must therefore optimize the full path from image or video preprocessing through autoregressive generation:

• Model Compression: Quantization and pruning [164] must be calibrated across the language backbone, vision encoder, and connector. A compression setting that preserves text fluency can still degrade grounding if it distorts visual embeddings, connector projections, or cross-modal calibration.
• Hardware-aware Design: Hardware-aware MLLM serving must account for vision-encoder cost, KV-cache growth, image or frame batching, and CPU–GPU–NPU data movement. Optimizing only the transformer decoder can leave preprocessing, visual projection, or memory transfer as the dominant latency source.
• Adaptive Computation: Conditional computation [165] can be extended to choose image resolution, frame sampling rate, token pruning, or early-exit behavior according to input complexity. The challenge is to save computation without discarding visual evidence needed for spatial, temporal, or fine-grained reasoning.

Training Efficiency

Training efficiency is shaped by heterogeneous multimodal batches, uneven modality loss scales, and the cost of pairing high-resolution visual inputs with long text contexts:

• Efficient Optimization: Large-batch training and gradient accumulation must handle variable image counts, frame counts, resolutions, and text lengths. Stable optimization requires balancing modality-specific losses and preventing high-cost visual examples from dominating throughput.
• Curriculum Learning: Multimodal curricula can order data by task complexity, modality mix, resolution, or temporal length. The main difficulty is defining progression rules that improve cross-modal reasoning rather than merely making early training cheaper.
• Pre-training Strategies: Efficient pre-training objectives must balance contrastive alignment, captioning, masked multimodal modeling, and instruction-style supervision without excessive visual-token cost. Work on large-scale contrastive learning [166] remains important, but MLLMs also need objectives that preserve fine-grained grounding after connector compression.

Memory Management

Memory pressure in MLLMs comes from activations, KV cache, visual-token sequences, connector states, and cross-modal attention. These costs rise quickly with image resolution, number of images, and sampled video frames:

• Gradient Checkpointing: Gradient checkpointing [167] should consider which visual, connector, and language activations are cheapest to recompute. The optimal policy may differ for high-resolution images, long videos, and text-heavy batches.
• Attention Memory Optimization: Linear attention [168] and kernel-based methods [158] may reduce memory use, but MLLMs must preserve cross-modal evidence flow. Separate strategies may be needed for intra-image attention, temporal attention, and language-to-vision attention.
• Mixed Precision Training: Mixed precision training [169] can reduce memory use, but numerical sensitivity may differ across visual features, connector layers, and language tokens. Precision choices should therefore be checked for both text quality and visual grounding.

6.2. Cross-Modal Learning and Representation

6.2.1. Alignment of Different Modalities

Creating unified representations that effectively capture information across modalities is a central challenge in MLLM development. The goal is to develop representations that can meaningfully capture and integrate information from different modalities while preserving the unique characteristics and relationships within each modality. This challenge is fundamental to enabling sophisticated cross-modal reasoning and generation capabilities.

Alignment Quality as a Consequence of First Principles

Section 2.2 introduced the information bottleneck and data-processing view of multimodal learning. In the present challenge context, the practical implication is that cross-modal alignment has no zero-error target: every connector must decide which visual evidence to preserve, compress, or discard under a fixed capacity budget. CLIP-style contrastive alignment, Q-Former-style token compression, and MLP projection therefore represent different positions on the same trade-off curve between retained task-relevant information and computational cost. The sections below focus on the concrete architectural and training consequences of this constraint rather than repeating the derivation.

Joint Embedding Spaces

The joint embedding space is introduced in Section 2.3; here we focus on the outstanding alignment-quality problems rather than restating the construction. These problems revolve around preserving modality-specific information while supporting meaningful cross-modal operations:

• Contrastive Learning: Methods like CLIP [7] use contrastive learning to align visual and textual representations through a self-supervised learning approach. While these methods have shown strong performance in image–text alignment, extending these approaches to more modalities and fine-grained alignments remains challenging. The challenge includes designing appropriate contrastive objectives that can handle multiple modalities simultaneously and capture fine-grained semantic relationships across modalities.
• Cross-modal Autoencoders: Techniques like multimodal autoencoders [170] aim to learn shared representations through reconstruction objectives. These approaches attempt to find common latent spaces that can capture the essential information from each modality while enabling cross-modal generation and translation. Balancing modality-specific and shared information in these models is an ongoing research direction, requiring careful consideration of architecture design and training objectives.
• Optimal Transport: Recent work by Chen et al. [171] uses optimal transport theory to align cross-domain embeddings, providing a principled framework for learning alignments between different representation spaces. Scaling these approaches to large-scale MLLMs and multiple modalities presents both opportunities and challenges. The mathematical foundations of optimal transport offer promising directions for achieving more precise and theoretically grounded cross-modal alignments, but computational scalability and adaptation to multiple modalities remain significant challenges.

Temporal Alignment in Video–Text Models

Handling temporal aspects in multimodal data, particularly for video understanding, presents unique challenges that go beyond the complexity of static multimodal content. The dynamic nature of video data, combined with the need to align and understand relationships across different modalities over time, introduces significant computational and modeling challenges. This temporal dimension adds layers of complexity to the already challenging task of multimodal integration.

• Long-term Dependencies: Capturing long-term dependencies across modalities in video understanding tasks is computationally challenging and requires sophisticated architectural solutions. These dependencies can span seconds, minutes, or even longer periods, making it difficult to maintain relevant context over time. Approaches like hierarchical transformers [172] show promise but require further development for multimodal settings. These architectures attempt to build representations at multiple temporal scales, from fine-grained frame-level features to high-level semantic concepts that span longer durations. The challenge lies in effectively combining these hierarchical representations while maintaining computational efficiency and meaningful cross-modal relationships.
• Asynchronous Events: Dealing with asynchronous events across modalities (e.g., delayed narration in videos) requires sophisticated temporal modeling that can handle complex temporal relationships. This challenge is particularly evident in real-world scenarios where different modalities may not be perfectly synchronized or may have varying temporal granularity. Recent work on temporal attention mechanisms provides a starting point for addressing this challenge, offering ways to learn flexible temporal alignments between modalities. These mechanisms must be capable of handling varying temporal scales and maintaining coherent cross-modal understanding despite temporal misalignments.
• Efficient Video Processing: Processing high-resolution video data in MLLMs is computationally intensive, requiring careful consideration of resource utilization and efficiency. The challenge is compounded by the need to process multiple frames while maintaining temporal coherence and cross-modal relationships. Techniques like dynamic sparse attention [173] could be adapted for efficient video processing in multimodal contexts, allowing models to focus computational resources on the most relevant temporal and spatial regions. This includes developing methods for adaptive frame sampling, temporal pooling, and efficient feature extraction that preserve important temporal dynamics while reducing computational overhead.

6.2.2. Transfer Learning and Generalization

Enabling effective knowledge transfer between modalities and tasks is crucial for the development of versatile MLLMs. The ability to leverage knowledge across different modalities and adapt to new tasks efficiently represents a fundamental challenge in multimodal learning. This capability is essential for creating models that can generalize effectively and adapt to new situations with minimal additional training.

Cross-Modal Transfer

Facilitating knowledge transfer between modalities presents several challenges that must be addressed to create truly adaptive and generalizable multimodal systems:

• Zero-shot Cross-modal Transfer: Enabling MLLMs to perform tasks in one modality based on knowledge from another without specific training examples is a significant challenge that requires sophisticated architectural and training approaches. Work on frozen language models for visual learning provides insights into potential approaches, but generalizing this approach to multiple modalities and tasks remains an open problem. The challenge lies in creating representations that can effectively bridge different modalities while maintaining the specific characteristics and requirements of each modality. This includes developing methods for abstract reasoning that can translate concepts learned in one modality to meaningful applications in another.
• Few-shot Learning: Developing few-shot learning techniques that effectively leverage knowledge across modalities is crucial for adaptive MLLMs that can quickly learn from limited examples. Recent work on meta-learning in multimodal contexts [70] shows promise but requires further investigation for large-scale models. The challenge involves creating learning algorithms that can effectively utilize prior knowledge across modalities to accelerate learning in new situations. This includes developing methods for efficient adaptation that can leverage cross-modal relationships while maintaining model stability and performance.
• Negative Transfer: Preventing negative transfer, where learning in one modality degrades performance in another, is a significant challenge that requires careful consideration of learning dynamics and knowledge representation. Techniques like gradient surgery could be adapted for multimodal settings to mitigate negative transfer by identifying and preventing harmful parameter updates. This includes developing methods to detect and prevent interference between modalities while maintaining beneficial knowledge transfer.

Domain Adaptation in Multimodal Settings

MLLMs often struggle when faced with domain shifts, particularly when these shifts occur differently across modalities. The challenge of domain adaptation becomes more complex in multimodal settings due to the need to handle shifts in multiple modalities simultaneously while maintaining cross-modal relationships:

• Cross-modal Domain Adaptation: Developing techniques that can adapt to domain shifts in multiple modalities simultaneously is challenging due to the complex interactions between modalities and the need to maintain coherent cross-modal relationships. Recent work on multi-source domain adaptation [174] provides a foundation for addressing these challenges, but extending these approaches to large-scale MLLMs remains an open problem. This includes developing methods that can effectively handle varying degrees of domain shift across different modalities while maintaining model performance and cross-modal understanding.
• Unsupervised Multimodal Adaptation: Creating unsupervised domain adaptation techniques for multimodal data is crucial for real-world deployments where labeled data in target domains may be scarce or unavailable. Approaches like MUDA show promise in addressing this challenge but require scaling to more complex multimodal scenarios. The challenge involves developing methods that can effectively leverage unlabeled data across modalities to adapt to new domains while maintaining model performance and reliability.
• Continual Adaptation: Enabling MLLMs to continuously adapt to changing domains across modalities without forgetting previously learned knowledge is a significant challenge that requires sophisticated approaches to memory and learning. Techniques like elastic weight consolidation [175] need careful adaptation for multimodal continual learning scenarios. This includes developing methods for selective parameter updates that can preserve important knowledge while allowing for adaptation to new domains and tasks.

6.3. Model Robustness and Reliability

6.3.1. Adversarial Robustness

MLLMs are vulnerable to adversarial attacks, particularly those that exploit the interaction between modalities. The multimodal nature of these models introduces new attack surfaces and vulnerabilities that must be carefully addressed to ensure reliable and secure deployment.

Cross-Modal Adversarial Attacks

Developing robust MLLMs requires addressing various types of adversarial attacks that can exploit vulnerabilities in cross-modal processing and integration:

• Multimodal Adversarial Examples: Creating defense mechanisms against adversarial examples that span multiple modalities is challenging due to the complex interactions between different types of inputs and the potential for attacks to exploit cross-modal dependencies. Recent work on attacks against audio–visual models highlights the complexity of cross-modal adversarial attacks and the need for sophisticated defense mechanisms. This includes developing methods that can detect and mitigate attacks that target multiple modalities simultaneously or exploit inconsistencies in cross-modal processing.
• Certified Robustness: Extending certified robustness techniques to multimodal settings is an open problem that requires new theoretical frameworks and practical implementations. Approaches such as randomized smoothing [176] provide certified robustness for unimodal classifiers, but they need adaptation to handle the complexities of multiple input modalities and their interactions. The challenge involves developing certification methods that can provide bounded, assumption-explicit guarantees about model behavior across different modalities and types of inputs.
• Transferability of Attacks: Understanding and mitigating the transferability of adversarial examples across modalities and model architectures is crucial for developing robust MLLMs. Work on cross-domain adversarial transferability by Naseer et al. [177] provides related initial evidence that perturbations can transfer across data domains and model families; analogous cross-modal transfer in MLLMs requires more comprehensive study. This includes investigating how adversarial perturbations in one modality can affect processing in other modalities and developing defense mechanisms that can handle these complex attack scenarios.

Robustness to Input Perturbations

Ensuring consistent performance under various input conditions is crucial for reliable MLLM deployment in real-world applications. The challenge of maintaining robust performance becomes particularly complex in multimodal settings, where perturbations can affect different modalities independently or in combination. Understanding and addressing these challenges is essential for developing MLLMs that can operate reliably in diverse and unpredictable environments.

• Visual Robustness: Developing models robust to visual noise, occlusions, and transformations is challenging and requires sophisticated approaches to maintain performance across a wide range of visual conditions. This challenge is particularly acute in real-world scenarios where lighting conditions, camera angles, and image quality can vary significantly. Techniques like adversarial training [178] need adaptation for multimodal contexts to improve visual robustness without compromising performance on clean data. This includes developing methods that can maintain cross-modal understanding even when visual inputs are degraded or partially obscured, while also ensuring that defensive mechanisms do not interfere with the model’s ability to extract meaningful features from clean inputs.
• Linguistic Variations: Addressing robustness to linguistic variations, including typos, dialects, and non-standard language use, is crucial for creating MLLMs that can effectively serve diverse user populations. This challenge becomes particularly important in multilingual and multicultural contexts where language use can vary significantly from standard forms. Recent work on text perturbation strategies [179] could be extended to multimodal settings, providing ways to systematically evaluate and improve robustness to linguistic variations while maintaining cross-modal understanding. This includes developing methods that can handle variations in text while preserving semantic relationships with other modalities.
• Cross-modal Consistency: Ensuring consistent outputs when information across modalities is perturbed or conflicting presents unique challenges that require careful consideration of how different modalities interact and influence each other. The challenge involves developing methods that can maintain coherent outputs even when different modalities provide contradictory or noisy information. Developing evaluation metrics and training objectives for cross-modal consistency is an active area of research, requiring new approaches to quantifying and optimizing the alignment between different modalities under various perturbation scenarios.

6.3.2. Fundamental Limits: When Is Hallucination Unavoidable?

Object hallucination (the generation of plausible-sounding but image-ungrounded content) is routinely described in the MLLM literature as a reliability problem to be reduced via better alignment, RLHF, or visual grounding losses. We use this subsection to distinguish errors that may be reduced by better grounding or connector design from errors that reflect biased or incomplete training distributions; separating these sources clarifies what future architectural work is likely to improve.

Let $p_{\mathrm{data}}(y\mid x_v)$ denote the conditional distribution of ground-truth descriptions y given a visual input $x_v$, and let $p_{\mathrm{train}}(y\mid x_v)$ denote the distribution induced by the web-scraped training corpus. Any model trained by (penalized) maximum-likelihood on $p_{\mathrm{train}}$ converges—in the infinite-data, infinite-capacity limit—to $p_{\mathrm{train}}$ itself, not to $p_{\mathrm{data}}$.

Whenever the training distribution contains spurious co-occurrence patterns (“sky $\to$ blue”, “beach $\to$ umbrella”, “kitchen $\to$ refrigerator”), a Bayes-optimal predictor on $p_{\mathrm{train}}$ will also place probability mass on the spurious completion even when the underlying image does not support it. Under this view, the POPE gap between “random” and “popular” splits (e.g., $86\%$ vs. $79\%$ precision for GPT-4V [144]) can be interpreted as evidence consistent with popular-object priors in the training or evaluation distribution, rather than as proof of a single alignment failure.

This component of the error is better treated as data-induced and reducible mainly through improved data curation, explicit causal/counterfactual supervision, and evaluation designs that separate visual evidence from language priors.

Other hallucination phenomena, by contrast, are more directly architectural. A model that is forced to commit to a next-token distribution before receiving enough visual evidence (as in compressed connectors that feed a small, fixed number of visual tokens into the LM) may hallucinate more often when the visual-token budget is too small, although the relationship between token budget and hallucination rate must be established empirically for each architecture. Resolution scaling, richer connectors, higher-resolution encoders, and grounding-aware training objectives can reduce attribute and relational hallucinations that depend on fine-grained visual grounding [20].

The practical implication is that “zero-hallucination MLLM” is an unrealistic target for open-ended generation: as argued from first principles in Section 2.2, the achievable hallucination rate is constrained by training-distribution co-occurrences, imperfect visual evidence, and representation bottlenecks. Architectural work can reduce measured hallucination under specified data and deployment conditions, but it cannot by itself remove every source of visually unsupported generation. We recommend that future failure mode studies explicitly disentangle these components—for example, by evaluating the same model on (i) a popular-object split whose failures are likely dominated by training-distribution priors and (ii) an adversarial split whose failures are likely dominated by visual-token compression and attention bottlenecks—rather than reporting only a single aggregate hallucination rate.

6.3.3. Handling Missing or Noisy Modalities

Real-world applications of MLLMs often involve scenarios where some modalities are missing or corrupted, making robust handling of incomplete or degraded inputs essential for practical deployment. This challenge is particularly relevant in applications where sensor failures, network issues, or other technical limitations may result in missing or degraded modalities.

Graceful Degradation

Developing MLLMs that maintain reasonable performance with partial or noisy inputs is crucial for ensuring reliable operation in real-world conditions. The ability to gracefully handle degraded inputs while maintaining as much functionality as possible represents a key challenge in multimodal system design:

• Modality Imputation: Developing methods to infer or reconstruct missing modalities could improve robustness by providing substitute inputs when original modalities are unavailable. Recent work on cross-modal generation [161] provides a foundation for addressing this challenge, but adapting these techniques for real-time inference in MLLMs is an open problem. This includes developing efficient methods for generating high-quality imputations that maintain semantic consistency with available modalities while being computationally feasible for real-time applications.

Uncertainty Quantification

Reliable deployment of MLLMs requires well-calibrated uncertainty estimation, particularly in multimodal contexts where different sources of uncertainty can interact in complex ways. Understanding and quantifying uncertainty is crucial for making informed decisions about model outputs and identifying situations where additional information or human intervention may be needed:

• Calibration Techniques: Developing calibration methods for multimodal outputs is challenging due to the diverse nature of different modalities and the need to maintain consistent calibration across various types of outputs. Recent work on temperature scaling [180] needs extension to handle multimodal outputs effectively. This includes developing methods that can provide well-calibrated uncertainty estimates across different modalities while accounting for their unique characteristics and potential interactions.
• Bayesian MLLMs: Exploring Bayesian approaches to uncertainty quantification in MLLMs is a promising direction for providing principled uncertainty estimates in multimodal settings. Techniques like variational inference [181] need adaptation to handle the complexities of multimodal architectures, including the development of appropriate prior distributions and efficient inference methods that can scale to large multimodal models. This includes addressing challenges related to computational efficiency and the handling of different types of uncertainty across modalities.
• Out-of-distribution Detection: Identifying out-of-distribution inputs in multimodal settings is crucial for safe deployment, as it enables models to recognize situations where their predictions may be unreliable. Recent work on contrastive training for OOD detection [182] could be extended to multimodal scenarios, providing ways to identify unusual or potentially problematic inputs across different modalities. This includes developing methods that can effectively detect out-of-distribution samples while considering the joint distribution of multiple modalities.

6.4. Interpretability and Explainability

6.4.1. Visualizing Cross-Modal Attention

Understanding how MLLMs attend to and integrate information from different modalities is crucial for interpretability and trust in these systems. The visualization of attention mechanisms in multimodal contexts presents unique challenges due to the complex interactions between different modalities and the need to represent these interactions in an interpretable manner. This understanding is essential not only for model development and debugging but also for building user trust and enabling effective human oversight.

Attention Map Analysis

Analyzing attention patterns in MLLMs presents unique challenges in multimodal contexts, requiring sophisticated approaches to visualize and interpret how models integrate information across different modalities:

• Multi-head Attention Visualization: Techniques like attention rollout [183] need adaptation for multimodal scenarios to capture complex cross-modal interactions effectively. The challenge lies in developing visualization methods that can meaningfully represent attention patterns across different modalities while maintaining interpretability. This includes addressing questions of how to visualize attention between different types of tokens (e.g., text tokens, image patches, audio segments) and how to represent the hierarchical nature of attention in deep networks. The visualization must be both technically accurate and intuitively understandable to humans, potentially requiring different levels of abstraction for different audiences.
• Temporal Attention Analysis: For video-based MLLMs, visualizing attention over time presents additional challenges that require consideration of both spatial and temporal dimensions. Work on temporal attention could be extended to multimodal temporal data, providing insights into how models integrate information across time and modalities. This includes developing methods to visualize how attention patterns evolve over time, how different modalities influence each other temporally, and how the model maintains coherence across longer sequences. The challenge involves creating visualizations that can effectively represent these complex temporal relationships while remaining comprehensible to human observers.
• Cross-modal Attention Flows: Developing methods to visualize how information flows between modalities through attention mechanisms is an open challenge that requires innovative approaches to representation and visualization. Techniques like attention flow [183] could be adapted for cross-modal settings, providing insights into how information is integrated across modalities. This includes developing methods to track and visualize how information from one modality influences the processing of others, how different modalities contribute to final predictions, and how attention patterns reflect the model’s understanding of relationships between modalities.

Feature Attribution Methods

Extending feature attribution techniques to multimodal settings presents unique challenges that require careful consideration of how to attribute importance across different types of inputs while maintaining consistency and interpretability:

• Gradient-based Methods: Techniques like Integrated Gradients [184] need careful adaptation to handle multiple input modalities consistently while providing meaningful attributions. The challenge involves developing methods that can appropriately scale and compare gradients across different types of inputs, accounting for the different characteristics and scales of each modality. This includes addressing questions of how to normalize attributions across modalities, how to handle interactions between modalities, and how to present these attributions in a way that is meaningful to human observers. The development of these methods must consider both the technical accuracy of the attributions and their practical utility for understanding model behavior.
• Perturbation-based Methods: Methods like LIME [185] require extension to generate meaningful perturbations across different modalities while maintaining semantic coherence. The challenge lies in developing perturbation strategies that are appropriate for each modality while considering cross-modal dependencies and constraints. This includes determining how to generate realistic perturbations that preserve semantic relationships between modalities, how to sample perturbations effectively in high-dimensional multimodal spaces, and how to aggregate results across different types of perturbations. The development of these methods must balance the need for comprehensive exploration of the input space with computational feasibility and interpretability of results.
• Unified Attribution Frameworks: Developing frameworks that provide consistent attributions across modalities is crucial for understanding how different inputs contribute to model decisions. Recent work on unified saliency maps [186] provides a starting point but requires further development for complex MLLMs. This includes creating methods that can meaningfully compare and combine attributions across different modalities, handling challenges related to different scales and characteristics of different input types, and developing presentation methods that can effectively communicate these unified attributions to users. The framework must be both theoretically sound and practically useful for understanding model behavior in real-world applications.

6.5. Concept-Based Explanation Challenges

Multimodal Large Language Models (MLLMs) have expanded the technical design space for AI systems by combining text, images, audio, video, and other modalities within shared reasoning and generation pipelines. Their progress is significant, but practical deployment also exposes reliability, interpretability, privacy, fairness, and security challenges that researchers and practitioners must address with careful evaluation and governance.

6.5.1. Concept-Based Explanations

As MLLMs are applied to more complex tasks, it becomes increasingly important to move beyond low-level feature attribution and towards higher-level concept-based explanations. This shift can make MLLMs more interpretable, trustworthy, and useful in real-world applications.

Multimodal Concept Discovery

The identification of interpretable concepts across modalities is an important challenge in enhancing the explainability of MLLMs. This area of research may help connect model behavior with explanations that human users can inspect and critique.

• Unsupervised Concept Discovery: While techniques like TCAV [187] have shown promise in single-modality scenarios, there is a pressing need to extend these approaches to discover concepts that span multiple modalities. This extension is non-trivial, as it requires algorithms capable of identifying abstract concepts that manifest differently across diverse data types. For instance, the concept of “joy” might be expressed through positive words in text, upbeat melodies in audio, and smiling faces in images. Developing methods that can autonomously discover such cross-modal concepts would significantly enhance our ability to interpret MLLM decision-making processes.
• Cross-modal Concept Alignment: The challenge of aligning concepts across modalities represents a fundamental hurdle in multimodal understanding. This task involves developing algorithms that can recognize when different modalities are expressing the same underlying concept, even when the surface-level representations are vastly different. For example, aligning the visual concept of a “cozy home” with its textual descriptions and associated sounds requires modeling both the individual modalities and their interrelationships. Progress on this challenge could lead to MLLMs that more reliably relate concepts across modalities, enhancing their usefulness in diverse scenarios.
• Hierarchical Concept Learning: The development of frameworks for learning hierarchical concept structures that integrate information across modalities represents a frontier in MLLM research. Such frameworks would need to capture not only the relationships between concepts within a single modality but also how these hierarchies interact and align across different modalities. This approach could lead to more nuanced and context-aware interpretations of multimodal data. For instance, a hierarchical concept structure might relate high-level concepts like “transportation” to more specific concepts like “cars” and “bicycles” across visual, textual, and auditory modalities, enabling richer and more coherent multimodal reasoning.

Compositional Explanations

The complexity of multimodal reasoning demands explanatory approaches that are inherently compositional, capable of breaking down complex decisions into understandable components while preserving the richness of cross-modal interactions.

• Neuro-symbolic Methods: The integration of symbolic reasoning with neural networks, as exemplified by the work of [188], holds significant promise for providing more interpretable explanations in multimodal contexts. These hybrid approaches aim to combine the flexibility and learning capabilities of neural networks with the transparency and logical rigor of symbolic systems. In the context of MLLMs, neuro-symbolic methods could enable the generation of explanations that are both data-driven and logically structured, potentially offering insights into how the model combines information across modalities to arrive at its conclusions. For example, a neuro-symbolic MLLM might explain its classification of a scene as “dangerous” by providing a logical chain of reasoning that incorporates visual cues (e.g., presence of smoke), textual context (e.g., news reports of a fire), and audio information (e.g., sound of sirens).
• Program Synthesis: Techniques for synthesizing programs that explain model decisions, such as those explored by [189], represent a powerful approach to generating interpretable explanations. Extending these methods to multimodal reasoning tasks presents both challenges and opportunities. The goal would be to generate executable programs that can recreate the MLLM’s decision-making process in a human-readable format. This approach could be particularly powerful for explaining complex multimodal interactions, as it would allow for step-by-step tracing of how information from different modalities is combined and processed. For instance, a synthesized program might explain how an MLLM determines the mood of a movie scene by detailing the steps it takes to analyze the visual composition, dialogue sentiment, and musical score, and how it weighs and combines these factors.
• Natural Language Explanations: The generation of coherent natural language explanations that integrate information from multiple modalities remains one of the most significant challenges in making MLLMs interpretable. This task requires not only the ability to reason across modalities but also the capacity to translate that reasoning into clear, concise, and contextually appropriate language. The difficulty lies in capturing the nuances of multimodal interactions without oversimplifying or losing critical information. Advances in this area could lead to MLLMs that provide clearer explanations for their outputs, enhancing their usefulness in fields such as education, healthcare, and decision support systems. For example, a clinical decision-support MLLM might explain a candidate output by referencing specific visual features from medical imaging, relevant passages from the patient’s medical history, and audio cues from recorded patient interviews, all synthesized into a coherent narrative that a healthcare professional can understand and verify.

Table 17. Challenges in multimodal large language models: overview.

Challenge Category	Key Problems	Representative Approaches	Open Problems
Architectural Scalability	Quadratic attention cost; modality-specific vs. unified encoders	Performer [158]; hybrid encoders	Optimal modality fusion; multimodal scaling laws
Cross-modal Alignment	Semantic gap between visual and language spaces	CLIP [7]; contrastive learning	Fine-grained alignment; 3+ modality alignment
Hallucination & Reliability	Factual errors; object hallucination in captions	RLHF; HACL [51]	Grounding-robust generation; calibrated uncertainty
Interpretability	Black-box decision-making; cross-modal attribution	TCAV [187]; attention visualization	Concept-level explanations across modalities
Evaluation & Benchmarking	Saturation of existing benchmarks; metric inadequacy	MMBench; MMMU; HallusionBench	Compositional generalization metrics; cultural diversity
Bias & Fairness	Dataset biases; demographic disparities	Adversarial debiasing; diverse data curation	Intersectional bias; evaluation across languages
Computational Efficiency	Latency, memory, energy, carbon proxy, and inference cost	Quantization; gradient checkpointing; resource cards	Hardware-aware design; adaptive computation; efficiency reporting
Privacy & Ethics	Data leakage; deepfake generation misuse	Differential privacy; watermarking	Consent-aware training; regulation alignment

Note: the bias and fairness challenge listed above is examined from a technical perspective here (data-level causes and measurement), while the privacy and ethics row is included as an operational risk category for training and deployment. The broader ethical dimensions—societal impact, mitigation strategies, privacy governance, and accountability frameworks—are discussed in Section 7.

summarizes the major challenge categories discussed in Section 6, along with representative approaches and open problems.

6.6. Evaluation and Benchmarking

Robust evaluation methodologies and comprehensive benchmarks are essential for measuring progress and identifying limitations in MLLM systems.

6.6.1. Comprehensive Multimodal Benchmarks

The creation of benchmarks that effectively evaluate the capabilities and limitations of MLLMs is a cornerstone challenge in advancing the field. These benchmarks must be carefully designed to capture the full spectrum of MLLM abilities while also probing for potential weaknesses and biases.

Task Diversity

Ensuring that benchmarks cover a wide range of multimodal tasks is essential for comprehensively evaluating MLLM performance:

• Cross-modal Reasoning: The development of tasks that require complex reasoning across modalities represents a frontier in MLLM evaluation. While datasets like CLEVR [190] have set a high standard for visual reasoning tasks, extending this approach to truly multimodal scenarios presents significant challenges. Such tasks might involve, for example, answering questions about a scene that require integrating information from visual, textual, and auditory inputs. For instance, a cross-modal reasoning task might present a video clip of a busy street scene along with a textual description and ambient audio, then ask questions that require the MLLM to synthesize information across all three modalities to infer complex relationships or predict outcomes.
• Open-ended Generation: Creating evaluation protocols for open-ended multimodal generation tasks presents unique challenges, particularly in assessing creativity and coherence across modalities. These tasks might include generating a story with accompanying illustrations, creating a multimedia presentation on a given topic, or composing music with lyrics that match a provided image. The difficulty lies not only in generating content that is coherent within each modality but also in ensuring that the generated elements are semantically aligned and enhance each other across modalities. Evaluation metrics for such tasks must go beyond traditional measures of quality for individual modalities and consider the holistic impact and coherence of the multimodal output.
• Long-form Understanding: Designing benchmarks for long-form multimodal content understanding, such as video story comprehension or multimedia document analysis, addresses a critical gap in current evaluation frameworks. These tasks require MLLMs to maintain context and track complex narratives or arguments across extended multimodal inputs. For example, a benchmark might involve summarizing a lengthy documentary film, requiring the model to integrate visual cues, spoken dialogue, background music, and on-screen text over an extended period. Such tasks test not only the model’s ability to process individual modalities but also its capacity to synthesize information over time and across modalities to form coherent, high-level understandings.

Fairness and Representation

Ensuring that benchmark datasets are inclusive and unbiased is an ongoing challenge that requires continuous attention and innovation:

• Cultural Diversity: Developing strategies for creating culturally diverse multimodal datasets that represent a wide range of global perspectives is crucial for ensuring that MLLMs can perform effectively across different cultural contexts. This challenge involves not only collecting data from diverse sources but also ensuring that the tasks and evaluation criteria are culturally sensitive and relevant. For instance, a truly diverse benchmark might include tasks that require understanding cultural nuances in gestures, idioms, or social cues across different societies, testing the MLLM’s ability to navigate complex cultural landscapes.
• Intersectionality: Designing benchmarks that assess model performance across intersectional categories, considering multiple demographic factors simultaneously, is essential for understanding how MLLMs perform for diverse user groups. This approach recognizes that individuals’ experiences and identities are shaped by the intersection of various factors such as race, gender, age, and socioeconomic status. Benchmarks incorporating intersectionality might, for example, evaluate an MLLM’s ability to understand and generate content relevant to older women from minority ethnic backgrounds, ensuring that the model’s performance is robust across diverse intersectional identities.
• Bias Detection: Creating tools and metrics for identifying and quantifying biases in multimodal datasets and model outputs is a critical component of responsible MLLM development. This challenge involves developing sophisticated analytical techniques that can detect subtle biases across different modalities and their interactions. For example, a bias detection tool might analyze whether an MLLM consistently associates certain visual characteristics with particular personality traits in generated text descriptions, or whether it shows preferences for certain types of voices when generating audio content to match text or images.

6.6.2. Metrics for Multimodal Performance

The development of appropriate metrics to evaluate MLLM performance is crucial for meaningful progress in the field. These metrics must capture not only the quality of outputs in individual modalities but also the coherence and effectiveness of multimodal integration.

Cross-Modal Coherence Metrics

Evaluating the consistency and coherence of MLLMs across modalities presents a complex challenge that requires innovative approaches:

• Semantic Alignment Measures: Developing metrics that assess the semantic alignment between generated content in different modalities is essential for ensuring that MLLMs produce coherent multimodal outputs. These measures must go beyond surface-level similarity to capture deep semantic relationships. For instance, a semantic alignment metric might evaluate how well the emotional tone of generated text matches the mood conveyed by an accompanying generated image or musical piece. This could involve developing new embedding techniques that can represent semantic content across modalities in a comparable space, allowing for quantitative assessment of alignment.
• Perceptual Similarity Metrics: Creating metrics that correlate with human judgments of cross-modal similarity and coherence is crucial for developing MLLMs that produce outputs that are not only technically correct but also intuitively coherent to human users. This challenge involves bridging the gap between computational measures and human perception. Approaches might include developing large-scale human evaluation datasets to train machine learning models that can predict human judgments of multimodal coherence, or creating novel perceptual models that simulate human cross-modal processing.
• Temporal Coherence Measures: For video-based tasks, developing metrics that evaluate coherence over time across modalities is particularly challenging. These metrics must capture not only the moment-to-moment alignment of different modalities but also the overall narrative or thematic coherence across an extended temporal sequence. This might involve developing new techniques for analyzing the temporal dynamics of multimodal content, such as methods for tracking the evolution of themes or emotions across visual, auditory, and textual components of a video over time.

Compositional Generalization Metrics

Assessing the ability of MLLMs to combine concepts across modalities in novel ways is crucial for understanding their potential for creative and flexible multimodal reasoning:

• Systematic Generalization: Designing evaluation protocols that test for systematic generalization in multimodal contexts, similar to SCAN [191] but extended to multiple modalities, is essential for ensuring that MLLMs can apply learned concepts and relationships to novel situations. These protocols might involve creating carefully constructed test sets that require the model to apply known concepts in new multimodal combinations. For example, a test might assess whether a model that has learned to associate certain visual textures with tactile descriptions can generate appropriate cross-modal content for entirely new texture-description pairs.
• Few-shot Composition: Developing metrics to evaluate few-shot compositional abilities across modalities addresses the important challenge of assessing how well MLLMs can quickly adapt to new multimodal tasks with minimal examples. This is particularly relevant for real-world applications where the ability to quickly learn and apply new multimodal concepts is crucial. Metrics in this area might evaluate how effectively a model can learn to generate appropriate audio given a new combination of visual and textual inputs after seeing only a few examples, testing the model’s ability to rapidly compose learned unimodal concepts into novel multimodal outputs.
• Out-of-distribution Composition: Creating benchmarks that assess compositional generalization to novel combinations of modalities or concepts is crucial for understanding the robustness and flexibility of MLLMs. These benchmarks would test the model’s ability to handle inputs or tasks that fall outside the distribution of its training data, particularly in terms of how different modalities are combined. For instance, a benchmark might evaluate how well a model trained on image–caption pairs can handle tasks involving image–audio–text triads, assessing its ability to compose learned bimodal relationships into novel trimodal outputs.

Table 18. Common failure modes in Vision–Language MLLMs.

Failure Mode	Symptom	Root Cause	Mitigation/Evaluation
Object Hallucination	Model names objects absent from the image	Over-reliance on language priors; weak visual grounding	POPE benchmark; HACL contrastive training [51]
Attribute Confusion	Incorrect color, size, or spatial relations	Insufficient cross-modal alignment during training	GQA spatial questions; attribute-specific probing
Counting Errors	Incorrect enumeration of objects	Limited compositional spatial reasoning	CountBench; synthetic counting datasets
OCR Failures	Garbled or missing text in images	Low-resolution training images; tokenization mismatch	TextVQA; DocVQA; high-res image patching
Cross-modal Inconsistency	Text contradicts image content	Modality imbalance in training	HallusionBench; NLI-based consistency checks
Sycophancy	Model agrees with incorrect user assertion	RLHF reward hacking; user-pleasing bias	Adversarial prompting evaluation; calibration metrics
Temporal Confusion	Errors in video ordering or event causality	Limited temporal position encodings	Video-QA benchmarks; next-event prediction tasks
Prompt Sensitivity	Different phrasings yield contradicting answers	Lack of instruction robustness	Paraphrase consistency testing; ensemble voting

summarizes common failure modes and their typical symptoms, root causes, and mitigations.

6.7. Structured Gap Analysis: Explored vs. Unexplored Design Choices

The rapid proliferation of Multimodal Large Language Models has produced a rich but uneven exploration of the architectural design space. Most published models select from a narrow set of visual encoders, language model backbones, connector modules, and training strategies, leaving large regions of the combinatorial space untested. This subsection maps the current coverage of that design space and identifies under-explored architectural combinations that may warrant future investigation. By making the gaps explicit, we aim to guide researchers toward experiments that would maximally expand the community’s empirical knowledge rather than replicate well-trodden configurations.

Table 19. Design-space coverage matrix: Visual Encoder × LM Backbone. Each cell lists representative models that use the given encoder family as the primary visual encoder with the given backbone family. “—” indicates no widely published standalone primary-encoder pairing occupies that cell as of early 2026.

Visual Encoder	LLaMA/Vicuna	Qwen	InternLM	Mistral	Phi	Gemma
CLIP ViT-L/14	LLaVA-1.5 [23]; ShareGPT4V	—	—	—	—	—
CLIP ViT-G/14	—	—	—	—	—	—
EVA-CLIP	MiniGPT-4; LLaVA-NeXT; CogVLM [9]	—	—	—	—	—
SigLIP	PaLI-X (PaLM)	Qwen2-VL; MiniCPM-V [10]	—	—	Phi-3-Vision [192]	PaliGemma
InternViT	—	—	InternVL 1–2	—	InternVL2-Phi3 [192]	—
DINOv2	—	—	—	—	—	—

reveals several structural patterns. The overwhelming majority of published vision–language models pair a CLIP-family encoder (ViT-L or EVA-CLIP) with a LLaMA- or Vicuna-based language model, forming a densely populated cluster in the upper-left region of the matrix. Outside this cluster, coverage drops sharply. SigLIP has begun to receive attention—notably through PaliGemma, Phi-3-Vision [192], and MiniCPM-V [10]—but most SigLIP pairings remain untested. InternViT is almost exclusively used with InternLM inside the InternVL family, leaving its compatibility with other backbone families unknown. Most strikingly, DINOv2, a self-supervised Vision Encoder that captures visual features complementary to CLIP’s contrastive representations (e.g., stronger local texture and shape information), has barely been explored as a standalone primary encoder in widely used MLLM pipelines.

Two further points are worth noting. First, Yi-VL [11] pairs a CLIP ViT-H/14 encoder with the Yi backbone family, which lies outside the six column headers of Table 19 and thus further illustrates how the encoder–backbone matrix is sparsely instantiated once one looks beyond LLaMA/Vicuna and Qwen. Second, multi-encoder systems such as Cambrian-1 [8] do not fit any single cell of the matrix at all: Cambrian-1 combines CLIP, SigLIP, DINOv2, and ConvNeXt into a mixture-of-encoders routed by a Spatial Vision Aggregator, representing an emerging line of work that challenges the single-encoder assumption underlying most of the models tabulated above.

Connector × Training Strategy Coverage

The choice of connector module that bridges the visual encoder to the language model interacts closely with the training recipe employed.

Table 20. Connector module × training strategy coverage. ✓ = well-studied combination with multiple published models; sparse = one or two published instances; — = no known model.

Connector	Contrastive Pre-Train → Generative FT	Generative-Only (No Contrastive Stage)	Instruction Tuning w/o Contrastive Pre-Train
Q-Former	✓ (BLIP-2, InstructBLIP)	sparse	—
Linear / MLP Projection	sparse	✓ (LLaVA-1.5, LLaVA-NeXT)	✓ (ShareGPT4V)
Cross-Attention Layers	✓ (Flamingo)	—	—
Perceiver Resampler	✓ (Flamingo, IDEFICS)	—	—

summarizes which combinations have been validated in practice and which remain open.

Several gaps are notable. The Q-Former connector, popularized by BLIP-2, has almost always been preceded by a contrastive pre-training stage; whether instruction tuning alone (without contrastive alignment) can reach competitive performance with a Q-Former connector is largely unknown. Conversely, the simple MLP projection popularized by LLaVA has been validated primarily in generative-only and instruction tuning regimes but rarely with an initial contrastive stage. Cross-attention connectors and the Perceiver Resampler have been explored almost exclusively within proprietary or semi-open models (Flamingo, IDEFICS) and have seen limited adoption with fully open-source language model backbones such as LLaMA-3, Mistral, or Qwen.

Under-Explored Directions

Beyond the specific cells highlighted in the tables above, several broader gaps deserve attention. First, smaller efficient backbones such as Phi-2/3 and Gemma-2B/7B are a growing but still under-studied direction for multimodal work. Most published MLLMs target 7B–70B parameter language models; systematic studies of how visual encoder quality interacts with backbone scale in the 1–3B range are scarce, despite the practical importance of edge-deployable multimodal models. Second, DINOv2 and other self-supervised encoders capture visual features—such as fine-grained texture, shape, and depth cues—that are complementary to the semantic features learned by CLIP-style contrastive training [7]. Integrating these representations, either as a replacement for or in combination with CLIP encoders, could improve performance on tasks requiring detailed spatial understanding (e.g., robotic manipulation, medical imaging), yet very few published models have attempted this. Third, non-English multimodal training remains a significant gap. While multilingual text-only LLMs have advanced considerably, multimodal pre-training datasets are overwhelmingly English-centric. This limits MLLM applicability in multilingual and low-resource-language settings and risks encoding Anglo-centric visual–linguistic biases. Efforts to construct multilingual image–text corpora and to evaluate MLLMs on non-English visual question answering and captioning benchmarks are urgently needed.

6.8. Synthesis of Challenges and Future Directions

The challenges reviewed in Section 6 are linked rather than isolated. Progress in visual encoders, connectors, and language model backbones has made cross-modal reasoning more capable, but reliable deployment still depends on robustness under distribution shift, reduced hallucination, transparent efficiency reporting, interpretable explanations, privacy safeguards, and fairness-aware evaluation.

Across these categories, a recurring pattern is that performance gains from scaling must be paired with evaluation protocols that expose failure modes under realistic conditions. Concept-based explanations remain especially important because they can connect low-level feature attribution with higher-level descriptions that users can inspect, contest, and apply in domain-specific workflows.

These technical challenges also motivate the ethics discussion that follows. Privacy, consent, bias, provenance, misuse, and equitable access cannot be treated as afterthoughts once an MLLM has been trained; they shape data collection, model evaluation, deployment constraints, and post-deployment monitoring.

Thus, Section 6 should be read as the technical basis for the following ethics section and for the final conclusions: future MLLM work needs stronger grounding, more reproducible evaluation, fairer and more multilingual data, interpretable explanations, and deployment practices that state limitations clearly.

7. Ethical Considerations and Responsible AI

The multimodal capabilities of MLLMs amplify several ethical concerns that already exist in text-only LLMs—bias, privacy, misuse, and equitable access—while introducing new risks specific to cross-modal generation [193]. Biases can compound across modalities: a model may simultaneously produce stereotyped imagery and reinforcing text [194,195]. Privacy risks grow as models ingest and generate increasingly sensitive visual and textual content [196,197].

The potential for deepfake-style misuse across text, image, and audio demands layered safeguards including provenance tracking, watermarking where appropriate, abuse monitoring, and user-facing disclosure, while recognizing that watermarking and metadata-based provenance can fail under editing, conversion, or adversarial removal [198]. Governance frameworks must balance innovation with accountability [199], while ensuring equitable access across languages and communities [200]. The following subsections examine bias mitigation and privacy protection in detail.

7.1. From Technical Bias to Societal Impact and Governance

The previous section (Section 6, “Bias & Fairness” row of the challenges taxonomy) treats bias as a technical phenomenon: a data-distribution problem measurable with fairness metrics and partially correctable by adversarial debiasing, re-weighting, and post-processing. That technical layer is necessary but not sufficient. Here we close the loop by tracing the same phenomenon through two further layers that a systems-oriented review of MLLMs should include: the social impact of biased multimodal outputs on the people who encounter them, and the governance countermeasures required to make the technical and impact layers accountable. We organize this subsection as a three-stage progression (technical challenge → social impact → governance), deliberately avoiding re-explaining the technical details already given in Section 6.

7.1.1. Technical Layer (Recap and Pointer)

Biases in MLLM training data propagate into outputs across all modalities, and in multimodal systems they compound across modalities (a model may simultaneously produce stereotyped imagery and reinforcing captions) [201,202,203]. The technical mitigation catalogue—adversarial debiasing, re-weighting, data augmentation, post-processing calibration, and the standard fairness metrics (demographic parity, equalized odds, disparate impact, calibration within groups, individual fairness)—is presented in the challenges taxonomy (Section 6, Table 17). We deliberately do not re-enumerate these techniques here; the point of Section 7 is to examine the layers that sit above them.

7.1.2. Social Impact Layer: Who Is Affected, How, and Why It Matters

The consequences of biased MLLMs are not symmetric across the population. Four impact channels deserve particular attention for multimodal systems:

• Downstream decision harm. When MLLMs feed into hiring, lending, criminal-justice risk scoring, or clinical triage, modest benchmark-level disparities can translate into materially different admission, loan approval, or treatment outcomes depending on thresholds, deployment context, and human oversight. Unlike text-only bias, multimodal bias can cascade: stereotyped image captions feed into search-ranking and content-moderation pipelines that further reduce visibility for already-underrepresented groups.
• Representational harm. Stereotyped imagery and captions shape how entire communities are perceived by users of generative tools, independent of whether any formal decision is made; this is the “who gets to look normal?” problem.
• Access asymmetry. MLLMs trained overwhelmingly on English and high-resource-language image–caption pairs perform substantially worse in other languages and cultural contexts, concentrating the benefits of the technology in populations already over-represented in training data.
• Cross-modal amplification. A model whose text head is only mildly biased and whose image head is only mildly biased can still produce strongly biased outputs because the two modalities reinforce each other; this interaction is qualitatively absent from text-only or vision-only systems and is under-studied.

These impact channels motivate why purely technical metrics—however well-calibrated—cannot substitute for domain-specific impact assessment: a model can satisfy demographic parity on a benchmark while still producing outcomes that harm the populations the benchmark was meant to protect.

7.1.3. Governance Countermeasures: Making Technical and Impact Layers Accountable

The governance layer is where the technical and impact layers are tied to concrete mechanisms for accountability. We highlight four that are specific to MLLMs rather than being generic AI-governance boilerplate:

• Model cards and multimodal disparity reporting. Model cards for multimodal systems should report per-group performance for each modality plus joint modalities, since, as noted above, cross-modal interactions produce disparities that single-modality metrics miss.
• Pre-deployment impact audits. Domain-specific audits (clinical, legal, educational) that explicitly evaluate representational and access harms, conducted by parties independent of the developer and informed by the affected communities, should be considered a prerequisite for deployment in high-stakes settings.
• Data-source transparency and consent. Because MLLM bias is rooted in web-scale image–text scraping, governance frameworks can require disclosure of the dataset provenance used for pre-training and provide meaningful opt-out or consent mechanisms for content creators whose data is used.
• Continuous monitoring and redress. Bias behaviour of deployed MLLMs drifts as users, prompts, and fine-tuning data change; governance frameworks should specify continuous post-deployment monitoring with clearly defined redress pathways for affected individuals, akin to GDPR Article 22 safeguards for solely automated decisions, including human intervention and contestation rights where applicable.

In short: the technical layer produces metrics; the impact layer tells us which metrics matter and for whom; the governance layer determines whether the metrics are actually measured, reported, and acted on. None of these three layers can stand alone, and presenting only the technical layer—as much of the current MLLM literature does—can leave the analysis incomplete.

Figure 15 summarizes how bias mitigation connects technical controls, impact assessment, and governance accountability.

7.2. Privacy and Data Protection

Multimodal Large Language Models (MLLMs) often rely on large datasets, and these datasets may contain sensitive personal information. This can range from personal images and medical records to social media posts, financial details, and location data. The inclusion of such data creates significant concerns about privacy and data protection, making it crucial for those developing and deploying these models to adopt rigorous ethical practices.

One primary concern is the unintentional leakage of sensitive data. These models can sometimes memorize and reproduce parts of their training data, leading to the potential exposure of private information during interactions with users. For instance, a model trained on medical records might inadvertently generate content containing personal health information, violating confidentiality and privacy regulations [204,205,206].

Another major challenge lies in obtaining proper consent for data usage. In many instances, the data used to train these models might be scraped from public sources without the explicit consent of the individuals involved. This creates ethical dilemmas, especially when individuals are unaware that their data is being used to train AI systems. Ensuring transparent data collection practices and obtaining informed consent is vital for maintaining trust and complying with data protection regulations [204,207,208,209].

The ethical principle of data minimization encourages developers to collect and store only the data strictly necessary for their specific task. By limiting the amount of sensitive information incorporated into these models, developers reduce the potential for harm and align their practices with privacy regulations emphasizing the necessity and proportionality of data collection.

Protecting user privacy in the development and deployment of these models is complex and multifaceted. It involves navigating challenges related to data consent, preventing data leaks, anonymizing sensitive information, and adhering to strict regulations. Developers and deployers must prioritize privacy at every stage of the model lifecycle to ensure that the sensitive information of individuals is respected and protected. Strong privacy safeguards cannot guarantee ethical use by themselves, but they can reduce risk and improve accountability when integrated into dataset governance, training, deployment, monitoring, and incident response [210,211,212].

Operational Privacy Controls for MLLMs

To safeguard user privacy, several techniques can be employed in the MLLM development process:

• Differential Privacy: By introducing carefully calibrated noise into gradients, training procedures, or model outputs, differential privacy helps limit the leakage of sensitive information about individual data points. Its guarantees depend on the specified mechanism, privacy budget, implementation, and composition accounting [213,214].
• Federated Learning: Instead of centralizing all training data in a single location, federated learning enables MLLMs to be trained collaboratively across multiple decentralized devices or institutions. Each participant keeps raw data locally and shares model updates, which may still leak information without protections such as secure aggregation, differential privacy, and attack monitoring. This approach is particularly valuable in domains such as healthcare, where data sharing is restricted by privacy regulations [215].
• Data Minimization and Anonymization: Collecting and retaining only the minimum amount of data necessary for the specific task at hand reduces the risk of privacy breaches. Additionally, techniques such as data anonymization, where personally identifiable information is removed or obfuscated, can help protect user privacy while still allowing MLLMs to learn from the data [216,217].

Table 21. Comparison of privacy-preserving techniques for MLLMs.

Technique	Mechanism	Privacy Guarantee	Utility Cost	Representative Basis/Example
Differential Privacy	Calibrated noise injection into gradients or outputs	Formal $(\epsilon ,\delta )$-DP under a specified mechanism, budget, and composition accounting	Accuracy loss at small $\epsilon$	[213,214]
Federated Learning	Local training; model updates shared	Raw data remains local; updates may still leak information	Comm. overhead	[215]
Data Anonymization	PII removal; k-anonymity; generalization	Reduces re-identification risk	May reduce task performance	[216,217]
Data Minimization	Collect only task-necessary data	Reduces attack surface	Limited training diversity	GDPR Article 5 principle
Secure Aggregation	Cryptographic aggregation of updates	Hides individual updates under protocol assumptions	High computational cost	MPC-based FL protocols
Watermarking	Embed traceable signals in model outputs	Provenance tracking; misuse detection support	Quality and robustness trade-off	Output steganography methods

compares the main privacy-preserving techniques by mechanism, guarantee, utility cost, and representative work.

Figure 16 visualizes the corresponding privacy controls and their trade-offs across the MLLM lifecycle.

7.3. Summary

The ethical landscape of MLLMs spans bias mitigation, privacy protection, transparency, and accountability—challenges that intensify as these models process increasingly diverse and sensitive multimodal data. Addressing these concerns requires integrating ethical considerations into every stage of the development lifecycle, from dataset curation through deployment and monitoring. The following section synthesizes the survey’s findings and identifies concrete open problems for future research.

8. Conclusions

This final section summarizes the advances enabled by Multimodal Large Language Models (MLLMs), the limitations that remain, and the responsibilities involved in their development and deployment.

Figure 17 presents the thematic roadmap of this survey, tracing the progression from foundational concepts to open challenges.

8.1. Summary of Key Findings

This survey has traced the rapid evolution of vision–language MLLMs from their NLP and computer vision foundations through current architectures, training methodologies, applications, and open challenges. Several findings stand out. First, the architectural convergence around a visual encoder–connector–LM backbone pipeline has proven remarkably effective, yet our design-space analysis (Table 19 and Table 20) reveals that most encoder–backbone combinations remain unexplored. Second, our unified benchmark comparison (Table 4) shows that recent open-source models report scores close to or above some proprietary systems on selected perceptual benchmarks, but Section 2.3 emphasizes that such comparisons are behavioral rather than controlled because proprietary data, post-training, and serving stacks are undisclosed. Third, recurring failure modes—spatial reasoning errors, object hallucination, multi-image comparison errors, long-video temporal confusion, and multi-turn context loss—persist across both open and closed systems, underscoring the need for evaluation that goes beyond aggregate accuracy. Finally, critical gaps remain in non-English multimodal resources, efficient processing of long-video and high-resolution inputs, transparent efficiency reporting, and privacy-preserving training methods.

8.2. Open Problems and Future Research Directions

While the preceding sections have summarized the state of the art, several concrete research problems remain open. We identify six directions where progress would most directly advance the field.

Cross-Modal Fairness Metrics

Current fairness evaluations in MLLMs largely inherit single-modality metrics (e.g., demographic parity on text outputs). However, bias can emerge from the interaction of modalities—a model may produce equitable text descriptions of individual faces yet systematically associate certain demographic groups with negative scene contexts when image and text are processed jointly. Designing fairness indicators that operate at the cross-modal level, capturing intersectional biases that surface only through modality combination, is an important and under-explored problem. Concrete technical candidates include causal-intervention-based fairness learning, where counterfactual image edits (e.g., changing perceived demographic attributes) are used to measure and penalize output disparity, and inter-modal contrastive fairness constraints that enforce equitable representation distances across modality pairs during training.

Privacy-Preserving Multimodal Training

Differential privacy (DP) techniques have been applied to text-only LLMs, but extending DP to image–text training introduces unique challenges. Visual data carries identity-revealing information (faces, license plates, handwriting) that is entangled with semantically useful scene content. Research is needed on privacy mechanisms that protect visual identity without significantly degrading scene understanding, as well as on federated learning frameworks that can handle the heterogeneous data distributions typical of multimodal datasets across institutions.

Efficient Attention for Long Video and High-Resolution Images

The quadratic cost of standard self-attention limits current MLLMs to short video clips and downsampled images. Extending multimodal reasoning to hour-long videos or gigapixel medical images will require sub-quadratic attention variants—sparse attention, linear attention, or state-space models—that maintain cross-modal alignment quality. Our gap analysis (Table 19) shows that most efficient-backbone combinations remain unexplored, suggesting a large space for architectural innovation.

Non-English and Low-Resource Multimodal AI

Much vision–language training data is English-centric, and many evaluation benchmarks assume English captions and questions. This creates a significant gap in MLLM capabilities for non-English languages, particularly low-resource ones. Beyond simply building multilingual benchmarks, future work must address the alignment of visual concepts across cultures—for instance, the visual representation of “breakfast” differs fundamentally between East Asian and Western contexts, and a model trained predominantly on English-captioned data may fail to ground culturally specific visual concepts. Technical approaches include cross-lingual visual–semantic transfer via shared visual encoders with language-specific projection heads, and culturally adaptive instruction tuning using community-curated, region-specific image–text datasets.

Hallucination Mitigation Beyond Detection

Object hallucination—where a model describes entities absent from an image—remains pervasive across both open-source and proprietary MLLMs, as documented in our case studies. Current approaches focus on post hoc detection (e.g., POPE benchmarks), but prevention through architectural or training-time interventions is less studied. Promising directions include grounding-aware decoding, contrastive visual instruction tuning, and inference-time visual verification modules that cross-check generated text against image features.

Closing the Open-Source–Proprietary Performance Gap

Our empirical comparison (Table 4) reveals that open-source models now report competitive scores on several perceptual benchmarks, but the comparison is not methodologically controlled: proprietary systems do not disclose training data scale, data quality, post-training cost, or serving hardware. Bridging the remaining reasoning gap on tasks such as MMMU likely requires advances in high-quality instruction tuning data curation, more effective knowledge distillation from larger models, and training regimes that encourage deeper reasoning rather than surface-level pattern matching.

8.3. Lessons from Diminishing Returns and Abandoned Approaches

A candid survey should acknowledge not only promising directions but also approaches that have shown diminishing returns or have been superseded in practice. We highlight three such lessons:

(i)

Naive model scaling. While early MLLM development showed consistent gains from scaling model parameters (LLaVA-7B → LLaVA-13B yields 1–3 point improvements), recent evidence suggests diminishing marginal returns. The jump from 7B to 72B parameters in models like Qwen2-VL and LLaVA-OneVision yields modest improvements on perception benchmarks (MMBench: 83.0 → 86.5 for Qwen2-VL) that may not justify the 10× increase in compute and memory cost. Data quality, instruction diversity, and training recipe appear to matter more than raw scale at current parameter counts, as demonstrated by 7–8B models matching or exceeding much larger predecessors.

(ii)

Q-Former and fixed-query compression. The Q-Former mechanism, introduced in BLIP-2 as an elegant solution to visual-token compression, remains influential but is less common among several recent high-scoring open-source models that use simpler MLP projections (LLaVA-1.5, InternVL, Qwen2-VL, LLaVA-OneVision). In our benchmark comparison, several MLP-based systems report higher scores than the Q-Former-based systems shown here (BLIP-2: 65.0 VQAv2; InstructBLIP: 36.0 MMBench), but these comparisons are not controlled connector ablations because model scale, data, training recipes, and evaluation protocols differ. The fixed query count can act as an information bottleneck that discards spatial detail critical for fine-grained tasks, and the additional cross-attention layers may provide diminishing benefit as encoder–backbone pairings grow stronger.

(iii)

Benchmark saturation. The community’s heavy reliance on VQAv2 and GQA has led to effective saturation: top models cluster within 1–2 points, and further improvements are difficult to distinguish from noise in the absence of statistical significance testing. Several newer models in our comparison either de-emphasize these benchmarks or omit one or both from their primary reporting, signaling a community-wide recognition that they are less discriminative among state-of-the-art systems than newer, more challenging evaluations. The field’s pivot toward benchmarks such as MMMU, DocVQA, and MathVista is a positive development, but these newer benchmarks face their own risks of rapid saturation and data contamination as they become standard training targets.

The trajectory of MLLM research points toward systems that are not only more capable but also more equitable, efficient, and trustworthy. Realizing this trajectory demands sustained collaboration among researchers, developers, and policymakers—grounded in concrete technical milestones and honest assessment of what has and has not worked, rather than aspirational narratives of capability alone.