From 2D Vision–Language Models to Volumetric Medical AI: Large Language Models and Foundation Models for 3D Medical Imaging

Abstract

Multimodal large language models (MLLMs) and vision–language models (VLMs) have rapidly entered medicine, demonstrating promising performance in clinical reasoning, radiology report generation, and visual question answering (VQA). However, many current multimodal architectures and pretrained visual backbones remain fundamentally rooted in two-dimensional (2D) image processing, even though major clinical imaging modalities, including computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and echocardiography, are inherently volumetric or temporal. This narrative review examines the transition from 2D vision–language systems to volumetric multimodal AI, tracing the evolution from 2D and slice- or projection-based approaches through sequential and video-like methods to three-dimensional (3D) vision foundation models and native 3D VLMs/MLLMs. We examine their representational and computational trade-offs, evaluation gaps, and clinically grounded benchmarks. Approaches differ substantially in how they represent and preserve 3D information. Slice- and projection-based methods offer computational efficiency but may discard spatial context, whereas sequential and native volumetric approaches increasingly model relationships across the full imaging study. Recent 3D foundation models and multimodal systems demonstrate the feasibility of reusable volumetric representations and language-enabled 3D image interpretation, but face barriers in computational cost, training-data scale, evaluation methodology, and clinical reliability. Only 53% of Med-Gemini-3D reports were judged clinically acceptable, and natural language processing (NLP) metrics such as BLEU and ROUGE correlate poorly with diagnostic correctness. True 3D multimodal medical intelligence remains in its early stages. Future progress requires efficient volumetric representation strategies, clinically grounded evaluation frameworks, standardized benchmarks, and robust cross-institution validation.

1. Introduction

The emergence of large language models (LLMs) and multimodal foundation models has reshaped the landscape of medical artificial intelligence over the past three years. Generalist systems such as Med-PaLM M and Med-Gemini have demonstrated that a single model can reason across radiology images, pathology slides, electronic health records, and free-text instructions, often achieving strong performance on narrowly defined benchmarks [1,2,3].

In medical imaging, multimodal systems have begun to perform a broad range of tasks that previously required dedicated task-specific models, including report generation, visual question answering, image captioning, and diagnostic reasoning [4,5,6].

Yet a structural mismatch persists between how these systems perceive the world and how clinical imaging is actually acquired. Many early multimodal medical AI systems operate on single or limited two-dimensional (2D) images [4,5,6].

By contrast, many major medical imaging studies are not single 2D images. A non-contrast chest CT may contain hundreds of axial slices that together form a volumetric representation of the thorax. Brain MRI often includes several volumetric sequences, each emphasizing different tissue properties. Optical coherence tomography (OCT) captures many B-scans across the retina, and echocardiography is fundamentally a video stream of cardiac structures in motion. Reducing such data to a single representative slice discards precisely the spatial and temporal context that defines clinical interpretation.

A growing body of work has begun to address this gap. Recent volumetric multimodal systems, including CT-CHAT [7], M3D-LaMed [8], Merlin [9], Med-Gemini-3D [3], Med-2E3 [10], BrainGPT [11], Med3DVLM [12], and CT-Agent [13], have demonstrated that large language model-based architectures can process and reason over full 3D medical imaging studies rather than isolated 2D images.

In parallel, video foundation models such as V-JEPA [14] are being repurposed for ordered medical sequences such as OCT volumes [15], and cardiac video models such as EchoCLIP [16] establish that temporal dynamics can be modeled at scale.

We argue that much of the current generation of multimodal medical LLMs remains shaped by 2D inductive biases, even when applied to volumetric data, which may limit their ability to preserve anatomical consistency, spatial relationships, and longitudinal context. The transition to three-dimensional (3D) multimodal medicine is therefore not merely an engineering optimization but a broader paradigm shift with implications for architecture, dataset construction, evaluation methodology, and clinical safety.

This review focuses on three main issues: (i) the transition from 2D vision–language models (VLMs) to 3D multimodal medical AI (Figure 1); (ii) the architectural and representational challenges involved in preserving spatial and volumetric information; and (iii) the evaluation, validation, and clinical deployment challenges that must be addressed before volumetric multimodal systems can be used reliably in practice.

Figure 1. Evolution of Medical AI Paradigms—from task-specific deep learning to native 3D multimodal systems. Each row shows the characteristic input type, representation strategy, and principal limitation of the corresponding paradigm. The indicated years represent approximate periods of emergence and increased research activity rather than discrete chronological boundaries; these paradigms overlap and continue to develop in parallel.

Figure 1 summarizes this trajectory as six stages of development, from task-specific deep learning models trained on isolated 2D slices, through general-purpose 2D vision foundation models (e.g., contrastive language–image pretraining (CLIP) and Self-distillation with no labels (DINO)), 2D multimodal VLMs that combine image encoders with instruction-tuned LLMs, sequential or video-adapted approaches that process ordered slice stacks, and 3D vision foundation models, to 3D-native multimodal VLMs that ingest full volumes and generate text.

Across this trajectory, approaches increasingly aim to preserve volumetric spatial context, generally at the cost of greater computational and data-curation requirements.

2. Methods

This narrative review examines the evolution of multimodal medical artificial intelligence systems from predominantly 2D vision–language architectures toward native volumetric and 3D medical AI. Consistent with the conventions of a narrative review, we did not pre-register a protocol or perform a formal quantitative synthesis or meta-analysis. A structured literature search was conducted in PubMed and Google Scholar for studies published between January 2023 and April 2026. Earlier foundational studies were identified through citation tracking and targeted supplementary searches and were included to provide historical and technical context; they were not part of the formally screened 2023–2026 corpus. Search terms covered large language models, vision–language models, foundation models, 3D and volumetric medical imaging, radiology report generation, visual question answering, and major volumetric or temporal imaging modalities, including CT, MRI, OCT, and echocardiography. Google Scholar searches were conducted using Publish or Perish [17]. Given the narrative rather than systematic scope of the review, Google Scholar was used as a supplementary discovery source, and screening was limited to the first 30 relevance-ranked results from each query, thereby focusing on the most relevant literature while maintaining a manageable screening scope. Complete database-specific search strings and search dates are provided in Supplementary Table S1.

The database searches identified 649 records, with an additional 16 records identified through citation tracking and supplementary searches. After duplicate removal, 634 unique records remained. Titles and abstracts were screened for relevance to the scope of the review. At this stage, records were excluded if they were non-medical, focused exclusively on text-based models, did not involve medical imaging, were unrelated to the transition toward volumetric or temporal medical AI, or addressed narrow task-specific applications without relevance to the architectural transition examined in this review. Relevant reviews and surveys were retained at this stage to contextualize the field and support citation tracking.

Full texts of potentially relevant studies were subsequently assessed for inclusion in the narrative synthesis. Studies focused exclusively on task-specific segmentation, without broader relevance to volumetric representation learning or multimodal medical AI, were excluded. Among eligible primary studies, systems were selected for detailed discussion when they introduced a distinct architectural or representation strategy, illustrated a major stage in the evolution from 2D to volumetric multimodal AI, or provided evidence relevant to the technical and evaluation challenges examined in this review. A total of 40 studies were ultimately included in the narrative synthesis (Figure S1). Because this is a narrative rather than systematic review, the selection was intended to capture representative and architecturally informative developments rather than provide an exhaustive inventory of all published models.

Included systems were categorized according to imaging modality, dimensionality (2D, slice-based, multi-view, sequential/video-adapted, or native 3D), training strategy, model architecture, primary clinical task, output format, validation approach, and accessibility. The synthesis focused on architectural trends, representational and computational trade-offs, evaluation limitations, and clinically relevant considerations underlying the transition from 2D vision–language systems to volumetric multimodal medical AI.

Use of artificial intelligence tools: During the preparation of this manuscript, the authors used ChatGPT (OpenAI, San Francisco, CA, USA; GPT-5 series; https://chatgpt.com/) and Claude (Anthropic, San Francisco, CA, USA; Claude Opus 5; https://claude.ai/) to support background exploration, provide feedback on manuscript structure, and assist with language refinement, clarity, and readability. Figure 1 and Figure 2 and the graphical abstract were generated using ChatGPT (GPT-5 series) from content and text prompts defined by the authors and were subsequently reviewed and revised by the authors. The authors have reviewed and edited the output and take full responsibility for the content of this publication.

Figure 2. Simplified conceptual workflow for training and inference in medical vision–language models. During training, medical images are paired with textual supervision, such as radiology reports or visual question–answer pairs. A vision encoder extracts imaging features, which are mapped by a connector or projection module into a representation compatible with a pretrained language model (LLM). The visual representation is combined with a textual instruction or question, and the generated output is compared with the target text to optimize trainable components. During inference, the trained system receives a new medical image and instruction or question to generate a report or answer. Specific architectures differ in how volumetric images are encoded and in which components are frozen or fine-tuned.

3. From 2D Vision–Language Models to 3D Medical AI

3.1. Two-Dimensional Medical Vision–Language Models

The first wave of multimodal medical vision–language systems was predominantly 2D in design. These systems adapted general vision–language architectures to biomedical images and showed that image–text instruction following could be useful in medicine before true volumetric reasoning was available.

LLaVA-Med was one of the earliest general-purpose biomedical VLMs. It adapted the LLaVA framework to biomedical images using image–caption pairs from PubMed Central and additional GPT-generated instruction data. The resulting model was designed to handle diverse biomedical image types, including radiographs, microscopy, and histology [4]. Med-Flamingo, an early few-shot multimodal medical assistant, extended OpenFlamingo to medical image–text data drawn from textbooks and publications, enabling medical visual question answering (VQA) with limited in-context examples [5].

Within radiology, chest X-ray interpretation and report generation became one of the earliest and most active application areas for medical VLMs [18]. This was a natural starting point because chest radiographs are common, clinically important, and inherently 2D. R2Gen introduced memory-driven transformer architectures for automated radiology report generation [19], establishing a foundation for subsequent image-to-report systems. More recent multimodal models, including CheXagent [20] and multimodal radiology-report generation frameworks [21,22], expanded beyond report generation to support instruction following, visual question answering, and conversational interaction. Together, these systems demonstrated the potential of transformer-based and later LLM-based architectures to generate clinically plausible, though not necessarily clinically reliable, radiology reports and support visual question answering. However, their reasoning remained largely tied to single-image 2D inputs.

Despite differences in architecture and target application, these early systems shared several common design assumptions. First, most systems relied on visual encoders that were originally trained on large collections of natural images and only later adapted to biomedical data. Common examples included CLIP- and DINO-based architectures. Second, the input was typically a single 2D image, so the model had no native concept of slice ordering, depth, or temporal continuity. Third, evaluation was usually organ- or modality-specific, which may obscure the limits of generalization to volumetric modalities. The main limitation of this paradigm is therefore not its performance on 2D tasks per se, but its inability to reason across multiple slices or preserve volumetric context, limitations that become critical when similar systems are applied to CT, MRI, or OCT.

3.2. Slice-Based Adaptation of Volumetric Imaging

Because most visual encoders were originally designed for 2D inputs, one of the simplest strategies for applying them to volumetric imaging is to represent each study using selected slices or 2D projections. Slice-selection approaches may use a single representative slice, often the middle slice of the scan (middle-slice selection), multiple slices from predefined anatomical locations (fixed anatomical landmark sampling), or slices considered most informative by the model (saliency-guided top-k slice sampling).

A related strategy is to transform part or all of the 3D volume into one or more 2D projection images (projection-based imaging). Maximum-intensity projection (MIP), for example, creates a 2D image by displaying, along each viewing ray, the highest-intensity voxel. MIP is widely used in CT angiography and vascular imaging to visualize contrast-enhanced vessels and other high-intensity structures [23]. Slab MIP restricts this projection to a thick axial or coronal slab, typically 10–30 mm, preserving some local volumetric context while remaining compatible with conventional 2D encoders. Because slab MIPs are already familiar in routine radiology workflows, they may provide a practical bridge between single-slice analysis and fully volumetric AI systems, although their use in contemporary multimodal foundation-model pipelines remains comparatively underexplored.

Taken together, these volume-to-2D adaptation strategies are computationally attractive and substantially lower the barrier to leveraging large-scale 2D pretraining. However, they remain preprocessing compromises. Small lesions outside selected slices or projection slabs can be missed, and inter-slice spatial relationships that are important for assessing mass effect, midline shift, vascular continuity, or disease progression over time may be only partially preserved.

Evidence from ophthalmology illustrates this limitation. In volumetric OCT, models designed to learn relationships across the full image volume, rather than analyzing isolated images or slices, have matched or outperformed leading slice-based approaches [15]. These findings suggest that preserving volumetric context may offer advantages over volume-to-2D adaptation strategies and motivate approaches that process imaging studies as ordered sequences rather than isolated images.

3.3. Sequential and Video-like Approaches

The limitations of slice-selection and projection-based approaches motivated a natural next step: processing all slices in their original order rather than reducing a volume to a small number of representative images. This approach is analogous to video analysis, where information is derived not only from individual frames, but also from the relationships between them. In this paradigm, volumetric imaging studies are treated as ordered sequences rather than collections of isolated images.

Spatial volumetric structure and temporal dynamics represent distinct modeling problems. CT, MRI, and volumetric OCT encode spatial relationships across three-dimensional anatomy, whereas echocardiographic video is inherently spatiotemporal, capturing both structural information within individual frames and temporal changes in cardiac structure and function across frames. Sequential and video-like modeling provides a common computational strategy for capturing relationships across ordered images, whether the ordering represents spatial progression through a volume or temporal progression over time [3,15,16].

MS-VLM exemplifies this sequential strategy by first encoding individual 2D slices and then modeling inter-slice dependencies across the ordered slice representations to construct a volumetric representation [24].

Med-Gemini-3D similarly leverages relationships across slices by repurposing Gemini’s video encoder for CT interpretation, replacing the temporal axis of video with the depth dimension of a scan [3]. The model was trained on 657,719 CT volumes with paired radiology reports across multiple anatomical regions and evaluated for report generation on a subset of 92 non-contrast head/neck CT studies. In ophthalmology, a V-JEPA-based model adapted a video foundation model to volumetric OCT by treating retinal slices as a sequence, enabling joint modeling of anatomical relationships across multiple slices rather than relying on a single B-scan. This approach matched or outperformed RETFound, a widely used slice-based retinal foundation model [15]. In cardiology, EchoCLIP represents an intermediate step between static image and spatiotemporal foundation models. Rather than learning directly from full video sequences, it is trained on randomly sampled 2D echocardiographic frames paired with clinical reports in a shared embedding space. At inference, similarity scores computed from embeddings of multiple sampled frames are aggregated to generate study-level predictions, enabling accurate zero-shot estimation of left ventricular ejection fraction and detection of implanted cardiac devices and prior cardiac procedures on external validation [16].

These examples show why sequential and video-like approaches are an important conceptual advance: they preserve slice order, volumetric continuity, or temporal information that single-image methods discard. Accordingly, adapting video-based models to 3D medical imaging has been proposed as a promising strategy for volumetric interpretation [25]. However, most video encoders were originally optimized for natural video, where the relevant patterns involve camera motion, object movement, and occlusion, whereas 3D medical images have distinct characteristics, including the spatial organization of slices and multi-sequence or multiphase information, that require domain-specific adaptation [25]. This limitation motivates the transition toward models designed specifically to learn volumetric representations.

3.4. Three-Dimensional Vision Foundation Models

The next step beyond sequential and video-like approaches has been to develop models that learn directly from the full 3D volume, rather than treating it as a sequence of slices. These models, known as 3D vision foundation models, are designed to learn general representations of volumetric imaging data such as CT and MRI.

Unlike vision–language models (VLMs), their primary goal is not to generate reports or answer clinical questions, but to understand and represent the imaging data itself. As a result, they can later be adapted for tasks such as disease classification, organ segmentation, image retrieval, or multimodal AI systems.

Conceptually, these models occupy the fifth paradigm in Figure 1. They learn from large collections of imaging data, often without requiring paired reports or question–answer data. This distinguishes them from 3D-native VLMs, which require image–text pairs at scale and therefore face a greater data bottleneck. Most 3D vision foundation models are trained using self-supervised or contrastive learning strategies and are evaluated on downstream classification, segmentation, and retrieval tasks.

Representative 3D foundation models reflect several complementary attempts to build reusable visual understanding of volumetric medical images. Early efforts often focused on specific imaging domains or anatomical regions. For example, TANGERINE [26] was developed as an open-source thoracic CT foundation model for lung-cancer screening at modest computational cost, while LCTFound [27] applied diffusion-based self-supervised pretraining (DDPM) on 2D lung CT slices for nodule detection and characterization. More recently, FM-HCT [28], pretrained on 361,663 non-contrast head CT scans, demonstrated the use of large-scale volumetric pretraining as a transferable backbone for multiple head CT disease-detection tasks.

A second direction aims to move beyond single-organ models toward broader volumetric representations. Pillar-0 [29] represents this trend by spanning multiple CT and MRI modalities across abdomen, chest, head, and breast regions. A related line of work focuses less on a specific organ and more on how 3D representations are learned: VoCo/VoComni [30] introduces a volumetric contrastive objective that exploits consistent anatomical position priors in 3D medical images as a self-supervised learning signal, and SuPreM [31] shows that large-scale supervised pretraining on AbdomenAtlas can transfer across organ segmentation and disease classification tasks.

In MRI, Decipher-MR [32] extends this foundation-model paradigm across diverse anatomies, sequences, and pathologies. Pretrained on 200,000 MRI series using both self-supervised visual learning and image-report contrastive learning, it provides reusable representations for classification, segmentation, localization, visual grounding, and cross-modal retrieval.

Merlin [9] is a hybrid case and provides a bridge to the next stage of the field. Its language head allows it to function partly as a VLM, but its 3D encoder was also designed as a stand-alone foundation model. Segmentation-oriented foundation models such as SAM-Med3D [33] and STU-Net [34], although outside the primary scope of this review, illustrate a complementary role for 3D foundation models. Such models can potentially serve as a preprocessing step for multimodal systems by identifying and isolating relevant anatomical regions, thereby reducing the amount of volumetric information that must be processed.

Together, these models show that 3D foundation modeling is not a single technical path. Some systems focus on clinical deployment in a defined setting, others on learning better general 3D representations, and others on bridging volumetric vision with language. These vision-only or vision-dominant systems provide the visual infrastructure for the 3D-native multimodal systems discussed in Section 3.5, and serve as benchmark references against which future volumetric VLMs should be evaluated.

In practice, 3D vision foundation models can be used in several ways. They may be applied directly to a new task when zero-shot inference is supported, used as fixed visual representations with a lightweight task-specific model trained on top, fully fine-tuned for a specific downstream task, or efficiently fine-tuned using methods such as LoRA [35] or adapter modules when connected to LLMs. These choices affect how much labeled data, computation, and validation are required. They also influence how model outputs are evaluated. As these models become more capable, it is increasingly important to verify that their outputs are truly supported by the underlying imaging findings. Datasets and benchmarks such as RadGenome-Chest CT [36] move evaluation beyond simple disease classification by incorporating structured annotations of findings and anatomical relationships, allowing more detailed assessment of model reasoning.

3.5. Three-Dimensional Multimodal Systems

The most recent generation of systems aims to combine volumetric medical imaging with language. Unlike 3D vision foundation models, which primarily learn visual representations, these systems are designed to interpret full 3D studies and generate clinically meaningful textual outputs, including reports, answers to clinical questions, and multimodal dialog.

At a conceptual level, a medical VLM combines a vision encoder with a language model through a connector or projection module (Figure 2). During training, medical images are paired with textual supervision, such as radiology reports or question–answer pairs, and selected model components are optimized to integrate visual representations with language. During inference, the trained system processes a new image together with a textual instruction or question to generate a report or answer.

A central challenge when applying this framework to 3D medical imaging is the mismatch between the scale of volumetric medical imaging and the context limitations of current LLMs. These studies often contain hundreds of slices or frames, generating far more information than can be processed directly by a language model.

Several design strategies have emerged to address these challenges. One of the earliest approaches was CT-CLIP [7], which learned associations between volumetric imaging studies and their accompanying radiology reports, allowing imaging and text to be represented within a shared framework. CT-CHAT subsequently extended this concept into a conversational interface that enabled direct interaction with CT studies. Together, these systems illustrate one strategy for volumetric multimodal AI: linking imaging data with language.

Other systems focus on a different challenge—how to condense the enormous amount of information contained in a volumetric study into a form that a language model can process efficiently. M3D-LaMed [8], Med-2E3 [10], and Med3DVLM [12] represent different attempts to achieve this goal. M3D-LaMed uses a pretrained 3D vision encoder with spatial pooling to reduce the number of visual tokens, Med-2E3 combines 3D and 2D encoders to capture complementary global and slice-level information, and Med3DVLM uses an efficient 3D encoder and multi-level feature projection to preserve spatial detail while limiting token length. These systems support tasks such as report generation, visual question answering, and image–text retrieval.

BrainGPT [11] represents a different strategy, relying on a 2D vision encoder rather than a native 3D encoder. It processes CT slices individually and uses a Perceiver Resampler to aggregate the resulting visual features before integration with the language model.

While the systems described above primarily focus on report generation or multimodal question answering, some approaches seek to support more complex forms of clinical reasoning. CT-Agent [13], for example, moves beyond a single end-to-end multimodal model by using an LLM-based planner to coordinate anatomy-aware reasoning modules, query processing, and retrieval of relevant examples for report generation and anatomy-level question answering.

Together, these systems represent an early generation of 3D multimodal AI. Although their architecture differ, they all seek to combine volumetric image understanding with language-based reasoning while addressing challenges related to scale, context length, and computational cost; however, the extent to which clinical reliability has been evaluated varies substantially across systems.

4. Current 3D Multimodal Systems: A Comparative Overview

The preceding sections highlight that there is no single architectural solution for volumetric multimodal AI. Current systems differ substantially in how they represent three-dimensional imaging data, integrate visual and language information, and support clinical reasoning. As a result, direct comparison is challenging because models have been developed and evaluated on different imaging modalities, anatomical regions, datasets, clinical tasks, and evaluation metrics.

To provide a practical overview of the current landscape, Table 1 summarizes representative 3D-aware multimodal systems. We use 3D-aware as an umbrella term for systems that incorporate information across a volumetric imaging study, including slice-based, sequential, hybrid, and native 3D approaches. Native 3D refers more specifically to systems whose visual encoder directly processes volumetric inputs using 3D operations or volumetric tokenization, rather than encoding individual 2D slices followed by aggregation. Preference was given to generalizable or multi-task systems and, where available, models with publicly released code or weights. Rather than ranking models according to reported performance, the goal is to compare their capabilities, design choices, and intended use cases. These differences have important implications for computational requirements, scalability, clinical deployment, and the types of tasks that can be supported.

Table 1. Representative 3D multimodal LLM/VLM systems. Columns describe training data, model backbone, imaging modality and anatomy, primary task or intended use, and access conditions.

Model	Year	Training Data	Backbone	Modality/Anatomy	Primary Task/Intended Use	Independent External Validation	Access
CT-CHAT [7]	2024/2026	CT-RATE; 25,692 chest CT studies and reports; >2.7 M derived Q&A pairs	CT-CLIP 3D vision encoder + Llama 3.1 70B	Chest CT	Visual Q&A and report generation	Yes	Code, model weights, and data publicly available
M3D-LaMed [8]	2024	M3D-Data: M3D-Cap (120K 3D image–text pairs) + M3D-VQA (662K instruction–response pairs)	M3D-CLIP-pretrained 3D ViT + Llama-2-7B.	CT (multi-organ)	Visual Q&A, report generation, image–text retrieval across body regions, positioning, and segmentation across body regions	No	Code, model weights, and data publicly available
Med-Gemini-3D [3]	2024	CT-US1: 657,719 CT training volumes with paired radiology reports (multi-anatomy)	Gemini video-based multimodal architecture	Three-dimensional CT (multi-anatomy); evaluated on head/neck CT	Head CT report generation and multimodal reasoning (research prototype)	No	Proprietary-not publicly accessible
Merlin [9]	2024/2026	15,331 abdominal CT studies + 1.8M electronic health record (HER) diagnosis codes + 6M report tokens	I3D ResNet152 + Clinical Longformer; RadLlama-7B for report generation	Abdominal CT	Phenotype prediction; zero-shot findings classification; image-report retrieval; report generation	Yes	Code and model weights publicly available; CT-report dataset available under data-use agreement
Med-2E3 [10]	2024/2025	CT-RATE (non-contrast chest CT) or M3D-Data (whole-body CT); sampled report-generation and VQA training sets	M3D-CLIP 3D encoder + SigLIP-256 2D vision encoder + Phi-3-mini LLM	General CT (multi-organ)	Visual Q&A and report generation	No	Code and model weights publicly available; uses public datasets
BrainGPT [11]	2024/2025	3D-BrainCT corpus: brain CT scans + paired neuroradiology reports; 15,238 training scans (365,928 sampled slices)	Two-dimensional CLIP ViT-L/14 vision encoder + LLaMA-7B	Brain CT	Structured brain CT report generation for common acute neuroradiology findings	Yes	Code and model weights publicly available; 3D-BrainCT available subject to approvals
Med3DVLM [12]	2025/2025	M3D-Data: M3D-Cap (120K 3D image–text pairs) + M3D-VQA (662K instruction–response pairs)	DCFormer-small 3D vision encoder + Qwen2.5-7B-Instruct	CT (multi-organ)	Visual Q&A, report generation, and image–text retrieval	No	Code and model weights publicly available; uses public datasets
CT-Agent [13]	2025/2026	CT-RATE (47,149 training CT volumes) + RadGenome-Chest CT (24,128 training scans; 624,876 region-aligned reports; 1.91M derived QA pairs	Two-dimensional CLIP ViT-B/16 + LLaVA-Med-v1.5 anatomy-aware plugins; DeepSeek-V3 planner	Non-contrast chest CT	Region-guided visual Q&A across 10 anatomical regions; report generation	No	Code and model weights not publicly available; uses public datasets

Note: For models with two years listed, the first indicates the year of the preprint release and the second indicates the year of the peer-reviewed publication. Med-Gemini-3D [3] and M3D-LaMed [8] remained preprints as of the final literature search (30 April 2026); the remaining studies summarized in Table 1 are peer-reviewed publications.

Table 1 focuses on 3D medical LLMs and VLMs that connect volumetric imaging with language to support outputs such as radiology reports, answers to clinical questions, image–text retrieval, or multimodal dialog. For each model, we summarize the reported training data, model backbone, supported anatomy and imaging modalities, primary tasks or intended uses, and accessibility.

Foundation models whose primary purpose is to learn general visual representations rather than interact directly with users are summarized separately in Table 2.

Table 2. Prominent 3D vision foundation models used in volumetric medical AI. These models do not all produce free-text reports natively, but serve as encoders, pretraining backbones, or benchmark references. Columns summarize the underlying training data, model backbone, imaging modality, primary downstream applications, and access conditions.

Model	Year	Training Data	Backbone	Modality	Primary Task/Intended Use	Access
TANGERINE [26]	2025/ 2026	98,588 thoracic chest low-dose CT (LDCT) volumes from SUMMIT + 27 public datasets	Three-dimensional masked autoencoding (MAE) with ViT-Large encoder	Thoracic LDCT	Lung-cancer screening, nodule detection, thoracic disease classification	Code and model weights publicly available; most training data publicly available
LCTFound [27]	2025/ 2025	LungCT-28M: 105,184 lung CT scans (>28M images) from 5 centers; 14 lung diseases + healthy scans	Cross-attention 2D U-Net; diffusion (DDPM) pretraining	Lung CT	Disease diagnosis and prognosis; segmentation; CT reconstruction and enhancement; surgical navigation	Code and model weights publicly available; anonymized subset of pretraining data available on request
Pillar-0 [29]	2025	143,749 CT volumes + 11,543 breast MRI exams from UCSF	Atlas 3D vision encoder + Qwen3-8B text encoder	CT and MRI (abdomen-pelvis, chest, head, breast)	Radiologic findings classification; lung-cancer risk prediction; hemorrhage detection	Code and model weights publicly available; data availability not reported
CT-CLIP [7]	2024/ 2026	CT-RATE; 25,692 chest CT studies and reports; >2.7M derived Q&A pairs	CT-CLIP 3D vision encoder	Chest CT	Zero-shot findings classification; image-report retrieval; encoder for downstream chest CT tasks	Code, model weights, and data publicly available
VoCo [30]	2024/ 2024	~1600 3D CT volumes from 3 public datasets (BTCV, TCIA COVID-19, and LUNA)	Swin-UNETR	Chest and abdominal CT; evaluated on brain MRI	Transfer learning for 3D segmentation and classification; volumetric feature extraction	Code available; pretraining datasets publicly available
Merlin * [9]	2024/ 2026	15,331 abdominal CT studies + 1.8M electronic health record (EHR) diagnosis codes + 6M report tokens	I3D ResNet152 + Clinical Longformer;	Abdominal CT	Phenotype prediction; zero-shot findings classification; image-report retrieval; report generation	Code and model weights publicly available; CT-report dataset available under data-use agreement
Decipher-MR [32]	2025/ 2026	203,233 MRI series from >22,000 studies	Three-dimensional ViT-Base + PubMedBERT	Three-dimensional MRI, multi-anatomy	Classification; cross-modal retrieval; segmentation; localization and visual grounding	Pretrained weights not publicly available; proprietary training data; architecture and methods described
FM-HCT [28]	2025/ 2026	361,663 non-contrast head CT scans	Three-dimensional ViT-Base	Three-dimensional Head CT	Disease detection, including hemorrhage, brain tumor, edema, hydrocephalus, and Alzheimer’s Disease and Related Dementias (ADRD)	Code publicly available; model weights available upon request under institutional agreement; the internal data is unavailable

Note: For models with two years listed, the first indicates the year of the preprint release and the second indicates the year of the peer-reviewed publication. Pillar-0 [29] remained a preprint in the final literature search (30 April 2026); the remaining studies summarized in Table 2 are peer-reviewed publications. * Merlin is included in both tables because it occupies a hybrid position, functioning both as a multimodal system and as a reusable 3D foundation model.

Several patterns emerge from this comparison. Although all systems are designed to process volumetric imaging data, the underlying mechanisms span a wide spectrum, including native 3D encoders (M3D-LaMed, Merlin, Med3DVLM), slice-based multimodal architectures (BrainGPT), hybrid 3D/2D designs (Med-2E3), video-inspired architectures (Med-Gemini-3D), and agentic frameworks that operate over slice-based representations (CT-Agent). Evaluation remains dominated by model-specific datasets and evaluation protocols, with relatively limited independent external validation and clinician-based assessment, although these are incorporated in some systems. Openness also varies: several academic systems provide code, weights, or both, whereas proprietary systems such as Med-Gemini-3D remain inaccessible, complicating reproducibility and direct comparison.

Building on the 3D vision foundation models introduced in Section 3.4, Table 2 summarizes prominent 3D vision foundation models and related multimodal systems that primarily function as visual encoders, pretraining backbones, or benchmark references rather than clinician-facing language models. These models differ in their training strategies, imaging domains, downstream applications, adaptation requirements, and accessibility.

Several patterns emerge from this comparison. Most 3D vision foundation models are designed primarily as reusable visual encoders or pretraining backbones and therefore require task-specific adaptation for downstream applications. In contrast, CT-CLIP [7] and Merlin [9] demonstrate zero-shot capabilities, while Decipher-MR [32] supports zero-shot cross-modal retrieval (image-to-text and text-to-image matching) but uses task-specific decoders for most other downstream tasks. The models also vary in training scale, imaging scope, and accessibility. Validation strategies differed across studies. Independent external validation was reported for CT-CLIP [7], Merlin [9], FM-HCT [28] and Pillar-0 [29]. Other studies evaluated generalizability using different strategies, including transfer learning and fine-tuning on previously unseen datasets for VoCo [30], zero-shot out-of-domain cross-modal retrieval for Decipher-MR [32], domain generalization to unseen datasets for TANGERINE [26] and external validation after task-specific fine-tuning for LCTFound [27]. These differences complicate direct comparisons of model robustness and generalizability across studies. Architecturally, most models in Table 2 learn directly from volumetric inputs using 3D encoders, whereas LCTFound [27] is predominantly pretrained on individual 2D CT images and subsequently adapted to volumetric downstream applications, illustrating that models applied to 3D imaging do not necessarily learn native 3D representations.

5. Evaluation Challenges

Evaluation of 3D multimodal medical AI operates across qualitatively different task types. For classification tasks, including disease detection, severity grading, and finding identification, standard metrics such as area under the receiver operating characteristic curve (AUC), F1 score, sensitivity, and specificity provide interpretable measures of performance and facilitate comparison with existing task-specific deep learning models. These metrics are well-understood by both AI researchers and clinicians and remain appropriate primary measures for tasks with defined binary or categorical labels.

Evaluation of foundation models introduces an additional challenge because performance depends not only on the learned representation but also on the downstream adaptation strategy. Frozen-encoder evaluation, using linear or other lightweight probes, provides a relatively standardized approach for assessing representation quality without modifying the pretrained backbone. Fine-tuning can provide a more complete assessment of downstream adaptability, but comparisons are more sensitive to differences in the extent of parameter updating, optimization strategy, task-specific architecture, and amount of labeled training data. Generalization evaluation requires further distinction between performance on internal held-out data, previously unseen datasets or domains, and independent external cohorts, as well as whether target-domain data were used for adaptation. These differences should be explicitly reported when comparing foundation models across studies, and standardized benchmarks are needed to enable more consistent and meaningful comparisons of representation quality, downstream adaptability, and generalizability.

Evaluation becomes more challenging for generative tasks, where the goal is not to assign a predefined label but to produce free-text outputs. Radiology report generation provides a useful example of this challenge. Traditional text-similarity metrics, such as BLEU and ROUGE, primarily quantify lexical overlap between generated and reference reports and do not necessarily reflect the clinical quality or diagnostic completeness of radiology reports. As a result, reports with high BLEU or ROUGE scores may still contain clinically important errors or omissions, motivating the development of clinically grounded evaluation metrics [11,37,38,39].

More recent evaluation methods aim to address these limitations. General semantic similarity metrics such as BERTScore [40] capture similarity beyond exact word overlap, whereas radiology-specific metrics such as RadCliQ [38] were designed to better reflect clinically relevant report quality and correlate with radiologist assessments. However, these remain indirect measures of diagnostic accuracy. The FORTE framework, introduced with BrainGPT, provides one example of this approach, evaluating structured components of radiology reports such as degree, landmark, feature, and impression separately [11]. Med-Gemini-3D provides an example of clinician-based evaluation of volumetric report generation. In 92 non-contrast head/neck CT studies (27 normal and 65 abnormal), a board-certified radiologist compared the generated and original reports using a rubric based on their implications for patient management. Overall, 53% of generated reports were considered clinically acceptable, while only 17% were rated as equivalent or superior to the original reports. Among reports containing clinically consequential errors, approximately half involved missed findings and half involved hallucinations [3].

More recently, LLM-as-judge approaches, in which a separate LLM evaluates generated reports against reference reports or predefined clinical criteria, have emerged as a complementary strategy for assessing report quality, although their reliability and potential evaluator biases require further validation [11,18,41].

Turing-style evaluations, in which radiologists attempt to distinguish AI-generated reports from human-written reports, provide useful information about report realism. However, even a report that appears indistinguishable from one written by a radiologist may not improve clinical decision-making. Reader studies that compare clinician performance with and without AI assistance therefore provide a more direct assessment of potential clinical utility, although such studies remain uncommon for current 3D systems.

Hallucination, the generation of clinically plausible but unsupported findings, deserves particular attention in LLM-based systems and is not adequately characterized by conventional report-generation metrics such as BLEU and ROUGE. To address this gap, benchmarks such as MedVH [42] and Med-HallMark were developed to evaluate model behavior under challenging or deliberately misleading inputs. Such studies demonstrate that multimodal medical models can generate clinically plausible findings that are unsupported by the underlying images [37,43], highlighting hallucination assessment as an important component of safety evaluation for clinical deployment.

For multimodal question–answering and reasoning tasks, evaluation remains particularly challenging because correct answers depend not only on accurate image interpretation, but also on appropriate clinical reasoning, making it difficult to determine whether errors arise from image interpretation, reasoning, or both. This complexity has limited the development of widely adopted benchmarks that comprehensively assess both capabilities. A further challenge, particularly relevant to 3D systems, is spatial and volumetric consistency. A reliable model should identify the same finding across adjacent slices, localize pathology to the correct anatomical region, and produce stable conclusions when similar studies are acquired repeatedly. RadGenome-Chest CT [36], which links report statements to anatomically grounded segmentation masks, enables evaluation of spatial grounding. However, standardized metrics for volumetric consistency remain underdeveloped.

Taken together, these challenges highlight the need for evaluation frameworks that extend beyond conventional performance and text-similarity metrics to assess clinical validity, hallucination, reasoning performance, spatial grounding, volumetric consistency, uncertainty, subgroup performance, and clinician-AI performance.

6. Future Directions

Evaluation on real-world data and development of standardized benchmarks are important priorities for the field. Most published systems are benchmarked on curated academic datasets; validating them on clinically representative real-world data, spanning scanner diversity, rare pathology, and underrepresented demographics, is a prerequisite for safe deployment. Translation toward clinical use will require a progressive evaluation pathway, moving from retrospective validation to multi-institutional testing, prospective evaluation, clinician-assistance studies, dedicated safety assessment, and ultimately regulatory assessment and post-deployment monitoring. There is currently no broadly adopted multi-anatomy 3D evaluation suite analogous to MedQA [44] in the text domain. A shared benchmark covering visual question answering, report generation, hallucination, spatial grounding, and volumetric consistency across diverse anatomical regions would substantially accelerate reproducible comparison and meaningful assessment of clinical progress.

On the technology side, both video-inspired architectures and native 3D transformers continue to evolve rapidly. Models such as Med3DVLM and M3D-LaMed demonstrate that direct volumetric processing is becoming increasingly feasible [8,12]. At the same time, hybrid systems that combine specialized image-analysis modules, such as segmentation or lesion-detection networks, with LLM-based reasoning are likely to remain important in the near term and may outperform fully general-purpose models for many clinical tasks [13,36]. This modular approach may increasingly evolve toward agentic systems, in which specialized components perform complementary tasks such as anatomical region localization, retrieval, structured finding extraction, and report generation, rather than relying on a single end-to-end model [13,45].

Several technical challenges remain unresolved. The way imaging studies are prepared before being processed by AI models can substantially affect performance, yet these preprocessing steps are often reported inconsistently, making it difficult to compare results across studies. The representation of clinically relevant CT windows is another important preprocessing consideration, as radiologists routinely use different window settings to evaluate different tissues.

Beyond imaging alone, future multimodal systems will likely integrate volumetric studies with longitudinal electronic health records, laboratory results, pathology, and genomic data, enabling patient-level rather than scan-level reasoning.

Within the literature identified in this review, chest CT, abdominal CT, and neuroimaging were more prominently represented, while fewer examples were identified in musculoskeletal and orthopedic imaging, pediatric and fetal imaging, breast MRI, ear, nose, and throat (ENT) imaging, and gastrointestinal imaging. Broader representation of these domains may be influenced by factors such as differences in imaging practices, data availability, greater anatomical and technical complexity, and, in some cases, increased privacy and annotation challenges. Addressing these gaps may benefit from targeted dataset development, broader data sharing, and closer collaboration with subspecialty experts.

Longitudinal analysis, the comparison of imaging studies acquired at different time points in the same patient, represents an important future direction for 3D medical AI. Rather than interpreting a single CT or MRI examination in isolation, these systems would need to identify, quantify, and describe changes that occur over time. This capability extends beyond both single-scan reporting and risk-prediction models, requiring direct comparison of volumetric studies acquired at different time points and the generation of clinically meaningful descriptions of interval change. Such comparisons are central to many clinical decisions, including monitoring disease progression, assessing treatment response, tracking lesion growth, and detecting post-operative complications.

Despite its clinical importance, longitudinal volumetric reasoning remains largely unexplored in current multimodal LLMs, particularly at scale, in part because datasets containing serial imaging studies with expert-annotated descriptions of interval change remain scarce. Beyond data availability, longitudinal analysis also introduces technical challenges, including registering and aligning volumes across time points and handling scanner or protocol drift.

Developing datasets and models capable of comparing studies across time, therefore, represents a major next step for the field.

7. Conclusions

Multimodal medical AI is undergoing a transition from 2D vision–language systems toward models capable of understanding and reasoning over volumetric imaging data. Recent advances have demonstrated that integrating 3D medical images with natural-language interaction is technically feasible and can support tasks such as report generation, question answering, and clinical reasoning.

This transition is being pursued through multiple architectural strategies, ranging from slice-based and video-inspired approaches to native 3D encoders, hybrid 2D/3D designs, and increasingly modular or agentic systems.

Despite this progress, 3D multimodal medical AI remains at an early stage. Many current systems continue to rely on simplified representations of volumetric data, evaluation frameworks remain immature, and evidence of real-world clinical benefit is still limited. Important challenges include robust evaluation, hallucination detection, generalization across institutions and patient populations, and extension to underrepresented clinical domains.

The field is beginning to move beyond proof-of-concept demonstrations and is now entering a phase focused on reliability, rigorous validation, and clinical relevance. How successfully these challenges are addressed will determine the ultimate impact of 3D multimodal AI on medical practice.