Systematic Analysis of Vision–Language Models for Medical Visual Question Answering

Abstract

General-purpose vision–language models (VLMs) are increasingly applied to imaging tasks, yet their reliability on medical visual question answering (Med-VQA) remains unclear. We investigate how three state-of-the-art VLMs—ViLT, BLIP, and MiniCPM-V-2—perform on radiology-focused Med-VQA when evaluated in a modality-aware manner. Using SLAKE and OmniMedVQA-Mini, we construct harmonised subsets for computed tomography (CT), magnetic resonance imaging (MRI), and X-ray, standardising schema and answer processing. We first benchmark all models in a strict zero-shot setting, then perform supervised fine-tuning on modality-specific data splits, and finally add a post-hoc semantic option-selection layer that maps free-text predictions to multiple-choice answers. Zero-shot performance is modest (exact match ≈20% for ViLT/BLIP and 0% for MiniCPM-V-2), confirming that off-the-shelf deployment is inadequate. Fine-tuning substantially improves all models, with ViLT reaching ≈80% exact match and BLIP ≈50%, while MiniCPM-V-2 lags behind. When coupled with option selection, ViLT and BLIP achieve 90–93% exact match and F1 across all modalities, corresponding to 95–97% BERTScore-F1. Our novel results show that (i) modality-specific supervision is essential for Med-VQA, and (ii) post-hoc option selection can transform strong but imperfect generative predictions into highly reliable discrete decisions on harmonised radiology benchmarks. The latter is useful for medical VLMs that combine generative responses with option or sentence selection.

1. Introduction

1.1. Motivation

Advances in artificial intelligence over the past decade have transformed how images and text are processed jointly, from generic captioning to complex visual reasoning. Visual question answering (VQA) was introduced as a benchmark task where models must answer natural-language questions about images, requiring them to combine visual recognition, language understanding, and commonsense reasoning [1]. These vision–language systems have rapidly improved through large-scale pre-training and transformer-based architectures, and are now being deployed in applications that demand high reliability, including healthcare [2].

In medicine, VQA is a key paradigm for building interactive decision-support tools that can answer clinicians’ questions on radiology, pathology, and other diagnostic images. Medical visual question answering (Med-VQA) extends general VQA to clinical images and domain-specific questions, such as “What abnormality is present in this CT scan?” or “Which anatomical structure is enlarged?” [3]. Compared to natural images, medical data present additional challenges: images are often noisy or low contrast, may represent 3D anatomy through 2D slices, and questions contain specialised terminology and implicit clinical context. Recent surveys emphasise that Med-VQA remains substantially more challenging than its general-domain counterpart, with models needing not only accurate perception but also medically grounded reasoning and robust uncertainty handling [3].

1.2. Context

The field has progressed through a series of specialised datasets and architectures. Early Med-VQA benchmarks such as VQA-RAD, PathVQA, and VQA-Med focus on radiology or pathology images with curated question templates and at a relatively small scale [4]. More recent datasets have introduced richer semantics, bilingual annotations, and ontology links. SLAKE is a semantically labelled bilingual Med-VQA dataset that covers CT, MRI, and chest X-ray images with expert-annotated questions, answers, and structured medical concepts [5]. OmniMedVQA, in contrast, aggregates images from 73 public medical datasets into a large, heterogeneous benchmark spanning 12 imaging modalities (including CT, MRI, and X-ray) and over 20 anatomical regions. It is designed to explicitly stress-test Large Vision–Language Models (LVLMs) on realistic, noisy clinical data [6]. Alternative benchmarking frameworks such as BESTMVQA provide unified pipelines for evaluating Med-VQA models across datasets and architectures, highlighting issues of data scarcity and reproducibility.

On the modelling side, most early Med-VQA systems adapt architectures from general VQA, for example via CNN-RNN pipelines or attention-based fusion, and refine them for clinical images and questions. Recent methods introduce asymmetric cross-modal attention networks, graph-based reasoning, contrastive pre-training, and domain-specific pre-training to bridge the semantic gap between medical images and textual questions [7]. In parallel, foundation VLMs such as Vision-and-Language Transformer (ViLT) and Bootstrapping Language-Image Pre-training (BLIP) have shown strong performance on general VQA and related tasks by pre-training on large-scale image–text pairs [8]. These models process images and questions with a unified transformer backbone and can be fine-tuned efficiently for downstream tasks, making them attractive candidates for medical applications where computational resources and deployment constraints matter.

More recently, large multimodal models (LMMs) (Whilst VLMs make use of two modalities (images and text), LMMs can handle multiple modalities.) such as Med-Flamingo, LLaVA-Med, and GPT-4V have been explored for medical imaging tasks, including VQA, report generation, and triaging [2]. Studies on OmniMedVQA and related benchmarks consistently show that even state-of-the-art LVLMs struggle when faced with real clinical images, multi-step reasoning, and fine-grained modality-specific questions [6]. At the same time, new evaluation suites such as HEAL-MedVQA and hallucination-focused analyses indicate that these models can produce confident but incorrect answers, underscoring reliability and safety concerns for clinical deployment [9].

1.3. Scope

Despite this growing body of work, two important gaps remain. First, much of the recent literature concentrates on either bespoke medical vision–language models or very large LVLMs that require substantial resources and domain-specific pre-training [10]. In contrast, comparatively few studies systematically examine how general-purpose VQA models—pre-trained only on natural images (e.g., VQAv2) and web-scale captions—behave on realistic medical VQA benchmarks when applied “as is” and after lightweight supervised adaptation. Secondly, although datasets like SLAKE and OmniMedVQA explicitly include multiple imaging modalities, most evaluation protocols aggregate performance across modalities or treat modality as a descriptive attribute rather than a central axis of analysis [5]. OmniMedVQA does report modality-wise scores for LVLMs, but primarily in zero-shot or prompt-based settings and without detailed investigation of how small to mid-sized VQA models can be fine-tuned on modality-specific subsets. As a result, there is limited understanding of how imaging modality (CT vs. MRI vs. X-ray) interacts with model architecture, fine-tuning strategy, and answer representation in Med-VQA.

In this study, we address these gaps by focusing specifically on modality-aware medical VQA across CT, MRI, and X-ray radiology images. We build a harmonised benchmark by combining the SLAKE dataset with a curated subset of the OmniMedVQA benchmark, restricting both to questions containing CT, MRI, and X-ray images and standardising annotations into a unified schema. On top of this shared dataset curation pipeline, we systematically evaluate three widely used general-purpose vision–language models: (a) ViLT, a classification-style VQA model pre-trained and fine-tuned on VQAv2; (b) BLIP, a generative vision–language model capable of open-ended VQA—without any medical pre-training; and (c) MiniCPM-V-2, an end-side multimodal large language model built on a SigLIP-based visual encoder and a lightweight MiniCPM language backbone—connected via a perceiver resampler. Our work aims to answer the following research questions:

• How well do general-purpose models (like ViLT, BLIP, and MiniCPM-V-2) perform in a strict zero-shot setting on CT-, MRI-, and X-ray-specific Med-VQA subsets constructed from SLAKE and OmniMedVQA data?
• Can supervised fine-tuning on modality-specific training subsets reduce the performance gap between these models and specialised pretrained Med-VQA models? and
• Can a simple post-hoc answer-selection strategy, which matches model-generated outputs to multiple-choice options via semantic similarity, further improve reliability?

To answer the above, we conduct a comprehensive evaluation using a suite of lexical and semantic metrics, including exact match, token-level F1, ROUGE-L, WUPS, and BERTScore, comparing zero-shot, fine-tuned, and post-processed configurations for each modality.

1.4. Contributions

i.

We propose a novel benchmark for medical VLMs derived from well-known datasets and covering three imaging modalities: Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and X-ray radiography. The benchmark includes a reproducible curation pipeline that merges SLAKE [5] and OmniMedVQA [6] into aligned CT, MRI, and X-ray subsets, enabling fair cross-modality comparisons under a unified experimental setup.

ii.

We present a general evaluation methodology and conduct a systematic analysis of general-purpose MiniCPM, ViLT and BLIP models on modality-specific Med-VQA subsets. We quantify the performance limitations of non-medical VQA models in both zero-shot and supervised settings. In addition, our methodology introduces and evaluates a semantic answer-matching procedure that converts free-form outputs into robust multiple-choice predictions, bridging the gap between generative and discriminative VQA modalities.

iii.

Our novel results reveal that MRI-based questions are more challenging than CT and X-ray-based questions. They also demonstrate that modality-aware fine-tuning yields substantial gains in both lexical and semantic correctness without requiring large, fully medical foundation models. These findings offer practical guidance on when and how general-purpose VQA architectures can be safely and effectively adapted for medical visual question answering across radiology modalities.

2. Related Works

This section reviews previous work, starting from general-purpose vision–language models (VLMs) and moving towards specialised medical visual question answering, relevant datasets, and evaluation metrics. We focus on explaining how existing studies handle different imaging modalities, and highlight that systematic, modality-aware analyses across CT, MRI, and X-ray are still limited.

2.1. Foundation VLMs and Multimodal LLMs

Large-scale vision–language pre-training has led to a new generation of foundation models that can be adapted to many downstream tasks, including VQA. ViLT (Vision-and-Language Transformer) is one of the earliest “minimal” vision–language models. It replaces the heavy CNN or region-based visual backbone with a Transformer that operates directly on patch embeddings along with text tokens, achieving up to 60× improvements in speed over previous vision–language pre-training (VLP) models while remaining competitive on VQAv2 (71.3% accuracy on the test set when fine-tuned as a 3129-class classifier) [8]. This efficiency and classification-style formulation make ViLT an attractive baseline for our work, as it can be fine-tuned separately on CT-, MRI-, and X-ray–specific answer vocabularies without prohibitive computational cost.

BLIP (Bootstrapping Language–Image Pre-training) extends this line of work with a multimodal encoder–decoder architecture designed to handle both understanding and generation tasks, including image–text retrieval, captioning, and open-ended VQA [11]. The key idea is to bootstrap cleaner supervision from noisy web image–text pairs using a captioner–filter pipeline. BLIP then achieves state-of-the-art results across several benchmarks, improving VQAv2 performance by around 1.6 VQA points compared with prior models and substantially boosting retrieval and captioning scores. Because BLIP exposes a generative VQA interface, it is well-suited to our setting where free-form answers must later be mapped to multiple-choice options.

A recent narrative review of multimodal large language models (MLLMs) outlines the emerging trend of models that accept text, images, audio, and video as input while generating language outputs [12]. It distinguishes two main architectural families: unified embedding–decoder models, in which image patches are embedded and concatenated with text tokens before being passed through a decoder stack, and cross-modality attention models, in which visual and textual streams interact via dedicated cross-attention layers within the transformer backbone. The review also clarifies how standard vision encoders (e.g., ViT) can convert image patches into representations compatible with language models. Although it is not an empirical study, it serves as a practical guide for practitioners by summarising current MLLM designs (including recent multimodal LLaMA variants) and offering conceptual guidance for designing multimodal pipelines.

Recent multimodal large language models (MLLMs) push these ideas further. MiniCPM-V-2 is a lightweight but strong multimodal LLM built from a SigLIP-400M vision encoder and a 2.4B-parameter MiniCPM language model, connected via a perceiver resampler. It targets efficient “end-side” deployment on consumer GPUs and mobile devices while maintaining competitive performance on benchmarks such as TextVQA and OCRBench [13]. In the biomedical space, Med-Flamingo adapts the Flamingo architecture to medical images, learning from multimodal medical knowledge sources and enabling few-shot generative Med-VQA. On datasets such as VQA-RAD, it improves clinician-rated answer quality by up to 20% over prior baselines [14]. LLaVA-Med follows a different route: starting from a general LLaVA-like model, it performs visual instruction tuning using figure–caption data from PubMed Central and GPT-4–generated instruction-following data, achieving strong performance on multiple Med-VQA datasets in under 15 h of training on eight A100 GPUs [15].

2.2. General-Domain Visual Question Answering

The task of visual question answering (VQA) was formalised by [16], who introduced the VQA dataset with 0.25M images, 0.76M questions, and about 10M answers, combining images from MS COCO and abstract scenes. They framed VQA as an open-ended, natural-language question answering problem over images and showed that solving it requires both visual understanding and language reasoning, beyond what generic captioning systems provide. Early deep learning approaches, such as “Ask Your Neurons”, jointly trained CNN-RNN pipelines to map images and questions to answers, and already highlighted the role of language priors and dataset biases [17].

The later-introduced VQAv2 [18] is a “balanced” version of the dataset, containing 265,016 images and 1.1 million questions. It was constructed so that each question is paired with two similar images that yield different answers. This design explicitly reduces language-only shortcuts and forces models to attend to visual content, becoming the de facto standard benchmark for VQA. Many modern VQA systems—including ViLT and BLIP—treat VQA as a multi-class classification problem over the most frequent 3100 answers, fine-tuning transformer-based vision–language encoders on VQAv2 and often achieving 70–75% accuracy on test-dev. These works establish the baseline practice we adopt in this paper: ViLT is used with a classification head over modality-specific answer vocabularies, while BLIP and MiniCPM-V-2 generate free-form answers that are later mapped to discrete options.

General VQA research has also explored robustness, ambiguity, and reasoning, for example through studies of question rephrasing, compositional generalisation, and causal analysis of language vs. vision contributions [19]. However, most of these analyses are carried out on natural images and everyday questions; how these insights transfer to medical imaging modalities remains underexplored. Models of different AM types were categorized into single-hop and multi-hop. Considering traditional VQA models as well as models with the capacity to read out text in images [20]. Furthermore, [21] gave a brief description of model with attention mechanism which showed excellent results on VQA. Allowing visualizing different analyses as well. [22] also explored a transformer-based encoder–decoder framework for medical visual question answering, combining Vision Transformer (ViT) image features with a transformer text encoder to jointly model visual and textual information. The fused representations were decoded autoregressively to generate answers, and the approach was validated on VQA-RAD and PathVQA, demonstrating improved performance over prior methods.

2.3. Medical Visual Question Answering Datasets and Benchmarks

The development of Med-VQA has been tightly coupled with the creation of specialised datasets. VQA-RAD is one of the earliest radiology-oriented datasets, consisting of 315 images and about 3.5K question–answer pairs collected from the MedPix database [23]. Questions were written by radiologists and include both open-ended and yes/no formats, covering tasks such as identifying anatomical structures, describing abnormalities, and interpreting imaging findings. PathVQA extends Med-VQA to pathology by extracting 4998 images from textbooks and online resources and generating 32,799 open-ended questions via a semi-automated NLP pipeline, with every question manually verified [24]. These datasets demonstrated feasibility but are limited in size and modality coverage. Ref. [25] introduced VQA-CP dataset for constrained experimentation on language bias. They categorized the existing methods into three classes and presented the causes of language bias.

SLAKE (Semantically Labelled Knowledge-Enhanced) is a more recent bilingual Med-VQA dataset that explicitly integrates structured medical knowledge. It contains 642 radiology images (282 CT, 181 MRI, 179 X-rays) and around 14,028 English/Chinese question–answer pairs, annotated with semantic labels and linked to a medical knowledge graph [5]. SLAKE covers multiple body regions and question types (e.g., “abnormality”, “organ”, and “colour”) and provides ontology-level annotations, making it a natural choice for evaluating knowledge-aware Med-VQA systems.

PMC-VQA scales Med-VQA to 227K question–answer pairs and 149K images derived from PubMed Central figures, with 80% of images being radiology. The authors also introduce MedVInT, a generative model pre-trained on PMC-VQA and fine-tuned on benchmarks such as VQA-RAD and SLAKE, achieving substantial improvements over prior models in terms of accuracy and BLEU, demonstrating superior performance [26]. OmniMedVQA pushes this idea further by aggregating data from 73 different medical datasets into a single benchmark spanning 12 imaging modalities (including CT, MRI, X-ray) and >20 anatomical regions, and demonstrates that even strong LVLMs struggle on many of the 200K+ VQA pairs [6].

Beyond datasets, Lin et al. [3] provide a comprehensive survey of Med-VQA methods, highlighting challenges such as data scarcity, domain shift, and the need for trustworthy evaluations. More recently, HEAL-MedVQA introduces a 67K-pair benchmark focusing on hallucination and localisation. In addition to VQA pairs, it includes doctor-annotated segmentation masks and new evaluation protocols that test whether models attend to correct regions or rely on spurious text cues [9].

These datasets collectively show that Med-VQA spans diverse modalities and question types. However, most studies report aggregate performance across modalities or focus on a single modality (e.g., pathology or radiology). The modality labels available in SLAKE, and OmniMedVQA are rarely used to build modality-specific training and evaluation subsets, which is the central focus of our work.

2.4. Medical VQA Models and Multimodal Clinical Assistants

A first generation of Med-VQA models adapted general VQA architectures (CNN-RNN, attention-based fusion) to medical images, sometimes with additional knowledge graphs or external clinical ontologies. Lin et al. point out that these approaches are limited by small datasets and hand-crafted fusion mechanisms. More recent work explores asymmetric cross-modal attention, graph reasoning over anatomical structures, and domain-specific pre-training to better align questions with image regions and structured knowledge [16].

With the rise of LVLMs, models such as LLaVA-Med and Med-Flamingo have become prominent. LLaVA-Med trains a biomedical vision–language assistant by first aligning a general model with biomedical figure captions and then instruction-tuning it with GPT-4-generated question–answer pairs. On three Med-VQA datasets (including VQA-RAD, SLAKE, and PathVQA), LLaVA-Med matches or surpasses previous supervised baselines in terms of overall accuracy, while supporting conversational interactions [27]. Med-Flamingo takes a few-shot learning perspective, extending Flamingo to medical domains and demonstrating improved clinician-rated answer quality (20% gain) on several benchmarks, including custom VQA splits designed to avoid data leakage [28].

More targeted models such as R-LLaVA incorporate region-of-interest prompts or localisation-aware training to better ground answers in specific image areas. R-LLaVA-7B, for example, reports state-of-the-art accuracies on SLAKE-EN (89.5% for open-ended and 90.1% for closed-ended questions) by explicitly modelling spatial focus [29]. HEAL-MedVQA similarly emphasises grounding and hallucination robustness, showing that many current LMMs can answer correctly while still attending to irrelevant or incorrect regions, thus motivating localisation-aware training protocols [9]. Ref. [30] also presented multiple fusion proposals and an abstract fusion framework that can be used with popular VQA models.

Despite recent advances in (Med-)VQA, they rely on large, domain-specialised LVLMs and focus on overall performance and hallucination behaviour rather than systematically dissecting performance per imaging modality. Furthermore, they provide limited guidance on how smaller, general-purpose models can be adapted efficiently to different radiology modalities using modest supervision, which is the problem we address in this work.

3. Materials and Methods

3.1. General Methodology

Dataset harmonisation (input to all pipelines).

We first load the SLAKE dataset, restrict it to English-language questions, and retain the image path, question text, reference answer, and modality. We then load OmniMedVQA-Mini (default test split) and retain the image, question, ground-truth answer, modality, and native multiple-choice options. Modality strings from both datasets are normalised into three canonical classes (CT, MRI, X-ray), and all other modalities are discarded. All records are mapped to a unified schema $\{imagePath,question,canonicalAnswer,modality,datasetId\}$ followed by a quality-control pass to ensure that every referenced image file exists and is readable. This procedure yields three modality-specific subsets—$D_{\left|\mathrm{CT}\right|}=3574, D_{\left|\mathrm{MRI}\right|}=1840,\mathrm{and} D_{\left|\mathrm{X}-\mathrm{ray}\right|}=2372$ image–question pairs—for a total of 7783 supervised examples.

For each modality m, we partition $D_m$ into training and test sets using a single deterministic split with test_size = 0.2 and seed = SPLIT_SEED, implemented via Hugging Face datasets. The resulting frozen split is reused across all models and configurations. Approximate train–test sizes are $\left|\mathrm{CT}\right|=2857/714, \left|\mathrm{MRI}\right|=1472/368, |\mathrm{X}-\mathrm{ray}|=1898/474.$ All answer strings are normalised (lowercasing, trimming, and collapsing whitespace) prior to label construction and metric computation.

Our modality-aware evaluation is defined at the level of image–question pairs rather than patients or studies, as neither SLAKE nor OmniMedVQA-Mini provides patient-level identifiers. Harmonisation performs schema alignment and concatenation only, without generating or duplicating records. The benchmark thus consists solely of original dataset examples grouped by imaging modality (CT, MRI, and X-ray).

Zero-shot benchmarking (Section 3.2.1).

For each modality $m\in \{\mathrm{CT},\mathrm{MRI},\mathrm{X}-\mathrm{ray}\}$, we evaluate the entire subset $D_m$ without any training. We benchmark three general-purpose vision–language models:

• ViLT (dandelin/vilt-b32-finetuned-vqa), a classification-style VQA model over a fixed VQAv2 answer vocabulary;
• BLIP (Salesforce/blip-vqa-base), an encoder–decoder model that generates short free-text answers;
• MiniCPM-V-2 (openbmb/MiniCPM-V-2), a chat-style multimodal large language model using the official model.chat interface.

For every sample, models are run in inference mode (GPU when available, gradients disabled), predictions and reference answers are normalised, and we compute Exact Match (EM), token-level F1, ROUGE-L, WUPS, WUPS@0.9, and BERTScore-F1. These scores form the zero-shot baseline per model and modality.

Supervised fine-tuning regimes (Section 3.3).

For each modality-specific training set and its frozen test set, we apply three supervised fine-tuning strategies.

ViLT (multi-label classification). A modality-specific label vocabulary is built from normalised training answers, defining label2id and id2label mappings with $\mathtt{config}.\mathtt{num}\_\mathtt{labels}=K_m$. A custom collator produces pixel values, tokenised questions, and a multi-hot label tensor $Y\in {\{0,1\}}^{B\times K_m}$. The model is optimised end-to-end using binary cross-entropy with logits over all labels (no PEFT).

BLIP (sequence-to-sequence generation). The BLIP processor constructs encoder inputs from image and question. Ground-truth answers are tokenised, an end-of-sequence token is appended, and padded positions are masked with $-100$. Training minimises the standard autoregressive negative log-likelihood over answer tokens. The full encoder–decoder stack is trained with a schedule matched to ViLT (small batch size with gradient accumulation, ∼20 epochs, learning rate $5\times {10}^{-5}$), in full precision (no fp16).

MiniCPM-V-2 (LoRA-based instruction tuning). Training and test splits are converted to JSONL SFT format, where each record contains an image path and a two-message conversation (user: <image> + question; assistant: normalised answer). LoRA adapters are attached only to the language model self-attention projections (q/k/v) with rank $r=8$, scaling $\alpha =16$, and dropout $0.05$, while the SigLIP-400M vision encoder and base LLM weights remain frozen. Training uses per-device batch size 4, gradient accumulation 16 (effective batch $\approx 16$), 10 epoch, learning rate $2\times {10}^{-4}$, and fp16, following the official OpenBMB training loop. After training, each model is evaluated on its modality-specific frozen test set using the same metric stack as in zero-shot benchmarking. For MiniCPM-V-2, we did not use any custom prompt engineering or task-specific templates. All zero-shot evaluations followed the official OpenBMB model.chat(…) interface with the model’s default chat template and decoding settings, using only the image and the raw question as input.

Multiple-choice reformulation (Section 3.4.1 and Section 3.4.2).

For OmniMedVQA-Mini, the native options field is expanded into option_A–option_D, and a correct_option label is derived by matching the normalised ground-truth answer to the normalised options. For SLAKE, synthetic options are generated using only the SLAKE answer distribution by constructing a top-10 high-frequency answer pool and an extended pool of all unique answers. For each question, three unique distractors are sampled, the correct answer is inserted to form four distinct options, and correct_option $\in \{\mathtt{A},\mathtt{B},\mathtt{C},\mathtt{D}\}$ is recorded. The augmented SLAKE and OmniMed tables are concatenated, and modality-specific train–test splits are recreated using the same seeds and fractions as above; options are carried along but not used in the loss.

Post-hoc semantic option selection (Section 3.4.3).

For each fine-tuned model and modality, inference is run on the MCQ test split to obtain a normalised free-text prediction ${\widehat{y}}_i$ for each example. We compute BERTScore-F1 between ${\widehat{y}}_i$ and each non-empty option $o_j$, and the option with the highest similarity score is selected and treated as the calibrated prediction. Evaluation metrics (EM, F1, ROUGE-L, WUPS, WUPS@0.9, BERTScore-F1) are computed on pairs $\left\{({\widehat{o}}_i,y_i)\right\}$, and MCQ accuracy is reported on the subset with a valid correct_option.

All experiments were implemented in Python 3.12.11 using PyTorch 2.10.0 and the Hugging Face ecosystem. We used the datasets library to access SLAKE and OmniMedVQA-Mini, and Transformers to load the following models: ViLT (dandelin/vilt-b32-finetuned-vqa), BLIP (Salesforce/blip-vqa-base), and MiniCPM-V-2 (openbmb/MiniCPM-V-2). All models were evaluated on test data, with gradients disabled during inference.

3.2. Benchmarking Pipeline

Here we describe the zero-shot benchmarking of ViLT, BLIP, and MiniCPM-V-2 on harmonised CT, MRI, and X-ray subsets derived from SLAKE and OmniMedVQA-Mini. Full implementation details (including libraries, exact loading utilities, and metric code) are provided in Appendices Appendix A.1, Appendix A.2, Appendix A.3, Appendix A.4.

3.2.1. Data Source and Modality-Specific Subsets

We used two publicly available Med-VQA datasets: SLAKE and OmniMedVQA-Mini. SLAKE is a bilingual, semantically labelled dataset with CT, MRI, and X-ray images and physician-annotated question–answer pairs. For this study, we restricted SLAKE to English questions and, for each item, retained the image filename, question text, reference answer, and modality label. OmniMedVQA-Mini is a compact multiple-choice subset of OmniMedVQA that provides, for each image–question pair, four answer options, a ground-truth answer, and a modality string.

Both datasets were mapped to a common schema with four core fields: image path, question, canonical reference answer, and modality. For OmniMedVQA-Mini, the gt_answer field was used as the canonical answer; SLAKE answers were kept as supplied. Modality strings were normalised to three labels—CT, MRI, and X-ray—via a lookup table. Any samples whose modality could not be resolved to one of these three categories were discarded. The harmonised SLAKE and OmniMed tables were then concatenated, with a dataset_id flag added. Three disjoint subsets (CT, MRI, X-ray) were created by filtering on modality. A final quality-control pass verified that all referenced image files existed and were readable. For the benchmarking pipeline, these modality-specific subsets were used exclusively as evaluation sets; no training or validation splits were defined at this stage (see Appendix A.2 for the full curation logic).

To assess any data leakage, we analysed the train–test split at the image level by visualising the distribution of image IDs across all QA pairs for each modality. The resulting plots (Appendix A.3) confirm that while the same image may appear in both splits with different questions, no identical image–question pairs are shared between training and testing.

3.2.2. Model Configurations and Zero-Shot Inference

We evaluated three general-purpose vision–language models in a strict zero-shot setting, i.e., without any additional medical pre-training or task-specific fine-tuning:

ViLT (dandelin/vilt-b32-finetuned-vqa). ViLT is a transformer-based vision–language model that jointly encodes image patches and text tokens in a single transformer stack and is fine-tuned on VQAv2 as a multi-class classifier over a fixed answer vocabulary. In our pipeline, each SLAKE or OmniMed sample is processed by the ViLT processor to obtain joint image–question embeddings. The VQA classification head yields logits over the VQAv2 answer set; the highest-scoring label is mapped back to text using the checkpoint’s id2label mapping and used as the prediction. We do not modify this answer vocabulary, so ViLT can only output answers that appear in VQAv2.

BLIP (Salesforce/blip-vqa-base). BLIP is a multimodal encoder–decoder model designed for both understanding and generation. We use the VQA base checkpoint in generative mode. For each image–question pair, the BLIP visual encoder and text encoder produce a fused representation that conditions an autoregressive decoder. The decoder generates an answer token by token up to a short maximum length; the decoded string is then normalised (lowercasing and whitespace cleanup) and treated as the prediction. BLIP is not constrained to a fixed answer list and can produce paraphrases or clinically equivalent variants of the reference answer.

MiniCPM-V-2 (openbmb/MiniCPM-V-2). MiniCPM-V-2 is a multimodal LLM that combines a SigLIP-based vision encoder with a MiniCPM language backbone. We use the official chat interface. Each sample is formatted as a single-turn “assistant” dialogue where the radiology image and question are passed together to the chat method, which returns a short natural-language answer. This answer is normalised in the same way as the BLIP outputs.

All models were run on GPU (NVIDIA A100 40 GB VRAM), set to evaluation mode, and executed with no_grad/inference-mode contexts. Random seeds were fixed at the start of each run to make iterative processes such as dataset shuffling and pre-processing as deterministic as possible. Further model-loading and processor details are given in Appendix A.1 and Appendix A.2.

3.2.3. Evaluation Protocol and Metrics

For each modality-specific subset (CT, MRI, and X-ray) and each model, the pipeline iterated over all samples, produced a normalised prediction ${\widehat{y}}_i$ from the image and question, and paired it with the normalised reference answer $y_i$, for $i=1,\dots ,N$. We then computed the metrics below. It should be noted that all metrics were computed separately for each model and modality, forming a modality-aware zero-shot baseline for the fine-tuning and post hoc option-selection pipelines. The implementation details for metric computation are summarised in Appendix A.

   Exact Match (EM)

$$\mathrm{EM}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbf{1}[{\widehat{y}}_i=y_i]\times 100,$$

where $\mathbf{1}[·]$ denotes the indicator funcion [31].

Token-level F1. Let $T(·)$ denote the set of tokens in a string. For each prediction–reference pair,

$$P_i={\displaystyle \frac{|T\left({\widehat{y}}_i\right)\cap T\left(y_i\right)|}{\left|T\left({\widehat{y}}_i\right)\right|}},R_i={\displaystyle \frac{|T\left({\widehat{y}}_i\right)\cap T\left(y_i\right)|}{\left|T\left(y_i\right)\right|}},$$

and

$$\mathrm{F}{1}_i=\left\{\begin{array}{cc}{\displaystyle \frac{2P_i{R}_i}{P_i+R_i},}\hfill & P_i+R_i>0,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

We report the mean token-level F1, ${\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\mathrm{F}{1}_i\times 100$.

ROUGE-L. ROUGE-L measures the longest common subsequence overlap between ${\widehat{y}}_i$ and $y_i$, computed using the evaluate implementation [32,33].

WUPS and WUPS@0.9. Wu–Palmer similarity is computed per answer pair using WordNet synsets, yielding a score in $[0,1]$. We report the mean WUPS and the proportion of examples with $\mathrm{WUPS}\ge 0.9$ (denoted WUPS@0.9) [34].

BERTScore-F1. BERTScore-F1 is a semantic similarity score computed between the contextual embeddings of ${\widehat{y}}_i$ and $y_i$ using a pre-trained transformer encoder [35]. We report the average BERTScore-F1 $\times 100$. Appendix A.5 provides the detailed implementation steps for Evaluation the models.

3.3. Fine-Tuning Pipeline

Building directly on the benchmarking pipeline, we next introduced supervised fine-tuning on the harmonised, modality-specific SLAKE + OmniMedVQA subsets. The same curation, CT/MRI/X-ray separation, text normalisation, and evaluation metrics as in the zero-shot experiments were re-used (see Section 3.1 and Section 3.2 and Appendix A). The new elements are: (i) explicit train–test splits per modality, and (ii) model-specific optimisation strategies for ViLT, BLIP, and MiniCPM-V-2. Full implementation details are provided in Appendices Appendix B.1, Appendix B.2, Appendix B.3, Appendix B.4.

3.3.1. Train–Test Splits

After harmonisation, the corpus comprised 7783 image–question pairs across three radiology modalities (3571 CT, 1840 MRI, 2372 X-ray). Each row is a single image–question pair, and because SLAKE and OmniMedVQA contain more QA items than images, the same radiology image can appear in multiple pairs; we therefore treat each pair as an independent supervised example.

For each modality $m\in \{\mathrm{CT},\mathrm{MRI},\mathrm{X}-\mathrm{ray}\}$, we applied the stratified 80/20 train–test protocol with fixed seed described in Section 3.1. The split behaviour and absence of QA-level leakage are illustrated in Figure 1 and documented in detail in Appendix A.5.

3.3.2. ViLT: Multi-Label Classification Fine-Tuning

For ViLT, we trained separate classification heads for CT, MRI, and X-ray on modality-specific answer vocabularies.

Label space. From each training split, we constructed the set of unique, normalised answers $Y_m$ and defined the mappings

$$\mathtt{label}\mathtt{2}\mathtt{id}:Y_m\to \{0,\dots ,K_m-1\},\mathtt{id}\mathtt{2}\mathtt{label}:\{0,\dots ,K_m-1\}\to Y_m,$$

where $K_m=\left|Y_m\right|$. These mappings were injected into the ViLT configuration via config.num_labels = $K_m$, config.id2label, and config.label2id. To avoid undefined labels at evaluation time, an optional filter (FILTER_OOD_TEST_ANSWERS = True) removed test examples whose answers were not present in the corresponding training label set. The fraction of removed test samples is: 0.42% (CT), 1.09% (MRI), and 0.84% (X-ray).

Collator and loss. A custom VQACollator wrapped the ViLT processor, loading images and questions and constructing a multi-hot label tensor $\mathbf{Y}\in {\{0,1\}}^{B\times K_m}$ for each batch, where $y_{ik}=1$ if example i has answer label k. This tensor was passed as labels to ViltForQuestionAnswering, which optimises a binary cross-entropy–with–logits objective over all labels [36]:

$${\mathcal{L}}_{\mathrm{ViLT}}=-{\displaystyle \frac{1}{B{K}_m}}\sum _{i=1}^B\sum _{k=1}^{K_m}\left[y_{ik}log\sigma \left(z_{ik}\right)+(1-y_{ik})log\left(1-\sigma \left(z_{ik}\right)\right)\right],$$

where $z_{ik}$ are the logits for example i and label k, $\sigma (·)$ denotes the sigmoid function, B is the batch size, and $K_m$ is the number of distinct answer labels for modality m.

Training configuration. For each modality, a fresh ViltForQuestionAnswering model was initialised from the VQAv2-fine-tuned checkpoint and trained end-to-end (without Parameter-Efficient Fine-Tuning (PEFT)). Further elaboration is presented in Appendix B.2

3.3.3. BLIP: Generative Sequence-to-Sequence Fine-Tuning

BLIP was fine-tuned as an encoder–decoder model that autoregressively generates the ground-truth answer text. A custom BlipVQACollator used the BLIP processor to obtain encoder inputs from the image and question, and tokenised the target answer sequence $y_i=(y_{i,1},\dots ,y_{i,T_i})$ with an end-of-sequence token. Padded positions were set to $-100$ in the label tensor so that they did not contribute to the loss. The model was optimised using the standard negative log-likelihood objective:

$${\mathcal{L}}_{\mathrm{BLIP}}\left(\theta \right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{t=1}^{T_i}log{p}_{\theta }\left(y_{i,t}\mid y_{i,<t},x_i,q_i\right),$$

where $x_i$ denotes the image, N is the batch size (the number of image–question–answer examples in the current batch), $q_i$ is the question, $y_{i,t}$ is the t-th token in the ground-truth answer sequence of length $T_i$, and $\theta$ is the BLIP parameters [13].

For each modality, a fresh BlipForConditionalGeneration model was initialised from the VQA-pretrained checkpoint and trained using the same high-level schedule as ViLT but in full precision (fp16 = False) to avoid numerical instabilities. At inference time, answers were generated with constrained decoding (maximum 16 new tokens, 3 beams, do_sample = False) before normalisation (see Appendix B.3).

3.3.4. MiniCPM-V-2: LoRA-Based Instruction Fine-Tuning

For MiniCPM-V-2, we followed the official OpenBMB instruction-tuning recipe and applied parameter-efficient fine-tuning with LoRA on the language backbone.

Conversation-style SFT data. For each modality, a helper make_sft_json_for_split converted the Hugging Face dataset split into JSONL files expected by the MiniCPM training scripts. Each example was represented as a single-turn conversation consisting of: (1) an image field pointing to the radiology image on disk; and (2) a conversations list with two messages: a user message containing the question and an image placeholder, and an assistant message containing the normalised ground-truth answer. The resulting JSON files were validated for image-path existence and schema correctness (see Appendix B.4).

LoRA and training setup. A customised finetune.py script extended the official MiniCPM training loop to attach LoRA adapters to the self-attention projections of the language model. We used LoRA with rank $r=8$, scaling $\alpha =16$, dropout $0.05$, and targeted modules $\mathtt{l}\mathtt{l}\mathtt{m}..\ast \mathtt{l}\mathtt{a}\mathtt{y}\mathtt{e}\mathtt{r}\mathtt{s}.<\mathtt{i}>.\mathtt{s}\mathtt{e}\mathtt{l}\mathtt{f}\_\mathtt{a}\mathtt{t}\mathtt{t}\mathtt{n}.(\mathtt{q}\_\mathtt{p}\mathtt{r}\mathtt{o}\mathtt{j}{k}_{p}roj|v_{p}roj\left)\right|$.

The SigLIP-400M vision encoder and the base MiniCPM weights were kept frozen throughout training. The training hyperparameters were:

-

Per-device train and evaluation batch size $=4$;

-

Gradient accumulation steps $=16$ (effective batch size $\approx 16$);

-

Number of epochs $=10$;

-

Learning rate $=2\times {10}^{-4}$;

-

fp16 = True;

-

Evaluation and checkpointing at each epoch with save_total_limit = 1.

The supervised fine-tuning objective is defined as

$${\mathcal{L}}_{\mathrm{SFT}}\left(\theta \right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{1}{\left|A_i\right|}}\sum _{t\in A_i}log{p}_{\theta }\left(y_{i,t}|y_{i,<t},v_i\right),$$

where N denotes the number of training examples, $y_{i,1:T_i}$ is the tokenised full conversation sequence for example i (including both the user prompt and the assistant response), and $A_i\subset \{1,\dots ,T_i\}$ indexes only the assistant answer tokens (i.e., positions not masked by the ignore index, such as $-100$). The term $v_i$ represents the visual tokens extracted from the input image by the vision encoder and provided to the multimodal model alongside text, and $p_{\theta }(·)$ denotes the model’s autoregressive next-token distribution parameterised by $\theta$ [13].

After training, a separate inference script loaded the fine-tuned checkpoint (base model plus LoRA adapters) and generated answers for the modality-specific test JSON files using the MiniCPM chat interface (see Appendix B.4).

3.3.5. Inference and Metrics After Fine-Tuning

Inference and evaluation after fine-tuning were kept identical to the benchmarking stage to enable direct comparison across settings.

• For ViLT, the model produced logits over the modality-specific label set. The predicted answer index was $arg{max}_k\sigma \left(z_{ik}\right)$, which was mapped back to text via the corresponding id2label mapping.
• For BLIP, answers were generated with the same constrained decoding configuration employed during training-time evaluation.
• For MiniCPM-V-2, answers were produced through the chat interface conditioned on both the image and the question, and the resulting text was normalised using the same routine as in the benchmarking stage.

For each modality-specific test set, we computed Exact Match (EM), token-level F1, ROUGE-L, WUPS, WUPS@0.9, and BERTScore-F1, following the metric definitions in Section 3.2.3 and the implementations described in Appendix A.5.

3.4. Option Selection After the Fine-Tuning Pipeline

In the final stage of our study, we extended the supervised fine-tuning pipeline by recasting both datasets into a unified four-option multiple-choice format and adding a post hoc semantic option-selection layer on top of each model’s free-text prediction. The training objective remained unchanged: all models were still trained to generate the ground-truth answer text, while the multiple-choice structure was used only at evaluation time. Through these experiments, we can compare the performance of models, before post hoc option selection, which our fine-tuned pipelines can provide us, and after post hoc option selection performance whose detailed implementation steps are given in Appendices Appendix C.1, Appendix C.2, Appendix C.3, Appendix C.4.

3.4.1. Multiple-Choice Harmonisation of SLAKE and OmniMedVQA

Starting from the harmonised SLAKE + OmniMedVQA corpus and modality-specific subsets, we attached four candidate options (A–D) and a correct_option label to every question–answer pair.

OmniMedVQA options. For OmniMedVQA-Mini, which already provides an options field, the list of candidates was expanded into explicit columns option_A–option_D, truncating or padding to exactly four entries. A correct_option label was derived by normalising the ground-truth answer and comparing it with the normalised option texts. If the answer matched exactly one option after normalisation, the corresponding label (A–D) was stored; otherwise, correct_option was left undefined, and the example was excluded from MCQ-accuracy computation. This procedure preserved OmniMedVQA’s manually curated distractors while aligning them with our normalisation and scoring pipeline.

Synthetic options for SLAKE. SLAKE does not provide candidate options, so we constructed synthetic distractors using only the SLAKE internal answer distribution. Initially, a global answer pool was built by normalising all answers in the full SLAKE split and counting frequencies. From this distribution, we selected the (TOP_K_SYNTH_ANSWERS = 10) most frequent normalised answers and mapped each normal form to a single canonical surface string observed in the dataset, forming a high-frequency pool. In parallel, we constructed an extended pool containing all distinct normalised answer strings to ensure broader coverage.

For a sample with ground-truth answer a, we normalised it to $a_{\mathrm{norm}}$ and invoke sample_unique_distractors($a_{\mathrm{norm}}$, need_k = 3), which operates as follows. First, the high-frequency pool was shuffled and candidates whose normalised form differed from $a_{\mathrm{norm}}$ and from each other were selected. Second, if fewer than three distractors were found, the procedure fell back to the extended pool, always enforcing uniqueness.

To assess the hardness of the synthetic distractors constructed for SLAKE, we measured the semantic similarity between each distractor and the ground-truth answer using BERTScore-F1, where higher similarity indicates greater confusability. Across approximately 6000 distractor instances, the similarity distribution has a mean and median of 0.56 (SD 0.14; IQR 0.48–0.67), with a pronounced upper tail (90th percentile 0.76, maximum 0.85) that reflects the presence of genuinely hard, semantically close alternatives. At the same time, lower-similarity cases (minimum 0.20) correspond to easier but still valid distractors drawn from the dataset’s answer vocabulary. Overall, this distribution shows that the synthetic options span a realistic range of difficulty and are neither trivial nor near-duplicate, yielding a balanced, non-degenerate MCQ setting suitable for post hoc calibration without artificially simplifying the task.

The three distractors were randomly shuffled, the correct answer string was inserted at a random position among four slots, and we asserted that all four normalised options were distinct. The final candidates were stored as option_A–option_D, and the position of the correct answer was encoded as correct_option $\in \{\mathtt{A},\mathtt{B},\mathtt{C},\mathtt{D}\}$.

This procedure equipped every SLAKE item with four plausible, dataset-consistent options without relying on external knowledge or model-generated distractors (see Appendix C.1 for detailed implementation).

3.4.2. Combined MCQ Dataset per Modality

For ViLT and BLIP, the augmented SLAKE and OmniMedVQA tables—containing image, question, answer, modality, dataset_id, option_A–option_D, and correct_option—were concatenated into a single dataset (combo). We then:

• Filtered combo by modality (CT, MRI, X-ray), as in previous stages;
• Applied the same stratified random split as in Section 3.2 (typically 80/20 train–test, with TRAIN_PERCENT = TEST_PERCENT = 1.0 for these runs);
• Retained all option columns and the correct_option label in both training and test splits, even though options were not used by the loss during training.

The fine-tuning configuration (loss functions, optimisers, and learning-rate schedules) was identical to the Section 3.2.3 The only change was that each example now carried a consistent four-option MCQ scaffold that was later used during evaluation (Appendix C.2).

3.4.3. Semantic Option Selection with BERTScore

After fine-tuning on the MCQ-augmented data, we introduced a post hoc decision layer that maps each model’s free-text prediction to the most semantically similar option.

For each test example, we had the model’s normalised free-text prediction $\widehat{y}$, and a set of non-empty options $\{o_1,\dots ,o_4\}$ and the dataset-provided correct_option label.

We computed BERTScore-F1 between $\widehat{y}$ and each candidate option $o_j$ using a pre-trained bert-base-uncased encoder. Let $s_j={\mathrm{BERTScore}}_{\mathrm{F}1}(\widehat{y},o_j)$. BERTScore derives contextual embeddings for both strings, matches tokens by cosine similarity, and computes precision, recall, and F1 over these soft matches, providing a semantic similarity measure that is robust to paraphrasing.

If the BERTScore computation failed for any reason (for example, because the BERTScore model cannot be loaded or executed correctly in the present environment), we fell back to a lexical token-level F1 between the normalised prediction and option texts:

$$\mathrm{F}{1}_{\mathrm{lex}}(p,t)={\displaystyle \frac{2\left|T\right(p)\cap T(t\left)\right|}{\left|T\right(p\left)\right|+\left|T\right(t\left)\right|}},$$

where $T(·)$ denotes the set of tokens in a string. The selected option was then

$$(\widehat{\mathsf{\ell }},\widehat{o})=arg\underset{{\widehat{\mathsf{\ell }}}_j\in \{\mathtt{A},\mathtt{B},\mathtt{C},\mathtt{D}\},{\widehat{o}}_j}{max}{s}_j,$$

returning both the option label $\widehat{\mathsf{\ell }}$ and its text $\widehat{o}$. If no valid options were available (e.g., all options are empty), we retained the original free-text prediction and marked the MCQ label as undefined. However, there was no such sample that we could mark as undefined. We also included an error analyses for post hoc option selection by computing the BertScore-F1 between the model’s raw free text and the selected option text. The plots are provided with the generic implementation of this logic in best_option_for_prediction (see Appendix C.3).

3.4.4. Integration with ViLT, BLIP, and MiniCPM-V-2

The option-selection layer is integrated uniformly across all three models, without modifying their respective training objectives.

ViLT (classification-style VQA). For each test example, a raw prediction was obtained by applying the fine-tuned classifier, taking the argmax over the output logits, and decoding the resulting index via id2label. This raw text prediction was passed to best_option_for_prediction together with the four stored options. We retained both the raw prediction and the selected option text. Evaluation metrics (EM, F1, ROUGE-L, WUPS, WUPS@0.9, and BERTScore-F1) were computed between the selected option text and the normalised ground-truth answer, while MCQ accuracy was obtained by comparing the selected label $\widehat{\mathsf{\ell }}$ with correct_option.

BLIP (generative VQA). A free-text answer was generated using the same constrained decoding configuration as in Section 3.2.3. The decoded and normalised string played the role of $\widehat{y}$ and was processed by the same best_option_for_prediction function. The resulting option texts and labels were then used for metric computation in exactly the same manner as for ViLT.

MiniCPM-V-2 (chat-style LVLM). The LoRA-based instruction fine-tuning remained unchanged. A dedicated evaluation script loaded the fine-tuned checkpoint, iterated over the modality-specific JSON files (containing image paths, conversation fields, options, and correct_option), extracted the image and question, and called the official model.chat interface to produce $\widehat{y}$. The script then applied best_option_for_prediction to the stored options, recorded both the raw and selected predictions, and computed the semantic metrics (EM, F1, ROUGE-L, WUPS, WUPS@0.9, and BERTScore-F1) together with MCQ accuracy.

Overall, this pipeline preserves a strictly free-text learning objective during training, while evaluation is performed in a multiple-choice space via semantic snapping onto dataset-consistent options. This design enabled us to quantify the performance gains achievable through simple, post hoc calibration on top of fine-tuned medical VQA models.

4. Results

4.1. Baseline Results

In the zero-shot setting, all three models were evaluated directly on the harmonised SLAKE–OmniMedVQA without any medical fine-tuning; see

Table 1. Modality-wise benchmark results on SLAKE+Omni Med VQA (zero-shot).

Dataset	Modality Type	Exact Match	Average F1	Rouge-L	WUPS	WUPS@0.9	BertScore-F1
Salesforce/blip-vqa-base
SLAKE—Omni Med VQA	CT—3571	24.56	25.02	25.07	64.87	28.48	72.42
	MRI—1840	18.86	19.88	20.15	50.62	23.48	65.59
	X-ray—2372	23.19	23.69	26.39	65.58	28.54	70.59
dandelin/vilt-b32-finetuned-vqa
SLAKE—Omni Med VQA	CT—3571	23.55	23.97	23.99	55.08	24.84	68.21
	MRI—1840	16.47	16.66	16.93	46.10	19.78	61.51
	X-ray—2372	18.59	18.89	18.92	54.88	24.24	63.47
MiniCPM-V2 (OpenBMB)
SLAKE—Omni Med VQA	CT—3571	0.00	3.16	6.88	70.47	18.96	35.49
	MRI—1840	0.00	1.96	7.68	58.54	13.86	36.39
	X-ray—2372	0.00	2.92	7.45	67.93	21.29	35.90

.

Across modalities, BLIP showed the strongest overall zero-shot performance. Its exact match accuracy was 24.6% on CT, 18.9% on MRI, and 23.2% on X-ray, slightly higher than ViLT (23.6% CT, 16.5% MRI, 18.6% X-ray). Token-level F1 followed the same pattern, with BLIP reaching ≈25% F1 on CT and ≈20–24% F1 on MRI/X-ray, whereas ViLT remained a few points lower on each modality. Both models exhibited only moderate lexical overlap, confirming that generic VQA pre-training alone is insufficient for reliable medical VQA.

Semantic metrics showed a similar trend. BLIP achieved WUPS scores of 64.9% (CT), 50.6% (MRI), and 65.6% (X-ray), and BERTScore-F1 of 72.4%, 65.6%, and 70.6%, respectively. ViLT’s WUPS and BERTScore were consistently lower (e.g., WUPS 55.1% CT, 46.1% MRI, 54.9% X-ray; BERTScore-F1 68.2%, 61.5%, 63.5%), indicating weaker semantic alignment to the gold answers even when predictions were partially correct.

In contrast, MiniCPM-V-2 performed poorly in strict lexical terms under pure zero-shot usage. Exact match was 0% on all three modalities, with F1 in the 2–3% range. Nonetheless, its semantic metrics showed reasonable performance: WUPS reached 70.5% (CT), 58.5% (MRI), and 67.9% (X-ray), and BERTScore-F1 was around 35–36%, suggesting that the model often produced medically related but lexically mismatched answers that failed the strict EM criterion.

Overall, the zero-shot results show that (i) BLIP transfers best among the three general-purpose models, (ii) ViLT retains some useful cross-domain signal but lags behind BLIP in both lexical and semantic alignment, and (iii) MiniCPM-V-2’s raw generations are semantically related but too unconstrained to score well under exact-match evaluation. These findings motivate the subsequent modality-specific fine-tuning and post hoc option-selection pipelines.

4.2. Supervised Fine-Tuning Results

After modality-specific fine-tuning on the harmonised SLAKE–OmniMedVQA subsets, our three models showed substantial gains over their zero-shot performance; see

Table 2. Modality-wise fine-tuning results on SLAKE + Omni Med VQA after fine-tuning.

Dataset	Modality Type	Exact Match	Average F1	Rouge-L	WUPS	WUPS@0.9	BertScore-F1
Salesforce/blip-vqa-base
SLAKE—Omni Med VQA	CT—714	81.72	84.33	84.79	91.78	83.26	93.51
	MRI—368	73.55	79.39	79.96	78.43	66.67	91.31
	X-ray—474	71.28	75.07	82.08	88.65	78.51	92.17
dandelin/vilt-b32-finetuned-vqa
SLAKE—Omni Med VQA	CT—714	82.70	84.57	84.35	91.58	82.98	93.93
	MRI—368	76.03	80.03	80.29	77.37	64.74	91.83
	X-ray—474	78.09	80.95	81.25	88.50	77.87	91.50
MiniCPM-V2 (OpenBMB)
SLAKE—Omni Med VQA	CT—714	79.97	81.52	81.68	89.13	78.43	93.00
	MRI—368	68.48	74.30	76.23	73.36	61.14	88.73
	X-ray – 474	59.70	65.52	70.20	78.66	64.14	85.88

.

4.2.1. Overall Effect of Fine-Tuning

Exact-match accuracy on CT increased from ≈24% to ≈82–83% for BLIP and ViLT, and from 0% to ≈80% for MiniCPM. On MRI, accuracies rose into the mid-70% range for BLIP and ViLT and high-60% for MiniCPM. On X-ray, BLIP and ViLT reached ≈71–78% EM, while MiniCPM improved to ≈60%. All secondary metrics (F1/ROUGE-L/WUPS/BERTScore) followed the same trend, typically landing in the mid-70s to low-90s after fine-tuning.

4.2.2. Comparison Between Models

ViLT achieved the strongest strict performance overall. CT: EM $\approx 82.7\%$ with all other metrics in the mid-80s to low-90s. MRI: EM $\approx 76.0\%$, slightly higher than BLIP (73.6%) and MiniCPM (68.5%). X-ray: EM $\approx 78.1\%$, again ahead of BLIP (71.3%) and MiniCPM (59.7%).

BLIP was competitive with ViLT, especially in semantic metrics. CT: EM 81.7%, with BERTScore-F1 $\approx 93.5\%$. MRI and X-ray: EM 73.6% and 71.3%, with BERTScore-F1 $\ge 91\%$ across modalities, indicating strong semantic alignment even when exact strings differ.

MiniCPM-V2 (OpenBMB) benefited markedly from fine-tuning but remained behind the VQA-specialised models. CT: EM $\approx 80.0\%$, secondary metrics mostly in the low-80s to high-80s, BERTScore-F1 $\approx 93.0\%$. MRI: EM $\approx 68.5\%$; X-ray: EM $\approx 59.7\%$, with corresponding semantic scores below those of BLIP and ViLT.

4.2.3. Modality Trends

• CT emerged as the easiest modality after fine-tuning, with all models achieving their highest accuracies and BERTScore values there.
• MRI remained the most challenging: while ViLT and BLIP both exceeded 73% EM, their other metrics dipped compared with CT, and MiniCPM lagged further behind.
• X-ray performance was intermediate, but again consistently higher for ViLT and BLIP than for MiniCPM.

In summary, modality-specific supervised fine-tuning transforms all three backbones from marginal zero-shot baselines into reasonably strong medical VQA models, with ViLT leading on strict accuracy, BLIP excelling in semantic similarity, and MiniCPM showing solid but clearly lower performance across all modalities.

4.3. Fine-Tuning with Post-Hoc Option Selection

Finally, we retained the fine-tuned models from Section 4.2 but replaced raw free-text evaluation with a post-hoc option-selection step: each generated answer was mapped to the most semantically similar multiple-choice option before scoring. This substantially increased reliability for all three backbones, especially the generative ones; see

Table 3. Modality-wise fine-tuning with post-hoc option selection on SLAKE+Omni Med VQA.

Dataset	Modality Type	Exact Match	Average F1	Rouge-L	WUPS	WUPS@0.9	BertScore-F1
Salesforce/blip-vqa-base
SLAKE—Omni Med VQA	CT—714	90.76	91.22	91.48	93.95	87.25	96.48
	MRI—368	87.77	87.77	87.77	78.04	66.58	95.65
	X-ray—474	93.25	93.61	93.61	91.33	81.65	97.57
dandelin/vilt-b32-finetuned-vqa
SLAKE—Omni Med VQA	CT—714	92.12	92.45	92.59	94.67	89.03	96.72
	MRI—368	92.84	92.84	92.84	79.81	70.25	96.87
	X-ray—474	92.34	92.87	92.86	90.63	82.34	96.54
MiniCPM-V2 (OpenBMB)
SLAKE—Omni Med VQA	CT—714	80.95	82.77	82.91	90.00	79.69	92.86
	MRI—368	76.63	80.47	81.35	75.71	64.95	91.27
	X-ray—474	73.84	78.42	78.87	85.39	73.42	89.81

.

ViLT (dandelin/vilt-b32-finetuned-vqa) had a very stable performance across CT, MRI, and X-ray data.

Compared with the plain fine-tuning stage, option selection consolidated the ViLT predictions and lifted it into the ∼92–93% EM and F1 range on all three modalities, with consistently high BERTScore-F1 (>96%).

BLIP (Salesforce/blip-vqa-base) benefited strongly from option selection, converting its semantically rich generations into reliable discrete answers.

BLIP now matches or slightly exceeds ViLT on X-ray (93.25% EM vs. 92.34%) and is only marginally behind on CT and MRI, with very high BERTScore-F1 across all modalities.

MiniCPM-V2 (OpenBMB) showed clear improvements under the same option-selection regime, but still lagged behind the two VQA-specialised models.

In sum, the option-selection pipeline shows that, once models are fine-tuned on modality-specific data, constraining their outputs to the multiple-choice answer set via semantic matching can raise both Exact Match and F1 above 90% for ViLT and BLIP, while also improving WUPS and BERTScore-F1. MiniCPM-V2 also benefits from this approach but remains behind the VQA-specialised backbones across all metrics and modalities.

4.4. Stratified Performance Analysis by Answer Type

We further report a per-model stratification by answer type, with results weighted across CT, MRI, and X-ray. For BLIP, option selection improved accuracy from 82.39→94.53 on yes/no (n = 494), 71.43→92.00 on number (n = 56), 74.10→88.78 on anatomy-term-like (n = 722), and 44.36→89.09 on other/free-text answers (n = 275), where n is the per–answer-type test set size used for the comparison. MiniCPM showed similar gains: 75.30→92.11 (yes/no), 62.50→91.07 (number), 55.96→72.16 (anatomy-term-like), and 39.27→72.73 (other). For ViLT, gains were smaller where raw exact match was already high—86.44→96.36 (yes/no) and 85.10→92.34 (anatomy-term-like)—but remained substantial for number (80.36→93.00) and other/free-text (56.18→84.27) answers. Overall, the improvements were not limited to short categorical responses; the largest gains frequently occurred for free-text answers, consistent with the post-hoc option selector calibrating semantically correct but non-identical predictions into the discrete option space.

5. Discussion

This study set out to examine how state-of-the-art general-purpose vision–language models behave on radiology-focused medical VQA when (i) used off-the-shelf, (ii) fine-tuned on modality-specific CT, MRI and X-ray subsets derived from SLAKE and OmniMedVQA, and (iii) coupled with a post-hoc option-selection layer. By harmonising two complementary Med-VQA datasets and systematically evaluating ViLT, BLIP and MiniCPM-V-2, we provide a controlled picture of how far generic VQA backbones can be pushed towards reliable performance in this medical setting. Our discussion is centred on answering the research questions posed in Section 1.3.

5.1. How Well Do General-Purpose VLMs Perform in a Strict Zero-Shot Setting?

In the zero-shot regime, all three models performed poorly in strict terms, with exact-match accuracies around the low-20% range for ViLT and BLIP and essentially 0% for MiniCPM-V-2; despite reasonable WUPS and BERTScore values. This confirms that simply deploying large VQA or multimodal LLMs “as is” on medical images is inadequate. The OmniMedVQA authors similarly report that both general-domain LVLMs and several medical-specialised LVLMs struggle on their benchmark, often underperforming expectations from natural-image VQA. Our zero-shot results are consistent with that broader observation: pre-training on large natural-image corpora provides limited semantic signal and is insufficient to meet even moderate accuracy requirements in radiology VQA.

Across models, BLIP was the strongest zero-shot baseline, slightly outperforming ViLT on all three modalities in both exact-match and semantic metrics. This aligns with the BLIP design as a unified understanding-and-generation framework that bootstraps high-quality supervision from noisy web image–text pairs and achieves state-of-the-art natural-image VQA scores. ViLT, in contrast, is a minimalist monolithic transformer that trades some accuracy for architectural simplicity and speed. MiniCPM-V-2 produced semantically related but highly unconstrained answers, reflected in reasonable WUPS and BERTScore scores but near-zero exact matches. Together, these findings reinforce that domain shift, specialised terminology, and a structured answer space remain major obstacles for off-the-shelf deployment of general VLMs in medical VQA.

5.2. Can Supervised Fine-Tuning on Modality-Specific Training Subsets Reduce the Performance Gap Between General-Purpose VLMs and Specialised Ones?

Introducing supervised fine-tuning on the harmonised SLAKE+OmniMedVQA splits dramatically changed this picture. All three backbones showed large gains, but the magnitude and character of the improvement depended on the underlying architecture and training objective.

ViLT benefited directly from fine-tuning. Because it is already trained as a multi-class classifier on VQAv2, re-training its classification head and encoder on the modality-specific medical label spaces lifted exact-match accuracy into the ≈80% range across CT, MRI and X-ray, with F1, ROUGE-L, WUPS and BERTScore all in the high-70s to low-90s. This suggests that, given a well-curated medical label space and sufficient supervised data, a relatively compact, task-aligned VQA backbone can adapt effectively to radiology.

BLIP also improved substantially after fine-tuning but retained a characteristic gap between semantic and lexical metrics: BERTScore-F1 and WUPS were high, yet EM and F1 remained noticeably lower than for ViLT. This behaviour is consistent with the generative nature of BLIP. It is pre-trained for both understanding and generation and tends to produce fluent paraphrases rather than exact strings. In our results, the BLIP answers often captured the correct clinical meaning but failed strict string equality, which is penalised by EM.

MiniCPM-V-2, despite being a modern multimodal LLM, continued to lag behind the specialised VQA backbones after LoRA-based instruction tuning. It did achieve solid improvements compared with its zero-shot baseline, especially on CT, but its EM and F1 scores on MRI and X-ray remained clearly lower than those of ViLT and BLIP. This suggests that scale and generality alone do not guarantee strong performance on Med-VQA subsets; specialised pre-training objectives and architectures still matter.

Our metrics show that even after fine-tuning, MRI questions yield lower WUPS@0.9 and F1 than CT and X-ray for all three backbones. This systematic gap indicates that modality-specific factors—not just overall dataset size—shape how well general VLMs can be adapted to radiology VQA. Box 1 is the key take-away from the results.

Box 1. Key finding on modality complexity using SLAKE + Omni Med VQA data.

Across all models and pipelines, a stable modality pattern emerged: CT was consistently the easiest setting, X-ray intermediate, and MRI the most challenging.

5.3. Can a Simple Post-Hoc Answer-Selection Strategy Further Improve Reliability on Benchmarks with Predefined Answer Sets?

The third pipeline shows that a large part of the residual error after fine-tuning is due to mismatches between free-form generation and the discrete answer space. By reframing evaluation as a multiple-choice decision problem and selecting the option that is semantically closest (via BERTScore) to the model’s generated answer, both ViLT and BLIP moved into a near-ceiling regime: exact-match and F1 exceeded 90% on all three modalities, and BERTScore-F1 approached or exceeded 96%.

For ViLT, option selection consolidates an already strong classifier, raising EM from ≈80% to >92% and improving semantic metrics further. For BLIP, the effect is even more pronounced. While fine-tuned BLIP alone lags ViLT in strict EM, the option-selection layer effectively converts its high-quality semantic predictions into reliable discrete choices, yielding performance comparable to—or slightly better than—ViLT on X-ray and only marginally lower on CT and MRI. These observations are in line with the BLIP original design goals of flexible transfer to both understanding and generation tasks.

MiniCPM-V-2 also benefits from option selection, with clear gains in all metrics, particularly on CT and X-ray. However, its scores remain noticeably below those of ViLT and BLIP. This indicates that while post hoc calibration is broadly helpful, it cannot fully compensate for weaker task adaptation in the underlying model.

Importantly, the option-selection pipeline does not change the training objective: the models are still fine-tuned on open-ended answers. The improvements arise purely from re-projecting predictions into the dataset’s answer space at evaluation time. This suggests that similar post hoc calibration layers—using semantic matching, cross-encoders, or retrieval-augmented scoring—may be a practical way to increase the reliability of generative Med-VQA systems without modifying the base training loop.

5.4. Relation to Prior Med-VQA Literature

Our findings sit alongside a growing body of work on medical VQA benchmarks. SLAKE [5] introduced semantically labelled, knowledge-enhanced questions spanning multiple modalities and anatomical regions, explicitly to support robust Med-VQA evaluation. OmniMedVQA extended this idea to a far larger scale and demonstrated that many existing LVLMs—both general and medical-specialised—struggle on realistic, multimodal clinical data. Other datasets such as PMC-VQA likewise report that out-of-the-box VLMs achieve modest scores and require careful adaptation [37].

Against this backdrop, our results show that, on harmonised CT, MRI, and X-ray subsets drawn from SLAKE and OmniMedVQA, relatively lightweight adaptations—modality-specific fine-tuning and post hoc option selection—are sufficient to push ViLT and BLIP into very high accuracy ranges. This does not contradict the earlier benchmarks. Our study focuses on three modalities and a merged dataset with unified answer processing, whereas OmniMedVQA evaluates across 12 modalities and a much broader variety of question types [6]. Rather, it illustrates that when the problem is carefully scoped and curation is controlled, general-purpose VQA backbones can become competitive Med-VQA solvers.

6. Conclusions and Future Directions

In this work, we systematically evaluated three general-purpose vision–language models—ViLT, BLIP, and MiniCPM-V-2—on harmonised Med-VQA subsets derived from SLAKE and OmniMedVQA-Mini, focusing explicitly on CT, MRI, and X-ray questions. Across all experiments, our findings were consistent. First, zero-shot performance was inadequate for medical use, with exact match confined to ≈24–25% for BLIP, ≈16–24% for ViLT, and 0% for MiniCPM-V-2 despite moderate semantic similarity scores. Second, modality-specific supervised fine-tuning was essential: ViLT reached 78–83% exact match across modalities, BLIP improved to 71–82%, and MiniCPM-V-2 to 60–80%, with CT being consistently easier than MRI and X-ray. Finally, adding a simple post hoc option-selection layer that maps free-text predictions to multiple-choice answers yielded the strongest results: ViLT and BLIP achieved 90–93% exact match, F1, and ROUGE-L on all three modalities, while MiniCPM-V-2 reached 74–81% exact match with BERTScore-F1 up to ≈93%. Taken together, these results show that carefully curated, modality-aware supervision combined with lightweight semantic calibration can transform generic VQA backbones into highly accurate solvers on radiology-focused Med-VQA benchmarks, while also clarifying the remaining performance gap for current multimodal LLMs.

This work has several practical limitations that suggest clear directions for future research. First, all experiments were constrained due to limited GPU resources, which restricted batch size, training duration, hyperparameter exploration and, in the case of MiniCPM-V-2, the proportion of data used for fine-tuning. Running the same pipelines on more powerful single or multi-GPU setups could enable deeper optimisation and potentially stronger performance. Second, we focused on two Med-VQA datasets (SLAKE and OmniMedVQA-Mini) and three imaging modalities (CT, MRI, and X-ray), so extending the current harmonisation and option-generation pipeline to additional datasets and modalities (e.g., ultrasound, histopathology, dermoscopy, and larger hospital cohorts) would be important to assess generalisability. Third, we evaluated only medium-scale backbones (ViLT-B/32, BLIP-VQA-base, and MiniCPM-V-2). Future work should explore larger, more powerful multimodal models—and medical-specialised LVLMs—within the same modality-aware, post hoc option-selection framework to determine how much additional benefit can be gained from increased parameter counts and more advanced architectures.