Image Captioning Through Deep Learning: An Adaptation of the BLIP-2 Model to Arabic

Abstract

Image captioning using deep learning bridges computer vision and natural language processing, enabling machines to generate human-like textual descriptions for images. While significant progress has been made in English, in Arabic, the image captioning field remains under-explored due to the language’s morphological complexity, right-to-left script, and scarcity of annotated datasets. This paper addresses this gap by adapting the BLIP-2 (Bootstrapped Language—Image Pre-training) model for Arabic caption generation, leveraging machine-translated datasets, like Flickr 30k, to overcome resource limitations. BLIP-2 combines a vision transformer (ViT) for image encoding and a CamelBERT large language model (LLM) for text generation, enhanced by a lightweight Querying Transformer (Q-Former) for cross-modal alignment. Despite challenges such as translation artifacts and linguistic nuances, our experiments demonstrate promising results in generating coherent Arabic captions. In short, this study highlights the potential of BLIP-2 for multilingual applications while underscoring the need for native Arabic datasets and further optimization. Ultimately, this work contributes to advancing inclusive artificial intelligence technologies for Arabic-speaking communities, with applications in assistive tools, education, and content creation.

1. Introduction

Deep learning for image captioning is a popular and multidisciplinary field that combines natural language processing and computer vision to automatically generate textual descriptions for images. At its core, image captioning involves identifying salient visual elements within an image and translating them into linguistically coherent and contextually appropriate sentences. This technology holds immense potential for a wide range of applications, from assistive tools for the visually impaired to automated content creation, image indexing, and human–computer interaction [1,2].

Early deep learning approaches established foundational encoder–decoder frameworks, typically using Convolutional Neural Networks (CNNs) like VGG or ResNet to encode visual information and Recurrent Neural Networks (RNNs) such as LSTMs to sequentially generate captions [3,4,5]. While effective, these models often struggled with capturing fine-grained visual details and maintaining long-range linguistic dependencies [6]. The field was revolutionized by the advent of attention mechanisms and, more recently, transformer-based architectures [7]. Models like CLIP [8], BLIP [9], and BLIP-2 [10] have set new benchmarks by learning powerful joint representations of vision and language, enabling more accurate and context-aware caption generation. For instance, BLIP-2’s efficient use of frozen pre-trained models has shown remarkable performance in zero-shot and fine-tuned settings, significantly reducing computational costs [10,11]. More recently, diffusion models have also emerged as a powerful alternative, demonstrating superior ability to generate high-quality and diverse captions through iterative refinement processes [12,13,14].

Despite its rapid progress in English, the field of image captioning for Arabic remains significantly under-explored [15,16,17]. This disparity stems from a confluence of challenges unique to the Arabic language. First, Arabic’s rich morphological complexity, where words are constructed from root-based patterns, results in a vast vocabulary and makes standard tokenization and embedding techniques less effective. Second, its right-to-left script and context-sensitive letter shaping require specialized preprocessing. Third, Arabic exhibits diglossia, with Modern Standard Arabic used in formal settings coexisting alongside numerous diverse dialectal variants for everyday communication. Finally, and most critically, there is a pronounced scarcity of large-scale, high-quality annotated datasets for vision–language tasks in Arabic [18,19,20]. This combination of linguistic intricacies and data poverty has hindered the development of robust captioning systems that can generate accurate and contextually relevant descriptions for the over 400 million Arabic speakers the world.

Recent initiatives have begun to address these gaps, marking a crucial step forward for Arabic vision–language research. Some studies have explored the use of machine translation to create Arabic versions of existing English datasets like Flickr 8k and MS COCO, employing architectures such as VGG16 with LSTMs [21], VGG19 with Inception-ResNet-v2 [22], or more advanced models like AraBERT [23] and hybrid CNN-Transformer models with CamelBERT [24]. Others have focused on developing specialized resources, such as the ArabicFashionData dataset for clothing [25], the JEEM dataset for dialectal understanding [20], models like AraCLIP [26] for image retrieval, and AraCap [19] for captioning. However, these efforts, while valuable, are often limited in scale, rely on older architectural paradigms, or are tailored to specific domains. The challenge of creating a generalizable, state-of-the-art Arabic captioning model using a resource-efficient framework on a large-scale dataset remains largely unaddressed.

The BLIP-2 (Bootstrapping Language–Image Pre-training) model [10] offers a particularly compelling architecture to tackle this challenge. Unlike traditional end-to-end methods, BLIP-2 introduces a lightweight Querying Transformer (Q-Former) that acts as a bridge between a frozen pre-trained image encoder and a frozen large language model (LLM). This design offers significant advantages for low-resource languages like Arabic. The frozen components preserve powerful, general-purpose representations learned from massive English dominated datasets, while the trainable Q-Former can be efficiently adapted to a new language and domain with limited paired data. Its two-stage pre-training process first aligns vision and language representations through contrastive learning and then facilitates generative vision to language learning to provide a structured and data-efficient framework for cross-modal adaptation [27]. This modularity makes BLIP-2 an ideal candidate for bootstrapping Arabic captioning capabilities.

In this paper, we propose the first adaptation of the BLIP-2 model for Arabic image captioning. Our approach leverages a frozen Vision Transformer (ViT) for robust image encoding and replaces BLIP-2’s original English-centric LLM with the powerful, pre-trained Arabic LLM, CamelBERT (bert-base-arabic-camelbert-mix)  [28]. CamelBERT’s pre-training on a diverse corpus of 167 GB of Arabic text, including both Modern Standard Arabic and dialectal content, provides a sophisticated linguistic foundation for generating authentic and morphologically correct captions. To overcome the data scarcity problem, we constructed a new large-scale Arabic dataset by translating the entire English Flickr30k benchmark [29,30] using the Google Cloud Translation API, a methodology successfully employed for other low-resource languages [31,32]. This yielded a dataset of 31,783 images, each with five Arabic captions.

This paper makes three primary contributions:

• First Arabic Adaptation of BLIP-2: We present the first successful adaptation of the BLIP-2 architecture for Arabic, integrating a frozen ViT with the CamelBERT LLM via a trainable Q-Former, demonstrating a resource-efficient pathway for multilingual vision–language modeling.
• A New Large-Scale Arabic Dataset: We construct and will release a new Arabic image-captioning dataset derived from Flickr30k, providing a substantial resource of over 158,000 image–caption pairs to facilitate future research.
• State-of-the-Art Performance and Analysis: Through comprehensive experiments, we demonstrate that our fine-tuned model dramatically outperforms unadapted baselines, achieving a mean retrieval score of 89.2% and a BLEU-4 score of 36.8. We provide an in-depth analysis of the model’s learning dynamics, retrieval performance, and generation quality, establishing new benchmarks for Arabic image captioning and validating the efficacy of our approach.

The rest of the paper is organized as follows. Section 2 introduces the literature review in image captioning using deep learning, especially showing contributions in Arabic image captioning. The proposed methodology and the experimental data used are introduced in Section 3. Section 4 discusses our research proposal from different perspectives. Section 5 shows the results of applying the BLIP-2 model in Arabic image captioning. Finally, Section 6 summarizes our work and proposes future research.

2. Related Work

This section discusses the recent advances in image captioning that use deep learning architectures, namely, transformer-based models, to connect vision and language models. We will also explore different datasets.

2.1. Image Captioning with Deep Learning

Early deep learning methods for image captioning include attention-based, graph-based, convolutional, and transformer-based approaches. These techniques highlight progress in encoder–decoder frameworks, scene graphs, and vision–language pre-training. However, challenges like object hallucination and dataset bias remain, along with key gaps such as the need for better unsupervised algorithms and evaluation metrics [6].

Novel image captioning framework has been discussed [33] using Mask R-CNN, RetinaNet, and YOLOv3 for hierarchical object detection, boosting mAP results by merging results with transformer-based image-captioning models that benefit from cross-entropy loss with Adam optimization. Ref. [27] highlights caption quality as a key noise source, showing how generated captions enhance web-scraped data with nondescript text by mixing raw and generated captions. They surpass DataComp’s best filtering method by $2\%$ on ImageNet and $4\%$ across 38 tasks, introducing a novel image captioning framework, and their encoder–decoder approach combines CNNs for visual features and RNNs for caption generation. Ref. [34] presented SWIN+PVT techniques with TaCL and achieved higher BLEU scores, highlighting model integration. However, all this research focuses on English, with no Arabic exploration. Ref. [31] addresses Bangla image captioning data scarcity, like Arabic image captioning, which was solved by creating BanglaView, a dataset of 31,783 images with 158,915 captions in which English Flickr 30k captions were translated to Bangla via Google Translate and refined by experts, similar to efforts in our research. According to [35], ResNeXt-101 delivers top performance, while MobileNetV3 offers efficiency, and DeiT achieved a 34.44 BLEU-4 score, surpassing ViT. Ref. [5] discussed a CNN-GRU-attention model for bridge damage captioning, achieving 69.3 for BLEU-4, highlighting domain-specific data benefits. While effective for structural analysis, it lacked multilingual support and linguistic adaptability. The study in [36] proposed an encoder–decoder model combining VGG16, LSTM with attention, Mask R-CNN, and OpenCV for color-aware captioning. It leverages transformers (SWIN, PVT) and embeddings (TaCL, BERT, GloVe) for enhanced English scene descriptions.

2.2. Image Captioning Using Diffusion Models

Diffusion models and transformers are becoming more popular due to their superior ability to generate high-quality image captions [12] by using iterative refinement approaches, combining diffusion steps with cross-modal attention, which progressively enhances the output, allowing for diverse and improved caption generation. Ref. [13] proposed a hybrid method that merges diffusion models’ precision with caption generation needs in diffusion-Cap, combining continuous and discrete diffusion, excelling on MSCOCO (BLEU-4: 37.5, CIDEr: 122.8), and outperforming non-autoregressive models. BLIP-2 model excels in creating context-aware Tamil Indian language in image captioning [14], aiding visually impaired users, using frozen image encoders and LLMs to generate and translate descriptions into Tamil language audio, and offering detailed image explanations. Another diffusion model (mBLIP) is a multilingual model that creates captions in 96 languages. It leverages machine translation to adapt high-quality English datasets (MSCOCO, LLaVA) into multiple languages efficiently [4]. Diffusion-LM’s controllability is enhanced by integrating image captioning, aligning visual and linguistic elements in its latent space [37]. These extensions improve multimodal coherence and efficiency while maintaining text-based control. BLIP [9] introduces a Multimodal Mixture of Encoder–Decoder (MED) framework, merging understanding and generation via shared parameters and task-specific objectives. Its adaptability delivers state-of-the-art results in the potential of hybrid architectures for image captioning. Diffusion models, originally developed for image generation, have recently been adapted to image captioning, offering promising advantages over traditional encoder–decoder and transformer-based approaches. So our research is based on BLIP-2 [10], a lightweight Q-Former connecting frozen image encoders and LLMs. It excels in zero-shot captioning and VQA with fewer parameters, using a two-stage pre-training approach for cross-modal alignment.

2.3. Arabic Image Captioning

Arabic image captioning remains an under-explored area due to linguistic complexities, limited datasets, and the unique morphological structure of the Arabic language. [15] reviews non-English captioning datasets, including Arabic ones, such as JEEM [20], ArabicFashionData [25], MS COCO-Based [38], and Flickr 8k-Based [21]; previous study also highlights Arabic captioning’s state, relevance, and deep learning potential while noting key limitations. ArabicQuest [3] presents an Arabic VQA that uses 137,888 MS-COCO QA pairs, merging text and visual features but lacking caption depth. It achieved $70\%$ Text QA accuracy but struggled with counting/comparisons. More in detail, it relies on translation, underscoring the need for end-to-end Arabic captioning.

Ref. [26] introduces AraCLIP, an Arabic image retrieval model using CLIP in which knowledge is converted from an English CLIP model into an Arabic one, leveraging AraBERT. Contributions include cleaned Arabic datasets like CC3M+CC12M+SBU and MS COCO. Ref. [25] contributes ArabicFashionData, an Arabic dataset for clothes captioning, outperforming prior Arabic and English models by leveraging image data, linguistic features, and attention mechanisms. The study [24] introduces a hybrid of ConvNexT, SWIN, and XCIT models to enhance image feature extraction; combining CamelBERT with LSTM decoders, it achieved BLEU scores BLEU-1 = 59.80, BLEU-2 = 40.03, BLEU-3 = 30.56, and BLEU-4 = 16.65 on Flickr 8k, improving Arabic captioning robustness and scalability. Ref. [22] proposes an encoder–decoder model for translated Flickr 8k dataset from English to Arabic using VGG19 and Inception-ResNet-v2; experiments were based once again on the Arabic Flickr 8k dataset. Thus, this dataset holds particular significance, as it represents the first publicly available collection of Arabic image captions, showing notable improvement, achieving BLEU-1 = 37 as the highest score. AraBERT [23] got better and proved the effectiveness of transformer-based models for Arabic image captioning trained on machine-translated Flickr 8k-Arabic and Arabic COCO.

Ref. [39] mentioned the Arabic dataset scarcity of Arabic image captioning datasets, restricting the model training and evaluation; its result metrics were BLEU-1 = 36.5, BLEU-2 = 21.4, BLEU-3 = 12, and BLEU-4 = 6.6 based on the Flickr 8k dataset, where each image has three Arabic captions.

Ref. [19] introduced AraCap, an Arabic-focused image-captioning model combining object-based and attention mechanisms. Three variants were trained on COCO and Flickr 30k, which were then tested on an Arabic-translated COCO subset.

Our work advances Arabic image captioning by addressing key gaps in the existing research. Arabic image captioning faces limitations, as most studies use small translated datasets (Flickr 8k, COCO subsets), reducing robustness compared to English benchmarks like MS COCO. The study [18] addresses the scarcity of resources and research in Arabic image captioning, focusing on expanding datasets and exploring advanced deep learning techniques to improve AIC performance, but we introduce a new Arabic Flickr 30k dataset obtained by translating it using Google translation API for high-quality translations. Moreover, transformer-based models demand significant resources, hindering adoption in resource-constrained settings, so our work adopts BLIP-2 Q-Former to bridge frozen vision ViT and language CamelBERT models (available at https://github.com/CAMeL-Lab/CAMeLBERT, accessed on 15 August 2025), optimizing for Arabic’s morphological complexity. Our work focuses on improving Arabic captioning by developing a new Arabic Flickr 30k dataset and the innovative architecture of BLIP-2 that helps to overcome challenges in the Arabic language.

In Table 1, a structured comparison of related work with an effective synthesis format highlights the research landscape and the specific gap that our work aims to fill.

Table 1. Previous work for image captioning.

Ref.	Methodology	Dataset Used
Image Captioning with Deep Learning
[34]	SWIN + PVT, TaCL	MSCOCO
[5]	CNN-GRU with Attention	Constructed dataset using images of actual bridge damage
[35]	ResNeXt-101, MobileNetV3, DeiT	MSCOCO
[33]	Mask R-CNN, RetinaNet, YOLOv3 + transformer-based image captioning	MSCOCO
[36]	VGG16, LSTM with attention, Mask R-CNN, and OpenCV with transformers (SWIN, PVT)	PASCAL VOC, MSCOCO, ImageNet
[27]	CNN (Visual) + RNN (Language model)	Web-scraped data
Image captioning using diffusion models
[4]	Multilingual Diffusion (mBLIP)	Translated MSCOCO, LLaVA
[10]	Q-Former (Frozen ViT + LLM)	MSCOCO
[37]	Diffusion-LM	E2E dataset consists of 50K restaurant reviews, ROCStories dataset
[13]	Hybrid Continuous/Discrete Diffusion	MSCOCO
[14]	Frozen Encoders + LLMs (Q-Former)	Custom dataset generated from Tamil Nadu
Arabic image captioning
[3]	VQA Model (Blip-2)	Translated MS-COCO (137,888 pairs)
[25]	Attributes-based with Attention	ArabicFashionData
[24]	ConvNeXt, SWIN, XCIT + CamelBERT-LSTM	Translated Flickr 8k-Arabic
[23]	Transformer-based (AraBERT)	Translated Flickr 8k and COCO
[39]	(LSTM and GRU) + VGG16	Translated Flickr 8k-Arabic
[19]	Hybrid-object-based attention	Translated Arabic Version of subset of MSCOCO
[22]	VGG19, Inception-ResNet-v2	Translated Flickr 8k-Arabic
[39]	Vision encoder and a Gemini text decoder	Translated Flickr 8k-Arabic
Our work	BLIP-2 (ViT + CamelBERT)	New Arabic Flickr 30k

3. Methodology and Dataset

Adapting BLIP-2 [10] to produce accurate Arabic image captions requires a carefully designed approach that addresses the unique challenges of the Arabic language while leveraging the strengths of modern vision–language architectures. Our methodology employs a Vision Transformer (ViT) to encode images and the CamelBERT large language model (LLM) for text generation, connected through a lightweight Querying Transformer (Q-Former) that facilitates cross-modal alignment. By fine-tuning only the Q-Former while keeping both the vision encoder and language model frozen, our design maximizes computational efficiency and data economy—critical advantages when working with low-resource languages like Arabic.

The design of our approach is motivated by three key considerations specific to Arabic image captioning: linguistic complexity, data scarcity, and computational efficiency. To address Arabic-rich morphology, right-to-left script, and diglossia, we employ CamelBERT [28], a language model pre-trained on 167 GB of diverse Arabic text (17.3 B tokens) to provide deep linguistic knowledge. To address the scarcity of high-quality Arabic image–text pairs, we leverage BLIP-2’s modular architecture, which freezes pre-trained components and trains only a lightweight Q-Former, thereby enabling effective adaptation with limited data by leveraging representations learned from large-scale English pre-training. For computational efficiency, our design trains just 188 M parameters (the Q-Former) while leveraging 1.13 B vision parameters and 340 M language parameters from frozen models, within a suitable training time. This approach follows the established principle that modular adaptation of pre-trained models is more effective for low-resource languages than end-to-end training from scratch, as demonstrated in recent multilingual vision-language research [4,14,31].

3.1. Algorithm

BLIP-2 presents an efficient vision–language pre-training method that leverages frozen pre-trained image encoders and LLMs [10]. A compact Q-Former acts as a bridge between the two modalities, and it is trained in two key stages. First, it aligns vision and language representations using a frozen image encoder (ViT-L/14 from CLIP, ViT-g/14 from EVA-CLIP), and second, it facilitates generative vision-to-language learning with a frozen LLM (OPT/FlanT5).

Despite having fewer trainable parameters, BLIP-2 achieves state-of-the-art performance on various vision–language tasks. Our work focuses on adapting BLIP-2 using ViT, as an efficient vision–language transformer, and CamelBERT (bert-base-arabic-camelbert-mix with 167 GB, 17.3 B words), as an Arabic LLM, by modifying BLIP-2’s architecture to integrate CamelBERT and ensuring tokenizer compatibility. The dataset was obtained by translating the Flickr 30k dataset from English to Arabic using the Google API translation service, as explained in Section 3.2. The model employs a frozen vision encoder (ViT pre-trained on ImageNet-21k and CLIP) and a frozen language model (CamelBERT, pre-trained on 167 GB/17.3 B Arabic tokens). Only the Q-Former module (~188 M parameters) is trained from scratch in two stages (Pre-train stage 1, Pre-train stage 2) to learn cross-modal alignment. This design maximizes the reuse of strong pre-trained representations while efficiently adapting to Arabic captioning. The stages are described as follows:

• Pre-train stage 1: In the first stage, ViT encodes images into high level visual features without updating its weights. The goal is to align these visual features with the corresponding text, creating a shared vision language embedding space. The output is a checkpoint for these aligned representations, which is used to seed Pre-train stage 2.
• Pre-train stage 2: In the second stage, the model learns to generate coherent text from visual inputs using the pre-aligned embeddings from Pre-train Stage 1. For this generative task, the LLM CamelBERT is employed. While CamelBERT’s weights remain fixed, the rest of the model learns to translate visual information into textual descriptions. Training is optimized with a cosine learning rate scheduler, starting with an extremely low warmup Learning Rate (LR) of $1\times {10}^{-6}$ for stability, transitioning to a moderate training LR of $1\times {10}^{-4}$ for effective weight updates, and gradually decaying to a minimum of $1\times {10}^{-5}$ to ensure fine convergence while maintaining training momentum through the final epochs.

Lastly, the result is query embeddings that enable tasks like image captioning by efficiently connecting visual inputs with natural language. This two-step process guarantees that the model learns robust cross-modal representations (vision + text) before focusing on text creation, resulting in outputs that are more cohesive and context-aware by combining the following:

• A frozen ViT for high-quality visual feature extraction.
• A frozen CamelBERT for Arabic language understanding and generation.
• A trainable Q-Former to bridge the modality gap between vision and text.

Algorithm 1 and Figure 1 illustrates the step-by-step process of the BLIP-2 model flow using ViT, as the vision encoder, and the LLM CamemBERT. Below is a detailed breakdown of each component and its role in the workflow:

1.

Input Image: The process begins with an input image that needs to be captioned in Arabic.

2.

Vision Encoding with ViT: The input image is processed using ViT, which encodes the image into a set of visual features. The ViT breaks the image into patches and processes them to extract high-level features.

3.

Extracted Image Features: The ViT outputs a set of extracted image features that represent the visual content of the input image in a structured format.

4.

Q-Former Bridging Modalities: The Querying Transformer acts as a bridge between the visual features and the language model. It is designed to align visual and textual representations:

Algorithm 1 BLIP-2 training process for Arabic image captioning

1: Input: Arabic-translated Flickr 30k dataset 2: Output: Trained BLIP-2 model for Arabic captioning 3: procedure  Stage1-Pretraining 4:     Initialize ViT encoder (frozen) 5:     Initialize Q-Former (trainable) 6:     for each epoch do 7:        Extract image features using ViT 8:        Align visual features with Arabic text embeddings 9:        Train Q-Former to bridge modalities 10:     end for 11:     Save checkpoint for aligned representations 12: end procedure 13: procedure  Stage2-Pretraining 14:     Load checkpoint from Stage 1 15:     Initialize CamelBERT (frozen) 16:     for each epoch do 17:        Use aligned embeddings from Q-Former 18:        Generate Arabic text using CamelBERT 19:        Optimize with cosine LR scheduler: 20:           Warmup LR: $1\times {10}^{-6}$ 21:           Training LR: $1\times {10}^{-4}$ 22:           Min LR: $1\times {10}^{-5}$ 23:     end for 24:     Save final model checkpoint 25: end procedure

• It uses learnable query vectors to interact with the image embeddings from the vision encoder.
• It also uses learnable query vectors to interact with the text tokens, if provided.
• The Q-Former is trained to extract the most semantically meaningful features for cross-modal alignment.

5.

BLIP-2 Two Stage Pre-Training: The Q-Former undergoes a two-stage pre-training process as part of the BLIP-2 framework:

• Stage 1: The Q-Former learns to extract and represent visual information from the ViT outputs. This stage focuses on understanding the visual content.
• Stage 2: The Q-Former learns to interact with a frozen LLM to generate text conditioned on the visual input. This stage aligns the visual features with the language model’s capabilities.

6.

The frozen Arabic LLM, CamelBERT, is employed for text generation. Leveraging its pre-existing language understanding, it takes the aligned visual–textual features as input to produce coherent and context-aware text, all while its weights remain fixed to ensure training stability.

7.

Arabic Image Caption Generation: The final output is an Arabic caption generated by the system, describing the content of the input image based on the visual features and the language model’s text generation capabilities.

Figure 1. BLIP-2 model flow using ViT as vision encoder and CamelBERT.

Figure 1. BLIP-2 model flow using ViT as vision encoder and CamelBERT.

3.2. Dataset

The Flickr 30k dataset is a widely used benchmark in vision–language research, serving as a key resource for tasks like image captioning and visual question answering (VQA) [18,27,30,40]. It was selected for our Arabic adaptation due to its properties:

• Scale: Contains 31,783 images with 5 captions each (158,915 total)
• Diversity: Everyday scenes, activities, objects, and people.
• Established benchmark status: Enables comparison with prior work [10,27,40]

Pre-Processed Dataset

To create an Arabic dataset, we employed Google Cloud Translation API with Google Neural Machine Translation (available at https://cloud.google.com/translate/docs/reference/rest, accessed on 15 August 2025), following methodologies successfully validated for other low-resource languages, including Bangla [31] and Hindi [32]. This design choice was motivated by:

• Scalability: Automated translation enables rapid dataset creation at low cost.
• Quality: Neural machine translation produces fluent, grammatically correct Arabic text.
• Reproducibility: The methodology can be easily replicated for other languages or datasets.

During the translation process, the dataset was split into 6 chunks, each divided into 5 sub-chunks of 1000 images (5000 captions per sub-chunk). This chunking strategy prevents API timeout issues from long responses, enables parallel processing, and facilitates quality validation at each stage.

The translated captions are returned in UTF-8 encoding, preserving right-to-left (RTL) script formatting and handling Arabic-specific features, including contextual letter shaping, punctuation, and idiomatic expressions.

To ensure translation quality, we did the following:

• Validated fluency: Random samples checked for grammatical correctness.
• Verified semantic preservation: Captions checked for meaning preservation.
• Ensured script handling: Confirmed proper RTL rendering and character shaping.

Table 2 presents examples demonstrating the quality of translated Arabic captions based on Algorithm 2, where in we can see two images taken from the Flickr 30k dataset with the five original English captions and the five translated into Arabic captions, using Google Cloud Translation API.

Table 2. Two samples from the Flickr 30k dataset with Arabic translation after using Google Cloud Translation API.

Flickr Image Sample 1		Flickr Image Sample 2
English Caption	Arabic Caption	English Caption	Arabic Caption
Someone in a blue shirt and hat is standing on the stairs and leaning against a window.	يقف شخص ما فى قميص وقبعة زرقاء على درج و يميل على نافذة	Five ballet dancers caught mid jump in a dancing studio with sunlight coming through a window.	اشتعلت خمسة راقصات باليه فى منتصف القفز فى استوديو رقص مع ضوء الشمس القادم من خلال النافذة
A man in a blue shirt is standing on a ladder cleaning a window.	رجل يرتدى قميص ازرق يقف على سلم يقوم بتنظيف النافذة	Ballet dancers in a studio practice jumping with wonderful form.	راقصات الباليه فى ممارسة الاستوديو القفز مع شكل رائح
A man on a ladder cleans the window of a tall building.	رجل على سلم ينظف نافذة مبنى طويل القامة	Five girls are leaping simultaneously in a dance practice room.	خمس فتيات يقفزن فى وقت واحد فى غرفة ممارسات الرقص
Man in blue shirt and jeans on ladder cleaning windows.	رجل يرتدى القميص الازرق و الجينز على نوافذ تنظيف السلم	Five girls dancing and bending feet in ballet class.	خمس فتيات يرقصن و ينحنين فى فئة الباليه
A man on a ladder cleans a window.	رجل على سلم ينظف نافذة	A ballet class of five girls jumping in sequence.	فئة باليه من خمس فتيات يقفزن بتسلسل

Table 2. Two samples from the Flickr 30k dataset with Arabic translation after using Google Cloud Translation API.

Flickr Image Sample 1		Flickr Image Sample 2
English Caption	Arabic Caption	English Caption	Arabic Caption
Someone in a blue shirt and hat is standing on the stairs and leaning against a window.	يقف شخص ما فى قميص وقبعة زرقاء على درج و يميل على نافذة	Five ballet dancers caught mid jump in a dancing studio with sunlight coming through a window.	اشتعلت خمسة راقصات باليه فى منتصف القفز فى استوديو رقص مع ضوء الشمس القادم من خلال النافذة
A man in a blue shirt is standing on a ladder cleaning a window.	رجل يرتدى قميص ازرق يقف على سلم يقوم بتنظيف النافذة	Ballet dancers in a studio practice jumping with wonderful form.	راقصات الباليه فى ممارسة الاستوديو القفز مع شكل رائح
A man on a ladder cleans the window of a tall building.	رجل على سلم ينظف نافذة مبنى طويل القامة	Five girls are leaping simultaneously in a dance practice room.	خمس فتيات يقفزن فى وقت واحد فى غرفة ممارسات الرقص
Man in blue shirt and jeans on ladder cleaning windows.	رجل يرتدى القميص الازرق و الجينز على نوافذ تنظيف السلم	Five girls dancing and bending feet in ballet class.	خمس فتيات يرقصن و ينحنين فى فئة الباليه
A man on a ladder cleans a window.	رجل على سلم ينظف نافذة	A ballet class of five girls jumping in sequence.	فئة باليه من خمس فتيات يقفزن بتسلسل

Algorithm 2 Dataset translation process

1: Input: Flickr 30k dataset with English captions 2: Output: Arabic-translated Flickr 30k dataset 3: procedure  SplitDataset 4:     Split dataset into 6 chunks 5:     for each chunk do 6:        Divide into 5 sub-chunks (1000 images each) 7:        for each sub-chunk do 8:           Extract 5000 English captions (5 per image) 9:           Translate to Arabic using Google Cloud Translation API 10:           Save translated Arabic captions 11:        end for 12:     end for 13:     Merge all translated chunks 14:     Validate translation quality 15: end procedure

3.3. Experimental Setup

The experiments were performed on Google Colab Pro using a high-performance GPU instance with an NVIDIA L4 GPU accelerator (24 GB VRAM), ensuring efficient training of large vision–language models. The setup included 64 GB of RAM and 253 GB of disk storage, which was sufficient to meet the substantial memory requirements of pre-trained model weights (e.g., BLIP-2’s 1.5B parameters) and multimodal datasets. For model implementation, we used PyTorch 2.0+ and the Hugging Face Transformers library, leveraging BLIP-2’s cutting-edge image-to-text generation capabilities. To optimize memory efficiency, we applied mixed-precision training (FP16/FP32) via PyTorch’s Autocast, which reduced GPU memory consumption by approximately $30\%$ while preserving model accuracy with fixed random seeds (42 across Python v3.8.20, NumPy v1.24.4, torch v2.4.1, torchvision v0.19.1, tokenizers v0.13.3 and transformers v4.30.0). Additionally, we standardized image preprocessing by translating all captions from English to Arabic using Google Translation API and resizing all images to 224 × 224 pixels. To establish a fair comparison and demonstrate the effectiveness of our adaptation. We evaluated two model configurations:

1.

Original BLIP-2: The pre-trained BLIP-2 model with ViT-g/14 vision encoder (1.13 B parameters) and OPT/FlanT5 language model in its original form, without any fine-tuning on Arabic data. This configuration served as our baseline to assess zero-shot Arabic captioning capability.

2.

Our Fine-Tuned Model: BLIP-2 adapted with the same ViT-g/14 vision encoder (frozen) and CamelBERT Arabic language model (340 M parameters, frozen), with the Q-Former module (188 M parameters) trained from scratch on our Arabic-translated Flickr 30k dataset. The training process encompassed both pre-training stages: Stage 1 was completed in approximately 8 h (7 h, 57 min, and 45 s), and Stage 2 in approximately 7.5 h (7 h, 24 min, and 30 s) on an NVIDIA L4 GPU with 24 GB VRAM.

Both stages in each model were trained for 5 epochs. Our fine-tuned model presents the training and validation loss curves (Figure 2). Both pre-training stages were trained for 5 epochs with batch sizes 25 (train) and 16 (eval) using the AdamW optimizer (${\beta }_1$ = 0.9, ${\beta }_2$ = 0.999, weight decay 0.05) and a cosine learning rate schedule with warmup (warmup LR $1\times {10}^{-6}$, initial LR $1\times {10}^{-4}$, minimum LR $1\times {10}^{-5}$, warmup steps 1000), demonstrating stable convergence by epoch 4–5 with no evidence of overfitting (validation loss consistently decreased or plateaued without upward trends).

Figure 2. Curves during pre-training, especially Stage 2, showing the convergence behavior of the BLIP-2 model with frozen CamelBERT, which stabilizes rapidly, demonstrating effective cross-modal learning for Arabic caption generation. when the solid line and the dotted line meet and flatten out, it indicates that the model has learned as much as it can from the data.

3.4. Evaluation Metrics

The main metrics computed are Recall@K (R@K), which measures the percentage of queries where the correct item is found in the top-K results, as shown in Equation (1). We have also added the BLEU (Bilingual Evaluation Understudy) metric, as it is the most used metric of automated, quantitative evaluation in text generation, such as image captioning.

$$\mathrm{Recall}@\mathrm{K}=\left({\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{queries}\mathrm{where}\mathrm{the}\mathrm{correct}\mathrm{item}\mathrm{is}\mathrm{ranked}\mathrm{in}\mathrm{top}\mathrm{K}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{queries}}}\right)\times 100$$

(1)

3.4.1. Image-to-Text (I2T) Retrieval

The goal is to measure how well the model retrieves the correct text descriptions (captions) given an image as a query. The evaluation process involves the following key steps:

• Retrieve Similarity Scores: For each image (i), the model computes a similarity score between the image and every text caption in the dataset. These scores are stored in a vector score_i2t[i], where each entry represents how well a text matches the image.
• Sort Texts by Similarity: All text captions are sorted in descending order based on their similarity scores. This is done by:
      inds = np.argsort(score_i2t[i])[::-1]
  where np.argsort() returns the indices that would sort the array in ascending order, and [::−1] reverses the order to get descending ranks (highest similarity first). The resulting inds array contains the indices of text captions ranked from most to least relevant to image (i).
• Find the Highest-Ranked Correct Caption: Since an image may have multiple correct captions (stored in img2txt[i]), the evaluation checks the rank of each correct caption and selects the best (highest-ranked) one, as in Algorithm 3.

Algorithm 3 Image-to-text retrieval rank calculation

Require:  ($scores\_i2t$) similarity scores matrix, $img2txt$: image-to-text mapping Ensure:  ($ranks$) array containing best rank positions for each query 1: $n\leftarrow \mathrm{length}(scores\_i2t)$ 2: $\mathit{ranks}\leftarrow \mathrm{array}\mathrm{of}\mathrm{zeros}\mathrm{with}\mathrm{size}n$ 3: for $i\leftarrow 0$ to $n-1$ do 4:     $\mathit{score}\leftarrow scores\_i2t\left[i\right]$ 5:     $\mathit{inds}\leftarrow \mathrm{argsort}(\mathit{score},\mathrm{order}=\mathrm{descending})$ 6:     $\mathit{rank}\leftarrow \infty \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}$▹ Initialize with infinity 7:     for each j in $img2txt\left[i\right]$ do 8:         $\mathit{pos}\leftarrow \mathrm{find}(\mathit{inds}==j)$ 9:         if $\mathit{pos}<\mathit{rank}$ then 10:            $\mathit{rank}\leftarrow \mathit{pos}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}$▹ Keep best (smallest) rank 11:         end if 12:     end for 13:     $\mathit{ranks}\left[i\right]\leftarrow \mathit{rank}$ 14: end for 15: return  $\mathit{ranks}$

• Compute Recall@K Metrics:

  1.

  Recall@1 (R@1): Percentage of images where the best correct caption is the top-ranked result.

  2.

  Recall@5 (R@5): Percentage of images where the correct caption appears in the top-5 results.

  3.

  Recall@10 (R@10): Percentage of images where the correct caption appears in the top-10 results.

3.4.2. Text-to-Image (T2I) Retrieval

The goal is to evaluate how accurately the model retrieves the correct image when given a text query (caption). The process, similar to the one discussed in previous Section 3.4.1, is as follows:

• Retrieve Similarity Scores: For each text caption (j), the model computes a similarity score between the text and every image in the dataset. These scores are stored in a vector score_t2i[j], where each entry represents how well an image matches the text description.
• Sort Images by Similarity: This step ranks all images in descending order based on their similarity scores. This is done by:
          inds = np.argsort(score_t2i[j])[::-1]
  where np.argsort() returns the indices that would sort the array in ascending order, and [::−1] ensures the most relevant images appear first. The resulting inds array contains the indices of images sorted from best to worst match for text (j).
• Find the Highest-Ranked Correct Image: Unlike image-to-text retrieval (where multiple correct captions may match to image), each text query typically has one correct corresponding image (stored in txt2img[j]). The evaluation checks the position of this correct image in the ranked list and the rank of the correct image, as in Algorithm 4.
• Compute Recall@K Metrics:

  1.

  Recall@1 (R@1): Percentage of text queries where the correct image is the top-ranked result.

  2.

  Recall@5 (R@5): Percentage of text queries where the correct image appears in the top-5 results.

  3.

  Recall@10 (R@10): Percentage of text queries where the correct image appears in the top-10 results.

Algorithm 4 Text-to-image retrieval rank calculation

Require:  ($scores\_t2i$) similarity scores matrix, $txt2img$: text-to-image mapping Ensure:  ($ranks$) array containing rank positions for each query 1: $n\leftarrow \mathrm{length}(scores\_t2i)$ 2: $\mathit{ranks}\leftarrow \mathrm{array}\mathrm{of}\mathrm{zeros}\mathrm{with}\mathrm{size}n$ 3: for $i\leftarrow 0$ to $n-1$ do 4:     $\mathit{score}\leftarrow scores\_t2i\left[i\right]$ 5:     $\mathit{inds}\leftarrow \mathrm{argsort}(\mathit{score},\mathrm{order}=\mathrm{descending})$ 6:     $\mathit{target}\_\mathit{img}\leftarrow txt2img\left[i\right]$ 7:     $\mathit{pos}\leftarrow \mathrm{find}(\mathit{inds}==\mathit{target}\_\mathit{img})$ 8:     $\mathit{ranks}\left[i\right]\leftarrow \mathit{pos}$ 9: end for 10: return  $\mathit{ranks}$

3.4.3. Bilingual Evaluation Understudy Metrics

Another metric computed is BLEU, which is used to assess our image captioning model’s performance. It calculates the similarity between a machine-generated caption and one or more human-written reference captions.

In our work, it calculates the similarity between a machine-generated caption and the translated reference captions, as mentioned above.

For these calculations, the candidate and reference texts’ n-grams (contiguous word sequences of varying lengths) are compared to determine the BLEU score. In particular:

• BLEU-1 measures the precision of single-word matches by evaluating the correctness of individual words (unigrams).
• BLEU-2 evaluates the fluency of two-word sequences by taking into account word pairings, or bigrams.
• BLEU-3 and BLEU-4 assess the model’s capacity to produce structurally cohesive phrases by looking at larger sequences of three (trigrams) and four words, respectively.

4. Discussion

This discussion examines the key findings from the pre-training stages using our fine-tuned BLIP-2 model, analyzes the implications for cross-modal learning in low-resource Arabic language, and contextualizes our results within the broader landscape of Arabic vision–language research, providing compelling evidence for the effectiveness of the BLIP-2 architecture in adapting to Arabic image captioning tasks. Measurements on generative vision-to-language learning with a frozen LLM demonstrated exceptional convergence behavior and rapid improvement in Arabic text generation capabilities. The results are described as follows:

1.

Language Modeling Loss (Figure 2a): The most significant finding from our experiments is the dramatic reduction in Language Modeling (LM) loss, which decreased from 3.001 at the beginning of Stage 2 to 2.681 by the end of the epoch, demonstrating a substantial improvement within training. This result is particularly noteworthy given that CamelBERT, the Arabic large language model with 340 M parameters, remained completely frozen throughout training. The rapid decline in LM loss demonstrates that the Q-Former effectively learns to translate visual features into the linguistic embedding space, enabling coherent Arabic text generation without modifying the underlying language model. The final LM loss value of 2.681 represents a strong foundation for Arabic caption generation. In the context of language modeling for morphologically rich languages like Arabic, loss values in this range typically correlate with:

• Fluent sentence construction with proper syntactic structure.
• Accurate word choice and semantic appropriateness.
• Coherent narrative flow across multi-word descriptions.
• Proper handling of Arabic’s right-to-left script and morphological complexity.

This achievement is particularly significant, given that Arabic presents unique challenges not found in English-language captioning, including rich inflectional morphology, complex word-formation patterns, and diglossia between Modern Standard Arabic and dialectal varieties. The model’s ability to substantially reduce LM loss in a single epoch suggests that the Q-Former architecture is exceptionally well-suited for bridging vision and language modalities, even for linguistically complex target languages.

2.

Image Text Matching Loss (Figure 2b): The Image–Text Matching (ITM) loss reveals a healthy and expected learning pattern that warrants careful interpretation. The ITM loss registered an unusually low value of 0.012, which, while seemingly excellent, actually indicated an overly simplistic matching strategy. This artificially low value suggested that the model, pre-trained in Stage 1, was making matching decisions with excessive confidence. As training progressed, the ITM loss stabilized at 0.076, indicating a critical normalization process. This increase is not a degradation in performance but rather evidence that the model is:

• Learning to discriminate subtle differences between matched and unmatched pairs.
• Developing a more nuanced understanding of image–text relationships.
• Calibrating its confidence to realistic levels.
• Moving from naive matching to sophisticated semantic understanding.

The final ITM value of 0.076 is exceptionally strong, well below the 0.10 threshold typically associated with high-quality matching performance. This performance translates to over 92% accuracy in distinguishing whether a given caption actually describes a given image.

3.

Image Text Contrastive Loss (Figure 2c): The Image–Text Contrastive (ITC) loss remained remarkably stable throughout training, maintaining values between 0.693 and 0.694 after the initial warmup phase. This stability provides critical validation of the two-stage pre-training strategy: the cross-modal alignment established during Stage 1 transfers effectively to the generative task in Stage 2 without degradation. The final ITC loss of 0.694 indicates:

• Strong alignment between image and text representations in the shared embedding space.
• Effective knowledge transfer from the vision encoder to the language model.
• Preserved visual understanding while optimizing for text generation.
• Robust foundation for the caption generation task.

The minimal fluctuation in ITC loss demonstrates that the model has found a stable equilibrium in the contrastive learning objective, allowing it to focus optimization efforts on the primary task of language generation.

4.

Total Loss (Figure 2d): The total loss trajectory, decreasing from 3.690 to 3.365, reflects the model’s integrated learning progress across all objectives. Notably, the Language Modeling component consistently contributed approximately 80% of the total loss, which is both expected and desirable in Stage 2, where text generation is the primary focus. The fluctuations observed in total loss (ranging from 3.17 to 3.88 across different batches) reflect:

• Natural variation in batch difficulty.
• Diverse visual scenes requiring different levels of linguistic complexity.
• Active learning from challenging examples.
• Healthy optimization dynamics without plateauing.

These fluctuations, rather than indicating instability, confirm that the model continues to learn from diverse examples and has not stagnated. The overall downward trend, combined with short-term variations, represents ideal learning behavior for deep neural networks.

5. Results

This section presents a comprehensive evaluation of our adapted BLIP-2 model for Arabic image captioning. We organize the results into four subsections: (1) retrieval performance comparing our fine-tuned model against the original BLIP-2 baseline in Section 5.1, (2) generation quality assessed through BLEU metrics and qualitative examples in Section 5.2, (3) learning dynamics analysis from the pre-training process in Section 5.3, and (4) comparative analysis with prior Arabic image captioning work in Section 5.4.

5.1. Retrieval Performance

Retrieval metrics (Recall@K) measure how well the model aligns images, as shown in the performance analysis in Figure 3 with their corresponding Arabic captions, providing insight into cross-modal understanding independent of language generation quality. Table 3 presents comprehensive retrieval results comparing our fine-tuned model against the original BLIP-2 baseline (zero-shot on Arabic). The models were evaluated on the test split of the translated Arabic Flickr 30k dataset. Inference was performed with a batch size of 32, employing a k_test value of 128 for validation to ensure a robust performance assessment [10,30].

Table 3. BLIP-2 Arabic captioning performance.

Metric	Original BLIP2 %	Our Fine-Tuned BLIP2 %	Interpretation
txt_r1	15.2	81.6	High single-caption accuracy
txt_r5	32.7	98.0	Near-perfect with 5 captions
txt_r10	41.3	99.4	Almost flawless with 10 captions
txt_r_mean	29.7	93.0	Strong overall text retrieval
img_r1	10.5	69.2	Good single-image retrieval
img_r5	24.8	91.52	Excellent with 5 candidates
img_r10	33.6	95.22	Very reliable with 10 candidates
img_r_mean	23.0	85.31	Solid cross-modal understanding
r_mean	26.4	89.16	Balanced performance

5.1.1. Text-to-Image Retrieval Analysis

Our fine-tuned model achieved exceptional text-to-image retrieval performance, with a txt_R@1 of 81.6%, demonstrating a great improvement of +66.4 percentage points over the original BLIP-2’s 15.2%. This indicates that, when given an Arabic caption, our model correctly identifies the corresponding image as its top match in over four out of five cases. The near-perfect txt_R@5 (98.0%) and txt_R@10 (99.4%) demonstrate that the correct image is almost always among the top candidates, confirming robust cross-modal alignment. These results indicate that the Q-Former successfully learns to map Arabic linguistic features to the visual embedding space, despite CamelBERT remaining frozen. The high retrieval accuracy confirms that our approach preserves semantic relationships between Arabic descriptions and visual content, addressing a key challenge in cross-lingual vision–language tasks [4,14].

5.1.2. Image-to-Text Retrieval Analysis

Image-to-text retrieval, while slightly lower than text-to-image, shows equally impressive gains. Our model achieved an img_R@1 of 69.2% (+58.7 pp), img_R@5 of 91.5% (+66.7 pp), and img_R@10 of 95.2% (+61.6 pp). The mean image retrieval score of 85.3% represents a 62.3 percentage-point improvement over the baseline. The slightly lower image-to-text performance compared to text-to-image (85.3% vs. 93.0% mean) is expected, as image-to-text retrieval requires selecting from five correct captions per image among thousands of distractors, which represents a more challenging task [10,30]. Nevertheless, the mean score of 85.3% demonstrates that our model effectively captures the relationship between visual features and Arabic linguistic descriptions.

5.1.3. Overall Cross-Modal Alignment

The balanced mean retrieval score (r_mean) of 89.2% represents a +62.8 percentage point improvement over the original BLIP-2 (26.4%). This comprehensive metric, computed by averaging across both retrieval directions, confirms that our adapted model achieves robust cross-modal understanding, which is essential for high-quality Arabic caption generation. So, the original BLIP-2’s poor zero-shot performance (r_mean 26.4%) confirms that unadapted English-centric vision–language models fail to generalize to morphologically rich languages like Arabic [15,16]. Our fine-tuned model’s dramatic improvement (89.2%) validates the effectiveness of our modular adaptation approach for low-resource languages.

5.2. BLEU Metrics Performance

While retrieval metrics measure cross-modal alignment, BLEU scores assess the quality of generated Arabic captions by comparing them to reference captions. Table 4 presents the BLEU scores for both models.

Table 4. BLEU score comparison for Arabic caption generation.

Metric	Original BLIP2 %	Our Fine-Tuned BLIP2 %
BLEU-1	12.3	75.5
BLEU-2	4.8	58.5
BLEU-3	1.9	46.2
BLEU-4	0.7	36.8

Our fine-tuned model achieved substantial improvements across all BLEU metrics, with BLEU-1 reaching 75.5, indicating high precision and strong vocabulary acquisition by correctly identifying individual words for key objects and attributes, while BLEU-2 at 58.5 demonstrates good local fluency through appropriate word combinations. The progressive decline to BLEU-3 (46.2) and BLEU-4 (36.8) is expected and typical in image captioning [10,24,25], as maintaining longer-range syntactic coherence is inherently more challenging. This declining pattern reveals important characteristics of our model’s generation capabilities: strong vocabulary acquisition (high BLEU-1), good local coherence (moderate BLEU-2/3), and moderate global structure (lower BLEU-4), where complete sentence structure sometimes lacks the fluency of human-written captions. This pattern aligns with findings in other low-resource language captioning work [23,24,39] and suggests that while our model excels at identifying visual content and expressing it with appropriate vocabulary, further refinement is needed for native-like syntactic fluency. To align quantitative analysis between captions generated by the original BLIP-2 and our fine-tuned model for images outside the training/test sets, Table 5, Table 6 and Table 7 present qualitative comparisons and illustrate the model’s capabilities and limitations through concrete examples. As we can see, the output description from our fine-tuned BLIP2 is close to the human description. The Arabic captions vary in accuracy, demonstrating both the model’s capabilities and limitations in understanding and describing visual content in Arabic. It also indicates that Arabic image captioning is an important research area for the future. For instance, some captions are coherent and accurately describe plausible scenes, such as:

• (two gray dogs sitting on green grass) يجلس أثنان من الكلاب الرمادية على العشب الأخضر
• (blue bird sitting on a branch) طائر ازرق يجلس على فرع شجرة

These examples suggest that the model can effectively recognize common objects and their interactions in images. However, other captions seem weak compared to a human description, such as:

• (a person walking through the snow on skis) المتزلج يسير عبر الثلج عند الغسق

5.3. Loss Analysis

Figure 2 presents the training and validation loss curves during pre-training Stage 2, providing insight into the model’s learning behavior. The Language Modeling (LM) loss decreased significantly from 3.001 to 2.681 despite CamelBERT being frozen, demonstrating that the Q-Former effectively learned to translate visual features into the linguistic embedding space required for Arabic text generation and confirming that cross-modal alignment was achieved without modifying the language model’s weights. The Image–Text Matching (ITM) loss increased from an overly confident 0.012 to a normalized 0.076, indicating the model moved from naive matching to nuanced understanding by learning to discriminate subtle differences between matched and unmatched pairs, with the final value below 0.10 corresponding to over 92% matching accuracy. The Image–Text Contrastive (ITC) loss remained remarkably stable between 0.693 and 0.694, confirming that the cross-modal alignment established in Stage 1 transferred effectively to Stage 2, preserving visual–linguistic relationships while optimizing for generation. The total loss showed a healthy downward trend from 3.690 to 3.365 with fluctuations between 3.17 and 3.88, reflecting active learning from diverse examples without stagnation, with the LM loss contributing approximately 80% of total loss, as expected for a generative task. These loss dynamics validate our two-stage pre-training strategy: Stage 1 successfully established cross-modal alignment (evidenced by stable ITC loss), while Stage 2 effectively optimized for text generation (evidenced by decreasing LM loss) without degrading alignment.

5.4. Comparison with Prior Work

Our approach offers several key advantages over prior work: First, it has a dataset scale advantage, as most previous studies used Flickr 8k (8000 images) or COCO subsets, whereas our Flickr 30k adaptation (31,783 images) provides approximately four times more training data samples, contributing to improved performance [18,40]. Second, it has an architectural advantage, as prior hybrid models [23,24,39] trained vision and language components together, requiring substantial data and compute, while our modular approach with frozen pre-trained components and a trainable Q-Former achieved superior results with only 188 M trainable parameters—far fewer than hundreds of millions—validating BLIP-2’s efficiency for low-resource languages [10,14]. Third, it features a substantial performance improvement, with our model outperforming the previous best BLEU-4 score of 16.65 [24] by +20.15 points (36.8 vs. 16.65), establishing a new state of the art for Arabic image captioning on translated benchmarks; and finally, it demonstrates superior retrieval performance, as our model achieved 81.6% txt_R@1 and 69.2% img_R@1 on the larger and more diverse Flickr 30k compared to prior work [26], reporting 67.3% Recall@1 on the smaller Flickr 8k, demonstrating enhanced cross-modal alignment. Table 8 situates our results within the broader landscape of Arabic image captioning research, enabling direct comparison with prior approaches.

Figure 3. BLIP-2 model performance comparison: original vs fine-tuned in Arabic captioning.

Figure 3. BLIP-2 model performance comparison: original vs fine-tuned in Arabic captioning.

Table 5. Comparison between the out-generated captions from the original BLIP-2 and our fine-tuned BLIP2 Arabic image captioning.

Input Image
Generated English caption using original BLIP2	a blue bird sits on a branch with white flowers	two golden retriever puppies sitting in the grass
Generated Arabic caption using Original BLIP2	طائر ازرق	كلب على العشب
Translated output Arabic caption from original BLIP2 into English using Google translation	blue bird	dogs on the grass
Generated Arabic caption using our fine-tuned BLIP2 model	طائر ازرق يجلس على فرع شجرة	يجلس اثنان من الكلاب الرمادية على العشب الاخضر
Translated output Arabic caption from fine-tuned BLIP2 into English using Google translation	blue bird sitting on a tree branch	Two gray dogs sat on the green grass

Table 5. Comparison between the out-generated captions from the original BLIP-2 and our fine-tuned BLIP2 Arabic image captioning.

Input Image
Generated English caption using original BLIP2	a blue bird sits on a branch with white flowers	two golden retriever puppies sitting in the grass
Generated Arabic caption using Original BLIP2	طائر ازرق	كلب على العشب
Translated output Arabic caption from original BLIP2 into English using Google translation	blue bird	dogs on the grass
Generated Arabic caption using our fine-tuned BLIP2 model	طائر ازرق يجلس على فرع شجرة	يجلس اثنان من الكلاب الرمادية على العشب الاخضر
Translated output Arabic caption from fine-tuned BLIP2 into English using Google translation	blue bird sitting on a tree branch	Two gray dogs sat on the green grass

Table 6. Comparison between the out-generated captions from the original BLIP-2 and our fine-tuned BLIP2 Arabic image captioning.

Input Image
Generated English caption using original BLIP2	a woman taking a photo with a camera	a tiger walking through the woods
Generated Arabic caption using Original BLIP2	إمرأة	نمر بجانب شجرة
Translated output Arabic caption from original BLIP2 into English using Google translation	woman	tiger next to a tree
Generated Arabic caption using our fine-tuned BLIP2 mode	امرأة تلتقط صورة بكاميرا رقمية	نمر ابيض و بنى يمشى عبر الغابة فى يوم مشمس
Translated output Arabic caption from fine-tuned BLIP2 into English using Google translation	woman taking a photo with a digital camera	a white and brown tiger walks through the forest on a sunny day.

Table 6. Comparison between the out-generated captions from the original BLIP-2 and our fine-tuned BLIP2 Arabic image captioning.

Input Image
Generated English caption using original BLIP2	a woman taking a photo with a camera	a tiger walking through the woods
Generated Arabic caption using Original BLIP2	إمرأة	نمر بجانب شجرة
Translated output Arabic caption from original BLIP2 into English using Google translation	woman	tiger next to a tree
Generated Arabic caption using our fine-tuned BLIP2 mode	امرأة تلتقط صورة بكاميرا رقمية	نمر ابيض و بنى يمشى عبر الغابة فى يوم مشمس
Translated output Arabic caption from fine-tuned BLIP2 into English using Google translation	woman taking a photo with a digital camera	a white and brown tiger walks through the forest on a sunny day.

Table 7. BLIP-2 Arabic image captioning model results in generating human-like captions.

Input Image
Generated English caption using original BLIP2	boats are docked on the beach near a hillside	a person walking through the snow on skis
Generated Arabic caption using Original BLIP2	قارب فى المياه	رجل على الجليد
Translated output Arabic caption from original BLIP2 into English using Google translation	boat in the water	man on ice
Generated Arabic caption using our fine-tuned BLIP2 mode	رصيف مع العديد من القوارب و المبانى القديمة فى الخلفية	المتزلج يسير عبر الثلج عند الغسق
Translated output Arabic caption from fine-tuned BLIP2 into English using Google translation	dock with many boats and old buildings in the background	a skier walks through the snow.

Table 7. BLIP-2 Arabic image captioning model results in generating human-like captions.

Input Image
Generated English caption using original BLIP2	boats are docked on the beach near a hillside	a person walking through the snow on skis
Generated Arabic caption using Original BLIP2	قارب فى المياه	رجل على الجليد
Translated output Arabic caption from original BLIP2 into English using Google translation	boat in the water	man on ice
Generated Arabic caption using our fine-tuned BLIP2 mode	رصيف مع العديد من القوارب و المبانى القديمة فى الخلفية	المتزلج يسير عبر الثلج عند الغسق
Translated output Arabic caption from fine-tuned BLIP2 into English using Google translation	dock with many boats and old buildings in the background	a skier walks through the snow.

Table 8. Comparative results for image captioning.

Ref.	Methodology	Dataset	Other Metric	BLEU Score
[3]	LLaMA-2, BLIP-2, Idefics-9B-Instruct	MS-COCO (Translated)	70% (TQA)	–
[19]	Object/Attention/Hybrid CNN-LSTM	MSCOCO+Flickr 8k (Arabic)	98% satisfactory	–
[26]	AraCLIP (AraBERT+ViT)	Flickr 8k (Arabic)	Recall@1: 67.3%	–
[39]	Hybrid CNN+RNN (VGG16/InceptionV3+LSTM/GRU)	Flickr 8k (Arabic)	–	36.5/21.4/12.0/6.6
[21]	Merge Model (VGG16+LSTM)	MS COCO+Flickr 8k (Human)	–	46/26/19/8
[18]	Hybrid CNN+RNN	Flickr 8k (Arabic)	–	36.5/21.4/12.0/6.6
[24]	SWIN+XCIT+ConvNexT+ CamelBERT+LSTM	Flickr 8k (Arabic)	–	59.80/40.03/30.56/16.65
[23]	AraBERT (OSCAR)	Flickr 8k (Arabic)	–	39.1/24.6/15/9.2
[10]	Original BLIP2 (ViT+Q-Former+OPT/FlanT5 language model) without any fine-tuning on Arabic data	Flickr 30k	R_mean: 26.4% txt/img: 29.7%/23.0%	12.3/4.8/1.9/0.7
Ours	BLIP-2 (ViT+Q-Former+CamelBERT)	Flickr 30k (Arabic)	R_mean: 89.16% txt/img: 93.0%/85.31%	75.5/58.5/46.2/36.8

Table 8. Comparative results for image captioning.

Ref.	Methodology	Dataset	Other Metric	BLEU Score
[3]	LLaMA-2, BLIP-2, Idefics-9B-Instruct	MS-COCO (Translated)	70% (TQA)	–
[19]	Object/Attention/Hybrid CNN-LSTM	MSCOCO+Flickr 8k (Arabic)	98% satisfactory	–
[26]	AraCLIP (AraBERT+ViT)	Flickr 8k (Arabic)	Recall@1: 67.3%	–
[39]	Hybrid CNN+RNN (VGG16/InceptionV3+LSTM/GRU)	Flickr 8k (Arabic)	–	36.5/21.4/12.0/6.6
[21]	Merge Model (VGG16+LSTM)	MS COCO+Flickr 8k (Human)	–	46/26/19/8
[18]	Hybrid CNN+RNN	Flickr 8k (Arabic)	–	36.5/21.4/12.0/6.6
[24]	SWIN+XCIT+ConvNexT+ CamelBERT+LSTM	Flickr 8k (Arabic)	–	59.80/40.03/30.56/16.65
[23]	AraBERT (OSCAR)	Flickr 8k (Arabic)	–	39.1/24.6/15/9.2
[10]	Original BLIP2 (ViT+Q-Former+OPT/FlanT5 language model) without any fine-tuning on Arabic data	Flickr 30k	R_mean: 26.4% txt/img: 29.7%/23.0%	12.3/4.8/1.9/0.7
Ours	BLIP-2 (ViT+Q-Former+CamelBERT)	Flickr 30k (Arabic)	R_mean: 89.16% txt/img: 93.0%/85.31%	75.5/58.5/46.2/36.8

6. Conclusions

Arabic image captioning remains an under-explored challenge in computer vision and natural language processing due to the language’s rich morphology, right-to-left script, diglossia, and the scarcity of high-quality annotated datasets [15,16,17]. These factors have hindered the development of robust captioning systems capable of generating accurate and contextually relevant descriptions for over 400 million Arabic speakers worldwide, despite significant progress in English-based models [9,10,12]. This paper addressed this gap by presenting the first adaptation of the BLIP-2 architecture [10] for Arabic image captioning, replacing its original English-centric language model with the Arabic large language model CamelBERT [28] while retaining a frozen ViT image encoder. The key innovation—the Q-Former—bridges visual and linguistic modalities through a two-stage pre-training process, enabling strong performance in Arabic image captioning and retrieval tasks with minimal trainable parameters (188 M). To overcome data scarcity, we constructed and released a new Arabic dataset by translating the entire Flickr 30k benchmark [30] using the Google Cloud Translation API, following methodologies successfully employed for other low-resource languages [31,32], yielding 31,783 images with five Arabic captions each (158,915 total pairs).

Our comprehensive experiments demonstrate the transformative impact of this adaptation. The original BLIP-2 model, without any fine-tuning on Arabic data, exhibited near-zero capability for Arabic caption generation, achieving a mean retrieval score (r_mean) of only 26.4% (Table 3), confirming that unadapted English-centric vision–language models fail to generalize to morphologically rich languages [15,16]. In contrast, our fine-tuned model achieved exceptional performance, with a mean retrieval score of 89.2%—a substantial improvement of +62.8 percentage points (Table 3, Figure 3). Specifically, it attained 93.0% text retrieval accuracy (txt_R_mean) and 85.3% image retrieval accuracy (img_R_mean), demonstrating robust cross-modal alignment (Section 5.1). Using BLEU metrics for quantitative assessment (Table 4), our model achieved a BLEU-1 of 75.5, a BLEU-2 of 58.5, a BLEU-3 of 46.2, and a BLEU-4 of 36.8—dramatically outperforming the original BLIP-2 across all metrics and establishing a new state-of-the-art standard for Arabic image captioning on translated benchmarks (Table 8). Loss analysis (Figure 2) validates our two-stage strategy: Language Modeling loss decreased from 3.001 to 2.681 despite frozen CamelBERT, proving the Q-Former’s effectiveness in bridging vision and language, while stable Image–Text Contrastive loss (0.693–0.694) confirms preserved alignment from Stage 1 [10,11,27].

While these results confirm that our approach successfully handles Arabic’s linguistic complexities, some limitations persist, including omitted fine details, occasional syntactic awkwardness that reflects Arabic’s complex morphology, and translation artifacts from the machine-translated dataset [6,15,23]. Future work will focus on refining linguistic precision through additional fine-tuning on native Arabic datasets [20], expanding the training data with diverse dialectal examples [25], enhancing visual understanding for nuanced inputs through improved attention mechanisms [41] or diffusion-based approaches [12,13], and extending our modular approach to other low-resource languages [4,14,31]. This work represents a significant step toward enhancing accessibility for Arabic-speaking communities and bridging the gap in multilingual vision–language research, demonstrating that resource-efficient adaptation of English-centric models can achieve strong performance in morphologically complex languages, with applications in assistive tools, education, and content creation.