MFEAM: Multi-View Feature Enhanced Attention Model for Image Captioning

Abstract

Image captioning plays a crucial role in aligning visual content with natural language, serving as a key step toward effective cross-modal understanding. Transformer has become the dominant language model in image captioning. Existing Transformer-based models seldom highlight important features from multiple views in the use of self-attention. In this paper, we propose MFEAM, an innovative network that leverages the multi-view feature enhanced attention. To accurately represent the entangled features of vision and text, the teacher model employs the multi-view feature enhanced attention to guide the student model training through knowledge distillation and model averaging from both visual and textual views. To mitigate the impact of excessive feature enhancement, the student model divides the decoding layer into two groups, which separately process instance features and the relationships between instances. Experimental results demonstrate that MFEAM attains competitive performance on the MSCOCO (Microsoft Common Objects in Context) when trained without leveraging external data.

1. Introduction

Images and text serve as two important mediums for conveying information. Images encompass more extensive information, but they are more challenging to interpret. On the other hand, text conveys information in a more straightforward manner. Early researches utilize Convolutional Neural Network (CNN) [1,2,3,4] and Recurrent Neural Network (RNN) [1,2,4,5,6]. RNN facilitates the transmission of information over different time steps, efficiently capturing temporal dependencies within the data. Nevertheless, RNN encounter difficulties such as gradient vanishing and explosion, which significantly limit the capacity to capture long-distance dependencies, especially when processing lengthy sequences. The Long Short-Term Memory (LSTM) network [7,8,9,10] represents a substantial advancement over RNN. The advent of the Transformer has rectified the computational inefficiency of LSTM in handling lengthy sequences, facilitating parallel processing of the input sequence and enhancing training speed. In addition, vision-language pre-trained models have dramatically enhanced the capability of cross-modal applications, such as Contrastive Language–Image Pretraining (CLIP) [11], by learning cross-modal alignment representations on large-scale image-text pairs. Nowadays, image captioning methods have significantly advanced with the adoption of Transformer-based models [12,13,14,15], while the visual feature extraction phase is swiftly advancing, transitioning to multimodal architectures [11,16,17,18] trained on large-scale datasets.

Despite advances in architectures and structural networks, most contemporary image captioning methods utilize a single language model to generate captions. Recently, the structure of two language models [13,19] has been employed to learn visual representations through knowledge distillation and model averaging [20,21,22]. Meanwhile, many image captioning datasets [23,24,25,26,27] contain large amounts of images and text, resulting in highly complex and diverse input features. As a result, Transformer-based image captioning models are capable of effectively extracting visual and textual features, but the characteristics of the input result in suboptimal performance in accurately expressing the entangled features of vision and text. To address this challenge, we introduce the multi-view feature enhanced attention to strengthen the representation of entangled features of vision and text. Through the integration of the Mean Teacher Learning approach, the teacher model can attain a superior comprehension of important features, facilitating more effective parameter updates for the student model.

In this paper, we propose a knowledge distillation framework named MFEAM, which is built upon a multi-view feature enhanced attention. MFEAM adopts an encoder–decoder architecture as its foundational framework. The CLIP model functions as the visual encoder, while the language decoder comprises the teacher model and the student model. The teacher model employs the multi-view feature enhanced attention to interpret important features from both visual and textual views. Through knowledge distillation and model averaging, the teacher model can gradually follow the state of the student model through time. In addition, inspired by the Guided-Embedding Network [28], we divide the decoder of the student model into two groups to separately process instance and relationship features, which helps prevent performance degradation caused by overemphasizing features. We formulate and evaluate strategies that implement this interaction paradigm during training phase. Experimental results demonstrate that MFEAM attains competitive performance on the MSCOCO when trained without leveraging external data.

• We present a novel architecture called MFEAM, which incorporates the multi-view feature enhanced attention. By overcoming the limitations of relying on a single language model as the decoder and integrating the advanced Mean Teacher architecture, the model significantly improves upon baseline performance.
• To strengthen the ability of the model in handling the entangled features of vision and text, we propose the multi-view feature enhanced attention (MFE). MFE is made up of the self-enhanced module and the sliding window module, which collaboratively assist the teacher model in comprehending entangled features from both visual and textual views.
• To mitigate performance degradation caused by overemphasis on certain features, a novel feature processing strategy is adopted in the student model. Specifically, the decoder layers are divided into two groups, with one focusing on processing instance features and the other responsible for capturing relational features between instances.

2. Related Work

2.1. Image Captioning

There has been a marked enhancement in the effectiveness of image captioning in recent years. Similar to machine translation, image captioning research [13,14,29,30,31] follows the encoder–decoder architecture. The visual encoder extracts spatial information and learns visual representations from images. Lu et al. [9] use an attention mechanism module to predict the next word, deciding whether to emphasize visual content or rely on language signals. Traditional language models have mainly relied on RNNs, but they struggle with information loss and training difficulties in long sequences. Yao et al. [32] address this by demonstrating a two-layer LSTM structure designed to investigate the connections between visual features and components of the generated caption. However, with the emergence and success of the Transformer architecture [33], the encoder–decoder structure used for image captions has increasingly converged towards the Transformer paradigm. Our approach is predicated on Transformer-based image captioning and improves the capacity of the model to process the entangled features of vision and text through the introduction of the multi-view feature enhanced attention.

2.2. Knowledge Distillation

Knowledge distillation is first introduced in [34], where it enhances the effectiveness of the student model by transferring the teacher model’s supervisory signal, demonstrating broad applicability in multimodal tasks. In generation tasks, distillation techniques can improve the ability of the model to capture long-range semantic dependencies through feature-level or outline-level constraints, while reducing the overfitting issue resulting from limited training data. Besides logits, other forms of knowledge, such as intermediate representations and attention maps [13,35,36,37], have been utilized to transfer the information embedded within models. Zhou et al. [38] present the POS-SCAN. Utilizing a Part-of-Speech tagger, the model preserves noun words during the calculation of the matching score, and the subsequently re-trained POS-enhanced POS-SCAN fulfills the criteria of the downstream task. Bajpai et al. [39] present a novel self-distillation structure, which improves model performance by introducing an online learning algorithm to dynamically select the optimal threshold. In our case, we present a knowledge distillation architecture utilizing the multi-view feature enhanced attention. The teacher model enhances the student model using the multi-view feature enhanced attention. The teacher model leverages the multi-view feature enhanced attention to facilitate a deeper understanding of features within the student model.

2.3. Vision-Language Pretrained Model

In recent years, pre-training techniques based on large-scale cross-modal alignment have achieved notable advancements in multimodal comprehension. The CLIP model proposed by OpenAI is trained on extensive image-text pairs, enabling its visual encoder to extract semantically highly generalized visual features [11]. Models such as Google’s ALIGN extend the training dataset and model scale, and they verify the effectiveness and versatility of the cross-modal feature learning pre-training strategy [40,41]. These models lay the foundation for multimodal generation tasks through a unified embedding space.

2.4. Dynamic Training Strategy

The dynamic training strategy enhances the model’s robustness by adaptively modifying the optimization process. The Mean Teacher architecture [19] uses the exponential moving average parameter to generate a regularization target, which is proven to be effective in semi-supervised classification tasks. In the generation task, reinforcement learning is used for multi-objective optimization, and the diversity and accuracy of the generated text are further improved by directly optimizing CIDEr [42] and other indicators. Methods such as SCST [5] provide technical support for training stability.

3. Method

As shown in Figure 1a, the main network follows the encoder–decoder paradigm, where the visual encoder utilizes the CLIP model [11] for feature extraction. The language decoder, which comprises the teacher and student model, is responsible for transforming extracted features into captions. The next section provides an in-depth review of the teacher model, elucidating the fundamental principles of the multi-view feature enhanced attention within the teacher model. In addition, we provide a detailed description of the architecture of the student model, which divides the six decoder layers into two structurally identical groups to effectively alleviate performance degradation resulting from an overemphasis on local features. Finally, we elucidate the dual-stage training strategy that is frequently used for image captioning.

3.1. Visual Encoder

As depicted in Figure 1a, we employ the CLIP network as the visual encoder to extract visual features $F=\left\{f_1,f_2,f_3,\dots ,f_n\right\}$, where $f_i\in R^{1\times d}$. CLIP converts image information into high-dimensional feature vectors, capturing both global and local features within the image. Specifically, for an image I, we derive the feature embedding $e^I\in R^d$ by using the CLIP model. A two-layer MLP is utilized to produce the feature set. Formally,

$$e^I=CLIP\left(I\right)$$

(1)

$$\left\{f_1,f_2,\dots ,f_n\right\}=MLP\left(e^I\right)$$

(2)

The visual features are fed into the language decoder, where the attention mechanism is employed to enhance information interaction among the features. Compared to conventional convolutional neural network approaches, CLIP provides more semantically rich and generalizable feature representations, enabling it to better accommodate image understanding tasks across diverse scenarios.

3.2. Teacher Language Model

As depicted in Figure 1b, the teacher model is the Transformer-based language model, incorporating the multi-view feature enhanced attention. The features F are sent to the encoder. After being processed by multiple encoding layers, the contextual features $F_c$ are obtained. These contextual features are passed to each decoding layer to facilitate the development of image descriptions. During the decoding stage, the multi-view feature enhanced attention not only captures the dependencies within the partially generated sequence but also further extracts important details from the $F_c$.

The key idea of the MFE is to refine attention weights from two complementary views—the textual view and the visual view—improving the semantic representation capability across modalities of the model. Specifically, the self-enhanced module strengthens attention weights by analyzing their distribution, encouraging the model to focus more on semantically informative tokens. In parallel, the sliding window module applies a convolutional kernel over the visual attention map, enhancing the model’s spatial awareness and its ability to capture local structure. The outputs from both views are then fused and used to reweight the value vectors, resulting in more expressive multi-modal representations. The overall architecture is illustrated in Figure 2. Firstly, we compute this attention weight $at{t}_i$ using the input vectors $q_i$ and $k_i$, as defined by the following equation:

$${att}_i=Softmax\left({\displaystyle \frac{q_i{k}_i^T}{\sqrt{d_k}}}\right)$$

(3)

Compared to the self-attention mechanism, which performs calculations between ${att}_i$ and $v_i$, the multi-view feature enhanced attention separately processes the attention weights from two views-visual features ${att}_i^I$ and textual features ${att}_i^T$.

For the attention weights of textual features ${att}_i^T$, we enhance the textual features by the self-enhanced module.

$$\alpha =1-max\left({att}_i^T\right)+min\left({att}_i^T\right)$$

(4)

$${EA\_att}_i^T=\left(\left(e^{\alpha }+1\right)·\gamma \right)·{att}_i^T$$

(5)

In Equation (4), $\alpha$ reflects the flatness of the attention distribution. When $max\left({att}_i^T\right)$ and $min\left({att}_i^T\right)$ are close, $\alpha$ becomes larger, indicating that the model lacks a strong semantic focus. A higher $\alpha$ increases the overall magnitude of the attention to encourage more discriminative attention weights. In Equation (5) we use $\gamma$ as a weight parameter to prevent excessive enhancement of ${att}_i^T$. Over-enhancing ${att}_i^T$ can lead to biased understanding of features by the model, resulting in an unstable training process. This adaptive mechanism enables more stable and effective modeling of textual focus in ambiguous or diffuse semantic contexts.

For the attention weights of visual features ${att}_i^I$, we enhance the visual features by the sliding window module. In contrast to the self-enhanced module, the sliding window module employs a convolution-based approach, where the convolutional kernel slides over the attention weight matrix to capture localized features and enhance the spatial awareness of the model. The convolutional kernel is defined as a 3 × 3 sharpening kernel.

$$H=\left[\begin{array}{ccc}0& -1& 0\\ -1& 5& -1\\ 0& -1& 0\end{array}\right]$$

(6)

$$EA\_at{t}_i^I=Conv2D\left(at{t}_i^I\right)$$

(7)

where $Conv2D$ denotes the two-dimensional convolution operation. The sliding window mechanism applies convolutional kernels over the surrounding regions of each position, enabling the model to effectively capture local variations.

Lastly, we fuse the attention weights of visual features and textual features, and perform a dot product operation with $v_i$ to obtain the output $M_i$.

$${EA\_att}_i=Concat\left({EA\_att}_i^I,{EA\_att}_i^T\right)$$

(8)

$$M_i={EA\_att}_i·v_i$$

(9)

where ${EA\_att}_i$ is the attention weight obtained after the fusion of ${EA\_att}_i^T$ and ${EA\_att}_i^I$.

In this way, the teacher model optimizes attention weight allocation from visual and textual feature views, enabling MFEAM to emphasize important information while avoiding redundant processing of unnecessary information. During the subsequent knowledge distillation process, the teacher model assists the student model understand features effectively, enhancing overall performance and accuracy.

3.3. Student Language Model

The student model adopts the Transformer-based architecture. Different from Transformer that process features layer by layer, inspired by the Guided-Embedding Network [28], we divide the structure of six decoder layers into two groups, while maintaining the identical architecture. As illustrated in Figure 3, one group functions as the Instance Decoder, responsible for detecting objects in the image and generating the corresponding features for each instance. The other group, termed the Relationship Decoder, focuses on identifying interaction relationships among objects.

As shown in Figure 3, the features F are sent to the encoder. After processing through multiple encoder layers, the encoder generates the contextual features $F_c$. During the decoding stage, two different attention mechanisms are employed to extract important information from $F_c$. In the instance decoder, the features generated by each decoder layer are fed into the relationship decoder. The calculation process is as follows:

$$R_0^{ID}=SelfAttn\left(F_c\right)$$

(10)

$$R_i^{ID}=CrossAttn\left(R_{i-1}^{ID},F_c\right)$$

(11)

where $i=1,2,3$. In the relationship decoder, to guide the interaction features $R_i^{ID}$ to explore informative regions, we take $R_i^{ID}$ as input and send it to the i-th relationship decoder layer.

By leveraging the output of the instance decoder, each interaction feature in the relationship decoder can be directly linked to the corresponding object. In MFEAM, the decoder structure of the student model can better focus on the interaction relationships between targets after receiving feature knowledge from the teacher model. This mechanism helps mitigate performance degradation caused by excessive emphasis on features.

3.4. Interaction Strategy

The interaction strategy consists of knowledge distillation and model averaging. In the training phase, we leverage the teacher model to provide regression targets. This process is facilitated by knowledge distillation, where the teacher model updates its parameters according to the exponentially weighted. Given the time step $\tau$, the loss function is as follows:

$$L\left({\theta }_s\right)=\underset{{\theta }_s}{min}{\sum }_{\tau }{\left(f\left({\theta }_t\right)-f\left({\theta }_s\right)\right)}^2$$

(12)

where ${\theta }_s$ and ${\theta }_t$ are the parameter sets of the student model and the teacher model, respectively. $f\left({\theta }_s\right)$ and $f\left({\theta }_t\right)$ are the predicted probabilities by the student model and the teacher model.

In model averaging, at the time step $\tau$, the student model dynamically adjusts the teacher model based on the historical average of its model parameters, helping the teacher model provide a smoother and more stable learning target.

$$\sigma {\theta }_t+\left(1-\sigma \right){\theta }_s\to {\theta }_t$$

(13)

where $\sigma$ is a target decay rate. During the training process, the value of $\sigma$ is fixed. This method allows the teacher model to sustain a weighted average of the student model’s sequential states, achieving a type of model ensembling.

3.5. Training Strategy

The training procedure is divided into two main stages as follows: an initial phase utilizing cross-entropy loss and a subsequent stage based on self-critical sequence optimization. The cross-entropy loss encourages the model toward producing more precise and semantically relevant textual outputs by assigning a higher penalty to low probabilities by taking the logarithm of the probability the model assigns to the true word. The learning rate is modified according to the Transformer’s learning rate scheduling methodology.

$$L_{XE}\left(\theta \right)=-{\sum }_{t=1}^{T}log\left(p_{\theta }\left(y_t^{*}\mid y_{1:t-1}^{*},I\right)\right)$$

(14)

where t indicates time, and $\theta$ denotes the set of all model parameters.

In the self-critical sequence training phase (SCST) [5], the model first samples a sequence and computes its evaluation metric score. Simultaneously, a baseline sequence is generated using greedy decoding as a reference. The disparity in scores between the sampled and baseline sequences is used as a reward signal, encouraging the generation of higher-scoring sequences.

$${▽}_{\theta }L\left(\theta \right)=-{\displaystyle \frac{1}{k}}\sum _{i=1}^k\left(\left(r\left(y_{1:T}^i\right)-b\right){▽}_{\theta }logD\left(y_{1:T}^i\right)\right)$$

(15)

where $y_{1:T}^i$ is the i-th sentence, k is the beam size, $r\left(·\right)$ is the reward function, and $b={\displaystyle \frac{1}{k}}{\sum }_{i=1}^k\left(r\left(y_{1:T}^i\right)\right)$ is the baseline, calculated as the average reward obtained for the sampled sequences.

4. Experiments

4.1. Experimental Setup

4.1.1. Dataset

This study employs the MSCOCO dataset [43], a prominent standard in image captioning. This dataset has more than 120,000 images and every image is accompanied by five manually created annotations. In our tests, we employ the Karpathy split method to partition the dataset to ensure a thorough and balanced dataset for assessing the efficacy of our proposed model.

4.1.2. Evaluation Metrics

To quantitatively evaluate the quality of the generated captions, we adopt several widely used benchmark metrics, including BLEU-1/4 [44], METEOR [45], ROUGE [46], CIDEr [42], and SPICE [47]. These metrics are abbreviated as B-1, B-4, M, R, C, and S. In our experimental analysis, we employ the official evaluation scripts released by MSCOCO, ensuring a fair comparison from previous models.

4.1.3. Implementation Details

In our study, the presented model was developed with PyTorch 1.9.0, a widely used framework for deep learning. The visual encoder comprised of three layers, each with a dimensionality of 512. Similarly, the decoder also consisted of three layers, each incorporating a feed-forward network with a dimensional size of 2048. In the experimental setup, the Adam optimizer is employed with a beam size of 5. For word representation, we employed Byte Pair Encoding (BPE) [48]. Additionally, standard sinusoidal positional encoding [33] was employed to ensure the positional representation of every word.

During the cross-entropy training phase, we set the batch size to 50. During the SCST fine-tuning phase, we employed a reduced batch size of 30. Our experiments were conducted on a standard desktop machine operating on Ubuntu 22.04, equipped with three RTX 8000 GPUs (NVIDIA, Santa Clara, CA, USA).

4.2. Experimental Results

Table 1. Performance of MFEAM on the MSCOCO dataset.

Method	B-1	B-4	M	R	C	S
Up-Down [4]	79.8	36.3	27.7	56.9	120.1	21.4
GCN-LSTM [9]	80.5	38.3	28.5	58.3	127.6	22.0
AoANet [51]	80.2	38.9	29.2	58.8	129.8	22.4
EURAIC [56]	80.9	39.5	29.4	59.4	130.3	-
HIP [49]	-	39.1	28.9	59.2	130.6	22.3
VAT [29]	81.2	39.0	29.3	59.4	131.8	22.8
X-Transformer [52]	80.9	39.7	29.5	59.1	132.8	23.4
DLCT [53]	81.4	39.8	29.5	59.1	133.8	23.0
RSTNet [54]	81.8	40.1	29.8	59.5	135.6	23.3
DIFNet [50]	81.7	40.0	29.7	59.4	136.2	23.2
BENet [55]	82.1	40.3	30.0	59.7	137.6	23.6
MFEAM	82.8	40.8	30.5	60.2	140.1	24.3

presents a comparative analysis of the SCST training results between MFEAM and other image captioning models. In the comparative analysis, we incorporate models based on LSTM and methods that focus on regional attention mechanisms (i.e., Up-Down [4], HIP [49]), approaches utilizing the dual information flow fusion (such as DIFNet [50]), methods enhanced by graph network structures (i.e., GCN-LSTM [9]), models employing self-attention mechanisms (i.e., AoANet [51]), and models with decoders primarily structured around Transformers (i.e., X-Transformer [52], DLCT [53], RSTNet [54], BENet [55], VAT [29] and EURAIC [56]).

As shown in Table 1, MFEAM achieves consistently superior performance across standard evaluation metrics such as BLEU-1, BLEU-4, METEOR, ROUGE, CIDEr, and SPICE. For example, on the MSCOCO Karpathy test split, MFEAM achieves the highest CIDEr score of 140.1, outperforming prior Transformer-based models. This improvement stems from the combination of the multi-view feature enhanced attention and the Mean Teacher Learning approach, which strengthens the model’s ability to encode cross-modal semantics.

We compare MFEAM with several strong baselines in terms of parameter size and inference speed. As shown in Figure 4, MFEAM achieves the highest CIDEr score, while also having the smallest parameter size and second-fastest inference time. These results highlight the efficiency of our architecture and its suitability for real-time or resource-constrained scenarios.

4.3. Ablation Study

As presented in

Table 2. Results of enhancement modules when training with cross-entropy loss.

SEM	SWM	B-1	B-4	M	R	C	S
-	-	77.8	37.7	28.6	57.5	122.8	21.5
-	√	78.2	38.1	28.9	57.8	123.5	21.8
√	-	78.6	38.9	29.1	58.6	125.3	22.1
√	√	79.2	39.7	29.3	59.1	127.3	22.4

Note: “√” indicates that the corresponding enhancement module (SEM or SWM) is enabled, while “-” indicates that the module is not used.

, different combinations of the enhancement module are evaluated to comprehensively demonstrate the impact of the multi-view feature enhanced attention. As illustrated in

Table 3. Ablation study on the use of the model averaging when training with cross-entropy loss.

Model	Model Averaging	B-1	B-4	M	R	C	S
CLIP-VIT-B/16	-	78.0	38.1	28.9	58.2	123.1	21.8
CLIP-VIT-B/32	-	76.3	36.6	28.1	56.8	118.3	21.1
CLIP-RN50	-	75.9	36.3	27.8	56.6	115.9	20.6
CLIP-RN101	-	76.6	37.1	28.1	57.1	118.7	21.0
CLIP-RN50×4	-	77.3	37.8	28.5	57.7	122.0	21.5
CLIP-RN50×16	-	78.7	39.1	29.1	58.8	126.7	22.3
MFEAM	√	79.2	39.7	29.3	59.1	127.3	22.4

Note: “√” indicates that model averaging is enabled in the corresponding configuration, while “-” indicates that it is not used.

, Experiments are performed to validate the efficacy of the model averaging. Furthermore, as shown in

Table 4. Results of different image feature extraction modules when training with cross-entropy loss.

Model	MFEAM	B-1	B-4	M	R	C	S
CLIP-VIT-B/16	-	77.8	38.0	29.1	58.0	122.5	21.8
CLIP-VIT-B/16	√	78.8	38.9	29.2	58.6	125.7	22.0
CLIP-VIT-B/32	-	76.5	36.7	28.1	56.9	118.1	21.0
CLIP-VIT-B/32	√	76.5	36.9	27.9	56.9	119.7	21.1
CLIP-RN50	-	75.7	36.2	28.1	56.7	115.7	20.8
CLIP-RN50	√	76.3	36.6	28.2	56.8	116.2	20.7
CLIP-RN101	-	76.5	37.2	28.4	57.2	118.6	21.2
CLIP-RN101	√	77.1	37.4	28.5	57.5	119.0	21.1
CLIP-RN50×4	-	77.2	37.6	28.7	57.6	121.9	21.6
CLIP-RN50×4	√	77.9	38.3	28.6	58.0	122.4	21.7
CLIP-RN50×16	-	78.0	38.8	29.4	58.6	125.0	22.2
CLIP-RN50×16	√	79.2	39.7	29.3	59.1	127.3	22.4

Note: “√” indicates that the proposed MFEAM architecture is used, while “-” indicates that a single language model serves as the language decoder.

, comprehensive ablation experiments are conducted to assess the impact of the Mean Teacher architecture.

Role of the multi-view feature enhanced attention. Table 2 presents the experimental results of the enhancement modules, where the models employing individual enhancement modules, namely the self-enhanced Module (SEM) and the sliding window module (SWM), are compared with the proposed MFEAM model. As can be seen, MFEAM achieves competitive performance compared to models that process entangled features from either the visual or textual view alone. The evaluation metrics of models using only the self-enhanced module or the sliding window module indicate that relying on a single enhancement module leads the teacher model to focus solely on either the visual or textual modality, which limits its ability to capture rich and comprehensive features. In contrast, MFEAM offers a more diverse mechanism for processing image features, enabling a more precise representation of entangled features of vision and text from both views.

Role of the model averaging. Table 3 presents the results obtained without applying the model averaging operation. In this setting, knowledge is transferred from the teacher model to the student model by optimizing a distillation-based objective that encourages the student to mimic the output behavior of the teacher. Meanwhile, instead of relying solely on static supervision, the teacher model dynamically incorporates guidance from the student model by tracking and utilizing the exponential moving average of the student’s parameters, thereby promoting a mutually reinforcing optimization process. This collaborative interaction the teacher model to provide a smoother and more stable learning target for the student model, thereby effectively reducing uncertainty during training. The performance of MFEAM which uses CLIP-RN50×16 and model averaging significantly surpasses that of the approach using CLIP-RN50×16 without model averaging. Model averaging enables the teacher model to retain a temporally weighted accumulation of the student model’s evolving parameters. This mechanism effectively serves as a form of model ensembling, enhancing the generalization capacity of the teacher model.

Role of the Mean Teacher architecture. We evaluate the performance of MFEAM using several image feature extraction modules, as indicated in Table 4. For models without the MFEAM architecture, a single language model is designated as the language decoder. The integration of all previously discussed components within the MFEAM framework leads to a substantial performance improvement, highlighting the effectiveness of the architecture. These results empirically validate the suitability of adopting the mean teacher learning paradigm. The results in Table 4 demonstrate that MFEAM attains the highest performance when employing the CLIP-RN50×16 in conjunction with the Mean Teacher architecture, where the distillation weight is set to 0.1. This configuration facilitates an effective balance between the supervisory signal provided by the teacher model and the representational learning capacity, leading to the enhanced alignment of visual and textual modalities.

4.4. Qualitative Studies

Figure 5 showcases image examples with ground truth and captions produced by CaMEL [13] and MFEAM, respectively. The qualitative examples show that captions produced by MFEAM adhere more closely to the surrounding context and exhibit markedly smaller semantic deviations. As illustrated in Figure 5, MFEAM accurately recognizes the spatial dependencies between objects present in the visual scene. The model also excels at capturing fine-grained information, particularly evident in its correct identification of both the cat’s color and its behavior in the third panel of Figure 5. These results suggest that the proposed MFEAM architecture conveys scene content and details with higher fidelity than CaMEL, offering improved alignment between visual semantics and textual descriptions. This advantage is attributed to the multi-view feature enhanced attention, which enables better cross-modal interaction. Taken together with the quantitative gains, these qualitative observations provide strong evidence for the superiority of MFEAM in generating contextually rich and semantically accurate captions.

4.5. Discussion

The results presented in Table 1 demonstrate that MFEAM achieves state-of-the-art performance across all major evaluation metrics, including BLEU, METEOR, ROUGE, CIDEr, and SPICE. In particular, MFEAM attains a CIDEr score of 140.1, outperforming strong baselines such as BENet (137.6), RSTNet (135.6), and DLCT (133.8). This improvement can be attributed to the dual-channel attention refinement strategy introduced by the multi-view feature enhanced attention, as well as the decoupled decoding structure in the student model.

In addition to the quantitative gains, qualitative results in Figure 5 further illustrate the effectiveness of MFEAM in generating accurate, context-aware, and semantically rich captions. Compared to CaMEL, MFEAM produces descriptions with smaller semantic deviations and better alignment with spatial and relational cues in the image.

Furthermore, compared with recently proposed methods, MFEAM offers a competitive advantage by achieving high caption quality without relying on external data or excessive model complexity. These results confirm the effectiveness of our framework in bridging visual and textual modalities while maintaining computational efficiency.

5. Conclusions

In this paper, we present a novel architecture named MFEAM, which incorporates the multi-view feature enhanced attention and knowledge distillation framework for image captioning. The model employs the CLIP architecture to encode image features, followed by a combination of knowledge distillation and model averaging to transfer the learned features. This mechanism enables the teacher model to guide the training process effectively. Specifically, the teacher model incorporates the multi-view feature enhanced attention mechanism, which integrates the self-enhanced module and the sliding window module to process the entangled features of vision and text from diverse views. This design enriches the generation of descriptions by improving the understanding of complex visual and textual interactions. In addition, a novel feature processing strategy is introduced by dividing the decoder layers into two groups in the student model as follows: one dedicated to capturing instance-specific features and the other to modeling inter-instance relational features. This approach mitigates potential performance degradation caused by excessive feature emphasis. Experimental results on the MSCOCO demonstrate that MFEAM outperforms existing methods in terms of accuracy and caption quality. In future work, we aim to further reduce the model’s parameter count and improve inference speed by exploring lightweight attention modules and compression strategies, enabling broader applicability in real-time and resource-constrained scenarios.