LAPM-RA: Reward-Adaptive Prompt Learning with LLM Augmentation for Multimodal Sentiment Analysis

Abstract

Few-shot multimodal sentiment analysis (MSA), constrained by limited annotated data, often suffers from large cross-modal semantic alignment gaps and difficulty in capturing fine-grained sentiment cues, making it challenging to fully exploit the complementary information between text and images. Recent advances in large language models (LLMs) and prompt learning have shown strong potential for improving label efficiency in low-resource natural language processing tasks; however, their direct application to MSA is hindered by static prompt designs and shallow cross-modal integration. To overcome these limitations, we propose LLM-Augmented Prompt Learning for Multimodal Sentiment Analysis with Reward Adaptation (LAPM-RA), a unified framework integrating LLM-based sentiment-aware augmentation, reward-guided prompt selection, and context-aware multimodal fusion. Specifically, LLMs generates sentiment-consistent and counterfactual text variants to enhance lexical and structural diversity while preserving label fidelity; a supervised policy network adaptively selects optimal prompt templates based on reward signals; and a lightweight gating mechanism integrates textual and visual embeddings contextually. Extensive experiments on multiple benchmarks validate the effectiveness and robustness of LAPM-RA over competitive baselines.

1. Introduction

Multimodal sentiment analysis (MSA) aims to infer sentiment polarity by jointly modeling heterogeneous signals from multiple modalities, such as textual content and visual information [1,2]. It has been widely studied in applications such as social media monitoring, opinion mining, and online product analysis, where sentiment understanding often depends on the interaction between linguistic expressions and visual contexts [3,4]. Compared with text-only sentiment analysis, MSA is more challenging because different modalities may provide complementary, ambiguous, or even conflicting affective cues. Sentiment representation can be studied from two complementary perspectives: categorical representations describe affect with discrete labels such as positive and negative, whereas dimensional representations use continuous affective dimensions such as valence and arousal to characterize sentiment intensity and affective ambiguity. This challenge becomes more pronounced in few-shot scenarios, where only a small number of labeled image-text pairs are available for training. Under such limited supervision, models often struggle to learn reliable cross-modal sentiment associations and generalize to diverse sentiment expressions.

Existing MSA studies have made substantial progress by designing cross-modal fusion mechanisms [5], cross-modal attention and alignment models [6,7], and adaptive interaction mechanisms [8]. These methods improve the integration of textual and visual information, but they typically rely on sufficient labeled data to learn stable multimodal correspondences. When annotations are scarce, limited training examples may fail to cover the diversity of sentiment expressions, lexical variations, and visual contexts. As a result, fusion-based models can easily overfit to sparse training samples and become sensitive to subtle changes in textual or visual cues. Therefore, few-shot MSA requires not only effective multimodal modeling, but also mechanisms that can expand the limited training space and improve the adaptability of model learning.

Prompt learning provides a promising direction for low-resource learning by reformulating downstream tasks as language modeling problems and exploiting knowledge encoded in pretrained language models [9,10]. Recent studies have introduced prompt-based methods into few-shot MSA to better utilize limited supervision and guide multimodal sentiment prediction. For example, Yu and Zhang proposed a prompt-based vision-aware language modeling method that incorporates visual information into prompt learning to enhance few-shot multimodal sentiment classification [11]. Xiang et al. further proposed an adaptive multimodal prompt-tuning method that dynamically optimizes multimodal prompts to improve the utilization of scarce labeled samples [12]. In addition, syntax-aware prompt modeling and hierarchical reasoning enhancement have also been explored to improve semantic modeling and sentiment discrimination under few-shot conditions [13,14]. These studies show that prompt learning can effectively activate task-relevant knowledge in pretrained models and provide an important technical pathway for few-shot MSA.

Despite these advances, existing few-shot MSA prompting methods still face two limitations. The first limitation is the insufficient expansion of the few-shot sentiment space. PVLM improves few-shot MSA through vision-aware prompt formulation [11], HSAP enhances prompt learning with syntactic information and attention mechanisms [13], and adaptive multimodal prompt tuning improves the use of scarce labeled samples through learnable prompt adaptation [12]. However, these methods usually treat the available image-text pairs as fixed training resources, making it difficult to cover diverse sentiment expressions, lexical variations, and counterfactual decision boundaries under limited supervision. The second limitation is the weak coupling between prompt adaptation and task-specific feedback. Existing methods mainly advance prompt construction, visual prompt incorporation, or prompt tuning, but they do not explicitly connect prompt selection with feedback signals that reflect discriminative capability, semantic consistency, and cross-modal reliability. Therefore, few-shot MSA still requires a framework that can both expand the limited sentiment expression space and adaptively select prompts according to multimodal sentiment-related feedback.

To address these limitations, we propose LLM-Augmented Prompt Learning for MSA with Reward Adaptation (LAPM-RA), a unified framework for few-shot multimodal sentiment analysis. LAPM-RA advances few-shot MSA by organizing sentiment-aware augmentation, reward-adaptive prompt selection, and prompt-based verbalization into a task-specific learning framework. Specifically, the sentiment-aware augmentation module uses LLMs to generate sentiment-preserving and counterfactual textual variants, thereby expanding the few-shot sentiment expression space. The reward-adaptive prompt optimization module then selects effective prompt candidates using feedback signals that jointly consider discriminative capability, semantic consistency, and cross-modal reliability. Finally, prompt-based verbalization connects the selected prompts with sentiment labels in a semantically meaningful prediction space. By linking augmentation, prompt selection, and verbalized prediction, LAPM-RA enables these components to support each other under limited multimodal supervision.

Our main contributions are summarized as follows:

• We propose LAPM-RA, a task-specific LLM-augmented prompt learning framework for few-shot MSA. Instead of treating LLM augmentation, prompt selection, and verbalizers as independent components, LAPM-RA couples them to expand the few-shot sentiment space and support adaptive prompt-based prediction.
• We develop a sentiment-aware augmentation and reward-adaptive prompt learning strategy. It enriches the few-shot training space through sentiment-preserving and counterfactual variants, and adaptively selects multimodal prompts using reward signals derived from discriminative capability, semantic consistency, and cross-modal reliability.
• We perform experiments on multiple benchmark datasets to evaluate the contribution of LLM-based augmentation, adaptive prompt selection, verbalizer construction, and multimodal fusion under few-shot MSA settings.

2. Related Work

In this paper, we mainly focus on multi-modal sentiment analysis on text-image posts, which has become a popular research topic in recent years. Therefore, in the following, we mainly overview the related works of text-image scenario from two perspectives.

2.1. Multi-Modal Sentiment Analysis

Multimodal sentiment analysis (MSA) aims to infer sentiment polarity by jointly modeling heterogeneous information from multiple modalities, such as text, images, and speech. Compared with text-only sentiment analysis, MSA provides a more comprehensive understanding of affective expression by exploiting complementary cues from textual semantics, visual scenes, facial expressions, and object attributes. Early studies mainly focused on cross-modal representation and fusion. Niu et al. constructed a large-scale weakly labeled dataset for visual sentiment analysis and investigated how affective information can be captured from visual content [15]. Xu et al. analyzed sentiment factors in social media image-text data and highlighted the complementary roles of textual and visual information in sentiment recognition [16]. Building on this line of research, Xu et al. proposed MultiSentiNet, which leverages visual information to guide the extraction of sentiment-related textual keywords and integrates them with object- and scene-level visual features [17]. Xu et al. further introduced Co-Memory, a collaborative memory network designed to capture cross-modal interactions between textual and visual representations [18]. Yang et al. proposed a multi-view attentional network that selectively attends to sentiment-relevant information from different modalities for image-text emotion classification [19]. In parallel, Zadeh et al. developed the Tensor Fusion Network to explicitly model unimodal, bimodal, and trimodal interactions through tensor outer products [5], while Tsai et al. proposed the Multimodal Transformer to capture cross-modal dependencies in unaligned multimodal sequences [6]. These studies established cross-modal interaction modeling as a central problem in MSA, but their effectiveness often relies on sufficient labeled data and carefully designed fusion architectures.

Another important line of work concerns how sentiment is represented. Most text-image MSA benchmarks adopt a categorical paradigm, in which affect is represented by discrete polarity or emotion labels such as positive, negative, happy, or sad. This formulation is suitable for supervised classification and direct benchmark comparison. In contrast, dimensional sentiment analysis represents affect using continuous dimensions such as valence, arousal, and dominance, following psychological theories of affect [20]. Dimensional resources and evaluations, such as EmoBank with VAD annotations [21], the NRC VAD Lexicon [22], Chinese EmoBank-style resources, multilingual dimensional aspect-based sentiment analysis in SemEval-2026 Task 3 [23], and multimodal affect challenges such as MuSe 2023 and MuSe 2024 [24,25], show that dimensional signals can capture affective intensity and subtle emotional variation beyond polarity labels. These techniques can enhance MSA in several ways. First, multi-dimensional relation models can provide auxiliary supervision for learning relations among polarity, valence, arousal, and visual affective cues. Second, refined or affect-enriched word embeddings can inject valence–arousal information into textual representations and verbalizer words, making sentiment cues more fine-grained [26]. Third, dimensional scores can be used as quality-control signals for LLM-based augmentation, for example by checking whether a sentiment-preserving rewrite maintains similar valence or whether a counterfactual rewrite changes polarity without destroying image–text affective compatibility.

Recent research has moved beyond static feature concatenation toward more fine-grained, adaptive, and robust multimodal representation learning. Akhtar et al. proposed a multi-task learning framework that jointly models multimodal sentiment analysis and related tasks, such as emotion recognition, to exploit task-level complementarities [27]. Song et al. introduced a target-oriented multimodal sentiment classification method that incorporates target information through topic modeling and gated mechanisms, thereby reducing the influence of task-irrelevant visual content [28]. Yang et al. proposed a cross-modal contrastive learning method to enhance the consistency between textual and visual representations [29], and Sun et al. further explored sentiment-knowledge-guided cross-modal representation connection to obtain more compact and robust multimodal representations [7]. Liu et al. developed TsAFN, a two-stage adaptive fusion network that dynamically integrates unimodal and cross-modal features [8]. Chen et al. proposed a relevance-aware visual entity filtering network to suppress visual entities that are irrelevant to aspect-level sentiment prediction [30]. Wang et al. introduced DLF to disentangle language-dominant sentiment cues from multimodal interaction information [31], while Zhou et al. proposed a dual-path dynamic fusion method with learnable queries to enhance both global and local cross-modal feature integration [32]. In addition, Zhu et al. proposed proxy-driven robust MSA to mitigate the impact of missing or corrupted modalities by learning proxy representations [33], and Jian et al. developed MDF to disentangle and fuse modality-shared and modality-specific factors, alleviating the negative effects of cross-modal heterogeneity [34]. These efforts indicate that MSA has gradually shifted from merely combining multiple modalities to learning fine-grained, adaptive, and robust sentiment representations.

Despite these advances, most existing MSA methods implicitly assume that sufficient labeled multimodal samples are available during training. In real-world scenarios, however, high-quality image-text sentiment annotation is costly because it requires simultaneous understanding of textual semantics, visual content, and their affective correspondence. This issue becomes more pronounced in emerging domains, new events, or fine-grained sentiment categories, where only a small number of labeled examples may be available. Under such limited supervision, models may fail to capture diverse sentiment expressions and stable cross-modal sentiment associations. Therefore, few-shot MSA raises a more challenging question: how can models effectively exploit pretrained knowledge, expand the limited sentiment expression space, and learn reliable image-text affective correspondences from scarce labeled data? This challenge motivates the use of prompt learning and generative augmentation, which will be discussed in the following subsection.

2.2. Prompt Learning

Prompt learning has emerged as an effective paradigm for adapting pretrained language models (PLMs) to downstream tasks by reformulating task objectives as language modeling problems. Instead of fully fine-tuning all model parameters, prompt learning seeks to activate task-relevant knowledge already encoded in PLMs through carefully designed or learnable prompts. Petroni et al. showed that factual knowledge stored in language models can be elicited through cloze-style prompts [35]. Brown et al. further demonstrated that large language models can perform few-shot task adaptation by conditioning on natural-language instructions and a small number of in-context examples [9]. However, early discrete prompts usually depend on manual template engineering, and their performance can be highly sensitive to prompt wording. To reduce this dependence on hand-crafted templates, later studies explored continuous prompt learning. Liu et al. proposed P-tuning, which optimizes trainable prompt embeddings in the continuous space to guide pretrained models toward downstream tasks [36]. Li and Liang introduced Prefix-Tuning, which prepends trainable continuous vectors to Transformer layers while keeping the backbone model frozen [37]. Ding et al. developed OpenPrompt as a unified framework for implementing and comparing prompt-learning methods [10]. These studies shifted prompt design from manually written templates to learnable task-specific representations, making prompt learning particularly suitable for low-resource scenarios.

In few-shot multimodal sentiment analysis, prompt learning has been adopted to better exploit limited supervision and strengthen the semantic connection between textual and visual information. Yu and Zhang proposed prompt-based vision-aware language modeling, which incorporates visual information into prompt learning for few-shot multimodal sentiment classification [11]. Zhou et al. introduced a syntax-aware hybrid prompt model that combines syntactic information with prompt learning to enhance semantic representation under few-shot settings [13]. Xiang et al. proposed an adaptive multimodal prompt-tuning model, which improves the utilization of scarce labeled samples through adaptive prompt optimization [12]. You et al. further developed Hierarchical Reasoning Enhanced Few-Shot Multimodal Sentiment Analysis, which decomposes sentiment prediction into hierarchical reasoning processes to improve discriminative capability under limited supervision [14]. These studies demonstrate that prompt learning can effectively guide pretrained models to capture sentiment-relevant textual and visual cues from only a few labeled image-text pairs.

Nevertheless, existing prompt-learning methods for few-shot MSA mainly focus on prompt construction, prompt tuning, or reasoning decomposition, while treating the training samples themselves as fixed resources. PVLM and HSAP show that prompt-based modeling can effectively exploit pretrained language knowledge for few-shot MSA, and adaptive multimodal prompt tuning further improves prompt flexibility. However, these methods do not explicitly use LLMs to enlarge the sentiment expression space with sentiment-preserving and counterfactual variants, nor do they couple prompt selection with feedback signals related to generated-sample semantics and cross-modal reliability. Compared with these approaches, LAPM-RA is designed as an augmentation-prompt coupling framework: LLM-based sentiment-aware augmentation expands the limited training space, reward-adaptive prompt selection evaluates candidate prompts with task-specific feedback, and verbalizer-based prediction maps the selected prompt representation to sentiment labels. This coordinated design provides a clearer pathway for improving few-shot MSA, because data expansion, prompt selection, and sentiment verbalization are optimized toward the same multimodal sentiment prediction objective.

3. Methodology

3.1. Motivation

Few-shot multimodal sentiment analysis (MSA) remains challenging due to two limitations of existing approaches. First, the scarcity of annotated data restricts models to narrow contexts, limiting their ability to generalize across diverse linguistic expressions and cross-modal semantics. Without diversity, models become fragile when faced with contextual variations or domain shifts, highlighting the need for data enrichment strategies that can generate both sentiment-consistent examples and counterfactual variations while preserving label semantics. Second, most prompt-based methods rely on manually designed or fixed templates, which lack the flexibility to capture subtle, context-dependent sentiment cues and adapt to heterogeneous multimodal inputs. This static design constrains robustness and transferability, underscoring the necessity of adaptive mechanisms that can dynamically optimize prompts under more informative supervisory signals. These challenges together motivate the development of a unified framework that integrates sentiment-aware data augmentation with reward-adaptive prompt learning, enabling robust and generalizable performance in few-shot MSA. For clarity, XLM-R denotes XLM-RoBERTa-large throughout this paper. We use the public Hugging Face checkpoint as the text backbone (https://huggingface.co/FacebookAI/xlm-roberta-large (accessed on 2 July 2026)).

3.2. Formalization and Overall Architecture

To address the challenges identified above, we propose LAPM-RA (LLM-Augmented Prompt Learning for MSA with Reward-Adaptive Optimization), a unified framework designed for few-shot MSA. The task is defined as estimating the conditional probability distribution $P(y\mid x^t,x^v)$, where $x^t$ denotes the textual input, $x^v$ the visual input, and $y\in \mathcal{Y}$ the sentiment label. Unlike conventional classifiers that rely on static prediction heads, our framework leverages verbalized prompts embedded into a pre-trained language model (PLM), aligning label prediction with semantically meaningful representations.

As illustrated in Figure 1, LAPM-RA consists of four core components whose trainable modules are optimized jointly in an end-to-end manner: (1) LLM-based sentiment-aware augmentation, which generates sentiment-consistent and counterfactual rewrites to enrich data diversity; (2) Reward-guided prompt selection, which adaptively selects prompts based on validation-driven reward signals; (3) Context-aware multimodal fusion, which encodes textual and visual inputs and dynamically integrates them into a joint representation; (4) Prompt-based classification and joint training, which verbalizes sentiment labels within the PLM and incorporates multi-objective optimization to improve accuracy, consistency, and robustness.

By unifying data augmentation with adaptive prompt optimization under a multimodal fusion framework, LAPM-RA enables effective, label-efficient, and context-sensitive sentiment reasoning generalizable to low-resource multimodal scenarios.

3.3. LLM-Based Sentiment-Aware Augmentation

As the first component of LAPM-RA, we propose a sentiment-aware augmentation strategy powered by LLMs. This module is designed to enhance generalization and reduce overfitting to superficial lexical patterns. Unlike traditional augmentation techniques, such as synonym substitution or back-translation, which often introduce shallow variations without semantic control, LLMs enable the structured generation of diverse yet sentiment-consistent rewrites. We employ two complementary augmentation strategies:

Sentiment-Preserving Rewrites. These variants introduce lexical and syntactic diversity while keeping the sentiment unchanged. For each training instance $(x_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,y^i)$, we instantiate a natural language prompt conditioned on both the input text and its sentiment label. A representative template is:

The following is a [positive/negative] review: “{xⁱ_text}". Please rewrite it using different wording while preserving its sentiment.

Conditioned on this prompt, the LLM generates a sentiment-consistent variant ${\widehat{x}}_{\mathrm{text}}^i$, which is paired with the original image and label to form $({\widehat{x}}_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,y^i)$.

Counterfactual Rewrites. In addition to sentiment-preserving paraphrases, we generate counterfactual samples that explicitly flip the sentiment label while keeping the overall semantics of the text largely intact. The goal is to expose the model to minimal perturbations that alter sentiment polarity, thereby encouraging it to focus on causal sentiment cues rather than spurious correlations. A representative prompt is:

Rewrite the following review into an opposite sentiment while retaining its core meaning: “{xⁱ_text}".

Conditioned on this prompt, the LLM produces a counterfactual variant ${\tilde{x}}_{\mathrm{text}}^i={\mathcal{G}}_{\mathrm{cf}}(x_{\mathrm{text}}^i,y^i)$ with a new sentiment label ${\tilde{y}}^i\ne y^i$. Each counterfactual instance is thus constructed as $({\tilde{x}}_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,{\tilde{y}}^i)$.

This process enables the model to learn fine-grained decision boundaries by contrasting original and counterfactual pairs, improving robustness against spurious lexical patterns and enhancing causal interpretability of sentiment prediction.

In this work, counterfactual augmentation is text-centered: the textual modality is intervened by rewriting sentiment-bearing expressions, while the visual modality is kept fixed as contextual evidence. Keeping the original image unchanged is a deliberate single-modality intervention rather than an attempt to synthesize a completely new image-text post. If both text and image were modified simultaneously, the label change would be confounded by textual variation, visual-content variation, and possible artifacts from image generation. By holding the visual context constant, the factual pair $(x_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,y^i)$ and its counterfactual counterpart $({\tilde{x}}_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,{\tilde{y}}^i)$ form a controlled comparison that encourages the model to identify textual cues that causally alter sentiment polarity. The visual modality still participates in prediction through the multimodal fusion module, but it is not regenerated during counterfactual augmentation. To examine whether this text-centered counterfactual construction remains empirically valid, we provide a counterfactual validity and effectiveness analysis in the experimental section.

The final augmented dataset is constructed as:

$$D_{\mathrm{aug}}=D_{\mathrm{train}}\cup \left\{({\widehat{x}}_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,y^i)\right\}\cup \left\{({\tilde{x}}_{\mathrm{text}}^i,x_{\mathrm{vis}}^i,{\tilde{y}}^i)\right\}$$

(1)

where ${\widehat{x}}_{\mathrm{text}}^i$ denotes sentiment-preserving rewrites with the same label $y^i$, and ${\tilde{x}}_{\mathrm{text}}^i$ represents counterfactual rewrites paired with a flipped sentiment label ${\tilde{y}}^i\ne y^i$. This enriched corpus increases both intra-class diversity (via sentiment-preserving rewrites) and inter-class contrast (via counterfactuals rewrites), encouraging the model to focus on semantically grounded sentiment cues while improving its ability to distinguish fine-grained decision boundaries.

3.4. Reward-Guided Prompt Selection

We adopt a prompt-based classification framework, where each input is paired with a prompt template to condition the prediction process. While fixed prompts may be suboptimal across diverse inputs, we propose a supervised reward-guided selection mechanism that learns to assign suitable prompts to individual instances in a context-aware manner.

Let $\mathcal{P}=\{P_1,P_2,...,P_K\}$ denote the set of candidate prompt templates. The prompt pool $\mathcal{P}$ is constructed before training and remains fixed during each run. The prompt policy network learns to select among these predefined templates rather than generating new templates online. For each input x, the XLM-R text encoder generates a semantic representation:

$$s=\mathrm{XLM}-\mathrm{R}\left(x\right)$$

(2)

This representation is fed into a single-layer MLP to produce a softmax distribution over prompts in the set $\mathcal{P}$:

$${\pi }_{\theta }(P_j\mid x)={\displaystyle \frac{exp\left(w_j^{\top }s\right)}{{\sum }_{k=1}^{K}exp\left(w_k^{\top }s\right)}}$$

(3)

where $P_j\in \mathcal{P}$ is a specific prompt template chosen by the policy, and $w_j$ is a learnable parameter vector corresponding to prompt $P_j$. A prompt $P_j$ is then sampled from this distribution and applied to the downstream model for classification.

To supervise the selection process, we define a reward function $S_{\sup }(P_j,\mathcal{Z})$ based on a validation split $\mathcal{Z}=\{z_1,...,z_N\}$ of the training data. For each validation sample $z_i$, we compute the classifier’s margin between the correct label $c_i$ and the highest scoring incorrect label $c_{\mathrm{else}}$, conditioned on prompt $P_j$:

$$S_{\sup }(P_j,\mathcal{Z})=\sum _{z_i\in \mathcal{Z}}\left[\mathrm{PLM}(c_i\mid z_i,P_j)-\mathrm{PLM}(c_{\mathrm{else}}\mid z_i,P_j)\right]$$

(4)

This reward quantifies how effectively prompt $P_j$ enhances the model’s discriminative confidence across the validation set.

For each candidate prompt, the reward is computed by aggregating validation-margin scores over the validation split. Based on this prompt-specific reward, the prompt policy network is trained to favor prompts that obtain higher validation performance. The selection objective is formulated as the expected supervised reward over the training samples:

$$J\left(\pi \right)={\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{x\in \mathcal{D}}{\mathbb{E}}_{P_j\sim {\pi }_{\theta }(·\mid x)}\left[S_{\sup }(P_j,\mathcal{Z})\right].$$

(5)

For stable optimization, we further normalize these prompt-specific rewards across the candidate prompt pool to obtain a soft prompt-preference target:

$${\tilde{S}}_{\sup }\left(P_j\right)={\displaystyle \frac{exp(S_{\sup }(P_j,\mathcal{Z})/\tau )}{{\sum }_{k=1}^{K}exp(S_{\sup }(P_k,\mathcal{Z})/\tau )}}$$

(6)

where $\tau$ is a temperature parameter for reward normalization. In implementation, these prompt-specific rewards are estimated at the epoch level and kept fixed during the mini-batch updates of that epoch. The policy network is then optimized by matching ${\pi }_{\theta }(P_j\mid x)$ to the normalized reward target through the prompt supervision loss in the joint training objective.

3.5. Context-Aware Multimodal Sentiment Fusion

To effectively leverage sentiment cues from both textual and visual inputs, we propose a semantic fusion module that dynamically integrates multimodal representations through a lightweight gating mechanism. This module enables context-adaptive fusion and captures complementary affective information from different sources.

Modality Encoding. Given a textual input $x^t$ and a visual input $x^v$, we first encode each modality independently using pretrained backbones:

$$f^t=\mathrm{XLM}-\mathrm{R}\left(x^t\right),h^v=\mathrm{ResNet}-50\left(x^v\right),f^v=W_{\mathrm{img}}{h}^v+b_{\mathrm{img}}$$

(7)

The text encoder is the same XLM-R backbone used for prompt-based classification. For the visual modality, we use an ImageNet-pretrained ResNet-50 and remove its final classification layer. The standard global-average-pooled image feature $h^v\in {\mathbb{R}}^{2048}$ is projected by $W_{\mathrm{img}}$ and $b_{\mathrm{img}}$ to obtain $f^v$. Thus, $f^t\in {\mathbb{R}}^{d_h}$ and $f^v\in {\mathbb{R}}^{d_h}$, where $d_h=1024$ is the hidden size of XLM-R. The two modalities are therefore aligned to the same hidden dimension before multimodal fusion. The ResNet-50 feature extractor is kept frozen during training, while the projection layer and the subsequent fusion module are trainable.

Gating-Based Fusion. To determine the relative importance of each modality, we compute a scalar gate $\alpha \in (0,1)$ based on the concatenated features:

$$\alpha =\sigma (w^{\top }[f^t;f^v]+b_{\mathrm{gate}})$$

(8)

Here, $[f^t;f^v]\in {\mathbb{R}}^{2d_h}$ denotes the concatenation of the two features, $w\in {\mathbb{R}}^{2d_h}$ and $b_{\mathrm{gate}}\in \mathbb{R}$ are learnable gating parameters, and $\sigma (·)$ is the sigmoid function. The scalar $\alpha$ acts as a soft attention weight, controlling the contribution of each modality based on input context.

The gated fusion representation is computed as:

$$F^{\mathrm{fusion}}=tanh\left(W_t\left(\alpha f^t\right)+W_v\left((1-\alpha )f^v\right)+b_{\mathrm{proj}}\right)$$

(9)

where $W_t,W_v\in {\mathbb{R}}^{d^{\prime }\times d_h}$ are projection matrices for text and vision modalities, and $b_{\mathrm{proj}}\in {\mathbb{R}}^{d^{\prime }}$ is the fusion bias term. The $tanh(·)$ non-linearity encourages bounded, smooth semantic representations. The output $F^{\mathrm{fusion}}\in {\mathbb{R}}^{d^{\prime }}$ serves as the integrated multimodal sentiment embedding.

This fused vector is then passed to the downstream classifier for final prediction.

3.6. Prompt-Based Classification and Joint Training

We adopt a prompt-based classification framework in which sentiment prediction is performed through a pretrained language model (PLM), conditioned on a constructed verbalizer and multimodal fusion representation. All components are jointly optimized under a multi-objective training strategy to enhance predictive accuracy, robustness, and prompt adaptability.

Verbalizer Construction. For each sentiment label $y\in \mathcal{Y}$, we construct a verbalizer ${\mathcal{V}}_y=\{v_1,\dots ,v_{n_y}\}$, where each $v_i$ is a semantically related token generated by GPT-4o-mini. Given a label seed $q_y$, GPT-4o-mini first generates a candidate set ${\mathcal{C}}_y=\{c_1,\dots ,c_M\}$ of label-related words. We rank the candidates according to their semantic relevance to $q_y$ and retain the top-$K_v$ words after filtering. In our implementation, $M=10$ candidates are generated for each label seed and $K_v=5$ words are retained for the final verbalizer. Candidates with duplicated meanings, ambiguous polarity, or overlap between positive and negative verbalizers are removed. Because prediction is performed with a single masked token, candidates that are split into multiple subword tokens by the XLM-R tokenizer are also excluded. The final verbalizers use the same retained size $K_v$ for all datasets.

Prompt-Based Classification. Given a textual input $x^t$ and visual input $x^v$, we obtain modality-specific features $f^t$ and $f^v$, and compute a fused representation $F^{\mathrm{fusion}}$ using the gating mechanism described above. The fused continuous vector is not converted into natural-language text. Instead, it is projected into a short sequence of continuous context embeddings and inserted into the prompt input of the language model. This representation is then embedded into a prompt template such as:

“[CONTEXT] The review is [MASK]."

Here, $\left[CONTEXT\right]$ denotes a sequence of soft prompt embeddings rather than readable text. Specifically, a trainable projection ${\varphi }_{\mathrm{ctx}}$ maps the fused vector to m continuous context embeddings:

$$E_{\mathrm{ctx}}={\varphi }_{\mathrm{ctx}}\left(F^{\mathrm{fusion}}\right),E_{\mathrm{ctx}}\in {\mathbb{R}}^{m\times d_h}.$$

(10)

Here, m is the number of context embeddings and $d_h=1024$ is the hidden size of XLM-R. These context embeddings are concatenated with the token embeddings of the selected prompt template and then fed into XLM-R for masked-token prediction. The probability of predicting a token v at the masked position is denoted as $p_{\mathrm{PLM}}(v\mid \mathrm{prompt}\left(F^{\mathrm{fusion}}\right))$. The probability of a label y is then computed by summing over the token probabilities in its verbalizer:

$$p(y\mid x^t,x^v)=\sum _{v\in {\mathcal{V}}_y}{p}_{\mathrm{PLM}}(v\mid \mathrm{prompt}\left(F^{\mathrm{fusion}}\right))$$

(11)

Multi-Objective Joint Training. Our training objective combines three components:

First, the classification loss encourages the PLM to assign higher probability to tokens within the correct verbalizer set:

$${\mathcal{L}}_{\mathrm{cls}}=-{\displaystyle \sum _{i=1}^N}log\sum _{v\in {\mathcal{V}}_{y_i}}{p}_{\mathrm{PLM}}(v\mid \mathrm{prompt}\left(F_i^{\mathrm{fusion}}\right))$$

(12)

Here, N is the number of training instances, $y_i$ is the true sentiment label for the i-th example, ${\mathcal{V}}_{y_i}$ is its verbalizer, and $F_i^{\mathrm{fusion}}$ is the corresponding fused representation.

Second, we introduce a counterfactual consistency loss to encourage robustness against lexical variations. Let $F_i^{\mathrm{ori}}$ and $F_i^{\mathrm{cf}}$ denote the fused representations of the original and counterfactually augmented inputs, respectively. Their distance is minimized via:

$${\mathcal{L}}_{\mathrm{cf}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{∥F_i^{\mathrm{ori}}-F_i^{\mathrm{cf}}∥}_2^2$$

(13)

This term is used as a weak auxiliary regularizer rather than as a replacement for sentiment classification supervision. The original and counterfactual samples are still optimized with their corresponding sentiment labels through ${\mathcal{L}}_{\mathrm{cls}}$, while ${\mathcal{L}}_{\mathrm{cf}}$ only encourages the shared non-sentiment context and fixed visual background to remain stable under controlled textual sentiment intervention.

Third, the prompt supervision loss ensures that the learned prompt selection policy ${\pi }_{\theta }$ aligns with external reward signals. Given K prompt candidates $\{P_1,\dots ,P_K\}$, for each input $x^t$ we minimize the deviation between the policy’s predicted probability ${\pi }_{\theta }(P_j\mid x^t)$ and the normalized supervised reward ${\tilde{S}}_{\sup }\left(P_j\right)$ for prompt $P_j$:

$${\mathcal{L}}_{\sup }={\displaystyle \frac{1}{K}}{\displaystyle \sum _{j=1}^K}{\left({\pi }_{\theta }(P_j\mid x^t)-{\tilde{S}}_{\sup }\left(P_j\right)\right)}^2$$

(14)

Final Objective. The overall learning objective unifies the main classification task with two auxiliary objectives:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{cls}}+{\lambda }_1{\mathcal{L}}_{\mathrm{cf}}+{\lambda }_2{\mathcal{L}}_{\sup }$$

(15)

Here, ${\lambda }_1$ and ${\lambda }_2$ are hyperparameters controlling the weight of auxiliary tasks. All components, including the PLM, fusion encoder, and prompt selector, are jointly optimized in an end-to-end fashion to achieve holistic representation learning. The algorithmic details provided in Algorithm 1.

Algorithm 1 LAPM-RA Framework

Input: Training set $\mathcal{D}$; prompt set $\mathcal{P}$; validation set $\mathcal{Z}$

Output: Model parameters $\mathrm{\Theta }$ and prompt policy ${\pi }_{\theta }$

1: Build augmented dataset ${\mathcal{D}}_{aug}$ via sentiment-preserving and counterfactual rewrites

2: Initialize PLM, multimodal fusion module, and prompt policy ${\pi }_{\theta }$

3: for $t=1$ to T do

Sample minibatch $\mathcal{B}\sim {\mathcal{D}}_{aug}$

for each $(x^t,x^v,y)\in \mathcal{B}$ do

Sample prompt $P_j\sim {\pi }_{\theta }(·|x^t)$

Encode multimodal features $f^t\leftarrow \mathrm{XLM}-\mathrm{R}\left(x^t\right),f^v\leftarrow \mathrm{Proj}(\mathrm{ResNet}-50\left(x^v\right))$

Compute fused representation $F^{\mathrm{fusion}}$ via gating mechanism

Estimate $p(y\mid x^t,x^v)$ using prompt-based verbalization

end for

Compute ${\mathcal{L}}_{cls}$, ${\mathcal{L}}_{cf}$, and ${\mathcal{L}}_{sup}$

Update $\mathrm{\Theta }$ and ${\pi }_{\theta }$ by minimizing ${\mathcal{L}}_{total}$

end for

4: Refine prompt policy ${\pi }_{\theta }$ using validation reward $S_{\sup }$

return   $\mathrm{\Theta },{\pi }_{\theta }$

4. Experiments

4.1. Datasets

The experiments are conducted on two widely-used multimodal sentiment classification benchmarks. Detailed dataset statistics are summarized in

Table 1. Statistics of the benchmark datasets after preprocessing.

Dataset	Classes	Positive	Negative	Train	Val	Test
MVSA-Single	2	2683	1358	2425	404	1212
MVSA-Multiple	2	11,900	1500	8040	1340	4020
TumEmo-Binary	2	84,778	60,095	115,898	14,487	14,488

. Table 1 reports the final dataset statistics after preprocessing. For MVSA, neutral samples, annotation ties, and unavailable or corrupted image-text pairs are removed. For TumEmo, we construct TumEmo-Binary for polarity-oriented sentiment classification. Specifically, happy and love are mapped to positive sentiment, while angry, fear, and sad are mapped to negative sentiment. Labels without unambiguous positive/negative polarity are not included in the binary sentiment setting.

MVSA (https://mcrlab.net/research/mvsa-sentiment-analysis-on-multi-view-social-data/ (accessed on 2 July 2026)) [15]: This dataset serves as a standard benchmark for MSA, comprising social media posts paired with images and sentiment labels annotated by multiple human raters. It includes two subsets: MVSA-Single, in which each post is associated with a single image, and MVSA-Multiple, where posts contain multiple images. Final sentiment labels are determined via a majority voting strategy, with posts exhibiting annotation ties excluded. To align with a binary sentiment setting, neutral instances are removed, retaining only positive and negative examples. MVSA is widely employed to evaluate models’ capability to capture complementary information across modalities and to handle fine-grained sentiment cues.

TumEmo (https://github.com/YangXiaocui1215/MVAN (accessed on 2 July 2026)) [15]: This dataset serves as a standard benchmark for MSA, comprising social media posts paired) [19]: TumEmo is a large-scale multimodal dataset collected from Tumblr, annotated with seven emotion labels. For polarity-oriented sentiment classification, happy and love are mapped to positive sentiment, while angry, fear, and sad are mapped to negative sentiment. Labels without unambiguous positive/negative polarity are not included in the binary sentiment setting. For TumEmo, verbalizer candidates are expanded from the emotion labels included in each binary group and then merged into the corresponding positive or negative verbalizer. Thus, separate verbalizers are not constructed for the original seven emotion labels in the binary setting. The dataset offers high diversity and noise, posing greater challenges for multimodal understanding.

4.2. Baseline Methods

To comprehensively evaluate the effectiveness of the proposed LAPM-RA method, we compare it with a diverse set of baseline models spanning both unimodal and multimodal paradigms:

• BiLSTM [38]: A bidirectional long short-term memory network that captures contextual dependencies in textual sequences, commonly used for unimodal sentiment classification.
• BiACNN [39]: Combines convolutional neural networks (CNNs) with BiLSTM and an attention mechanism to enhance the identification of emotionally salient regions in text.
• OSDA [40]: A dual-attention model that simultaneously captures object-level and scene-level visual features to better reflect user sentiment.
• HSAN [16]: Performs multimodal sentiment analysis by jointly modeling semantic image captions and hierarchical text structure through layered attention.
• MultiSentiNet [17]: A vision-guided multimodal sentiment model that employs attention to extract emotional keywords from text and fuses them with object and scene-level visual features.
• Co-Memory [18]: A collaborative memory network that performs iterative reasoning over textual and visual modalities to improve cross-modal sentiment alignment.
• MVAN [19]: A state-of-the-art attention-based fusion model that jointly encodes visual and textual features for robust sentiment understanding.
• PVLM [11]: A prompt-based multimodal framework using fixed and learnable templates, applied to few-shot sentiment classification and aspect-based sentiment analysis.
• HSAP [13]: A syntax-enhanced prompting method that combines fixed and learnable prompts with biaffine attention and scaled dot-product attention to incorporate syntactic and semantic signals for improved few-shot multimodal sentiment classification.
• MER-CLIP [41]: A recent CLIP-based multimodal emotion and sentiment recognition model that uses CLIP representations and label-guided cross-modal decoding. Since the original method is designed for related multimodal emotion-recognition settings, we adapt it to the image-text sentiment classification task under the same few-shot evaluation protocol.
• LLaVAC [42]: A recent LLaVA-style multimodal sentiment classifier obtained by fine-tuning a large vision-language model for sentiment classification. We include it as a representative LVLM-based baseline and adapt it to the evaluated image-text datasets using the same training, validation, and test partitions.

4.3. Experiment Settings

In the few-shot setting, K denotes the number of labeled examples sampled from the training partition for each sentiment class. We set $K=50$, i.e., 50 examples per class are sampled as the few-shot training data, and another 50 examples per class are sampled from the remaining training data to construct a class-balanced validation set. For prompt-based and few-shot methods, the same few-shot training split, validation split, and fixed test split are used under each random seed. Conventional neural network-based baselines that are not designed for few-shot learning are trained on the full training partition following their standard supervised setting.

The validation split is used to compute aggregate prompt-level reward scores and for hyperparameter selection. Validation samples are not mixed into the few-shot training mini-batches as additional supervised instances. The test set is kept fixed across all runs, used only for final evaluation, and never used for prompt reward computation, model selection, or parameter optimization. We repeat the few-shot sampling and evaluation with five random seeds, $\{13,42,123,153,166\}$.

For hyperparameter fairness, all prompt-based methods use the same main training hyperparameters, including batch size, learning rate, optimizer, and training epochs. Other neural-network and vision-language baselines follow the recommended hyperparameter settings of their original models when available. For MER-CLIP and LLaVAC, we adapt the input interface to image-text pairs and replace the original task-specific prediction layer with a binary sentiment classifier for the evaluated datasets. Regarding implementation sources, the reported baseline results are obtained under our unified preprocessing, data split, and metric protocol. We follow the authors’ public repositories for LLaVAC settings and TumEmo/MVAN data, while BiLSTM, BiACNN, OSDA, HSAN, MultiSentiNet, Co-Memory, MVAN, PVLM, HSAP, MER-CLIP, and the LLaVAC classifier are implemented or adapted according to the corresponding papers.

For the main API-based generator, we interact with the model via its API in a prompt-driven conversational format to generate sentiment-consistent augmented samples and construct a prompt candidate pool. The augmented texts are then used for both training prompt-based classifiers and generating verbalizer tokens. To improve reproducibility, we further specify the LLM-based augmentation settings. The main API-based generator is GPT-4o-mini (gpt-4o-mini), which is used for sentiment-preserving rewrites, counterfactual rewrites, prompt candidates, and verbalizer candidates. The comparison generators are GPT-3.5 (gpt-3.5-turbo-0125), LLaMA3 (Meta-Llama-3-8B-Instruct), and GPT-2 (openai-community/gpt2), and they are evaluated with the same augmentation instructions when applicable. For each original training instance, we generate one sentiment-preserving rewrite and one counterfactual rewrite. For text augmentation, we set the sampling temperature to 0.7, the nucleus sampling probability p to 0.9, and the maximum output length to 128 tokens. For verbalizer generation, we set the sampling temperature to 0.3 and the maximum output length to 64 tokens. Empty outputs and exact duplicates are discarded before training.

The prompt settings are fixed across datasets. The sentiment-preserving prompt is: The following is a [positive/negative] review: “{text}”. Please rewrite it using different wording while preserving its sentiment. The counterfactual prompt is: Rewrite the following review into an opposite sentiment while retaining its core meaning: “{text}”. For prompt-pool construction, GPT-4o-mini is asked to generate concise sentiment-classification templates for image-text posts. We retain templates with a sentiment prediction slot and remove candidates with duplicated meanings, ambiguous label wording, or incomplete mask positions. The default prompt pool contains 10 templates and is shared across datasets. In the prompt-pool-size analysis, the 5-template setting uses a concise subset, while the 15-template setting adds valid templates with more diverse wording. For verbalizer generation, we use the label-specific instruction, e.g., List 10 words that are semantically similar to ‘positive’, such as ‘amazing’, ‘excellent’, etc., and replace the seed label for the negative class. For prompt-tuning and masked-token sentiment prediction, we use XLM-R as the pretrained language model. The same text backbone is used to obtain textual features for prompt selection and multimodal fusion. The training is conducted with a batch size of 32, a learning rate of $3\times {10}^{-5}$, a dropout rate of 0.5, and runs for 15 epochs. We use the Adam optimizer with a weight decay of $1\times {10}^{-5}$ to prevent overfitting. For visual feature extraction, we use ImageNet-pretrained ResNet-50. The final classification layer is removed, and the standard 2048-dimensional global-average-pooled visual feature is linearly projected to the hidden dimension of XLM-R before gated fusion. The ResNet-50 backbone is frozen, while the visual projection layer, fusion module, prompt policy network, and prompt-based classifier are trainable.

All experiments are conducted on a server equipped with an NVIDIA RTX 4090 GPU, an Intel(R) Core(TM) i9-12900K CPU (3.20 GHz), and 128 GB of memory. The software environment includes Python 3.9 and PyTorch 1.12 to ensure efficient model training and evaluation. We report Accuracy and Macro-F1 as the evaluation metrics. Accuracy measures the proportion of correctly classified samples, while Macro-F1 computes the unweighted average of class-wise F1 scores and is less sensitive to class-frequency imbalance after preprocessing. The implementation code and configuration files are available at https://github.com/ZZMix/LAPM-RA (accessed on 2 July 2026).

4.4. Main Results

Table 2. Accuracy and Macro-F1 results on three datasets under the revised five-seed protocol. Results are reported as mean ± standard deviation. The best result for each metric is shown in bold. Significance is tested against the strongest non-LAPM-RA baseline using a paired t-test with $p<0.05$.

Methods	MVSA-Single		MVSA-Multiple		TumEmo
	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
BiLSTM	0.6897 ± 0.0087	0.5688 ± 0.0108	0.6707 ± 0.0094	0.5777 ± 0.0112	0.6315 ± 0.0099	0.4839 ± 0.0121
BiACNN	0.7213 ± 0.0069	0.6558 ± 0.0091	0.6856 ± 0.0082	0.6836 ± 0.0076	0.6286 ± 0.0094	0.5165 ± 0.0107
OSDA	0.7213 ± 0.0078	0.6925 ± 0.0084	0.6981 ± 0.0071	0.6397 ± 0.0103	0.6382 ± 0.0090	0.5090 ± 0.0115
HSAN	0.6544 ± 0.0102	0.6409 ± 0.0089	0.6436 ± 0.0114	0.6250 ± 0.0096	0.5779 ± 0.0123	0.5604 ± 0.0101
MultiSentiNet	0.6685 ± 0.0098	0.6756 ± 0.0081	0.6757 ± 0.0086	0.6708 ± 0.0097	0.4962 ± 0.0132	0.4497 ± 0.0118
Co-Memory	0.7117 ± 0.0072	0.6498 ± 0.0104	0.6931 ± 0.0088	0.5737 ± 0.0120	0.6497 ± 0.0079	0.5395 ± 0.0109
MVAN	0.6914 ± 0.0080	0.6793 ± 0.0092	0.6916 ± 0.0076	0.6696 ± 0.0083	0.6489 ± 0.0095	0.5978 ± 0.0106
PVLM	0.7301 ± 0.0058	0.7052 ± 0.0074	0.7096 ± 0.0069	0.6957 ± 0.0066	0.6636 ± 0.0081	0.5938 ± 0.0093
HSAP	0.7364 ± 0.0053	0.7286 ± 0.0065	0.7117 ± 0.0078	0.7270 ± 0.0059	0.6602 ± 0.0087	0.6264 ± 0.0072
MER-CLIP	0.7192 ± 0.0084	0.6923 ± 0.0106	0.7064 ± 0.0091	0.6841 ± 0.0098	0.6513 ± 0.0114	0.5894 ± 0.0126
LLaVAC	0.7446 ± 0.0073	0.7161 ± 0.0095	0.7214 ± 0.0108	0.7043 ± 0.0089	0.6745 ± 0.0107	0.6121 ± 0.0130
LAPM-RA	0.7631 ± 0.0033	0.7417 ± 0.0041	0.7584 ± 0.0038	0.7295 ± 0.0045	0.6915 ± 0.0050	0.6402 ± 0.0054

reports Accuracy and Macro-F1 under the revised five-seed evaluation protocol. For each run, few-shot and prompt-based methods use the same sampled training split, validation split, and fixed test split. The reported values are the mean and standard deviation across five random seeds. This setting evaluates both average performance and stability under few-shot sampling variation.

Traditional approaches perform poorly under few-shot MSA. Single-modal baselines such as BiLSTM and BiACNN fail to exploit visual information, which is essential for capturing nuanced sentiment cues. Multimodal models such as OSDA, HSAN, MultiSentiNet, Co-Memory, and MVAN introduce visual features and cross-modal reasoning, but their architectures still rely heavily on sufficient supervised data. As a result, they show limited robustness and relatively large variation under the few-shot protocol.

Prompt-based methods represent an important step forward by leveraging pretrained language models. PVLM employs fixed and learnable prompts, while HSAP augments templates with syntactic cues and attention mechanisms. These approaches outperform most traditional baselines, confirming the value of pretrained knowledge in few-shot MSA. MER-CLIP and LLaVAC further provide comparisons with recent CLIP-based and LLaVA-style models. However, because they are adapted from related but not identical multimodal emotion or sentiment settings, their performance is less consistent across the evaluated image-text datasets.

Our proposed LAPM-RA achieves the best average performance across the reported metrics and shows lower standard deviations than most baselines. Its advantage comes from the combination of sentiment-aware LLM augmentation, reward-guided prompt selection, and context-aware multimodal fusion. These components help the model expand the limited few-shot training space, adapt prompts to input-specific sentiment cues, and integrate textual and visual evidence more effectively. Therefore, LAPM-RA shows competitive and relatively stable performance under the evaluated few-shot multimodal sentiment setting.

4.5. Ablation Study

To evaluate the contribution of each core component in the proposed LAPM-RA framework, we conduct ablation experiments by removing: (i) the dynamic prompt generation module (-DynamicPrompt), (ii) the reward-guided prompt selection policy (-ReinforcedPrompt), and (iii) the verbalizer-based label expansion (-Verbalizer). We further add targeted variants without counterfactual rewrites (-CF Rewrites), gated multimodal fusion (-Gated Fusion), and the counterfactual consistency loss (-CFLoss). The contribution of LLM-based augmentation is examined in the following counterfactual effectiveness analysis. The results are summarized in

Table 3. Ablation results of LAPM-RA under the revised five-seed protocol. Results are reported as mean ± standard deviation for Accuracy and Macro-F1.

Module	MVSA-Single		MVSA-Multiple		TumEmo
	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
-DynamicPrompt	0.7195 ± 0.0116	0.7102 ± 0.0128	0.6913 ± 0.0134	0.6896 ± 0.0121	0.6071 ± 0.0152	0.5930 ± 0.0160
-ReinforcedPrompt	0.7441 ± 0.0079	0.7215 ± 0.0095	0.7322 ± 0.0091	0.7053 ± 0.0102	0.6460 ± 0.0124	0.6322 ± 0.0113
-Verbalizer	0.7535 ± 0.0063	0.7391 ± 0.0070	0.7346 ± 0.0082	0.7160 ± 0.0089	0.6646 ± 0.0101	0.6203 ± 0.0117
-CF Rewrites	0.7523 ± 0.0072	0.7335 ± 0.0081	0.7439 ± 0.0086	0.7210 ± 0.0093	0.6745 ± 0.0108	0.6339 ± 0.0101
-Gated Fusion	0.7385 ± 0.0104	0.7194 ± 0.0116	0.7276 ± 0.0112	0.7102 ± 0.0123	0.6614 ± 0.0139	0.6215 ± 0.0144
-CFLoss	0.7578 ± 0.0054	0.7369 ± 0.0062	0.7507 ± 0.0059	0.7258 ± 0.0068	0.6813 ± 0.0075	0.6376 ± 0.0073
LAPM-RA	0.7631 ± 0.0033	0.7417 ± 0.0041	0.7584 ± 0.0038	0.7295 ± 0.0045	0.6915 ± 0.0050	0.6402 ± 0.0054

.

Removing the dynamic prompt generation module or gated fusion leads to notable performance drops, showing that adaptive prompt construction and multimodal integration are both necessary for capturing input-dependent sentiment cues. Excluding the reward-guided prompt selection produces a moderate decrease, which suggests that reward-informed prompt choice provides stable gains over non-adaptive prompt use. Removing counterfactual rewrites also reduces performance, indicating that controlled sentiment interventions provide useful supervision. The verbalizer has a dataset-dependent effect, with a more visible drop on TumEmo, while CFLoss yields a smaller but consistent improvement.

Overall, the ablation results confirm that each component contributes uniquely to the framework. Dynamic prompts enhance semantic flexibility, reward-guided selection improves input-awareness, gated fusion strengthens cross-modal sentiment modeling, counterfactual rewrites introduce controlled polarity variation, and verbalizers provide richer label supervision. Removing the counterfactual consistency loss leads to a smaller but consistent performance decline, suggesting that this loss acts as a useful auxiliary regularizer for stabilizing factual-counterfactual representations rather than forcing opposite-label samples to share the same prediction. Their complementary roles collectively help LAPM-RA maintain better multimodal sentiment classification performance under the evaluated scarce-data conditions.

To further examine modality dependence, verbalizer construction, and the number of generated samples, we add a targeted ablation analysis. Default Aug. corresponds to the current LAPM-RA configuration, which generates one sentiment-preserving rewrite and one counterfactual rewrite for each original training instance. Double and triple augmentation increase both augmentation types proportionally, generating two and three rewrites of each type, respectively.

As shown in

Table 4. Additional targeted ablation results under the revised five-seed protocol. Results are reported as mean ± standard deviation for Accuracy and Macro-F1. Default Aug. uses one sentiment-preserving rewrite and one counterfactual rewrite per original instance; Double Aug. and Triple Aug. generate two and three rewrites of each type, respectively.

Variant	MVSA-Single		MVSA-Multiple		TumEmo
	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
Text-only	0.7316 ± 0.0089	0.7068 ± 0.0095	0.7219 ± 0.0094	0.6942 ± 0.0102	0.6518 ± 0.0111	0.5965 ± 0.0120
Image-only	0.6359 ± 0.0134	0.5847 ± 0.0146	0.6192 ± 0.0141	0.5726 ± 0.0152	0.5614 ± 0.0158	0.4979 ± 0.0167
Fixed verbalizer	0.7552 ± 0.0056	0.7340 ± 0.0064	0.7481 ± 0.0068	0.7214 ± 0.0075	0.6819 ± 0.0083	0.6296 ± 0.0091
Default Aug.	0.7631 ± 0.0033	0.7417 ± 0.0041	0.7584 ± 0.0038	0.7295 ± 0.0045	0.6915 ± 0.0050	0.6402 ± 0.0054
Double Aug.	0.7604 ± 0.0048	0.7392 ± 0.0053	0.7549 ± 0.0057	0.7261 ± 0.0062	0.6881 ± 0.0067	0.6364 ± 0.0070
Triple Aug.	0.7576 ± 0.0059	0.7368 ± 0.0066	0.7521 ± 0.0064	0.7238 ± 0.0071	0.6849 ± 0.0078	0.6327 ± 0.0084

, the text-only variant performs better than the image-only variant but remains below the full multimodal model, indicating that textual cues are dominant while visual information still provides useful complementary evidence. The fixed-verbalizer variant underperforms the GPT-generated verbalizer setting, showing that the generated and filtered label words provide a more suitable sentiment-label mapping. For the augmentation quantity, the default setting gives the best overall results, while Double Aug. and Triple Aug. bring no consistent gain, likely because excessive generated text may introduce redundancy or noise.

4.6. Validation–Test Prompt-Ranking Consistency Analysis

To directly examine whether reward-guided prompt selection overfits to the small validation split, we conduct a validation–test prompt-ranking consistency analysis. For each dataset and each random seed, each candidate prompt template is evaluated individually. We record its validation reward, validation Macro-F1, and independent test Macro-F1, and then compute the Spearman rank correlation between validation-derived prompt scores and test Macro-F1 across prompt candidates. A positive rank correlation indicates that prompts preferred by the validation reward also tend to perform better on the unseen test set.

As shown in

Table 5. Validation–test prompt-ranking consistency analysis. Spearman correlations and ranks are reported as mean ± standard deviation over five random seeds. “Rank” denotes the test-set Macro-F1 rank of the prompt selected by validation reward among 10 candidate prompts, where a lower value indicates a better rank.

Dataset	#Prompts	$\mathit{\rho }$ (Val Reward, Test Macro-F1)	$\mathit{\rho }$ (Val Macro-F1, Test Macro-F1)	Rank
MVSA-Single	10	0.724 ± 0.083	0.758 ± 0.071	1.8 ± 0.8
MVSA-Multiple	10	0.647 ± 0.096	0.681 ± 0.088	2.2 ± 1.0
TumEmo	10	0.536 ± 0.112	0.594 ± 0.104	2.6 ± 1.1

, validation-derived prompt scores show positive rank correlations with independent test-set Macro-F1 across all datasets. The correlations are highest on MVSA-Single, moderate on MVSA-Multiple, and lower but still positive on TumEmo, which is consistent with the greater variability of the TumEmo setting. The reward-selected prompts also rank near the top among the 10 candidate prompts on the test set. These results suggest that the reward signal captures useful prompt-preference information rather than merely fitting accidental patterns in a particular validation split.

4.7. Counterfactual Validity and Effectiveness Analysis

To further examine whether the text-centered counterfactual samples remain valid multimodal sentiment instances, we conduct a post-hoc validity analysis on generated counterfactual samples. This analysis does not introduce an additional filtering module into the training pipeline; instead, it evaluates whether the generated counterfactual texts express the intended opposite sentiment, preserve the main non-sentiment content of the original text, and remain compatible with the fixed visual context. Specifically, we randomly sample generated counterfactual instances from each dataset and ask human annotators to judge each instance using four binary criteria: sentiment correctness, content preservation, image-text compatibility, and overall validity. The reported values are pass rates, i.e., the proportion of sampled counterfactual instances satisfying each criterion. Sentiment correctness measures whether the rewritten text matches the intended flipped label; content preservation measures whether the main topic or event is retained; image-text compatibility measures whether the rewritten text can still form a reasonable multimodal sentiment instance with the original image; and overall validity summarizes whether the sample is acceptable as a counterfactual training instance.

As shown in

Table 6. Post-hoc validity analysis of generated counterfactual samples. Values denote human-annotated pass rates for each criterion.

Dataset	Sentiment Correctness	Content Preservation	Image-Text Compatibility	Overall Validity
MVSA-Single	0.912	0.874	0.846	0.835
MVSA-Multiple	0.901	0.861	0.831	0.822
TumEmo	0.887	0.842	0.804	0.793

, most generated counterfactual samples satisfy the intended sentiment reversal and preserve the main non-sentiment content. The image-text compatibility score is lower than the sentiment correctness score, which is expected because the image is intentionally kept fixed while the textual sentiment is intervened. Nevertheless, the compatibility and overall validity scores indicate that most counterfactual samples remain usable under the fixed visual context. This supports the use of original images as controlled visual backgrounds rather than regenerating images together with text.

Table 7. Effectiveness analysis of sentiment-preserving and counterfactual augmentation under the revised five-seed protocol. Results are reported as mean ± standard deviation. SP denotes sentiment-preserving, and CF denotes counterfactual. Significance is tested against LAPM-RA using paired t-tests with $p<0.05$.

Variant	MVSA-Single		MVSA-Multiple		TumEmo
	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
w/o LLM aug.	0.7286 ± 0.0091	0.7134 ± 0.0102	0.7169 ± 0.0104	0.6997 ± 0.0113	0.6468 ± 0.0127	0.6076 ± 0.0138
w/o SP rewrites	0.7492 ± 0.0075	0.7296 ± 0.0084	0.7397 ± 0.0089	0.7172 ± 0.0097	0.6724 ± 0.0111	0.6285 ± 0.0119
w/o CF rewrites	0.7523 ± 0.0072	0.7335 ± 0.0081	0.7439 ± 0.0086	0.7210 ± 0.0093	0.6745 ± 0.0108	0.6339 ± 0.0101
LAPM-RA	0.7631 ± 0.0033	0.7417 ± 0.0041	0.7584 ± 0.0038	0.7295 ± 0.0045	0.6915 ± 0.0050	0.6402 ± 0.0054

further evaluates the effectiveness of counterfactual augmentation. Here, w/o LLM aug. removes the LLM-generated sentiment-preserving and counterfactual training samples while keeping the remaining prompt-based learning pipeline unchanged. Removing all LLM-based augmentation causes the largest performance drop, confirming the importance of generated data under the few-shot setting. Removing SP rewrites and removing CF rewrites both reduce performance compared with the full model, indicating that the two types of augmentation provide complementary benefits. In particular, the performance decrease after removing CF rewrites shows that controlled textual sentiment interventions under a fixed visual context contribute useful supervision beyond sentiment-preserving paraphrases alone.

4.8. Impact of Prompt Pool Size

We investigate the effect of prompt pool size on model performance under three settings: 5, 10, and 15 templates. As shown in Figure 2, using only 5 prompts leads to relatively poor results on MVSA-Single and TumEmo, reflecting insufficient diversity for adapting to complex multimodal inputs. MVSA-Multiple, however, is less affected, likely because its sentiment classification task involves lower semantic complexity. These findings suggest that overly small pools restrict the model’s ability to capture nuanced variations across modalities.

Expanding the pool to 10 prompts yields the best results across all datasets, striking a favorable balance between diversity and relevance. In particular, TumEmo exhibits substantial gains in both accuracy and Macro-F1, highlighting the benefits of richer prompt variation in challenging multimodal contexts. Nevertheless, further enlarging the pool to 15 prompts causes performance drops on MVSA-Multiple and TumEmo. We attribute this to two factors: increased uncertainty in selecting suitable prompts from a larger set, which introduces redundancy and noise, and the higher computational overhead associated with managing more candidates. Taken together, these results indicate that prompt pool size is a critical hyperparameter: in our experiments, a moderate size (around 10) provides a favorable trade-off between effectiveness and efficiency under the evaluated few-shot MSA settings.

4.9. Comparison Across Generative Models

We evaluate the impact of different LLMs used for sentiment-aware text augmentation and prompt construction, comparing GPT-2, LLaMA3, GPT-3.5, and GPT-4o-mini. As shown in

Table 8. Accuracy and Macro-F1 results of different generative models under the revised five-seed protocol. Results are reported as mean ± standard deviation, with bold values indicating the best performance. Significance is tested against the strongest non-best generator for each metric using paired t-tests with $p<0.05$.

Model	MVSA-Single		MVSA-Multiple		TumEmo
	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
LLaMA3	0.6419 ± 0.0148	0.6468 ± 0.0156	0.6435 ± 0.0139	0.5982 ± 0.0161	0.5947 ± 0.0168	0.6012 ± 0.0175
GPT-2	0.5800 ± 0.0184	0.5069 ± 0.0201	0.6215 ± 0.0167	0.5642 ± 0.0188	0.4857 ± 0.0213	0.4200 ± 0.0236
GPT-3.5	0.7360 ± 0.0068	0.7262 ± 0.0075	0.7444 ± 0.0056	0.7313 ± 0.0061	0.6517 ± 0.0094	0.6494 ± 0.0088
GPT-4o-mini	0.7631 ± 0.0033	0.7417 ± 0.0041	0.7584 ± 0.0038	0.7295 ± 0.0045	0.6915 ± 0.0050	0.6402 ± 0.0054

, GPT-4o-mini achieves the best accuracy on all three datasets and the highest Macro-F1 on MVSA-Single, while GPT-3.5 obtains slightly higher Macro-F1 scores on MVSA-Multiple and TumEmo. Its strong semantic understanding and expressive generation capabilities allow it to produce high-quality rewrites, counterfactuals, and verbalizers that enhance multimodal sentiment classification. GPT-3.5 performs comparably on several metrics, although its accuracy remains lower than GPT-4o-mini across the three datasets. LLaMA3 exhibits moderate performance, generally outperforming GPT-2 but still lagging behind GPT-3.5 and GPT-4o-mini, particularly in tasks requiring nuanced semantic control.

By contrast, GPT-2 yields the weakest results across all benchmarks, due to its limited generation capacity and insufficient contextual modeling. These results underscore that the quality of generated data plays a critical role in few-shot multimodal learning. API-based generators such as GPT-4o-mini can produce sentiment-consistent and stylistically diverse augmentations, which may contribute to better downstream performance in the evaluated few-shot settings. These results suggest that generative model choice is an important factor affecting the effectiveness of prompt-based multimodal learning frameworks such as LAPM-RA.

4.10. Hyperparameter Sensitivity Analysis

We conduct a sensitivity analysis to examine how the model’s performance is influenced by two key hyperparameters: batch size and learning rate. As shown in Figure 3, batch size affects both training dynamics and final accuracy in our experiments. Smaller batch sizes allow more frequent updates and better capture of sample-level nuances but may lead to unstable gradients. As batch size increases, performance improves up to a certain threshold beyond which excessively large batches may lead to weaker performance, possibly due to reduced sensitivity to fine-grained patterns. Our findings suggest that a moderate batch size provides a favorable balance between stability and adaptability in the evaluated setting.

Learning rate also plays a critical role in model optimization, as illustrated in Figure 4. While smaller values lead to stable convergence, they can slow training and get trapped in suboptimal solutions. Larger learning rates accelerate convergence but risk overshooting minima or causing instability. Across datasets, a learning rate of 3 $\times {10}^{-5}$ yields relatively stable results in our experiments, indicating its suitability for the evaluated prompt-based multimodal learning setting. Overall, these results show that batch size and learning rate jointly affect model performance, indicating that careful tuning of both is important under few-shot settings.

4.11. Cost and Deployment Analysis of LLM-Based Augmentation

Because LAPM-RA uses GPT-4o-mini to generate augmented text, we further analyze the computational cost and practical deployment implications of the LLM-based augmentation stage.

Table 9. Average generation cost per original training sample for LLM-based text augmentation. Each original sample is used to generate one sentiment-preserving rewrite and one counterfactual rewrite. Token counts and time are reported as sample-level averages over the two generated variants.

Generator	Input Tokens	Output Tokens	Time/Sample (s)	Deployment Mode
GPT-4o-mini	98	72	3.10	API-based
GPT-3.5	98	70	1.45	API-based
LLaMA3	101	76	1.20	Local GPU
GPT-2	96	58	0.35	Local GPU/CPU

reports the average sample-level generation cost. For each original training instance, the augmentation process generates one sentiment-preserving rewrite and one counterfactual rewrite. Therefore, the reported token counts and time are averaged at the original-sample level and cover these two generated variants, while prompt-pool construction and verbalizer generation are excluded because they are dataset-level or class-level operations rather than per-sample costs. For locally deployed models, token counts are computed using the corresponding model tokenizer and are used as a workload indicator rather than an API billing measure.

The results show that GPT-4o-mini introduces the largest preparation cost because it relies on API-based generation and has higher latency than the smaller generators. This is a practical limitation of using an API-based proprietary LLM for augmentation, especially when the number of training samples or generated variants becomes large. In addition, API availability, pricing changes, and model-version updates may affect exact reproducibility over time. Nevertheless, these limitations mainly concern the offline data-preparation stage rather than the deployed classifier. After augmented samples, prompt candidates, and verbalizer tokens are generated and cached, the final LAPM-RA classifier does not require GPT-4o-mini calls during inference. Therefore, GPT-4o-mini affects the preparation cost of augmented training data, but it does not increase the inference-time deployment cost of the trained model. For cost-sensitive deployment, GPT-3.5 or locally deployed LLaMA3 can be used as practical alternatives, although Table 8 suggests that reducing generator capacity may also reduce augmentation quality and downstream performance.

5. Conclusions and Limitations

In this paper, we propose LAPM-RA, a unified framework for few-shot MSA that integrates LLM-based augmentation, reward-adaptive prompt learning, and context-aware semantic fusion. To tackle the challenges of limited data and prompt rigidity, our approach enriches training with both sentiment-preserving and counterfactual samples, dynamically adapts prompt selection through reward guidance, and effectively balances textual and visual cues via adaptive fusion. Comprehensive experiments across benchmark datasets show that LAPM-RA achieves strong and competitive performance under few-shot conditions, with stable gains in Macro-F1 and overall accuracy. Overall, our work demonstrates the potential of combining generative augmentation with adaptive prompting for multimodal learning, offering a practical and extensible paradigm for low-resource sentiment understanding.

This study still has several limitations. First, LAPM-RA relies on LLM-generated augmentation, and the exact generated samples may be affected by API availability, model-version changes, and generator-specific bias. Although we use filtering and quality-control analysis to reduce low-quality generations, hallucinated, biased, or weakly aligned samples cannot be completely eliminated. Second, counterfactual augmentation in this work performs text-centered intervention while keeping the original image fixed. This design provides controlled sentiment variation, but some generated samples may still have imperfect image-text compatibility. Third, the current experiments focus on binary few-shot sentiment settings. Extending LAPM-RA to broader datasets, additional few-shot regimes, and full emotion-label settings such as seven-class TumEmo remains an important direction for future work. We use publicly available benchmark datasets under their original data-release conditions, and generated text resources can be shared only when permitted by the corresponding dataset licenses and API-use policies.